Thelonious Monk, the pianist and composer whose wry, angular melodies and unusual harmonic progressions are among the most striking contributions to the jazz repertory, died yesterday in Englewood Hospital in New Jersey at the age of 64. He had suffered a stroke on Feb. 5. Although Mr. Monk's music was rooted in the stride-piano tradition of Willie (The Lion) Smith, James P. Johnson and Duke Ellington, it stood apart from the main flow of jazz. ''He hasn't invented a new scheme of things,'' Paul Bacon wrote in the jazz magazine The Record Changer in 1948, ''but he has, for years, looked with an unjaundiced eye at music and seen a little something else. ''He plays riffs that are older than Bunk Johnson but they don't sound the same. His beat is familiar but he does something strange there, too. He can make a rhythm almost separate, so that what he does is inside or outside it. Monk is really making use of all the unused space around jazz, and he makes you feel that there are plenty of unopened doors.'' Randy Weston, a pianist who studied with Mr. Monk, has called him ''as complete an original as it is possible to be'' and he cites the unifying ''simplicity'' in his music. ''Not that his music isn't often complex to execute,'' Mr. Weston explained, ''but it always comes through so clear and accurate, so uncluttered. His music is simple in the sense that it has totality of personality. It's all him.'' Among his works were ''Round Midnight,'' ''Straight No Chaser'' and ''Well, You Needn't.'' The strange contours of Mr. Monk's tunes led the jazz critic Whitney Balliett to describe them as rippling ''with dissonances and rhythms that often give one the sensation of missing the bottom step in the dark.'' When Mr. Monk was a guest at a jazz class at Columbia University, the lecturer turned to him and asked if he would ''play some of your weird chords for the class.'' Mr. Monk bridled. ''What do you mean, weird?'' he asked. ''They're perfectly logical.'' ''Jazz,'' he said on another occasion, ''is my adventure. I'm after new chords, new ways of syncopating, new figurations, new runs. How to use notes differently. That's it. Just using notes differently.'' His way of doing things differently -but, as he insisted, logically - extended beyond playing and composing. During a performance with his group, he often got up from the piano and went into an intense shuffling dance. This was his method of conducting. He might be setting a rhythmic pulse to move his musicians into the rhythm he wanted or simply checking to make sure that the performance was swinging properly. Mr. Monk once explained the difference between himself and the bop musicians with whom he was often associated in the 1940's: ''They think differently, harmonically. They play mostly stuff that's based on the chords of other things, like the blues and 'I Got Rhythm.' I like the whole song, melody and chord structure to be different. I make up my own chords and melodies.'' Mr. Monk was a man of unwavering musical principles, strong opinions and sardonic wit. When a saxophonist, Sahib Shihab, told him that he was going back to the Juilliard School to study, the largely self-taught Mr. Monk remarked, ''Well, I hope you don't come out any worse than you sound now.'' Thelonious Sphere Monk, whose father was also named Thelonious, was born on Oct. 10, 1917, in Rocky Mount, N.C. When he was 4, his mother brought him and two other children to New York, but his father remained in the South. In the early 1930's, the family moved into a small apartment on West 63d Street in the area known as San Juan Hill, where Mr. Monk continued to live until a few years ago, when the building was torn down. He began playing in bands when he was 13. Four years later he joined a group that traveled with an evangelist (''She preached and feared and we played,'' he said). When the troupe reached Kansas City, Mary Lou Williams, the pianist, who became one of his close friends, heard him and reported later that ''he was playing the same style then that he is now.'' In the late 30's and early 40's, Mr. Monk was the pianist in the house band at Minton's Playhouse in Harlem, where young and then unknown musicians such as Charlie Parker, Dizzy Gillespie and Kenny Clarke came to jam. The music that became known as be-bop developed out of the sessions. Mr. Monk, however, was not part of the be-bop movement. He went his own way. To the public who found the ''boppers'' strange, he was even stranger and more difficult to accept. His first slight brush with success came in 1944. Bud Powell, the pace-setting pianist of the 40's who had been influenced by Mr. Monk, persuaded Cootie Williams, in whose orchestra Mr. Powell was playing, to record Mr. Monk's tune ''Round Midnight.'' In the same year he made his first records with a group led by the saxophonist Coleman Hawkins, who, unlike most jazz musicians who came up in the 1920's, was interested in the new jazz developments of the 40's. Mr. Monk's first records under his own name, for the Blue Note label in 1947, gained some attention in black areas of large cities. But white listeners tended to reject him because, as Alfred Lion of Blue Note Records pointed out, ''They thought he lacked technique.'' He was getting only occasional work in the late 40's, and even that was cut off completely when, in 1951, he and a friend were arrested for possession of narcotics. Although Mr. Monk was known to be ''clean'' and was widely considered to be innocent in this case, he took the full blame, refusing to be what he called ''a drag'' by implicating his friend. He served 60 days in jail but, what was much worse, he lost his cabaret card, without which he could not perform in a New York club. For the next six years, until 1957, he recorded occasionally. Most of his time was spent composing many of the tunes that became the core of his repertory, including ''Bemsha Swing,'' ''Blue Monk,'' ''Little Rootie Tootie'' and ''Pannonica,'' named for Baroness Nica de Koenigswarter. The Baroness had befriended several jazz musicians of the period, including Charlie Parker. In 1957, with the help of the Baroness, Mr. Monk regained his cabaret card and started to play at the Five Spot in New York. The club became a steady base for him. During the late 50's, he led a quartet that included John Coltrane, the saxophonist. In the 1960's, Mr. Monk finally gained the recognition that had eluded him for almost 20 years. He worked regularly with a quartet featuring Charlie Rouse, the tenor saxophonist, appearing in clubs and at concerts and festivals all over the world. He also gave concerts with large orchestras, playing his compositions at Town Hall in 1959, at Philharmonic Hall in 1963 and in Carnegie Hall in 1964. In the 1970's, his public appearances became infrequent because of illness. His last official performance was at Carnegie Hall in 1976, although he sat in informally one night after that at Bradley's on University Place. Mr. Monk, who lived in Weehawken, N.J., in recent years, is survived by his wife, Nellie; a son, Thelonious Monk Jr., who is a drummer, and a daughter, Barbara. A funeral service will be held Monday at 11 A.M. in St. Peter's Church, Lexington Avenue at 54th Street. The body will be at the Walter B. Cooke Funeral Home, 85th Street and Third Avenue, Saturday from 4 to 9:30 P.M. and Sunday from 11 A.M. to 9:30 P.M.