"It was an extremely dull grammar school, and I can't remember a single stimulating thing about Canterbury." Is Robert Wyatt, former Soft Machine drummer, singer and songwriter, being disingenuous?

For any English schoolchild suckled on the avant-garde "Canterbury sound" of the 1960s - the jazz-folk-pop fusions of Soft Machine, Caravan, Egg, Gong, Hatfield and the North and Kevin Ayers' Whole World - the Simon Langton school in Canterbury was a kind of English Haight-Ashbury, a happening centre of all things mystical, mythical, anti-establishment, acid-laced, and musical.

In their dreams, man. It was no such thing. Not even close. Yet, it was here, amid the chalk and gowns, that a nucleus of talented, literate and artistically inclined young musicians emerged to produce a sound that, while never likely to challenge Phil Spector's wall of sound in terms of commercial success, much less the Mersey sound of the Beatles, continues to haunt the byways and backwaters of British music.

Kevin Ayers, born in 1945 in Herne Bay, Kent, was one of the people drawn into the intimate circle of well-bred Canterbury schoolboys - Robert Wyatt, Mike Ratledge, Hugh and Brian Hopper - who happened to meet the real thing, Daevid Allen, in 1961. Allen, an Australian, had worked with the waywardly brilliant writer William Burroughs, author of The Soft Machine (1961), in the US, and in Paris with the avant-garde Californian composer Terry Riley, whose minimalist masterpiece In C was to be a huge influence on Steve Reich and Philip Glass, as well as Ayers's future British band, Soft Machine.

Allen had long hair. He had dropped acid. He recorded music in his houseboat on the Seine. He must have seemed amazing to a group of artistic, middle-class schoolboys told to get their hair cut and do their ties up.

Ayers was easily assimilated into this magic circle. The son of Rowan Ayers, a BBC producer, he was partly brought up in the Far East by his stepfather, a district officer in British Malaysia. In Canterbury, just a few miles from Herne Bay, where colonial officers went to retire, he discovered his latent musical talent with this group of Canterbury schoolboys jamming in the graceful Georgian house that belonged to Robert Wyatt's mother, Honor, where Daevid Allen rented a room.

Here they listened to the latest jazz while coming to terms with Dadaist art and surrealist poetry. At first Ayers thought the music "weird" and even "utter gibberish", especially played on cutlery and kitchen instruments. But, gradually, and with Wyatt drumming, Hugh Hopper on bass, his elder brother, Brian, playing lead guitar and saxophone, Richard Sinclair on rhythm guitar and Ayers singing, The Wilde Flowers, Ayers' tribute to Oscar Wilde, were formed in June 1963.

In Deal, not so far from Canterbury, I listened, last month, with an enthusiastic if never exactly wild flowering of dedicated Ayers supporters, Soft Machine aficionados and a budding generation of sixth-formers and college students, to Kevin Ayers material, drawn from a back catalogue of four re-released EMI CDs - Joy of a Toy, Shooting at the Moon, Whatevershebringwesing and Banana Amour.

Ayers is too self-deprecating, too elusive and knowing to ever have made it big. Yet his songs are special, some catchy and weird, others poppy and even rocky, yet others gone with the wind, ethereal, surreal, the stuff of early drafts of poems better sung than confined to print. They are delightful, but impossible to pin down.

"I would have made a very unlikely star with a voice like mine," he says in his rich, upper-middle-class baritone, in his house near Carcassonne in southern France. "I mean, a public school rocker with a plummy BBC accent... hardly."

And, yet, one of Ayers' first demo tapes includes a Lennon/McCartney-style number, She's Gone, proving that he could have cut it with the pop stars. "I did want to make some money. Not that I ever did," he says, "which is why I'll have to make some more records and go on tour again, but I get terribly bored with all the travelling, the hotels and the general waiting around. And, while a bit of recognition and celebrity is nice, a little goes a very long way."

Ayers tripped off to Ibiza with Daevid Allen. He has always liked sun, sea and freedom. And, the girls. "The girls, today, are my two daughters, a couple of wives in the past... and I've just bust up with my latest lady, sadly; not that I intend to write angst-ridden songs about it. You get past that phase. It's very charming to see the Rolling Stones banging on about teenage love - ooh, baby, yeah - but I'm afraid it's not quite my bag."

Freedom was something he felt denied him when The Wilde Flowers metamorphosed briefly into Mister Head and then Soft Machine, with Wyatt, Ratledge and Allen, in the summer of 1966. Ayers and Allen had met the Texan millionaire, Wes Brunson, who gave them the money to buy new equipment and a rehearsal studio near Canterbury. They formed Soft Machine after Allen had phoned Burroughs in the US to get approval for the name.

Soft Machine's music was a rainbow of sounds and songs drawn from gamelan to pop, via jazz and Terry Riley's minimalism. There was nothing quite like it. They played on the same bill as Pink Floyd at the International Times launch party in October 1966 and became regulars at the UFO club on London's Tottenham Court Road in the spring of 1967.

They were, along with the likes of Pink Floyd and the Nice, one of the underground bands of the moment. And, like most head-expanding underground bands, they were met with baffled incomprehension and even hostility outside London. "I only ever walked offstage once", says Ayers. "It was when the beer bottles started flying. Not my scene." The band moved to France where they were welcomed as Dadaist heroes and played venues like the Museum of Contemporary Art in Paris.

Allen was refused entry back into the UK that summer. He gave up Soft Machine, stayed in Paris and formed a new band, Gong, and later the Invisible Opera Company of Tibet, the Magick Brothers and, eventually, back to Gong again. Performing as a trio in London, Soft Machine was much admired by Jimi Hendrix who asked them to tour the US with him. "He thought we were terribly cute," says Ayers, "and absolutely no threat to his sensational stage presence. He was quite right."

The six-month schedule was punishing, especially as the band was writing and cutting its first album, Soft Machine Volume One. At first, Ayers went for the rock 'n' roll lifestyle hammer and tongs. "I was completely drunk with the whole thing. Girls lining up outside the door, free drink everywhere". But he soon tired of it.

"I went on a very strict macrobiotic diet and I didn't go out partying. I became alienated from everything that was going on around me - because of the violence and extremity of it. At its worst, it was plane, hotel, gig, hotel, plane, hotel, gig. Mike Ratledge and I would just stay in. He read books, while I used to lie on the floor and stare at the ceiling. At the end of 1968, I couldn't take it any more."

The sex, drugs and booze were one thing, the rock'n'roll, or what passed for it in Soft Machine's world, another. "Mike and Robert were far more musically literate than I was, and I think my simplicity bored them. They were going more in the direction of jazz and fusion, which didn't interest me. I was strictly pop. They were into what I consider to be self-indulgence. It's stuff you play for yourself, and 'fuck the audience'... so I took my simplicity elsewhere."

In 1970 Ayers formed the Whole World, a particularly eclectic band that included a teenage Mike Oldfield on lead guitar. Oldfield's demos of Tubular Bells were made on Ayers' tape-recorder. As Oldfield limbered up for global stardom, Ayers left for France to play with Daevid Allen's Gong.

From then on, like some gentle Ancient Mariner, Ayers has played and recorded for whoever will stop for him or whenever the mood takes him. He kept away from Britain during the punk explosion, recorded a strangely conventional rock album, Diamond Jack and the Queen of Pain, with local musicians in Spain, and the lovely Falling Up, with songs of wine and love, made in Madrid in 1988.

"The music executives were putting some pressure on, which is why I ended up making a couple of records that sound like anyone else's. My real records have always been messy things, but the industry doesn't like this. Nearly all best-selling albums have a consistent sound to make them easy to position and market. Well, I tried, but my heart was never in it. I think you have to have a bit missing upstairs, or just be hungry for fame and money, to play the industry game. I'm not very good at it."

Falling Up got Ayers back onto the British stage, but it was the acoustic Still Life with a Guitar (1992) that saw him happily on the road in Britain, the US and Japan. Playing small venues, he has found new audiences, and especially when he recorded Hymn with Ultramarine, whose techno-ambient sound gave us a Kevin Ayers for the DJ generation, as did his later tours with the Wizards of Twiddly, a band that spawned, among others, the Coral.

"Well, I suppose I must get down to making that new record," says Ayers, a charming, quietly spoken and intelligent man working on the fringe of an industry that thrives on hype, volume and the selling of sensitive souls. A soft machine in a hard world, it is little short of a miracle that this civilised chap is still charming us with songs that come as much from the brain, and art, as a heartful of soul.