Sam Shepard made his name unpicking the American psyche in Fool for Love. As the play opens in London, the author talks to playwright and fan Joe Penhall about writing, jazz and his father's alcoholism

Fool for Love is the play that, for most people, defines Sam Shepard, typifying his fascination with the American west and spawning the cartoon image of him that he lampooned as "Mr. Artistic Cowboy". Perhaps it's because this deceptively simple 1983 play, in which a torrid love affair in a cheap hotel in the Mojave desert morphs into incestual yearning, distils much of what intrigued him up until then, and probably couldn't be bettered. It may also be that the film version, fancifully directed by Robert Altman, starred Shepard himself, authentically menacing as the cowboy stuntman protagonist.

Somehow it makes no difference that Shepard has more or less avoided cowboy country in the theatre ever since. Anxious not to repeat himself and wary of that enduring perception of him, he is now the author of more than 40 plays. How ironic, then, that he is still defined in such narrow terms.

"It's strange," he says, "because you're only using this material as the springboard to something else entirely. The thing you're really after is what's going on between the characters underneath it all. I'm not trying to write a particular vision of the west, I'm just using it because I grew up there and I like the place. I'm not interested in making philosophical or political statements about it - I'm interested in making characters happen in their element and allowing them to be."

So while it is a thrill to see the play in a new West End production, with a nuanced performance from Juliette Lewis, it seems a pity that commercial theatre producers overlook Shepard's most interesting work, gravitating instead toward the flashily iconic, more easily digested Americana that is easy to "cast up" with stars - a process he defines as "boring and stupid".

Shepard is my favourite playwright, and his plays are probably the reason I started writing. When I don't know why I'm doing it any more, it's Shepard I turn to. People have remarked on the influence of Pinter and Mamet on my work - because everybody admires Pinter and Mamet - but for me, Shepard's always been the boss, no question.

Read "La Turista" about two bewildered tourists in Mexico, or "Silent Tongue" about native American Indians, or "A Lie of the Mind, " with its damaged female protagonist affecting a European accent, and you will begin to understand the eclecticism of the real Sam Shepard. The product of an impoverished, peripatetic childhood, moving as a teenager from Illinois to southern California, with frequent sojourns across the border to Mexico, he dropped out of agricultural college and ran away to New York with plenty of spleen to vent in plays that embraced mainstream American culture and gave voice to those outside of it.

"To me it was just the condition of being an outsider," he says, "being literally, geographically from somewhere else. I don't mind being part of the mainstream, but there was nothing to be part of. I guess there was an anger about that, but at the same time I'm glad about it - it gives me my vantage point as an artist."

We meet for lunch at Pastis, the fashionable grill in Manhattan's meat-packing district, on a baking afternoon, his beloved Sonny Rollins blasting from the sound system as if in his honour. At 6ft 3in, with crooked teeth and big, raw-boned hands, he is not someone you would want to mess with. But as he apologises profusely for his lateness (he was writing and forgot the time) and peruses the menu through scholarly wire-rimmed spectacles, he seems more like the Pulitzer prize-winning New York intellectual he otherwise is.

At his best, Shepard articulates the strange divided self that is the American psyche, born of fear and puritanism, the progenitor of exciting, decadent, secular iconography, awash with violence, cruelty and destructiveness, yet continually yearning for spiritual release.

"There's a kind of madness that exists in the American psyche," he offers. "The American culture is intrinsically violent. It's in the constitution, the right to bear arms. To me, that's where the madness comes from. My ancestors came over on the Mayflower. The very first white kid born in the Plymouth colony was my ancestor. He was called Peregrine White - it's a hawk, a peregrine falcon."

Shepard's plays are widely referred to as "mythic", and the myth with which he has accomplished the most is that of the American family. It is savagely repudiated in Shepard's trilogy of "Curse of the Starving Class," "Buried Child" and "True West." In these disturbing plays, the family is the seat of madness, alcoholism and infanticide. Shepard's own disillusionment with his upbringing allows him to penetrate the hypocrisy of a nation that is intrinsically violent and coercive, yet peddles the cosy cliche of "family values" at every opportunity.

These plays are part of a continuum of work in which Shepard has explored the character of his complex, contradictory father. Originally from a farming family in Illinois, Sam Shepard Rogers joined the air force as a fighter pilot during the second world war. When he returned, a haunted man, he took to the bottle and embarked on various doomed enterprises until finally he drank it all away.

"He was a brilliant guy," says Shepard. "He was a language teacher for a while, he spoke fluent Spanish, he was a Fulbright scholar, but he used to systematically self-destruct. All along his dream was journalism. Before the war he went to Chicago to try to break into the newspaper business but then the war broke out. I remember being in sixth grade and my dad driving me to school one day and he told me about this. It's strange, isn't it?

"I grew up in an incredibly unstable environment. Very violent, very crazy, where you're always threatened and you grow up gun-shy. I left home under a great deal of duress at the age of 18 and severed the connection for five years. I eventually got in touch with my mother, but my dad was drinking perpetually by that stage. He'd gone off to Mexico on his own and he was pretty much out of the loop. It takes a long time to get over that. You never get over the feeling that you're under assault or under siege."

Shepard rarely talks about his father, prefering to let the plays speak for themselves. In True West, his funniest and most accomplished play, he lampoons his family's dysfunction and his own artistic pretensions, as two very different brothers fight about their father's decline into alcoholism, knowing it is possibly their own destiny. (Intriguingly, Shepard has no brother himself, only sisters.) One brother, well-heeled scriptwriter Austin, has retreated into his lucrative work, appalled by the squalor. The other brother, Lee, is a trailer-trash petty criminal who himself lives in squalor in the Mojave desert, in thrall to the old man. Each brother yearns for the perceived freedom of the other, and when a Hollywood producer commissions Lee and not Austin to write a cowboy screenplay, Austin mordantly observes: "He's lost his mind. He thinks we're the same person."

"Buried Child," in which city boy Vince returns to his rural family with his uncomprehending girlfriend, won Shepard a Pulitzer prize in 1979. His father "was actually working in a bookstore somewhere when he discovered I'd won a Pulitzer. He wrote me a note saying, 'I don't understand what the hell you're writing, I can't make hide nor hair of this shit, but I have to congratulate you nevertheless.'"

Not long after, Rogers was killed lurching across a Mexican highway in the arms of his beloved alcohol. The event haunts Shepard's 2000 play, "The Late Henry Moss" - an often painful work in which Shepard's sense of humour failed him, or was abandoned. It is a harrowing, nightmarish vision of loss concerning two brothers arguing about their father's suspicious death, while the horrified, booze-sodden father wakes up on a slab, not believing he really is dead. It is about his father, of course - but more than anything it is about the death of the spirit.

Shepard's work harvests such ugly personal experiences that I wonder how he can bear to offer it up for public consumption. "I don't think you can do anything if you're right slap in the middle of it. The only successful thing I've ever seen that does that is O'Neill's "Long Day's Journey into Night. " The only way I can do it is to step a little bit away, and the characters somehow become human. If you have enough distance that it becomes funny, if you can illuminate the humour, even though what they're going through is very raw, then it's possible. That may be why The Late Henry Moss is a little hard to take. If the humour drops away then it just becomes deadly - so deadly and remorseful that it's hard to enjoy. What humour brings to it is the 'humaneness' of it."

Shepard's detractors complain that he continues to write about the same obsessions - as he puts it, "fathers and sons". Others would prefer it if he stuck just to that. Either way, his refusal to engage with the "issues" of the day and his insistence that politics is "boring" have cost him. In 1990, he wrote "States of Shock," which, like last year's God of Hell, was wrongly perceived as an anti-war play and was ungratefully received by critics who dismissed him as a superannuated 1960s radical.

"How long does an issue last anyway?" he counters. "It has a life span that's very short. The Cuban Missile Crisis is gone, nobody talks about that any more. I'm not interested in writing political dogma plays and I don't consider "States of Shock" or "God of Hell" as necessarily political plays. They're investigating a certain strain of American fanaticism. If you want to associate that with the right then that's fine, but it could easily go in the other direction. There were certain aspects of the Clinton administration that I thought were ridiculous - like his non-involvement in Bosnia. I thought it was outrageous to let people slaughter each other because of his pacifism, when he could have done something about it."

These plays, Shepard insists, relate to his father as well. Thanks to him, "I understand the military mentality implicitly. It absolutely believes in violence and exists in order to enforce its will. "States of Shock" just happened to coincide with the first Gulf war. I don't see it as doing anything other than what I was trying to do all along. God of Hell was inspired by Ubu Roi, this so-called avant-garde play about a monstrous, devouring, cannibalistic maniac who swallows up everything. It's not about the Democrats or the Republicans, that's boring. They're all businessmen one way or another, fuck all of them. I don't vote. I write plays."

It was in New York in the 1960s that Shepard first defined himself as a playwright. What influenced him, he says, was jazz. He was working as a busboy at the Village Gate, where the greats of the era played every night. "I saw Sonny Rollins at the Five Spot in 1964 with Eric Dolphy and Mingus," he says. "Sonny had a mohawk haircut and shades, a big guy, he blew the walls off the place. Dolphy to me was a genius - he goes to places I don't even think Coltrane went to. You see as far as influences go, I look to other forms.

"Jazz seems like theatre to me. Musicians standing up in front of you is a vital thing. You felt that you were in the presence of something real. An exchange was going on."

Writing loquacious, freeform short plays that mimicked the improvisational structure of the music, Shepard was lauded by Edward Albee and became the new crown prince of the avant garde. One of his most significant achievements is to write about class with the elegance of Harold Pinter, Joe Orton and Arnold Wesker - whom he greatly admires - while articulating America's own unique relationship with it. And this tension between rich and poor, educated and uneducated, thinkers and bullies, the free and the oppressed, could be attributed, too, to his love of jazz.

"All of these jazz clubs were white and all the performers were black. You could feel the knives being thrown, very aggressive and antagonistic from the musicians - and the white audience would just sit there, really appreciating it. I found it somewhat inspiring. There's no doubt that I felt a lot of anger about things in America then. There was a pissed-offness about the 60s that gets covered over by flower power now, but it was an angry time."

His other great inspiration was rock. He has collaborated with Dylan, and in 1971 wrote and performed the play Cowboy Mouth with his then lover, Patti Smith, with whom he is still friends. "Her writing blew me away," he says. "She's a really great poet and she wasn't even singing back then, she was just doing poetry readings. I said: 'Y'know, you can get up there and do this poetry reading and make like you're a rock star - or you can actually be a rock star.'"

Eventually, drugs, drink and his consuming affair with Smith took their toll. "I kind of wore myself thin here in New York through the 60s and I needed to change and learn," he says. "What I really needed was to change the way I was doing things, not the environment - but, of course, I thought it was the environment."

In 1972, Shepard escaped to London for a couple of years - and it was then that his writing took a quantum leap forward. He was asked to direct his friends Bob Hoskins, Stephen Rea and Kenneth Cranham in his first major play, "Geography of a Horse Dreamer," at the Royal Court. "It was a great experience, which changed the way I approached writing, because I started writing for the actors. I'd shape things and move them around and realise what was possible or not possible. You understand the rhythmic structures much better when you work directly with actors. It's like a piece of music, it shifts and changes."

Hemingway said that writing is all about rewriting and, having directed, Shepard learned to rewrite. "With the early stuff I never rewrote anything. It was the arrogance of youth. 'Fuck it, if you don't understand it, I'll just write another one.'

"I was riding those plays, like you'd ride a horse. You'd go as hard as you could, then get off and get on another one and go again and not really give a shit about how you're riding them. You'd just leave them wet and go and get on the next one. But then I started to become interested in the form and what was possible. How you have to pay attention to certain details of language, of structure, of rhythm, in order to make a play that wasn't just opening a window and letting wind rush through the room. Now," he concludes, "I love rewriting. I go over and over and over stuff - it's become a fetish."

After directing two underrated independent films ("Far North," "Silent Tongue") he decided it "isn't my medium", and has turned instead to prose. At the moment he is working on another collection of short stories, following his well-received volumes "Hawk Moon" and "Motel Chronicles," "Cruising Paradise" and "Great Dream of Heaven." Unusually for a playwright, Shepard is a surprisingly strong prose writer. His short stories remind me of Richard Ford's in their muscular, ironic turn of phrase; Ford calls Shepard's latest collection of stories "irresistible".

For the past two years, Shepard has done most of his writing in New York, periodically forsaking his 107-acre Kentucky ranch. "The only thing I really like about New York is that it forces me to write. The claustrophobia actually helps the writing. I retreat from it into this work attitude and I get a lot of work done."

He is notoriously protective of his privacy, hating to talk about his family, but he credits Lange, whom he married just after his father's death in 1984, with bringing him stability, and clearly dotes on his children.

"You know, Flann O'Brien has this great quote," he offers: "'I am my own father and my son.' There have been moments when I've been with my oldest son, Jessie, where I feel that he's the father and I'm the son. Or he's my brother, you know what I mean? These things are tenuous to say the least."

In 1984, Shepard was nominated for an Oscar for his performance as the astronaut Chuck Yaeger in Phil Kauffman's "The Right Stuff." But he is suspicious of Hollywood, worried it might coarsen his writing talent, and freely admits that these days he only acts when he needs the money. He has written more about the difficulty and joy of the creative process than any other playwright, but he's not one for the suffering artist cliche.

"It's just a way of life and I think it's useless to spend much time indulging in the pain and ecstasy of it. Just do it. I've discovered a lot more about how to do it and how to free yourself up now. You get tied up in so many different bundles and when you unwrap one of them all of a sudden a whole lot of stuff starts to spill out. It's thrilling, but it's almost like you have to trick yourself into the freedom of it. It's not enough just to want to write. There's something else that occurs that that allows you the freedom to really write for yourself. And when that happens, it becomes a revelation. It becomes something you're glad to be involved in".