With a new collection of short stories to his name and two of his plays currently showing in New York, the notoriously private Pulitzer prize-winner discusses masculinity, his battle with drink and his 'tumultuous' relationship with Jessica Lange. 

Where do you even begin with Sam Shepard? With his Pulitzer prize? His Oscar nomination? The fact that he's routinely described as "America's greatest living playwright?" Or if you're going to be superficial about it – and I am, just for a moment – maybe the place to start is with the image of him as the tall, taciturn test pilot, Chuck Yeager, the cowboy-ish character he played in The Right Stuff; a man whose life was spent exploring the outer edge of what is and isn't possible.

But then I speak to Patti Smith on the phone and ask her what her impression was of Sam Shepard the first time she met him back in 1970 (shortly before they began an affair), and it's the first thing she says too: "He was just everything that one could want. He was – still is – a very handsome man. And he had this animal magnetism. It was almost visceral. He was so high energy and had a real glint in his eyes. He was born for rock'n'roll. I had no idea who he was when I met him. He was a drummer in a band, the Holy Modal Rounders, at the time and he just had something in him that made him a great, great performer. I just thought he was the future of rock'n'roll. I had no idea that actually he was this great writer too." If you had to invent an all-American literary hero, he'd be something like Sam Shepard. With his slow, western drawl, and his love of the open road and the empty badlands way out west, he's always seemed like the authentic voice of a certain sort of American manhood; telling stories – of suffocating families and wretched lovers – from the forgotten, inbetween places of the American outback. He wrote the screenplay for Paris, Texas, the great, atmospheric Wim Wenders film, and played another cowboy-ish character in Robert Altman's adaptation of Shepard's stage play Fool for Love, fixing an image in the public imagination of both him and a remote, fly-blown America a world away from the metropolises on either coast. But then Sam Shepard is that man. He comes to New York for work but his heart is with his horses back at the ranch in Kentucky that he shares with the actress Jessica Lange, his partner now for nearly 30 years.

All this, then, and a literary reputation that it's hard to overstate. According to Christopher Bigsby, professor of American literature at the University of East Anglia, who I consult on the matter, he's simply the most significant playwright of the past 50 years. His biography groans with accomplishments, he's written nearly 50 plays, acted in dozens of films, directed others, and written the screenplays for still more. And then there's the books about him, the academic treatises on his art, a Cambridge companion to his work, critical exegeses of his themes, analyses of his stagecraft… oh, the list goes on and on.

The one thing he isn't, though, is much of a talker. He doesn't often give interviews but when he does he's routinely described as "taciturn" and "private"; his answers are "curt" or "terse". He's "famously press-skittish". Worse, I read time and again of how he's "notoriously protective of his privacy" and won't answer personal questions. Which is a shame because there are so many personal questions I want to ask him... He's spoken extensively about his relationship with his alcoholic father before, but not about his own drinking: last year he was arrested for driving under the influence and ordered to attend an alcohol rehabilitation programme.

He'll talk about the work but there's nothing I read which gives much sense of him as a man. I can't help but feel a pang for the journalist who asked him if, one day, he might turn their conversation into dialogue in one of his plays. "We're not having a dialogue, this is question and answers," he says curtly. "Dialogue is like jazz. Dialogue is creative.'"

I am prepared for the worst, then, and when he ambles into the restaurant he's chosen near New York's Times Square, it seems this is probably just as well.

How long have we got, I ask, while fumbling with my tape recorder.

"Well," he says sitting down and ordering tea, "that all depends on the questions."

It's a heart-sinking moment and, as it turns out, a completely misleading one. Because it transpires that Sam Shepard isn't actually cold or taciturn or intimidating at all. Or at least the Sam Shepard I meet isn't, because it turns out that there seem to be several different Shepards co-existing side by side.

He's dressed in country clothes – a checked shirt and a nondescript jacket – and, unlike most writers, he has an outdoors complexion; a lived-in face. But what's most noticeable is his sense of humour. It's a lovely, gentle thing; he pokes fun at me, at himself; and when I listen back to the tape, I realise something more shocking still: he doesn't just laugh, and on occasion guffaw, he actually giggles. Sam Shepard is a giggler.

The private, difficult Sam Shepard is nowhere to be seen. Or at least not for a good three hours of tea drinking and conversation that is remarkably relaxed. The restaurant, an unpretentious place he's chosen, is deserted when we arrive. It gradually fills with the pre-theatre dinner crowd, becomes loud and noisy, and has started  to empty again by the time I finally blow it and ask a question too far. Nice, easy Sam vanishes instantly, replaced in a second by cautious, wary Sam. "Oh, he's a very charming guy," Patti Smith tells me. "Very compassionate and thoughtful about other people's feelings. But he's not one for bullshit either."

But then I ought to know something of the idea of two Sam Shepards, existing side by side, because it's how he wrote himself in his most famous play, True West: as two warring brothers, Austin the Hollywood screenwriter, and Lee the desert drifter, two sides of the same Sam Shepard coin, intellect versus instinct locked in an eternal battle for supremacy.

Perhaps the most astonishing thing of all about Shepard's talent is the sheer range of it. He's risen to the top of his field in almost everything he's tried his hand at, but, despite all the diversions, the acting and the directing and the music playing, he is, at heart, a writer. Who simply can't stop writing. Not one but two of his plays are currently playing in New York – Ages of the Moon, a new work, and A Lie of the Mind, a modish revival directed by Ethan Hawke. On top of which, a new collection of short stories, Day Out of Days, has just been published. It's the kind of success that most writers would maim and kill for, although it's largely beside the point, says Shepard.

"The funny thing about having all this so-called success is that behind it is a certain horrible emptiness. All this stuff is happening. And yet it is not what you are after as a writer. Even though they are relatively successful. Ages of the Moon has sold out, the book is doing well, and yet it's not The Thing. And then you're left… there's this feeling… what is it, then? And, I guess, it's the writing itself which is important."

His sheer output is evidence of Shepard's drive to write. He burst on to the off-off-Broadway scene in 1964, writing in his off-duty hours from waiting tables in the Village, enthralling his audience with his exotic tales of the badlands way out west, puncturing the greatest American myths, and he hasn't stopped writing since. It's the process, I say, not the results, that makes you happy?

"Yeah, yeah, yeah. Although happy isn't the exact word. It makes you feel that you're not useless. That you're at least putting your hand in. I think without writing I would feel completely useless."

These days he seems to have it all: as much professional success as he can handle, a long and steadfast relationship, three children, the ranch in Kentucky and bolt holes in New York and New Mexico. And, in some ways, he's the American dream personified: he was born Samuel Shepard Rogers in Fort Sheridan, Illinois, the son of a second world war bomber pilot. As a child he was "Steve Rogers" but after a short stint at college studying animal husbandry he lit off across America, finally landing in New York, where he emerged as "Sam Shepard". His life is the ultimate act of self-creation; he came from nowhere, was little-read and poorly educated, and he turned himself into one of America's leading literary lights.

"And yet still feel so unfulfilled?" he says, and ponders on it for a moment or two. But then anyone with even the slenderest acquaintance with Shepard's work knows that "the American dream" is to be treated with circumspection; in Shepard's universe it's a false concept to be blown wide apart and splattered across all surfaces.

"The great thing for me, now, is that writing has become more and more interesting. Not just as a craft but as a way into things that are not described. It's a thing of discovering. That's when writing is really working. You're on the trail of something and you don't quite know what it is."

He writes on a manual typewriter, and refuses to so much as look at the internet. "I have a cellphone but I have no Google, I have no gaggle."

Really? But everything you've ever wondered, ever, is out there, I say.

"No, no, no! The things that I wonder about most are not on the internet, I promise you that."

He's still, even after all these years, he says, an outsider. "I'm inhabiting a life I'm not supposed to be in… and at certain times in my life I have felt a wrongness. And not a moral wrongness but a sense that this isn't what I was born to be doing." The writers who he responds most to are those who seem to share a sense of "aloneness", and "writing is almost a response to that aloneness which can't be answered in any other way".

For Shepard, the heart of this, seemingly, and a recurring theme in his work, is bound up with the relationship he had with his alcoholic, abusive father. It's there in True West, Fool for Love, Curse of the Starving Class, Buried Child and A Lie of the Mind, and even now, at the age of 66, it troubles him still. In Fool for Love, written almost three decades ago, the main character is haunted by the chilling possibility that he is turning into his father. Back then it was a fear; now, he says, it has become a fact.

"You think about it, you talk about it, analyse it, and then all of a sudden you have become the thing that you were most vehement against. It's very Greek. They invented this shit. Or at least gave it a name."

He's been sober, he says, since the drink-driving incident a year ago. "And prior to that I was sober for four years and then I relapsed. It's a constant struggle. It's such a knucklehead disease because you refuse to see it. It wasn't until the 90s that I actually started going to AA and made a real compact with myself to quit. And I did quit for four years. And then I picked it up again. It's like being a junkie. I think I have that sort of thing in my blood, in my psyche. I can become addicted very easily, although the curious thing is that I have two sisters who are not. So I don't know. Maybe it's just a toss of the dice."

It's the sort of thing a Sam Shepard character might say. In the new book, Day Out of Days, characters wander through the pages, lost within their own lives (one of the most memorable features a man trapped in a public toilet who is literally driven mad when he's forced to listen to Shania Twain on an endless loop). They struggle for personal agency or a sense that they're in control of their own lives.

"And they never are," he says. "That's the one thing about being an author as opposed to being in one's life is that you have the illusion that you can bring some form to it. Which is the beautiful part of it. You don't feel that you are so much in chaos. I don't know what it would be like if I didn't have some form, short stories or plays or whatever."

He feels "blessed", he says, to have discovered writing. "It fulfils something in me that I don't know how I'd serve otherwise." His father was a bright man, the winner of a Fulbright scholarship, a fluent speaker of Spanish, but he never found that outlet. Or at least the outlet he found was drink. He struggled with the return to civilian life after the war, moving his family from airbase to airbase, training as a Spanish teacher, until he was sacked for drinking, and then moving the family to Duarte, California, where he attempted to farm, his drinking increasing year by year. "The alcohol just completely deranged him," says Shepard.

Roxanne, his younger sister, told People magazine back in the 80s: "There was always this kind of facing off between them [Shepard and his father], and it was Sam who got the bad end of that. Dad was a tricky character because he was a charismatic guy when he wanted to be. And at the other side he was like a snapping turtle. With him and Sam it was that male thing. You put two virile men in a room and they're going to test each other."

It's this quality, of a simmering, barely controlled violence that disrupts and distorts all of Shepard's families, that is at the heart of much of his best work. In Shepard's world, romantic love as the meeting of two souls and the family as the nurturing heart of American life are nothing but delusions. "They're wonderful retreats from the illusion of being protected from spinning off the planet. But I don't believe it. And I never did."

So you didn't celebrate Valentine's Day then?

"Oh yes. We just did. I bought her a couple of bottles of wine. I don't drink."

It's not the most romantic gift, I say.

"They were two really good bottles of wine. Really good ones. Oh, and a tape measure. Because she was putting up a painting."

Love in Shepard's universe is never straightforward, never wholly life-enhancing; it's life-destroying, too, a struggle for power or control; a curse as well as a blessing. He and Lange have survived but the relationship was "tumultuous" from the outset. "I mean, we have long periods of relative calm. But then you know…"

But you've always seemed like such an incredible match.

"Yeah, well, we're definitely an incredible match. But, you know, not without fireworks… although at this point, you know, she's the only woman I could live with. Who could live with me! What other woman would put up with me?"

She is, he says, the most honest person he's ever met. "I've never known her, ever, to lie about anything. And I couldn't say that about…"

Yourself?

"About myself. About anybody. Men lie all the time."

Really?

"You don't know that?" he says and raises his eyebrows. "Whereas Jessica has this absolute honesty. I think it's a direct quality of the midwest, of that background that she's from."

While the children were growing up, that's where they lived, in Jessica's hometown in Minnesota, down the road from her mother (and with Jessica's daughter from her relationship with Mikhail Baryshnikov, Shura). It's the equivalent, today, of Brad and Angelina deciding to settle in a suburb of Wisconsin. But then, although Shepard and Lange have both appeared in movies, and been nominated for Oscars – Shepard, one; Lange, six (and she's won two) – they've always refused to be movie stars.

There's a couple of great quotes from Jessica about you, I say.

"Is there? My God. What? Actually, no. Just give me the good ones."

She said: "No man I've ever met compares to Sam in terms of maleness."

"Well, that's a double-edged sword."

Really? I took it as a compliment.

"This morning she had a conversation with me about France, because she was in Paris in the 70s, about the gay scene in Paris, which she was very enchanted with. She was talking about a couple of incidents, and at the end of it I said: 'Well, that's very charming.' And so I think she now thinks I'm a homophobe because she said: 'Asshole!' and stormed out of the room. I thought, 'Oh my God, well obviously I'm not sophisticated enough to talk about the gay 70s in Paris.'"

He was married once before, to another actress, O-Lan Jones. She was 19 at the time, he was 26. Their son, Jesse, was born shortly after the wedding, and then Shepard met Patti Smith. The attraction was instantaneous, as was their affair, an intense, full-throttle romance, conducted mostly at the Chelsea Hotel. It was Shepard who encouraged Patti Smith to become a performer. "She already had this incantatory, lyrical, chanting way of talking, all she needed was a little shove. She was inhibited by not knowing guitar. I said: 'Guitar is just a back-up for your voice. You're not going to be Jeff Beck, don't worry about it. Just learn these chords and you'll be able to back yourself up.' And then it turned out she has this extraordinary voice too."

Reading about the Jones-Shepard-Smith triangle, it all seems very 60s somehow, an amicable bohemian ménage à trois. When I speak to Patti Smith, though, she puts me straight: "It was the early 70s. And it wasn't that amicable."

Shepard had decided to return to his wife and baby. "And it was painful," says Smith. "We knew it was going to end and we were in a room at the Chelsea Hotel. And instead of sitting around and moping, Sam said: 'Let's write a play.' And I said: 'I don't know how to write a play.' And he said: 'I'll be one character, and you can be the other.' And we just sat up all night, him writing a line and then pushing the typewriter across the table to me, and then I'd write a line."

The result was Cowboy Mouth, which opened at the American Place Theatre with Sam Shepard and Patti Smith playing themselves, in a double bill with Shepard's play Back Bog Beast Bait in which O-Lan played a character based on Patti. It was too much, and without warning, Shepard quit, and fled with O-Lan and Jesse to London.

There are so many of these ruptures in the story of your life, I say to Shepard. You're doing one thing and then suddenly you're doing something else.

"I know. I don't why it had to be so traumatic. It very definitely felt like these were earthquakes when they happened. They're terrible and yet on the other side of the coin they're ecstatic. Like when I met Jessie. It was terrible leaving my oldest boy at that time. He was 13, which is a really hard age. And, in one way, I can't forgive myself for that. And, in another way, I'm glad of the life that I've had with Jessie. What's the trade-off? It's always felt like that. The other thing that's kind of amazed me is that I've had absolutely no qualms about setting off into unknown territory. I've never been afraid to just start something new."

It was on the set of the film Frances that he met Lange. I tell him that one critic I read claimed that after meeting Jessica his depiction of male-female relationships became more complex and interesting. He says that you started writing meatier parts for women.

"Hmm. I guess that's true. Fool for Love came out of my relationship with Jessica and that's pretty powerful."

Fool for Love features a tumultuous relationship between two characters, Eddie and May, who both attract and repulse each other. And who, it turns out, are half-brother and sister.

I was looking at photographs of you and Jessica next to each other and I was struck by how similar you look, I say.

"We do, kinda."

Is the theme of incest in Fool for Love in some way borne out of that?

"I'm sure there's something about that. I'm sure when you're looking for someone, you're looking for some aspect of yourself, even if you don't know it... What we're searching for is what we lack. You lack something and your hope is that it'll be fulfilled by who you find."

His relationship with his father has had such a profound effect upon his life, his work, it's inevitable that he must have reflected upon his own effect upon his children, Jesse, 39, Hannah, 24, and Samuel Walker, 22.

He hesitates when he replies. "I would like to think… you can never determine how you are going to influence someone, particularly your children. I mean, they are all musicians in some way or another, so I feel as though… I think that's a result… And my daughter is also a really good writer. Really good."

The thing about your children compared to you, I say, is that they had a very stable…

"Stable?"

Oh, is that the wrong word?

"Well, relatively stable."

They haven't had the childhood that you had...

"They haven't had an abusive childhood. On the other hand, they have a different set of problems."

Having a father who is very successful…"

"And a mother," he says. "Yeah. There's a lot of stigmas. My youngest boy is very, very shy. He doesn't want anything to do with celebrity. And my daughter, she's not crazy about it. None of them covet fame."

He shies away from speaking about his sons but he seems happy enough to talk about Hannah, his daughter, currently studying for a PhD at the University of Galway.

"I never thought about having a daughter and then I had a daughter and it was a remarkable thing. It was very different from having a son and your response to it. With a son, it's much more complex. And it's probably because of my stuff in the past. With a daughter, I was surprised at how simple it is."

It's to her, he says, that he intends to leave his notebooks, "because she's the one who's asked for them."

He's obsessed with his notebooks, he says; they travel with him wherever he goes, "like gremlins". And he fishes his current one out of his coat and shows it to me. On the inside back cover he's written the places it's been to with him over the year – Sicily, Kentucky, New Mexico – and then he flicks through the pages and says, "Look at this! Look at these drawings." And he shows me some stick men, riding the sort of horses I drew aged eight. "You know, I was sitting in the University of Texas where they have the original manuscript of Watt by Mr Beckett and it was amazing because there were all these drawings on them, so I sat there one afternoon and copied them!"

It's almost as if Sam Shepard has spent his life circling around Samuel Beckett. It was discovering his plays as a young man that first inspired him to write, and Patti Smith says that in those days he never went anywhere without a copy of one or other of his plays on him. "Of course, now he's read everything. He's always discovering something new, whether it's Japanese death poetry or some new Venezuelan writer or whatever."

Not meeting Beckett is his greatest regret, he says. "My greatest literary regret."

Do you think you're starting to look like him, I say, tongue-in-cheek, although there's an element of truth to it; he's still recognisable from his cinematic glory days but his face is craggier now, crisscrossed with experience. He guffaws, enjoying the joke.

"No! It'd be flattering if I did but I think my features are a little bit more savage."

Themes of regret and remorse, of time passing and humans ageing have started to creep into his work. "I don't believe people who say, 'I have no regrets'. How can you not have regrets?"

Death, he says, changes all perspectives. When I ask him how old his father was when he died, he replies immediately. "A year older than I am. He was 67."

Does that weigh on you?

"I think about it. But it doesn't weigh on me because of the way in which he died." His father was run down by a car while drunk. "So I don't worry about it that way. I don't worry about the way I'm going to die…

But do you think about death?

"Yeah. There's not a day goes by. But that has always been the case. We're all haunted by it in one way or another. And it's the easiest thing in the world to push it away, you just get a cappuccino. But, yes, you're haunted by it in a different way [as you get older]. I feel its presence. I feel it in sleep, in dreams, in waking. Particularly in the morning."

Do you think about the things that you would lose?

"No. You feel that you're diminishing in some way. You feel that your senses are diminishing. I don't see as well. I'm not as quick as I used to be. Things like that. Knock on wood, I'm not sick. I don't how people deal with that… I mean life is tough enough. And now you're going to die! Wow!"

In Ages of the Moon his central character, Ames, has been unfaithful to his wife. "She discovers this note, this note from this girl, which to this day I cannot for the life of me remember," says Ames. "Some girl I would never in a million years have ever returned to for even a minor blow job."

"Minor?" asks his friend, Byron.

In his earliest plays, Patti Smith says, his characters had to act. "They had to do something, kick a door down or whatever. Now they tend to be more introspective. They're more likely to examine what they're doing and why."

And Shepard too. His life is in his plays, he's always said that. And so I ask him. About Ames's infidelities. About whether that's been a source of regret for him too.

"I'm not going to talk about that. You're not going to sucker me into that one! When did you think I was born?"

Oh dear. It's a classic interview mistake: the question too far. He's amicable enough, and we carry on for five or so more minutes, but I've got the other Sam. He looks the same but I can tell he's scanning the horizon for an escape route; it's Sam Shepard, the cowboy, the character in all his plays; the desert drifter, shifty, cautious, suspicious of strangers. The giggles are over. And then he's gone, with the briefest of handshakes and a rush to the door. It's not an entirely inappropriate ending. Shepard's world is a place of blundering people and blundered words; where plots are never neatly tied up and truths are only ever hinted at, never fully revealed, least of all to the characters themselves.