Herring once grew a Hitler-style toothbrush moustache and recorded the reactions, in a dedicated exploration into prejudice, or possibly just an attempt to see how many times he could get hit in the face in the space of one month (he describes it to me as, "a genuine experiment. In madness"). He certainly enjoys courting controversy, but his main motivation seems to be a real desire to ask questions and leave the audience to debate the answers. At his gigs he asks probing questions like what is love, why do we respect the dead but not the living, and how large is the average penis size? His thirst for knowledge continues in We're All Going To Die! his latest show about death which plays at the Leicester Square Theatre from October 8-13.

"Death seemed like a big enough subject to take on," he explains, "I've always been slightly obsessed with it since I was a little kid, not in a macabre way, just in a, 'What if I died?' worried way."

I meet Herring at the Bush Theatre in Shepherd's Bush, near the home he shares with his wife, fellow comedian and children's author Catie Wilkins. He's densely built, solid and short, with sparkling blue eyes. We sit on a bench in the sunny garden. He seems very comfortable in the leafy surroundings, like a cheeky garden gnome. Unlike some comedians, meeting Herring is not like attending a stand-up show with an awkward audience of one. He is naturally funny but not forcefully so. The conversation is less an interview, more like a chat with an amusing, rather brilliant, slightly self-deprecating friend.

Herring's parents were Christian (C of E) and he recalls attending church a few times each year. "I was religious probably until I was about eight but I always questioned stuff very early on," he says. In Christ on a Bike, a show he first performed in Edinburgh in 2000 and revived in 2011, he questioned the first page of the new testament in the Abraham begat speech — a masterclass in satire combined with an impressive feat of memorisation, pointing out the ridiculous inconsistencies in the religious text. But the idea that the word of God should be questioned and criticised came very early to young Herring.

"That was something I did when I was eight years old," he says. "I remember saying to my mum, "what if someone is married and then that person dies and then they get remarried, what happens in heaven?" My grandmother has just died and she had a boyfriend after my grandfather died 25 years ago, so are they all on a cloud together? Is her boyfriend's wife also there? And is it a bit awkward? If it isn't awkward, are we even anything like we are in real life? If not, what's the point? What I question about the afterlife is how many people just accept it and don't question it, because I think living for an eternity would be much worse than dying."

There's a certain philosophical calm to his words. He's far from Buddha-like, but he's dropped the angry edge that used to drive his comedy. Herring enjoyed some early success as a double act with stand up Stewart Lee, whom he met at Oxford, making the successful late-Nineties TV show, This Morning with Richard not Judy. When the duo split, Lee went on to co-write Jerry Springer — The Opera and started his own BBC TV show, Stewart Lee's Comedy Vehicle, which is commissioned up until 2015, while Herring, to put it delicately, was exploring other mediums of expression. "Maybe six or seven years ago, I would be more cross about people not recognising my genius," he says, "but I have realised I am happy where I am and the level I've got to and I'm not sure I want that many more people to know about me."

Although he still dabbles in TV and radio (he's currently working on a script set in Russia called Ra-Ra-Rasputin, which he describes as "like 'Allo 'Allo, mixed with Blackadder mixed with the music of Boney M., but it's also a satire of what's going on now in politics. With knob jokes."), Herring's main focus is now on his stand-up shows and the Leicester Square Theatre Podcast, a meandering, beautiful, in-depth series of interviews with comedians, created and controlled by Herring. It was on Herring's LSTP that Stephen Fry first revealed that he tried to take his own life in 2012. "It was very interesting to hear him talk about that and he did a fantastic job of spreading the message that someone as successful and amazing as him can, through depression, have something as illogical and crazy happen to him. I was a bit astonished by it. I was planning to ask him what happens when we die but then I didn't get into it, it didn't seem appropriate to discuss it."

He admits he may have been initially resentful of Lee's early success, which was based on a type of comedy that the pair had begun developing together, but the route Herring has taken allows him much greater creative freedom than a TV format, he says.

"If I took the Leicester Square Podcast to TV, they would go, 'It's got to be half an hour and you can't do things like ask if people have ever tried to suck their own cocks' and well then, it's become a very different show. Why it works is it's an hour and half long."

Interestingly, Lee might agree with this philosophy — after commanding huge stadium venues, he has recently taken to playing smaller gigs with just a handful of audience members, under the guise of comedy alter-ego Baconface.

"Stew has that sort of superiority and is very analytical about comedy and I really am a massive fan of his," Herring says, laughing, "but its all well and good taking the piss out of bad comedy but you sort of have to be able to do good comedy as well, er, which Stew certainly can, but I kind of wanted to be funny rather than be laughing at other people's failure to be funny."

So maybe the bitterness at his friend's success hasn't completely gone away, but he is certainly living a more balanced existence now. He married Wilkins in 2012 and they seem very happy. This morning, he tells me, he made her porridge with strawberries in a heart-shape on top. They have an ongoing joke that they are each allowed to sleep with five celebrities outside of the marriage, which is the sort of joke you can only make about your marriage if things are going swimmingly. Her list is limited to just one person: Sawyer from Lost. His includes Amy Pond, "but only the character, not the actress," he clarifies. "If the actress is prepared to stay in character then I will have sex with her. I will make that very clear beforehand.  Ann Widdecombe I've got on there as a wild card just because that's more of a challenge and also she's never had sex, so... she might, you know, calm down a bit, so I might be providing a public service. But only if she wants to."

Despite his jokes about kids being nothing more than 'sexcrement', the 46-year-old isn't at all averse to having children but, he says, this won't stop him making fun of parents' infatuation with their own children. "I've always loved kids," he says, "and I just think it's a funny thing to think of children as sexcrement. The routine is trying to make you look at the world in a different way." This gentler form of poking fun, as opposed to a sardonic onslaught, has emerged in his routines in the last few years.

"I was experimenting with bad taste stuff ten years ago that I wouldn't do now, I think I've got that on a better level. But also comedy is about being allowed to take risks and say things you wouldn't say in the real world. I’m investigating where our boundaries lie by sometimes overstepping those boundaries. That’s why I did a joke about it [rape] and that’s why I didn't do a rape."

He's referring to the joke he made in his Metro column about handling hecklers, writing that his retort to one rowdy female audience member was to say: "‘You're a bit talkative, aren't you? You're loquacious. It's annoying. You're the one woman in the world where a man would put Rohypnol in your drink and then leave you in the pub." Political columnist Owen Jones and New Statesman writer Laurie Penny took issue with him on Twitter, saying he had made an insensitive rape joke, with Herring claiming in a tweet at the time that, "The joke is not about rape".

"Sometimes my jokes are meant to be tasteless," he says, explaining that comedians' jokes are often taken out of context and also that some comedians can get away with more than others. "Aziz Ansari [a popular US comedian] did an amazing bit about paedophilia and if you wrote the words down and got someone else to read it, it would sound terrible, but because he's so charming, 3000 people were charmed by this thing he did about being such a pretty kid that paedophiles were too embarrassed to ask him out in case he rejected them."All Going To Die: Richard Herring's latest show explores death and the afterlife

In We're All Going to Die, Herring does a skit about a real magazine called Railways and the Holocaust, he holds up a copy in the routine, pointing out that it's strange to conflate the two topics; 'railways' and 'the Holocaust' in one publication, and goes on to pick it to pieces in wonderfully pedantic detail, noting that the sub-heading, 'the trains that shamed the world' implies some sinister motive on behalf of the aforementioned locomotives.

"I say, 'if this was a joke you'd be offended by it but because its serious its OK'. Some bloke came up to me afterwards and said, 'Do you think that's funny about the railways and the Holocaust? You think that's something to make a joke about?' People hear a word and that's it. So this guy had just picked out the word holocaust and I said, 'Well, how were you not offended when I mentioned 9/11 or about Nelson Mandela being close to death?' Some people just genuinely don't have a sense of humour and that's why people like the easier comics, because if you don't understand jokes there is a place to laugh and you can tell from the rhythm where the joke's coming."

I'm not sure this entirely excuses the rape joke but it certainly points at the grey area between offensive and pointing out the inconsistencies in public thought where Herring gets some of his best material.

"If we laugh at terrible things that sort of takes their power away. The stage version of me is the sort of pedantic person who would take an idea like that and stretch it out. It's me getting caught up in my own logic to create this different view on the world."

In one of his most surreal stretches of logic yet, Herring has recently taken to playing snooker with himself, something he used to do as a lonely child. He has decided to inflict this onto the public in a series of podcasts (37 and counting), with commentary, also from himself. His motives for this aren't entirely clear.

"I don't think there's enough sport played by incompetent people, with spectators," he says. "If you get the joke you get the joke and if you don't you're annoyed by it. I just wish one of those annoyed people wasn't my wife. [The Me1 vs Me2 Snooker podcast] got 35,000 listeners for the first one, 5,000 listened to the latest one. I'm trying to whittle it down to one listener. I think it's the cleverest and best thing I ever did, it's like performance art, but also very irritating. My wife really hates it. I might divorce her over this, but I don't know if the courts will accept that as a reason. Maybe Ann Widdecombe will be sitting there waiting on the bench."

Richard Herring: We're All Going To Die, is on at the Leicester Square Theatre, 6 Leicester Place, WC2H 7BX, leicestersquaretheatre.com, richardherring.com/wagtd