Asked what he misses most about being Labour leader, Ed Miliband answers: “The pulpit...the chance to speak to the country and shape the national debate."

Perhaps unsurprisingly, what he misses least is “the intrusion on your family life. The sense that you are never able to turn off.”

Oh and he’s not pining for the bear pit of Prime Minister’s Questions “one iota”.

“That was just a grind. It holds the Prime Minister to account, but it never wins the leader of the opposition any favours,” he tells the Ham&High Podcast.

Perhaps fellow north Londoner Keir Starmer should take note from a man whose bruising 2015 election defeat reaped such personal and political fallout. So how do Ed’s wife Justine and sons Samuel and Daniel feel about his return to shadow cabinet? A greyer Miliband gestures with his hand to denote ambivalence.

“I am pleased I came back, it’s much less stressful than being leader, but they would say I work too hard. I don’t find relaxing and not working very easy to be honest.”

When he’s not watching the Boston Red Sox or Unforgotten box-sets, the Darmouth Park resident has become a pond swimmer – after reading a New York Times listicle of top swimming spots and realising he’d never been to the one on his doorstep.

“I’m definitely a convert,” he says, revealing that the downside is crossing paths with ex Labour spin doctors on their morning workout.

“I ran into Alastair Campbell at the lido the other day because the ponds were closed. As I arrived at 7am with a lot of people, Alistair greeted me with: ‘Pond life – we’re having to queue because of you.’

“Thanks very much and good morning to you!”

Although he’s written a book Go Big: How To Fix Our World, Ed isn’t offering Starmer any tips.

“Everyone’s always got a view of what you should be doing, you get so much advice.”

It’s inspired by his podcast Reasons to be Cheerful, in which he and Geoff Lloyd interview the world’s doers and fixers tackling big problems. But it’s also a manifesto about the power of ideas.

"People confuse winning at the first attempt with thinking it can never happen, but good ideas if you fight for them tend not to die. Change seems impossible until it happens. Part of the DNA of being in politics is you’ve got to have optimism. My dad used to say ‘history is on our side’. Things do get better. Struggle produces positive change."

Ham & High: Left to right: Sir Keir Starmer, Tristram Hunt and Ed Miliband at Haverstock School. Picture: Nigel Sutton.Left to right: Sir Keir Starmer, Tristram Hunt and Ed Miliband at Haverstock School. Picture: Nigel Sutton. (Image: © Nigel Sutton email pictures@nigelsuttonphotography.com)

Marxist academic Ralph Miliband and human rights campaigner Marion Kozak, who both survived the Holocaust, brought up their sons David and Ed in Primrose Hill.

“My first political memory is Harold Wilson resigning. Play School was cancelled and I was extremely upset,” he chuckles.

Growing up under Thatcherism, attending inner London comprehensive Haverstock School, when he wasn’t sneaking off to watch Dallas, Ed joined the debate in his “very political household”.

“There’s two choices; to rebel or to get with the programme. Our parents encouraged us to have opinions and invited us into the conversation, it gave me an excitement about politics. My dad was very egalitarian. People would come to dinner, I or David would pipe up, and if there was any sense that the guest was patronising us he would step in to defend us.”

His most vivid memory was meeting anti-Apartheid campaigners Joe Slovo and Ruth First who was later assassinated by the South African government.

Ham & High: Ed MilibandEd Miliband (Image: Jenny Smith Photography)

“It made me feel that politics was about big things. It really mattered.”

And although their Holocaust experience “wasn’t really talked about”, Ed sensed it “must be quite defining”.

“I think about the dislocation that it caused them, it made them almost quasi religious in their belief that you have a responsibility to make the world a better place, which if I am being completely frank, is quite a burden."

He has fond memories of Primrose Hill Primary as a “lovely, safe place” but Haverstock, which he left with four A-levels and a place at Oxford, was an altogether edgier prospect.

“The teaching was high quality – but it had its rough elements. I remember feeling a bit anxious at times. I see from my son that safety and bullying is taken much more seriously today than it was then.”

If the podcast and book have helped to humanise Miliband, they have also been a platform for discussing big issues.

“Politics, the day to day yah-boo stuff can feel very off-putting. We saw there was a desire to cover big ideas in an accessible non-partisan way. We’d get people saying: ‘I’m a Conservative and would never dream of voting for you but I like some of your ideas.'”

Miliband believes Britain is at a crossroads. “We are in injury time for the Thatcherite settlement that was adapted by Tony Blair, and people across the political spectrum are looking for the new solution. My constituency voted 70%for Brexit, they wanted something different. The question is what does the new contract look like?”

He says he was trying to “move the debate on” during his election bid and reflects that he “could and should have been bolder”, because small change won’t solve huge threats like climate change and inequality.

“Lots of things I said that were controversial then – ‘Red Ed' and all that. You now can’t find anyone who disagrees with them. It takes time to move a debate. I am on a mission to persuade people that it’s not hopeless, and if I can say that having lost a general election there must be something to it!”

Astoundingly he remains optimistic about the political process but adds: “It’s not politicians that make change, it’s people, but the role of politics is to try to make sense of our collective experience and what the future looks like.”

Will his kids follow him into politics?

“I don’t think so, which I am probably quite pleased about. They care about the world but without too much of an overwhelming responsibility for it.”

Click here to subscribe to the the Ham&High Podcast.

Ham & High: Go Big is published by Vintage price £18.99Go Big is published by Vintage price £18.99 (Image: Supplied)