Regé-Jean Page plays the Duke of Hastings in the hit Netflix period drama

Prodigal Son star Tom Payne is reminiscing on what could have been.

The actor, 38, appeared in the most recent episode of PEOPLE’s Defend Yourself — a new weekly celebrity interview series airing Tuesdays at 10 a.m. ET on YouTube — and revealed that he auditioned for Bridgerton but didn’t quite make the cut.

“I auditioned for one of the parts — the big heartthrob hunk guy — but I’m not tall enough,” he said, referring to the role of the Duke of Hastings, played by Regé-Jean Page.

“You can’t walk into the room and look up at anyone,” Payne added of his height. “It’s just not going to work for that kind of character.”

This Is What Phoebe Dynevor Says Her Bridgerton Costar Regé-Jean Page Smells Like

Still, the Walking Dead alum noted that he does have a rather close connection to the hit Netflix period drama: it was filmed in his hometown in England.

“Bridgerton is actually filmed in Bath, where I’m from, so there are locations in Bridgerton which are still there which you can go and look at and marvel at,” he said. “And it’s very nice to see my hometown on screen.”

Based on the popular books by Julia Quinn, Bridgerton season 1 followed the esteemed families of British high society and their romantic pursuits through the lens of a mysterious gossip columnist, Lady Whistledown (voiced by Julie Andrews).

Bridgerton‘s Regé-Jean Page: 7 Things You Need to Know About Netflix’s New Heartthrob

The series was one of Shonda Rhimes’ long-awaited projects as part of the eight-show deal she made with Netflix in 2017, and it was announced last month that the show would be getting a second season.

Page caused quite the stir with his portrayal of Simon Basset, the brooding Duke of Hastings, while costar Phoebe Dynevor stole fans’ hearts as the delightful Daphne Bridgerton — so much so, in fact, that their onscreen chemistry stirred up rumors of a real-life romance.

But during a joint interview with Access, the actors laughed off rumors about them dating off-screen. “I think everything you need to know is on camera. That’s why we presented it so beautifully for you,” Page joked.