Prince Harry Answers Reader Question About Growing Up in a Palace During 'Spare' Book Event

Prince Harry says that living in royal residences isn't quite like the storybooks.

The Duke of Sussex, 38, joined Dr. Gabor Maté on Saturday to discuss his groundbreaking memoir, Spare, in a virtual book event. Prince Harry and Maté, an expert specializing in trauma, addiction, stress and childhood development, spoke about living with loss, healing and how past experiences shape our identities today during their intimate conversation.

Following the wide-ranging exchange, which touched on family, relationships and fatherhood, Harry took a few questions from the virtual audience — and the last question ended the event on a lighthearted note.

"As a royal child, did you get to roam the castles freely, and did you find any trap doors?" Maté asked on an attendee's behalf. With a chuckle, Harry replied, "That's the assumption, that there were trap doors all over castles. As kids, it was an adventure, running around that kind of a place. What an experience."

"Not so many trap doors, lots of fun things to investigate, but I guess, it's weird. When you're brought up as a child in that environment, you think that is normal, but it's not. It's anything but normal," he continued. "So no, some secret doorways and some secret cupboards. None so sophisticated as you pull a book and then all of the sudden a thing opens. I probably would have gotten stuck in there," he joked.

Prince Harry and Prince William were mainly raised at Kensington Palace in London by their parents Princess Diana and King Charles. Both boys began boarding school at the Ludgrove School at age 8, before moving on to Eton. During school breaks, they'd also spend time at Highgrove House, Charles' country home in Gloucestershire, Balmoral Castle, their grandmother Queen Elizabeth's beloved retreat in the Scottish Highlands, and more.

Harry and his wife Meghan Markle now call the U.S. home. The couple is raising their two children, Archie, 3, and Lilibet, 1, in Montecito, California. Last week, it was revealed that the Duke and Duchess of Sussex have been asked to vacate their U.K. home base of Frogmore Cottage in Windsor.

PEOPLE exclusively revealed the news of the livestream event last Friday, six weeks after Spare hit shelves. The book became an instant New York Times bestseller and remained at the #1 spot on the New York Times Hardcover Nonfiction bestseller list for six consecutive weeks.

English-language sales for Spare top more than 4 million copies with over 1.43 million units purchased on the first day it was available, making Spare the fastest-selling nonfiction book of all time.

The week of the book's release, Prince Harry opened up to PEOPLE about what he hopes his family takes away from the text.

"I want people to read my memoir and come to their own conclusions — I don't want to tell anyone what to think of it, and that includes my family. This book and its truths are in many ways a continuation of my own mental health journey," he told PEOPLE. "It's a raw account of my life — the good, the bad and everything in between."