The chat show host, 43, has become pals with Harry and his wife Meghan Markle and lives just an hour and a half away from the couple’s sprawling mansion in Montecito, California.
Asked if he had been to the couple’s £11million home, James said: ‘I have. It was more a sort of play date scenario [with the kids]. Then we went out for dinner after, it was lovely.’
James and Julia are parents to children Max, 11, Carey, seven, and Charlotte, four, while Prince Harry and Meghan share Archie, three, and Lilibet, 11 months.
Harry previously appeared as a guest on The Late Late Show shortly after he moved to Los Angeles with Meghan.
Adding that he knows Harry better than Meghan, James said he finds it hard to be ‘judgemental’ of the pair, seemingly referring to their decision to step down as senior members of the royal family and move to the US.
He said: ‘I have a huge amount of respect and admiration for Harry, and anyone who has been around him and spent time with him would feel exactly the same way.
‘I think he is a devoted and loving husband and father and I think what they have done [in turning away from royal life] is incredibly brave and I’ll always be in their corner.’
Although he is quitting his CBS chat show in 2023, James said he is also a big fan of Prince William and Kate Middleton and would love to have them as guests but added he is not ‘holding [his] breath’.
James and his wife Julia are planning to relocate to the UK after his time on his CBS show has come to an end.