Castle met Barbara Castle during the Second World War. She later recalled: "I did not take much to him at first sight: tall, dapper, with a neat moustache, he looked a bit like David Niven, and I suspected him of being just a slick journalist. I gradually changed my mind." The couple were married in 1944.
In 1950 Tom Hopkinson sent James Cameron and Bert Hardy to report on the Korean War. While in Korea the two men produced three illustrated stories for Picture Post. This included the landing of General Douglas MacArthur and his troops at Inchon. Cameron also wrote a piece about the way that the South Koreans were treating their political prisoners. Edward G. Hulton considered the article to be "communist propaganda" and Hopkinson was forced to resign. Castle took over as editor but six months later he was sacked by Edward G. Hulton.
In 1973 Castle had open-heart surgery. He recovered and the following year was made a life peer and was an active member of the House of Lords.
I did not take much to him at first sight: tall, dapper, with a neat moustache, he looked a bit like David Niven, and I suspected him of being just a slick journalist. I gradually changed my mind. His marriage had broken up and he was waiting for his decree absolute, so he had a lot of time to spare for me. There then ensued one of the most unusual courtships in Cupid's history. He would meet me outside the Fish Division and listen with all the appearance of enthralled interest while I regaled him with stories about dehydrated mullet and fish hooks for Iceland, to say nothing of the mysteries of snoek. As a younger man (he was then in his early thirties), he had been very active in NUJ politics, organizing everything from press balls to strikes, and had been recognized as a bit of a lad, but I soon found that he was a serious socialist.