Joseph Fry was born in Sutton Benger in Wiltshire in 1728. The son of a well-known member of the Society of Friends, he was educated as a doctor. As a young man he moved to Bristol where he set up business as an Apothecary. He advised his patients on diet and was a strong believer in the health qualities of cocoa powder.

In 1756, Fry started making his own chocolate at a factory in Bristol. Fry was also involved in several other business ventures including the manufacture of china and soap. Joseph Fry also joined with William Pyne to form a type-foundry. In 1758 Oxford University commissioned the company to create a fount of Greek types. Fry was also involved in publishing books. This included editions of classic authors such as Virgil and Milton.

When Joseph Fry died in 1787 his chocolate business was taken over by his three grandsons: Joseph, Francis and Richard. Under their management the company continued to expand. In 1919 J S Fry & Sons merged with Cadbury Brothers.