James Covey was born in Africa in about 1820. When a child he was kidnapped and sold as a slave. He was purchased by Portugese couple living in Mani who employed him on their farm.

Covey was later obtained by a slave-trader but while being transported to America, the Portuguese slave-ship was intercepted by a British armed vessel. Covey was taken to Sierra Leone and given his freedom.

Covey was taught to read and write the English language by members of the Church Missionary Society.

In November 1838, Covey enlisted as a sailor on board the British brig of war Buzzard, commanded by Captain Fitzgerald. It was on board this vessel, when at New York, in October 1839, that he was asked to become a an interpreter in the investigation into the Amistad Mutiny.