James Stephen, the second of seven children of James Stephen (1733-1779), and his wife, Sibella Milner (1738–1775), was born on 30th June 1758 at Poole, Dorset. In 1773 he was sent to Winchester College, but for financial reasons, he was forced to return home to Kennington.

After the death of his mother he began studying law. He entered Lincoln's Inn on 23rd September 1775 and studied at Marischal College in Aberdeen, for two years. He had to abandon his studies because of a shortage of money. On his return to London he assisted with his father's practice as a conveyancer.

On the death of his father in September 1779, Stephen became a reporter for The Morning Post. He also reviewed books for the newspaper. In 1781 his uncle died and with his inheritance he finished off his legal education. According to his biographer, Patrick C. Lipscomb: "After passing the bar on 26 January 1782 he made no serious effort to practise, but continued his studies and attempted to collect a legal library."

On 17th June 1783 he married Anna Stent (1758-1796). Later that year he decided to emigrate to the West Indies. He stopped off at Bridgetown and witnessed the trial of four slaves accused and convicted of murder. His biographer argued: "The manifest injustice of the proceedings, the obvious innocence of the accused, and the knowledge that there was good reason to believe that a white man was the murderer led Stephen to vow never to own a slave and to become a committed opponent of slavery."

James Stephen worked as a lawyer on St Kitts. Over the next few years his wife gave birth to seven children. In September 1794 the family returned to London where he became involved in the campaign against slavery. He contributed anti-slavery propaganda to The Morning Chronicle, and he served as counsel for the Sierra Leone Company, presenting evidence before the House of Lords, as part of the campaign against slavery.

After the death of his wife in December 1796, Stephen became converted to Evangelical Christianity. He joined the Clapham Set, a group of evangelical members of the Anglican Church, centered around Henry Venn, rector of Clapham Church in London. He became friends with William Wilberforce and on 15th May 1800 he married his sister, Sarah, the widow of Thomas Clarke.

Stephen was elected a member of the London Abolition Committee on 23rd May 1804. The following year he helped to draft an order in council passed by the government on 15th August to prohibit the importing of slaves into the recently conquered Dutch colony of Guiana. In 1806 he provided Lord Grenville with the draft of a bill to abolish the foreign slave trade, which parliament passed and which effectively outlawed over three-quarters of the existing trade. After the passing of the Abolition of the Slave Trade Act in 1807 Stephen helped to organize the African Institution.

On 25th February 1808 he was elected to the House of Commons for Tralee. In June 1810 Henry Brougham complained that the Abolition of the Slave Trade Act was ineffective. Brougham argued that Britain was doing nothing to end "this abominable commerce". Yet she was always ready to use her power "when the object is to obtain new colonies and extend the slave trade; then we can both conquer and treat; we have force enough to seize whole provinces where the slave trade might be planted and skill enough to retain them and the additional commerce in slaves their cultivation requires." James Stephen replied: "we have at least delivered ourselves as a nation from the guilt and shame of authorising that cruel and opprobrious traffic... If we have effected nothing more I shall rejoice and bless God to the last hour for this happy deliverance."

In 1812 Stephen changed seats, becoming MP for East Grinstead. According to Patrick C. Lipscomb: "Acutely aware of his want of a classical education Stephen spoke infrequently in parliament, mainly on issues relating to the anti-slavery movement and religion. As a parliamentary speaker Stephen lacked polish, wit, and humour, and was frequently too vehement in tone.... Arguably Stephen's greatest contribution to the anti-slavery movement was his advocacy of slave registration and his drafting of the proposals for registering slaves on the island of Trinidad. This order, as finally passed on 26 March 1812, was later extended to the islands ceded by France at the end of the Napoleonic wars - Mauritius, St Lucia, and Tobago. It also served as a model for the proposed registration of slaves in the older slave colonies."

Stephen resigned his parliamentary seat on 14th April 1815. He now concentrating on writing and published in two volumes, The Slavery of the British West India Colonies Delineated (1824 and 1830). The book provides a comprehensive and detailed picture of the British West Indian slave system in the early nineteenth century.

James Stephen died at Bath on 10th October 1832 and was buried at Stoke Newington.