James Speed was born in Kentucky in 1812. He studied law at Transylvania University and worked as a lawyer in Louisville. He joined the Whig Party and was a strong opponent of slavery.

Elected to the Kentucky Senate in 1861 he became the leader of the pro-Union forces in the state during the American Civil War. In 1862 he controversially introduced a bill to "confiscate the property" of those supporting the Confederacy in Kentucky.

In November, 1863, Abraham Lincoln selected Speed to replace Edward Bates as his Attorney General. After the assassination of Lincoln he became associated with the Radical Republican group and advocated he vote for male African Americans. Disillusioned with the increasingly conservative policies of President Andrew Johnson, Speed resigned from the Cabinet in July, 1866.

Speed returned to his law practice in Louisville. Speed's radical views were unpopular in Kentucky and his attempt to be elected to the Senate in 1867 ended in failure. James Speed, who described Abraham Lincoln as the "best and greatest man I ever knew", died in 1887.