Caleb Smith was born in Boston, Massachusetts, in 1808. He studied law and was admitted to the bar in 1828. Smith worked as a lawyer in Connersville, Indiana and also became involved in local politics. He joined the Whig Party and was elected to the House of Representatives.
When he was elected president Lincoln considered appointing the radical Schuyler Colfax as his Secretary of the Interior. However, he eventually selected the more conservative Smith. Lincoln told Colfax that he was a young man with a promising career in front of him, whereas the age of Smith meant it was "now or never".
Smith advised Abraham Lincoln against attempting to relieve Fort Sumter believing it was important to do everything possible to avoid war with the Confederacy. Once the American Civil War started he was the main supporter of a negotiated peace. Smith was also the Cabinet's leading opponent of the recruitment of black regiments.
Unhappy with Lincoln's increasing radicalism, Smith resigned from office in December, 1863. Abraham Lincoln appointed him Judge of the Indiana District but he died a few months later.