James Gardner was born in Paisley, Renfew, Scotland in 1829. The family moved to Glasgow and in 1850. His older brother, Alexander Gardner, was interested in the socialist ideas being advocated by Robert Owen. Inspired by the New Harmony community established by Robert Dale Owen and Fanny Wright in Indiana, Gardner helped establish the Clydesdale Joint Stock Agricultural & Commercial Company. The plan was to raise funds and to acquire land in the United States.
In 1850, Alexander Gardner, took his brother and sister, Jessie Sinclair, and six others to the United States. He purchased land and established a cooperative community close to Monona, in Clayton County, Iowa. Gardner returned to Scotland to help raise more money and to recruit new members.
Alexander Gardner returned with the rest of the family in 1856. When he arrived at the Clydesdale colony, he discovered that several members were suffering from tuberculosis. His sister had died from the disease and her husband was to follow soon afterwards.
On the outbreak of the American Civil War there was a dramatic increase in the demand for Gardner's work as soldiers wanted to be photographed in uniform before going to the front-line. The following officers were all photographed at the Matthew Brady Studio: Nathaniel Banks, Don Carlos Buell, Ambrose Burnside, Benjamin Butler, George Custer, David Farragut, John Gibbon, Winfield Hancock, Samuel Heintzelman, Joseph Hooker, Oliver Howard, David Hunter,John Logan, Irvin McDowell, George McClellan, James McPherson, George Meade, David Porter, William Rosecrans, John Schofield, William Sherman, Daniel Sickles, George Stoneman, Edwin Sumner, George Thomas, Emory Upton, James Wadsworth and Lew Wallace.
In July, 1861 Matthew Brady and Alfred Waud, an artist working for Harper's Weekly, travelled to the front-line and witnessed Bull Run, the first major battle of the war. The battle was a disaster for the Union Army and Brady came close to being captured by the enemy.
Soon after arriving back from the front Matthew Brady decided to make a photographic record of the American Civil War. He sent Gardner, Alexander Gardner, Timothy O'Sullivan, William Pywell, George Barnard, and seventeen other men to travel throughout the country taking photographs of the war. Each one had his own travelling darkroom so that that collodion plates could be processed on the spot.