Sherman was the son of Elias Huntington Sherman and Clarissa Wilmarth. He worked on the family farm until 1855, when he enrolled in pre-college studies at the Brandon Seminary, and then Burr Seminary in Manchester, Vermont. He entered Middlebury College in 1856, graduating in 1860. Upon graduating from college, he went on to teach at Brandon and South Woodstock Seminaries.
With the outbreak of the American Civil War, in May 1862, Sherman enlisted in the 9th Vermont Infantry. This regiment was captured at the September 1862 Battle of Harpers Ferry, a part of the Maryland Campaign. Following the capture of his regiment, the Union Army sent Sherman to Camp Douglas. Annoyed by this forced idleness, Sherman resigned his commission in January 1863 (he had attained the rank of lieutenant).
Sherman now enrolled at the University of Chicago to study law. He graduated in 1864 and set up a law practice. In 1876, he was elected to the Illinois House of Representatives. The next year, Shelby Moore Cullom, Governor of Illinois, appointed Sherman as judge advocate of the first brigade of the Illinois National Guard, a position he held for many years. He won re-election to the Illinois House of Representatives in 1878. In 1879, he was appointed master in chancery of the United States District Court for the Northern District of Illinois. In 1882, he was elected president of the Illinois State Bar Association. He was also active in the activities of the American Bar Association.