Born in Natchez, Mississippi in 1828, James W.M. Harris studied at Amherst College and returned to his native state to read law under General Sparrow. In 1852 Harris married Mary C. Lum (1831-1871), and the couple took up residence near Vicksburg at Avenel Plantation. The young couple soon had a growing family, with the birth of daughters Annie Laura (Lola) in 1853, Caroline (Carrie) in 1855, and Natalie in 1859. Harris continued to practice law in Vicksburg, but the family cotton plantation, Avenel, also provided income. The couple's lives would change, however, with the coming of the Civil War.

Like many southerners, the couple's lives changed dramatically with the advent of war. Harris and his two brothers, Nathaniel and William, served the Confederate States of American in various capacities. Both Nathaniel and William joined the army, with the former eventually rising to the rank of General, and the latter serving as quartermaster. Meanwhile, James undertook a position as Third Auditor of the Post Office Department in Richmond, a position that took him away from Avenel and his family for the duration of the war. Because of the Union occupation of Vicksburg, Mary and the three children spent much of the war in Eufaula, Alabama. At the end of the War in 1865, the family was reunited and returned to Warren County to once again make Avenel home. Harris resumed his law practice, but the family moved to New York where Mary died in childbirth in 1871. Harris moved to the Washington Territory shortly before his death in 1885, where he attempted to become territorial governor. Upon his death, Harris's body was returned to New York, where he was interred next to his wife Mary.