The Nelson family has a long and rich history in the state of Virginia beginning with Thomas Nelson (1677–1745) who came to Yorktown in the early years of the 18th century and is often identified as Scotch Tom. Among his sons was William Nelson (1711–72), colonial governor of Virginia (1771–71), sometimes known as "President Nelson," as he held the title, President and Commander in Chief of the Colony and Dominion of Virginia; and Thomas Nelson (1716–82), known as Secretary Nelson, who served as secretary of the Virginia Colony. One of William's son's, Thomas Nelson (1738–89) represented Virginia in the Continental Congress, was a signer of the Declaration of Independence, and was Governor of Virginia in 1781. Governor Thomas Nelson and his wife, Lucy Grymes had eleven children, all of whom were born at Yorktown. Among Governor Thomas Nelson and Lucy Grymes's children were Thomas Nelson Jr. (b. 1764), Philip Nelson (b. 1766), and Francis Nelson (b. 1767).

Thomas Nelson Jr. married Frances Page in 1795 and had four children, including Rev. George Washington Nelson (b. 1805), who married Jane Crease in 1834. They had three children, including George Washington Nelson, Jr. (b. 1840), also known as "Wash." He attended University of Virginia from 1858 to 1860, but, in May 1861, enlisted in the Confederate Army. Within a year, he was a Captain commanding the Hanover Light Artillery, soon to be promoted to Major. In October 1862, outside New Market in Millwood, Clarke County, Va., he was captured by Union forces and spent the rest of the war as a prisoner. Confined first at Atheneum Prison, West Virginia, he was transferred to Camp Chase, Ohio; Johnson's Island, Ohio; Pt. Lookout then Hammond General Hospital, Maryland; and then to Ft. Delaware, Delaware in June 1864. On 20 August 1864, he was moved to Morris Island outside Charleston, South Carolina, where he become one of what is known as the Immortal 600, a group of confederate prisoners placed by the Union in the line of Confederate fire, in retaliation for a similar action by the Confederate Army. "Wash" was moved again to Ft. Pulaski, Georgia, where he was finally released in June 1865. His correspondence with cousin Mollie Scollay, written while he was a prisoner of war, is a prominent part of this collection. He and Mollie were married in October 1865. George Washington Nelson, Jr. was ordained as a Episcopal priest in 1875 and had been rector of St. James Episcopal Church in Warrenton, Va. for over twenty years at the time of his death on 30 May 1903. Mollie died on 12 June 1923.

Philip Nelson married Sarah Ann Burwell in 1789 and moved to Clarke Co. Va. together with Sarah's brother, Robert Carter Burwell in 1790. Robert Carter Burwell was the builder of the home "Rosney" and, about twenty years later, the nearby mansion at Long Branch Plantation, both of Clarke County. Burwell died while serving in the military during the War of 1812, after which Philip and Sarah Nelson, who had lived at Rosney, inherited Long Branch.

Francis Nelson of Mont Air, Hanover Co. Va. married Lucy Page about 1792 and had fourteen children. Among them were Sally Page Nelson (b. 1801), Hugh Mortimer Nelson (b. 1811), and Fannie Burwell Nelson (b. 1810).

Sally Page Nelson married Dr. Samuel Scollay of Smithfield, Jefferson County, Va. (later West Virginia) in 1841. Among their children was Mary Scollay (b. 1844), also known as Mollie. It is Mollie's correspondence with George Washington ("Wash") Nelson, Jr., her future husband, while he was a prisoner of war that forms a large part of this collection. "Wash" and Mollie were cousins, as their fathers, Thomas Nelson, Jr. and Francis Nelson, were brothers. They were both grandchildren of Governor Thomas Nelson.

Hugh Mortimer Nelson (Sr.) married Adelaide Holker (b. 1816) of Boston in 1836. They settled in Baltimore for a short time, where Hugh began his career as a lawyer. In 1842, Hugh moved back to Virginia with his wife and three-year old daughter Nannie and purchased a struggling Long Branch from his uncle Philip Nelson. The correspondence between Adelaide and daughter Nannie while the latter was attending a New York school in 1857 comprises another significant portion of this collection. Son Hugh Nelson, Jr. was born in 1847. Another daughter, Lucy, was born in 1842, but died as an infant. Hugh Mortimer Nelson, Sr. represented Clarke County at the Virginia (Secession) Convention of 1861, where he initially maintained his support for the Union. Following the events at Ft. Sumter and Virginia's decision to secede, he raised a cavalry company from Clarke County and served under J.E.B. Stuart's command for a time before attaining the rank of Major under General Richard S. Ewell. Wounded at the Battle of Gaines Mill/Seven Days Battles on 26 June 1862, he died in Albemarle County on 6 August 1862. Adelaide was left in charge of Long Branch. She died in 1875 of pneumonia after a long struggle to keep the property in the family.

Fanny Burwell Nelson, one of Hugh Mortimer Nelson, Sr's sisters, never married and moved from Mont Air, Hanover County to Smithfield, Jefferson County, Va. (later West Virginia) around 1846. She moved from Smithfield to Long Branch in 1878 when it belonged to Hugh Mortimer Nelson, Jr. and his wife, Sally Page Nelson (b. 1866), daughter of George Washington ("Wash") Nelson, Jr. and Mary (Mollie) Scollay Nelson. Fanny Burwell Nelson died at Long Branch in 1896.