Desegregation in Leesburg was a gradual and generally peaceful occurrence that resulted in desegregated lunch counters as early as 1961. Loudoun County schools began desegregating in 1963, and were completely integrated by 1968, when the African-American high school, Douglass High, closed. However, in the summer of 1963, parts of Leesburg remained segregated, including the baseball field and swimming pool.

Leesburg volunteer firemen's swimming pool, which was built in 1956, was a public pool for white swimmers only. After successfully protesting to integrate both the Tally-Ho Movie Theater and Village Lanes Bowling Alley in the early summer of 1963, Leesburg's African-American community, including leader Gene Ashton (1946- ) and his sister Gertrude (Ashton) Evans (1948- ), turned its focus to the swimming pool. Even after several weeks of peaceful protests, they did not have any success; the firemen persistently refused to let blacks in. The swimming pool remained open for the remainder of the summer, but was still segregated. In 1965, one year after President Lyndon B. Johnson (1908-1973) signed the Civil Rights Act of 1964 prohibiting discrimination based on color, four African-American children were again refused entry to the Firemen's Swimming Pool. They and their parents filed federal suit under the aegis of Civil Rights Act. The following spring, the court ruled in favor of the children and ordered the firemen to allow black swimmers into the pool. The firemen refused and closed the pool to avoid having to integrate. The pool remained closed and in 1968, the land was sold and the pool was filled in with rocks and cement. It was not until 1990 that Leesburg again had a public swimming pool and not until 2009 that it had an outdoor public pool.