The 1850 census gives us a glimpse of the household at Ellenborough. Benjamin's biographer Joshua E. Kastenberg noted their property was valued higher than any other in St. Mary's County with 45 enslaved laborers. Ann Delia was at home but Susan may have been at school. Others may be relatives including Emily Long at 45 listed as Insane and a Pauper.
When the Civil War began Martha was in her mid 40s with two grown girls; Susan was married to George F. Maddox. Benjamin had become a spokesman for Southern secession in a polarized Maryland. He'd given an indication of his political and personal opinions in a failed 1858 petition to the Maryland Legislature to re-enslave the state's 80,000 freed people. Maryland remained in the Union but many of his neighbors were sympathetic to the Confederate cause. In 1863 he was elected to the U.S. Congress to represent the southernmost Maryland counties.