Originally constructed in 1763, Folly Castle is a Georgian-style home built by Peter Jones II, the great grandson of Peter Jones for whom Petersburg is named. After inheriting vast amounts of land in Petersburg, Peter parceled and sold twenty-eight lots on High Street. As a result, he was able to finance his new home at the age of twenty-three.

Folly Castle's name originates from two distinct sources. During the colonial times, "castle" was a common reference for large dwellings. Because Peter was a bachelor, his home became known as his "folly." Hence, the name "Folly Castle" was derived.

After Peter's death at the age of thirty, Folly Castle remained in the family for approximately one hundred years thereafter. Peter's entire estate eventually went to his niece, Sarah Newsum. Sarah continued developing lots within Petersburg, including the once popular Gillfield Race Track. Sarah's husband Major Erasmus Gill, a Revolutionary War soldier, was quite a gambler. In order to pay off his massive debts, Erasmus sold large tracts of Sarah's estate, which reduced the sizable land on which Folly Castle stood.

Folly Castle stayed in the family following Sarah's death in 1825 as her daughter, Martha, resided in the home. In 1855, Martha arranged for the house to be moved one hundred feet towards West Washington Street. Moving the house forward placed it on a street which had become a fashionable Petersburg address. At Martha's death in 1874, Folly Castle stood on merely eleven acres of land. Martha's son William Hilton acquired the home and sold off lots surrounding Folly Castle. He also made extensive renovations reflecting its present-day Greek Revival architectural style. When William died in 1895 the property was sold outside the family.

Mary Deffenbaugh established Folly Castle as a guest house and tea room in 1928. For nearly forty years she catered to Petersburg's elite. Brides descended the beautiful staircase to greet well-wishers assembled in the great drawing rooms below. Mary transformed Folly Castle into a "home away from home" for international tobacco buyers as well as for soldiers and their wives during World War II.