James G. Varn and his younger brother, Rev. Little Berry Varn, after whom the Town of Varnville was later named, had a sawmill on the north side of the tracks and sold the right-of-way to the railroad. The Varns then laid out a town on the north side of the tracks. The Town of Varnville was born and a post office opened in 1872.
The town was formally incorporated December 24, 1880. Varnville's history is intertwined with the logging industry, agriculture and the railroad.