Following Lord Dunmore's bombardment of Norfolk in January 1776, the Virginia legislature ordered the construction of a fort on this site, and another at Fort Nelson across the river, to protect the area of Norfolk Town. The Forts were not sufficiently manned and were ineffective against Sir George Collier's British fleet in May 1779. About 1810 Fort Norfolk was rebuilt with the present exterior brick walls and many of the interior buildings were constructed. The fort was manned by the Virginia militia during the War of 1812, but no shots were fired. The Navy acquired use of the fort as an ammunition depot in 1849 and built the large magazine in 1856. Following the Confederate evacuation of Norfolk in May 1862, Federal troops used the fort briefly as a prison. From 1863 to 1878 it was again a Naval installation. From 1921 through 1983 the old fort was occupied by the Army Corps of Engineers. Since 1991 the Norfolk Historical Society has used the fort as its headquarters and as an historical attraction.