From the marker next to the lighthouse:

OLD POINT COMFORT LIGHT
The lighthouse, built in 1802, is the oldest standing structure at Fort Monroe.
It remains an active navigational aid, the property of the U.S. Coast Guard.
During the War of 1812, the tower was used as a lookout by a British
invasion force while they attacked Washington. The adjacent house
was the lightkeepers' quarters until the light was automated in 1973
when the house became Army property.