As with many Norfolk towns, the name can be traced back to a Saxon ancestry, with the word probably deriving from the Saxon word for wood. A clearing and a crossing place for roads on the wooded high ground of the Cromer-Holt ridge was a natural place for a settlement to grow up. The ridge itself was formed from boulder clay, deposited during the last ice age.

Holt gains a mention in King William's Domesday book of 1086. It had its own market and its own port - Cley is listed as being the port of Holt. With five water mills and twelve plough teams it can been seen as a busy settlement. Originally held by Walter Giffard, the manor passed to Hugh, Earl of Chester, who in turn left it to the De Vaux family. With a well-established market and annual fair days on April 25th and November 25th, Holt grew as the local centre of trade.

There are a few glimpses of the town in the medieval period, with De Vuaux's bailiffs being recorded as being less than honest with the townsfolk - stealing goods from stallholders, and holding some of their animals to ransom. The plague of 1348 had a devastating effect in the Holt deanery - 23 priests are recorded as dying during this period, and there must have been many more of their parishioners besides. At the time of the Peasants revolt, later in the 14th century, local leader Geoffrey Litster preached unrest in Holt market place. In 1588 the threat of invasion from the Spanish Armada led to the fortification and garrisoning of nearby Weybourne, which must have led to much activity in Holt.

There is evidence of a church in Holt for over 1000 years. Associated with the work of the church in Holt was the Guild of St Mary the Virgin. Such guilds provided relief for the por of the parish until the guilds were supressed in the time of Henry VIII.

Holt becamse victim to a devastating fire in 1708. The fire destroyed most of the town within a space of three hours. The church was badly damaged - its thatched chancel burned, lead melted from the windows and the flames spread up the steeple. Total damage was over