Records of the first settlement in Whaddon date from Anglo Saxon times, when the village is referred to in the Anglo-Saxon Chronicle (AD 966-75) as Hwætædun, meaning 'hill where wheat is grown'. Wadone, with its 14 villagers and nine small-holder households, is next mentioned in the Doomsday Book, listed as part of the reward by William the Conqueror to a Norman baron, Walter Giffard, Seigneur de Longueville, for his service during the Conquest. The manor reverted to the Crown in 1164 and was then granted to Richard de Humetis, Constable of Normandy. It passed through several prominent families throughout the medieval period.
The village is at the centre of the ancient Whaddon Chase, an important medieval hunting forest, in existence since the 13th century. Whaddon Chase is designated an area of 'Special Landscape Interest'.
Arthur, 14th Lord Grey de Wilton, inherited the manor house in Whaddon (Whaddon Hall) in the mid-16th century and greatly extended it. Queen Elizabeth I is thought to have paid a visit, in 1568. The Elizabethan poet, Edmund Spenser, who was Grey’s secretary, frequently stayed at the hall and is even reported to have written The Fairie Queene sitting under an oak tree in the grounds! Other famous Whaddon residents include Dr Richard Cox, Bishop of Ely, and tutor to King Edward VI, who was born here in 1500; and Browne Willis (1682-1760), the well-known antiquary and author, who lived in Whaddon Hall and left his library to Oxford University.
Whaddon Hall became the home of the Lowndes family from 1783 when Mr William Lowndes Selby took possession of the Hall. In 1813 his son took again the family name of Lowndes after that of Selby, and so the name of Selby-Lowndes became associated with the village. The present Whaddon Hall, dating from around 1820, is at least the fourth to stand on the site.
During World War II Whaddon Hall served as headquarters of Section VIII (Communications) of MI6. The "Station X" wireless interception function was transferred here from Bletchley Park in February 1940. That facility served in a number of capacities, the most critical of which was the transmission of Ultra intelligence to allied generals in the field. In addition, specialist secret communications equipment used by secret agents and resistance groups, as well as in Allied naval vessels and aircraft, was designed and built in Whaddon. A plaque on the village hall, unveiled in May 2016, commemorates the village’s significant wartime role.