Hartrow Manor House and estate are traceable back to the time of Domesday (1087) when Roger de Lisieux held it of William de Mohun of Dunster Castle. The property through subsequent centuries was then connected with the Vinar (or Venas), Hartrow and Sydenham families until passing through the Wyndham family it came into the possession of William Lacy around 1565; he built himself a house c.1580 which occupies the site of the present building. The property passed into the estate of Thomas Rich in 1711, whose son died unmarried at the age of 23 in 1727, leaving the major part of his estate to his fiancée, Margaret Hay.
The old house built by William Lacy was pulled down by Thomas Sweet-Escott, who built part of the present house, including the Great Hall, the windows of which were filled with Dutch and Flemish 17th century stained glass and some earlier English glass.
In 1932 this was offered for sale, along with many pieces of choice furniture, oak panelling etc. that had enriched the interior, and which had been brought together by the then owner and his predecessors.
Owners of Hartrow after Margaret Hay:
Bickham Escott, nephew of Margaret Hay 1731-1801 (inherited in 1753)