The buildings erected by Reinfrid under the Norman influence then prevailing in England must have been a slight advance upon the destroyed fabric, and we know that during the time of his successor, Serlo de Percy, there was a certain Godfrey in charge of the building operations, and there is every reason to believe that he completed the church during the fifty years of prosperity the monastery passed through at that time.  But this was not the structure which survived, for towards the end of Stephen’s reign, or during that of Henry II., the unfortunate convent was devastated by the King of Norway, who entered the harbour, and, in the words of the chronicle, ’laid waste everything, both within doors and without.’  The abbey slowly recovered from this disaster, and if any church were built on the ruins between 1160 and the reconstruction commenced in 1220, there is no part of it surviving to-day in the beautiful ruin that still makes a conspicuous landmark from the sea.

It was after the Dissolution that the abbey buildings came into the hands of Sir Richard Cholmley, who paid over to Henry VIII. the sum of L333 8s. 4d.  The manors of Eskdaleside and Ugglebarnby, with all ’their rights, members and appurtenances as they formerly had belonged to the abbey of Whitby,’ henceforward belonged to Sir Richard and his successors.  Sir Hugh Cholmley, whose defence of Scarborough Castle has made him a name in history, was born on July 22, 1600, at Roxby, near Pickering.  He has been justly called ‘the father of Whitby,’ and it is to him we owe a fascinating account of his life at Whitby in Stuart and Jacobean times.  He describes how he lived for some time in the gate-house of the abbey buildings, ’till my house was repaired and habitable, which then was very ruinous and all unhandsome, the wall being only of timber and plaster, and ill-contrived within:  and besides the repairs, or rather re-edifying the house, I built the stable and barn, I heightened the outwalls of the court double to what they were, and made all the wall round about the paddock; so that the place hath been improved very much, both for beauty and profit, by me more than all my ancestors, for there was not a tree about the house but was set in my time, and almost by my own hand.  The Court levels, which laid upon a hanging ground, unhandsomely, very ill-watered, having only the low well, which is in the Almsers-close, which I covered; and also discovered, and erected, the other adjoining conduit, and the well in the courtyard from whence I conveyed by leaden pipes water into the house, brewhouse, and washhouse.’

In the spring of 1636 the reconstruction of the abbey house was finished, and Sir Hugh moved in with his family.  ‘My dear wife,’ he says, ’(who was excellent at dressing and making all handsome within doors) had put it into a fine posture, and furnished with many good things, so that, I believe, there were few gentlemen in the country, of my rank, exceeded it....  I was at this time made Deputy-lieutenant and Colonel over the Train-bands within the hundred of Whitby Strand, Ryedale, Pickering, Lythe and Scarborough town; for that, my father being dead, the country looked upon me as the chief of my family.’