Loder, in his History of Framlingham, thus describes the former state of the structure: “This castle, containing an acre, a rood, and eleven perches of land, within the walls now standing, but anciently a much larger quantity before the walls enclosing the same were demolished, was in former ages very fair and beautiful, standing within a park (long since disparked) on the north side of the town; fortified with a double ditch, high banks, rampires, and stone walls 44 feet high and 8 feet thick; in these walls were thirteen towers, 14 feet higher than the walls, built four-square—whereof two were watch-towers, one looking towards the east and the other towards the west: and the rooms within the castle were very commodious and necessary, capable to receive and contain abundance of people.
“In the first court was a deep well, of excellent workmanship, compassed with carved pillars, which supported its leaden roof, and though out of repair, was in being in the year of our Lord 1651. A chapel stood in the same court, adjoining to the east watch-tower; which in the reign of Henry viii. was hung with cloth of arras, of the history of Christ’s passion; and a lamp of the value of seven shillings was usually burnt before the altar there. On the side of the court, towards the west watch-tower, was the hall, covered with lead; and over the gate thereof were formerly cut in stone the arms of Brotherton impaled with Bouchier, quartering Louvain, supported with a lion and an eagle. Divers other arms there were in the rest of the buildings, some cut on stone and some on timber, to be seen in the year of our Lord 1651—as Bygods, Brothertons, Seagraves, Mowbrays, Howards, and St. Edmund’s, the king and martyr. Between the hall and chancel, fronting the great castle gate, was a large chamber, with several rooms, and a cloyster under it, pulled down A.D. 1700; for which, when standing, in the reign of King Henry viii., there was one suit of hangings of the story of Hercules; which are supposed to be those still remaining at the seat of Lord Howard, of Walden.