The mound on which the castle stood was originally of Celtic construction, but was afterwards converted into one of the fortresses which the Normans built in the eastern part of Cornwall as rallying-points in case of any sudden insurrection among the “West Welshmen.”  The occupation of the fortress by the Normans was the immediate cause of the foundation of the town of Lostwithiel, to which a charter was granted in 1196 by Robert de Cardinan, the then owner of the castle and the surrounding country.

An exchequer deed showed how the castle and town of Lostwithiel came into the possession of the Dukes of Cornwall: 

Know ye present and to come that I, Isolda-de-Tracey, daughter and heir of Andrew de Cardinan, have granted to Lord Richard, King of the Romans, my whole Manor of Tewington....  Moreover I have given and granted to the aforesaid Lord the King, Castle of Restormell and the villeinage in demesne, wood and meadows, and the whole Town of Lostwithiel, and water of Fowey, with the fishery, with all liberties, and free customs to the said water, town, and castle, belonging.  Whereof the water of Fowey shall answer for two and a half knights fees (a “knight’s fee” being equal to 600 acres of land).

In the year 1225 Henry III gave the whole county of Cornwall, in fee, to his brother Richard, who was created Earl of Cornwall by charter dated August 12th, 1231, and from that time Restormel became the property of the Earls of Cornwall.  Afterwards, in 1338, when the Earldom was raised to a Dukedom, the charter of creation settled on the Duchy, with other manors, the castle and manor of Restormel, with the park and other appurtenances in the county of Cornwall, together with the town of Lostwithiel:  and it was on record that the park then contained 300 deer.  Richard, Earl of Cornwall and King of the Romans, caused extensive alterations and improvements in the castle at Restormel, and often made it his residence, and kept his Court there.  He was elected King of the Romans or Emperor of the Holy Roman Empire at Frankfort on January 13th, 1256, and crowned at Aix-la-Chapelle, November 27th, 1257.  Edward the Black Prince, upon whom the Dukedom was confirmed when only seven years old, paid two visits to Restormel.  The first of these was in 1354, possibly while his expedition to France was being prepared at Plymouth, and the second in 1363.

In the time of the Civil War the commanding position of the castle caused it to be repaired and held by the Parliamentarians; but after the disastrous defeat of their army under the Earl of Essex in 1644 it was garrisoned by Sir Richard Grenville for the King.  In recent times it was again visited by royalty, for on Tuesday, September 8th, 1846, the royal yacht Victoria and Albert sailed into Fowey and landed a royal party, who drove to Restormel Castle.  It revived old memories to read the names of the party who came here on that occasion, for in addition to Queen Victoria and her husband, Prince Albert, there were the Princess Royal and the Prince of Wales, Lady Jocelyn, Miss Kerr, Mdlle.  Geuner, Lord Spencer, Lord Palmerston, Sir James Clark, Mr. Anson, and Col.  Grey.