This book, published in 1978, is at times written as a journal or in first person narrative, is the story of an eventful summer late in the life of Charles Arrowby, a famous man of the British theatre. Arrowby is an actor, playwright, director, and romancer of women.

The novel begins with Charles having just retired from what he tends to think of as the glittering superficiality of the theatre world and moving to what he also tends to think is the relative peace of coastal England. He has purchased a house near the sea, Shruff End, has moved in with very little, and seems resolved to let go of old emotional habits, practices and beliefs. He plans to live simply, eat purely, and think quietly.

Charles' friends from London seem to have other ideas. A series of letters and uninvited visitors repeatedly disrupts Charles' refuge, the most disturbing of which involve previous lovers. After a particularly emotional encounter, Charles suddenly realizes that a woman almost run down by the departing visitor's car is in fact his childhood sweetheart, Hartley. They had loved each other non-sexually as teenagers and vowed to marry each other, but shortly before Charles moved to London to begin theatre school, Hartley broke off their relationship and suddenly and completely disappeared. This loss has haunted Charles and all his other relationships for his entire life.

After seeing Hartley, Charles becomes obsessively focused on rebuilding their past innocent, purely loving relationship. Complications appear in the form of Hartley's emotionally violent husband and runaway son (Titus), Charles' repeatedly visiting ex-lovers, his strangely distant cousin James, and Hartley's own ambivalence about returning to the past. At one point Charles forcibly confines Hartley to his home, where she becomes nearly deranged in her desperation to return to her other life. Charles is finally persuaded to let her go.

Once Hartley has gone, life takes on a kind of forced normalcy at Shruff End - until, following a drunken evening with friends, Charles is pushed into the sea. He survives, imagining he was rescued by James. Shortly after he recovers, however, he receives another shock. Hartley's son, who had been living with him and whom he had come to regard as a son of his own, drowns in the sea. These traumas send Charles even further into fixated fantasies and imaginings about Hartley and her husband, whom he blames for both the attempt on his life and the death of Titus.

Eventually Charles learns that he was pushed into the sea by the angry ex-husband of one of his ex-lovers, that Hartley and her husband are emigrating to Australia, and that several of his friends whom he once believed he could absolutely control and manipulate are getting on with their lives. He also discovers, following James' sudden and slightly mysterious death, that he's been left heir to his estate. He sells Shruff End, moves to London, and as he sorts through his feelings and impressions and experiences, begins to move back into the life he left behind.