American novelist Henry James, now fully entrenched in the society of England, seeks to recover from a series of professional disappointments and personal tragedies. When his play, Guy Domville, ends in humiliation, Henry flees to Ireland, where Lord and Lady Wolseley entertain him. Unfortunately, due to his family's low origins, these nobles cannot fully accept him either, and Henry suffers further embarrassment at their table.

Returning to England, Henry throws himself into his writing with full vigor. The idea of a ghost story begins to intrigue him, and he develops this concept, using himself and his sister Alice as inspiration. This proves to be a habit of Henry's, appropriating the lives of family and friends for his short stories and novels. Failing his family and friends in their time of need is another habit, as the novel slowly reveals Henry's guilt over the deaths of his cousin, Minny Temple, his sister, Alice, and close friend, Constance Fenimore Woolson.

With Alice, Henry could not protect her from the world and blames himself for not preventing her bitterness. This leads him to be very attentive to and protective of little girls later in life. Sadly, though, despite this feeling, Henry did not provide his own sister with the utmost care. When Alice falls ill, Henry attends to her and takes her around Europe, but only so much as his own convenience will allow. Similarly, he does not visit Minny Temple when she lay dying, despite the fact he was residing near her quarters in Italy during the last days of her life. Constance's case, in particular, is a source of grief, since Henry could have prevented her death simply by visiting her, but he refused to make the time. They had developed an intense friendship, and Constance came to rely on Henry for her happiness. Henry made a comment about visiting her in Venice during the winter, and when he later made it clear that he had no real intention to do so, Constance's seasonal depression sent her over the edge. Constance committed suicide by jumping off her apartment building.

While Henry struggles with the demands of social decorum, his own homosexuality, and a contentious relationship with his famous brother William, the deaths of the three witty women overshadows his every action. Henry gains a degree of confidence and security when he moves from his rented London studio to a more peaceful and permanent residence at Rye and employs a Scottish typist to facilitate his writing once his hand begins to deteriorate.

Finally, coming to some degree of resolution with Constance's suicide, Henry meets the young sculptor Hendrik Andersen in Rome, and they begin a flirtation. While Henry does not manage to advance the relationship before Andersen travels to America, he does make some progress and gains confidence in his abilities as a suitor. Later, William visits Henry at Rye, and they resolve much of their bickering. Henry attends to his ailing brother, taking him to the best heart surgeon in London. William and his wife Alice also tell Henry that their psychic has given them a message: Henry's mother is watching over him, and he is not alone. Henry is not normally given to accepting the word of mediums, but this woman had specifically mentioned two significant objects in his home, a painting from Constance Fenimore Woolson's apartment and a bust sculpted by Hendrik Andersen. Once William retires for the night, Alice gives Henry another message from the psychic: he will be the last of the James brothers, and Alice will be by his side when he dies.