A famous painter, Thomas Bayber, reveals a portrait that he has kept hidden for more than 30 years of his career. The painting shows a younger version of Thomas with two sisters, Natalie and Alice, and is missing two panels. Thomas enlists an art critic, Dennis Finch, and an art appraiser, Stephen Jameson, to find the missing panels of the painting--and, in turn, to find the two Kessler sisters, with whom he lost touch more than 30 years before.Though Finch and Stephen initially don't get along, they agree to take on the task--Finch, out of a sense of duty to Thomas, and Stephen needing the help to re-invigorate his flailing career. Together they search for clues to uncover the mystery of the Kessler sisters, which is complicated further when Thomas has a heart attack and becomes virtually comatose.

In flashbacks, the truth about Thomas's relationship with the Kessler sisters is revealed. Thomas was their neighbor at a summer lake house, where he carried on a secret affair with 17-year-old Natalie, who has changed dramatically over the summer, becoming sullen and resentful of her family. 14-year-old Alice also has a crush on Thomas, and the two seem to have a connection, until she discovers the truth about him and Natalie. Several years later, Alice returns to the summer house and meets up with Thomas again. Thomas tells her she misunderstood his relationship with Natalie, which Alice believes, and the two begin a romantic relationship. Alice discovers that Thomas has been lying to her and leaves, stealing a figurine of a bird as she goes.

Alice soon discovers that she is pregnant with Thomas's child. Alice keeps this from him, and also doesn't reveal the truth about the father to Natalie. She gives birth in her parents' home, after which Natalie tells her that the baby was born stillborn, though Alice thinks she remembers hearing the baby cry. Natalie shortly thereafter sells the house and forces Alice to move to a town named Orion. Alice has rheumatoid arthritis, and her condition has worsened over the years, forcing her to be dependent on Natalie as her caregiver.

In modern day, unaware that Thomas is searching for her, Alice's complicated relationship with Natalie continues, even after Natalie's death. Going through Natalie's things, she discovers that Natalie was pregnant as a teenager but forced to have an abortion by their parents, which left her unable to have children. Fiercely jealous of Alice-- not only for being able to have a child, but having a child with her former lover Thomas-- Natalie lied to Alice about her baby. She raised the child as her own and sent letters and photographs to Thomas, taunting him with the child he would never meet. Alice discovers the last known address for her daughter and travels to Santa Fe, New Mexico, to meet her.

In the meantime, Finch and Stephen have managed to piece together the same information and meet up with Alice and her daughter, Agnete, in Santa Fe. Thomas dies before he can be reunited with Agnete and Alice, but Finch and Stephen are able to find the missing panels of the painting. In the process, they also discover that Stephen is Thomas's biological son, which is why he was chosen to search for the paintings. Stephen and Finch have bonded over the journey, which they also realize was part of Thomas's intent when they find his last letter. Finch, Stephen, Alice, and Agnete began the novel feeling alone but have now been brought together and have each other to depend on.