While I Was Gone is a story of how the forgotten past and lessons not learned come back to influence the present and future. Jo Becker has an almost idyllic life with a devoted husband and three grown daughters with talent and promise. She has not told the full story of her past, but it is a past that will return.

Jo has a successful veterinary practice in Adams Mills, a rural community in western Massachusetts. She and her husband Daniel, a minister, live in a picturesque farmhouse with plenty of land around to take their dogs on long walks. Jo enjoys her home and her career until one day mention of a person from her past returns and makes her remember events from more than twenty years earlier.

In the late 1960s, Jo had been previously married and been employed as a teacher in Philadelphia. She grows first dissatisfied with her job and then her marriage. She quits her teaching job and takes a job as a waitress in a bar. The environment in the bar is unlike any she has experienced, and she longs for more freedom and adventure. After lying to her mother and her husband about her intended destination, Jo takes a bus to Boston. Once there she answers an ad regarding a vacancy in a house inhabited by several other people her age. She uses a fake name but is accepted as a new housemate, and soon Jo finds the life she had been seeking in the free-spirited lifestyle of the house.

Jo enjoys the company of all the housemates, and she enjoys the endless string of parties. She grows particularly close to one resident, a young woman of similar age named Dana. Dana is in many ways the kind of person that Jo wants to become. She is free in her granting of affection without becoming entangled in traditional expectations of a young woman. Jo's happy life at the group house is cut short after the brutal murder of Dana.

Years later in Jo's life in Adams Mills, she reencounters one of the group house residents, Eli, who needs veterinary care for his dog. Jo is at first delighted to see an old acquaintance, and then she finds that she is attracted to Eli. Jo increasingly fantasizes about having an affair with Eli, and when the two agree to meet to discuss old times and the tragedy surrounding Dana's death, Jo's fantasies only grow. For their second meeting Jo chooses a hotel bar. The setting and opportunity seem right for the start of an affair, but Jo's attraction to Eli abruptly ends when he shocks her by confessing to having murdered Dana all those years ago.

Jo tells her husband about the confession, and in doing so informs him of her intention to have an affair. Jo then has to find a way to repair her marriage. She must also decide what to do with the information Eli has given her, and she must do so in a way that minimizes further damage to her family.