Garden of Shadows by V.C. Andrews is the final book published in the Dollangager series, but a prequel to the first novel, Flowers in the Attic. Olivia Winfield is a tall, unhappy spinster when she meets Malcolm Neal Foxworth. Olivia believes Malcolm is her knight in shining armor, come to rescue her from her loveless life. However, instead of giving love, Malcolm opens a world of secrets and obsession. Olivia soon finds herself emotionally abused to such a degree that she participates in cruel acts she never thought herself capable of. Garden of Shadows will answer a lot of questions for fans of the Dollanganger series, but it will also reveal a few new truths that will create new questions.

Olivia Winfield lost her mother at a critical age and wishes she still had her to talk to, especially when it comes to the long line of men who avoid Olivia due to her unusual height. Olivia's father has encouraged her to get an education in anticipation of her inability to find a man willing to marry her. Olivia has just about given up on finding love when Malcolm Neal Foxworth walks into her life. Malcolm has come to discuss business with her father, but he is so impressed with Olivia's independence that he asks her to dinner. Olivia is so happy with Malcolm's attentions that she agrees to marry him after only a few dates. It never occurs to Olivia to be concerned that Malcolm has never attempted to so much as share a goodnight kiss.

After the wedding, when Olivia arrives at her new home, Foxworth Hall, she discovers that her husband has little interest in playing the devoted husband. Malcolm wants a woman who can run his home with little input from him, who can represent him suitably in public, and who can provide an heir to his empire. Malcolm has gotten all that, and in return Olivia has gotten a cold, disturbed husband and a lonely existence. The birth of her two sons brings Olivia unending joy, but only irritates her husband, especially when the doctor announces that Olivia will be unable to have any more children. Malcolm wants a daughter now that he has an heir and feels as though Olivia has purposely chosen to prevent him from having all his desires.

Three years into their marriage, Malcolm's father, Garland, comes home with his bride. Malcolm dislikes his father's laid-back attitude and resents his intrusion into both Malcolm's home and business. Malcolm also claims to dislike Garland's extremely young bride, who reminds him of his mother. Olivia believes that Malcolm dislikes Alicia because of the child she is expecting. Olivia, out of a misguided sense of loyalty and jealously, attempts to place Alicia in a dangerous situation in the attic to cause a miscarriage. Fortunately, Alicia is not harmed and she gives birth a short time later to a healthy boy, Christopher Garland Foxworth.

The two couples live together for three years. Over this time, the glaring differences between the two becomes more obvious. Garland and Alicia are highly affectionate, often disappearing for hours at a time in the afternoon to make love. Olivia and Malcolm are never affectionate with one another. Olivia is fascinated with Garland and Alicia, often finding herself listening to their lovemaking through the thin wall in her room or following them to the lake to watch their encounters there. One afternoon, Olivia follows Alicia to the lake only to find Malcolm attempting to seduce her.

A short time later, Olivia is drawn to Alicia's room by screams in the night. Olivia discovers that Garland has had a fatal heart attack after fighting with Malcolm. Malcolm had been in Alicia's room, forcing himself on her. Over the next few months, Malcolm goes to Alicia again and again. Alicia tells Olivia that it is rape, but Olivia finds herself wondering if Alicia only wants her to believe her innocent. Alicia also announces she is pregnant. Olivia talks her into having the baby in secret and giving it to her and Malcolm. When Alicia agrees, Olivia goes to Malcolm with all she knows. Olivia promises to hide Alicia and care for her, then to take the child as her own. However, in exchange for this benevolence, Malcolm is to set up a trust fund for both his and Olivia's sons.

Olivia cuts Alicia's hair to keep her as unattractive as possible to Malcolm. Over the months, Olivia is the only person Alicia ever sees. Alicia appears to have some kind of mental breakdown in which she imagines Garland is still alive. When the time of the birth comes, Malcolm attends to Alicia with a midwife, forcing Olivia to wait in her room alone, making noises to convince the servants of her own impending delivery. When the child is born, it is discovered to be a perfect baby girl. Malcolm names her Corinne, for his own mother.

Malcolm spoils Corinne, keeping Olivia as far from her day-to-day care as possible. Corinne grows to be a spoiled, but delightful child. When Corinne is blooming into adolescence, her eldest brother, Mal, is killed in a motorcycle accident. A few short months later, her other brother, Joel, is also killed in an accident. Olivia and Malcolm are beside themselves with grief. John Amos, Olivia's cousin and new butler to Foxworth Hall, helps them find solace in the Bible. When Alicia writes to inform them she is dying, Olivia and Malcolm agree to take on her son Christopher and put him through college. Christopher is a bright, delightful young man who immediately begins an affectionate relationship with them all. John Amos warns Olivia frequently that there is something unnatural about Christopher's relationship with Corinne, but she refuses to listen until the day she finds them making love in the Swan Room.

Olivia and Malcolm are stunned by Christopher and Corinne's relationship, especially since they are the only ones who know the two are half-brother and sister. Malcolm attempts to convince Corinne the inappropriateness of the relationship, but Corinne will not listen. Corinne and Christopher run away, taking a new last name and marrying. Olivia keeps tabs on them through a private detective and is informed each time they have a child. When Corinne writes to let them know of Christopher's death, Olivia makes the decision not to tell Malcolm about the children. In fact, Olivia comes up with a scheme, with John Amos's help, to hide the children until Malcolm's death.