When Stephen is allowed to leave his father's apartment to go to the family's vacation home in Tarumi, Japan, on his own, he counts it a victory. To celebrate, he buys a journal and begins making entries in the journal to document his journey and his time in Tarumi. It's those journal accounts that become the book, Samurai's Garden.

The book begins with Stephen's account of his diagnosis of tuberculosis, his family's reaction, and the events that brought him to be on a train bound for Tarumi. As Stephen tells it, his parents were searching for a place for Stephen to recuperate but were also looking for a way to get him away from his younger sister, Penelope, before he infected her. When Stephen arrives in Tarumi, he's met by Matsu, caretaker of the family property. Stephen first believes that his time in Tarumi will be too quiet and expects boredom. He plans to paint and expects to be homesick at once. But he soon discovers that he is entranced by the quiet and forms a friendship with Matsu. That friendship is enhanced when Stephen meets Sachi—the woman Matsu helped through her discovery of her own infection with leprosy. Matsu quickly becomes a complicated character and Stephen is inextricably caught up in Matsu's life, which is interwoven with Sachi's own.

Matsu's younger sister was among those in the village infected when the village experienced an outbreak of leprosy. All who were infected felt they had dishonored their families, and Matsu's sister took her own life in an effort to bring honor to her family. Sachi says she tried to do the same, but didn't have the courage. It was Matsu who helped Sachi find her way to Yamaguchi—the Village of the Lepers. Matsu's best friend is Kenzo. Kenzo was betrothed to Sachi until her discovery of the leprosy. When Sachi ran away, Kenzo sent messages to Sachi by way of Matsu, but never again saw her until he happened upon her one day at Matsu's house. Kenzo soon committed suicide himself.

Throughout the story, there are two gardens that are important to Matsu, Sachi, and Stephen. Matsu's garden is filled with water and color. He has a fish pond and a bridge, and the three work hard at the garden after a storm causes major damage. The second garden is Sachi's and is, in many ways, the opposite of Matsu's. Sachi's garden is dry and there's a single flower that blooms in the center of her garden. Rocks are arranged to resemble water and Sachi rearranges the rocks to resemble ripples from a single point or from a wave. She explains that she finds solace in the arrangements and in the act of arranging. As Stephen befriends Matsu and Sachi, he discovers that both are complicated humans with loves, dreams ,and desires. He also learns about himself through the process.

Stephen hears the news of the escalating war between China and Japan. He worries about his family's safety, as the Japanese advance toward Hong Kong. The war means that some Japanese in Tarumi are hostile toward Stephen because he is Chinese. He meets a young woman, Keiko, and is instantly attracted to her. She returns the affection, but her father hates Stephen because of his race. When Keiko's brother is killed serving with the army against China, Keiko says she can no longer see Stephen.

Stephen is also forced into an adult role as he learns that his parent's marriage is in jeopardy. Stephen's mother tells him of his father's affair and asks him to intervene. Stephen avoids the situation as much as possible, but admits that his image of his father changed once he realized that there really was an affair and that his father didn't intend to end it. Stephen reluctantly prepares to go home . He knows that he must return to his mother in Hong Kong, but he has become caught up in his friendships in Tarumi. Though he toys with the idea of remaining at the beach house, he does return home after more than a year away. The book ends as Stephen boards the train—the first leg on his journey back to his family and his life before he was diagnosed with tuberculosis.