Afternoon of the Elves is the story of a special friendship between two girls, which blossoms because of a "magical" elf village created in the backyard of one of the girls.

Hillary Lenox is nine years old, and the protagonist of the novel. She has an ideal home life, she is obedient and kind, and she is well-liked at school. She strikes up a unique friendship with Sara-Kate Connolly, eleven years old, who is in many ways the opposite of Hillary. Sara-Kate dresses oddly, she is gruff and short-tempered, she is disliked by Hillary's classmates, and she behaves in eccentric ways.

Sara-Kate shows Hillary what she claims is a magical elf village built in the backyard of her home. There are tiny homes made of leaves and twigs, a well with a bottlecap for a bucket, and a Ferris wheel made out of two bicycle wheels. Hillary is enchanted by the prospect of a community of magical elves. Over the next weeks, Hillary aids Sara-Kate in observing the village, gathering food for the elves, and other duties. Though Hillary never sees an elf, she swears they exist. Sara-Kate is full of wisdom about the habits and culture of the elves, but alarmingly secretive about her own life, especially her never-seen mother. The girls grow close in their mutual passion for the village.

One day, Sara-Kate doesn't show up at school, and after some time, Hillary dares to creep into Sara-Kate's home, a place into which she has never been invited. There, she discovers Sara-Kate clutching her sickly mother in a rocking chair. The sight traumatizes Hillary, and Sara-Kate yells at Hillary to leave and never come back. Hillary attends school like normal for a time, but can't shake the image of Sara-Kate.

Hillary returns to Sara-Kate's home, not mentioning the mother, and they decide to repair the elf village, which has been destroyed by neglect and a snowfall. The two work hard and restore the village. Sara-Kate reveals a lack of money, and therefore of food, and Hillary volunteers to help. Hillary steals money from her mother's purse, buys groceries, and steals a pack of bologna from the grocer. The two girls have a party in Sara-Kate's living room making bologna sandwiches.

Mrs. Lenox knocks at the door, wondering where Hillary is. She sees the alarming neglect of the home, and demands to see Sara-Kate's mother. The physical and mental illness of Mrs. Connolly is exposed. As Hillary sadly watches from her own home, Sara-Kate's home is invaded over the next days by many strangers, who clean it up and eventually sell it to another family. Rumors in the newspapers claim that Sara-Kate has moved away to Kansas, and that her mother has been institutionalized.

Still clinging to the magic of the elf village, Hillary tells her mother that she intends to rescue the elf village, and build it in a weedy patch of their own backyard.