Family is the story of an aristocratic Chinese family at the beginning of the 20th century. The Kao family lives in a well-appointed compound in the city of Chengtu and enjoys all the material and cultural benefits the city has to offer. Three brothers in the Kao family, Cheuh-hsin, Cheuh-min, and Cheuh-hui, are the main characters of the story. Cheuh-hsin, the oldest, is married and has a son, and the other two brothers are students in their late teenage years.

Cheuh-hsin suffers greatly because he has never been able to stand up for himself. He is a big believer in filial piety, and he has always been willing to do what his elders tell him to do, even when his own dreams are sacrificed. For example, Chueh-hsin wanted to marry his cousin Mei and pursue a career in scientific academia, but his grandfather arranged a marriage with Jui-cheuh and got him a job in town. Chueh-hsin accepted the marriage and job quietly, but they have eaten away at him.

Cheuh-min is the middle brother. He is not as docile as his older brother, but he is not openly rebellious either. He is intelligent and gentle, and he falls in love with his cousin Chin. His love for Chin puts him in a very similar situation with his older brother, but he watches what happens to Cheuh-hsin and vows to stand up for himself and for Chin. He temporarily runs away in order to avoid a marriage that his grandfather arranges with the daughter of a family friend. His grandfather renounces the arrangement on his deathbed, and Cheuh-min is saved from Cheuh-hsin's fate.

The youngest brother, Cheuh-hui, is defiant and rebellious. He does not agree with most of the family's traditions and is never afraid to speak out against them. The elder members of the family learn that they cannot push Chueh-hui around as they have done to Cheuh-hsin. Cheuh-hui falls in love with one of the servants, Ming-feng. He even promises that he will marry her. When she is to be sent as a concubine to one of the Master Kao's friends, she drowns herself in the lake rather than live in such misery. Ming-feng's suicide pushes Cheuh-hui even farther into rebellion.

Tragedy heaps upon tragedy as Cousin Mei dies from a broken heart and then Jui-cheuh, Cheuh-hsin's wife, dies in childbirth after the family agrees to send her outside of the city to give birth in order to protect the corpse of the Venerable Master Kao. Each tragedy pushes Cheuh-hsin deeper into despair. At the same time, Cheuh-hui decides that he cannot live in such an immoral environment. After Jui-cheuh's death, Cheuh-hui decides that he must get away from the family in order to survive morally. He gets his older brother to help him financially and leaves for Peking where he hopes to start a new life that is closer in line with his ideals.