Ginger Man follows Sebastian Dangerfield, a young law student from a wealthy American family, as he drops out of Dublin University law school, makes a mess of his early marriage, and drinks his way through the Dublin pubs. Dangerfield is a womanizer whose only ambition is not to work. He finds himself in squalor and poverty, moving to London, wrecking relationship after relationship, and finally discovering it will be twenty more years before he can expect any inheritance from his wealthy father's death.

As the novel begins, Dangerfield is struggling while studying law at Dublin University and living in poverty in a spooky, rundown house on the edge of a cliff in Ireland. His wife and baby daughter are away, and he has his friend O'Keefe over. Dangerfield buys food and liquor on credit, and they get drunk and wreck the house. Marion comes back and is furious with his drunkenness. After the backyard falls down the cliff in a storm, the Dangerfields move to a tiny house in Dublin. The plumbing is bad and the toilet pipes break over the kitchen. Dangerfield is broke, and he and Marion fight.

Dangerfield meets a woman named Chris who works at a local laundry, and they begin having an affair. Dangerfield starts feeling guilty, though, and goes back to Marion for a time before getting bored and frustrated and going back to find solace with Chris. One day, Dangerfield leaves his fly open on the train, and when he gets home, he has a big fight with Marion. She has written to Dangerfield's wealthy father, complaining about how he mistreats her and their child. Dangerfield is furious, thinking that he'll be disowned. However, Dangerfield's father sends Marion money. She threatens to leave him, and Dangerfield gets drunk and smashes up a bar. He takes refuge at Chris's, but she is upset at his lying about what's happened. He leaves her house the next day, only to find that Marion's moved out.

Dangerfield follows Marion to her new address and insinuates himself in the nice little house she's rented. They rent out a room to a woman named Lilly Frost. For a while, Dangerfield is happy but continues with his drunken behavior. One day, he finds Marion has taken all the money and moved away. He begins an affair with Lilly. Meanwhile, his former landlord and other debts have followed him. While drunk at a party, he meets yet another woman, named Mary. He gets her to give him money to move to London, and he leaves Lilly.

In London, Dangerfield hooks up with some old friends. He writes Mary to come join him. His father dies, and he expects a big inheritance but discovers that it will be twenty years before he will see a penny of the money. When the money goes away, he fights with Mary and drives her away. However, at the end of the novel, he meets an old friend who's now wealthy and flush with money. Dangerfield makes up with Mary and plans for a future with her.