Why doesn't Lady Partridge like Pat Crowther in the novel, The Line of Beauty?

Lady Partridge doesn't like Pat Crowther because he talked all during dinner about racism, or "the colored question," as she put it.