What is the order of power in the Derby household?

Obviously, Clay and Mr. Derby have the most power. Then, the slaves and women are treated pretty similarly. Clay speaks just as harshly toward Polly as he does toward Noah and Amari, and he tells her to keep her mouth shut. However, neither man ever hits Polly, as they do Noah and Amari, so her race, not her sex, must in some way make her seem superior to the slaves.