What irony does Lavender see in his own life in the novel, Little Big Man?

Lavender is a freed slave, working as gardener and handyman for the Pendrakes. He chafes under the restrictions of his wife, finding it ironic that he has been freed from the slavery of men only to find himself the slave of a woman.