What did Hugo and James agree was an ironic twist in their love relationships in this book?

Hugo and James agreed that it was ironic that they each loved the other sister who did not care for them.