Why does Genji treat the Third Princess coldly after she gives birth to a son?

The Third Princess gives birth to a boy and names him Kaoru. Genji becomes cold toward her and does not want to invite her or the child into his home. The Third Princess had an affair with Kashiwagi and Genji knows that the child is not his.