Why does Flashman suspect Elspeth's child is not his in the novel, Flash for Freedom!?

Flashy doubts he is the biological father of his wife's unborn child. Like himself, Elspeth enjoys sleeping around and they seem to have an unspoken agreement that their marriage is open.