Well, that's just a myth. He was a man, after all... It seems that he was faithful to his wife and, if he wasn't, then he was absolutely discreet, as there are no news (and absolutely none!) of an extramarital affair. On the other hand, he fathered three children before his marriage and at least one after Isabel's death.