As for the idea that one or the other was infertile, it makes you wonder why then there was so much gossip surrounding their marriage which didn't arise in the case of other childless couples.
I think there was so much gossip about
their childlessness in particular for a couple of reasons. Firstly, there weren't many high profile couples of their exalted rank who remained childless. I really can't think of any other couple of the Imperial Family contemporary with Ella and Serge who were childless. So they must have stood out in their family because of this.
Secondly, Ella was touted as one of the most beautiful women of her time. Many men were probably jealous of Serge for having married her and were eager to spread gossip about him. The fact that they had no children was seen as just plain odd in those days, as all women were expected to want children, and all men wanted an heir to carry on their name and inherit their fortune, so the idea that a couple had no children
by choice was probably dismissed as quite preposterous.
Of course, infertility was a reality they knew about and I do believe this may have been the case with this couple. Hence the statement made by Alexander III that poor Ella and Serge would never know the joys of parenthood. The fact that Serge may very well have been homosexual doesn't factor in very much for me, as other gay men fathered children. They knew it was their duty.
The fact that Ella and Serge remained childless fanned people's imagination that they had an unhappy marriage or that something was wrong with one of them, or it fueled their speculation about Serge's homosexuality. This couple was already the object of gossip simply because they seemed to be such an odd pairing, and the fact that they remained childless did nothing to quiet people's assumptions that they had an unhappy marriage.