I read that Serge was quite pocessive of Ella. While he encourged her to stand with officers during parties (instead of chatting with the ladies), he made sure that she and Paul were not overly close. However Ella and Paul did became close and when he married Alexandra of Greece, she accepted her as a sister. However Ella was very disapointed in Paul over his later mistress Olga (later Princess Paley).
It's conjecture how close Ella & Paul were before his marriages. There is no evidence that they were engaged in an adulterous affair. Certainly, Ella had no sympathy for Paul after he broke his pledge to the emperor, devastated his brother Serge, and eloped with his mistress. Apparently, Ella did not want Marie & Dmitri to become permanent sqatters in her household & deeply resented her husband's unqualified love for them. According to Marie Pavlovna Ella never forgave Paul.
Not conjecture is Serge's strong feelings for Paul's wife Alexandra of Greece.
Marie Pavlovna, quote:
"My uncle was profoundly attached to my mother. The early death which came to her, as I have said, at Alinskoe, left him with a sense of bereavement for which he could never be consoled. He ordered the room in which she had spent her last hours to be kept intact, exactly as it was when she died there. He had the rooms locked, & throughout the remainder of his life himself guarded the key, allowing no one else to enter."
Marie Pavlovna wrote that her Aunt Ella only tolerated Paul & Alexandra's children in her house because that was what her husband wanted. The exact reason for Ella's ("confessed") jealousy is unclear.
If Serge was possessive of his wife, there is evidence that Ella was even more possessive of her husband.