Well what does one expect from a family with 6 young boys? Alexei himself wasn't exactly the most well mannered and he was only 1. I can only imagine what 6 would be like. Alexei's fragile health did probably have some roll but I'm not sure that was the main reason since she allowed him to horse around with other boys that were personally selected.
Alexander Mikhailovich's theory on the rift among the families, who were very close in the beginning, was the fact that Xenia was having healthy son after healthy son while Alexandra struggled to have one, and when she did he had hemophilia. I don't know how much of that is true but I personally think there is some truth to it. Also their disapproval of Rasputin later on made things worse on Alexandra's side.
There are many photos of Xenia's children with OTMAA, some as late as the war. Although these photos usually only include Nicholas and not Alexandra.