Elisabeth disliked most of Habsburg family except her husband, children and brother in law Maximilian.
But I mean, that only Ludwig Victor and Charlotte didn't like her. Ludwig Victor had very complicated and malicious charatcter, therefore many people from the court (not only Sisi) shivered against him an his intrigues. Franz Joseph said about his youngest brother, that he always meddled in affairs of other people.
I think, Charlotte considered her for something less, because she was far more rich bride (daughter of King Leopold I. of Belgium), then Elisabeth ever was. Charlotte was perhaps jealous, that despite of her qualities, Elisabeth was the Empress, not she.