It's hard to believe that Sissi would feel interested at having an affair (physical relationships) at knowing her character only a bit, she disliked intimacy with men (she showed it very well at allowing her husband having mistresses) plus not wanting to be pregnant anymore. Two, i doubt that she would like to take the risk of being involved on a scandal after looking the problems of her sisters Marie and Sophie. Three her lonely character was other thing.
I read one time about her that she liked her letting be loved but not letting be touched.
Of course, I'm not saying that she didn't feel certain love for some men, but was only platonic and idealized more that a real affair.