Why is it a big deal? Because in the Orthodox Church that kind of thing is for a husband and wife only. It is in a very real way a binding of souls, because in marriage two people are made into one person. In a very real way, if we 'sleep' with someone, we have become eternally a part of that person.
Now of course there are people, including 'nominally' Orthodox, and probably even fairly sincere Orthodox who have not kept themselves pure until marriage, but that is something that will always be on their consience until it is Confessed.
Whether he did such things or not is irrelevant to his Sainthood, so even if he did it doesn't tarnish his halo. One of the most beloved of all our Saints is Saint Mary of Egypt who became a harlot at twelve, but later repented.