I wouldn't say his life was quiet and discreet. It might have been after he was sent to the Castle of Klesheim in 1875 after leading a scandalous life in Vienna. In fact, the Emperor forced him to go there because he had threatened a man in a spa after he violently refused to have sexual intercourse with him.
He was also a gossipy man, always talking about everyone. He was in very good terms with Sisi, until she realised she was also one of the victims of his poisoned tongue.
In 1914 he was officially declared mad, although I don't know if this was true, it was just an illness produced by old age or a way of legitimating the Emperor's decision. As the declaration was published 39 years after he was expelled from the Court, I would say the most plausible option would be the second one.