I believe the decision regarding Young Alfred's education was made by Marie, Duchess of Coburg, Ilyala, certainly not Queen Victoria. As far as I know, it was Prince Albert, anyway, who promised his brother that his second son (Alfred Snr.) could be his (Ernest's) heir.
Alfred Junior was removed from his family at the age of only eight to be educated for his future role and that family appeared to have no idea he was being ill-treated by his tutor and the bad company he got into later on.
What about Charles Albany then? Later on, when he was being groomed to be future Duke of Coburg, his mother and sister moved to be with him.