Illegitimate not at all! He was recognised by Henry VI as being "born within the marriage" and had full inheritance rights. Illegitimacy is a legal definition and not a biological term! Thus Edward IV too was definitely legitimate, french archer or no.
As to the question of whether the Prince of Wales was Henry's biological son, I have the following thoughts:
It was not unusual for couples to fail to conceive for several years after marriage: this happens. Other examples: Marie Antoinette, Anne of Austria, Catherine de' Medicis. This naturally leads to a lot of talk, especially if the husband is a bit odd!
It was definitely lucky for Margaret to have a child at this point. I wonder though, if the child was not Henry's why she didn't have any more children, just to make sure. Having only one son was really putting all her eggs in one basket, as she found out to her cost at Tewkesbury.
So to me, the fact she only had the one, suggests that Henry may have finally managed to father a child. The alternative would be that her adulterous relationship ended with the birth of the prince (she was 25), and she was never tempted again.