Henry was adamant that 1) he had to have Anne and 2) Anne would give him the male heir he so desperately needed. As such, he may have felt that he could 'afford' to not recognise Henry Carey (unlike Henry FitzRoy) since Anne was about to give him a healthy, unquestionably legitimate male heir - to recognise Henry Carey would be to officialise, in a way, his relationship with Mary Boleyn, and thus render his marriage to Anne questionable and make him a hypocrite (see: his divorce with Catherine of Aragon).
As for Elizabeth Howard, she gave her husband a child almost every year, so I'm guessing by 1509 or thereabouts she wasn't "all that". The rumours about her and Henry stretch back to the start of his reign, mainly because of the great favour her husband enjoyed at court. Thomas Boleyn was a popular courtier, highly educated, refined, ambitious, etc etc, the grandson of another influential courtier, the wealthy Earl of Ormonde, and he took active steps to furhter his daughters' education, so I don't think its too far-fetched to say Anne/Mary/George's charm etc came from their father rather than their mother.
Considering there was discussion between the Papacy and Henry about making Henry FitzRoy King of Ireland and marrying him to his half-sister the Princess Mary (in order to secure the succession), Catholic views on incest were quite different that what they are today: - 1) certain positions were deemed more sinful than certain combinations, missionary with a half-brother being "less bad" than other positions with a non-relative and 2) Pope could grant dispensations for pretty much anything except descendant-ancestor marriages. Even so I think its very far-fetched and highly unlikely Henry and Elizabeth Howard ever had an affair.
Someone said Henry had very few mistresses...I'm pretty sure that was not the case. Lady Hastings, Bessie Blount, Mary Boleyn, Madge Shelton, Anne Basset, a flemish noblewoman whose dowry he promised to pay but then didn't (Yvonne, Yvette, something like that), plus the little harem he kept at a house called "Jericho" and possibly many others.