Willie Shakespeare, in his fabulous tragedy "Richard III", suggested that the Tower had a very earlier orige, because it have been built by Julius Caesar
But it is a legend, not the historical truth
I think that the Tower was built by William I, the Conqueror who came from Normandy. He wished to erect an impregnable fortress using Caen stone, as a great demostration of his power. Later, king Richard the Lionheart ordered to enclose the Tower with high embattled walls of stone.
People who was sent to the Tower...The first arrival of royal prisoners happened in the reign of Edward III: he ordered to conduct his own brother-in-law, king David II of the Scots, to the Tower after he had been taken prisoner at the Battle of NevilleĀ“s Cross. King John of France and his son prince Philip, both of them captured in battlefield by the famous Black Prince of Wales, were also hosted at the Towers before the french court payed a great ransom.
The lancastrian king Henry IV was also a prisoner at the Tower during the reign of his cousin Edward IV.
I remember, specially, George duke of Clarence and poor Margaret of Salisbury, and also the son of Margaret, Henry lord Montagu, with his wife and children. The Queen of the Nine Days, Jane Grey, was also imprisoned there with her husband Guildford Dudley. But also Anne Boleyn, her first cousin Catherine Howard and lady Jane Rochfort. Elizabeth I and Robert Dudley, later earl Leicester, were shortly imprisoned at the Tower. A lot of years later, Robert Devereux, earl of Essex, was there.