If Mary had died at birth or during childhood, she would (most probably) have been succeeded by her father's second cousin, James Hamilton, 2nd Earl of Arran (c.1519-1575).
He was the son of James Hamilton, 1st Earl of Arran (d. 1529) and his second wife, Janet Beaton. His paternal grandmother, Mary Stewart (c. 1450-c. 1488) was the daughter of King James II. The validity of his parents marriage was questioned, but the Scottish Parliament declared him Heir presumptive in 1543(after King James V's death). Arran was made Governor of Scotland and ruled the war-torn country until 1554, when he surrendered the governorship to Marie de Guise. He was nicknamed the "Inconstant Governor," because he frequently changed his mind. He shifted Scotland's political alliances and even his own religion when he felt it was necessary. No one trusted him much.
Unlikely...
If the 1st Earl of Arran's second marriage had been found invalid, the Scottish throne might have gone to Matthew Stuart, 13th Earl of Lennox (1516-1571). His paternal grandmother, Elizabeth Hamilton was the sister of the 1st Earl of Arran, and thus the granddaughter of King James II. Lennox hated Arran. He had strong royal ambitions and wanted the governorship for himself. He married Lady Margaret Douglas, Henry VIII's niece, in a bid for English support. (Their son Lord Darnley ended up marrying Mary)
source: Oxford Dictionary of National Biography
