Burke's royal families of the world lists the following children of Nicholas and Alexandra (I've only put the years of birth of the children who lived):
1. Alexander b.1818
2. Maria b. 1819
3. A daughter, stillborn at Peterhof, 1820
4. Olga b. 1822
5. A daughter, b and d at St Petersburg, 4 Nov 1823
6. Alexandra b. 1825
7. Elisaveta b 7 June 1826; d young
8. Constantine b. 1827
9. A daughter, b and d at St Petersburg 17 October 1829
10. Nicholas b. 1831
11. Mihail b. 1832
I wondered whether 'Elisaveta' was mistaken for Nicholas' niece Elizabeth who was born in 1826, and died in 1845. However, Burke lists the latter in the proper place and as having born in Moscow on 26 May 1826 and marrying Adolf Duke of Nassau. So Burke clearly saw two Elizabeths somewhere - but it is very strange not to have a date of death. For the first stillborn daughter, in 'Romanov relations' a letter from Grand Duke Nicholas to his sister Anna says "You already know from Mama about the cruel accident which happened to my wife which was followed by a miscarriage" and in 'Chere Annette', their mother's letters to Anna, she says "our dear Alexandrine was delivered safely and easily of a baby girl, already two weeks dead." In October 1823 Maria Feodorovna wrote to Anna that "Charlotte [Alexandra] is a little indisposed with stomach cramps and being a little delayed but its not serious" and there is no further information on what might have been another stillbirth in November and Maria Feodorovna writes in the same month rather cheerfully about a visit Anna's husband was making at the time to St Petersburg. The birth of Elizabeth Mihailovna ('Elisa') is mentioned in May 1826, coinciding with the death of the Empress Elizabeth - but no mention is made of the birth of another Elizabeth to Alexandra in June. Nicholas writes to Anna in the autumn of 1829 with much information about the state of Alexandra's health, which was poor, but there was no mention of a pregnancy or any child. So I haven't been able to find any other supporting evidence for any of the stillbirths or daughters who allegedly died young except for the one in 1820.