I know OTMAA were referred to with diminutive forms of their given names (ex: Tanya, Masha, Nastya) within their family
I'm not so sure, actually. With the exception of Maria and Anastasia, the written instances of OTMAA's diminutives are actually quite few.
What they used in daily speech is possibly another matter. I do know that at least one of the courtiers' memoirs points out that Maria was called "Masha" by her sisters, which implies that they did not
use other diminutives amongst themselves.
Which diminutive form did Alix use for her children
I'm aware of seven letters from the empress to Olga in which she uses "Olenka," one to Tatiana using "Tatianochka," and two to Maria using "Mashenka." These were all written between 1909 and 1910, and I've never seen those forms used elsewhere in the IF's correspondence or diaries. I don't recall the empress ever using any diminutives for Anastasia. For Aleksei she used English nicknames like "Sunbeam" and "Baby."
Were there forms that they generally didn't use?
I've never seen a single instance of "Olya," "Olishka," or "Tanya" in the imperial family's letters or diaries. Pretty sure I've never seen "Alyosha" either. The only place I see those diminutives are on fansites.
What about Nicholas?
Anastasia routinely signed her wartime letters to the tsar as "Nastasya," "Nastanka," and "Nastaska." However, I don't believe I've ever seen a written instance of the tsar using a traditional Russian diminutive to address or refer to any of his children.
Or OTMAA to each other?
The only usage of diminutives I've seen between the siblings are letters to and from Maria that use "Mashka" or "Masha." In letters they used their full given names. In their diaries they most often use initials to indicate each other, although Olga often refers to Anastasia as "Schvybs."
Also, is there a difference in meaning between the -sha and -shka endings? (Masha vs Mashka) Is one more endearing?
My understanding is that the longer the nickname, the more intimate its use becomes. So from least to most endearing: Masha, Mashka, Mashenka.