Well it was orignally built by Henry VIII merely as a retreat next to his newly aquired hunting ground (St James's) and it wasnt changed much untill Whitehall burned down and then it became the Base of Soverign. (Whitehall itself althought the largest royal palace in the world was described as a miseray of different ages and style buildings in no order or pattern). The lay out of St James's was distrupted further when it caught fire in the late 1700's - it was rebuilt with the additions of Clarence House and York House (now Lancaster House) - this was the first stage of moving from St James's to Buckingham.
What i thinkam trying to say is that St James's was never intended as the seat of the sovereign merely a retreat. Untill another Palace burnt down and it itself caught fire, which ultimately made it "all over the place" as it is now. Not to mention the several alterations that it has undergone since its last major rebuild with the buildings being divided, reopened and divided again as different apartments and offices have been created. St James's never had the planning of a grand palace like Buckingham Palace did, even though it shared the role of Buckingham Palace for many years.