I wonder if when looking at the area from above (Google), you may be incorrectly thinking that the various vast green areas of Green Park, Hyde Park, etc. are part of the palace's grounds? Those are not actually. Buckingham Palace gardens are those immediately behind (north and west side of) the palace, and are not open to the public; they are totally walled in and private. Well, private when one considers the hundreds of staff and visitors wandering about every day.
The vast lawn and lake beyond on the west side is probably the least private parts, as someone walking there would be visible from a variety of areas of the palace. But the gardens to the north side (visible out the windows of the Queen and Duke's private apartments in that wing) are probably considered the "private gardens" for the royals. Looking closely on Google maps, I've noticed winding paths through the trees on that side, what appears to be a tennis court and maybe even a summer or tea house. That's where I suspect HM and HRH wander when they wish to have privacy outdoors when in residence at BP.