The main entrance to Whitehall Palace, situated on Whitehall, next to the Banqueting House, roughly on the junction with today’s Horse Guard’s Avenue. See this map and pp.480-1 of the Latham & Matthews Companion.
The overlays that highlight 17th century London features are approximate and derived from Wenceslaus Hollar’s maps:
Open location in Google Maps: 51.504809, -0.126085
Chart showing the number of references in each month of the diary’s entries.