The Navy Office moved to Greenwich Palace for safety during the Plague.
The overlays that highlight 17th century London features are approximate and derived from Wenceslaus Hollar’s maps:
Open location in Google Maps: 51.483013, -0.006695
"Thursday 24 August 1665
"... At noon down to Sir J. Minnes and Lord Bruncker to Greenwich to sign some of the Treasurer’s books, and there dined very well; and thence to look upon our rooms again at the King’s house, which are not yet ready for us."
I read this to say that the Navy Offices were to be in the King's House at Greenwich Palace. Pepys is never this specific on the location again.
Chart showing the number of references in each month of the diary’s entries.