Berkeley House was later demolished for the construction of Devonshire House (more at Wikipedia).
The overlays that highlight 17th century London features are approximate and derived from Wenceslaus Hollar’s maps:
Open location in Google Maps: 51.507270, -0.142756
Berkeley House, which John, Lord Berkeley, erected at a cost of over £30,000 on his return from his tenure of the viceroyalty of Ireland...was constructed from 1665 to 1673. The house was later occupied by Barbara Villiers, Duchess of Cleveland, a mistress of Charles II. The house, a classical mansion built by Hugh May, had been purchased by William Cavendish, 1st Duke of Devonshire in 1697 and subsequently renamed Devonshire House.
On 16 October 1733, the former Berkeley House, while undergoing refurbishment, was destroyed by fire. The cause was attributed to careless labourers. http://en.wikipedia.org/wiki/Devonshire_House#The…
Berkeley House, PICCADILLY, stood where Devonshire House now stands, on the site of a farm called "Hay Hill Farm," a name still preserved in the surrounding streets. It was designed about the year 1665 by Hugh May (the brother of Bap. May) for John, Lord Berkeley of Stratton (d. 1678), the hero of Stratton fight, one of the minor battles of the Civil War under Charles I. The gardens were very extensive, including Berkeley Square and the grounds now attached to Lansdowne House, as well as those belonging to Devonshire House.
---London, Past and Present. H.B. Wheatley, 1891.
Devonshire House, earlier Berkeley House, Piccadilly is in the middle of this map http://www.motco.com/map/81002/SeriesSearchPlates…
Berkely House, Piccadilly
Chart showing the number of references in each month of the diary’s entries.