Daily entries from the 17th century London diary
Located roughly where Buckingham Palace now stands, at what was then the western edge of London, it was bought by Baron Arlington in 1665. It burned down in 1674 and rebuilt.
Goring House marked the western edge of London in 1660 where Buckingham Palace now stands.
Goring House
Possibly the first house erected within the site was that of a Sir William Blake, around 1624. The next owner was Lord Goring, who from 1633 extended Blake’s house and developed much of today’s garden, then known as Goring Great Garden. He did not, however, manage to obtain freehold interest in the mulberry garden. Unbeknown to Goring, in 1640 the document “failed to pass the Great Seal before King Charles I fled London, which it needed to do for legal execution”. (It was this critical omission that helped the British royal family regain the freehold under King George III.)
http://en.wikipedia.org/wiki/Buckingham_Palace#Goring_House