From Survey of London: volumes 29 and 30: St James Westminster, Part 1 (1960) by F. H. W. Sheppard, at British History Online:

St. James’s Market was established under the aegis of the Earl of St. Albans to serve the growing number of people who had come to live in the new buildings in the vicinity. In March 1663 and again in the following June inquisitions were held at the Green Dragon tavern in Pall Mall to enquire into a proposal that three markets should be held each week in St. James’s Fields and two others in the Haymarket. It was decided that these would not be prejudicial to the interests of the King or of the merchants and traders in the area, and so on the following 21 November the King granted by letters patent the right to hold the markets to John Hervey and John Coell, as trustees for the Earl of St. Albans.

Read more.

The map location is taken from this map of 1681-2.