Google Maps leads for a whole city

A list of every dentist, café or plumber in a city is where a lot of sales and research work starts. One search does not give you all of them, because Google Maps shows the places for the part of the map you are looking at. This guide searches a grid of points across the city, keeps each place once, and writes a CSV.

One search around a point

GET /v1/google-maps/places takes a query, as you would type it into Google Maps, and a point: lat, lng and radius_m in metres.

curl -H "Authorization: Bearer $SCRAPEFIELD_KEY" \
  "https://api.scrapefield.com/v1/google-maps/places?query=dentist&lat=41.1496&lng=-8.6110&radius_m=1000"

Each page holds up to 20 places, each a full listing: name, category, formatted_address, location, rating, user_ratings_total, formatted_phone_number, website, business_status and the rest, in the places reference. A page costs 3 credits: $2.34 per 1,000 pages on the smallest pack.

Two things to know first

• The radius is not a fence. It sets how much of the map the search looks at, and Google also shows places just outside it. In a test on 4 October, a 2 km search in central Porto returned places up to 6.7 km away. Filter by location yourself, as the script below does.
• How many pages a search has is Google’s choice. Coffee shops around one point in Manhattan gave 7 full pages, 140 places; dentists within 2 km of central Porto stopped after one. When a search has no more, meta.next_cursor is null. A later page that comes back empty is not charged.

A whole city

Search a grid of points across the area. Points 1.4 times the radius apart make neighbouring searches overlap. So a place is often found more than once: keep it by its place_id. In Python:

import csv, math, os, requests

BASE = "https://api.scrapefield.com/v1/google-maps/places"
HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPEFIELD_KEY']}"}

def search(query, lat, lng, radius_m, pages=3):
    # up to `pages` pages of places around one point, and whether there were more
    found, cursor = [], None
    for _ in range(pages):
        params = {"query": query, "lat": lat, "lng": lng, "radius_m": radius_m}
        if cursor:
            params["cursor"] = cursor
        body = requests.get(BASE, params=params, headers=HEADERS).json()
        if "error" in body:
            raise RuntimeError(body["error"]["message"])
        found += body["data"]
        cursor = body["meta"]["next_cursor"]
        if cursor is None:
            break
    return found, cursor is not None

def grid(south, west, north, east, step_m):
    # points across a box, step_m apart
    lat = south
    while lat <= north:
        lng = west
        while lng <= east:
            yield lat, lng
            lng += step_m / (111_320 * math.cos(math.radians(lat)))
        lat += step_m / 111_320

south, west, north, east = 41.135, -8.655, 41.175, -8.585  # central Porto, about 6 by 4.5 km
radius = 1_000
places, crowded = {}, []
for lat, lng in grid(south, west, north, east, step_m=radius * 1.4):
    found, more = search("dentist", lat, lng, radius)
    if more:
        crowded.append((lat, lng))
    for p in found:
        loc = p["location"]
        inside = loc and south <= loc["lat"] <= north and west <= loc["lng"] <= east
        if inside and p["business_status"] != "CLOSED_PERMANENTLY":
            places[p["place_id"]] = p

with open("dentists.csv", "w", newline="") as f:
    out = csv.writer(f)
    out.writerow(["name", "address", "phone", "website", "rating", "ratings", "place_id"])
    for p in places.values():
        out.writerow([p["name"], p["formatted_address"], p["formatted_phone_number"],
                      p["website"], p["rating"], p["user_ratings_total"], p["place_id"]])

print(len(places), "places;", len(crowded), "points had more than 3 pages")

That box is 20 points. At up to 3 pages each, it is at most 60 pages, 180 credits, and the 2,000 trial credits cover it. Run on 4 October, it found 65 dentists in under a minute, and no point was crowded. A field Google does not show for a place, such as a website, is null, and the CSV writes it as an empty cell.

Getting the grid right

• A crowded point still had a next page after its third. There are more places there than three pages show: search that part again with a smaller radius, rather than more pages.
• The cost grows with the area. Halving the radius makes four times as many points. Start with a radius of 1 to 2 km in a city centre and 5 km outside it, and look at how many points were crowded.
• One kind of business at a time. The query is what you would type into Google Maps, in the local language where it helps: dentista in Porto, as well as dentist.

For hundreds of points, run the searches side by side, up to your limit of calls at once. To bring the list into a spreadsheet instead, see Google Maps places into a Google Sheet.