← All articles

Tutorials

Web Scraping With Gemini: A Practical Python Guide

Alex R. · July 19, 2026 · 9 min read

Traditional web scrapers are brittle: you write CSS selectors or XPath to pull each field, and the moment a site tweaks its markup, your script breaks. Large language models like Google Gemini change that. Instead of hard-coding where the data lives, you describe what you want in plain English and let the model read the page semantically. This guide shows two practical ways to scrape with Gemini in Python, how to keep your token bill down, and why residential proxies are still the part you cannot skip.

Why Use Gemini for Web Scraping?

LLM-assisted scraping trades fragile selectors for semantic understanding. The wins are real:

It is the same shift we covered for other models — see web scraping with Claude and how to use ChatGPT for web scraping. Gemini's advantage is a generous free tier, strong long-context handling, and a built-in tool that can fetch pages for you.

Two Ways to Scrape With Gemini

There are two paths, and picking the right one is most of the battle:

  1. The URL Context tool — Gemini fetches the page itself from a URL you provide. Zero fetching code, great for simple public pages.
  2. Fetch the HTML yourself — you download the page with requests or Playwright (through a proxy), then hand the cleaned text to Gemini. More code, but the only option for protected, geo-restricted, or JavaScript-heavy sites.

The URL Context Tool: Let Gemini Fetch the Page

Gemini can retrieve a handful of public URLs per request and answer questions about them directly. With the current google-genai SDK it looks like this:

from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_GEMINI_API_KEY")

resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=(
        "From https://example.com/product/1 extract the product name, "
        "price and rating. Return only JSON with keys name, price, rating."
    ),
    config=types.GenerateContentConfig(
        tools=[types.Tool(url_context=types.UrlContext())],
    ),
)
print(resp.text)

This is the fastest way to prototype. The catch: Gemini fetches from Google's own servers, so it only works on simple, public, static pages. Anything behind a login, a paywall, aggressive anti-bot protection, or heavy client-side JavaScript will return an error or an empty page — and you cannot control the exit IP or location. For those targets, you fetch the HTML yourself.

When You Need to Fetch the HTML Yourself

You take over fetching whenever the target is hardened or dynamic:

All four are solved the same way: route your requests through residential proxies so each looks like an ordinary user browsing from a real home connection, then pass the finished HTML to Gemini.

Setup: API Key and Client

Grab a free API key from Google AI Studio, then install the SDK and helpers:

python -m venv venv
source venv/bin/activate
pip install google-genai requests beautifulsoup4

Gemini offers several models. For scraping, gemini-2.5-flash is the sweet spot — fast, cheap, and more than capable of pulling fields out of a page. Reserve the larger pro models for genuinely complex reasoning.

Preparing HTML to Save Tokens

Feeding raw HTML straight to Gemini is the most common mistake. Markup, inline scripts, and styles can be 90% of a page's bytes and burn through your token limit for nothing. Strip the page down to visible text first with BeautifulSoup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "svg", "noscript"]):
    tag.decompose()
text = soup.get_text(" ", strip=True)

This routinely cuts payload size by 80% or more, which means lower cost and faster responses. For very long pages, truncate to the section you care about before sending.

Extracting Structured Data With Gemini

The reliability trick is to force JSON output with response_mime_type so you never have to parse loose prose or strip code fences. Here is the full manual pipeline — fetch through a SpyderProxy residential proxy, clean, extract, and load into a Python dict:

import json
import requests
from bs4 import BeautifulSoup
from google import genai
from google.genai import types

proxy = "http://USERNAME:[email protected]:12321"
proxies = {"http": proxy, "https": proxy}
client = genai.Client(api_key="YOUR_GEMINI_API_KEY")

# 1. fetch the page through a residential proxy
html = requests.get("https://example.com/product/1", proxies=proxies, timeout=20).text

# 2. strip boilerplate so tokens are spent on content, not markup
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "svg", "noscript"]):
    tag.decompose()
text = soup.get_text(" ", strip=True)[:12000]

# 3. let Gemini pull structured fields out of the cleaned text
prompt = (
    "Extract the product name, price (number only) and rating from this page text. "
    "Return JSON with keys name, price, rating.\\n\\n" + text
)
resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=prompt,
    config=types.GenerateContentConfig(response_mime_type="application/json"),
)
data = json.loads(resp.text)
print(data)

Because the proxy IP is a real residential address, the target sees an ordinary visitor and returns the full page — the same reason residential proxies are the default for web scraping and AI data collection.

Saving and Reusing Scraped Data

Once Gemini returns structured records, persist them. JSON keeps nested fields intact; CSV is cleaner for flat, tabular data headed to a spreadsheet or analytics (see JSON vs CSV). A common pattern is to scrape to JSON to preserve everything, then export a flattened CSV for analysis — useful when you are building datasets for machine learning.

Scaling Up: Loop, Rotate, Retry

To run across thousands of pages, wrap the pipeline in a loop, rotate IPs, and add retries with backoff. Rotating residential proxies give you a fresh IP per request automatically, so no single address gets rate-limited or banned. Keep Gemini calls batched sensibly to stay under model rate limits, and cache pages you have already fetched so a re-run does not pay twice. For the fundamentals of a resilient collector, see how to build a web scraper in Python.

Limitations, Anti-Bot Defenses, and Cost

Gemini is a superb parser. The fetching — looking like a real user, from the right country, at scale — is still the hard part, and it is exactly what a good residential network handles.

Frequently Asked Questions

Can Gemini fetch web pages on its own?

Yes, via the URL Context tool, which retrieves a small number of public URLs per request. It only works on simple, public, static pages — anything behind a login, paywall, anti-bot system, or heavy JavaScript needs you to fetch the HTML yourself, ideally through a residential proxy.

Do I still need proxies if I use Gemini?

For anything beyond a few simple public pages, yes. Gemini parses content but does not disguise your requests. To fetch protected, geo-restricted, or rate-limited pages at scale without being blocked, you route requests through residential proxies so each looks like a real user.

Which Gemini model is best for scraping?

gemini-2.5-flash is the practical default — fast, inexpensive, and accurate at extracting fields from cleaned page text. Save the larger pro models for tasks that need heavy reasoning, since they cost more per page.

How do I stop Gemini from wasting tokens on HTML?

Strip scripts, styles, and navigation with BeautifulSoup and send only the visible text, truncated to the relevant section. This commonly cuts payload by 80% or more, lowering cost and speeding up responses.

Is scraping with Gemini legal?

Using Gemini to parse pages does not change the rules. Scrape only public data, respect robots.txt and terms of service, avoid login-gated content you are not authorized to access, and comply with privacy laws for any personal data. The legality depends on the site and your jurisdiction, not the tool.

Conclusion

Gemini turns web scraping from a selector-maintenance chore into a describe-what-you-want workflow: fetch a page, strip it to text, and let the model return clean JSON. Use the URL Context tool for quick public pages, and fetch the HTML yourself through residential proxies for everything hardened, geo-locked, or dynamic. The model handles the parsing; the proxy handles the access.

Feed your Gemini scraper pages it can actually reach: SpyderProxy residential proxies from $2.75/GB — ethically sourced, 195+ countries, built to beat blocks and rate limits.

Ready to collect data without getting blocked?

Start now ↗