Quick answer: A scraping API returns X (Twitter) profile and post data as clean JSON from a single HTTP request. You skip the headless browser, the fingerprint rotation and the parser that breaks on every redesign. Teams use it to build AI training sets, run sentiment and brand monitoring, vet influencers and track competitors. Pricing runs per request rather than per month, from $3.00 down to $1.33 per 1,000 requests, and you can try it with no account on our free X profile lookup.
What a scraping API replaces
Anyone who has maintained a social scraper knows the shape of the work. The first version takes an afternoon. Keeping it alive takes a year.
A working X scraper needs a headless browser or a reverse-engineered internal endpoint, a pool of residential IPs so requests do not all originate from one datacenter block, fingerprint rotation so the browser does not look automated, session handling, retry and backoff logic, and a parser mapped to a DOM or JSON shape that the platform changes without notice. None of that is the thing you actually wanted. You wanted follower counts and post text.
A scraping API moves that whole stack behind an HTTP call. You send a handle, you get structured data. When the target changes, the provider fixes it and your code does not move. The trade is straightforward: you pay per request instead of paying an engineer to babysit infrastructure.
If you would rather build it yourself, that is a legitimate choice and we sell the residential proxies you would need for it. Our API versus web scraping comparison lays out where each approach wins.
Why X data is used for AI training
Short-form social text has properties that make it useful for model work and hard to source elsewhere.
- It is conversational and current. Book and article corpora skew formal and lag reality by months or years. Posts carry slang, abbreviations, sarcasm and reactions to events that happened this week.
- It comes with engagement signal. Every post carries likes, reposts and replies. That is a ready-made relevance or quality weight you do not have to annotate by hand.
- It is naturally labelled by author. Pulling by handle gives you clean per-author sets, which matters for style transfer, authorship attribution and persona work.
- It is timestamped. Ordered, dated text supports temporal evaluation — testing whether a model knows something it should not have known yet.
Common jobs: fine-tuning a model on a domain's actual vocabulary, building sentiment sets where the labels come from engagement rather than annotators, assembling evaluation sets that postdate a model's training cutoff, and retrieval corpora for agents that answer questions about a niche.
For the wider picture on sourcing, see our guides to LLM training datasets and AI data collection.
What one request returns
A profile lookup returns the full public record for an account in a single response.
| Group | Fields |
|---|---|
| Identity | Handle, display name, numeric user ID, verification status, protected flag |
| Profile | Bio, avatar URL, banner URL, website, location |
| Audience | Follower count, following count, total post count, listed count |
| History | Account creation date |
A timeline request returns posts with text, post ID, creation timestamp, and like, repost and reply counts. No HTML parsing on your side, no login, no cookies to keep warm.
Six things teams build with it
1. AI training and evaluation sets. Pull by handle or by topic, filter on engagement, and you have a domain-specific corpus with quality signal attached.
2. Sentiment and brand monitoring. Track how a product or company is discussed over time. Engagement counts let you weight loud posts above ignored ones instead of treating every mention equally.
3. Influencer vetting. Follower count alone is trivially inflated. Follower-to-engagement ratio, posting cadence and account age together are much harder to fake, and all three come back in the same response.
4. Competitor tracking. Watch what rival accounts post, which posts land, and how fast their audience is moving.
5. Lead qualification. Enrich a list of handles with bio, location, website and audience size to prioritise outreach.
6. Trend and market research. Aggregate posts across accounts in a sector to spot vocabulary and sentiment shifting before it reaches formal coverage. The same approach applies to financial data scraping.
Paging through a full timeline
Timeline requests return twenty posts. Each response includes a next_cursor and a has_more flag; pass the cursor back to get the next page. Unlike depth-capped endpoints, the cursor walks the account's full public history rather than stopping at a fixed number of posts.
Each page costs one credit. A 400-post backfill is 20 credits; 10,000 posts across 50 accounts is roughly 500. Budget in pages, not posts.
Two practical notes. Page sets are snapshot-based, so a cursor eventually goes stale — if you get an expiry message, re-request without a cursor to start a fresh set. And write each page to durable storage as it arrives rather than holding a long backfill in memory, so a failure halfway through costs you one page and not the whole run.
What it costs
Credits are bought in packs. One request is one credit, credits do not expire, and there is no subscription or monthly minimum.
| Pack | Price | Per 1,000 |
|---|---|---|
| 1,000 requests | $3.00 | $3.00 |
| 5,000 requests | $13.50 | $2.70 |
| 25,000 requests | $57.50 | $2.30 |
| 100,000 requests | $190.00 | $1.90 |
| 500,000 requests | $665.00 | $1.33 |
The structural difference from the official API is not only the headline number. The official API sells tiers with a monthly commitment and gates timeline depth by tier, so an idle month still costs full price. Per-request pricing means 4,000 lookups this month and none next month costs you 4,000 lookups. For bursty research work — collect a dataset, go quiet for six weeks, collect another — that difference dominates.
Scraping public data responsibly
Being straight about this matters more than a reassuring sentence.
This API reads public profiles and posts only. Protected accounts, direct messages and anything behind a login are not accessible and will not be. Scraping public pages is broadly lawful in most jurisdictions, and courts in the US have repeatedly declined to treat access to public web pages as unauthorised access under the CFAA.
That is not the whole question. Three others sit alongside it, and they have different answers:
- Platform terms. Automated collection generally conflicts with X's terms of service. That is a contractual matter between you and the platform, not a criminal one, but it is real and you should know you are making that choice.
- Copyright. Individual posts are the author's expression. Collecting them is one thing; republishing or training a commercial model on them raises questions that are actively being litigated.
- Privacy law. Public does not mean unregulated. Under GDPR, personal data stays personal data when it is publicly posted, and building a dataset about identifiable EU residents needs a lawful basis.
Practical guidance: collect only the fields you actually need, do not build profiles on private individuals, honour deletion requests, and if the output is a commercial model or a redistributed dataset, get a lawyer to look at it. We are not able to give you legal advice, and a vendor telling you it is all fine would be selling you something.
Making your first request
Try it with no account first on the free profile lookup tool. It is rate-limited per IP and needs no key. When you want it in code, buy a credit pack and your API key appears in the dashboard.
curl "https://data.spyderproxy.com/twitter/profile?access_key=YOUR_KEY&url=spyderproxy"A timeline page, then the page after it:
curl "https://data.spyderproxy.com/twitter/tweets?access_key=YOUR_KEY&url=spyderproxy&limit=20"
curl "https://data.spyderproxy.com/twitter/tweets?access_key=YOUR_KEY&url=spyderproxy&limit=20&cursor=NEXT_CURSOR"Collecting a full timeline in Python:
import requests
KEY = "YOUR_KEY"
BASE = "https://data.spyderproxy.com/twitter/tweets"
def timeline(handle, max_pages=50):
cursor, pages = None, 0
while pages < max_pages:
params = {"access_key": KEY, "url": handle, "limit": 20}
if cursor:
params["cursor"] = cursor
meta = (r := requests.get(BASE, params=params, timeout=30).json()).get("meta", {})
for post in r["data"]["tweets"]:
yield post
if not meta.get("has_more") or not meta.get("next_cursor"):
return
cursor, pages = meta["next_cursor"], pages + 1
for post in timeline("spyderproxy"):
print(post["created_at"], post["likes"], post["text"][:80])The max_pages guard is deliberate. Cursor pagination on a prolific account will happily run for thousands of pages and spend credits you did not mean to spend.
Where this is going
X is the first endpoint in a scraping API line, not the whole of it. The pattern is the same each time: the platform-specific breakage is our problem, you get JSON. More endpoints are rolling out.
Related reading: AI web scraping tools, advanced web scraping in Python, best proxies for X, and the X Data API product page.
Frequently asked questions
What is a scraping API?
A scraping API is an HTTP endpoint that returns structured data from a website without you running the scraper. You send a handle or URL and get JSON back. The provider owns the part that breaks: proxy rotation, browser fingerprinting, CAPTCHA handling, and the parser that needs rewriting every time the target ships a redesign.
Can I use scraped tweets to train an AI model?
Teams routinely build training and evaluation sets from public posts, and a scraping API is the usual collection layer. What you may then do with that data is governed by the platform's terms, by copyright in the individual posts, and by privacy law where the authors live. Treat collection and licensing as two separate questions and get the second one reviewed.
How is this different from the official X API?
The official API sells access in tiers with a monthly commitment, and timeline depth is gated by tier. A scraping API charges per request, so 4,000 lookups in one month and none the next costs you only the 4,000. It also needs no app review, no OAuth flow, and no developer account.
How many posts can I pull from one account?
Twenty per request. Each response carries a cursor for the next page, and the cursor pages through the account's full public history rather than stopping at a fixed depth. One credit per page, so a 400-post backfill costs 20 credits.
Do I need my own proxies to use it?
No. Proxy rotation happens on our side and is priced into the credit. If you are building your own scraper instead, that is exactly the layer you would need residential IPs for.
Do credits expire?
No. Credits sit on your account until you spend them, with no subscription and no monthly minimum. Buying a larger pack lowers the per-request price and nothing else changes.