· Charlotte Will · webscraping  · 8 min read

Advanced Strategies for Scraping Real-Time Data Feeds

Discover advanced strategies and techniques for scraping real-time data feeds effectively. Learn how to extract dynamic content, handle rate limits, and use tools for seamless integration. Perfect for both beginners and experienced developers.

Discover advanced strategies and techniques for scraping real-time data feeds effectively. Learn how to extract dynamic content, handle rate limits, and use tools for seamless integration. Perfect for both beginners and experienced developers.

Introduction

Most scrapers run against pages that sit still. A price page, a category listing, a directory of articles. You fetch it, you parse it, you are done.

Real-time data feeds break that assumption. The value is in the timestamp, which means the page you scraped ten seconds ago is already stale, and the endpoint serving it changes shape without warning. Stock quotes, sports scores, weather radar, flight status, crypto order books: each one updates on its own clock, often behind a JavaScript layer, often through a websocket rather than a plain request.

Scraping these well takes a different set of techniques than scraping a static page. The parsing gets harder, but the real work is in scheduling, deduplication, and not getting your access revoked halfway through.

Find the underlying feed before you scrape the page

Most sites that display live data hold it in a more convenient place than the rendered page. The visible table is built from JSON that the browser requests for you, and that JSON endpoint is usually faster, lighter, and more stable than parsing the DOM.

Open your browser’s developer tools, switch to the Network tab, and reload the page. Watch what fires while the numbers update. A request that returns a JSON payload every few seconds is your real target:

import requests

FEED_URL = "https://example.com/api/quotes/stream"

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
    "Accept": "application/json",
})

response = session.get(FEED_URL, timeout=30)
response.raise_for_status()
payload = response.json()

for quote in payload["quotes"]:
    print(f"{quote['symbol']:>8}  {quote['last']:>10}  {quote['ts']}")

The difference in cost is large. A rendered page might be 400 KB of HTML and 3 seconds of browser startup. The JSON feed is often under 20 KB and returns in under 200 ms, which is the difference between a scrape that keeps pace with the update interval and one that does not.

Where the data arrives over a websocket, the connection stays open and messages arrive as they happen. Python’s websockets library handles this cleanly:

import asyncio
import json
import websockets

WS_URL = "wss://example.com/feed/quotes"

async def consume():
    async with websockets.connect(WS_URL, ping_interval=20) as ws:
        async for raw in ws:
            message = json.loads(raw)
            if message.get("type") != "quote":
                continue
            print(f"{message['symbol']} @ {message['last']} ({message['ts']})")

asyncio.run(consume())

A websocket removes the polling problem entirely, since you are pushed updates rather than asking for them. The tradeoff is that the connection must be kept alive, and a dropped connection needs a reconnect that resumes without duplicating records.

Deduplicate on the update timestamp

Real-time feeds repeat themselves. A quote that has not moved is often re-sent on every tick, and a reconnect replays the last several messages. Storing every row you receive means your dataset fills with identical values carrying different identifiers.

The fix is to key on whatever the feed uses to identify a version, usually a timestamp or a sequence number:

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class FeedState:
    """Track the newest version of each record we have accepted."""

    latest_ts: dict[str, str] = field(default_factory=dict)

    def is_new(self, record_id: str, ts: str) -> bool:
        previous = self.latest_ts.get(record_id)
        if previous is not None and ts <= previous:
            return False
        self.latest_ts[record_id] = ts
        return True

state = FeedState()

def on_quote(quote: dict) -> Optional[dict]:
    if not state.is_new(quote["symbol"], quote["ts"]):
        return None
    return quote

Comparing timestamps as strings works when the feed uses a fixed-width ISO format, which most do. When it does not, parse into a datetime and compare properly, because lexicographic comparison on 2024-1-9 against 2024-1-10 gives the wrong answer.

For a feed that carries a monotonic sequence number instead, keep a single high-water mark rather than a per-record map:

class SequenceTracker:
    def __init__(self) -> None:
        self.high_water = 0

    def accept(self, seq: int) -> bool:
        if seq <= self.high_water:
            return False
        self.high_water = seq
        return True

Handle rate limits without falling back to bursts

Live feeds tend to publish limits in the response headers. Reading them is more reliable than guessing, because the numbers change with your plan and your load:

from dataclasses import dataclass

@dataclass
class RateLimitState:
    remaining: int = 0
    reset_at: float = 0.0
    limit: int = 0

def read_rate_limit(response, now: float) -> RateLimitState:
    limit = int(response.headers.get("X-RateLimit-Limit", 0))
    remaining = int(response.headers.get("X-RateLimit-Remaining", 0))
    reset = int(response.headers.get("X-RateLimit-Reset", 0))
    return RateLimitState(
        remaining=remaining,
        reset_at=float(reset),
        limit=limit,
    )

def should_wait(state: RateLimitState, now: float) -> bool:
    if state.remaining > 0:
        return False
    return state.reset_at > now

When the limit is exhausted, wait for the reset rather than sleeping a fixed interval. A fixed sleep either wastes quota or hits the wall, and neither outcome helps. When you do need to slow down, a small random jitter on top of the base interval keeps a fleet of workers from synchronising into a burst that looks like an attack:

import random
import time

def polite_delay(base_seconds: float = 1.0, jitter: float = 0.4) -> None:
    time.sleep(base_seconds + random.uniform(0, jitter))

Push data out of the scrape loop

The most common way real-time scraping fails is architectural. A scrape loop that both fetches and writes, synchronously, means a slow write delays the next fetch, which delays the write, and the whole thing falls behind the update interval within minutes.

Separate the two. Fetch into a queue, and let a separate process drain the queue into storage:

import queue
import threading
import time

records: "queue.Queue[dict]" = queue.Queue(maxsize=1000)

def fetch_loop(url: str) -> None:
    while True:
        try:
            payload = session.get(url, timeout=30).json()
        except requests.RequestException:
            time.sleep(5)
            continue
        for record in payload["quotes"]:
            records.put(record)

def write_loop(conn) -> None:
    batch = []
    while True:
        record = records.get()
        batch.append(record)
        if len(batch) >= 200:
            conn.executemany(
                "INSERT INTO quotes (symbol, last, ts) VALUES (?, ?, ?)",
                [(r["symbol"], r["last"], r["ts"]) for r in batch],
            )
            conn.commit()
            batch.clear()

The queue’s maxsize matters. An unbounded queue lets a slow consumer accumulate an unbounded backlog, which means the data you eventually store is minutes stale even though the scrape itself never fell behind. A bounded queue applies backpressure: the fetcher blocks when the writer cannot keep up, which surfaces the problem instead of hiding it behind a growing buffer.

Reconcile against the authoritative source

Live feeds drift out of sync with the source of truth. A price moves on the exchange and the feed that carries it is delayed, or a game is postponed and the score endpoint still shows the old value. If your feed feeds a decision, you need a reconciliation step that compares your accumulated data against a slower, authoritative source.

A practical pattern is to treat the stream as the fast path and a periodic full fetch as the correction:

def reconcile(fast_rows: list[dict], authoritative_rows: list[dict]) -> list[dict]:
    """Return rows where the authoritative source disagrees with the stream."""
    authoritative = {row["id"]: row for row in authoritative_rows}
    corrections = []
    for row in fast_rows:
        truth = authoritative.get(row["id"])
        if truth is None:
            continue
        if truth["value"] != row["value"]:
            corrections.append(truth)
    return corrections

Run this on a schedule rather than on every tick. The stream is fast enough that small discrepancies resolve themselves within a few updates, and chasing them immediately mostly produces false alarms.

What to watch in production

Four numbers tell you whether a live scrape is healthy, and each one catches a different failure:

SignalReadingWhat it means
Fetch latency, p95RisingThe source is throttling you, or your connection is degrading
Gap since last accepted recordGrowingThe feed stalled, or your dedupe logic is rejecting everything
Queue depthSteady at maxThe writer cannot keep up, and stored data is falling behind
Block rateAbove a few percentYour access pattern is no longer acceptable

The second one catches the failure that looks like success. A dedupe bug that rejects every record produces a pipeline that reports healthy fetches and an empty dataset. Track the time since the last accepted row, not just the time since the last request.

Conclusion

Scraping real-time data feeds is mostly a scheduling problem wearing a scraping costume. Find the JSON endpoint or websocket behind the rendered page, deduplicate on the version field, respect the rate limit headers instead of guessing at intervals, and keep the fetch path separate from the write path with a bounded queue between them.

The techniques for handling dynamic content are covered in detail in Scraping Dynamic Content Loaded by JavaScript Frameworks, and if you are pulling data at volume rather than from one endpoint, Advanced Techniques for Real-Time Web Scraping covers the throughput side. For the extraction step that turns a feed into structured rows, What is Real-Time Data Extraction with Web Scraping? walks through the parsing choices.

FAQs

  1. How do I scrape real-time stock prices? Look for the JSON endpoint behind the quote table rather than scraping the rendered page. Poll it on a short interval, or connect to the websocket if one is offered, and deduplicate on the quote timestamp so unchanged prices do not fill your table.

  2. What is the best way to extract real-time sports scores? Most sports sites load scores from a JSON feed that the page requests on a timer. Find that feed in the Network tab, then poll it and key your records on the game ID plus the update timestamp.

  3. How do I handle rate limits when scraping real-time weather data? Read the rate limit headers on every response and wait for the reset timestamp when the remaining count hits zero. Add a small random jitter to your base interval so parallel workers do not synchronize into a burst.

  4. How do I scrape data that loads with JavaScript? Use a headless browser to render the page when the data never appears in the initial HTML, or call the underlying API directly when you can identify it. The second option is usually faster and more reliable.

  5. How often should I run a real-time scrape? Match the interval to the fastest thing that changes on the page. Polling faster than the source updates wastes quota and risks the block, and a scrape slower than the update rate misses data the reader can see.

    Back to Blog

    Related Posts

    View All Posts »
    Implementing Geospatial Data Extraction with Python and Web Scraping

    Implementing Geospatial Data Extraction with Python and Web Scraping

    Discover how to implement geospatial data extraction using Python and web scraping techniques. This comprehensive guide covers practical methods, libraries like BeautifulSoup, Geopy, Folium, and Geopandas, as well as real-time data extraction and advanced analysis techniques.

    What is Web Scraping for Competitive Intelligence?

    What is Web Scraping for Competitive Intelligence?

    Discover how web scraping can revolutionize your competitive intelligence efforts. Learn practical techniques, tools, and strategies to extract valuable data from websites. Enhance your market research and analysis with actionable insights.