Playwright: scraping and automating modern websites

Feeding an LLM good data often means collecting it from the web first — and the classic "fetch the HTML and parse it" approach falls apart on modern sites, where the content you want is drawn by JavaScript after the page loads. Playwright solves that by driving a real browser.

What it is

Playwright is a browser-automation library that controls Chromium, Firefox, and WebKit. It loads a page exactly as a user's browser would — running the JavaScript, waiting for content — and lets you interact with and read the fully-rendered result. That makes it far more capable than raw HTTP requests for dynamic pages.

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/products');
await page.waitForSelector('.product');           // wait for JS to render

const titles = await page.$$eval('.product h2',
  (els) => els.map((e) => e.textContent?.trim()));

await browser.close();

The patterns that make it reliable

• Wait for elements, not timers — waitForSelector beats arbitrary sleeps and cuts flakiness.
• Handle pagination and infinite scroll explicitly — click "next" or scroll, then wait for new content.
• Be resilient — pages change; select on stable attributes and fail loudly when structure shifts.

Beyond scraping: it's browser automation

Reading pages is only half of what Playwright does. Because it drives a real browser, the same API automates actions — clicking through a flow, filling and submitting forms, uploading files, logging in, and asserting that a page did what it should. That's why Playwright is a leading tool for end-to-end testing, not just data extraction: the mechanics of 'scrape this' and 'automate this flow' are identical. It's also why an AI agent can use it to act on the web, not just observe — which is exactly what the Playwright MCP server exposes, letting an agent drive a browser through MCP (see that post).

Scrape (and automate) responsibly

Capability isn't permission. Respect robots.txt and terms of service, rate-limit your requests, avoid personal data, and prefer an official API when one exists — and when you're automating actions rather than just reading, be doubly careful, because now you're changing state on someone else's site, not just observing it. Getting the data isn't the hard part; doing it without being reckless is.

If a browser can see it, Playwright can read it — and do it. Whether you should is a separate question — ask it first.