Playwright Web Scraping - Upstash Documentation

Every box can come with its own browser. Create one with browser: true and you get a managed, headless Chromium that boots on first use. Nothing to install, no apt-get, no Chromium binary to keep up to date.

This guide scrapes a JavaScript-heavy site with that built-in browser. We read rendered pages through the SDK, extract structured data against a schema, then connect Playwright over CDP to harvest many pages deterministically.

Note

The browser can only be provisioned when the box is created. It cannot be enabled on an existing box. See Browser for the full surface.

1. Installation#

npm install @upstash/box playwright-core zod

playwright-core is enough. You connect to the box's Chromium instead of launching one locally, so there are no browser binaries to download on your side either.

Set your environment variable:

.env

UPSTASH_BOX_API_KEY=box_xxxxxxxxxxxxxxxxxxxxxxxx

2. Create a box with a browser#

scripts/scrape.ts

import "dotenv/config"import { Box } from "@upstash/box"const box = await Box.create({  runtime: "node",  browser: true,})const tab = await box.browser.tab.create("https://books.toscrape.com", {  waitUntil: "domcontentloaded",})console.log(`Box ready: ${box.id}`)console.log(tab.id, tab.url, tab.title)

The first tab.create boots Chromium. The tab handle is addressed by its Chrome DevTools Protocol target id, so it stays valid across navigations and you can re-attach to it later with box.browser.getTab(id).

3. Read the rendered page#

content() returns the tab's current title, URL, visible text, and links from the real DOM, including anything JavaScript rendered after load. No model is involved, so this costs no tokens.

scripts/scrape.ts

const page = await tab.content()console.log(page.title)console.log(page.text.slice(0, 200))for (const link of page.links ?? []) {  console.log(link.text, link.href)}

This is already enough for a crawler: follow the links you care about with tab.goto(url), read each page with content(), and chunk the text into a dataset.

Note

Client-rendered pages can hydrate after domcontentloaded. If page.text comes back short, wait briefly and call content() again until it settles.

4. Extract structured data with a schema#

When you want typed fields instead of raw text, extract() hands the page to a DOM-aware agent inside the box and validates the result against a Zod schema:

scripts/scrape.ts

import { z } from "zod"const catalog = await tab.extract(  "extract every book in the listing with its title and price",  z.object({    books: z.array(      z.object({        title: z.string(),        price: z.string(),      }),    ),  }),)console.table(catalog.books.slice(0, 5))

The result is parsed with your schema before it is returned, so a successful call always gives you the shape you asked for. Capture a screenshot next to it if you want provenance for what was on screen:

import { writeFile } from "node:fs/promises"await writeFile("catalog.png", await tab.screenshot({ fullPage: true }))

Note

extract uses an LLM and is metered. It needs a model provider key on the box or your account. See AI Actions for the model override and the full list of providers.

5. Scrape many pages with Playwright over CDP#

An AI call per page gets expensive fast. For repeatable multi-page work, drive the same browser with Playwright. cdpUrl() returns an authenticated WebSocket URL that chromium.connectOverCDP accepts directly:

scripts/scrape.ts

import { chromium } from "playwright-core"const browser = await chromium.connectOverCDP(await box.browser.cdpUrl())try {  const ctx = browser.contexts()[0] ?? (await browser.newContext())  const p = await ctx.newPage()  const books: { title: string | null; price: string | null }[] = []  for (let n = 1; n <= 3; n++) {    await p.goto(`https://books.toscrape.com/catalogue/page-${n}.html`, {      waitUntil: "domcontentloaded",    })    books.push(      ...(await p.$$eval("article.product_pod", (rows) =>        rows.map((row) => ({          title: row.querySelector("h3 a")?.getAttribute("title") ?? null,          price: row.querySelector(".price_color")?.textContent?.trim() ?? null,        })),      )),    )  }  console.log(`${books.length} books scraped`)} finally {  await browser.close()}

This is an ordinary Playwright script. Migrating an existing one is usually a single line: chromium.launch() becomes chromium.connectOverCDP(await box.browser.cdpUrl()), and your selectors, actions, and assertions stay as they are.

Warning

The CDP URL carries its auth token in the URL. Anyone who has it gets full control of the browser, so treat it as a secret.

CDP and the SDK drive the same browser and the same tabs. A page opened by Playwright shows up in box.browser.listTabs(), and a tab created by the SDK is visible to Playwright, so you can mix scripted steps with AI steps in one run.

6. Compile the selectors once, then scrape for free#

The two previous sections combine into the pattern worth using in production: let the model read the layout once and emit selectors, then run every later scrape deterministically with no model tokens at all.

This is a file of its own rather than a continuation of the script above, so it opens its own box and browser:

scripts/compile-and-scrape.ts

import "dotenv/config"import { Box } from "@upstash/box"import { chromium } from "playwright-core"import { z } from "zod"const PAGE = (n: number) => `https://books.toscrape.com/catalogue/page-${n}.html`const RECIPE_PATH = "recipe-books.json"const RecipeSchema = z.object({  itemSelector: z.string(),  title: z.object({ selector: z.string(), attr: z.string() }),  price: z.object({ selector: z.string(), attr: z.string() }),})type Recipe = z.infer<typeof RecipeSchema>const box = await Box.create({ runtime: "node", browser: true })// Ask for the selectors, not the data. One metered call per layout.async function compile(): Promise<Recipe> {  const tab = await box.browser.tab.create(PAGE(1), { waitUntil: "domcontentloaded" })  return await tab.extract(    [      "You are compiling a scraper for the repeating list of books.",      "Return `itemSelector`: a CSS selector matching each book's container.",      "Return `title` and `price` field specs: each with a CSS `selector`",      'RELATIVE to the container and `attr` ("text" or an attribute name).',      "If the visible text is truncated, prefer an attribute holding the",      "full value (such as an anchor's title attribute).",      "Selectors must be generic, no :nth-child tied to one item.",    ].join(" "),    RecipeSchema,  )}// Deterministic harvest: same recipe, every page, zero model tokens.async function scrape(recipe: Recipe, pages: number) {  const browser = await chromium.connectOverCDP(await box.browser.cdpUrl())  try {    const ctx = browser.contexts()[0] ?? (await browser.newContext())    const p = await ctx.newPage()    const rows = []    for (let n = 1; n <= pages; n++) {      await p.goto(PAGE(n), { waitUntil: "domcontentloaded" })      rows.push(        ...(await p.$$eval(          recipe.itemSelector,          (items, fields) =>            items.map((row) => {              const read = (field: { selector: string; attr: string }) => {                const el = field.selector ? row.querySelector(field.selector) : row                if (!el) return null                return field.attr === "text" ? el.textContent?.trim() ?? null : el.getAttribute(field.attr)              }              return { title: read(fields.title), price: read(fields.price) }            }),          { title: recipe.title, price: recipe.price },        )),      )    }    return rows  } finally {    await browser.close()  }}// Reuse the recipe cached on the box, or compile and cache a fresh one.let recipe: Recipetry {  recipe = RecipeSchema.parse(JSON.parse(await box.files.read(RECIPE_PATH)))} catch {  recipe = await compile()  await box.files.write({ path: RECIPE_PATH, content: JSON.stringify(recipe, null, 2) })}const rows = await scrape(recipe, 3)console.log(`${rows.length} books`, rows[0])

Leave that tab open. Closing the last tab shuts the browser down, and the CDP connection in scrape() then fails with The browser is shutting down.

Caching the recipe on the box filesystem means the box carries its own knowledge: the next run reuses it and calls no model at all.

Validate the harvest (row count, non-empty fields, price format) on every run. When validation fails, the site's layout changed: recompile the recipe with one extract call and cache the new one. That way the AI cost is paid once per layout, not once per page.

A full runnable version of this is the AI-compiled scraper example.

7. Reuse the box across runs#

A browser box has nothing to install, so there is no setup cost to snapshot away. What is worth keeping is the box itself: the cached recipe, the cookies, and a logged-in session.

Keep the box id and re-attach on the next run:

scripts/run-scrape-job.ts

const box = process.env.SCRAPER_BOX_ID  ? await Box.get(process.env.SCRAPER_BOX_ID)  : await Box.create({ runtime: "node", browser: true })

The box pauses when idle and resumes on demand with its filesystem intact. It bills until you delete it, so delete it when you are done with the target site:

await box.delete()

To log in once and have every later script reuse that session, see the login once, reuse session example.

Next steps#

Reading pages for content, screenshot, and extract.
AI actions for observe and act, including replaying a resolved action with no LLM.
Live view and recordings to watch or replay a scrape.
Browser cookbook for runnable examples across crawling, automation, and testing.