Web Scraping JavaScript-Rendered Content with Python + Playwright

Many websites today load data dynamically using JavaScript. If you've ever tried scraping one using requests or BeautifulSoup, you probably ran into this frustrating problem:

"The page looks empty — where's the data?!"

This post shows how to solve it using Playwright, a modern automation library that works like Selenium, but faster and more powerful.

---

🧠 The Problem

Here’s what happens with traditional scraping:

pythonCopyEditimport requests
from bs4 import BeautifulSoup

url = "https://example.com/products"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

print(soup.find_all("div", class_="product-card"))

The result? Often [], because the real data gets loaded by JavaScript after the HTML is served.

---

⚙️ The Solution: Headless Browsing with Playwright

Playwright can render JavaScript, just like a real browser, then let you scrape the fully loaded page.

---

🚀 Step 1: Install Playwright

bashCopyEditpip install playwright
playwright install

---

📄 Step 2: Scrape with Playwright

pythonCopyEditfrom playwright.sync_api import sync_playwright

def scrape_js_site(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url)
        page.wait_for_timeout(2000)  # Wait for JS to load

        # Extract rendered HTML
        content = page.content()
        browser.close()
        return content

---

🧪 Step 3: Use BeautifulSoup After JS Renders

pythonCopyEditfrom bs4 import BeautifulSoup

html = scrape_js_site("https://example.com/products")
soup = BeautifulSoup(html, "html.parser")

for card in soup.select(".product-card"):
    title = card.select_one(".title").text
    price = card.select_one(".price").text
    print(f"{title}: {price}")

You now get real data, even if it was loaded by JavaScript!

---

🛠 Tips & Tricks

• page.wait_for_selector(".product-card") is more reliable than wait_for_timeout

• You can also click buttons, fill forms, or scroll with Playwright

• Headless mode keeps it fast — but you can disable it for debugging

---

🧵 Wrapping Up

If you’re scraping modern web apps, requests just won’t cut it. Playwright lets you automate and extract content from dynamic, JS-powered websites — with less pain and more power than Selenium.

Give it a try, and let me know if you'd like a full Playwright tutorial for logging in, scrolling pages, or downloading files!