FetchWebsiteText — Analysis Tool Box

Introduction

Modern websites often render their real content client-side, so a plain HTTP request returns a shell of JavaScript rather than the text you actually need. FetchWebsiteText answers "what does this page actually say, once it's fully rendered?" by driving a headless browser (via Browserless) to execute the page's JavaScript, then stripping the resulting HTML down to clean, readable text. Reach for it when scraping React/Vue/Angular sites, monitoring content that changes over time, or gathering source text for downstream NLP — anywhere a naive scraper would come back empty.

Teaching Note

Common use cases include:

Web scraping for research and data collection
Content monitoring and change detection
Competitive intelligence gathering
Automated content extraction for text analysis or NLP
Archiving web content for documentation
Extracting text from JavaScript-heavy websites

Parameters

Parameter	Type	Default	Description
`url`required		—	The URL of the website from which to fetch text content. Must be a valid HTTP/HTTPS URL.
`browserless_api_key`		`None`	API key for the Browserless service. If not provided, the function attempts to load it from environment variables (typically stored in a .env file).

Returns

A string with the extracted text content of the website, with HTML tags removed and excessive newlines (4 or more) normalized to three newlines for improved readability.

Example

python

from analysistoolbox.data_collection import FetchWebsiteText
import os

# Fetch text using an API key from environment variables
api_key = os.getenv('BROWSERLESS_API_KEY')
text = FetchWebsiteText(
    url='https://example.com/dynamic-page',
    browserless_api_key=api_key
)