This is the star of the show. Instead of guessing that an article lives in `div#main-content > article.post`, you can now ask Kreuzberg for it directly. It analyzes the DOM for clusters of text-dense nodes, link density, and common structural patterns to identify primary and secondary content blocks.

The Old Way (BeautifulSoup):

from bs4 import BeautifulSoup
import requests

url = "https://some-news-site.com/article"
html = requests.get(url).text
soup = BeautifulSoup(html, 'html.parser')

# Hope this selector doesn't change!
article_div = soup.find('div', class_='article-body')
if article_div:
    text = article_div.get_text(separator='\n', strip=True)
else:
    # ... try another selector? Ugh.
    text = ""
    

The New Way (Kreuzberg v3.11):

import kreuzberg

url = "https://some-news-site.com/article"
doc = kreuzberg.fetch(url)

# Just ask for the main content
article = doc.get_article()

print(article.text)
# Also available: article.title, article.publish_date
    

The `get_article()` method returns a rich object containing the cleaned text, the inferred title, and even attempts to find a publication date from the page metadata. It’s not just shorter; it’s more robust and resilient to website changes.

Intelligent Table Extraction: From HTML to DataFrame in One Step

Scraping HTML tables is a special kind of hell. Nested tables, `colspan` and `rowspan` attributes, or—worst of all—tables built with `div` tags can make parsing a nightmare. Kreuzberg v3.11’s new `get_tables()` method handles this brilliantly.

It not only parses standard `

` elements but also identifies table-like structures made from other tags. Best of all, if you have Pandas installed, it returns a list of DataFrames directly.

import kreuzberg

# A page with a complex, messy table
url = "https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)"
doc = kreuzberg.fetch(url)

# Returns a list of pandas DataFrames for each table found
tables_as_dfs = doc.get_tables(as_pandas=True)

# Now you can work with the data immediately
gdp_df = tables_as_dfs[2] 
print(gdp_df.head())
    

This single feature can save hours of tedious parsing and data cleaning code for data-heavy projects.

The Noise Reduction Filter: Your Automatic Cleaner

How much time have you wasted writing code to remove things like "Share on Twitter," "Related Posts," or cookie consent banners? The new `NoiseReduction` filter is a pre-processor that strips this junk out before you even start extracting.

It uses a list of common noise patterns (e.g., social sharing widgets, navigation bars, footers) to simplify the HTML, leaving you with a much cleaner document to work with. It's enabled by default but can be configured.

Kreuzberg’s philosophy is simple: automate the 90% of scraping that is repetitive, so you can focus on the 10% that requires your unique attention.

Kreuzberg vs. The Classics: A Quick Comparison

So, where does Kreuzberg fit in? It doesn't replace everything. Here’s a high-level look at how it stacks up against the established players.

Feature	Kreuzberg v3.11	BeautifulSoup + Requests	Scrapy
Primary Use Case	Intelligent text & data extraction from articles, posts, and data pages.	General-purpose HTML parsing and tree navigation. A flexible toolkit.	Large-scale, asynchronous crawling of entire websites. A full framework.
Ease of Use	Extremely high for common tasks. Almost zero-config.	Moderate. Requires manual selector definition and cleaning logic.	Steep learning curve. Requires understanding of Items, Spiders, and Pipelines.
Key Strength	Semantic understanding. Automatically finds and cleans content.	Unmatched flexibility. You can extract anything if you can write the selector.	Performance, scalability, and architecture for massive scraping jobs.
Resilience to Change	High. Less reliant on specific class names or IDs.	Low. A small CSS change can break the scraper.	Low to Moderate. Suffers from the same selector fragility as BeautifulSoup.

# Install or upgrade Kreuzberg
pip install --upgrade kreuzberg

# For the DataFrame feature, make sure you have pandas
pip install pandas
    

import kreuzberg

url = "https://your-target-article-url.com/"
doc = kreuzberg.fetch(url)
article = doc.get_article()

print(f"Title: {article.title}\n---\n{article.text[:500]}...")