Beautiful Soup Free Python Library: HTML & XML Parsing Made Easy

Getting Started with Beautiful Soup

The recommended and easiest way to install Beautiful Soup is using pip. Please use the following command a smooth installation.

pip install beautifulsoup4 

Work with Multiple Parsers

The Beautiful Soup library doesn’t do all the parsing itself — instead, it sits atop mature parsers like Python’s built-in html.parser, the fast lxml parser, and the browser-like html5lib. Depending on needs, you can prioritize speed or standards-compliance. This architecture gives you flexibility to choose the parser that best fits your needs, trade parsing speed for flexibility, try different parsing strategies without changing your Beautiful Soup code, and so on. The following simple code shows how developers can use different parsers with Beautiful Soup.

# Using different parsers with Beautiful Soup
html_doc = "
Sample content
"

# Using lxml parser (fast)
soup_lxml = BeautifulSoup(html_doc, 'lxml')

# Using html5lib parser (slower but more lenient)
soup_html5lib = BeautifulSoup(html_doc, 'html5lib')

# Using Python's built-in parser
soup_builtin = BeautifulSoup(html_doc, 'html.parser')

Extracting Links from Web Pages

Scraping links is a common web scraping task, particularly useful for building web crawlers. The open source Beautiful Soup library has included support for loading web pages and extracting links from it inside Python applications. The following simple code example demonstrates fetching a live webpage using the requests library, then extracting all anchor tags inside Python applications.

from bs4 import BeautifulSoup
import requests

# Fetch a real webpage
url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

# Extract all links
links = soup.find_all('a')

print(f"Found {len(links)} links:")
for link in links:
    href = link.get('href')
    text = link.get_text().strip()
    print(f"Text: {text} | URL: {href}")

# Filter links by criteria
external_links = []
for link in soup.find_all('a'):
    href = link.get('href')
    if href and href.startswith('http'):
        external_links.append(href)

print(f"\nFound {len(external_links)} external links")

Searching Specific HTML Elements

One of Beautiful Soup’s biggest strengths is how naturally it lets you search HTML, almost like reading plain English. Instead of dealing with complex DOM APIs, you describe what you want, and Beautiful Soup finds it for you. Suppose you want to find all the links on a page. You can use the find_all method as shown in the following code example. The method returns a list of all link tags. We then iterate through them to extract the URL and text to get the visible link name using Python API.

# Extract all anchor tags
links = soup.find_all('a')

for link in links:
    href = link.get('href')
    text = link.text.strip()
    print(f"Text: {text} | URL: {href}")

Automatic Encoding Handling

One of Beautiful Soup's most convenient features is its automatic encoding conversion. The library automatically converts incoming documents to Unicode and outgoing documents to UTF-8, eliminating one of the most common headaches in web scraping. Please remember that you only need to think about encodings when the document doesn't specify an encoding as Beautiful Soup can't automatically detect the encoding. In these rare cases, you simply specify the original encoding manually.