Real-Time News Extraction with Model Context Protocol

In this cookbook, we'll build a powerful news extraction server using the Model Context Protocol (MCP) and Hyperbrowser. This integration enables AI models to access real-time news from various sources through your local machine, transforming how they can retrieve and analyze current events.

With this setup, you'll be able to give AI models the ability to:

Search for latest news on any topic from Bing
Retrieve geographically relevant local news
Access structured news data with titles, summaries, and source information

The Model Context Protocol serves as a standardized bridge between AI systems and local tools, enabling assistants to work with dynamic web content they couldn't otherwise access. Unlike traditional integrations that require hardcoded API endpoints, MCP enables dynamic discovery of available tools and their specifications. This allows AI models to automatically understand what tools are available, what parameters they need, and what outputs they provide - all without requiring manual integration work for each new tool.

By the end of this cookbook, you'll have a robust news extraction system that dramatically expands what your AI can know about current events!

Prerequisites

Before starting, you'll need:

A Hyperbrowser API key (sign up at hyperbrowser.ai if you don't have one)
The MCP Python package (pip install mcp)
Requests for Python (pip install requests)
Pydantic for adding models when doing structured extraction (pip install pydantic)
Python 3.9+ installed

Store your API key in a .env file or set it as an environment variable as needed for the MCP client.

So, for example for Claude, we'd modify the claude_desktop_config.json like so:

{
  "mcpServers": {
    "hyperbrowser-news": {
      "command": "<PATH TO PYTHON>",
      "args": ["<PATH TO MAIN.PY>/main.py"],
      "env": {
        "HYPERBROWSER_API_KEY": "<HYPERBROWSER_API_KEY>"
      }
    }
  }
}

Step 1: Import Libraries and Set Up Environment

We start by importing the necessary packages for our news extraction server. The key components include:

Hyperbrowser: For headless browser automation and structured data extraction
FastMCP: The Model Context Protocol server implementation
Pydantic: For creating strongly-typed data models
requests: For retrieving location data when providing local news

Together, these libraries will enable us to create a robust news extraction system that can be discovered and used by AI assistants.

import os
import sys
import requests

from hyperbrowser import Hyperbrowser
from hyperbrowser.models.extract import StartExtractJobParams
from hyperbrowser.models.session import CreateSessionParams
from typing import List, Optional

from pydantic import BaseModel
from mcp.server.fastmcp import FastMCP
import urllib.parse

Step 2: Initialize the MCP Server

Now we initialize our Model Context Protocol server with a meaningful identifier.

mcp = FastMCP("hyperbrowser-news")

Step 3: Define Data Models for News Information

Before implementing our extraction tools, we define structured data models using Pydantic. These models serve multiple important purposes:

They provide a schema that Hyperbrowser uses for focused extraction
They ensure consistent data structure that AI models can rely on
They validate the extracted data to catch potential errors

Our data models include NewsInfo for individual news items and NewsInfoList as a container for multiple items. Each news item captures key attributes like title, link, summary, and source - essential information for meaningful news analysis.

class NewsInfo(BaseModel):
    title: str
    link: str
    summary: str
    source: str


class NewsInfoList(BaseModel):
    list: List[NewsInfo]

Step 4: Create the Bing News Search Tool

Our first MCP tool provides access to Bing's news search functionality. This tool:

Takes a topic query and optional page number parameter
Constructs a properly formatted search URL using URL encoding
Uses Hyperbrowser's extraction capabilities to gather structured news data
Returns the results as a standardized JSON response

The pagination support allows AI models to explore beyond the first page of results when needed, enabling more comprehensive news analysis. Hyperbrowser's extraction capabilities automatically structure the unorganized web data into our defined schema.

@mcp.tool()
def get_bing_news(news_topic: str, page: Optional[int]) -> str:
    """Get the latest news articles from Bing for a specific topic"""
    hb = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))
    search_query = urllib.parse.quote_plus(news_topic)
    search_url = f"https://www.bing.com/news/search?q={search_query}&first={(max((page-1),0) if page is not None else 0)*10}"
    resp = hb.extract.start_and_wait(
        StartExtractJobParams(urls=[search_url], schema=NewsInfoList)
    )

    if resp.data:
        return NewsInfoList.model_validate(resp.data).model_dump_json()
    else:
        raise ValueError("Could not get news from bing.")

Step 5: Create the Google News Search Tool

Next, we implement a tool for accessing Google News. This provides an alternative source of information, which can be valuable for comparison or when seeking broader coverage.

@mcp.tool()
def get_google_news(news_topic: str) -> str:
    """Get the latest news articles from Google News for a specific topic"""
    hb = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))
    search_query = urllib.parse.quote(news_topic)
    search_url = f"https://news.google.com/search?q={search_query}"
    resp = hb.extract.start_and_wait(
        StartExtractJobParams(urls=[search_url], schema=NewsInfoList)
    )

    if resp.data:
        return NewsInfoList.model_validate(resp.data).model_dump_json()
    else:
        raise ValueError("Could not get news from google.")

Step 6: Create the Local News Tool

Our most advanced tool enables AI models to retrieve geographically relevant news. This capability addresses a significant limitation of many AI systems - their inability to access location-specific information. The function works by:

Either accepting explicit location parameters or detecting location from IP address
Using multiple geolocation providers for reliable location detection
Setting up a geographically appropriate proxy through Hyperbrowser
Searching for local news through that proxy to ensure relevance

This approach enables AI assistants to provide news that's actually relevant to a user's local area - a capability that dramatically enhances contextual awareness for location-specific queries.

@mcp.tool()
def get_local_news(city: Optional[str], country: Optional[str]) -> str:
    """Get the local news for a location. If the location isn't provided, then it will be inferred from your IP."""
    hb = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))
    if city is None or country is None:
        resp = requests.get("https://ip.oxylabs.io/location")
        if resp.status_code == 200:
            location_data = resp.json()
            # Try to get city from different providers, falling back through the list
            providers = ["ip2location", "maxmind", "dbip"]
            for provider in providers:
                if provider in location_data["providers"]:
                    city = location_data["providers"][provider]["city"]
                    country = location_data["providers"][provider]["country"]
                    break

    if city is None or country is None:
        raise ValueError("Could not get city or country info")
    else:
        print(f"City: {city}, Country: {country}", file=sys.stderr)
        search_query = urllib.parse.quote("local news")
        search_url = f"https://www.bing.com/news/search?q={search_query}"
        resp = hb.extract.start_and_wait(
            StartExtractJobParams(
                urls=[search_url],
                schema=NewsInfoList,
                session_options=CreateSessionParams(
                    use_proxy=True, proxy_city=city, proxy_country=country  # type: ignore
                ),
            )
        )

        if resp.data:
            return NewsInfoList.model_validate(resp.data).model_dump_json()
        else:
            raise ValueError("Could not get news from google.")

Step 7: Running the MCP Server

Now we'll launch our MCP server to make our news extraction tools available to AI models. The server uses stdio (standard input/output) as its transport mechanism, making it compatible with a wide range of AI integration platforms including Claude Desktop, Cline, Cursor, and Windsurf.

When an AI model connects to this server, it will automatically discover all three of our news tools along with their documentation, parameter types, and return types - all through the standardized MCP protocol. This dynamic discovery is a key advantage of MCP over traditional hardcoded integrations.

if __name__ == "main":
    # Initialize and run the server
    mcp.run(transport="stdio")

Conclusion

In this cookbook, we've built a powerful real-time news extraction server using the Model Context Protocol and Hyperbrowser. This combination enables AI models to access current news content that would otherwise be unavailable to them due to training cutoffs or access limitations.

By leveraging MCP, we've created a standardized interface that allows any compatible AI to:

Search for the latest news on any topic of interest
Access structured news data with titles, sources, and summaries
Retrieve geographically relevant local news

All without requiring custom training or hardcoded integrations for each news source.

Next Steps

To take this news extraction system further, you might consider:

Creating specialized tools for financial, sports, or technology news with more indepth analysis
Adding historical news search capabilities
Implementing source credibility scoring

The MCP protocol opens up possibilities far beyond news - any web-based or local data source can be made available to AI models using this same pattern, dramatically expanding their real-time knowledge capabilities.