Building a Content Discovery Agent with Hyperbrowser and GPT-4o

In this cookbook, we'll build an intelligent agent that can find related content from content creators across the web. Given a link to one piece of content (like a video, article, or post), our agent will:

Identify the creator/channel/profile that published the content
Find the homepage or profile page of that creator
Discover other content published by the same creator

We'll use these tools to build our agent:

Hyperbrowser for web scraping and data extraction
OpenAI's GPT-4o for reasoning about web content and finding connections

By the end of this cookbook, you'll have a versatile agent that can help you discover more content from creators you enjoy!

Prerequisites

To follow along you'll need the following:

A Hyperbrowser API key (sign up at hyperbrowser.ai if you don't have one, it's free)
An OpenAI API key (sign up at openai.com if you don't have one, it's free)

Both API keys should be stored in a .env file in the same directory as this notebook with the following format:

HYPERBROWSER_API_KEY=your_hyperbrowser_key_here
OPENAI_API_KEY=your_openai_key_here

Step 1: Set up imports and load environment variables

First, we'll import the necessary libraries and load our environment variables from the .env file.

import asyncio
import json
import os

from dotenv import load_dotenv
from hyperbrowser import AsyncHyperbrowser
from hyperbrowser.tools import WebsiteExtractTool, WebsiteScrapeTool
from openai import AsyncOpenAI
from openai.types.chat import (
    ChatCompletionMessageParam,
    ChatCompletionMessageToolCall,
    ChatCompletionToolMessageParam,
)

load_dotenv()

Step 2: Initialize clients

Next, we'll create instances of Hyperbrowser and OpenAI's API clients using our API keys.

hb = AsyncHyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))
llm = AsyncOpenAI()

Step 3: Create a tool handler function

This function processes tool calls from the LLM and executes the appropriate web extraction or scraping operations. It then returns the results back to the agent for further analysis.

Currently, it's set up to use both the scraping and structured extraction tool, but should be easy to set up with other tools as well.

async def handle_tool_call(
    tc: ChatCompletionMessageToolCall,
) -> ChatCompletionToolMessageParam:
    print(f"Handling tool call: {tc.function.name}")

    try:
        if (
            tc.function.name
            == WebsiteExtractTool.openai_tool_definition["function"]["name"]
        ):
            args = json.loads(tc.function.arguments)
            content = await WebsiteExtractTool.async_runnable(hb=hb, params=args)
            return {"role": "tool", "tool_call_id": tc.id, "content": content}
        elif (
            tc.function.name
            == WebsiteScrapeTool.openai_tool_definition["function"]["name"]
        ):
            args = json.loads(tc.function.arguments)
            content = await WebsiteScrapeTool.async_runnable(hb=hb, params=args)
            return {"role": "tool", "tool_call_id": tc.id, "content": content}
        else:
            raise ValueError(f"Tool not found: {tc.function.name}")

    except Exception as e:
        err_msg = f"Error handling tool call: {e}"
        print(err_msg)
        return {
            "role": "tool",
            "tool_call_id": tc.id,
            "content": err_msg,
            "is_error": True,  # type: ignore
        }

Step 4: Implement the agent loop

This is the core of our agent's functionality. The agent loop manages the conversation between the model and its tools, allowing it to make multiple tool calls as needed to gather information about the content creator and their other works.

Simply put,

it takes in a list of messages that include the system prompt, and the user query.
Sends them to Open AI,
Processes the tool calls (if any)
Continues processing until the "stop" message is called

async def agent_loop(messages: list[ChatCompletionMessageParam]) -> str:
    while True:
        response = await llm.chat.completions.create(
            messages=messages,
            model="gpt-4o",
            tools=[
                WebsiteExtractTool.openai_tool_definition,
                WebsiteScrapeTool.openai_tool_definition,
            ],
            max_completion_tokens=8000,
        )

        choice = response.choices[0]

        # Append response to messages
        messages.append(choice.message)  # type: ignore

        # Handle tool calls
        if (
            choice.finish_reason == "tool_calls"
            and choice.message.tool_calls is not None
        ):
            tool_result_messages = await asyncio.gather(
                *[handle_tool_call(tc) for tc in choice.message.tool_calls]
            )
            messages.extend(tool_result_messages)

        elif choice.finish_reason == "stop" and choice.message.content is not None:
            return choice.message.content

        else:
            print(choice)
            raise ValueError(f"Unhandled finish reason: {choice.finish_reason}")

Step 5: Design the system prompt

The system prompt dicatates the behavior of the LLM. Our prompt establishes what the agent should do: find the creator/channel of the provided content, and then if needed, locate their homepage, and discover other content they've published.

Note that we also provide a reasonably detailed description of the tools that the LLM can use.

SYSTEM_PROMPT = """
You are a helpful assistant that can help me find content on the internet. You will be supplied with url - {link}, containing content belonging to a certain channel/profile. You will be required to find other content belonging to the same channel/profile. You can either 
- Get the relevant content directly from the page itself, or 
- You can scrape the provided link, to get the homepage of the channel/profile. You can scrape the homepage to get the other content belonging to the same channel/profile.

To do this, you have access to two tools:
1. You can use the 'extract_data' tool to get the structured data from the webpage, although you will have to provide the formatted json schema. The json schema must have a object at the root/top level.

2. You can use the 'scrape_website' tool to scrape the website and get the markdown content directly from the website.

You will return to me two things
- The url of the homepage of the channel/profile
- The list of urls of the other content belonging to the same channel/profile.
""".strip()

Step 6: Create a factory function for generating content discovery agents

Now we'll create a factory function that generates specialized content discovery agents. This function takes a content URL as input and returns a function that can discover other content from the same creator.

Some minor cleaning is also done on the link to ensure that it is properly formatted.

from typing import Coroutine, Any, Callable


def make_social_media_agent(
    link_to_profile: str,
) -> Callable[..., Coroutine[Any, Any, str]]:
    # Popular documentation providers like Gitbook, Mintlify etc automatically generate a llms.txt file
    # for documentation sites hosted on their platforms.
    if not (
        link_to_profile.startswith("http://") or link_to_profile.startswith("https://")
    ):
        link_to_profile = f"https://{link_to_profile}"

    sysprompt = SYSTEM_PROMPT.format(
        link=link_to_profile,
    )

    async def solve_code(question: str) -> str:
        return await agent_loop(
            [
                {"role": "system", "content": sysprompt},
                {"role": "user", "content": question},
            ]
        )

    return solve_code

Step 7: Test the agent with a real content URL

Let's test our agent by creating an instance for a YouTube video and asking it to find other content from the same channel. This will demonstrate the full workflow:

The agent receives a link to a YouTube video
It uses web scraping tools to identify the channel that published the video
It finds the channel's homepage
It discovers other videos published by the same channel
It returns both the channel URL and a list of other content URLs

You'll see the tool calls being made in real-time as the agent works.

link_to_social_media_profile = "https://www.youtube.com/watch?v=9066onUKCl8"

question = "Get me the list of videos belonging to the channel/profile"

agent = make_social_media_agent(link_to_social_media_profile)

response = await agent(question)

print(response)

Handling tool call: scrape_webpage

The video belongs to the YouTube channel "Mentour Now!". Here are the details you requested:



- **Channel Homepage**: [Mentour Now!](https://www.youtube.com/@MentourNow)

  

- **List of Videos from the Channel**:

  1. [Did Airbus just Change EVERYTHING?!](https://www.youtube.com/watch?v=9066onUKCl8)

  2. [WHO is Winning the FIGHT between AIRBUS and QATAR Airways?!](https://www.youtube.com/watch?v=0KSkobkfKtQ)

  3. [THIS Is the LAST THING Boeing Needs!](https://www.youtube.com/watch?v=6If2jSyeAus)

  4. [This Engine Could Seriously Threaten Boeing!](https://www.youtube.com/watch?v=57RlvnjPKM4)

  5. [Boeing 777-10 - The PERFECT 747 Replacement?](https://www.youtube.com/watch?v=sH5q1h9UiZA)

  6. [The B-36 Peacemaker Had 10 ENGINES - Why Did It Fail?](https://www.youtube.com/watch?v=8gfHLU6wxJA)

  7. [This New Airbus Is About to Change Everything...](https://www.youtube.com/watch?v=dyRvlyqM4mY)



These are some of the recent videos posted by the channel "Mentour Now!".

Conclusion

In this cookbook, we built a powerful content discovery agent using Hyperbrowser and OpenAI's GPT-4o. This agent can:

Take any content URL (like a video, article, or social media post)
Identify the creator, channel, or profile that published it
Find the creator's homepage or profile page
Discover other content published by the same creator
Return a structured list of related content

This pattern can be extended to create more sophisticated content discovery tools or be integrated into larger applications like content recommendation engines, research assistants, or social media monitoring tools.

Next Steps

To take this further, you might consider:

Adding support for more platforms (Twitter, Substack, Medium, etc.)
Implementing content filtering based on topics or keywords
Creating a web interface where users can paste content links
Adding content summarization for each discovered piece
Building a database to track creators and their content over time

Happy content discovering!