Building an Ingredients-Based Recipe Suggester with Hyperbrowser and GPT-4o

In this cookbook, we'll create a powerful AI-powered recipe recommendation system that can:

Analyze photos of ingredients using GPT-4o's vision capabilities
Extract and prioritize the most useful ingredients in the image
Autonomously search for recipes that use those ingredients
Return curated recipe suggestions with links

This approach combines:

OpenAI's GPT-4o for image understanding and ingredient extraction
Hyperbrowser's autonomous agents to navigate recipe websites and gather recommendations

By the end of this cookbook, you'll have a versatile tool that can look at what's in your fridge and tell you exactly what you can cook!

Prerequisites

Before starting, make sure you have:

A Hyperbrowser API key (sign up at hyperbrowser.ai if you don't have one)
An OpenAI API key with access to GPT-4o
Python 3.9+ installed

Both API keys should be stored in a .env file in the same directory as this notebook.

HYPERBROWSER_API_KEY=your_hyperbrowser_key_here
OPENAI_API_KEY=your_openai_api_key_here

Step 1: Set up imports and initialize clients

We'll start by importing the necessary libraries and initializing our API clients.

import os
from typing import Optional

from dotenv import load_dotenv
from hyperbrowser import AsyncHyperbrowser
from hyperbrowser.models.beta.agents.browser_use import StartBrowserUseTaskParams
from openai import AsyncOpenAI
from openai.types.chat import ChatCompletionMessageParam

import base64


load_dotenv()

hb = AsyncHyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))
llm = AsyncOpenAI()

Step 2: Create the ingredient analysis function

Next, we'll create a function that can extract ingredients from a food image. This function:

Encodes the image as base64 for transmission to the OpenAI API
Crafts a prompt that instructs the model to identify only food ingredients
Prioritizes ingredients by their culinary usefulness
Returns a standardized list of the top 8 most useful ingredients

By limiting to 8 ingredients and prioritizing them, we ensure the recipe search will focus on the most important components rather than getting distracted by minor items in the image.

Here's an example image that we'll be working off of -

An example list of ingredients

undefined

async def analyze_ingredients_from_image(image_path: str) -> Optional[str]:

    with open(image_path, "rb") as image_file:
        image_data = base64.b64encode(image_file.read()).decode("utf-8")

        messages: list[ChatCompletionMessageParam] = [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Please analyze this image and list all the ingredients you can see in it. Focus only on food ingredients and kitchen items. Return them as a simple list. Do not return specific products (like doritos), but answer with generic names (like tortilla chips). Answer only with specifics (like bread slices), do not offer alternatives (like Bread slices (or chips/crisps) or any alternative names, only the most common name. Also order them in order of how useful they would be for cooking, and give me only the top 8.",
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
                    },
                ],
            }
        ]

        response = await llm.chat.completions.create(
            model="gpt-4o", messages=messages, max_tokens=1000
        )

        # Extract ingredients from response
        ingredients_text = response.choices[0].message.content

        return ingredients_text

Step 3: Create the recipe finder function

Now we'll create a function that takes our list of ingredients and searches for suitable recipes. We'll use Hyperbrowser's browser_use agent, which can:

Navigate to recipe websites autonomously
Enter our ingredients into search fields
Find appropriate recipes that use our available ingredients
Extract and return recipe names and URLs

This approach is significantly more flexible than using a fixed API, as the agent can adapt to different website layouts and search functionality.

async def get_recipes_from_ingredients(ingredients_list: str):

    # Start browser session and get recipes
    browser_session = await hb.agents.browser_use.start_and_wait(
        StartBrowserUseTaskParams(
            task=f"""Go to https://www.reciperadar.com, and from the list of ingredients, get me a list of things I can cook. You should be able to enter the ingredients on the page itself. Do not navigate away from this page. Do not search before you've entered all the ingredients. In some cases you might not be able to enter a specific ingredient. In that case, feel free to remove it.
 
 Return to me the URLs of the things I can make along with the name of the recipe. I want 3 recipes.
 
 I'm not too picky, but preferably I'd want italian recipes.
 
 Here is the list of ingredients that I have: {ingredients_list}""",
            use_vision=True,
            max_steps=30,
        )
    )

    return browser_session.data

Step 4: Test the system with a real image

Let's test our recipe suggester by analyzing a real image of ingredients and seeing what recipes it can find. We'll trace the complete workflow:

First, we'll analyze the image to extract ingredients
Then we'll use those ingredients to search for suitable recipes
Finally, we'll display the recommended recipes with their URLs

This demonstrates the end-to-end process from image to recipe suggestions.

ingredients_list = await analyze_ingredients_from_image("./assets/ingredients.jpg")
print(f"Got ingredients {ingredients_list}")

Got ingredients Sure! Here are the top 8 food ingredients and kitchen items from the image:



1. Lettuce

2. Parsley

3. Green onions

4. Avocados

5. Baby spring mix

6. Cucumber

7. Black beans

8. Cherry tomatoes

if ingredients_list is None:
    raise Exception("No ingredients list could be found")
else:
    recipes = await get_recipes_from_ingredients(ingredients_list)
    print(recipes.final_result)

I have successfully entered all ingredients and extracted the first 3 recipes. The recipes are:

1. Avocado Ranch BLT Salad in Bacon Cups - https://tastykitchen.com/recipes/salads/avocado-ranch-blt-salad-in-bacon-cups/

2. 11 Layer Salad - https://tastykitchen.com/recipes/salads/11-layer-salad/

3. BBQ Tofu Salad - https://tastykitchen.com/recipes/salads/bbq-tofu-salad/



These are not italian recipes, but I used all the ingredients you provided.

Next Steps

To take this further, you might consider:

Adding dietary preference filters (vegetarian, gluten-free, etc.)
Implementing quantity recognition for more precise recipe matching
Creating a mobile interface for easier kitchen use
Expanding to multiple recipe websites for greater variety
Including nutritional information in the recommendations

Conclusion

In this cookbook, we've built a powerful ingredients-based recipe suggester that combines:

ChatGPT computer vision for ingredient extraction from images
Autonomous web agents for recipe discovery
Natural language processing for prioritizing ingredients

This system demonstrates how AI can bridge the gap between raw data and actionable insights, showcasing how vision models, autonomous agents, and NLP can be combined to create powerful developer tools and automation workflows; in this case to reduce food waste and eat more healthily.

Happy cooking!