Computer Actions

Computer Actions let you drive a live session using screen-level primitives like click, type, drag, scroll, and screenshot. Use them when DOM automation is unreliable or impossible (canvas apps, remote desktops, non-standard controls, or stubborn overlays).

When To Use Computer Actions

You need full-screen interaction, not just DOM selectors.
The page uses canvas or custom rendering where selectors are unreliable.
You want a robust fallback when Playwright/Puppeteer actions fail.

How It Works

Every session includes a computerActionEndpoint. The SDKs wrap this for you with client.computerAction.* (Node.js) and client.computer_action.* (Python). You can pass either a session ID or the full session object, and passing the session object is recommended.

Coordinates are pixels relative to the top-left of the session screen. The default screen size is 1280x720. If you change screen in session creation, adjust your coordinates accordingly.

Quickstart

This example navigates using keyboard input, scrolls, and captures a screenshot.

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

async function main() {
  const session = await client.sessions.create();

  try {
    // Focus address bar and navigate
    await client.computerAction.pressKeys(session, ["Control_L", "l"]);
    await client.computerAction.typeText(session, "https://example.com");
    await client.computerAction.pressKeys(session, ["Return"]);

    const shot = await client.computerAction.screenshot(session);

    console.log(shot.screenshot); // base64
  } finally {
    await client.sessions.stop(session.id);
  }
}

main().catch(console.error);

from hyperbrowser import Hyperbrowser
import os
from dotenv import load_dotenv

load_dotenv()

client = Hyperbrowser(api_key=os.environ["HYPERBROWSER_API_KEY"])

session = client.sessions.create()

try:
    # Focus address bar and navigate
    client.computer_action.press_keys(session, keys=["Control_L", "l"])
    client.computer_action.type_text(session, text="https://example.com")
    client.computer_action.press_keys(session, keys=["Return"])

    shot = client.computer_action.screenshot(session)

    print(shot.screenshot)  # base64
finally:
    client.sessions.stop(session.id)

Action Examples

Click

const response = await client.computerAction.click(
  "session-id",  // or session object
  500,  // x coordinate
  300,  // y coordinate
  "left",  // button: "left" | "right" | "middle" | "back" | "forward" | "wheel"
  1,  // number of clicks
  false  // do not return screenshot (default: false)
);

console.log(response.success);
console.log(response.screenshot);  // base64 if requested

response = client.computer_action.click(
    "session-id",  # or session object
    x=500,
    y=300,
    button="left",  # "left" | "right" | "middle" | "back" | "forward" | "wheel"
    num_clicks=1,
    return_screenshot=False  # do not return screenshot (default: False)
)

print(response.success)
print(response.screenshot)  # base64 if requested

Type Text

const response = await client.computerAction.typeText(
  "session-id",
  "Hello, World!",
  false  // do not return screenshot (default: false)
);

response = client.computer_action.type_text(
    "session-id",
    text="Hello, World!",
    return_screenshot=False  # do not return screenshot (default: False)
)

Press Keys

Uses the xdotool format for keys: https://github.com/sickcodes/xdotool-gui/blob/master/key_list.csv

const response = await client.computerAction.pressKeys(
  "session-id",
  ["Control_L", "a"],  // Key combination
  false  // do not return screenshot (default: false)
);

response = client.computer_action.press_keys(
    "session-id",
    keys=["Control_L", "a"],  # Key combination
    return_screenshot=False  # do not return screenshot (default: False)
)

Move Mouse

const response = await client.computerAction.moveMouse(
  "session-id",
  500,  // x
  300,  // y
  false  // do not return screenshot (default: false)
);

response = client.computer_action.move_mouse(
    "session-id",
    x=500,
    y=300,
    return_screenshot=False  # do not return screenshot (default: False)
)

Drag

const response = await client.computerAction.drag(
  "session-id",
  [
    { x: 100, y: 100 },
    { x: 200, y: 200 },
    { x: 300, y: 300 },
  ],
  false  // do not return screenshot (default: false)
);

response = client.computer_action.drag(
    "session-id",
    path=[
        {"x": 100, "y": 100},
        {"x": 200, "y": 200},
        {"x": 300, "y": 300},
    ],
    return_screenshot=False,  # do not return screenshot (default: False)
)

from hyperbrowser.models import Coordinate

response = client.computer_action.drag(
    "session-id",
    path=[
        Coordinate(x=100, y=100),
        Coordinate(x=200, y=200),
        Coordinate(x=300, y=300),
    ],
    return_screenshot=False,  # do not return screenshot (default: False)
)

Scroll

const response = await client.computerAction.scroll(
  "session-id",
  500,  // x position
  300,  // y position
  0,  // scroll x delta
  100,  // scroll y delta
  false  // do not return screenshot (default: false)
);

response = client.computer_action.scroll(
    "session-id",
    x=500,
    y=300,
    scroll_x=0,
    scroll_y=100,
    return_screenshot=False  # do not return screenshot (default: False)
)

Screenshot

const response = await client.computerAction.screenshot("session-id");
console.log(response.screenshot);  // base64

response = client.computer_action.screenshot("session-id")
print(response.screenshot)  # base64

Get Started

Browser Sessions

Session Configuration

Web API

Agents

HyperAgent Usage

Scraping

Integrations

When To Use Computer Actions

How It Works

Quickstart

Action Examples

Click

Type Text

Press Keys

Move Mouse

Drag

Scroll

Screenshot

​When To Use Computer Actions

​How It Works

​Quickstart

​Action Examples

​Click

​Type Text

​Press Keys

​Move Mouse

​Drag

​Scroll

​Screenshot

​Related

When To Use Computer Actions

How It Works

Quickstart

Action Examples

Click

Type Text

Press Keys

Move Mouse

Drag

Scroll

Screenshot

Related