Computer Actions

Computer Actions let you drive a live session using screen-level primitives like click, type, drag, scroll, and screenshot. Use them when DOM automation is unreliable or impossible (canvas apps, remote desktops, non-standard controls, or stubborn overlays).

When To Use Computer Actions

You need full-screen interaction, not just DOM selectors.
The page uses canvas or custom rendering where selectors are unreliable.
You want a robust fallback when Playwright/Puppeteer actions fail.

How It Works

Every session includes a computerActionEndpoint. The SDKs wrap this for you with client.computerAction.* (Node.js) and client.computer_action.* (Python). You can pass either a session ID or the full session object, and passing the session object is recommended. Cursor position, current-cursor helpers, and modifier keys require Node SDK 1.1.0+ or Python SDK 1.10.0+.

Coordinates are pixels relative to the top-left of the session screen. The default screen size is 1280x720. If you change screen in session creation, adjust your coordinates accordingly.

Using cURL

You can also send JSON requests directly to the computerActionEndpoint returned when you create a session. The full URL already includes its authentication token, so no additional authentication header is needed. The cURL examples require curl and jq. Save your session creation response as session.json, then set the endpoint once in your shell:

COMPUTER_ACTION_ENDPOINT="$(jq -er '.computerActionEndpoint' session.json)"

Keep the full URL, including its query parameters. Each action is a POST to this same URL. JSON action names use snake case, such as press_keys, while fields such as returnScreenshot, numClicks, and scrollY use camel case.

Quickstart

This example navigates using keyboard input, scrolls, and captures a screenshot.

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

async function main() {
  const session = await client.sessions.create();

  try {
    // Focus address bar and navigate
    await client.computerAction.pressKeys(session, ["Control_L", "l"]);
    await client.computerAction.typeText(session, "https://example.com");
    await client.computerAction.pressKeys(session, ["Return"]);

    const shot = await client.computerAction.screenshot(session);

    console.log(shot.screenshot); // base64
  } finally {
    await client.sessions.stop(session.id);
  }
}

main().catch(console.error);

from hyperbrowser import Hyperbrowser
import os
from dotenv import load_dotenv

load_dotenv()

client = Hyperbrowser(api_key=os.environ["HYPERBROWSER_API_KEY"])

session = client.sessions.create()

try:
    # Focus address bar and navigate
    client.computer_action.press_keys(session, keys=["Control_L", "l"])
    client.computer_action.type_text(session, text="https://example.com")
    client.computer_action.press_keys(session, keys=["Return"])

    shot = client.computer_action.screenshot(session)

    print(shot.screenshot)  # base64
finally:
    client.sessions.stop(session.id)

# Focus the address bar and navigate
curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"press_keys","keys":["Control_L","l"]}'

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"type_text","text":"https://example.com"}'

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"press_keys","keys":["Return"]}'

# Allow time for the page to load before interacting
sleep 2

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"scroll","x":500,"y":300,"scrollX":0,"scrollY":100}'

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"screenshot"}' \
  -o screenshot.json

jq -r '.screenshot' screenshot.json # base64

Action Examples

The cURL examples use COMPUTER_ACTION_ENDPOINT set above. Successful actions return JSON with success: true. Add "returnScreenshot": true to an action to include a base64 screenshot in the response; the screenshot action always returns one. Click, drag, and scroll accept optional keys in xdotool format, such as Control_L or Shift_L. These keys are held during the action and released afterward. Omit keys for an unmodified action.

Cursor Position

Read the pointer’s current position in screen pixels. Like other actions, this returns ComputerActionResponse; check the shared data variant before reading x and y. You can also request a screenshot with returnScreenshot (Node.js/cURL) or return_screenshot (Python).

const response = await client.computerAction.cursorPosition("session-id");
if (response.success && response.data && "x" in response.data) {
  console.log(response.data.x, response.data.y);
}

from hyperbrowser.models import ComputerActionResponseDataCursorPosition

response = client.computer_action.cursor_position("session-id")
if response.success and isinstance(response.data, ComputerActionResponseDataCursorPosition):
    print(response.data.x, response.data.y)

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"cursor_position","returnScreenshot":false}'

Click

const response = await client.computerAction.click(
  "session-id",  // or session object
  500,  // x coordinate
  300,  // y coordinate
  "left",  // button: "left" | "right" | "middle" | "back" | "forward" | "wheel"
  1,  // number of clicks
  false,  // do not return screenshot (default: false)
  ["Control_L"]  // optional modifier keys
);

console.log(response.success);
console.log(response.screenshot);  // base64 if requested

response = client.computer_action.click(
    "session-id",  # or session object
    x=500,
    y=300,
    button="left",  # "left" | "right" | "middle" | "back" | "forward" | "wheel"
    num_clicks=1,
    return_screenshot=False,  # do not return screenshot (default: False)
    keys=["Control_L"]  # optional modifier keys
)

print(response.success)
print(response.screenshot)  # base64 if requested

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"click","x":500,"y":300,"button":"left","numClicks":1,"returnScreenshot":false,"keys":["Control_L"]}'

Click at the Current Cursor

Use clickAtCursor() in Node.js, or omit both x and y from Python’s click(), to click without moving the pointer. Node.js click() still requires coordinates.

const response = await client.computerAction.clickAtCursor("session-id");

response = client.computer_action.click("session-id")

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"click","button":"left","numClicks":1}'

Type Text

const response = await client.computerAction.typeText(
  "session-id",
  "Hello, World!",
  false  // do not return screenshot (default: false)
);

response = client.computer_action.type_text(
    "session-id",
    text="Hello, World!",
    return_screenshot=False  # do not return screenshot (default: False)
)

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"type_text","text":"Hello, World!","returnScreenshot":false}'

Press Keys

Uses the xdotool format for keys: https://github.com/sickcodes/xdotool-gui/blob/master/key_list.csv

const response = await client.computerAction.pressKeys(
  "session-id",
  ["Control_L", "a"],  // Key combination
  false  // do not return screenshot (default: false)
);

response = client.computer_action.press_keys(
    "session-id",
    keys=["Control_L", "a"],  # Key combination
    return_screenshot=False  # do not return screenshot (default: False)
)

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"press_keys","keys":["Control_L","a"],"returnScreenshot":false}'

Move Mouse

const response = await client.computerAction.moveMouse(
  "session-id",
  500,  // x
  300,  // y
  false  // do not return screenshot (default: false)
);

response = client.computer_action.move_mouse(
    "session-id",
    x=500,
    y=300,
    return_screenshot=False  # do not return screenshot (default: False)
)

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"move_mouse","x":500,"y":300,"returnScreenshot":false}'

Drag

const response = await client.computerAction.drag(
  "session-id",
  [
    { x: 100, y: 100 },
    { x: 200, y: 200 },
    { x: 300, y: 300 },
  ],
  false,  // do not return screenshot (default: false)
  ["Shift_L"]  // optional modifier keys
);

response = client.computer_action.drag(
    "session-id",
    path=[
        {"x": 100, "y": 100},
        {"x": 200, "y": 200},
        {"x": 300, "y": 300},
    ],
    return_screenshot=False,  # do not return screenshot (default: False)
    keys=["Shift_L"],  # optional modifier keys
)

from hyperbrowser.models import Coordinate

response = client.computer_action.drag(
    "session-id",
    path=[
        Coordinate(x=100, y=100),
        Coordinate(x=200, y=200),
        Coordinate(x=300, y=300),
    ],
    return_screenshot=False,  # do not return screenshot (default: False)
    keys=["Shift_L"],  # optional modifier keys
)

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"drag","path":[{"x":100,"y":100},{"x":200,"y":200},{"x":300,"y":300}],"returnScreenshot":false,"keys":["Shift_L"]}'

Scroll

scroll() moves the pointer to the supplied coordinates before scrolling. Its coordinate arguments remain required in both SDKs.

const response = await client.computerAction.scroll(
  "session-id",
  500,  // x position
  300,  // y position
  0,  // scroll x delta
  100,  // scroll y delta
  false,  // do not return screenshot (default: false)
  ["Control_L"]  // optional modifier keys
);

response = client.computer_action.scroll(
    "session-id",
    x=500,
    y=300,
    scroll_x=0,
    scroll_y=100,
    return_screenshot=False,  # do not return screenshot (default: False)
    keys=["Control_L"]  # optional modifier keys
)

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"scroll","x":500,"y":300,"scrollX":0,"scrollY":100,"returnScreenshot":false,"keys":["Control_L"]}'

Scroll at the Current Cursor

Use scrollAtCursor() (Node.js) or scroll_at_cursor() (Python) to scroll without moving the pointer. Both deltas default to zero, so pass a nonzero delta to scroll. You can optionally hold modifier keys, just as with scroll().

const response = await client.computerAction.scrollAtCursor(
  "session-id", 0, 2, false, ["Control_L"]
);

response = client.computer_action.scroll_at_cursor(
    "session-id", scroll_y=2, keys=["Control_L"]
)

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"scroll","scrollX":0,"scrollY":2,"returnScreenshot":false,"keys":["Control_L"]}'

Screenshot

const response = await client.computerAction.screenshot("session-id");
console.log(response.screenshot);  // base64

response = client.computer_action.screenshot("session-id")
print(response.screenshot)  # base64

curl --fail-with-body -sS "$COMPUTER_ACTION_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"action":"screenshot"}' \
  -o screenshot.json

jq -r '.screenshot' screenshot.json # base64

Get Started

Browser Sessions

Session Configuration

Web API

Agents

HyperAgent Usage

Scraping

Integrations

When To Use Computer Actions

How It Works

Using cURL

Quickstart

Action Examples

Cursor Position

Click

Click at the Current Cursor

Type Text

Press Keys

Move Mouse

Drag

Scroll

Scroll at the Current Cursor

Screenshot

​When To Use Computer Actions

​How It Works

​Using cURL

​Quickstart

​Action Examples

​Cursor Position

​Click

​Click at the Current Cursor

​Type Text

​Press Keys

​Move Mouse

​Drag

​Scroll

​Scroll at the Current Cursor

​Screenshot

​Related

When To Use Computer Actions

How It Works

Using cURL

Quickstart

Action Examples

Cursor Position

Click

Click at the Current Cursor

Type Text

Press Keys

Move Mouse

Drag

Scroll

Scroll at the Current Cursor

Screenshot

Related