Prepare an environment

Your environment must execute the requested actions and capture screenshots. Keep the same browser or desktop session available throughout the task. Use a browser for web applications or a VM for native desktop applications.

Set up a local browsing environment

Use a browser automation library such as Playwright or Selenium to execute actions and capture screenshots. These libraries run in your environment.

Recommended safeguards for local browser automation:

• Run the browser in an isolated environment.
• Pass an empty env object so the browser does not inherit host environment variables.
• Disable extensions and local file-system access where possible.

Install Playwright:

• Python: pip install playwright and then playwright install
• JavaScript: npm i playwright and then npx playwright install

Then launch a browser instance. Keep the browser and page alive while you run the remaining steps. In Python, those steps belong inside the with sync_playwright() block:

Start a browser instance

import { chromium } from "playwright";

const browser = await chromium.launch({
  headless: false,
  chromiumSandbox: true,
  env: {},
  args: ["--disable-extensions", "--disable-file-system"],
});
const page = await browser.newPage({
  viewport: { width: 1280, height: 720 },
});

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=False,
        chromium_sandbox=True,
        env={},
        args=["--disable-extensions", "--disable-file-system"],
    )
    page = browser.new_page(viewport={"width": 1280, "height": 720})

Set up a local virtual machine

For a desktop application, provide a VM or container and translate the returned actions into operating system input events.

Create a Docker image

The following Dockerfile starts an Ubuntu desktop with Xvfb, x11vnc, and Firefox:

Dockerfile

FROM ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y \
    xfce4 \
    xfce4-goodies \
    x11vnc \
    xvfb \
    xdotool \
    imagemagick \
    x11-apps \
    sudo \
    software-properties-common \
    firefox-esr \
 && apt-get remove -y light-locker xfce4-screensaver xfce4-power-manager || true \
 && apt-get clean && rm -rf /var/lib/apt/lists/*

RUN useradd -ms /bin/bash myuser \
    && echo "myuser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
USER myuser
WORKDIR /home/myuser

RUN x11vnc -storepasswd secret /home/myuser/.vncpass

EXPOSE 5900
CMD ["/bin/sh", "-c", "\
    Xvfb :99 -screen 0 1280x800x24 >/dev/null 2>&1 & \
    x11vnc -display :99 -forever -rfbauth /home/myuser/.vncpass -listen 0.0.0.0 -rfbport 5900 >/dev/null 2>&1 & \
    export DISPLAY=:99 && \
    startxfce4 >/dev/null 2>&1 & \
    sleep 2 && echo 'Container running!' && \
    tail -f /dev/null \
"]

Build the image:

docker build -t cua-image .

Run the container:

docker run --rm -it --name cua-image -p 5900:5900 -e DISPLAY=:99 cua-image

Create a helper for shelling into the container:

Execute commands on the container

import { execFile } from "node:child_process";
import { promisify } from "node:util";

const execFileAsync = promisify(execFile);

async function dockerExec(
  containerName,
  executable,
  args = [],
  { decode = true, env = {} } = {}
) {
  const environmentArgs = Object.entries(env).flatMap(([name, value]) => [
    "--env",
    `${name}=${value}`,
  ]);
  const output = await execFileAsync(
    "docker",
    [
      "exec",
      ...environmentArgs,
      containerName,
      executable,
      ...args.map(String),
    ],
    {
      encoding: decode ? "utf8" : "buffer",
      maxBuffer: 10 * 1024 * 1024,
    }
  );
  return output.stdout;
}

const vm = {
  display: ":99",
  containerName: "cua-image",
};

import subprocess

def docker_exec(cmd: str, container_name: str, decode: bool = True):
    safe_cmd = cmd.replace('"', '\\"')
    docker_cmd = f'docker exec {container_name} sh -c "{safe_cmd}"'
    output = subprocess.check_output(docker_cmd, shell=True)
    if decode:
        return output.decode("utf-8", errors="ignore")
    return output

class VM:
    def __init__(self, display: str, container_name: str):
        self.display = display
        self.container_name = container_name

vm = VM(display=":99", container_name="cua-image")