Vercel Labs agent-desktop 1.0: computer use without pixels

Every small business has one desktop app with no API. The accounting package, the shipping label tool, the old POS back office. agent-desktop, now a Vercel Labs project, reached 1.0 this week, and it goes after exactly that app. It gives an AI agent computer use through the operating system's accessibility tree, not screenshots and pixel guessing.

What actually happened

agent-desktop tagged v1.0.0 on October 28 and v1.0.2 on October 30. The 1.0.2 notes show the repo moving from its creator's account to the vercel-labs org. It's Apache-2.0, written in Rust, and installs with npm install -g agent-desktop on macOS, Windows and Linux.

How it works, per the README:

• Snapshot, act, snapshot. agent-desktop snapshot --app Finder -i returns JSON with a ref for every element the agent can act on. The agent clicks a ref, then takes a new snapshot to check the result.
• Refs expire. If the UI changes, an action returns STALE_REF and the agent must look again. Errors say whether a retry is safe.
• Headless by default. Actions go through the accessibility API without moving your mouse. Physical input needs a --headed flag.
• 60 command names cover clicks, typing, windows, notifications and the clipboard. A C library ships too, for in-process calls.
• Permissions: macOS needs an Accessibility grant. Linux needs at-spi2-core. Windows needs none for apps at your terminal's integrity level.

The README is blunt about privacy. Snapshots, screenshots and clipboard reads "can contain private desktop content". Keep them out of public logs.

Why desktop computer use matters for your business

It reads structure, not pictures. Pixel-based agents break when a button moves 10 pixels. An accessibility tree names the button. That makes small, repeatable desktop jobs realistic: export yesterday's report, copy order numbers into the shipping tool, close the daily batch.

Treat it like an employee login. An agent with Accessibility permission can operate anything you can. Run it under its own OS user, on a machine that holds only the apps it needs. Do not let it near your banking app.

Logs are sensitive data. If a snapshot of your customer list lands in a debug log, that log is now a customer list. Set retention before the first run.

Start with read-only jobs. Have the agent pull data out of the legacy app before you let it type data in. Add a human check on anything that posts, pays or deletes.

Sources: vercel-labs/agent-desktop on GitHub, agent-desktop releases.