slate-desktop
Background computer use for Linux. Gives an agent its own Wayland seat (pointer, keyboard, focus, clipboard), per-window screenshots, and input, exposed over MCP so Claude Code and Codex can drive GUI apps without touching the human's mouse.
Status
Works, verified on sway 1.12 (headless and on a real display, HiDPI included); the SlateOS desktop runs it as a supervised user service. Needs a compositor that implements:
ext_transient_seat_manager_v1zwlr_virtual_pointer_manager_v1(v2) andzwp_virtual_keyboard_manager_v1ext_image_copy_capture_manager_v1withext_foreign_toplevel_image_capture_source_manager_v1for per-window captureext_foreign_toplevel_list_v1
That is wlroots-based compositors today. GNOME and KDE do not expose these to clients. Window geometry (for window-relative clicks) comes from sway IPC; on other compositors screenshots and absolute coordinates still work.
The daemon
slate-desktop daemon owns the agent seat for the whole session and must start before any application: toolkits only accept input from seats that existed when they started. The Slate sway profile runs it first (exec slate-desktop daemon). slate-desktop serve (MCP) and the CLI use the daemon when it is running and fall back to a private seat otherwise.
Borrowing the user's seat
seat: "user" actions put sway into a controlling binding mode (the profile binds Esc there to slate-desktop takeover-cancel), the panel shows a blinking "controlling", and after Esc further user-seat input is refused for a minute so the agent has to ask again. desktop_status reports the state.
The background screen
On the SlateOS desktop sway has a second, headless output (HEADLESS-1, workspace agent). desktop_launch opens windows there unless where: "here"; desktop_show and desktop_hide move windows between the screens; desktop_windows shows each window's location. The agent seat has one virtual pointer per output and routes clicks by layout coordinates, so windows on either screen can be driven the same way. Without a headless output (plain sway), everything opens on the user's screen.
Apps the agent launches
Most toolkits bind every seat that exists when they start, so they take the agent's input as well as the person's. Chromium and everything built on it (Chrome, Electron apps, CEF) bind one seat, the first: the person's. On SlateOS, desktop_launch (in the background) therefore starts a Chromium-based app, known from Chromium's resource files next to its program or in its Flatpak, on a Wayland socket of its own, $XDG_RUNTIME_DIR/slate/agent-wayland, which the daemon opens through wp_security_context_v1 with the sandbox engine dev.benchgrid.slate.agent and the agent seat's name as instance id. SlateOS's SwayFX shows the clients of that context the agent's seat and no other, so the app binds it and takes the agent's keyboard and pointer. The person's do not reach it: desktop_windows marks such windows agent_seat_only, and desktop_launch and desktop_show say so. Since these apps run one instance, a window the person opens later could land in the agent's; the agent is told to close them when done. Apps for the person open with where: "here", on the session's own socket. Like any client with a security context, the agent's apps get no privileged protocol (screen capture, virtual input, layer shell), Flatpak apps included. The module turns this on with SLATE_DESKTOP_AGENT_DISPLAY=1 where its SwayFX carries the patch; elsewhere desktop_launch uses the session's socket.
The locked screen
While a screen locker runs (swaylock), the person decides what the agent may do in windows: agentWhileLocked in the system's settings.json, which only root writes (slateos-settings, behind the person's password; SlateOS's Settings, AI), or services.slate.desktop.agentWhileLocked. Without either, pause.
| Choice | The agent may |
|---|---|
pause | look: windows, elements, text, window screenshots |
elements | also element actions, and launch, close, move and resize windows; its keyboard and pointer go to the lock screen on stock sway, so they are refused, and so are the element fallbacks that use them |
continue | everything, when the compositor exempts its seat from the lock: the daemon sends seat <name> lock_exempt yes (a SlateOS SwayFX patch), after which sway leaves that seat's focus alone and gives its input to the windows behind the lock screen, never to the lock screen; on a compositor without the patch it acts as elements |
seat: "user" input is always refused while the screen is locked. desktop_status reports the state under lock. The daemon also writes the time the agent last used a tool to $XDG_RUNTIME_DIR/slate/agent-active, which SlateOS's idle lock waits on when the person lets the agent keep the screen awake.
The accessibility tree
desktop_elements, desktop_read, desktop_element_click and desktop_element_set_text work through AT-SPI2, the accessibility bus every Linux toolkit speaks. Elements come with a role, a name, a value, states, window-relative extents and their actions; text comes back as text. desktop_element_click runs the element's own action (Action.DoAction) and desktop_element_set_text uses EditableText, so neither needs a pointer or a keyboard and both work in toolkits that ignore extra seats (GTK4). Pointer and typing fallbacks kick in when an element has no action or is not editable over the bus.
Applications are matched to compositor windows by pid (sway reports the pid of each view; the bus reports the pid of each connection), frames by title. Trees are read from the application's Cache in one call, with a walk from the frame when the cache is shallow (GTK4 caches lazily). Verified: Thunar and pavucontrol (GTK3), the settings app (GTK4/libadwaita), Firefox (about 2000 nodes in under three seconds).
Two switches must be on, and the SlateOS desktop profile sets both: the bus must say accessibility is enabled (org.a11y.Status IsEnabled, mirrored from the toolkit-accessibility gsettings key; the daemon also sets it at connect), and NO_AT_BRIDGE must not be 1 (NixOS sets it unless services.gnome.at-spi2-core is enabled). Firefox and Chromium additionally read GNOME_ACCESSIBILITY / ACCESSIBILITY_ENABLED, Qt reads QT_LINUX_ACCESSIBILITY_ALWAYS_ON.
MCP tools
desktop_windows, desktop_elements, desktop_read, desktop_element_click, desktop_element_set_text, desktop_screenshot [window], desktop_click, desktop_move, desktop_scroll, desktop_type, desktop_key, desktop_close, desktop_focus, desktop_window_set, desktop_arrange, desktop_launch (with where), desktop_show, desktop_hide, desktop_seats, desktop_status, desktop_takeover_cancel. Windows are named by id (a unique prefix of 8 or more digits will do), app_id or title; of several matches by name, a window the person can see wins. Window-relative coordinates match window screenshots, so an agent can look, then click what it saw. desktop_type and desktop_key take a window, focus it for the seat in use, verify the focus through the compositor and report which window received the input. Every action returns a screenshot of the result.
slash adds this server to the agent backend automatically when WAYLAND_DISPLAY is set and the binary sits next to slash.
Toolkit compatibility
Whether an app reacts to the agent seat depends on its toolkit binding every wl_seat, not just the first. Verified on sway 1.12:
| Toolkit / app | Agent seat | Notes |
|---|---|---|
| foot | works | binds all seats |
| Firefox, GTK3 (Thunar) | works | when the seat predates the app (the daemon guarantees that) |
| GTK 4 / libadwaita (gnome-text-editor 49, Settings) | works | GTK 4.22, when the seat predates the app (measured 2026-10-25; the "ignored" of earlier measurements came from seats created after the app) |
| Electron (Obsidian 1.14), Chromium | works when the agent launches it on SlateOS; ignored otherwise | one seat is bound: the agent's for apps it launches in the background (see above), the person's otherwise, where element actions or seat: "user" remain; the accessibility tree works, but text fields are not editable over the bus |
| WeChat 4.1 | works (pointer) | draws its own interface: no accessibility tree, screenshots and clicks only |
| Flatpak apps | as their toolkit | matched to their windows through the sandbox (the bus sees Flatpak's proxy, not the app); measured: Flatseal (GTK4), Obsidian, WeChat |
| Qt | untested | contributions welcome |
seat: "user" injects through the human's own seat and needs approval (slated tier Confirm). Pointer and keyboard both work this way on a real display. See docs/decisions/0007-toolkits-and-the-agent-seat.md for the compositor-side fix that removes the need for it.
CLI (for testing)
slate-desktop probe
slate-desktop windows
slate-desktop shot [IDENT] out.png
slate-desktop elements WINDOW [QUERY]
slate-desktop read WINDOW
slate-desktop glance [CON] # what the person sees, as JSON (the Slate prompt's <screen> block)
slate-desktop element-click ID [ACTION]
slate-desktop element-set-text ID TEXT
slate-desktop click X Y [left|right|middle]
slate-desktop type "echo hi"
slate-desktop key Return
slate-desktop launch foot
slate-desktop --seat user click 700 600 # borrow the human's seat (a Chromium app the user opened)
Each CLI call creates a fresh transient seat; serve keeps one for the life of the MCP session.
How typing works
Arbitrary Unicode is typed by generating an xkb keymap where every distinct character in the text has its own keycode (the wtype approach), uploading it to the virtual keyboard, then pressing those codes. Named keys and modifiers live in the same keymap so ctrl+l and Return work. One quirk: the first key event after a keymap upload is dropped between the compositor and the client, so a reserved no-op key is pressed after every upload.
Not yet
- A distinct "ghost" cursor for the agent seat (needs compositor support or a patch)
- Takeover beyond Esc (freeze the seat, hand a window back)
- Multiple outputs