C_W

Working with agents by ear

After eye surgery I couldn't look at a screen for long, so I steered my coding agents by voice. What made that usable, and the plugin that came out of it.

Topic
Experiments
Reading
3 min

After eye surgery I couldn’t look at a screen for long, and the agents still needed steering. So I asked one to stop sending me tables and just talk. A few days later that had turned into a plugin for Claude Code. It’s called vox and it’s open source.

Three things made it usable

Claude Code has a voice mode, but you hold a key to talk, and that didn’t work in my terminal. Having the reply read aloud was the easy part. These made it something I could work with all day:

  • An open mic that sends by itself. I talk, I stop, the message goes. A pause in the middle of a thought waits a little longer than a pause after a full sentence.
  • It knows my voice. An open mic hears everyone. Once the helper knew my voice, somebody else talking in the room no longer ended up in the session. It cancels echo too, so the agent doesn’t hear itself, and it stops talking the moment I start.
  • Big letters. The last words sit above the prompt in block letters I can read at a glance.
Diagram
flowchart TB
  M["Mic<br/>echo cancelled"] --> H["Helper<br/>types the words as I talk"]
  H --> V{"My voice?"}
  V -->|no| X["Dropped"]
  V -->|yes| P["A pause<br/>shorter after a full sentence"]
  P --> F["A second model<br/>reads the message again"]
  F --> S["Sent to the agent"]:::lit
  S --> R["The reply<br/>spoken as it streams"]

A fox instead of a status bar

One line above the prompt holds everything: a fox whose face is the state, the two switches, the mic as an equalizer and the words so far. Here it is in five states.

Clicking the fox opens the full view, where the last words stand five rows tall under a larger equalizer.

The agent had to learn to talk

A reply written for the screen is useless by ear. Nobody wants a table read out. So the plugin ships a writing style for the agent:

  • Answer first. The first sentence has to make sense on its own, because the listener may stop there.
  • One or two short sentences. No lists and no headings, since speech flattens them.
  • Choices by number. Options are read out as one, two, three, and I answer with one key.

It forgot, of course. On the first day it answered with a list and I had to remind it of its own rules. Now the plugin loads them again every 25 prompts.

What it gets wrong

Dictation mishears names. My terminal is called herdr, and it came out as “her” and “herder”. “Wait mode” became “weight mode”. A small dictionary fixes the words I use most, and a second speech model reads every message again before it’s sent, in about 0.06 seconds on an M2.

Speech recognition runs on the machine, and no audio leaves it.

Try it

/plugin marketplace add cwdx/vox
/plugin install vox@vox
/vox setup
macOS Linux
Speech say piper or espeak-ng
Dictation Swift, echo cancelled Python
Only my voice yes not yet

Dictation types into herdr panes, so it needs herdr. Much of vox was built by asking for it out loud, which is a decent test of a voice tool.

Open questions

  • How do you review code by ear? Short glances at the screen were still part of it.
  • Recognising one voice only works on macOS so far. What does it take on Linux?