Hash Devlog: Tracing the Missing Keystrokes

I was building ghost text for Hash, an AI-powered shell. Instead of blocking while the agent thinks, ghost text streams dimmed suggestions inline as tokens arrive - like Copilot, but for shell commands. The feature worked, but something was wrong with input handling. Press Enter to accept a suggestion: nothing. Press again: nothing. Third time: finally works.

Same with Ctrl+D to exit. One press, two presses, three presses, finally exits. Intermittent. Frustrating. Impossible to reproduce reliably.

I threw Claude at it repeatedly. Each session followed the same pattern: describe symptoms → confident fix → bug persists. After several rounds, I realized the problem wasn’t Claude’s reasoning - it was my inputs.

The Debugging Dead End

Me: Enter key doesn’t work after the inline ?? suggestion appears. Takes 2-3 presses.

Claude: The issue is likely that the ghost text streaming completion isn’t properly updating its state. Let me fix the state transition…

Fix applied. Bug persists.

Me: Still broken. Here’s a screenshot of the terminal.

Claude: Ah, I see. This is a CSIu issue, the problem is that the Enter is wrongly interpreted as an Esc…

Another fix. Bug persists.

This pattern repeated across multiple sessions. Claude was making reasonable guesses based on incomplete information. I was describing symptoms, not providing evidence.

Building Hash’s Trace System

After the fifth or sixth failed fix, I stopped and thought about what I’d actually need to debug this: a trace of exactly what keys were pressed, when, on which goroutine, and what handlers processed them.

Hash already had a PTY-specific tracing system, enabled through HASH_PTY_TRACE, for low-level I/O debugging, but that wasn’t the right tool. I needed application-level traces that could correlate keystrokes with state machine transitions across the editor, shell, and agent subsystems.

I used the superpowers:brainstorming skill from Obra to design the system:

❯ /superpowers:brainstorm build a trace system for debugging keystroke issues

⏺ Using brainstorming to design a trace system for Hash.

  What debugging problems are you encountering most often?

  1. Agent communication issues
  2. Completion problems
  3. Command parsing issues
  4. General shell flow — input handling, REPL loop

❯ 4, 3 — keystrokes getting swallowed, multiple presses needed

⏺ When hunting these bugs, how do you prefer to debug?

  1. Live tracing to stderr
  2. File-based logging — write to trace file, review after
  3. Both

❯ 2

⏺ What granularity of events do you need?

  1. High-level only — key pressed, action taken
  2. Detailed — raw key codes, timing, buffer contents
  3. Verbose — goroutine info, channel ops, full state dumps

❯ 3 — I suspect race conditions

⏺ What format should trace entries use?

  1. Human-readable
  2. JSON Lines — machine-parseable, easy to filter with jq
  3. Both

❯ 2 — I want to feed this to an LLM / Agent

We landed on a design with:

• Subsystem filtering via HASH_TRACE=editor,agent,shell,parser
• Timestamps with millisecond deltas between events
• Goroutine IDs on every entry
• Structured JSON for machine parsing

{"ts":"...","delta_ms":2.5,"goroutine":42,"sub":"editor","event":"key_read","data":{"raw":[27,91,65],"parsed":"ArrowUp"}}

The implementation took one Claude session using the superpowers:executing-plans skill.
Eight tasks, incremental commits, all tests passing.

Hash now had observability.

What the Trace Revealed

With the trace system in place, I reproduced the bug and captured a trace. Here’s what the log revealed:

Line 91: ghost_state complete (goroutine 29)
Line 92: Enter key_read (goroutine 27) — NO DISPATCH FOLLOWS
Line 93: Enter key_read (goroutine 39) — 3.2 seconds later
Line 94: key_dispatch (goroutine 1) — finally dispatched

The first Enter was read by goroutine 27 but never dispatched. The second Enter was read by goroutine 39 and processed normally.

Same pattern for Ctrl+D:

Line 97: Ctrl+d read (goroutine 27) — NO DISPATCH
Line 98: Ctrl+d read (goroutine 39) — NO DISPATCH
Line 99: Ctrl+d read (goroutine 83)
Line 100: key_dispatch (goroutine 1) — finally works

Multiple goroutines competing to read from stdin. Keys consumed by orphaned readers that weren’t connected to the dispatch loop.

I fed the trace to Claude:

❯ analyze the traces in @hash-trace.jsonl; I am experiencing several issues:
  1. even after the inline ?? ghost text appeared, the OSC 9;4;3 progress
     showed, and "Enter" did nothing. I had to repeat it multiple times to
     be accepted.
  2. when done, to exit hash, I had to Ctrl-D **3 times**

  what do you learn from the trace, and how do we fix it?

It identified the root cause immediately:

Every call to editor.New().Run() spawns a goroutine that blocks indefinitely on stdin.Read(). When the editor exits, the goroutine keeps running because it’s blocked on the read. On the next editor invocation, another goroutine is spawned. Now multiple goroutines compete for stdin—keys go to orphaned readers and get lost.

The fix was straightforward: make the reader goroutine interruptible. I switched from a blocking Read() to a polling loop that uses Unix select() with a 50ms timeout. Between polls, the goroutine checks a done channel—when the editor closes, it signals the channel and the reader exits cleanly on the next iteration. The 50ms latency is imperceptible during typing but prevents the pile-up of orphaned readers that were competing for stdin.

One trace. One conversation. One fix. Ghost text now accepts on the first Enter.

Takeaways

The trace system cost 30 minutes to build. The debugging loop before it? Hours of wasted effort.

The system is designed for Agent-assisted debugging: JSON Lines for easy parsing, goroutine IDs for concurrency issues, millisecond deltas for race conditions. Every trace entry is structured data that Claude can reason about.

This is now part of Hash’s development workflow. When something breaks, I capture a trace and feed it to Claude. The signal-to-noise ratio is incomparably better than screenshots and prose.

Using Hash’s Trace System

The trace system will ship in the next Hash release:

# Trace everything
HASH_TRACE=all ./hash

# Trace specific subsystems
HASH_TRACE=editor,agent ./hash

# Custom output path
HASH_TRACE=all HASH_TRACE_PATH=/tmp/debug.jsonl ./hash

# View traces
cat hash-trace.jsonl | jq .

If you’re hacking on Hash and hit something weird, capture a trace and paste it into Claude. Structured data beats screenshots every time.

Hash is in its very early stages, and is being actively developed on GitHub.