Cue

Problem Statement

Learning something new on your computer is weirdly indirect. You hit a wall in an app, screenshot it, switch to a chat window, describe what you're stuck on, paste the image, read the answer, then try to map "click the button in the top right" back onto your actual screen. And the assistants that do watch and listen ship your microphone audio and screenshots off to a stack of vendors — AssemblyAI for transcription, ElevenLabs for speech — each metering you by the character. For a companion you'd want open all day while you learn, that's a lot of friction and a lot of trust to hand out.

Hypothesis

What if a buddy lived on your screen, saw exactly what you see, and pointed at the thing instead of describing it? Showing beats telling when you're learning. And what if the whole voice loop ran on-device, so the only thing that ever left your machine was the question itself? Apple Silicon is fast enough to run speech-to-text and text-to-speech locally, which means a screen-aware learning companion can be genuinely helpful and genuinely private at the same time — one call to Claude, nothing else leaves.

Solution

Cue is a macOS menu bar learning buddy (no dock icon). You hold a key, talk to it like a person, and it sees your screen and talks back. It's a fork of Clicky by Farza, rebuilt around a learning-first persona and fully on-device voice:

• It points, it doesn't just tell. A blue cursor flies to the exact menu, button, or word it's explaining — Claude embeds [POINT:x,y:label] tags in its replies, and the cursor routes to the right element even across multiple monitors. "Open the color inspector" becomes the cursor actually flying to it
• It teaches, conversationally. Replies are written for the ear, not the eye — short and direct by default, but it goes deep the moment you ask it to. It describes what code does instead of reading it out verbatim
• It never dead-ends you. Instead of closing with "want me to explain more?", it plants a seed — a related concept, a more ambitious next step, something worth coming back for
• It remembers the whole session. One ongoing conversation, so "wait, back up" or "okay, now the next part" just works
• It works on anything on screen. Color grading in Final Cut, committing in Xcode, a dense doc, your writing — if it's on screen, you can point at it with your voice and ask
• It pulls in fresh info when it matters. For news, prices, or recent events, Claude's server-side web_search runs (capped at 3 per response, source cited naturally). For anything on your screen or stable facts, it doesn't bother
• The voice loop is fully on-device. WhisperKit transcribes on the Apple Neural Engine (~150ms for a 10-second utterance); Kokoro-82M speaks the reply via ONNX Runtime. No AssemblyAI, no ElevenLabs, no quotas — only the question reaches Claude

Technical Architecture

A menu bar app with two NSPanel windows: a control panel dropdown and a full-screen transparent cursor overlay. Push-to-talk (Control + Option) buffers audio locally, runs WhisperKit transcription on key-up, then streams the transcript plus a screenshot to Claude over SSE. The reply is spoken locally through Kokoro-82M once its model finishes downloading. Two API key paths are supported: direct mode (key read from the environment at runtime, never bundled into the binary) for personal use, and a Cloudflare Worker proxy for shipping signed builds without burning the key into the app.

Key Product Decisions

• Show, don't tell. The flying cursor is the whole thesis. Telling someone "it's in the top right" still leaves them hunting; flying a cursor to the exact element removes the translation step entirely. The system prompt biases toward pointing whenever it would genuinely help
• Teach, don't dead-end. The persona is tuned to end on a "seed" — a deeper concept or next-level technique — rather than a yes/no question. Dead-end questions force a nod; a seed gives you a reason to keep learning
• On-device by default, not as a setting. Privacy isn't a toggle buried in preferences — the voice stack just runs locally. The only network call is the one that's genuinely required (the model), and that's the design, not an option
• Graceful first launch. Kokoro is ~89MB and downloads on first run, so the app falls back to AVSpeechSynthesizer until it's ready. Cue is usable from the very first push-to-talk instead of making you wait on a download
• Two key paths for two audiences. Direct mode for "just me on my Mac," Worker proxy for "ship a build to someone else." Same app, the path is chosen automatically based on whether the env var is set
• Don't rename what users never see. The Clicky→Cue rebrand renamed every user-facing string and internal identifier, but deliberately kept the Xcode scheme name to avoid a deep .pbxproj rewrite that risked breaking the build. Old UserDefaults keys auto-migrate so existing users keep their settings

Impact & Metrics

Lessons Learned

• Showing is a different product than telling. The moment the assistant can point at the actual element, "how do I do this?" stops being a translation exercise. The cursor turns explanation into guidance — that's the feature people actually feel
• The Neural Engine changes what's possible locally. Whisper transcription in ~150ms on-device makes push-to-talk feel instant. The "send audio to a server" architecture exists mostly because people assume local inference is too slow — on Apple Silicon, it isn't
• Fork, don't rebuild. Starting from Clicky meant the cursor overlay, screenshot capture, and panel UI already worked. The interesting work was the learning-first persona and swapping the voice layer for an on-device one, not reinventing the menu bar plumbing
• Renaming a fork is deceptively deep. A real rebrand touches user-facing strings, internal Swift identifiers, the build product, and migration logic for existing users' preferences — not just the display name

Links

GitHub Repository ↗  ·  Clicky (upstream) ↗  ·  Kokoro-82M ↗