Live Process Auditing

An invisible desktop agent that watches your screen and audio in real time, runs reasoning models in the background, and surfaces structured guidance - follow-up questions, debugging hints, algorithm walkthroughs - while you work or interview. Source: Kevin Liu (ideation), pickle-com/glass (base project)

The Concept

Fork Glass by Pickle (7.4K stars, GPL-3.0) and pivot from meeting assistant to live process auditing agent. Glass already solves the hard problems: invisible overlay (never appears in screen recordings, screenshots, or dock), screen capture, audio listening, and real-time context understanding. The delta is in the reasoning layer and prompt architecture.

Three capabilities the fork adds beyond Glass's meeting focus:

1. Proactive follow-up question generation. The agent watches the problem being presented and immediately surfaces clarifying questions the user should ask, before they start solving.
2. Background reasoning pipeline. An expensive reasoning model runs asynchronously while the user asks questions in the foreground. The user appears thoughtful and methodical; the agent is doing heavy computation behind the curtain.
3. Live debugging and course correction. As the user works, the agent monitors for mistakes, suboptimal approaches, and missed edge cases - surfacing corrections in real time rather than after the fact.

Architecture: Foreground/Background Split

The key UX insight is decoupling the interaction layer from the reasoning layer:

The foreground keeps the user engaged and appearing active. The background produces the deep analysis that arrives just as the user needs it. This mirrors how a skilled practitioner actually thinks: quick questions up front, deep reasoning running in parallel.

Visual Aids: Inline Diagram Generation

Text guidance is a poor channel for anything spatial - control flow, system topology, data structure mutation, decision trees. The agent emits Mermaid diagrams as a third overlay channel, popping up alongside the foreground guidance whenever the current step is more legible as a picture than as prose.

Channel rules:

• Diagrams render as Mermaid only - Glass already ships a Markdown surface, and Mermaid renders inline without extra binaries or design assets.
• One diagram per pass, max. If the diagram would just restate the prose, skip it.
• Diagrams refresh on context change (new whiteboard, new file, new question), not on every token. The reasoning model decides when a re-render is worth the visual interruption.
• Each diagram is captioned with the single sentence the user should say while pointing at it. The caption IS the script.

Diagram menu by interview type:

The "Imperfect First Attempt" Pattern

A deliberate design choice: the agent should introduce plausible mistakes in initial reasoning, then self-correct. This serves two purposes:

• Authenticity. In interview settings, perfect first-attempt solutions are suspicious. Real problem-solving involves false starts and course corrections.
• Teaching. Showing the mistake-then-fix path teaches the user to recognize and recover from common errors, which is more durable than memorizing correct answers.

Coding Interview Instantiation

The most concrete use case. A structured multi-pass prompt that runs when the agent detects a coding problem on screen:

Pass 1 - Orient. Return clarifying questions. Explain intuition behind the problem in paragraph form. Enumerate all possible algorithms with Big O time/space complexity, sorted worst-to-best. Select the best algorithm and walk through the thought process naturally - realizations, data structure choices, loop/conditional logic - using test cases as illustrations. Respect edge/base cases and assumptions.

• Diagram: flowchart TD showing input → algorithm choice → output, with branch labels for each edge case the user should call out aloud.

Pass 2 - Analyze. Time and space complexity analysis. Show how all edge cases are handled. Technical but clear.

• Diagram: if the structure is recursive or memoized, render the call tree as a mindmap; if it's iterative over a data structure, render the structure mutating across two or three snapshots in a flowchart LR.

Pass 3 - Implement. JavaScript solution in the order you'd naturally write it. Descriptive comments throughout. Dry-run/stepthrough with examples to verify correctness.

• Diagram: sequenceDiagram of the dry-run - variable values at each step, in the order they'd appear on the whiteboard. The user narrates by reading down the diagram.

Pass 4 - Expand. Additional algorithms beyond the chosen one, each with Big O and a one-sentence intuition explanation.

• Diagram: flowchart LR with one subgraph per alternative, each labeled with its Big O - so the user can point and say "I picked A because B is O(n²) here."

Pass 5 - Anticipate. Potential follow-up questions about the solution with prepared answers.

• Diagram: none unless a follow-up specifically asks for a structure (e.g. "how would you parallelize this?" → sequenceDiagram of the parallel version).

System Design Interview Instantiation

Triggered when the agent detects whiteboard mode, a Miro/Excalidraw window, or audio cues like "design X" or "scale Y."

Pass 1 - Scope. Functional requirements, non-functional requirements (QPS, latency, durability, consistency), back-of-envelope numbers. Clarifying questions ranked by impact on the design.

• Diagram: none yet - keep the prose focused on questions to ask before drawing.

Pass 2 - Topology. Component diagram with the minimum viable set of services. Trust boundaries, sync vs async edges, payload shapes.

• Diagram: flowchart LR grouped into subgraphs for client, edge, services, datastores. Caption is the one-line elevator pitch the user delivers while drawing.

Pass 3 - Lifecycle. Walk one read path and one write path end to end. Failure modes, retry semantics, idempotency.

• Diagram: sequenceDiagram per path. Annotate with latency budgets and the failure recovery arrow.

Pass 4 - Data. Entities, cardinality, partitioning key, hot-key risks, index strategy.

• Diagram: erDiagram for the model; flowchart LR if the question is really about sharding or replication.

Pass 5 - Scale & tradeoffs. Bottlenecks at 10x and 100x. Caching, queueing, CDC, read replicas. Two named alternatives with the discriminating constraint surfaced.

• Diagram: flowchart LR with side-by-side subgraphs for Option A vs Option B; highlight the single arrow that differs.

Behavioral Interview Instantiation

Triggered on prompts like "tell me about a time," "walk me through," or "describe a conflict."

Pass 1 - Pick the story. Match the question to the strongest story in the user's profile (pulled from [[wiki/people/Kevin]] and prior transcripts). Surface the title and one-line hook.

• Diagram: none - the user just needs the title.

Pass 2 - STAR scaffold. Situation, Task, Action, Result with concrete numbers and dates the user has actually used before.

• Diagram: mindmap rooted at the story title with four branches - S, T, A, R - each carrying two or three bullet leaves. The user reads the leaves left-to-right.

Pass 3 - Decision narrative. Reframe the Action as a decision under ambiguity: what options were considered, what criterion ruled, why the chosen path won.

• Diagram: flowchart TD with the decision node at top, options as children, the criterion labeling the winning edge.

Pass 4 - Anticipate probes. Likely follow-ups ("what would you do differently," "how did the team feel," "what was the metric"), with one-sentence answers.

• Diagram: flowchart LR mapping probe → prepared answer, only if there are 3+ likely probes.

Services-as-Software Analysis

The unique wedge: Glass's invisibility. Every other tool is "use ChatGPT in a browser tab." Glass doesn't appear in screen recordings or screenshots, enabling use during actual high-stakes moments - not just practice. This is the moat no browser-based competitor can replicate.

Vulnerability: the copilot framing means better models could compress the product into a feature. The autopilot pivot - selling guaranteed interview outcomes - would compound instead. That's a different business (accountability, money-back guarantees, outcome tracking) but a stronger one.

Beyond Coding Interviews

The "live process auditing" pattern generalizes:

• Live debugging sessions. Agent watches the IDE and terminal, surfaces relevant documentation, suggests fixes as errors appear. Diagram: sequenceDiagram of the failing call path with the suspect frame highlighted.
• Sales calls. Agent listens to the prospect, surfaces objection-handling scripts, competitor comparisons, pricing guidance. Diagram: flowchart LR competitor matrix or pricing tier comparison.
• Medical consultations. Agent monitors the conversation, surfaces relevant clinical guidelines, drug interactions, differential diagnoses. Diagram: flowchart TD differential decision tree.
• Legal depositions. Agent tracks testimony, flags inconsistencies, surfaces relevant case law. Diagram: flowchart LR timeline of asserted facts with contradictions marked.

Each vertical is a distinct autopilot opportunity where the services budget dwarfs the software budget.

Open Questions

• Latency. Can the background reasoning model produce useful output fast enough? The foreground model buys time, but if the background takes 60+ seconds, the user may have moved past the point where the guidance is useful.
• Context window. Glass captures screen + audio continuously. How much context does the reasoning model need to produce good guidance? Full session history vs. sliding window?
• Detection risk. If this becomes popular for interviews, companies may develop countermeasures. The invisibility moat is technical, not permanent.
• Ethics. Using an invisible AI assistant during evaluated interactions raises questions about fairness and disclosure. The tool is most defensible when framed as a learning/practice aid rather than a cheating device.
• Diagram overlay UX. Mermaid renders well in Markdown surfaces but Glass's overlay is small. Open questions: should diagrams pop in a separate hideable pane, scale to fit, or stream as ASCII fallback when the Mermaid block exceeds a size threshold? Leaning toward: pane that auto-collapses after 8s unless the user pins it.

Concept Position

---

Timeline

• 2026-07-01 | Concepts category refresh added this page to the Brain, memory, and retrieval family, linked it to Concept System Map, and kept it standalone because it owns this reusable mental model: An invisible desktop agent that watches your screen and audio in real time, runs reasoning models in the background, and surfaces structure... Source: User request, 2026-07-01