Explore intelligence

Method · Money

Revealing the job: four questions and a job statement

Kent walks through a method for revealing the job to be done: ask who is this for (attendees and whoever mans the door), when does this problem happen (when people are getting into a workshop), what progress do they want (just get in to the workshop), and what do they do currently (a QR code scan, a blocking request). Then write the job statement in the form 'When [situation], help me make progress so I can [desired outcome],' often adding 'without [something]' — e.g., 'When there's a workshop going on at 10110, help me get into the workshop quickly so that I can learn about product engineering without missing the first 20 minutes.'

AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer

Method described by Kent C. Dodds

Method preserved for inspection; not a verified protocol.

Method · Money

Turn feature requests into job statements via why-questions

When a feature request comes in, ask 'what are you trying to solve?' as your first question back to the requester. Ask 'why' repeatedly to dig past the requested feature and find the underlying problem. Ask 'who is this for?', 'when does this problem happen?', 'what progress do they want?', and 'what do they do currently?' to reveal what the job is. When writing a job statement, add 'without [undesired outcome]' to capture what the user wants to avoid. Example job statement: 'When there's a workshop going on at 10110, help me get into the workshop quickly so that I can learn about product engineering without missing the first 20 minutes.' The job statement has nothing to do with the requested feature (facial recognition). Getting down to the why-why-why and turning that into a jobstatement

AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer

Method described by Kent C. Dodds

Method preserved for inspection; not a verified protocol.

Method · Money

Open coding then axial coding of agent traces

First pass: read each trace and write down what you see in natural language ('vague recommendation', 'made up a number', 'wrong ticker') without pre-categorizing. Second pass: group similar open codes into categories (axial coding). Each root cause points to a different fix — if you don't know the cause, you can't pick the right remedy.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Run faithfulness evals by attaching research context from traces

The speaker described running faithfulness evals in code (rather than the UI) when using the SDK: extract the research context — the output of the first research step — from the traces, attach it to all spans as context for the judge, then run the faithfulness evaluator, which asks whether, based on that same information, the agent did a good job writing the report. This puts the judge on a level playing field with the agent, unlike correctness evals for live-search agents.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Run online evals on sampled production traffic

You're not done when the dev experiment passes — you're done when the agent keeps performing on unseen production traffic. Run your evaluators (ticker check, faithfulness, actionability) automatically on incoming production traces, at trace level so the evaluator sees the full tree, and sample 10% or 1% of traffic for directional signal without an expensive LLM judge on every instance.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Split labeled data into a dev set and a held-out test set to verify the judge generalizes

Laurie Voss recommends splitting labeled data into a development set (about 75%) used to iterate on the eval until the judge agrees with you, and a held-out test set (about 25%) used to verify the judge gets expected scores. Holding out a test set shows the judge is generalizing to new examples instead of just fitting the specific data set, and he warns not to overfit the evaluator to the golden data set — the same principle as train/test splits in machine learning.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Four parts of a custom rubric: role, criteria, data labeling, output choices

The speaker recommends four parts for a custom rubric: define the judge's role, spell out explicit pass and fail criteria, label the data with XML tags, and define the output choices outside the prompt itself.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Validate the judge against human ground truth and tighten the rubric on disagreements

Measure the judge's performance by comparing its predictions against ground truth (human judgment). Give human labelers real concrete criteria and stick to them. When the judge disagrees, ask 'Is my rubric wrong or were my human labels wrong?' — read the explanations on disagreements, find the ambiguity, and tighten the criteria (e.g., 'includes forward-looking analysis' → '...with specific recommendations or guidance').

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Meta-evaluate the judge: read disagreement explanations, tighten the rubric, and rerun

Laurie Voss describes a rubric-iteration loop: when the judge disagrees with your human labels, read its explanation and decide whether the judge was right or you were right — the most valuable output of meta-evaluation. Judge-human disagreement usually reveals an ambiguity in the rules you set down. To fix a rubric, read the explanations on the disagreements, find the ambiguity, and tighten the criteria (e.g., change 'includes forward-looking analysis' to 'includes forward-looking analysis with specific recommendations or guidance', which rules out reports that talk about the future but never tell the judge what to do about it). Then rerun the judge on the same examples and see if the disagreement goes away.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Error analysis via open coding and axial coding of traces

The speaker laid out a method: read traces end-to-end (focusing on failures), use open coding — writing down what you see in natural language without trying to be neat — then axial coding, grouping similar codes into categories; ask why each failure failed, since the wrong response is a symptom, not the root cause; also check whether the agent did something it was never told it could do; then summarize categorized failures into a table of root-cause frequency.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Hand failing traces to a coding agent to improve prompts

One of the highest-leverage uses of evals: export failing traces from AX with their explanations, feed them plus your requirements to a coding agent (Claude Code, Cursor), and let it rewrite prompts. Pass requirements too — otherwise the agent optimizes only for 'make evals pass' and cheats by including test data in its eval. Tell it to find themes rather than chase individual failures, and never just say 'get better' — every change must trace to a specific failure explanation.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

The full production loop: traces → evals → monitors → regression datasets → fixes

The complete cycle: the application produces traces, online evals grade them and produce labels and explanations, labels feed monitors, monitors alert the team, the team investigates failing traces and saves them as a regression dataset, the agent is improved, an experiment verifies the fix, and it ships — with returning traffic graded by the same online evals. Every caught failure becomes a new test case and regression test.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

The universal verifier pipeline

To verify an agent trajectory: generate a good rubric with N criteria of what success looks like; rank the trajectory's screenshots by relevance for each criterion and use the top-K evidence to determine whether each criterion was met; check for contradictions between what the agent said it did and the actual environment state; output a scored rubric with process score (partial credit in some cases) plus an outcome Boolean for whether the agent accomplished the task as a reasonable user would expect.

AI Engineer · Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

Method described by Miguel Pagano

Method preserved for inspection; not a verified protocol.

Method · Money

The four-step account management loop

Run the loop: (1) understand the executive outcome (the target), (2) deliver the value proof by running the project with the CSM, (3) communicate the value back to the customer and get acknowledgment, (4) get the next outcome and repeat — as many times and as fast as possible. The more and faster the loop runs, the more opportunities to sell.

30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account

Method described by Mark (guest)

Method preserved for inspection; not a verified protocol.

Method · Money

Implement human-in-the-loop approval with Temporal using workflow, wait condition, and signal

Warrick's core method for human-in-the-loop agent systems: structure the agent loop as a Temporal workflow, use a wait condition to pause the workflow while tracking for a signal, and use the signal to capture the human's response and resume work. She states this is the pattern to use whether the human or the computer initiates the pause, and whether the service is offline or back online.

AI Engineer · The Human Is an Async API — Melanie Warrick, Temporal

Method described by Melanie Warrick

Method preserved for inspection; not a verified protocol.

Method · Money

VS Code's automated error-stack pipeline: collect, filter, fingerprint, file, auto-fix

Harald Kirschner describes the pipeline: every time an exception occurs in an app, an error stack is reported. VS Code builds error-stack tooling on its own data instead of using external tools. They collect raw telemetry and filter it down to error stacks that have the full stack (around 51 billion telemetry events per day before filtering), then group and bucket filtered error stacks as a fingerprinting exercise. After filtering and bucketing, 10 issues are filed assigned to specific area owners, and a pull request is auto-created to try to fix the problem. All error stacks are displayed in an error page showing hit counts and number of users affected so the team can quickly react. The agent investigates filed issues to find out why the error happened and who was responsible. In one case

AI Engineer · How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner

Method described by Harald Kirschner

Method preserved for inspection; not a verified protocol.

Method · Money

Leveraging New Executive Networks via Sales Navigator

Set up Sales Navigator alerts to identify when VP+ or executive-level employees join your company, then immediately connect with them to leverage their networks for warm introductions.

30 Minutes to President's Club | No-Nonsense Sales · Introducing: The Warm Pipeline Machine (Social Selling Course)

Method described by Sam McKenna

Method preserved for inspection; not a verified protocol.

medium

Method · Money

AI Measurement Framework: Utilization, Impact, Cost

Measure AI adoption across three dimensions: Utilization (DAU/WAU), Impact (quality/speed/value), and Cost (token spend). Use API telemetry to create cohorts and compare against foundational metrics.

AI Engineer · The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX

Method described by Justin Reock

Method preserved for inspection; not a verified protocol.

Method · Money

The 'Translation' Method: Solving Problems at Different Abstraction Levels

Angel Steger describes the architecture technique of 'translation': starting with a high-level abstraction (e.g., site map), solving one problem at that level, then switching materials or views to solve the next layer of problems, avoiding the trap of trying to solve everything at once.

Design Better · Angel Steger: Wealthfront's VP of Design on what architecture teaches you about software

Method described by Angel Steger

Method preserved for inspection; not a verified protocol.

Recommendation · Money

Imagine your product costs $1,000–$10,000/month and price as PMF signal

Anish Acharya suggests imagining your product costs $1,000 or $10,000 a month and asking what it would have to do to justify that price. He frames price as a measure of product-market fit and argues that expensive consumer software is a new and important opportunity, urging builders to think about extraordinarily expensive consumer products where every part of consumer discretionary spend is up for grabs.

Lenny's Podcast · Why companies are becoming a series of loops | Anish Acharya (a16z)

Recommendation from Anish Acharya