Explore intelligence

Report · Money

Once in production, eval data goes stale: skim real traffic, have a team score it manually, and check judge agreement to stay relevant

Once in production, there is a high chance your eval data will go stale, so you need to actively stay relevant. To stay relevant, skim real traffic: collect new user inputs and your agent's outputs, have an internal team score them manually, and check the judge's agreement against those scores. Your eval data set should be a living data set — continuously updated with fresh scored traffic. A living data set with an up-to-date judge is very difficult to break and provides reliability ahead of time. After skimming public traffic, you will have even more data to refine your evals. In production, skim the top of the data, get a little more data, and refine and refine.

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Claim from Tejas Kumar

Report · Money

Evals are afforded strategically: start with deterministic tests, then regex, then traces, then a judge with the cheapest model

To keep evals cheap, start with actual deterministic tests (unit tests). If deterministic tests can't be done, the next level up is regular expressions or pattern matching. If pattern matching fails, the next thing to do is examine your traces. Ideally, everyone is using some type of observability solution and has traces of each message envelope, inputs, outputs, tool calls, etc., which can be validated per object. Eventually you will come to a place where you need to use a judge, and then you start with the cheapest model. Evals are afforded strategically, because it will not be cheap if you use the most expensive model, and you want to build up to it. Regular expressions can be used for testing AI outputs but have their limits.

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Claim from Tejas Kumar

Report · Money

Judge training is iterative: investigate dissonance, find the knowledge gap, fill it with policy; agreement grows toward the threshold

When humans and AI don't agree, it's usually because of a knowledge gap — and the way to reach agreement is to introduce context to bridge that gap. To improve judge agreement, examine the cases where you disagreed and fill the knowledge gap with policy. In production you would supply policy to the judge via a RAG pipeline that retrieves enterprise context; in a demo you can just hard-code the policy. Write the policy explicitly (e.g., a 14-day return window; defective items can be returned) and interpolate it into the judge prompt. Building your judge is arduous iterative work — investigating failures and filling knowledge gaps is literally the job. After adding policy, check whether agreement is growing even if it still fails — improvement indicates you're on the right track. When the

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Claim from Tejas Kumar

Principle · Money

Engage the account immediately at AE-to-AM handoff and verify the executive outcomes

The minute the account is yours, engage right away so the value loop starts turning — the AM not being involved in implementation is a mistake. At handoff, verify (don't assume) the executive outcomes the AE used to win the deal, grab them, keep the CSM on target to them, and deliver those as the first set of value; that proof is what lets the next outcome happen.

30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account

Claim from Mark (guest)

Method · Money

Check whether your site is blocking AI crawlers via robots.txt

A short check the creator walks through: visit yourdomain.com/robots.txt and look for disallow rules next to GPTBot, OAI-SearchBot, ClaudeBot, or Google-Extended. If you see disallow rules next to any of these AI crawler names, you are blocking that AI crawler from your site.

Ahrefs · Learn 80% of AEO in 19 Minutes

Method described by Ahrefs

Method preserved for inspection; not a verified protocol.

Method · Money

How to identify your buyers and build content around them

Hormozi lays out a method: if you don't know who your buyers are, look at your customer base. Look at the people who spent the most money — the top 20% — and the common factors they have. Look at what messages and problems they had, then create messaging and video topics based on the problems of the people with the most money in your audience, and make content about that. Put UTMs on the links below the descriptions in your videos to track revenue attribution, and make CTAs inside the videos to take the next step, such as a lead magnet. That is how you can see on the back end how content translated into revenue for the business.

Alex Hormozi · How to become rich with social media (my exact playbook)

Method described by Alex Hormozi

Method preserved for inspection; not a verified protocol.

resolved

Method · Money

Agentic collections playbook: invoice fast, CC stakeholders, follow up weekly, escalate to humans when late

Automate accounts receivable collections with an agent — it is a universal pain point that most companies still handle manually. The playbook: send the invoice to the correct finance point of contact within 10–30 minutes of the deal closing; check whether a finance contact is listed on the contract and email the line-of-business person as primary and CC the finance contact (or vice versa); if no finance contact is listed, use enrichment to look up and guess the right finance recipient (most companies have standard AP or finance inboxes); CC multiple stakeholders on every collections invoice (contract signer, deal POC, finance contact, line-of-business owner) so someone can always escalate; have the agent monitor the collections inbox daily; send a first payment-reminder touchpoint one week

The Official SaaStr Podcast: SaaS | Founders | Investors · SaaStr 880: Agent Pricing is Chaos. Here's What We're Seeing From the Buyer Side. (The Agents #015)

Method described by The Official SaaStr Podcast: SaaS | Founders | Investors

Method preserved for inspection; not a verified protocol.

Method · Money

Coinbase's 'brain': markdown knowledge files agents ingest, with reviewer corrections fed back in

Coinbase's approach: each team and repository has a 'brain' implemented as markdown files that team members can review at any time. Every time an agent comes in to make a change to a service, it ingests the brain for that team and repo. When a human reviewer fixes something an agent missed, that context must go back into the brain so it is fixed in all future cases — the simple manual-edit-and-ship response is the wrong move. This causes the accept rate for one-shotted PRs to tick up over time, creating a recursive self-improvement system.

No Priors: Artificial Intelligence | Technology | Startups · Coinbase’s Everything Exchange: Agentic Finance, Stablecoins, and Tokenization with CEO Brian Armstrong

Method described by Brian Armstrong

Method preserved for inspection; not a verified protocol.

Method · Money

Building a task sandbox: choose harness, MCPs, network rules, and resources

Jim Clark walks through deciding what capabilities a task sandbox needs: which harness to use (OpenCode, Claude, Gemini, Codex), what MCPs are allowed to be used in the sandbox, what networking rules to apply (e.g., whether it needs access to api.github.com, whether it needs to surf the web in random ways or just needs nothing), and what resources and work trees it needs. Model capabilities and build the right sandbox for the task, then put the task in. Treat orchestrators as having a job of asking what the task is and what sandbox to put it in.

AI Engineer · How Many Credentials Should Your AI Agent Have? Zero. — Jim Clark, Docker

Method described by Jim Clark

Method preserved for inspection; not a verified protocol.

Method · Money

Iterate the rubric when the judge disagrees

When the judge disagrees with your labels, read its explanation and decide whether the judge was right or you were right — ask 'Is my rubric wrong or were my human labels wrong?' Tighten rubric criteria (e.g., instead of 'includes forward-looking analysis', write 'includes forward-looking analysis with specific recommendations or guidance'), then rerun the judge on the same examples and see if the disagreement goes away — this is rubric iteration.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Derive eval criteria from error analysis of traces

Don't invent criteria from whole cloth: read your traces, observe the specific ways reports go wrong, and write criteria matching those observed failures.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Measure the judge against human ground truth

Measure the judge's performance by comparing its predictions against ground truth — a human's judgment. Use AX annotations to attach human labels to spans right in the UI (set spans actionable or not actionable via the 'human actionable' annotation config), then compare what you said was actionable to what the judge says. Labeling is extremely time consuming, so do it ahead of time; it's work a non-technical staff member can do using their domain expertise.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Build failing-trace datasets and regression test sets

Filter your traces down to just the failing ones and turn them into a dataset so you can run your changed prompt against just them — this is faster and cheaper. Also save a regression test set from the cases where your eval is NOT failing, and check it less frequently to make sure a rubric change hasn't broken previously-passing cases.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Writing a custom eval rubric: role, criteria, and specificity

Define the judge's role first, give it context on what it's looking at and what you expect, list exactly what makes output good or bad with specific observable criteria, and avoid vague aspirations like 'a good response'. The 'you are an expert X evaluator' framing itself makes only a very small difference.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Include labeled good and bad examples in the rubric

Include examples of good and bad output — it most improves quality. Label examples with XML tags, otherwise the LLM treats the example as the thing to write and spits it back. Examples give the judge a concrete reference for your labels, dramatically improving consistency especially on edge cases; if output is too non-deterministic, an example grounds it in what you expect. A good actionable example report contains percentage revenue growth, a PE ratio, a con backed by a number, and a specific recommendation; a non-actionable one is correct but has no specific data or risks and just says investors should consider various factors.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Build golden datasets with concrete criteria and both-direction tests

Build golden datasets to measure whether your evaluator is actually doing its job. Give yourself real concrete criteria and stick to them when labeling — don't just say this was good, this was bad; be specific the same way you told the LLM to be specific. Test in both directions: have cases where a behavior should occur and cases where it shouldn't — if you only test whether it searches the web, you'll end up with an agent that always searches the web even when it doesn't need to.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Split labeled data into dev and test sets for rubric iteration

Split your labeled data into a development set (75%) and a test set (25%). Use the dev set to iterate on your eval — tweaking criteria and adjusting examples until the judge agrees with you. Hold out the remaining 25% to run the judge and see that it gets the scores you expect — that's how you know the judge is generalizing to new examples.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Automate prompt improvement with a coding agent driven by failure explanations

Laurie Voss recommends using a coding agent (e.g., Claude Code) to pull down your failing traces, modify your code, and improve your application instead of hand-writing prompt fixes yourself. Claude pulls down all of the failing traces, reads all of the explanations, and turns the explanations into actions it takes on your code base. Nothing in the automated improvement is guessed — every change the coding agent proposes traces back to a specific failure explanation. For example, if the judge said the output lacks specific recommendations, the coding agent will rewrite the prompt so that it does; if the judge said the output presents risks without supporting evidence, it will rewrite the prompt to fix that too. This is data-driven prompt engineering, and it can now be automated.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

POC-to-production pipeline for AI-ready document data

A repeatable pipeline: start a POC with ~40 pre-curated files to validate parsing, chunking, OCR, embedding, vector DB, and knowledge graph; then scale to 80,000+ files by automating discovery, tagging, sensitivity scanning, and quality checks.

AI Engineer · From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

Method described by Jeff Koss

Method preserved for inspection; not a verified protocol.

described

Method · Money

Universal verifier pipeline: rubric, evidence ranking, scored output

The verifier first generates a rubric with N criteria of what success looks like, ranks the most relevant screenshots in the trajectory for each criterion, uses the top-K evidence group to determine whether each criterion was met, and outputs a scored rubric with partial credit (process score) plus an outcome Boolean with an explanation.

AI Engineer · Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

Method described by Miguel Pagano

Method preserved for inspection; not a verified protocol.