Explore intelligence

Method · Money

Coinbase's 'brain': markdown knowledge files agents ingest, with reviewer corrections fed back in

Coinbase's approach: each team and repository has a 'brain' implemented as markdown files that team members can review at any time. Every time an agent comes in to make a change to a service, it ingests the brain for that team and repo. When a human reviewer fixes something an agent missed, that context must go back into the brain so it is fixed in all future cases — the simple manual-edit-and-ship response is the wrong move. This causes the accept rate for one-shotted PRs to tick up over time, creating a recursive self-improvement system.

No Priors: Artificial Intelligence | Technology | Startups · Coinbase’s Everything Exchange: Agentic Finance, Stablecoins, and Tokenization with CEO Brian Armstrong

Method described by Brian Armstrong

Method preserved for inspection; not a verified protocol.

Method · Money

Building a task sandbox: choose harness, MCPs, network rules, and resources

Jim Clark walks through deciding what capabilities a task sandbox needs: which harness to use (OpenCode, Claude, Gemini, Codex), what MCPs are allowed to be used in the sandbox, what networking rules to apply (e.g., whether it needs access to api.github.com, whether it needs to surf the web in random ways or just needs nothing), and what resources and work trees it needs. Model capabilities and build the right sandbox for the task, then put the task in. Treat orchestrators as having a job of asking what the task is and what sandbox to put it in.

AI Engineer · How Many Credentials Should Your AI Agent Have? Zero. — Jim Clark, Docker

Method described by Jim Clark

Method preserved for inspection; not a verified protocol.

Method · Money

Iterate the rubric when the judge disagrees

When the judge disagrees with your labels, read its explanation and decide whether the judge was right or you were right — ask 'Is my rubric wrong or were my human labels wrong?' Tighten rubric criteria (e.g., instead of 'includes forward-looking analysis', write 'includes forward-looking analysis with specific recommendations or guidance'), then rerun the judge on the same examples and see if the disagreement goes away — this is rubric iteration.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Derive eval criteria from error analysis of traces

Don't invent criteria from whole cloth: read your traces, observe the specific ways reports go wrong, and write criteria matching those observed failures.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Measure the judge against human ground truth

Measure the judge's performance by comparing its predictions against ground truth — a human's judgment. Use AX annotations to attach human labels to spans right in the UI (set spans actionable or not actionable via the 'human actionable' annotation config), then compare what you said was actionable to what the judge says. Labeling is extremely time consuming, so do it ahead of time; it's work a non-technical staff member can do using their domain expertise.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Build failing-trace datasets and regression test sets

Filter your traces down to just the failing ones and turn them into a dataset so you can run your changed prompt against just them — this is faster and cheaper. Also save a regression test set from the cases where your eval is NOT failing, and check it less frequently to make sure a rubric change hasn't broken previously-passing cases.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Writing a custom eval rubric: role, criteria, and specificity

Define the judge's role first, give it context on what it's looking at and what you expect, list exactly what makes output good or bad with specific observable criteria, and avoid vague aspirations like 'a good response'. The 'you are an expert X evaluator' framing itself makes only a very small difference.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Include labeled good and bad examples in the rubric

Include examples of good and bad output — it most improves quality. Label examples with XML tags, otherwise the LLM treats the example as the thing to write and spits it back. Examples give the judge a concrete reference for your labels, dramatically improving consistency especially on edge cases; if output is too non-deterministic, an example grounds it in what you expect. A good actionable example report contains percentage revenue growth, a PE ratio, a con backed by a number, and a specific recommendation; a non-actionable one is correct but has no specific data or risks and just says investors should consider various factors.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Build golden datasets with concrete criteria and both-direction tests

Build golden datasets to measure whether your evaluator is actually doing its job. Give yourself real concrete criteria and stick to them when labeling — don't just say this was good, this was bad; be specific the same way you told the LLM to be specific. Test in both directions: have cases where a behavior should occur and cases where it shouldn't — if you only test whether it searches the web, you'll end up with an agent that always searches the web even when it doesn't need to.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Split labeled data into dev and test sets for rubric iteration

Split your labeled data into a development set (75%) and a test set (25%). Use the dev set to iterate on your eval — tweaking criteria and adjusting examples until the judge agrees with you. Hold out the remaining 25% to run the judge and see that it gets the scores you expect — that's how you know the judge is generalizing to new examples.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

Automate prompt improvement with a coding agent driven by failure explanations

Laurie Voss recommends using a coding agent (e.g., Claude Code) to pull down your failing traces, modify your code, and improve your application instead of hand-writing prompt fixes yourself. Claude pulls down all of the failing traces, reads all of the explanations, and turns the explanations into actions it takes on your code base. Nothing in the automated improvement is guessed — every change the coding agent proposes traces back to a specific failure explanation. For example, if the judge said the output lacks specific recommendations, the coding agent will rewrite the prompt so that it does; if the judge said the output presents risks without supporting evidence, it will rewrite the prompt to fix that too. This is data-driven prompt engineering, and it can now be automated.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Method described by Laurie Voss

Method preserved for inspection; not a verified protocol.

Method · Money

POC-to-production pipeline for AI-ready document data

A repeatable pipeline: start a POC with ~40 pre-curated files to validate parsing, chunking, OCR, embedding, vector DB, and knowledge graph; then scale to 80,000+ files by automating discovery, tagging, sensitivity scanning, and quality checks.

AI Engineer · From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

Method described by Jeff Koss

Method preserved for inspection; not a verified protocol.

described

Method · Money

Universal verifier pipeline: rubric, evidence ranking, scored output

The verifier first generates a rubric with N criteria of what success looks like, ranks the most relevant screenshots in the trajectory for each criterion, uses the top-K evidence group to determine whether each criterion was met, and outputs a scored rubric with partial credit (process score) plus an outcome Boolean with an explanation.

AI Engineer · Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

Method described by Miguel Pagano

Method preserved for inspection; not a verified protocol.

Method

Recognizing and managing suspected differentiation syndrome on menin inhibitors

Maintain a high index of suspicion for differentiation syndrome in any patient starting a menin inhibitor: for revumenib, onset is roughly within one to two weeks; for ziftomenib it can occur later, up to one to two months, because of its longer half-life. Patients should come in for evaluation immediately if they develop fever, joint aches, bone aches, chest pain, trouble breathing, or rash. Workup includes checking blood counts for rising white count and differentiating cells, evaluating kidney function and joint/bone aches, obtaining a chest x-ray for pleural or pericardial effusions, and performing an echocardiogram if a pericardial effusion is found. At any suspicion, give high-dose dexamethasone 10 mg twice daily for three days regardless of symptom improvement — stopping steroids

Research To Practice · Acute Myeloid Leukemia — An Interview with Dr Ghayas Issa on the Role of Menin Inhibitors

Method described by Dr Ghayas Issa

Method preserved for inspection; not a verified protocol.

Consequential creator advice requires independent verification before Today can promote it.

Method · Money

The production-log-to-eval flywheel

Ship an AI application (e.g., a docs page with an AI chatbot), let customers use it, and real production logs start coming in. Sample a subset of production logs — maybe 10 to 20 rows — to create a dataset. From that sampled dataset, write your task, write your scoring system, and run evals; tweak your dataset, prompt, or score and run multiple different evals. After running a couple of evals and seeing which does best, you learn something about your system, make the code change, deploy it back into your application, and the whole flow repeats because those changes show up in your production logs.

AI Engineer · Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust

Method described by Jess (Braintrust)

Method preserved for inspection; not a verified protocol.

Method · Money

Cold call structure: 'heard the name tossed around' opener plus problem-first pitch

Open with the 'heard the name tossed around' opener — 'Hey [name], we work with a few other [portfolio company type] companies. It's [name] from [company]. Have you heard our name tossed around?' — mentioning mutual connections or shared context before pitching. Then lead with the problem before the solution (a 'problem proposition' or 'reverse pitch'). For objections, use the 'Mr. Miyagi method': agree with the objection and sell a test drive before fully addressing it.

30 Minutes to President's Club | No-Nonsense Sales · #606 - How To Build An Elite Outbound Sales Team (Get Reps Dialing Fast)

Method described by Armand Farrokh

Method preserved for inspection; not a verified protocol.

endorsed

Method · Money

Hiring funnel: recruiter screen, manager interview, and double mock cold call

Start with a 15-minute recruiter screen to filter out candidates who cannot compose a sentence or show basic professionalism, protecting the team's time. In the hiring manager interview, assess whether the candidate is 'smart' and 'cares' by asking why they want to get into sales and what they did to prepare. Then run a mock cold call twice: the first call will suck and that doesn't matter — evaluate whether the candidate listened to prep material and tried to apply it, and whether the gears are turning. Give no more than two points of feedback, coach them to lead with a problem statement instead of a pitch, then run a second call to see if they adapt. If they don't change anything on the second call, that's a massive red flag. After the mock calls you should know if you have a 'triple A'

30 Minutes to President's Club | No-Nonsense Sales · #606 - How To Build An Elite Outbound Sales Team (Get Reps Dialing Fast)

Method described by Armand Farrokh

Method preserved for inspection; not a verified protocol.

endorsed

Method · Money

Run five weekly meetings to operate an SDR team

Run an SDR team on five weekly meetings: Monday morning meeting (energy + activity focus), Tuesday teardown (cold call/email/account review), Wednesday one-on-ones (individual coaching), Thursday teardown, and Friday team dials (energy + activity). Bookend the week with energy-focused meetings on Monday and Friday to keep the team dialing.

30 Minutes to President's Club | No-Nonsense Sales · #606 - How To Build An Elite Outbound Sales Team (Get Reps Dialing Fast)

Method described by Armand Farrokh

Method preserved for inspection; not a verified protocol.

anecdotal

Method · Money

Build a guessed org chart from the leadership page and LinkedIn, then have your champion correct it

A concrete multi-step method: (1) go to the company's leadership page and manually add CEO, CFO, SVPs; (2) use LinkedIn and Sales Navigator to find additional CXO/SVP/VP-level people; (3) guess reporting lines (e.g., Scott reports into Jordan, COO Valentina reports into Jordan); (4) add people like Fatima and Nadia based on best guess; (5) make up an excuse for a one-on-one with the champion and walk them through the guessed chart so they correct it. Laura prefers doing this in a tool called 'Mirror' and has built charts down to six layers in some organizations.

30 Minutes to President's Club | No-Nonsense Sales · #608 - How Mapping the "Shadow" Org Chart Got Me to 305% of Quota

Method described by Laura

Method preserved for inspection; not a verified protocol.

Method · Money

Exec pairing playbook: map the org, get introductions, and match levels

A multi-step method for building multi-threaded executive relationships before a deal closes: map the customer's org chart and power base in a Miro board, seek to be corrected to learn the real org chart, get introductions from the champion to key people by framing why access is in the champion's interest, then play matchmaker by pairing peer levels (CEO to CEO, engineers to engineers, C-suite to C-suite) using the EA, a ghost-written message, and a prep doc.

30 Minutes to President's Club | No-Nonsense Sales · #608 - How Mapping the "Shadow" Org Chart Got Me to 305% of Quota

Method described by 30 Minutes to President's Club hosts and guest Laura

Method preserved for inspection; not a verified protocol.

The hosts present this as an endorsed playbook for enterprise deals; no outcome data is cited.