Explore intelligence

Report · Money

To train a judge: build a golden dataset of ~30 question-answer-score triples scored by a human, remove your scores, let the judge score, and compute agreement

To train a judge (not ML training): you have an LLM you want to use as your judge, and you make sure it agrees with you using a dataset. Step one is to have a long list of data — question, answer, score triples — where the score is one that you or a subject matter expert human has scored, a trustworthy score. Start with about 30 examples, then let it grow. To train the judge, remove your scoring from the dataset, keep only the question and answer, send that to the judge, and let it produce its own scores. Compute the delta between your scores and the judge's scores to find your percentage of agreement. Aim for 80–85% agreement, maybe a bit more. Under 80% you won't get much value, because even a team of human support agents only agree about 80% of the time. If your judge agrees 100% of the

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Claim from Tejas Kumar

Report · Money

Once in production, eval data goes stale: skim real traffic, have a team score it manually, and check judge agreement to stay relevant

Once in production, there is a high chance your eval data will go stale, so you need to actively stay relevant. To stay relevant, skim real traffic: collect new user inputs and your agent's outputs, have an internal team score them manually, and check the judge's agreement against those scores. Your eval data set should be a living data set — continuously updated with fresh scored traffic. A living data set with an up-to-date judge is very difficult to break and provides reliability ahead of time. After skimming public traffic, you will have even more data to refine your evals. In production, skim the top of the data, get a little more data, and refine and refine.

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Claim from Tejas Kumar

Report · Money

Evals are afforded strategically: start with deterministic tests, then regex, then traces, then a judge with the cheapest model

To keep evals cheap, start with actual deterministic tests (unit tests). If deterministic tests can't be done, the next level up is regular expressions or pattern matching. If pattern matching fails, the next thing to do is examine your traces. Ideally, everyone is using some type of observability solution and has traces of each message envelope, inputs, outputs, tool calls, etc., which can be validated per object. Eventually you will come to a place where you need to use a judge, and then you start with the cheapest model. Evals are afforded strategically, because it will not be cheap if you use the most expensive model, and you want to build up to it. Regular expressions can be used for testing AI outputs but have their limits.

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Claim from Tejas Kumar

Report · Money

Judge training is iterative: investigate dissonance, find the knowledge gap, fill it with policy; agreement grows toward the threshold

When humans and AI don't agree, it's usually because of a knowledge gap — and the way to reach agreement is to introduce context to bridge that gap. To improve judge agreement, examine the cases where you disagreed and fill the knowledge gap with policy. In production you would supply policy to the judge via a RAG pipeline that retrieves enterprise context; in a demo you can just hard-code the policy. Write the policy explicitly (e.g., a 14-day return window; defective items can be returned) and interpolate it into the judge prompt. Building your judge is arduous iterative work — investigating failures and filling knowledge gaps is literally the job. After adding policy, check whether agreement is growing even if it still fails — improvement indicates you're on the right track. When the

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Claim from Tejas Kumar

Principle · Money

Engage the account immediately at AE-to-AM handoff and verify the executive outcomes

The minute the account is yours, engage right away so the value loop starts turning — the AM not being involved in implementation is a mistake. At handoff, verify (don't assume) the executive outcomes the AE used to win the deal, grab them, keep the CSM on target to them, and deliver those as the first set of value; that proof is what lets the next outcome happen.

30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account

Claim from Mark (guest)

Report

Environment ID routes to the same sandbox; state persists across turns

The environment.id parameter enables routing to the exact same context so you can use the same sandbox. As long as you preserve the environment ID, it will return you to the original sandbox you had. Any packages installed or files created remain accessible to the model through each turn, as long as it passes back the interaction ID and the environment ID. You don't need to manage any persistence of state yourself; you only need to add a few lines of code.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Claim from Ivan Leo

Report

Mix and match inbuilt tools in a single API call; the model reasons autonomously

With the Interactions API you can mix and match different inbuilt tools, which wasn't possible before. Example agent setup: give the model Google Search (the same index Google has), the URL context tool to retrieve information from web pages, and a custom tool called File Incident, to look up the latest security reports for a React application. With mixed tools, the model can reason autonomously, figure out what information it needs, and generate a response — all in a single API call. Giving models internet access makes them powerful because they can work with information beyond their training cutoff date.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Claim from Ivan Leo

Report

Interaction ID preserves context across turns via previous_interaction_id

With the responses API, the first turn (e.g., 'hi, my name is Phil') returns an interaction ID; passing that ID back as previous_interaction_id on the second turn preserves all context. The interaction.id parameter lets you preserve and work with context and ensures you're not busting your cache.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Claim from Ivan Leo

Report

Managed agent provides a persistent sandbox out of the box with a single API call

In the past, building a coding agent required queueing in the harness, finding a sandbox provider, managing the infrastructure, and figuring out how to preserve context between runs. With the anti-gravity managed agent, a single API call gets you a persistent sandbox out of the box that you can keep hitting and treat as a personal cloud without any modifications.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Claim from Ivan Leo

Principle

Regulatory Strategy: Self-Regulation vs. Nationalization

Industry self-regulation (e.g., Hollywood's MPAA) can prevent heavy-handed government intervention. However, if risks are deemed 'non-zero' for catastrophic outcomes, the logical regulatory response is nationalization, as seen in critical infrastructure sectors like banking and power.

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Claim from Steven Sinofsky

medium

Principle

Grounding AI Safety in Concrete Risks

AI safety discussions should be grounded in concrete, identified risks (e.g., novel cybersecurity threats) rather than speculative super-intelligence scenarios. Security claims should be validated using physics-based and systems-based constraints (e.g., entropy, bit rates) rather than abstract 'super-intelligence' capabilities.

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Claim from Steven Sinofsky

Principle

The Shift to Probabilistic Programming

The center of AI innovation has shifted from model creation to the integration of models into traditional software systems. This involves using language models to return probabilities for conditional logic (e.g., 'if X percent') rather than binary true/false checks, reviving 1960s-70s probabilistic programming research.

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Claim from Martin Casado

medium

Commercial context: The creator presents this as their own offering and directs viewers to get it.

Principle

AI Agent Threat Model: Untrusted Oracles

AI agents should be treated as untrusted entities with oracle-like capabilities in threat models. Unlike human employees, they can execute thousands of security probes rapidly without fatigue, requiring granular access controls and monitoring of internal APIs for denial-of-service style attacks.

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Claim from Steven Sinofsky

Method · Money

Check whether your site is blocking AI crawlers via robots.txt

A short check the creator walks through: visit yourdomain.com/robots.txt and look for disallow rules next to GPTBot, OAI-SearchBot, ClaudeBot, or Google-Extended. If you see disallow rules next to any of these AI crawler names, you are blocking that AI crawler from your site.

Ahrefs · Learn 80% of AEO in 19 Minutes

Method described by Ahrefs

Method preserved for inspection; not a verified protocol.

Method · Money

How to identify your buyers and build content around them

Hormozi lays out a method: if you don't know who your buyers are, look at your customer base. Look at the people who spent the most money — the top 20% — and the common factors they have. Look at what messages and problems they had, then create messaging and video topics based on the problems of the people with the most money in your audience, and make content about that. Put UTMs on the links below the descriptions in your videos to track revenue attribution, and make CTAs inside the videos to take the next step, such as a lead magnet. That is how you can see on the back end how content translated into revenue for the business.

Alex Hormozi · How to become rich with social media (my exact playbook)

Method described by Alex Hormozi

Method preserved for inspection; not a verified protocol.

resolved

Method · Social

How to structure and deliver a compelling story

The narrator lays out a coherent method for telling a story: open with a hook line that grabs the group's attention before any scene-setting, follow it with substantive storytelling that delivers on the hook's promise, set the scene with enough detail for the audience to imagine it in their mind's eye (even though it's the least interesting part), set up the story in under 30 seconds, include only details that will pay off later, and end with a bang — usually a laugh or a lesson.

Charisma on Command · Matthew McConaughey’s #1 Charisma Skill

Method described by Charisma on Command narrator

Method preserved for inspection; not a verified protocol.

Method · Money

Agentic collections playbook: invoice fast, CC stakeholders, follow up weekly, escalate to humans when late

Automate accounts receivable collections with an agent — it is a universal pain point that most companies still handle manually. The playbook: send the invoice to the correct finance point of contact within 10–30 minutes of the deal closing; check whether a finance contact is listed on the contract and email the line-of-business person as primary and CC the finance contact (or vice versa); if no finance contact is listed, use enrichment to look up and guess the right finance recipient (most companies have standard AP or finance inboxes); CC multiple stakeholders on every collections invoice (contract signer, deal POC, finance contact, line-of-business owner) so someone can always escalate; have the agent monitor the collections inbox daily; send a first payment-reminder touchpoint one week

The Official SaaStr Podcast: SaaS | Founders | Investors · SaaStr 880: Agent Pricing is Chaos. Here's What We're Seeing From the Buyer Side. (The Agents #015)

Method described by The Official SaaStr Podcast: SaaS | Founders | Investors

Method preserved for inspection; not a verified protocol.

Method · Social

Three questions to develop theory of mind

To drastically improve your theory of mind, ask three questions: What is this behavior doing for them? Why do they need to do that? And what response are they trying to pull out of me? These questions help you look past what someone is doing and understand why they're doing it.

Charisma on Command · The Psychology That Disarms Emotional Games

Method described by Charisma on Command

Method preserved for inspection; not a verified protocol.

Method · Money

Coinbase's 'brain': markdown knowledge files agents ingest, with reviewer corrections fed back in

Coinbase's approach: each team and repository has a 'brain' implemented as markdown files that team members can review at any time. Every time an agent comes in to make a change to a service, it ingests the brain for that team and repo. When a human reviewer fixes something an agent missed, that context must go back into the brain so it is fixed in all future cases — the simple manual-edit-and-ship response is the wrong move. This causes the accept rate for one-shotted PRs to tick up over time, creating a recursive self-improvement system.

No Priors: Artificial Intelligence | Technology | Startups · Coinbase’s Everything Exchange: Agentic Finance, Stablecoins, and Tokenization with CEO Brian Armstrong

Method described by Brian Armstrong

Method preserved for inspection; not a verified protocol.

Method · Money

Building a task sandbox: choose harness, MCPs, network rules, and resources

Jim Clark walks through deciding what capabilities a task sandbox needs: which harness to use (OpenCode, Claude, Gemini, Codex), what MCPs are allowed to be used in the sandbox, what networking rules to apply (e.g., whether it needs access to api.github.com, whether it needs to surf the web in random ways or just needs nothing), and what resources and work trees it needs. Model capabilities and build the right sandbox for the task, then put the task in. Treat orchestrators as having a job of asking what the task is and what sandbox to put it in.

AI Engineer · How Many Credentials Should Your AI Agent Have? Zero. — Jim Clark, Docker

Method described by Jim Clark

Method preserved for inspection; not a verified protocol.