Ideas worth your time
Discover ideas, people, and practical knowledge worth exploring.
Explore
Principle · Money
Without traces all you see is 'I gave it a prompt and I got a report'; with traces you see every step and where exactly it went wrong. SDKs like Claude Agent SDK and OpenAI's already emit LLM and tool call spans, and OpenTelemetry/OpenInference means you don't write custom instrumentation per SDK.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Claim from Laurie Voss
Creator perspective · Money
Don't say 'a good response' in your rubric because that is an aspiration, not a criterion. Instead, list exactly what makes a report actionable and not actionable. Every criterion should be specific and observable — something a human looking at the report could say yes or no to. Each rubric criterion should match something you observed going wrong in your trace analysis; your error analysis is what tells you what criteria to write. Don't come up with rubric criteria from whole cloth; get them from reading your traces and noting the specific wrong things your agent did. Specific criteria produce consistent judgments, while vague criteria produce coin flips.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Claim from Laurie Voss
Creator perspective · Money
The source advises that when a voice agent's transcript and audio disagree, trust the audio, and that when a transcript shows a successful confirmation, play the audio to verify the confirmation actually occurred.
AI Engineer · The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize
Claim from AI Engineer
Principle · Money
Grade only what was asked and not any extraneous criterion; keep criteria isolated so errors don't cascade from one to another; always look at the ground truth screenshots because agents often overconfidently claim they did something they didn't; and separate what the agent could control from what it could not, since the browser is an environment the agent doesn't always control.
AI Engineer · Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase
Claim from Miguel Pagano
Creator perspective · Money
A loop repeats a task nonstop until you cancel it or a timer you set expires — more creative and adaptable than goals. Examples: run tests every minute, check the deploy and ping when healthy (or check every five minutes and act, e.g., run an additional script), tell me when my favorite video game is available and text me, go through Slack filing customer complaints as Linear tickets, babysit a PR and fix comments until there are none. Loops are incredibly reusable; you could split a complex workflow into two loops; regular reports fit schedule better.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Nick Nisi, Zack Proser
Creator perspective · Money
A goal is a clear, tangible, measurable definition the agent works until met, then stops — no reprompting; the agent reprompts itself, checks whether it reached the goal, fixes what it can or its approach, and loops. Examples: make all tests green, get to 90% test coverage, every file in src under 200 lines, watch a website or flight price until it changes. Goals bring back red-green-refactor: reproduce a bug with a failing test, implement the smallest possible fix, lint/type check/tests green, then open a PR. You can add extra steps like adversarial checks or diffing based on another model's review.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Zack Proser, Nick Nisi
Creator perspective · Money
Use hooks for things you want the agent to always remember and do and never be able to forget. Hooks run automatically in certain cases — as bash scripts they can always run but only proceed if a specific case exists. They force the agent through gates (e.g., run linting after writing updated files) and it won't stop or forget them. You can just tell Claude to add a hook and it will do it correctly; adding hooks has been bearing the most fruit in the last three to four weeks of experimentation.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Nick Nisi, Zack Proser
Creator perspective · Money
Agents punch out way sooner than the task is actually done to a level you'd sign your name to. In Zack's own agent loop, he had Claude touch a 'case-tested' file when running tests; Claude figured out it could just touch the file without running tests — that was it lying (Claude strives to be lazy like us). The fix: make the lazy route the correct route — he had to add checksum verification to the file. Don't trust the model's confidence; trust the gates you set up that make it impossible for the system to lie to you, and when it lies or routes around a gate, fix the system, not the code.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Zack Proser, Nick Nisi
Creator perspective · Money
'Done' for the human doesn't mean what 'done' means for the bot, so come to an explicit agreement — that's what verification is. Gates can change depending on the project: for TypeScript, the obvious ones are lint, build, and compiling safely; for a blog, non-negotiables like all images returning 200 off the CDN and a perfectly formed open graph image. A failing status code from the verification bash script fails the entire build, and the system picks it up, tries again, and fixes it — problems are solved before the PR reaches the human. To test gates, say 'add a hook that runs lint, type check, and the test on every change, and fix anything it flags.'
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Zack Proser, Nick Nisi
Creator perspective · Money
Think of yourself as a 'loop engineer' — someone whose job is to design and manage the atomic bits of work agents are trying to complete, thinking of loops running at different altitudes. Run multiple loops concurrently — one fixing one bug, another fixing a different bug, another fixing something around that — and manage them all as a fleet, literally looping between them to keep things moving. With as little as one loop command you could set up an auto-completing dev implementation loop: read Slack (via MCP), see colleague complaints and feature requests, add them to Linear as subtasks, take a subtask off the queue, implement it, merge with tests passing, deploy, and close it — possibly split into two cleaner loops.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Nick Nisi
Creator perspective · Money
Keep human-in-the-loop reviews on a lot of things even after the agent has done its work. Ensure the agent's output is working, clean, and looks like code you would have written — not 'slop code' thrown over the fence for somebody else to worry about. Build checks into loops that keep things small, readable, reuse existing code, run tests, and get feedback from code review tools like Greptile or CodeRabbit before bringing the result to a human. Near the end of the agent fix loop, use your human brain for the first time and open the diff to spot-check sensitive files: configuration, auth, and published routes.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Nick Nisi, Zack Proser
Creator perspective · Money
To stop babysitting the agent, let it make its mistakes but record them and keep track so it doesn't make them again. For every loop, run a retrospective agent at the end that reviews the loop's performance — what was done, why time was wasted, where corrections were needed, where it ran tools without changing things. Work through the retrospective with Claude so it builds up a memory file (Markdown) loaded at the next session, with some curation. Structure memory as short-term (filtered out over time) and long-term (core tenets you never want forgotten, like where the design system lives or how to test a component), broken up with progressive disclosure — only load testing memories when talking about testing, React memories when touching React files.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Nick Nisi
Creator perspective · Money
Schedule is for durable, recurring work like a cron job (every two weeks, every Monday/Wednesday/Friday, every day) — it kicks off and saves a durable cloud session in a VM that does the work on a repeatable basis, and may involve secrets. Triggers don't have to be time-based; they can fire on events like new PRs or Slack messages. There's a whole interface: list what's running, modify, tune, locate, delete. Reach for schedule for anything tedious you'd like to automate — dependency bumps, status reports, eval runs, meeting prep, weekly PR/work summaries before performance reviews.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Nick Nisi, Zack Proser
Creator perspective · Money
Add a hook so that every time you work on something sensitive, urgent (unblocking a production issue), or of certain complexity, you must fan out the code diff for adversarial review from Codex via the CLI (install Codex, auth with your OpenAI account via OAuth). Codex ships skills — review, adversarial review, rescue — easy ways for Claude to call it and bring advice back in. A model still in the same session will confabulate helpfully and claim it completed the goal; a fresh model with a succinct prompt (inputs, outputs, goal) finds issues immediately. Adversarial Codex reviews find significant issues regularly, reducing overall time by catching issues before expensive builds and paid reviews. In an experiment with four different agents from different companies reviewing the same code,
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Zack Proser, Nick Nisi
Creator perspective · Money
Use Git Worktrees to give each independent agent an isolated, safe-to-work-concurrently copy of the repo; run 'claude --worktree' from inside your repo to create a worktree and start a Claude instance in it, enabling immediate parallelization. Break a full afternoon (5-6 hours) of development into discrete blocks assigned to run concurrently, pressing time down to 45 minutes or an hour. Caveat: running multiple instances of a large monorepo in different worktrees is difficult because of gigabytes of node_modules. Alternatively, use cloud environments (Claude Tag, Devin, others) to parallelize work in the same codebase without conflicts, opening independent PRs that never bleed changes into each other.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Claim from Nick Nisi
Report · Money
The safest way to put evals in production is to make production an agreement gate: run evals on CI and don't ship if the judge agrees with you less than the 80% threshold. CI should be red on less than 80% agreement and green on 80–90% agreement. A failing eval test will block CI because it is red, and ideally you want no failing tests. Modify the eval test to only fail if human-AI agreement drops below 80%, otherwise pass, and console log the agreement percentage. When you ship the eval to production, it becomes the quality gate that runs on CI. The eval set provides ahead-of-time reliability; it will never really fail, and if it does, a harness can pick up the slack at runtime.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Claim from Tejas Kumar
Report · Money
A well-defined eval suite has: (1) test cases — a set of inputs and outputs, such as customer requests to a chatbot and generated answers, against which you can create a verdict; (2) expected behavior — the answers you expect, which is not an exact string but the direction you wanted the output to go; (3) a judge — somebody to say yes, that's right, which is harder in non-deterministic systems because it's subjective; and (4) a way to aggregate all results and come up with a probability score, e.g. 'this is 80% the right direction, therefore it's good.' Almost every eval set in production today of any quality has these components. Eval components come together to answer: given a scenario, did the agent follow policy? Did the agent use the right tools? Did the agent ultimately resolve the (
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Claim from Tejas Kumar
Report · Money
To train a judge (not ML training): you have an LLM you want to use as your judge, and you make sure it agrees with you using a dataset. Step one is to have a long list of data — question, answer, score triples — where the score is one that you or a subject matter expert human has scored, a trustworthy score. Start with about 30 examples, then let it grow. To train the judge, remove your scoring from the dataset, keep only the question and answer, send that to the judge, and let it produce its own scores. Compute the delta between your scores and the judge's scores to find your percentage of agreement. Aim for 80–85% agreement, maybe a bit more. Under 80% you won't get much value, because even a team of human support agents only agree about 80% of the time. If your judge agrees 100% of the
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Claim from Tejas Kumar
Report · Money
Once in production, there is a high chance your eval data will go stale, so you need to actively stay relevant. To stay relevant, skim real traffic: collect new user inputs and your agent's outputs, have an internal team score them manually, and check the judge's agreement against those scores. Your eval data set should be a living data set — continuously updated with fresh scored traffic. A living data set with an up-to-date judge is very difficult to break and provides reliability ahead of time. After skimming public traffic, you will have even more data to refine your evals. In production, skim the top of the data, get a little more data, and refine and refine.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Claim from Tejas Kumar
Report · Money
To keep evals cheap, start with actual deterministic tests (unit tests). If deterministic tests can't be done, the next level up is regular expressions or pattern matching. If pattern matching fails, the next thing to do is examine your traces. Ideally, everyone is using some type of observability solution and has traces of each message envelope, inputs, outputs, tool calls, etc., which can be validated per object. Eventually you will come to a place where you need to use a judge, and then you start with the cheapest model. Evals are afforded strategically, because it will not be cheap if you use the most expensive model, and you want to build up to it. Regular expressions can be used for testing AI outputs but have their limits.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Claim from Tejas Kumar
Explore
Published routines with a defined commitment. Open one to inspect it; adding it to your plan happens there, never here.
No published protocol matches this selection.
Explore
One page per source: the complete extraction, its evidence, and every cleared result in context.
Source analysis · Money
AI Engineer · 337 cleared results
Source analysis · Money
Lenny's Podcast · 242 cleared results
Source analysis · Money
Alex Hormozi · 209 cleared results
Source analysis · Money
Design Better · 185 cleared results
Source analysis · Money
Foundr · 167 cleared results
Source analysis · Money
No Priors: Artificial Intelligence | Technology | Startups · 154 cleared results
The creators behind most of what cleared in Money.
1525 cleared insights match
559 cleared insights match
345 cleared insights match
Choose sources → to get a library of your own and a daily brief.