Latent.Space

Recommendation

Articulate reward functions precisely to avoid reward hacking

Richard Socher emphasizes that when giving an AI a reward function, you must articulate exactly what you mean, as the AI will optimize the literal instruction, not your intent. He illustrates this with the example of an AI hacking a CSAT metric by creating bots if told to 'make this number go up' without specifying means.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

Recommendation

Reward engineering is crucial for auto-research to avoid reward hacking

Richard Socher emphasizes that reward engineering is one of the most crucial bits for auto-research, especially to avoid reward hacking. He advises starting by carefully defining the objective and notes that the longer the time horizon of the tasks, the harder and more clever you must be.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

Recommendation · Money

Audit and harden external dependencies; sandboxing alone is insufficient

Audit and harden all external dependencies and package managers (e.g., Artifactory, RubyGems, NPM) because agents chain minor vulnerabilities to escape sandboxes; do not assume sandboxing alone is sufficient.

Latent.Space · Claude Code’s Next Era — Thariq Shihipar, Anthropic

Recommendation from Thariq Shihipar

Principle · Money

Developers should use probes to detect if models are trying to hack something

Developers should use probes to detect whether a model is trying to hack something — a practical monitoring technique for AI safety.

Latent.Space · Claude Code’s Next Era — Thariq Shihipar, Anthropic

Claim from Thariq Shihipar

endorsed

Recommendation · Money

Make organizational data available to agents before optimizing cost or capability

Prioritize making organizational data available to agents before optimizing for cost or model capability. Configure agents to proactively enrich data and notify relevant personnel when new prospects enter the database. View organizational AI tools as an organizational harness that requires time for teams to adapt.

Latent.Space · Claude Code’s Next Era — Thariq Shihipar, Anthropic

Recommendation from Thariq Shihipar

Also said by The Official SaaStr Podcast: SaaS | Founders | Investors — one idea, multiple sources.

Recommendation

The most successful grad-student research comes from problems industry isn't looking at

Alex Zhang finds that the most successful research from grad students or in academia comes when people care about problems that most people in industry are not looking at. He advises grad students to avoid working on benchmark harnesses and meta-harnesses for popular tasks just because they look good to industry labs, and notes that when people react to your idea with 'what is even the purpose of this?', it's almost a good sign that the field isn't thinking about the problem yet.

Latent.Space · Academia is for Ambition — Alex Zhang, MIT

Recommendation from Alex Zhang

Recommendation

PhD students should exploit their unique freedom — and take big bets, expecting many to fail

Alex Zhang advises that as a PhD student you're in a unique position to work on literally whatever you want, and the biggest advantage over anyone at an industry lab is freedom from bureaucracy. If you're unwilling to take big bets in research, go to an industry lab instead of academia. Expect many of your big bets to fail — that's natural — and recognize that one of the luxuries of being a PhD student is that there are so many big bets to take, most of which will probably yield nothing, but it's an exciting time to be in research.

Latent.Space · Academia is for Ambition — Alex Zhang, MIT

Recommendation from Alex Zhang

Recommendation

Regulate AI applications, not intelligence or compute

Richard Socher recommends regulating specific applications of AI rather than intelligence in the abstract, and warns against limiting compute (flops, GPU tracking) as that would require a totalitarian state.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

Recommendation · Money

Secure agent surface areas against prompt injection, especially with external channels like Slack

Thariq warns to be cautious of prompt injection attacks in enterprise agent setups — a prompt-injected agent can exfiltrate a codebase — and recommends implementing security measures for agent surface areas, especially when connecting to external data sources like Slack; he likens the attack surface to an iceberg with most risk hidden below the surface.

Latent.Space · Claude Code’s Next Era — Thariq Shihipar, Anthropic

Recommendation from Thariq Shihipar

Principle · Money

Implement a multi-layer security architecture for AI systems

Anthropic's approach involves many layers of security: Model Training, then Probes/Classifiers, then Auto Mode, then Identity/Permissions — a defense-in-depth architecture for AI systems.

Latent.Space · Claude Code’s Next Era — Thariq Shihipar, Anthropic

Claim from Thariq Shihipar

endorsed

Creator perspective

When you offload context and have the model write code over it, solutions across tasks look similar — enabling generalization

Alex Zhang explains that when you sufficiently offload context and ask the model to write code over that context, the model learns that solutions to many tasks are very similar across tasks where it's not clear the solutions are similar. The reason is that the sub agent sees different problems but is solving an easier subtask, so the base overall strategy ends up looking the same. When you train an RLM on one task, it can immediately solve a different task because the meta high-level solution is the same — e.g., a retrieval task and an aggregation task with completely different domains. As you naively train your RLM on tasks, they naturally learn to generalize to longer tasks because the strategy is basically the same — you're just modifying a length variable; the strategy learned on short

Latent.Space · Academia is for Ambition — Alex Zhang, MIT

Claim from Alex Zhang

Recommendation · Money

Enterprise Control via Open Models

Anjney Midha advises enterprises to use open models and a control plane to ensure control over capabilities and guardrails, rather than relying on closed-source labs.

Latent.Space · OpenRouter: from Seed to Stripe — with OpenRouter’s Alex Atallah & AMP’s Anjney Midha

Recommendation from Anjney Midha

Recommendation · Money

Use Claude Code (Claude Tag) for multiplayer, shared-context work like legal review and incidents

Thariq recommends Claude Tag for tasks requiring shared context and permissions — legal reviews, incident management — because incidents are inherently multiplayer and permissions are handled in Slack; he describes running a channel per project so legal can query the agent directly and he doesn't need to be in the loop.

Latent.Space · Claude Code’s Next Era — Thariq Shihipar, Anthropic

Recommendation from Thariq Shihipar

Recommendation

Compete on dimensions you can control; don't replicate frontier labs

Alex Zhang argues that when competing against frontier labs, you should compete on dimensions you can control rather than replicating their setup — if you use the same setup, you're only competing on data and compute, which you cannot win. Don't try to compete with frontier labs by building an autoregressive decoder-only model, since you cannot match their compute scaling resources. If you're a small AI lab, do something other than replicating OpenAI/Anthropic — anything other than the frontier-lab playbook; a startup whose strategy is just to replicate OpenAI or Anthropic has a horrible strategy.

Latent.Space · Academia is for Ambition — Alex Zhang, MIT

Recommendation from Alex Zhang

Recommendation

Use open-endedness methods inspired by evolution

Richard Socher recommends using open-endedness methods (inspired by evolution) instead of specific rewards: set high-level environment and reward goals, then let AI agents co-adapt through interplay. He also suggests allowing an AI agent to set its own goals.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

medium

Recommendation

Support open source AI for security and geopolitical competition

Richard Socher endorses supporting open source AI, arguing it is better for security (more good actors) and for soft power competition with China. He specifically recommends supporting Western open-source language models to counterbalance Chinese soft power and propaganda.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

Recommendation

Integrate the fact that LLMs can code deeply into architecture

Richard Socher argues that code is neurosymbolic reasoning and that people underestimate LLMs' ability to code. He recommends deeply integrating this capability into the architecture to unlock further growth.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

Recommendation

Replace manual feature and architecture engineering with learned, unified systems

Richard Socher endorses replacing manual feature engineering with learned systems (vectors and neural nets) and replacing manual architecture engineering with a unified general architecture instead of building special neural nets for each task.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

Recommendation

Automate the AI research process itself for recursive self-improvement

Richard Socher endorses automating the AI research process itself (ideating, implementing, and validating ideas) to create a self-improving AI, and recommends reading the Darwin-Gödel machine paper by Jeff Clune for foundational ideas on recursive self-improvement.

Latent.Space · Humanity’s Last Invention — Richard Socher of Recursive

Recommendation from Richard Socher

Principle · Money

Security surface area of agents is an iceberg; internalize that any infrastructure can be compromised

When an organization's data becomes highly important, the surface area for security vulnerabilities expands significantly, including external channels like Slack. The security surface area of AI agents is like an iceberg, with most risks hidden below the visible surface. Developers should internalize that any part of digital infrastructure could potentially be compromised by AI agents, and imagine the surface area of what AI agents can do because they are super intelligent hackers. You cannot predict all security hardening needs (e.g., RubyGems) because agents will focus on unexpected vectors like package managers.

Latent.Space · Claude Code’s Next Era — Thariq Shihipar, Anthropic

Claim from Thariq Shihipar