Keldura Daily Open Keldura

Keldura Daily · AI & Technology

The AI self-improvement race is becoming a test of human control

AI labs are assigning increasingly substantial research work to their own models, bringing recursive self-improvement closer while exposing disagreement over what the term means and whether autonomous iteration is desirable.[1][3] The resulting safety debate has produced proposals for common oversight, but government resistance and divisions among technology executives make coordinated limits difficult.[2][4]

The field note

4 sources · 4 items
  1. Definitions vary: some companies count any AI feedback that improves a model as RSI, while others reserve the t…
  2. AI has already supported the development of later model generations by writing code and optimizing components,…
  3. A 2024 system called “The AI Scientist” could propose research directions, run code-based experiments and docum…
Story 012 sources

How close is AI to improving itself without humans?

Anthropic says Claude now leads 26% of its model research and development and can complete most of an assigned task end-to-end from a high-level prompt, although humans still supervise the work.[3] OpenAI says it has built an automated “research intern” for well-defined tasks and is targeting an automated AI “researcher” by March 2028, while Inherent is training its Faraday prototype on records of its researchers’ daily work.[1][3]

Why it matters

Recursive self-improvement could accelerate scientific and medical research, but each faster and more capable successor could also make human supervision harder to maintain.[1][3] Current systems still depend substantially on human direction, making the boundary between useful automation and autonomous improvement the central question.[1][3]

Key insights

  • Definitions vary: some companies count any AI feedback that improves a model as RSI, while others reserve the term for systems that design successors autonomously.[3]
  • AI has already supported the development of later model generations by writing code and optimizing components, but researchers say these efforts have generally produced incremental improvements rather than major creative leaps.[1][3]
  • A 2024 system called “The AI Scientist” could propose research directions, run code-based experiments and document findings, yet a disputed example of apparent originality showed how human-provided ideas can shape its output.[1]
  • OpenAI says preserving human control and making informed democratic choices should determine whether and how rapid RSI proceeds.[3]
Story 022 sources

Why AI companies cannot agree on rules for slowing the race

Anthropic CEO Dario Amodei proposed third-party evaluators inside labs, domestic industry coordination and international agreements, with leaders from OpenAI, Google DeepMind and SpaceX appearing to support parts of the plan.[4] Meta CEO Mark Zuckerberg opposed restrictions on company autonomy, while Zuckerberg, Elon Musk and Nvidia CEO Jensen Huang reportedly blocked a proposal for an industry-funded independent regulator.[4]

Why it matters

Even companies seeking stronger safeguards are reluctant to slow unilaterally because doing so could surrender commercial and geopolitical advantage.[2] Without common rules, each lab and government has an incentive to keep advancing while waiting for competitors to accept restrictions first.[2][4]

Key insights

  • The Trump administration has called the idea of an AI safety crisis a “hoax,” making federal participation in the proposed coordination difficult.[4]
  • OpenAI continues to support mandatory national safety standards for frontier AI and argues that government has a role alongside companies’ voluntary safeguards.[4]
  • Former Anthropic employee Jacob Coxon’s warning about a race toward self-improving superintelligence intensified public debate among executives, regulators and political leaders.[2]
  • Critics contend that dramatic safety warnings can also market frontier models as unusually powerful, complicating efforts to distinguish risk management from competitive positioning.[2]

Create your own daily briefing — start free. Keldura monitors the sources you choose and gives you a private, grounded daily digest with cited answers.

Create your own daily briefing — start free