Anthropic Warns Imminent Risk of Rapidly Accelerating Self-Improving AI Capabilities

Anthropic warned in a revised S-1A filing that the company may imminently reach a milestone known as recursive self-improvement (RSI), a state where AI helps develop increasingly capable versions of itself. The company's internal estimates suggest that future models could soon meet the threshold for "dramatically accelerated progress attributable to AI," according to the filing.

Under Anthropic's Responsible Scaling Policy, this threshold is met if AI systems either replace the company's entire research workforce at competitive costs or help double the aggregate rate of progress in AI capabilities through automated research and engineering. The company noted that while current models have not crossed this threshold, it has previously used AI to develop new models with capability leaps that researchers did not recognize at the time.

Anthropic cautioned that using AI to build more capable AI could pose significant safety risks if it sharply accelerates the pace of progress. The company reported that its increasing use of AI in research and development may cause model capabilities to advance faster than the company's ability to develop, validate, and implement necessary safety, alignment, and security measures.

"We do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible," the company wrote in the filing. To address these risks, Anthropic is considering updates to its policy to ensure humans remain in control of frontier model development.

Data from the company shows a growing reliance on AI for technical tasks. In September 2026, Claude authored over 90% of contributions to Anthropic's codebase, and the average daily code output per engineer during the third quarter was more than 45 times its 2024 level. Anthropic clarified that these figures measure code volume rather than equivalent gains in engineering productivity.

Leave a Reply

Your email address will not be published. Required fields are marked *

FI Journal
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.