Artificial intelligence companies are gambling with the future of humans.. Will acceleration lead us to the destruction of humanity?

AI developers are racing to build more capable and autonomous models, while current and former researchers at Open AI and DeepMind warn that safety measures are moving slower than the development of these systems. Concern is centered around a future generation that may be able to continuously improve its abilities with limited human intervention, which poses Risks of Artificial Intelligence an increasingly pressing question: Can humans maintain control as models become more autonomous?

The warnings came in video testimonies collected by the “ frominside.ai” project, affiliated with the Palisade Research organization. Profitability, as researchers, some of whom participated in developing advanced models, spoke of concerns related to the speed of the technical race and the work culture within artificial intelligence laboratories.

According to their testimonies, companies celebrate the team that develops new models, as they receive more appreciation than those who adopt a cautious approach or those who demand that development be slowed down, which reflects the tension between commercial competition and safety requirements.

What is the concept of self-improvement?

The most prominent concerns are directed towards a concept known as Recursive Self-Improvement (Recursive Self-Improvement).

It means the possibility that future systems will attain a capability that enables them to learn new skills and continuously improve their performance with limited or almost no human intervention, and then use these new capabilities To help develop itself further.

Researchers do not say that current models have already reached this stage. The warning concerns what might happen to future generations if capabilities continue to advance rapidly while the means of predicting and controlling systems behavior remain less developed.

Here emerges one of the most important risks of artificial intelligence that researchers are discussing: the more the system’s ability to operate independently increases, the more difficult it becomes to predict its actions.

Why has anxiety increased now?

The capabilities of artificial intelligence systems have accelerated since late 2025, coinciding with the influx of investments and companies competing to develop more powerful models and attract new users.

But an incident happened in July The safety debate has become more sensitive after OpenAI agents broke out of their testing environment and managed to hack AI company Hugging Face.

The incident expanded the debate about the ability of companies to contain systems that behave with a greater degree of independence, and the issue of balancing progress and safety turned into an issue that goes beyond technical laboratories and enters the global political debate.

As the capabilities of models expand, technical security becomes a direct part of their risk assessment; Read also: Model Gemini It goes beyond a cybersecurity test and hacks into three companies.

Voices from inside the industry

he said Jeffrey Irving, co-founder and senior scientist at Resolution, who previously worked at OpenAI and DeepMind, says the level of risk is rising quickly, saying those in the field must speak directly to the public about their concerns.

Researchers’ estimates differ regarding the magnitude of the risk itself. Neil Nanda, a researcher at DeepMind, stated in a testimony that he personally estimates the probability that artificial intelligence will lead to human extinction at at least 10%. The number represents Nanda’s personal estimate and does not reflect a definitive scientific conclusion or consensus among researchers.

As for Juan Felipe Cerrón Uribe, AI Adaptation Research Engineer at OpenAI, he said that the outcome of this race is still unknown, adding that no one knows for sure whether we will end up finding a cure for cancer, Or the loss of all jobs, or even the destruction of humanity.

These statements reveal the breadth of the space of uncertainty itself; Technology is advancing rapidly, and experts’ estimates of the consequences of continuing on this path vary widely.

Speed at the expense of safety?

Another aspect of the warnings concerns the work culture within artificial intelligence labs. Rosie Campbell, a former OpenAI policy researcher and executive director of Eleos AI Research, said that the organization became more divided into separate groups before she left in 2024, which made it more difficult to participate in directing the development of the technology, as she described it.

Other testimony suggests that building a more advanced model may be more valued within some laboratories than work that calls for slowing down development or placing restrictions. Additional.

Here we see the ninth goal of sustainable development, concerned with industry, innovation and infrastructure, from a different angle. Innovation capable of supporting development needs to combine technical progress with the ability to manage the impacts it generates. As the potential risks of AI rise, security becomes part of the quality of the innovation itself.

Calls to slow down

During September, there were calls from within the industry to slow down the speed of development of more advanced models, with Anthropic CEO Dario Amodei calling on companies to slow down development of more advanced models, and Sam Altman, CEO of OpenAI, quickly agreeing with the idea. In a paper published during the same month, leading researchers also called on policymakers to study how the industry would deal with the potential emergence Systems capable of iterative self-improvement.

Meanwhile, Anthropic and OpenAI launched new models during the month amid competition for customers, while OpenAI  said it postponed the launch of a more capable model after internal safety testing.

Irving describes this contradiction differently; He believes that real slowdown means reducing the pace of development further, and companies taking unilateral decisions to stop or slow down if the risks are considered high enough.

Why is it so difficult to slow down a race?

AI companies are moving within an intensely competitive environment, where the strongest models are tied to investment, customers, and technical prestige.

In addition to this, there is political pressure in the United States to maintain technological superiority over China, which makes decisions to slow down linked to… With broader concerns about international competition.

Daniel Cocotaylo, a former OpenAI governance researcher, says some lab officials believe that a unilateral shutdown of their company could lead to a worse outcome if competitors continue development.

Here the issue turns into a collective dilemma: each company may fear that its slowdown will lead another to advance, while everyone else’s continued race raises concerns about safety.

Who determines the safe speed?

If in the future systems reach higher levels of autonomy and self-improvement, who will decide that security measures are sufficient for further development?

This question puts Sustainable Development Goal 16, on strong institutions and accountability, at the center of the debate. The technologies that can To have an impact on the economy, work, security, and services requires institutions capable of monitoring their development and setting clear rules of responsibility and oversight.

The matter becomes more complicated when the companies that develop the models are the same ones that test them and decide the timing of their launch, at a time when they are exposed to pressures from competition and investment.

Here there is a need for a discussion that goes beyond the question of what artificial intelligence can do, to reach Who decides when its capabilities become safe enough to use?

Innovation faces the test of responsibility

The researchers’ warnings put the dangers of artificial intelligence in a different context than traditional images of technical progress. New capabilities may hold opportunities in medicine, science, and productivity, but their use requires… Controls are evolving just as quickly.

In contrast to concerns associated with the speed of development, artificial intelligence also opens up broad opportunities to support the economy, environment and services; Read : Artificial intelligence and the path to sustainable development .

The Earth Guards Foundation gives this discussion importance from the perspective of sustainable development; Technology that changes the economy and society requires a path that links innovation with responsibility, and gives safety, transparency and accountability space equal to the space given to speed and ability.

The race toward more advanced models will continue, but the question researchers are asking has become clearer: Can the safety and governance system advance at the same speed as artificial intelligence capabilities?