Max Tegmark and Yoshua Bengio warn that AI systems smarter than humans with the ability to act autonomously could be dangerous due to the lack of clear controls

General artificial intelligence could become a serious problem if adequate control over its development is not established. This is the warning from two of the leading AI experts, Max Tegmark from the Massachusetts Institute of Technology (MIT), and Yoshua Bengio from the University of Montreal, who have raised concerns about building systems that are too advanced.

The rapid growth of artificial intelligence has led to the creation of systems called “AI agents,” designed to act with greater autonomy and make decisions without direct human intervention.
Leading companies in the tech industry are promoting this evolution with the promise of enhancing productivity and improving everyday life. However, Bengio warns, in the CNBC podcast Beyond The Valley, that this approach carries a significant risk: granting AI the ability to act independently without effective oversight.

According to the researcher, the key to the problem lies in the agency of AI, meaning its ability to set and pursue its own goals. “We are creating a new form of intelligence on our planet without knowing if its decisions will align with our needs,” Bengio points out.
This uncertainty is what leads the experts to call for strict regulation before the technology advances too far.

The greatest fear among specialists is not only the autonomy of these systems but also the potential development of self-preservation mechanisms within AI. Bengio asks, “Do we want to compete with entities that are smarter than us? That’s not a very reassuring bet, is it? That’s why we need to understand how self-preservation could emerge as a goal in AI.”
This scenario could lead to a lack of control over advanced systems, making their evolution unpredictable.

The possibility that an AI might seek its own survival or make decisions that do not align with human needs raises both ethical and technical dilemmas. Although there is currently no AI with true consciousness or intentions, the ongoing trend toward increasingly autonomous systems is causing concern among researchers.

AI Tools Instead of Autonomous Artificial Intelligence:
Max Tegmark proposes a different solution to the development of autonomous agents, focusing on the creation of “tool AI.” This concept is based on designing intelligent systems with specific purposes and under a strict control framework, rather than allowing them to operate with full autonomy.

Examples of this approach include advanced tools for medicine, such as systems capable of suggesting cancer treatments, or autonomous vehicles designed with safety protocols to ensure human control at all times. Tegmark believes that, with proper safety standards, AI can evolve without becoming a risk.

“I believe that, optimistically, we can have almost everything we’re excited about with AI… if we just insist on having some basic safety standards before people can sell powerful AI systems,” says Tegmark.

The problem is that there are currently no global standards regulating the development of these systems, leaving the door open to potentially dangerous uses.

In 2023, the Future of Life Institute, led by Tegmark, urged companies to pause the development of AI systems that could match or surpass human intelligence, until proper control measures are established. Although this request has not materialized, the discussion around the risks of AI has gained significant relevance in recent months.

---

Silvia Cabrero Díez is a Spanish journalist specialized in technology and cybersecurity.

She currently works as a writer for Computer Hoy, where she has published numerous articles on artificial intelligence, electronic devices, and tech trends.

In addition to her work at Computer Hoy, Cabrero Díez has contributed to other digital media outlets like Xataka and Hobby Consolas, focusing on topics related to technology and science.

In her free time, she enjoys reading, watching series, and movies. Her journalistic approach is characterized by a clear and accessible style aimed at a general audience interested in technology.