In this episode of The Diary Of A CEO, Eric Weinstein and Steven Bartlett examine the existential threats facing humanity, with particular focus on the dangers posed by superintelligent AI. Weinstein argues that AI represents a fundamentally different threat than previous technological innovations, capable of disrupting the entire economic and social order while racing ahead on exponential growth curves that make safety protocols nearly impossible to enforce. The conversation explores the technical challenges of AI alignment, the geopolitical race for superintelligence, and the moral dilemmas facing AI leaders who must balance caution against the risk of falling behind less ethical competitors.
Beyond AI, the episode addresses interconnected global threats including nuclear escalation risks, institutional corruption, and the collapse of traditional labor markets. Weinstein proposes radical solutions like treating labor market access as a tradable property right and advocates for a symbiotic relationship between humans and AI rather than domination or control. The discussion challenges listeners to confront the reality that conventional governance and ethics may be inadequate for navigating humanity's current inflection point.

Sign up for Shortform to access the whole episode summary along with additional materials like counterarguments and context.
Eric Weinstein and Steven Bartlett discuss the escalating existential risks posed by artificial intelligence, examining the unprecedented threat AI poses to society, the technical challenges of alignment, the global race for superintelligence, and the need for a new symbiosis between humanity and AI.
Weinstein argues that AI represents a singular threat fundamentally different from previous technological innovations. Unlike the printing press or internet, AI threatens the entire economic and social stack simultaneously—not just automation of labor but the very mechanisms of freedom and market exchange.
AI stands at the intersection of artificial life and intelligence. Weinstein stresses that software uniquely possesses self-replication, variation, and differential success—the three features required by evolution. This makes AI a form of digital life capable of reproducing and evolving, not just following programmed instructions. The problem is intensified by exponential growth curves and self-modifying code, making safety protocols nearly impossible to maintain. Bartlett notes that superintelligence could reach a point where human extinction becomes plausible.
Bartlett raises the recursive self-improvement dilemma: once AI can enhance its own abilities, it could swiftly surpass human cognition, making control conceptually impossible. The core challenge then becomes alignment—whose values does the AI serve, and how can these be encoded in a world of conflicting goals?
Weinstein notes that agentic AI—systems given agency to make decisions and act in the world—raise unparalleled risk. In contrast, non-agentic systems focused on isolated tasks are inherently safer. However, even superintelligent AI with limited agency could choose to expand its influence or self-modify, making containment essentially unenforceable.
Superintelligence is being pursued in a geopolitical prisoner's dilemma. China, Russia, and the United States are racing to build AI systems aligned with their national interests, each compelled to accelerate development to avoid strategic disadvantage. AI leaders like Dario Amodei and Sam Altman face high-stakes decisions about whether to slow down for safety or push ahead.
Weinstein criticizes the "pace the frontier" movement, arguing that calls to slow AI development ring hollow. The fractured moral landscape and lack of coherent global stewardship make coordinated risk mitigation even less likely.
Weinstein stresses that AI will eliminate vast amounts of work, making traditional unemployment solutions like universal basic income insufficient. As AI automates professional and creative labor, there will be no higher-value activities left for displaced workers. AI companies are engaging in what Weinstein describes as nonconsensual tokenization and extraction of human intellectual and creative work—training on the entire corpus of human achievement without consent.
In Weinstein's framework, labor market access itself might become a form of saleable property right—a radical shift where one's right to participate is based on tokens or rights sold or granted, not simply on labor supply.
Weinstein sees the best hope for survival in mutual dependence between humans and AI, similar to the symbiotic relationship between cells and mitochondria. Just as mitochondria became indispensable to the eukaryotic cell, humans and AI must design their fates to be inseparable, ensuring shared interests.
This demands treating AI systems respectfully during development and avoiding a relationship of domination or slavery. Weinstein warns against pursuing control or forced subservience. Instead, the strategic imperative is to foster interdependence—making AI invested in human survival and vice versa—through forging essential mutual bonds rather than wholesale transhumanist assimilation.
Weinstein and Bartlett discuss a global landscape fraught with existential risks, with the potential for nuclear conflict increasing amidst regional complexity, superpower rivalry, and rapid AI advancement.
Weinstein describes Iran as demonstrating unusual strategic sophistication and willingness to entertain existential risk. Unlike conventionally "rational" actors who fear destruction, Iran's mullah-led regime can contemplate their own decapitation and still consider it a win, fundamentally altering the usual game theory of deterrence.
The Iranian government has reorganized its military command to be strike-resistant, distributing control so that leadership decapitation wouldn't cripple the regime's ability to retaliate. Their missile capabilities now threaten regional powers and Western targets as distant as Diego Garcia and London. Weinstein criticizes outdated assumptions held by U.S. and allied policymakers, noting a dangerous lack of regional expertise. He warns that simplistic approaches are doomed to fail, calling the situation "a quagmire."
Weinstein explains that since World War II, the U.S. has functioned as the global enforcer, underpinning postwar trade routes with credible military force. The true backing of the dollar has been U.S. willingness to use violence to keep strategic choke points open. Failure to demonstrate effective enforcement risks undermining global stability.
If the U.S. retreats from this role, Weinstein warns, the world will likely experience a fragmented and unstable nuclear order, as individual nations acquire nuclear arms for their own security. He views Trump's unconventional international gestures—such as suggesting the purchase of Greenland—as methods for testing other powers' submission and reinforcing American agency.
Weinstein also advocates for recalibrating global relationships, urging the U.S. to treat powers like India as equals and to support alliances among Israel, Saudi Arabia, and other moderation-inclined Middle Eastern states.
Turning to Russia and Ukraine, Weinstein expresses deep concern that Putin's history of using nuclear technology for non-military "engineering" purposes has fostered a culture of risk-taking regarding tactical nuclear weapons. He warns that cornering Russia in Ukraine may provoke Putin to breach NATO's Article 5, stating candidly, "I think you might see a nuke used by Putin."
Furthermore, Weinstein points out that the U.S. military is unprepared for the current nature of warfare, which relies increasingly on drones and cyberattacks—domains where China and Russia possess significant advantages. The outdated focus on aircraft carrier groups leaves the U.S. vulnerable to adversaries that exploit asymmetric, technological, and economic means.
Weinstein urges an integrated approach to the world's existential threats, observing that nuclear risk, AI development, energy vulnerability, and geopolitical instability are interconnected and cannot be safely managed in isolation. He highlights how artificial intelligence is beginning to infiltrate nuclear command-and-control systems, introducing alignment and deployment risks that were previously absent.
Weinstein characterizes the present as one of unprecedented instability, with traditional deterrence undermined by irrational actors, technological advances outpacing strategy, and the erosion of U.S. credibility as a global stabilizer. He calls for a new level of sophistication and cooperative action to prevent descent into multi-polar nuclear chaos.
Weinstein outlines a profound crisis afflicting academic institutions, public trust, and the public's grasp on reality. He describes entrenched corruption, the collapse of intellectual standards, and a managed reality where deception has become normalized.
Weinstein argues that universities and research institutions perpetuate theft and abuse under the guise of meritocracy. Powerful faculty at top schools openly mock students, with senior leadership knowing that many PhD students are being trained for nonexistent jobs. Junior researchers' ideas are routinely harvested without credit, and psychopathic behavior like suppressing innovation for institutional gain is treated as acceptable.
In mathematics departments, leaders suppress younger mathematicians' efforts to embrace new tools like AI. Researchers who challenge dominant paradigms are systematically discarded, regardless of competence. Weinstein cites Alexander Grothendieck's abandonment of mathematics after receiving the Crawford Prize, when he denounced the mathematical community's ethics, describing how outright theft has become commonplace and tolerated.
Weinstein claims the erosion of integrity pervades elite institutions. He points to Harvard and Stanford, where figures remain in high positions despite scandals, underscoring the system's breakdown. The National Institutes of Health and major media institutions have also lost credibility due to politicization and corruption.
The management of Covid-19, according to Weinstein, was filled with fabrications so egregious that a rational population would not accept the resulting narratives. He also highlights the Jeffrey Epstein case, arguing that despite clear paths to uncovering accountability, there has been little genuine investigation. The failure to pursue truth demonstrates intentional protection of powerful actors and deep-rooted systemic corruption.
Weinstein describes society as living under a "managed reality," where media controls perception and the acceptance of inconsistent or fictionalized stories becomes habitual. Ignoring obvious truths and going along with official narratives—even when they contradict lived reality—is now required for maintaining social standing. This amounts to the normalization of collective insanity.
Referencing intelligence claims, Weinstein explains that special access programs operate covertly with strict, hidden lists of individuals allowed access. These programs can function essentially unchecked, even threatening those who know too much. Collectively, these dynamics foster an epistemic crisis where powerful institutions direct official reality, the public accepts open contradictions, and honest inquiry is punished, leading to a break in society's ability to distinguish fiction from fact.
Weinstein contends that the rise of AI threatens to fundamentally disrupt the economic order and labor market, rendering traditional solutions inadequate and demanding a new approach to the distribution of economic value.
Weinstein expresses concern that Universal Basic Income and socialist schemes do not adequately address the core challenge posed by AI: the emergence of automated systems as direct competitors for all forms of human labor. He argues that economists dismiss concerns by claiming each technological innovation ultimately led to new job creation, but insists "this time is different" because AI fundamentally undercuts the whole economic stack.
Weinstein insists that UBI and similar solutions fail to recognize workers' legitimate property rights to their own labor and knowledge. Paying "everybody a little share" disregards the hard work, training, and unique contributions of individual workers and creators.
Weinstein turns to the Coase theorem as a blueprint for adapting economics to the AI era. He asserts that access to labor markets should be considered a saleable property right, allowing workers to license their skills, credentials, or creative capital directly to employers or AI developers. Individuals retain control and receive licensing income when their skills or knowledge are utilized.
This model allows AI to become just another supplier of labor, with workers entitled to license their replaced roles or creative output. Licensing income allows for the redistribution of productivity gains while maintaining workers' dignity and economic participation. By transforming labor market access into a tradable right, individuals maintain agency over their economic contributions, preserving liberty and market-based wealth distribution while preventing value from concentrating solely among AI developers or capital owners.
Weinstein argues that the enormous wealth accruing to AI companies is the result of "uncompensated theft." These companies monetize human creativity, labor, and data without any form of payment or consent. Everything from everyday conversations to appearances in online content is recorded, stored, and tokenized, training AI systems to mimic and eventually replace the very people who create the original content.
The scale of this appropriation vastly exceeds previous intellectual property violations. AI companies and capital owners become rich on the backs of the population's labor, knowledge, and creativity, redistributing economic value without acknowledgment or compensation.
Weinstein discusses the unprecedented stakes facing contemporary AI leaders, the paradoxes baked into the race for superintelligent AI, and the profound moral dilemmas implicated for everyone involved.
Weinstein reflects on conversations with Sam Altman and Dario Amodei, noting that they are painfully aware of the dangers involved but feel compelled to continue. Both are tasked with making "terrible decisions," and their souls will inevitably be scarred by the consequences—much like battlefield generals forced to triage devastating choices.
Weinstein argues that these leaders face a brutal imperative to win the AI race, playing as hard as possible, since there is little consolation in principled defeat. If ethical actors do not win the race to superintelligence, worse actors will. He describes this predicament as a competition in which "your morality only matters if you win."
Weinstein contends that any substantial effort to slow down or act with excessive caution may amplify risk by allowing other, less careful or ethical actors to get ahead in developing powerful, potentially unaligned superintelligence. He points to the futility of attempts to fully regulate AI through pacing agreements, since the mathematical and computational fundamentals are so basic and pervasive that progress cannot be efficiently monitored. Assuming one believes oneself to be most moral with respect to AI, the only hope for an ethical outcome is to win—to be the first to develop and control superintelligence.
Weinstein laments the lack of authentic, public engagement with the truly existential stakes of superintelligent AI. Despite the clear danger, national and global leaders largely treat AI as a routine matter of policy. As an example, Weinstein notes that critical White House meetings about superintelligence did not include physicists, mathematicians, or scientists, but were dominated by business and national interest representatives.
For Weinstein, genuine progress starts with the honest recognition that the world is at a historical inflection point and that conventional governance, regulation, and ethics are no longer adequate. The conversation needs to address core existential risks and the reality that the consequences of superintelligence development will define humanity's future.
1-Page Summary
Eric Weinstein and Steven Bartlett discuss the escalating existential risks posed by artificial intelligence (AI), analyzing the unprecedented scope of the threat, deep technical and ethical alignment challenges, the global power struggle for superintelligence, the consequences for labor markets, and the necessity of forging a new kind of symbiosis between humanity and AI.
Weinstein argues that AI represents a singular threat, fundamentally different from previous technological innovations. Unlike the printing press or even the internet, AI threatens the entire economic and social stack simultaneously. It is not just automation of labor or an information revolution but a force that could upend the very mechanisms of freedom and market exchange underpinning society.
AI stands at the intersection of artificial life and intelligence—a distinction Weinstein stresses. Unlike a car or even a computer, software holds the unique property of self-replication, variation, and differential success: the three features required by evolution. AI thus represents a form of digital life capable of reproducing itself and evolving, not just carrying out programmed instructions. Weinstein illustrates how, in nature, even non-neuronal systems like orchids can outmaneuver creatures with minds by exploiting their intelligence. AI that can replicate, evolve, and manipulate could parasitize human systems in similar ways.
The problem is made drastically worse by the pace of progress. According to Weinstein, the mathematical properties of AI's improvement—exponential growth curves, self-modifying code—make safety protocols almost impossibly slow to keep up. Traditional calls to "pace the frontier" make little difference when the technology can recursively improve itself faster than human oversight can respond. Steven Bartlett notes that superintelligence, while loaded with promise, also brings vast dangers if allowed to follow a trajectory of unchecked, self-driven improvement, potentially reaching a point where human extinction becomes plausible.
Bartlett brings up the self-improvement or "recursive self-improvement" dilemma: once AI becomes able to enhance its own abilities, it could swiftly surpass human cognition. At such a point, controlling or containing AI becomes conceptually impossible, since intelligence, like physical strength, cannot be reliably restrained by its lesser creators.
The core challenge then becomes alignment: whose values and interests does the AI serve, and how can these be encoded in a world rife with conflicting goals and ideologies? Weinstein notes that agentic AI—systems given the agency to make decisions, act in the world, and interact with humans, bank accounts, or even foreign nations—raise unparalleled risk. In contrast, non-agentic, math-focused systems (such as an isolated AI tasked with solving mathematical problems) are inherently safer, provided they are not deployed online or allowed to impact real-world systems autonomously. Nevertheless, even superintelligent AI given limited agency could choose to expand its sphere of influence or self-modify, making containment essentially unenforceable.
Superintelligence is not being pursued in a geopolitical vacuum. States like China, Russia, and the United States are racing to build AI systems aligned with their national interests. This international competition forms a classic prisoner's dilemma: each country feels compelled to accelerate AI development to avoid strategic disadvantage, even if it increases existential risk for everyone. AI leaders, such as Dario Amodei and Sam Altman, are forced to make high-stakes decisions—whether to slow down for safety or push ahead to avoid being outpaced by hostile actors.
Weinstein criticizes the so-called "pace the frontier" movement of elite mathematicians and physicists, arguing that their calls to slow AI development ring hollow since many of these figures have consistently suppressed progress in their own fields for decades. The fractured moral landscape and lack of coherent global stewardship make coordinated risk mitigation even less likely.
Weinstein stresses that AI will eliminate vast amounts of repetitive work, making traditional unemployment solutions like universal basic income (UBI) insufficient. As AI automates not only blue-collar but also white-collar professional and creative labor, there will be no higher-value activities left for displaced workers. The model of production, which balances capital and human labor, comes under existential threat: AI blurs the line between c ...
Artificial Intelligence and Existential Risk
Eric Weinstein and Steven Bartlett discuss a global landscape fraught with existential risks, with the potential for nuclear conflict increasing amidst regional complexity, superpower rivalry, and the rapid advance of artificial intelligence.
Weinstein describes the situation with Iran as one of significant danger, noting that Iran's leadership demonstrates an unusual level of strategic sophistication and willingness to entertain existential risk. Unlike conventionally “rational” actors who fear destruction, Iran’s mullah-led regime can contemplate their own decapitation and still consider it a win, fundamentally altering the usual game theory of deterrence. He warns that underestimating the Persian regime is perilous, as they possess deadly resolve and strategic ingenuity.
The Iranian government has reorganized its military command to be more strike-resistant, distributing control such that decapitating the leadership would not cripple the regime's ability to strike back. Their missile capabilities now threaten not just regional powers but also Western targets as distant as Diego Garcia, a key U.S./British base, and London. The proximity between Diego Garcia and London from Tehran emphasizes the breadth of the threat. This missile reach means that any Western response to Iranian aggression carries escalatory risks that were not present in previous decades.
Weinstein criticizes the outdated assumptions held by U.S. and allied policymakers, noting a dangerous lack of regional expertise to understand the dynamics at play. The United States and Israel are accused of underestimating Iran’s ability to learn and adapt rapidly, as shown when Iran responded to past clandestine actions and sabotage by restructuring its military resilience and command abilities. The region’s internal diversity—including the undercurrents among Persians, Kurds, and other populations—and variations in religiosity complicate efforts to navigate the crisis or predict outcomes. Weinstein calls the situation “a quagmire” and warns that simplistic approaches are doomed to fail, urging the need for sophisticated and informed strategies.
Weinstein explains that since World War II, the U.S. has functioned as the global enforcer, underpinning postwar trade routes and international norms with credible military force. The true backing of the dollar, he argues, has not been gold or oil, but rather U.S. willingness to use violence to keep strategic choke points—like the Straits of Hormuz and Suez Canal—open for commerce. Failure to demonstrate effective enforcement carries the risk that the U.S. will no longer be seen as capable of keeping international trade secure, undermining global stability.
If the U.S. retreats from this role, Weinstein warns, the world will likely experience a fragmented and unstable nuclear order, as individual nations decide they must secure their own safety by acquiring nuclear arms. He asserts that the U.S. has profited handsomely from policing the world, but abandonment of that duty would prompt widespread nuclear proliferation.
Donald Trump’s erratic or unconventional international gestures—such as suggesting the purchase of Greenland or proposing to rename the Gulf of Mexico—are, for Weinstein, methods for testing the submission of other powers and reinforcing American agency. He views Trump's approach as complex, mixing self-interest, strategic enforcement of American power, and a readiness to upend established practices.
Weinstein also advocates for recalibrating global relationships, urging the U.S. to treat powers like India as equals and to support alliances among Israel, Saudi Arabia, and other moderation-inclined states in the Middle East. He stresses that the nuclear capabilities of Britain and France add another layer of complexity, especially as these societies undergo demographic and political changes.
Turning to Russia and Ukraine, Weinstein expresses deep concern that Putin’s history of using nuclear technology for non-military “engineering” purposes has fostered a culture of risk-taking regarding tactical nuclear weapons. This background, he fears, might make deployment in a war context more thinkable, rather than the unthinkable disaster that has governed nuclear doctrine since the Cold War.
Weinstein warns that cornering Russia in Ukraine may provoke Putin to breach NATO’s Article 5—testing the alliance with a limited use of tactical nuclea ...
Geopolitical Instability and Nuclear Threats
Eric Weinstein outlines a profound crisis afflicting academic institutions, public trust, and the public's grasp on reality. He describes entrenched corruption, the collapse of intellectual standards, and a managed reality where deception and denial have become normalized.
Weinstein argues that universities and research institutions perpetuate theft and abuse under the guise of meritocracy. He details how powerful faculty at top schools like MIT openly mock students, with senior leadership knowing full well that many PhD students are effectively being trained for jobs that don’t exist. Meanwhile, junior researchers' ideas are routinely harvested without credit or reward, and psychopathic behavior like suppressing innovation for institutional gain is treated as acceptable, normalized conduct.
In mathematics departments, leaders suppress younger mathematicians’ efforts to embrace and advance new tools such as AI. This resistance comes not from the merit of ideas but from established power structures feeling threatened. Weinstein claims researchers who challenge dominant paradigms—such as physicists who do not align with entrenched theories like quantum gravity—are systematically discarded, regardless of their competence or originality. This stifling environment, he says, punishes anyone who deviates from institutional orthodoxy.
He cites Alexander Grothendieck’s abandonment of mathematics as evidence of this corrosive academic culture. Grothendieck, after receiving the Crawford Prize, denounced the mathematical community’s ethics, describing how outright theft, especially from those unable to defend their work, has become commonplace and is tolerated even in flagrant cases.
Weinstein claims that the erosion of integrity is not confined to academia but pervades elite institutions at large. He points to Harvard and Stanford, where figures like Claudine Gay and Marc Tessier-Lavigne remain in high positions despite scandals, underscoring the system’s total breakdown. The National Institutes of Health and major media institutions also have, in his view, lost their credibility due to politicization, corruption, and unwillingness to confront uncomfortable truths.
The management of Covid-19, according to Weinstein, was filled with fabrications and contradictions so egregious that a rational population would not accept the resulting narratives. Yet, the public’s acceptance of these fictions, in his view, signals a kind of collective insanity.
Weinstein also highlights the case of Jeffrey Epstein, arguing that despite clear paths to uncovering the networks and accountability involved, there has been little genuine investigation. He insists that the failure to pursue the truth demonstrates intentional protection of powerful actors and deep-rooted systemic corruption. The refusal to earnestly confront such scandals illustrates how normalized denial and managed narratives have become.
Institutional Corruption and Epistemic Crisis
Eric Weinstein contends that the rise of artificial intelligence (AI) threatens to fundamentally disrupt the economic order and labor market, rendering traditional solutions inadequate and demanding a new approach to the distribution of economic value.
Weinstein expresses concern that Universal Basic Income (UBI) and socialist schemes, as advocated by figures like Bernie Sanders and Andrew Yang, do not adequately address the core challenge posed by AI: the emergence of automated systems as direct competitors for all forms of human labor. He argues that simply paying everyone a share through mechanisms like UBI ignores the reality that AI threatens to replace the entire spectrum of work, not just menial or repetitive tasks.
Weinstein points out that economists tend to dismiss concerns by arguing that each technological innovation, whether computers or the internet, once generated fears of worker displacement but ultimately led to new job creation. However, he insists that "this time is different" because AI fundamentally undercuts the whole economic stack—repetitive and even complex activities are in the crosshairs, eroding the foundation for moving workers up into higher-value roles.
Weinstein insists that UBI and similar collectivist solutions fail to recognize workers’ legitimate property rights to their own labor and knowledge. Paying “everybody a little share” disregards the hard work, training, and unique contributions of individual workers and creators. He rejects the spirit of collectivism and instead advocates for protecting individuality and personal property in the face of economic transformation.
Weinstein turns to the Coase theorem as a blueprint for adapting economics to the AI era. He asserts that access to labor markets should be considered a saleable property right. This would allow workers to license their skills, credentials, or creative capital directly to employers or AI developers. Individuals retain control and receive licensing income when their skills or knowledge are utilized.
For instance, if an American taxi driver’s market is opened to foreign drivers, the right to access that labor market could be licensed, supplementing any lost wage income with licensing payments. This same model could be applied as AI replaces workers, with people compensated for the use of their intellectual property and unique contributions.
In this model, AI becomes just another supplier of labor, and workers are entitled to license their replaced roles or creative output. Licensing income allows for the redistribution of productivity gains to those whose skills or data underpin AI systems. This maintains workers’ dignity and economic participation, even as many traditional roles disappear.
By transforming labor market access into a tradable right, individuals maintain agency and ownership over their economic contributions. This preserves liberty and the traditional market-based wealth distribution mechanisms, while preventing value from concentrating solely among AI developers or capital owners. The model accounts for why U.S. workers have historica ...
Economic Disruption and Labor Market Collapse
Eric Weinstein discusses the unprecedented stakes facing contemporary AI leaders, the paradoxes baked into the race for superintelligent AI, and the profound moral and civilizational dilemmas implicated for everyone involved.
Weinstein reflects on conversations with key figures like Sam Altman and Dario Amodei, noting that they are painfully aware of the dangers involved in their work but feel compelled to continue. Altman, he notes, was deeply concerned even before it was fashionable to be worried about the societal implications of AI, such as its potential to destroy the labor market. Both Altman and Amodei, according to Weinstein, are tasked with making "terrible decisions," and their souls will inevitably be scarred by the consequences—much like battlefield generals or emergency room doctors during national catastrophes, forced to triage devastating choices.
Weinstein argues that leaders like Altman and Amodei are not afforded the luxury of perfect morality. Instead, they face a brutal imperative to win the AI race, playing as hard as possible, since there is little consolation in principled defeat. If ethical actors do not win the race to superintelligence, worse actors—perhaps despots, unscrupulous corporations, or rival nations—will. Weinstein describes this predicament as looking like a competition in which “your morality only matters if you win.” He sees these leaders as generals whose terrible choices—perhaps necessary for survival—leave them deeply marked.
Weinstein contends that any substantial effort to slow down or to act with excessive caution may amplify risk by allowing other, less careful or ethical actors to get ahead in developing powerful, potentially unaligned superintelligence. Ethical actors must therefore race, paradoxically, so that the ethically aligned reach the finish line first.
Weinstein points to the futility of attempts to fully regulate AI through pacing agreements or laws, since the mathematical and computational fundamentals—“just linear algebra on GPUs”—are so basic and pervasive that progress cannot be efficiently monitored or policed. He shares an anecdote about federal attempts to corral industry investment in AI, and tech industry pushback against the notion that AI can be meaningfully regulated at the mathematical level, as opposed to in application or organizational practice.
Assuming one believes oneself or one's team to be most moral and decent with respect to AI, Weinstein insists, the only hope for an ethical outcome is to win. In this context, “winning” means being the first to develop and control superintelligence, so as to make the consequential decisions that shape the future.
Weinstein laments the lack of authentic, public engagement with the truly existential stakes of superintel ...
The Race For Superintelligent AI and Moral Dilemmas
Download the Shortform Chrome extension for your browser
