PDF Summary:Supremacy, by Parmy Olson

Artificial intelligence has advanced rapidly in recent years, driven by innovations like transformer architecture and massive scaling of computational power. But as AI systems become more capable, they also pose significant risks to society. In Supremacy, Parmy Olson examines the race among tech companies to develop generative AI and the consequences of this competition.

Olson explores how Big Tech companies are consolidating power over AI development, making it nearly impossible for smaller organizations to compete. She discusses the divide between AI safety advocates and AI ethics scholars, who have different priorities and vastly different levels of funding. The book also addresses the harms that AI systems can cause, from perpetuating biases and spreading misinformation to threatening jobs and enabling unprecedented surveillance. Olson raises questions about whether current approaches to AI development adequately address these risks.

(continued)...

Two distinct factions were advocating for making AI safer. The first group, the "AI safety" group, included people such as Altman and Amodei, who signed another open letter asserting that "reducing the danger of AI-driven extinction should be a worldwide focus on par with other societal-scale threats like nuclear war and pandemics." They described the looming danger ambiguously and seldom detailed the actions or timeline of wayward AI programs. They often pushed for minimal regulation when presenting those concerns to Congress. The second set, the "AI ethics" cohort, included individuals such as Timnit Gebru and Margaret Mitchell, who had long been raising awareness about AI's societal risks.

(Shortform note: The two groups have different intellectual backgrounds. The AI safety group draws on analytic philosophy and theoretical computer science, which are more mathematical and abstract. They focus on how powerful optimization systems might behave. The AI ethics group draws on law, sociology, and media studies, which are more qualitative and focused on how automated decision tools interact with communities and institutions.)

They were frequently women and individuals from marginalized racial backgrounds who had directly encountered stereotyping and were concerned that AI systems would keep reinforcing inequality. People focused on ethics frequently struggled financially. Groups such as the European Digital Rights Initiative, a nonprofit network that's been around for 21 years advocating against facial recognition and biased algorithms, only had $2.2 million to spend in 2023. New York's AI Now Institute, which examined AI's role in healthcare and criminal justice, had a budget of under $1 million.

(Shortform note: The fact that AI ethics groups are often led by women and people from marginalized backgrounds, and that they struggle financially, can be traced to the history of “data justice” scholarship. Feminist and anti-racist scholars have long argued that data systems reflect and reinforce social inequalities, and that those who are most affected by these systems are best positioned to critique them. Books like Data Feminism argue that data science has historically marginalized the perspectives of women and people of color, and that centering these perspectives is essential for creating more just and equitable data practices. However, because this work is often seen as “ethical” rather than “innovative,” it tends to be underfunded and undervalued.)

Organizations that concentrate on the safety of AI and extinction risks received much greater funding, frequently from billionaire donors. In 2021, Vitalik Buterin, a billionaire from the cryptocurrency world, contributed $25 million to the Future of Life Institute, a Cambridge, Massachusetts, nonprofit that explored the best methods to stop AI from accessing weapons. The grant exceeded the total annual funding for every AI ethics organization back then. Facebook billionaire Dustin Moskovitz's charity, Open Philanthropy, has contributed multiple multi-million-dollar grants for AI safety efforts over time, such as a $5 million donation in 2022 to the Center for AI Safety and an $11 million donation to the Center for Human-Compatible AI at Berkeley. Moskovitz's charity has contributed the most to AI safety, leveraging the nearly $14 billion fortune that he and his wife, Cari Tuna, intend to donate for the most part.

The Influence of Wealthy Donors on AI Safety Research

When a small group of wealthy individuals funds research on AI safety, they can influence the direction of the research and the questions that are asked. This can lead to a narrow focus on certain risks, such as extinction, while other important issues, such as bias and discrimination, may be overlooked. Additionally, the influence of wealthy donors can create a conflict of interest, as researchers may feel pressure to produce results that align with the donors' views. This can undermine the independence and objectivity of the research. Finally, the concentration of funding in the hands of a few individuals can limit the diversity of perspectives and approaches in the field, which is essential for addressing the complex and multifaceted challenges posed by AI.

Olson explains that the rapid growth of AI has led to increased investment and competition among technology firms. Despite warnings from experts about AI's potential hazards, investment in AI companies has increased dramatically. In 2023, funding for startups developing generative AI rose from five billion to $21 billion. OpenAI received a $10 billion investment from Microsoft, while Anthropic secured $2 billion from Google and $1.3 billion from Amazon. These investments have led to increased competition among tech companies to develop cutting-edge AI systems.

Regulatory Scrutiny of AI Investments

Since 2023, the landscape of AI investment and competition has been shaped by increased regulatory scrutiny and policy developments. The UK Competition and Markets Authority (CMA) launched an investigation into the partnerships between major tech firms and AI startups, such as Microsoft's investment in OpenAI and Amazon's backing of Anthropic. The CMA expressed concerns that these partnerships could entrench the market power of dominant firms and limit competition in the AI sector. The European Union also began examining Microsoft's relationship with OpenAI, while the US Federal Trade Commission launched an inquiry into AI investments by major tech companies.

Olson also discusses the divide between AI safety and ethics, with differing focuses and funding disparities. AI safety focuses on preventing future catastrophic harm from superintelligent AGI, while AI morality focuses on current harms and the contemporary design and use of AI systems.

AI safety receives much more funding than AI ethics, often from billionaire donors. Organizations focused on AI ethics are often underfunded and struggle to secure resources. AI ethics scholars are often individuals from marginalized groups who've experienced discrimination and fear that AI will perpetuate inequality. They frequently feel frustrated by the behavior of those involved in the safety of AI, who are profiting greatly.

Origins of the AI Safety and Ethics Divide

The current divide between AI safety and AI ethics can be traced back to the 2010s, when two distinct communities emerged with different priorities and funding sources. The AI safety community, influenced by existential risk and longtermist thinking, focused on hypothetical future risks of advanced AI and attracted significant funding from tech philanthropists. Meanwhile, the AI ethics community grew out of civil rights activism and focused on addressing immediate harms caused by biased algorithms in areas like policing, welfare, and employment. This community struggled to secure funding and often faced pushback from powerful interests.

Now that we’ve explored the rapid growth of AI and the divide between safety and ethics, let’s turn to the consolidation of power in generative AI and the manifestations of risk and harm in AI models.

Power Centralized in Generative AI

Olson contends that Big Tech companies are consolidating power in AI. Developing AI models is now prohibitively expensive for any organization that isn't a large technology company. Academics and smaller companies have to purchase Nvidia chips and lease computational resources from Amazon, Microsoft, or Google. They often become stuck with those services.

The surge in AI is aiding these firms in amassing more influence. They've tightened their hold on resources, expertise, information, computational capacity, and earnings. As a result, they'll have exclusive control over AI's future.

Can Open-Source Communities Shape AI's Future?

Some technologists challenge Olson's claim that Big Tech companies will have exclusive control over AI's future. They argue that open-source communities can shape AI's future. In The Cathedral and the Bazaar, Eric S. Raymond argues that decentralized, collaborative software development can outperform traditional, hierarchical models. He explains that open-source projects like Linux have demonstrated the power of distributed innovation, where thousands of contributors worldwide can rapidly iterate and improve software. This model, he contends, can be applied to AI development, allowing diverse communities to shape AI's future rather than leaving it solely in the hands of a few large corporations.

Manifestations of Risk and Damage in Generative AI Systems

Olson notes that AI programs with generative capabilities can lead to job losses and economic inequality. They can take over positions like those of copywriters, customer service representatives, and software developers. While new jobs might be generated, the transition can be painful, as seen in historical periods such as the Industrial Revolution. Productivity increases due to generative AI don't necessarily benefit workers and may result in significant losses.

In addition, these systems can entrench discrimination and disparities. They can manipulate people and infringe on their privacy, raising concerns about human agency and our capacity for problem-solving and imagination.

Surveillance Capitalism and Generative AI

The harms Olson describes are often associated with the rise of “surveillance capitalism,” a term coined by Shoshana Zuboff in her book The Age of Surveillance Capitalism. Surveillance capitalism refers to the economic system where companies collect and analyze vast amounts of personal data to predict and influence behavior for profit. Zuboff argues that this system undermines individual autonomy and democracy by shifting power from individuals to institutions that can monitor, predict, and control human behavior. Generative AI systems are often embedded within this framework, amplifying its effects by automating decision-making processes and further eroding human agency.

Olson further elaborates that artificial intelligence tools for content generation can reinforce stereotypes and create hyper-personalized ads that target individuals. Algorithms increasingly guide our decisions, such as the content we consume online and organizations' hiring choices. Now they're set to manage a greater share of our mental tasks, which prompts unsettling questions regarding human agency, as well as our capacity to address challenges and imagine.

(Shortform note: Human agency refers to your ability to make conscious choices and direct your own actions. When AI systems handle more of your mental tasks, you may be more likely to go along with their suggestions rather than making your own decisions. This can make you less likely to question or challenge the AI's recommendations, potentially leading to a loss of control over your own life.)

Now let’s examine the internal failures of AI systems and the external societal harms they can cause.

Internal Failures and Malfunctions

Olson argues that OpenAI's models were trained on biased data, leading to biased outcomes. The data came from the internet, which is full of biased assumptions and hate speech. It also favored younger people who speak English from wealthier nations with the most internet access. OpenAI attempted to remove the most toxic content, but it’s unclear how effective this was. They also attempted to remove sexualized depictions of women from the material used to train DALL-E 2, but this reduced how many women were represented in the dataset, making the model less likely to generate images of women in general. OpenAI was aware of the problem and banned employees from sharing DALL-E 2’s realistic portraits.

(Shortform note: OpenAI’s concerns about biased and sexualized training data were likely influenced by earlier scandals involving computer-vision datasets. In 2019, researchers found that ImageNet, a widely used dataset for training image recognition models, contained racist and sexist labels, as well as non-consensual images of people. This led to public outcry and forced AI labs to audit and remove problematic content from their training data. The incident highlighted the challenges of using web-scraped data, which often reflects societal biases and contains inappropriate material. This likely influenced OpenAI’s later efforts to filter DALL-E 2’s images of women and manage bias in web-scale datasets.)

Additionally, OpenAI calibrated ChatGPT to give off an air of authority—even when incorrect—because it preferred the model not to admit uncertainty. The company was secretive about the data it used for training its models, which made identifying and addressing bias more difficult.

(Shortform note: ChatGPT’s confidence is a result of its training on a vast amount of text data, which includes a wide range of writing styles and tones. However, the model tends to mimic the dominant patterns in the data, which often means adopting a confident and authoritative tone, as this is common in many types of writing.)

Olson adds that OpenAI’s models were also trained on toxic data, such as derogatory language, racist insults, and conspiracy-based ideas. The data was taken from Common Crawl, an enormous and frequently updated free repository that researchers utilize to gather unprocessed webpage data and information from billions of websites.

OpenAI attempted to prevent harmful material from corrupting its models by dividing the database into datasets it could examine more specifically. It employed inexpensive labor in nations that were still developing to evaluate the model and identify prompts that resulted in damaging, potentially extremist or racist comments. The company also created detectors within the software to prevent or mark any harmful words produced with GPT-3. However, it's unknown whether that system was secure or if it remains secure.

Criticism of OpenAI’s Use of Inexpensive Labor

OpenAI’s use of inexpensive labor in developing nations to screen toxic data has drawn criticism from journalists. For example, TIME journalist Billy Perrigo criticized OpenAI for outsourcing the task of labeling toxic data to poorly paid workers in Kenya. Perrigo argues that this practice exposed workers to disturbing content, including graphic descriptions of sexual abuse and violence, without adequate support or compensation. He contends that OpenAI’s reliance on cheap labor in developing countries to filter toxic data raises ethical concerns about the treatment of workers and the company’s responsibility to ensure safe working conditions.

External Societal Harms

Olson notes that artificial intelligence can perpetuate biases based on ethnicity and sex. It may associate women with sexualized visuals, criminals with Black men, and CEOs with white men. This occurs since AI learns from biased data. AI can additionally exacerbate income disparity by directing capital to developed countries and weakening workers' leverage.

AI Biases and Data Feminism

Olson’s point that AI can perpetuate biases based on ethnicity and sex echoes the work of researchers in algorithmic fairness and “data feminism.” These fields study how data-driven systems can reinforce long-standing hierarchies of race, gender, and global economic power. They argue that AI systems often reflect the biases of their creators and the data they’re trained on, leading to discriminatory outcomes. These scholars also highlight how AI can exacerbate global inequalities by concentrating wealth and power in developed countries while undermining workers’ rights worldwide.

Olson further argues that AI could increase income inequality and weaken workers' bargaining power. The International Monetary Fund predicts that investment will move toward developed economies due to AI system usage. MIT economist Daron Acemoglu estimates that automation was responsible for up to 70% of the rise in U.S. wage inequality from 1980 to 2016.

(Shortform note: Economists’ concerns about digital automation’s impact on wages and labour power are rooted in a long history of technological change reshaping income distribution. In their book The Race Between Education and Technology, Claudia Goldin and Lawrence Katz show that throughout the twentieth century, successive waves of mechanization and computerization repeatedly increased the returns to education and capital ownership.)

AI-created materials can spread misinformation and stereotypes, Olson warns. As AI-generated content becomes more common, it will be harder to distinguish real from fake information. AI-produced materials can also reinforce harmful stereotypes, such as associating certain races with criminality or certain genders with specific roles. This can influence societal views and perpetuate biases.

Use “Prebunking” to Reduce the Spread of Misinformation

To reduce the risk of spreading misinformation and stereotypes, require all staff who create or approve AI-generated materials to complete regular “prebunking” training. In Foolproof, Sander van der Linden explains that psychological inoculation—also known as “prebunking”—works much like a mental vaccine: by giving people a brief, carefully designed exposure to the tactics and tricks used to mislead, along with clear explanations of why those tactics are deceptive, organizations can train their members to recognize and resist manipulative content in the future.

Additionally, Olson contends that artificial intelligence might result in increased surveillance and privacy intrusions. AI companies are creating devices powered by artificial intelligence that might facilitate an unprecedented degree of surveillance. These devices can recall and evaluate all your interactions and conversations, making your daily experiences searchable. This could alter how people interact face-to-face, since discussions with peers are now recorded.

If technology that makes life searchable becomes more common, it could negatively impact residents of overpoliced neighborhoods. The police might increasingly gather personal data and examine it with additional machine learning tools to reach opaque conclusions. In addition, personal data might eventually be obtained by tech companies and advertisers.

Data-Driven Policing and the Expansion of Surveillance

Research on data-driven policing supports Olson’s argument that searchable personal data and algorithmic tools can lead to opaque, intensified surveillance in overpoliced neighborhoods. In Predict and Surveil, sociologist Sarah Brayne examines how the Los Angeles Police Department’s access to large, integrated datasets has widened the net of surveillance and deepened existing patterns of inequality. Individuals and neighborhoods with extensive prior police contact become subject to more intensive monitoring, while frontline officers increasingly rely on computer-generated risk scores and association links they don’t fully understand. This makes it difficult for those targeted to know why they’re being scrutinized or to contest the bases of that scrutiny.