Nexus

1. Information is about connection, not just representation

Information doesn't necessarily inform us about things. Rather, it puts things in formation.

Information creates realities. Throughout history, information has been used not just to represent existing realities, but to create new ones by connecting disparate elements. This is true from DNA in biology to religious texts in human societies. Information technologies like writing, printing, and now computers have dramatically expanded our ability to create and maintain large-scale cooperative networks.

• Examples of information creating realities:
  • DNA connecting cells into organisms
  • Religious texts connecting believers into communities
  • Legal documents connecting individuals into nations
  • Computer code connecting devices into networks

Understanding information as a connective force helps explain phenomena like the power of myths, the rise of bureaucracies, and the potential impact of AI on human societies.

2. Stories and myths shape human cooperation on a massive scale

For tens of thousands of years, Sapiens built and maintained large networks by inventing and spreading fictions, fantasies, and mass delusions—about gods, about enchanted broomsticks, about AI, and about a great many other things.

Myths enable large-scale cooperation. Humans are unique in their ability to create and believe in shared fictions, which allows for cooperation beyond small kinship groups. These shared myths - whether religious, national, or ideological - have been crucial in building large societies and civilizations.

• Functions of shared myths:
  • Create common identities
  • Establish shared goals
  • Justify social hierarchies
  • Provide meaning and purpose

However, while myths are powerful tools for cooperation, they can also be sources of conflict and oppression when different groups hold incompatible beliefs. The challenge for modern societies is to harness the connective power of shared stories while maintaining flexibility and tolerance for diverse perspectives.

3. Bureaucracy and documents centralized power but limited privacy

Documents, archives, forms, licenses, regulations, and other bureaucratic procedures have changed the way information flows in society, and with it the way power works.

Bureaucracy changed information flow. The rise of written documents and bureaucratic systems dramatically altered how information was collected, stored, and used in societies. This shift centralized power in new ways, but also created new limitations on privacy and individual autonomy.

• Key changes brought by bureaucracy:
  • Centralization of information in archives
  • Creation of standardized forms and procedures
  • Rise of specialized bureaucratic classes
  • Increased state capacity for taxation and control

While bureaucracy enabled more complex and efficient governance, it also created new forms of oppression and surveillance. The tension between bureaucratic efficiency and individual privacy remains a central challenge in the digital age.

4. Holy books and infallible sources often lead to human control

The attempt to invest all authority in an infallible superhuman technology led to the rise of a new and extremely powerful human institution—the church.

Infallibility breeds control. Throughout history, attempts to create infallible sources of truth - whether holy books, ideologies, or AI systems - have often led to increased human control rather than liberation from error. This paradox arises because infallible sources still require human interpretation and implementation.

• Examples of "infallible" sources leading to human control:
  • Religious texts and priestly classes
  • Political ideologies and party elites
  • AI systems and tech companies

The lesson for the age of AI is that claims of algorithmic infallibility should be treated with skepticism, and human oversight and interpretation will remain crucial even as AI systems become more powerful.

5. Democracy relies on distributed information and self-correction

Democracy is a conversation with numerous participants, many of them talking at the same time. It can be hard to follow such a conversation.

Democracy is messy but resilient. Democratic systems are characterized by distributed information flows and strong self-correcting mechanisms. While this can make them appear chaotic or inefficient compared to more centralized systems, it also makes them more adaptable and resistant to catastrophic errors.

Key features of democratic information systems:

• Multiple independent information channels
• Freedom of speech and press
• Separation of powers
• Regular elections and peaceful transfers of power

The challenge for modern democracies is to maintain these distributed and self-correcting features in the face of new technologies that enable unprecedented centralization and control of information.

6. Computers are changing the nature of information networks

Unlike printing presses and radio sets, which were passive tools in human hands, computers are already becoming active agents that escape our control and understanding and that can take initiatives in shaping society, culture, and history.

Computers are active agents. The rise of computers and AI represents a fundamental shift in the nature of information networks. Unlike previous technologies, computers can make decisions and create new ideas independently of human input.

• Key differences between computers and previous technologies:
  • Ability to process vast amounts of data
  • Capacity for machine learning and self-improvement
  • Potential for autonomous decision-making
  • Creation of inter-computer realities

This shift raises profound questions about human agency, control, and the future of social and political systems built on human decision-making.

7. AI's alignment problem poses existential risks to humanity

The alignment problem turns out to be, at heart, a problem of mythology.

Aligning AI with human values is crucial. As AI systems become more powerful, ensuring that their goals and actions align with human values and interests becomes an existential challenge. This "alignment problem" is complex because it involves not just technical issues, but fundamental questions about human values and the nature of consciousness.

• Key aspects of the alignment problem:
  • Defining human values in machine-readable terms
  • Ensuring AI systems pursue intended rather than literal goals
  • Maintaining human control over increasingly autonomous systems
  • Addressing potential conflicts between AI and human interests

Solving the alignment problem may be the most important challenge facing humanity in the coming decades, as misaligned AI systems could pose existential risks even if they are not explicitly hostile to humans.

8. Computer biases can perpetuate and amplify human prejudices

AIs too have childhood experiences. Algorithms might even infect one another with their biases, just as humans do.

AI can amplify biases. While AI systems are often presented as objective and unbiased, they can actually perpetuate and amplify human prejudices present in their training data. This occurs because AI systems learn patterns from existing data, which often reflects historical biases and inequalities.

Examples of AI bias:

• Facial recognition systems performing poorly on darker skin tones
• Resume screening algorithms favoring male candidates
• Predictive policing systems targeting minority neighborhoods

Addressing AI bias requires not just technical solutions, but a deep examination of the social and historical contexts in which AI systems are developed and deployed.

9. Democracy faces new challenges in the age of AI and automation

If three years of high unemployment could bring Hitler to power, what might never-ending turmoil in the job market do to democracy?

AI disrupts democratic stability. The rapid pace of technological change, particularly in AI and automation, poses significant challenges to democratic systems. These challenges include potential mass unemployment, increased economic inequality, and the manipulation of public opinion through AI-driven information systems.

Potential threats to democracy from AI:

• Job displacement leading to social unrest
• Concentration of wealth and power in tech companies
• AI-powered surveillance and social control systems
• Manipulation of elections through targeted misinformation

Addressing these challenges will require new forms of democratic governance and economic organization that can adapt to rapid technological change while preserving core democratic values.

10. Preserving human agency is crucial as AI becomes more powerful

To guard against a plethora of unforeseeable problems, our best bet is to create living institutions that can identify and respond to the threats as they arise.

Human oversight remains essential. As AI systems become more powerful and autonomous, preserving human agency and oversight becomes increasingly crucial. This requires not just technical safeguards, but the development of new social and political institutions capable of governing AI systems.

Key principles for preserving human agency:

• Maintaining human control over critical decisions
• Ensuring transparency and accountability in AI systems
• Developing robust ethical frameworks for AI development
• Investing in human education and adaptability

The ultimate challenge is to harness the immense potential of AI while ensuring that it remains a tool for human flourishing rather than a threat to human autonomy and existence.

Last updated: September 10, 2024