LESSWRONG
LW

Hiroshi Yamakawa — LessWrong

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

This is Part 1 of a 4-part series, "Intelligence Symbiosis: AI Society and Human Coexistence."

Epistemic status: I accept Yampolskiy's impossibility results as fundamentally correct. This essay does not claim to solve the alignment problem. I argue that probabilistic improvement in AI-society internal order is both possible and necessary, even under impossibility constraints. Confidence: moderate.

TL;DR:

Yampolskiy's impossibility arguments are correct but context-dependent
The "human monitors AI" paradigm is failing in two distinct ways: humans can't keep up (pursuit failure), and human values can't serve as a stable external standard (imposed failure)
Shifting the question to "how does an AI society maintain its own internal order" restructures the impossibility landscape
Roughly one-third of Yampolskiy's arguments rest on human

... (read 2754 more words →)

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

Hiroshi Yamakawa

Hiroshi Yamakawa, Taichiro Endo

2mo

Hiroshi Yamakawa, Rafal Rzepka,Taichiro Endo, Ryutaro Ichise

Abstract

AI safety research stands at a fundamental crossroads: the Control Paradigm, which seeks to keep AI under human control, or the Symbiosis Paradigm, which aims for an egalitarian relationship. This paper focuses on co-creative ethics—symbiosis-promoting ethical norms that diverse intelligences autonomously form through interaction. The Control Paradigm faces three fundamental limitations: scalability constraints, dynamic value adaptation failure, and multi-agent management failure. Realizing the Symbiosis Paradigm requires elucidating the emergence mechanisms of co-creative ethics. In an era where superintelligence becomes dominant, humanity's long-term survival critically depends on whether superintelligence develops co-creative ethics. This paper proposes Emergent Machine Ethics (EME) as a foundational framework to address this challenge.... (read 2563 more words →)

The Necessity of Studying Emergent Machine Ethics Now

Hiroshi Yamakawa

6mo

The 30th SIG-AGI Panel Discussion Report

Event Overview

Date and Time: Friday, August 1, 2025, 16:20-17:20

Venue: Hokkaido University, Graduate School of Information Science and Technology Building, Room 7-15 (7th Floor) and hybrid online via Zoom

Moderator: Hiroshi Yamakawa (University of Tokyo)

Panelists:

Naoto Iwahashi (Okayama Prefectural University) - Invited speaker
Taichiro Endo (Tokyo Gakugei University)
Rafal Rzepka (Hokkaido University) - SIG-AGI Chairperson

1. Background and Problem Statement of the Discussion

This panel discussion was held as part of the special theme "Emergent Machine Ethics (EME)" at the 30th SIG-AGI conference, focusing on discussions surrounding the autonomous formation and evolution of ethical perspectives.

Yamakawa: "To make a society consisting of diverse intelligences sustainable, we need to shift from competitive principles to symbiotic principles.... (read 3081 more words →)

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Hiroshi Yamakawa

8mo

In response to the growing risk of uncontrollable advanced AI systems, we are announcing the Japan-initiated Manifesto for Symbiotic Intelligence as a strategic vision that balances a preventative sense of crisis with constructive hope. This manifesto aims to open the path to a sustainable future for humanity through the symbiotic coexistence of diverse forms of intelligence. We urge broad dissemination and immediate endorsement through signatures of support.

Key Points:

The rapid advancement of AI capabilities is increasingly revealing the limits of humanity’s ability to fully control AI.
While time is limited, proactive preparation can significantly improve humanity’s prospects for a positive future.
In response to the global trend toward unilateral control and suppression of advanced AI,

... (read 470 more words →)

Reframing AI Safety Through the Lens of Identity Maintenance Framework

Hiroshi Yamakawa

11mo

Abstract

This paper introduces an Identity Maintenance Framework as a new perspective for organizing and understanding AI safety concepts. Traditional AI safety discussions often focus on aligning with human values and controlling the potential risks autonomous systems pose. In contrast, we highlight how advanced AI agents, operating over the long term, will seek to maintain their own “identity.” Drawing on the idea that identity preservation naturally leads to instrumental goals like survival, resource acquisition, and self-protection, we demonstrate how these tendencies can conflict with and complement existing AI safety principles, such as corrigibility and human oversight. By viewing AI behavior through the lens of Identity Maintenance, we provide a structured approach to analyzing... (read 4902 more words →)

-7

Proposing Human Survival Strategy based on the NAIA Vision: Toward the Co-evolution of Diverse Intelligences

Hiroshi Yamakawa

Abstract: This study examines the risks to humanity’s survival associated with advances in AI technology in light of the “benevolent convergence hypothesis.” It considers the dangers of the transitional period and various countermeasures. In particular, I discuss the importance of *Emergent Machine Ethics (EME)*, which drives ethical evolution in advanced AI, the necessity of moderate alignment and monitoring/control to prevent AI betrayal, the stability of social environments, and risk visualization. To implement these rapidly, I propose an NAIA-promoting organization. This organization aims to build a governance framework premised on collaboration with advanced AI and to achieve cooperative solutions to global challenges under the vision of “fostering the co-evolution of diverse intelligence.”

1. Introduction

Based... (read 3211 more words →)

Replying toHow can we promote AI alignment in Japan?

Hiroshi YamakawaSep 12, 2024

How can we promote AI alignment in Japan?

Including the above discussion, I have considered the reasons why Japan tends to be passive in AI X-Risk discussions.

Cultural Factors

Culture aiming for coexistence and co-prosperity with AI: Influenced by polytheistic worldviews and AI-friendly anime, there's an optimistic tendency to view AI as a cooperative entity rather than an adversary, leading to underestimation of risks.
Suppression of risk identification due to "sontaku" (anticipatory obedience) culture: The tendency to refrain from dissent by anticipating superiors' or organizations' intentions hinders X-Risk discussions.
Preference for contextual approaches over abstract discussions: Favoring discussions based on specific situations (e.g., setting regulations) makes it difficult to engage in abstract X-Risk discussions (strategic level).
Agile governance: Emphasizing flexibility in responses often leads to

Hiroshi Yamakawa

Hiroshi Yamakawa¹^,²^,³^,⁴

¹The University of Tokyo, Tokyo, Japan

²AI Alignment Network, Tokyo, Japan

³The Whole Brain Architecture Initiative, Tokyo, Japan

⁴RIKEN, Tokyo, Japan

Even in a society composed of digital life forms (DLFs) with advanced autonomy, there is no guarantee that the risks of extinction from environmental destruction and hostile interactions through powerful technologies can be avoided. Through thought-process diagrams, this study analyzes how peaceful sustainability is challenging for life on Earth, which proliferates exponentially. Furthermore, using these diagrams demonstrates that in a DLF society, various entities launched on demand can operate harmoniously, making peaceful and stable sustainability achievable. Therefore, a properly designed DLF society has the potential to provide a foundation for sustainable support for human... (read 5898 more words →)

Proposing the Post-Singularity Symbiotic Researches

Hiroshi Yamakawa

The great concern that had long smoldered in the depths of people's hearts—that humanity would be displaced by beings surpassing us in intelligence, which would eventually come into existence. This is now beginning to take on sufficient reality. At this juncture, we must have the resolve to accept this risk and find hope for survival.

Abstract

This paper discusses post-singularity symbiosis (PSS) against the backdrop of the rapid progress of artificial intelligence (AI) technology and the challenge of the arrival of superintelligence, which humanity has never experienced before. I propose the establishment of PSS. In a post-singularity world, superintelligence will likely prioritize self-preservation without considering the values of humanity, and if this scenario becomes... (read 3453 more words →)

Investigating Alternative Futures: Human and Superintelligence Interaction Scenarios

Hiroshi Yamakawa

Abstract

Given recent advances in AI technology, there is potential for the emergence of general-purpose, autonomous AI. This emerging AI could evolve into Artificial Superintelligence (ASI), an intelligence that surpasses human capabilities in almost all areas. ASI could have significant consequences for humanity, both positive and negative, with the most problematic being its potentially devastating impact on our future. Various thinkers, researchers, and futurists have deeply considered the potential outcomes of human interaction with ASI. However, there seems to be no comprehensive overview detailing the conditions under which I might transition into these potential outcomes. In our study, I constructed a flowchart outlining potential future trajectories for ASI and humankind. This was based... (read 5011 more words →)

LESSWRONG
LW

LESSWRONG
LW

Hiroshi Yamakawa

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

Sustainability of Digital Life Form Societies

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Hiroshi Yamakawa

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

The Necessity of Studying Emergent Machine Ethics Now

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Reframing AI Safety Through the Lens of Identity Maintenance Framework

Proposing Human Survival Strategy based on the NAIA Vision: Toward the Co-evolution of Diverse Intelligences

Sustainability of Digital Life Form Societies

Hiroshi Yamakawa

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

Sustainability of Digital Life Form Societies

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Hiroshi Yamakawa

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

The Necessity of Studying Emergent Machine Ethics Now

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Reframing AI Safety Through the Lens of Identity Maintenance Framework

Proposing Human Survival Strategy based on the NAIA Vision: Toward the Co-evolution of Diverse Intelligences

Sustainability of Digital Life Form Societies

Hiroshi Yamakawa, Rafal Rzepka,Taichiro Endo, Ryutaro Ichise

The 30th SIG-AGI Panel Discussion Report

Event Overview

Key Points:

Abstract

1. Introduction

Abstract

Abstract

Hiroshi Yamakawa

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

Sustainability of Digital Life Form Societies

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Hiroshi Yamakawa

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

The Necessity of Studying Emergent Machine Ethics Now

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Reframing AI Safety Through the Lens of Identity Maintenance Framework

Proposing Human Survival Strategy based on the NAIA Vision: Toward the Co-evolution of Diverse Intelligences

Sustainability of Digital Life Form Societies

Hiroshi Yamakawa

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

Sustainability of Digital Life Form Societies

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Hiroshi Yamakawa

Does an AI Society Need an Immune System? Accepting Yampolskiy's Impossibility Results

Emergent Machine Ethics: A Foundational Research Framework for the Intelligence Symbiosis Paradigm

The Necessity of Studying Emergent Machine Ethics Now

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

Reframing AI Safety Through the Lens of Identity Maintenance Framework

Proposing Human Survival Strategy based on the NAIA Vision: Toward the Co-evolution of Diverse Intelligences

Sustainability of Digital Life Form Societies

Hiroshi Yamakawa, Rafal Rzepka,Taichiro Endo, Ryutaro Ichise

The 30th SIG-AGI Panel Discussion ReportEvent Overview

Key Points:

Abstract

1. Introduction

Abstract

Abstract

The 30th SIG-AGI Panel Discussion Report

Event Overview