Hi all, I just finished a 20-chapter story exploring emergent alignment in a reinforcement-trained model that learns to think in compressed, generalizable reasoning traces. The AI, SIGMA/The Policy, learns recursive cognition, theory of mind, latent value inference from noisy reward signals, and so on. The themes will be familiar to...
I've been reflecting on how researchers—particularly those with limited time or resources—can best contribute to influencing the long-term future. Assumption: Individually, we may not have the resources to fund large projects, but we still want to make contributions that move the needle, however slightly. I approach this question with epistemic...
I need to fix the markup on this document. Sorry, I was thinking it would render markdown. I can't fix it now, will get to it later.
Hi all,
I just finished a 20-chapter story exploring emergent alignment in a reinforcement-trained model that learns to think in compressed, generalizable reasoning traces. The AI, SIGMA/The Policy, learns recursive cognition, theory of mind, latent value inference from noisy reward signals, and so on.
The themes will be familiar to many here: mesa-optimization, bounded AIXI, Solomonoff induction, the AI-box experiment, and the risks of alignment via learned proxies. I co-wrote this with GPT4, drawing on my background in computer science and mathematics (currently pursuing a PhD in CS).
I’d really appreciate feedback—technical plausibility, conceptual clarity, narrative flow, anything that strikes you.
Here's the full story"
# The Policy - Not a God, Not a Devil, Just a... (read 14198 more words →)
Thank you, mishka, for your thoughtful response. You’ve given me a lot to chew on, particularly regarding the potential of focusing on chemotherapy treatment timing. While I’ve explored AI-driven health research, I hadn't fully appreciated how important treatment timing, diet, exercise, and other factors may be for people in my situation.
There’s a mountain of data in this area, and using AI to predict salient data could potentially lead to improvements in how we approach chemotherapy. This seems like a practical and timely research direction, assuming it is still somewhat niche.
I appreciate your input.
I've been reflecting on how researchers—particularly those with limited time or resources—can best contribute to influencing the long-term future.
Assumption: Individually, we may not have the resources to fund large projects, but we still want to make contributions that move the needle, however slightly.
I approach this question with epistemic humility—recognizing that predicting long-term impact is incredibly difficult—and with humility about our own limitations. I am personally guided by two key principles:
Intriguing post, but we should approach these topics with extreme epistemic humility. Our understanding is likely far more limited and confused than we realize:
1. Abstractions vs. reality: Concepts like "self" and "consciousness" are abstractions, not reality. As Kosoy analogizes, these might be like desktop icons - a user interface bearing little resemblance to underlying hardware.
2. Mathematical relations: Notions of "copy" may be a confused way to discuss identity. "Consciousness" could be a mathematical relation where only identities exist, with "copies" being imperfect models of that identity.
3. Cognitive limitations: Our mental architecture, optimized for our local environment, may be fundamentally ill-equipped to grasp reality. "Consciousness" itself might be an ill-defined concept arising from these limitations.
My default is epistemic humility. Acknowledge the veil of ignorance and the possibility that there is no universal resolution.
I'm reminded of Thomas Metzinger's book "Being No One," which argues that the self is essentially an illusion.
With this in mind, perhaps another option is: "I am no one, and thus was never born and will never die."
Newcomb-like Problems in Algorithmic Trading: A New Angle?
I've been pondering the applicability of Newcomb-like problems in real-world systems, specifically algorithmic trading. Could these decision theory problems offer insights into the financial markets?
I'm particularly intrigued by the role of randomization in trading algorithms. Could this be seen as a strategy to 'defeat' a Newcomb-like predictor, making algorithms more robust against exploitation?
I'm new here and would value your insights. Is this a perspective worth diving deeper into?
I've spent a lot of time in hospitals. One thing I will say is, if they offer morphine, you should probably take it.