Onid

Onid7mo*

Took me a while to get back to you because I wanted to read your whole sequence (linked to me by someone else) before responding.

Anyway, it was far better than anything I wrote, and the thinking was much clearer.

I’m also a bit surprised that I hadn’t seen more discussion about double-halferism. It’s a single, barely comprehensible paragraph on the Sleeping Beauty wikipedia page, and you can barely find any links to it when googling “Sleeping Beauty” or “anthropics.” All this, despite the fact that it’s clearly the correct solution!

Anyway, thank you for the good read. I’m hoping to take a second attempt at this problem that should be a bit more thorough. I think I had gotten most of the way there, reasoning wise, but there were pieces I was clearly missing.

Onid7mo

The problem doesn't specify, you are correct. But if you're trying to use this as a guide for figuring out how to assign probabilities to the origins of our universe, then the 50/50 reasoning is the correct one.

Onid7mo

I think we may be approaching these questions too differently, and I'm having trouble appreciating your question. I want to make sure I actually answer it.

The way I'm modelling the situation is this, the implication being that this is the closest to the way we would want to understand our universe:

1. A universe is created

2. Observers are "placed" in this universe as a result of the universe's inherent processes

3. You are "assigned" to one of these observers at random

In this framework, you don't necessarily get to verify anything. It's merely the case that if you were modelling the universe that way, then you would find that the probability of being in any given universe was determined only by step 1, unaffected by step 3.

Onid7mo

That comment about math was just intended as a rhetorical flourish. My apologies if it was over stated. In context, though, hopefully it was clear that I meant once you’ve established what question you’re actually asking, the math is straightforward.

Would you be willing to expand a little on your last sentence and explain a bit more about what you’re asking? I’m not sure I followed.

Algorithmic Improvement Is Probably Faster Than Scaling Now

7mo

[This has been cross-posted to OneManyNone.substack.com]

[IMPORTANT UPDATE: Comments on this piece - both here and on my substack - have convinced that there are significant issues with the arguments in this piece. I no longer stand by most of what I have written here.]

Today, I’m going to be writing about the Sleeping Beauty (SB) problem, a famous question in the field of philosophy. For those of you who don’t know what the SB problem is or why it’s so important our understanding of our own existence, I’ll explain it in detail in the first section. For the rest, you can skip past that and dive into the... (read 4677 more words →)

Replying toSentience matters

Onid3y

Sentience matters

I feel as if I can agree with this statement in isolation, but can't think of a context where I would consider this point relevant.

I'm not even talking about the question of whether or not the AI is sentient, which you asked us to ignore. I'm talking about how do we know that an AI is "suffering," even if we do assume it's sentient. What exactly is "suffering" in something that is completely cognitively distinct from a human? Is it just negative reward signals? I don't think so, or at least if it was, that would likely imply that training a sentient AI is unethical in all cases, since training requires negative signals.

That's not to say that all negative signals are the same or that maybe in some contexts it's painful or not, just that I think determining that is an even harder problem than determining if the AI is sentient.

Replying toAlgorithmic Improvement Is Probably Faster Than Scaling Now

Onid3y

Fair enough. But for the purposes of this post, the point is that capability increased without increased compute. If you prefer, bucket it as "compute" vs "non-compute" instead of "compute" vs "algorithmic".

Proposal: Tune LLMs to Use Calibrated Language

Why I Believe LLMs Do Not Have Human-like Emotions

I have been thinking a lot recently about the problems of hallucination and sycophancy in LLMs, having developed a couple of hypotheses on the matter. This post will discuss those hypotheses, and a proposal for how to mitigate these problems if my ideas hold. As with my last proposal, I make the disclaimer that it's pretty much impossible to know whether a deep learning idea works until you've tried it, so this is largely just thinking aloud.

My ultimate proposal is a scalable oversight method in the vein of Constitutional AI, where LLMs are used to continually reprogram other LLMs in order to enforce certain output behavior.

The Challenges

I suspect one of the big... (read 1213 more words →)

We Shouldn't Expect AI to Ever be Fully Rational

[EDIT: I don't know exactly why this received so many downvotes - whether it was my tone or arguments or both - but I accept that this was not, in the end, a strong post. I am a new writer here and I am still figuring out how to improve my argumentation.

That said, if we're willing to taboo the word "emotion" as a description of an internal state, then the specific thing I meant to argue - and which I still stand by - is that when an LLM produces emotionally charged outputs (for instance, when Bing chat loses it's temper) those outputs are not meaningfully associated with an internal state in... (read 1971 more words →)

Proposal: Align Systems Earlier In Training

Summary of Key Points^[1]

LLMs are capable of being rational, but they are also capable of being extremely irrational, in the sense that, to quote EY's definition of rationality, their behavior is not a form of "systematically promot[ing] map-territory correspondences or goal achievement."

There is nothing about LLM pre-training that directly promotes this type of behavior, and any example of this behavior in fundamentally incidental. It exists because the system is emulating rationality it has seen elsewhere. That makes LLM rationality brittle. It means that there's a failure mode where the system stops emulating rationality, and starts emulating something else.

As such, LLM-based AGI may have gaps in their reasoning and alignment errors that are fundamentally... (read 1633 more words →)

Inference Speed is Not Unbounded

Unfortunately, if AGI arrives in anything like the current industrial and political landscape, then it is all but inevitable that whoever builds it will not be able to hold onto it for long. Between leaks and espionage, it will not be long before the model is public and anyone can use it to do anything.

This means there is a huge risk from bad actors. Worse, this leaked model may be entirely unaligned, regardless of how much progress we make on alignment in the interim. This is because the current procedure for building and aligning LLMs does not actually even attempt alignment until very far into the process. But it is entirely possible... (read 3170 more words →)