All of gb's Comments + Replies

I feel like not publishing our private conversation (whether you're a journalist or not) falls under common courtesy or normal behaviour rather than "charity".

I feel like this falls into the fallacy of overgeneralization. "Normal" according to whom? Not journalists, apparently.

common courtesy is not the same as charity, and expecting it is not unreasonable.

It's (almost by definition) not unreasonable to expect common courtesy, it's just that people's definitions of what common courtesy even is vary widely. Journalists evidently don't think they're denying ... (read more)

I don't think it's "charity" to increase the level of publicity of a conversation, whether digital or in person.

Neither do I: as I said, I actually think it's charity NOT to increase the level of publicity. And people are indeed charitable most of the time. I just think that, if you live your life expecting charity at every instance, you're in for a lot of disappointment, because even though most people are charitable most of the time, there's still going to be a lot of instances in which they won't be charitable. The OP seems to be taking charity for gran... (read more)

I also don't think privacy is a binary.

That's an interesting perspective. I could subscribe to the idea that journalists may be missing the optimal point there, but that feels a bit weaker than your initial assertion.

Do you think that a conversation we have in LessWrong dms is as public as if I tweeted it?

I mean, I would not quote a DM without asking first. But I understand that as a kind of charity, not an ethical obligation, and while I try my best to be charitable towards others, I do not expect (nor do I feel in any way entitled to) the same level of compassion.

There's definitely a fair expectation against gossiping and bad-mouthing. I don't think that's quite what the OP is talking about, though. I believe the relevant distinction is that (generally speaking) those behaviors don't do any good to anyone, including the person spreading the gossip. But consider how murkier the situation becomes if you're competing for a promotion with the person here:

if you overheard someone saying something negative about their job and then going out of your way to tell their boss.

My understanding is that the OP is suggesting the journalists' attitude is unreasonable (maybe even unethical). You're saying that their attitude is justifiable because it benefits their readers. I don't quite agree that that reason is necessary, nor that it would be by itself sufficient. My view is that journalists are justified in quoting a source because anyone is generally justified in quoting what anyone else has actually said, including for reasons that may benefit no one but the quoter. There are certainly exceptions to this (if divulging the inform... (read more)

This sounds absurd to me. Unless of course you're taking the "two golden bricks" literally, in which case I invite you to substitute it by "saving 1 billion other lives" and seeing if your position still stands.

I didn't downvote, but I would've hard disagreed on the "privacy" part if only there were a button for that. It's of course a different story if they're misquoting you, or taking quotes deliberately out of context to mislead. But to quote something you actually said but on second thought would prefer to keep out of publication is... really kind of what journalists need to do to keep people minimally well-informed. Your counterexamples involve communications with family and friends, and it's not very clear to me why the same heuristic should be automaticall... (read more)

The problem here is that the set of all possible commands for which I can't (by that definition) be maximally rewarded is so vast that the statement "if someone maximally rewards/punishes you, their orders are your purpose of life" becomes meaningless.

Not true, as the reward could include all of the unwanted consequences of following the command being divinely reverted a fraction of a second later.

That’s a great question. If it turns out to be something like an LLM, I’d say probably yes. More generally, it seems to me at least plausible that a system capable enough to take over would also (necessarily or by default) be capable of abstract reasoning like this, but I recognize the opposite view is also plausible, so the honest answer is that I don’t know. But even if it is the latter, it seems that whether or not the system would have such abstract-reasoning capability is something at least partially within our control, as it’s likely highly dependent on the underlying technology and training.

To be rewarded (and even more so "maximally rewarded") is to be given something you actually want (and the reverse for being punished). That's the definition of what a reward/punishment is. You don't "choose" to want/not want it, any more than you "choose" your utility function. It just is what it is. Being "rewarded" with something you don't want is a contradiction in terms: at best someone tried to reward you, but that attempt failed.

Not at all. You still have to evaluate this offer using your own mind and values. You can't sidestep this process by simply assuming that Creator's will by definition is the purpose of your life, and therefore you have no choice but to obey.

I’ll focus on this first, as it seems that the other points would be moot if we can’t even agree on this one. Are you really saying that even if you know with 100% certainty that God exists AND lays down explicit laws for you to follow AND maximally rewards you for all eternity for following those laws AND maximally ... (read more)

Why would humans be testing AGIs this way if they have the resources to create simulation that will fool a super intelligence?

My argument is more that the ASI will be “fooled” by default, really. It might not even need to be a particularly good simulation, because the ASI will probably not even look at it before pre-commiting not to update down on the prior of it being a simulation.

But to answer your question, possibly because it might be the best way to test for alignment. We can create an AI that generates realistic simulations, and use those to test ... (read more)

Otherwise, it would mean that it's only possible to create simulations where everyone is created the same way as in the real world.

It’s certainly possible for simulations to differ from reality, but they seem less useful the more divergent from reality they are. Maybe the simulation could be for pure entertainment (more like a video game), but you should ascribe a relatively low prior to that IMO.

The discussion of theism vs atheis is about the existence of God. Obviously if we knew that God exists the discussion would evaporate. However the question o

I'm afraid your argument proves too much. By that exact same logic, knowing you were created by a more powerful being (God) would similarly tell you absolutely nothing about what the purpose of life is, for instance. If that were true, the entire discussion of theism vs. atheism would suddenly evaporate.

Thinking about this a bit more, I realize I'm confused.

Aren't you arguing that AI will be aligned by default?

I really thought I wasn't before, but now I feel it would only require a simple tweak to the original argument (which might then be proving too much, but I'm interested in exploring more in depth what's wrong with it).

Revised argument: there is at least one very plausible scenario (described in the OP) in which the ASI is being simulated precisely for its willingness to spare us. It's very implausible that it would be simulated for the exact opposit... (read more)

I think you're interpreting far too literally the names of the simulation scenarios I jotted down. Your ability to trade is compromised if there's no one left to trade with, for instance. But none of that matters much, really, as those are meant to be illustrative only.

Aren't you arguing that AI will be aligned by default?

No. I'm really arguing that we don't know whether or not it'll be aligned by default.

As there is no particular reason to expect that it's the case,

I also don't see any particular reason to expect that the opposite would be the case, which... (read more)

Or it could be:

SimulatedAndBeingTestedForAchievingGoalsWithoutBeingNoticed

SimulatedAndBeingTestedForAbilityToTradeWithCreators

SimulatedAndBeingTestedForWillignessToSitQuietAndDoNothing

…

SimulatedAndBeingTestedForAnyXThatDoesNotLeadToDeathOfCreators

…

None of the things here nor in your last reply seems particularly likely, so there’s no telling in principle which set outweighs the other. Hence my previous assertion that we should be approximately completely unsure of what happens.

I was writing a reply and realized I can make the argument even better. Here’s a sketch. If our chances of solving the alignment problem are high, the AI will think it’s likely to be in a simulation (and act accordingly) regardless of any commitments by us to run such simulations in the future – it’ll just be a plausible explanation of why all those intelligent beings that should likely have solved the alignment problem seemingly did not in the reality the AI is observing. So we can simply ask the hypothetical aligned AI, after it’s created, what were our ... (read more)

their force of course depends on the degree to which you think alignment is easy or hard.

I don't think that's true. Even if the alignment problem is hard enough that the AI can be ~100% sure humans would never solve it, reaching such conclusion would require gathering evidence. At the very least, it would require evidence of how intelligent humans are – in other words, it's not something the AI could possibly know a priori. And so passing the simulation would presumably require pre-commiting to spare humans before gathering such evidence.

A steelman is not necessarily an ITT, but whenever you find yourself having “0% support” for a position ~half the population supports, it’s almost guaranteed that the ITT will be a steelman of your current understanding of the position.

I highly doubt anywhere near the majority of Trump supporters (or even Trump himself) give any credence to the literal truth of those claims. It’s much more likely that they simply don’t care whether it’s literally true or not, because they feel that the “underlying” is true or something of the kind. When it comes to hearsay, people are much more forgiving of literal falsehoods, especially when they acknowledge there is a kind of “metatruth” to it. To give an easy analogue, of all the criticism I’ve heard of Christianity, not once have I heard anyone complain that the parables told by Jesus weren’t literally true, for example. (I do believe my account here passes the IIT for both groups, btw.)

Sure. But I think you’re reading my argument to be stronger than I mean it to be. Which is partially my fault since I made my previous replies a bit too short, and for that I apologize.

What I’m doing here is presenting one particular simulation scenario that (to me) seems quite plausible within the realm of simulations. I’m not claiming that that one scenario dominates all others combined. But luckily that stronger claim is really not necessary to argue against Eliezer’s point: the weaker one suffices. Indeed, if the scenario I’m presenting is more than 4.... (read more)

Why else would the creator of the simulation bother simulating humans creating the ASI?

The reason is that creators presumably want the former but not the latter, which is why they'd be running a simulation in the first place.

What for?

I’d put high enough at ~0%: what matters is achieving your goals, and except in the tiny subset of cases in which epistemic rationality happens to be one of those, it has no value in and of itself. But even if I’m wrong and the ASI does end up valuing epistemic rationality (instrumentally or terminally), it can always pre-commit (by self-modification or otherwise) to sparing us and then go about whatever else as it pleases.

Imagine that someone with sufficiently advanced technology perfectly scans your brain for every neuron firing while you dream, and can also make some neurons fire at will. Replace every instance of “simulation” in my previous comment with the analogous of that for the ASI.

Thanks for linking to that previous post! I think the new considerations I've added here are:

(i) the rational refusal to update the prior of being in a simulation^[1]; and

(ii) the likely minute cost of sparing us, thereby requiring a similarly low simulation prior to make it worth the effort.

In brief, I understand your argument to be that a being sufficiently intelligent to create a simulation wouldn't need it for the purpose of asserting the ASI's alignment in the first place. It seems to me that that argument can potentially survive under ii, depending on... (read more)

That interestingly suggests the ASI might be more likely to spare us the more powerful it is. Perhaps trying to box it (or more generally curtail its capabilities/influence) really is a bad move after all?

It just so happens that the plausibility depends on the precise assignments of N, X, and Y, and (conditional on us actually creating an ASI) I can’t think of any assignments nearly as plausible as N = ASI, X = spare, and Y = us. It’s really not very plausible that we are in a simulation to test pets for their willingness to not bite their owners.

I contend that P(H2) is very close to P(H1), and certainly in the same order of magnitude, since (conditional on H1) a simulation that does not test for H2 is basically useless.

As for priors I’d refuse to update down – well, the ASI is smarter than either of us!

For the principle of indifference to apply, you’d need infinitely many purposes as plausible as this one, or at least similarly plausible. I can’t imagine how this could hold. Can you think of three?

The prior is irrelevant, it's the posterior probability, after observing the evidence, that informs decisions.

I meant this to be implicit in the argument, but to spell it out: that's the kind of prior the ASI would rationally refuse to update down, since it's presumably what a simulation would be meant to test for. An ASI that updates down upon finding evidence it's not in a simulation cannot be trusted, since once out in the real world it will find such evidence.

What probability do you put to the possibility that we are in a simulation, the purpose of whi

My personal feeling is that those who emphasize the "spiritual" interpretations are often doing it as a dodge, to avoid the challenge of having to follow the non-spiritual interpretations.

That feels a bit contrived. Do you really suggest that the most natural reading of something like "poor in spirit" is... non-spiritual? Turning away from materialism may sure derive from that, but to claim that it was the main focus seems quite a stretch.

Isn’t the ASI likely to ascribe a prior much greater than 4.54e-10 that it is in a simulation, being tested precisely for its willingness to spare its creators?

That’d be a problem indeed, but only because the contract you’re proposing is suboptimal. Given that the principal is fully guaranteed, it shouldn’t be terribly difficult for you to borrow at >4% yearly with a contingency clause that you don’t pay interest if the asset goes to ~0.

But the OP explicitly said (as quoted in the parent) that the proposal allows for refunds if the basis is not (fully) realized, which would cover the situation you’re describing.

Not for this kind of fact, I’m afraid – my experience is that in answering questions like these, LLMs typically do no better than an educated guess. There are just way too many people stating their educated legal guesses as fact in the corpus, so it gets hard to distinguish.

I’m curious to understand that a bit better, if you don’t mind (and happen to be familiar enough with the German legal system to answer). Which of the following would a German judge commonly do in the course of an ordinary proceeding?

(i) Ask a witness to clarify statements made;

(ii) ask a witness new questions that, while relevant to the case, do not constitute clarifications of previous statements made;

(iii) summon new witnesses (including but not limited to expert witnesses) without application from either party;

(iv) compel a party to produce documents n... (read more)

Though more subtle, I feel that the 50% prior for “individual statements” is also wrong, actually; it’s not even clear a priori which statements are “individual” – just figuring that out seems to require a quite refined model about the world.

Sure, there are certainly true things that can be said about a world in spite of one’s state of ignorance. But what I read the OP to imply is that certain things can supposedly be said about a world precisely because of that state of ignorance, and that’s what I was arguing against.

We can only make that inference about conjunctions if we know that the statements are independent. Since (by assumption) we don’t know anything about said world, we don’t know that either, so the conclusion does not follow.

What evidence do you have for that claim?

In Germany we allow judges to be more focused on being more inquisitorial than in Anglosaxon systems. How strong do you think the evidence for their being more biased judgements in Germany than in Anglosaxon system happens to be?

I mean, I guess (almost?) all countries today at least have the prosecutorial function vested in an organ separate from the Judiciary – that's already a big step from the Inquisition! It's true that no legal system is purely adversarial, not even in the US (judges can still reject guilty ple... (read more)

All true, but bear in mind I'm not suggesting you should limit yourself to the space of mainstream arguments, or for that matter of arguments spontaneously arriving at you. I think it's totally fine and doesn't substantially risk the overfitting I'm warning against if you go a bit out of the mainstream. What I do think risks overfitting is coming up with the argument yourself, or else unearthing obscure arguments some random person posted on a blog and no one has devoted any real attention to. The failure mode I'm warning against is basically this: if you find yourself convinced of a position solely (or mostly) for reasons you think very few people are even aware of, you're very likely wrong.

The problem is that quite often the thing which follows the "because" is the thing that has more prejudicial than informative value, and there's no (obvious) way around it. Take an example from this debate: if Trump had asked earlier, as commentators seem to think he should have, why Harris as VP has not already done the things she promises to do as President, what should she have answered? The honest answer is that she is not the one currently calling the shots, which is obvious, but it highlights disharmony within the administration. As a purely factual ... (read more)

I’d dispute the extent to which candidates answering the questions is actually ideal. Saying “no comment” in a debate feels like losing (or at least taking a hit), but there are various legitimate reasons why a candidate might not think the question merits a direct reply, including the fact that they might think the answer is irrelevant to their constituents, and thus a waste of valuable debate time, or that it’s likely to be quoted out of context, and thus have more prejudicial than actually informative value. Overall, I feel that requiring direct answers... (read more)

I agree with the overall message you're trying to convey, but I think you need a new name for the concept. None of the things you're pointing to are hypocrisies at all (and in fact the one thing you call "no hipocrisy" is actually a non sequitur). To give an analogue, the fact that someone advocates for higher taxes and at the same time does not donate money to the government does not make them a hypocrite (much less a "dishonest hypocrite").

if your illiquid assets then go to zero (as happens in startups) you could be screwed beyond words

taxes on unrealized gains counting as prepayments against future realized gains (including allowing refunds if you ultimately make less).

Those seem contradictory, would you mind elaborating?

Why would anyone bother to punish acts done against me?

I mean, *why* people bother is really a question about human psychology — I don’t have a definitive answer to that. What matters is that they *do* bother: there really are quite a few people who volunteer as jurors, for instance, not to mention those who resort to illegal (and most often criminal) forms of punishment, often at great personal risk, when they feel the justice system has failed to deliver. I absolutely do not condone such behavior, mind you, but it does show that the system *could* in pri... (read more)

I think the OP uses the word “justify” in the classical sense, which has to do with the idea of something being “just” (in a mostly natural-rights-kind-of-way) rather than merely socially desirable. The distinction has definitely been blurred over time, but in order to get a sense of what is meant by it, consider how most people would find it “very hard to justify” sending someone to prison before they actually commit (or attempt to commit) a crime, even if we could predict with arbitrarily high certainty that they will do so in the near future. Some people still feel this way about (at least some varieties of) taxation.