I wrote a very brief comment to Eliezer's last post, which upon reflection I thought could benefit from a separate post to fully discuss its implications. Eliezer argues that we shouldn't really hope to be spared even though > Asking an ASI to leave a hole in a Dyson Shell,...
Epistemic status: prima facie unlikely in the usual framework, which I'll try to reframe. Corroborated by loads of empirical observations. YMMV, but if you've held some contrarian view in the past that you came to realize was wrong, this might resonate. In practical (and also not-so-practical) life, we often have...
I don't think it's "charity" to increase the level of publicity of a conversation, whether digital or in person.
Neither do I: as I said, I actually think it's charity NOT to increase the level of publicity. And people are indeed charitable most of the time. I just think that, if you live your life expecting charity at every instance, you're in for a lot of disappointment, because even though most people are charitable most of the time, there's still going to be a lot of instances in which they won't be charitable. The OP seems to be taking charity for granted, and then complaining about a couple of instances in which it... (read more)
I also don't think privacy is a binary.
That's an interesting perspective. I could subscribe to the idea that journalists may be missing the optimal point there, but that feels a bit weaker than your initial assertion.
Do you think that a conversation we have in LessWrong dms is as public as if I tweeted it?
I mean, I would not quote a DM without asking first. But I understand that as a kind of charity, not an ethical obligation, and while I try my best to be charitable towards others, I do not expect (nor do I feel in any way entitled to) the same level of compassion.
There's definitely a fair expectation against gossiping and bad-mouthing. I don't think that's quite what the OP is talking about, though. I believe the relevant distinction is that (generally speaking) those behaviors don't do any good to anyone, including the person spreading the gossip. But consider how murkier the situation becomes if you're competing for a promotion with the person here:
if you overheard someone saying something negative about their job and then going out of your way to tell their boss.
My understanding is that the OP is suggesting the journalists' attitude is unreasonable (maybe even unethical). You're saying that their attitude is justifiable because it benefits their readers. I don't quite agree that that reason is necessary, nor that it would be by itself sufficient. My view is that journalists are justified in quoting a source because anyone is generally justified in quoting what anyone else has actually said, including for reasons that may benefit no one but the quoter. There are certainly exceptions to this (if divulging the information puts someone in danger, for instance), but those really are exceptions, not the rule. The rule, as recognized both by common practice and by law, is that you simply have no general right to (or even expectation of) privacy about things you say to strangers, unless of course the parties involved agree otherwise.
This sounds absurd to me. Unless of course you're taking the "two golden bricks" literally, in which case I invite you to substitute it by "saving 1 billion other lives" and seeing if your position still stands.
I didn't downvote, but I would've hard disagreed on the "privacy" part if only there were a button for that. It's of course a different story if they're misquoting you, or taking quotes deliberately out of context to mislead. But to quote something you actually said but on second thought would prefer to keep out of publication is... really kind of what journalists need to do to keep people minimally well-informed. Your counterexamples involve communications with family and friends, and it's not very clear to me why the same heuristic should be automatically applied to conversations with strangers. But in any case, not even with the former your communication is "truly" private, as outside of very narrow exceptions like marital privilege, their testimony (on the record, for potentially thousands of people to read too) may be generally compelled under threat of arrest.
The problem here is that the set of all possible commands for which I can't (by that definition) be maximally rewarded is so vast that the statement "if someone maximally rewards/punishes you, their orders are your purpose of life" becomes meaningless.
Not true, as the reward could include all of the unwanted consequences of following the command being divinely reverted a fraction of a second later.
That’s a great question. If it turns out to be something like an LLM, I’d say probably yes. More generally, it seems to me at least plausible that a system capable enough to take over would also (necessarily or by default) be capable of abstract reasoning like this, but I recognize the opposite view is also plausible, so the honest answer is that I don’t know. But even if it is the latter, it seems that whether or not the system would have such abstract-reasoning capability is something at least partially within our control, as it’s likely highly dependent on the underlying technology and training.
To be rewarded (and even more so "maximally rewarded") is to be given something you actually want (and the reverse for being punished). That's the definition of what a reward/punishment is. You don't "choose" to want/not want it, any more than you "choose" your utility function. It just is what it is. Being "rewarded" with something you don't want is a contradiction in terms: at best someone tried to reward you, but that attempt failed.
I wrote a very brief comment to Eliezer's last post, which upon reflection I thought could benefit from a separate post to fully discuss its implications.
Eliezer argues that we shouldn't really hope to be spared even though
Asking an ASI to leave a hole in a Dyson Shell, so that Earth could get some sunlight not transformed to infrared, would cost It 4.5e-10 of Its income.
He then goes on to discuss various reasons why the minute cost to the ASI is insufficient reason for hope.
I made the following counter:
... (read more)Isn’t the ASI likely to ascribe a prior much greater than 4.54e-10 that it is in a simulation, being tested precisely for its willingness to
Epistemic status: prima facie unlikely in the usual framework, which I'll try to reframe. Corroborated by loads of empirical observations. YMMV, but if you've held some contrarian view in the past that you came to realize was wrong, this might resonate.
In practical (and also not-so-practical) life, we often have to make a call as to which theory or fact of matter is probably true. In one particularly popular definition of rationality, being rational is making the right call, as often as possible. If you can make the map correspond to the territory, you should.
I believe that in many cases, the best way to do so is not to adopt what I will... (read 414 more words →)
I feel like this falls into the fallacy of overgeneralization. "Normal" according to whom? Not journalists, apparently.
It's (almost by definition) not unreasonable to expect common courtesy, it's just that people's definitions of what common courtesy even is vary widely. Journalists evidently don't think they're denying you common courtesy when they behave the way most journalists behave.
... (read more)