All of siIver's Comments + Replies

It doesn't really matter whether the AI uses their full computational capacity. If the AI has a 100000 times larger capacity (which is again a conservative lower bound) and it only uses 1% of it, it will still be 1000 as smart as the human's full capacity.

AGI's algorithm will be better, because it has instant access to more facts than any human has time to memorize, and it will not have all of the biases that humans have. The entire point of the sequences is to list dozens of ways that the human brain reliably fails.

Because

"[the brain] is sending signals at a millionth the speed of light, firing at 100 Hz, and even in heat dissipation [...] 50000 times the thermodynamic minimum energy expenditure per binary swtich operation"

https://www.youtube.com/watch?v=EUjc1WuyPT8&t=3320s

AI will be quantitatively smarter because it'll be able to think over 10000 times faster (arbitrary conservative lower bound) and it will be qualitatively smarter because its software will be built by an algoirthm far better than evolution

"Less than a third of students by their own self-appointed worst-case estimate *1."

missing a word here, I think.

I think your post is spot on.

re-live. Although I'd rather live the same amount of time from now onward.

First question: I know you admire Trump's persuasion skills, but what I want to know is why you think he's a good person/president etc.

Answer: [talks about Trump's persuasion skills]

Yeah, okay.

This is an exceptionally well reasoned article, I'd say. Particular props to the appropriate amount of uncertainty.

Well, if you put it like that I fully agree. Generally, I believe that "if it doesn't work, try something else" isn't followed as often as it should. There's probably a fair number of people who'd benefit from following this article's advice.

I don't quite know how to make this response more sophisticated than "I don't think this is true". It seems to me that whether classes ore lone-wolf improvement is better is a pretty complex question and the answer is fairly balanced, though overall I'd give the edge to lone-wolf.

I don't know what our terminal goals are (more precisely than "positive emotions"). I think it doesn't matter insofar as the answer to "what should we do" is "work on AI alignment" either way. Modulo that, yeah there are some open questions.

On the thesis of suffering requiring higher order cognition in particular, I have to say that sounds incredibly implausible (for I think fairly obvious reasons involving evolution).

This looks solid.

Can you go into a bit of detail on the level / spectrum of difficulty of the courses you're aiming for, and the background knowledge that'll be expected? I suspect you don't want to discourage people, but realistically speaking, it can hardly be low enough to allow everyone who's interested to participate meaningfully.

Yeah, you're of course right. In the back of my mind I realized that the point I was making was flawed even as I was writing it. A much weaker version of the same would have been correct, "you should at least question whether your intuition is wrong." In this case it's just very obvious to me me that there is nothing to be fixed about utilitarianism.

Anyway, yeah, it wasn't a good reply.

This is the ultimate example of... there should be a name for this.

You figure out that something is true, like utilitarianism. Then you find a result that seems counter intuitive. Rather than going "huh, I guess my intuition was wrong, interesting" you go "LET ME FIX THAT" and change the system so that it does what you want...

man, if you trust your intuition more than the system, then there is no reason to have a system in the first place. Just do what is intuitive.

The whole point of having a system like utilitarinism is that we can fig... (read more)

This seems like something we should talk about more.

Although, afaik there shouldn't be a decision between motivation selection and capability controlling measures – the former is obviously the more important part, but you can also always "box" the AI in addition (insofar as that's compatible with what you want it to do).

That sounds dangerously like justifying inaction.

Literally speaking, I don't disagree. It's possible that spreading awareness has a net negative outcome. It's just not likely. I don't discourage looking into the question, and if facts start pointing the other way I can be convinced. But while we're still vaguely uncertain, we should act on what seems more likely right now.

I guess it's a legit argument, but it doesn't have the research aspect and it's a sample size of one.

This just seems like an incredibly weak argument to me. A) it seems to me that prior research will be influenced much more than the probability for an arms race, because the first is more directly linked to public perception, B) we're mostly trying to spread awareness of the risk not the capability, and C) how do we even know that more awareness on the top political levels would lead to a higher probability for an arms race, rather than a higher probability for an international cooperation?

I feel like raising awareness has a very clear and fairly safe upside, while the downside is highly uncertain.

Pretty sure it is. You have two factors, increasing the awareness of AI risk and of AI specifically. The first is good, the second may be bad but since the set of people caring about AI generally is so much larger, the second is also much less important.

I whole-heartedly agree with you, but I don't have anything better than "tell everyone you know about it." On that topic, what do you think is the best link to send to people? I use this, but it's not ideal.

Essentially:

Q: Evolution is a dumb algorithm, yet it produced halfway functional minds. How can it be that the problem isn't easy for humans, who are much smarter than evolution?

A: Evolution's output is not just one functional mind. Evolution put out billions of different minds, an extreme minority of them being functional. If we had a billion years of time and had a trillion chances to get it right, the problem would be easy. Since we only have around 30 years and exactly 1 chance, the problem is hard.

I often ask myself the question of "is this really a thing" when it comes to high level concepts like this. I'm very unsure on Akrasia, and you make a decent enough argument. It could very well not actually be a thing (beyond referring to sub-things).

More importantly, though, even if it were a thing, I agree that the strategy you suggest of focusing on the smaller issues is likely the better one.

I read the first post, which is excellent. Thanks for sharing.

Thanks! So UDT is integrated. That's good to hear.

Can someone briefly explain to me the difference between functional and updateless decision theory / where FDT performs better? That would be much appreciated. I have not yet read FDT because it does not mention UDT (I checked) and I want to understand why UDT needs "fixing" before I invest the time.

Ugh. When I heard about this first I naively thought it was great news. Now I see it's a much harder question.

In my situation, it is the same: you can "determine" whether your dial is set to the first or second position by making a decision about whether to smoke.

No.

You can not. You can't.

I'm struggling with this reply. I almost decided to stop trying to convince you. I will try one more time, but I need you to consider the possibility that you are wrong before you continue to the next paragraph. Consider the outside view: if you were right, Yudkowksy would be wrong, Anna would be wrong, everyone who read your post here and didn't upvote this revoluti... (read more)

it will mean that everyone in the future had their dial set to the second position.

No it won't. Nothing you wrote into the story indicates that you can change the box (in case of no forecaster). If you could, that would change everything (and it wouldn't be the smoking lesion anymore).

I know it was the intention, but it doesn't actually work the way you think.

The thing that causes the confusion is that you introduced an infallible decision maker into the brain that takes all autonomy away from the human (in case of there being no forecaster). This is basically a logical impossibility, which is why I just said "this is newcomb's problem". There has to be a forecaster. But okay, suppose not. I'll show you why this does make a difference.

In Newcomb's problem, you do in fact influence the contents of the opaque box. Your decision ... (read more)

I'm afraid you misunderstand the difference between the Smoking Lesion and Newcomb's problem. In the Smoking Lesion, if you are the kind of person who is affected by the thing which causes lung cancer and the desire to smoke, and you resist this desire, you still die of cancer. Your example is just Newcomb's problem with an infallible forecaster, where if you don't smoke you don't die of cancer. This is an inherent difference. They are not the same.

I'm pretty happy with this article... though one of my concerns is that the section on how exactly AI could wipe out humanity was a bit short. It wants to cure cancer, it kills all humans, okay, but a reader might just think "well this is easy, tell it not to harm humans." I'd have liked if the article had at least hinted at why the problem is more difficult.

Still, all in all, this could have been much worse.

I feel like I am repeating myself. Here is the chain of arguments

1) A normal person seeing this article and its upvote count will walk away having a very negative view of LessWrong (reasons in my original reply)

2) Making the valid points of this article is in no way dependent on the negative consequences of 1). You could do the same (in fact, a better job at the same) without offending anyone.

3) LessWrong can be a gateway for people to care about existential risk and AI safety.

4) AI safety is arguably the biggest problem in the world right now and extremel... (read more)

No, I fully acknowledge that the post tries to do those things, see the second half of my reply. I argue that it fails at doing so but is harmful for our reputation etc.

It's about a set of mannerisms which many people on LW have that are really bad. I don't know what you mean by woke.

I agree that it's clear that you should one box – I'm more talking about justifying why one-boxing is in fact correct when it can't logically influence whether there is money in the box. Initially I found this to be unnerving initially, but maybe I was the only one.

Reposting this from last week's open thread because it seemed to get buried

Is Newcomb's Paradox solved? I don't mean from a decision standpoint, but the logical knot of "it is clearly, obviously better two one-box, and it is clearly, logically proven better to two-box". I think I have a satisfying solution, but it might be old news.

But what makes you think that more complex story types allow many more possibilities?

Isn't that an inherent property of complexity? A larger set of elements -> a larger powerset of elements -> more possibilities. In fact the size of the powerset grows at 2^x. I think a second game of thrones would be less groundbreaking, but doesn't have to be worse... and the same goes for the 1000th GoT.

There seems to be a slowdown in more arty / complex stories this decade (than compared to the 90's for example).

With film and television creation being more de

Well, there is a provably finite possibility space for stories. You only have so many ways to arrange letters in a script. The question is whether it's meaningful.

To use some completely made-up numbers, I think the current possibility space for movies produced by the bottom 80% of people with the current style may be 90% covered. The space for the top 2%, on the other hand, is probably covered for less than 0.1% (and I resisted putting in more zeros there).

To get more concrete, I'll name some pieces (which I avoided doing in my initial post). Take Game of ... (read more)

I'd say no to both. I don't think any genre has come meaningfully close to completion, though I don't know classic of jazz very well.

Let's talk film. If I take a random movie that I didn't like, I find it very similar to others. If, however, I take one that I really like, I find that frustratingly few movies exist that are even similar.

I consider the possibility space to be a function of creativity/intelligence/competence (let's call it skill) of writing, and one that grows faster-than-linearly. The space of medium-skill writing may be nearing completion (... (read more)

While I would agree that those kinds of accusations are used unfairly at times, I don't think it's unreasonable to assign Yudkowsky's statements a higher a priori chance of being true.

Do people feel like the Newcomb paradox (one-boxing yields the better result, it is clearly preferable; two-boxing only means taking an additional 1000$ through a decision that can't possibly have an effect on the 1 million, it is clearly preferable) been resolved through Anna's post in the Sequences (or others)? I strongly feel that I have a solution with no contradictions, but don't want to post it if it's obvious.

In being ironically guilty of not addressing your actual argument here, I'll point out that flaws of LW, valid or otherwise, aren't flaws of rationality. Rationality just means avoiding biases/fallacies. Failure can only be in the community.

Yeah, this is pretty much my conclusion, too. If I had read this article a couple of years ago, it'd have helped me a lot.

I'd add that you should still overrule system 1 in some really important and rare cases, it's just not practical for recurring things.

I'm pretty split on this. I found the quotes from Ben Todd and Robert Wiblin to be quite harmless, but the quotes from Jacy Reese to be quite bad. I don't think it's possible to judge the scope of the problem discussed here based on the post alone. In either case, I think the effort to hold EA to high standards is good.

I don't find this convincing.

“Human intelligence” is often compared to “chimpanzee intelligence” in a manner that presents the former as being so much more awesome than, and different from, the latter. Yet this is not the case. If we look at individuals in isolation, a human is hardly that much more capable than a chimpanzee.

I think the same argument has been made by Hanson, and it doesn't seem to be true. Humans seem significantly superior based on the fact that they are capable of learning language. There is afaik no recorded instance of a chimpanzee... (read more)

Question: Regardless of the degree to which this is true, if everyone collectively assumed that Valence Utilitarianism (every conscious experience has value (positive or negative, depending on pleasantness/unpleasantness), each action's utility is the sum of all value it causes / changes / prevents) was universally true, how much would that change about Friendly AI research?

Well, I don't think this is even complicated. The super happies are right... it is normal for them to forcefully reform us, and it is moral for us to erase the babyeater species.

Suffice to say I preferred the normal ending.

Link is missing!

One thing to keep in mind is that, just because something already exists somewhere on earth, doesn't make it useless on LW. The thing that – in theory – makes this site valuable in my experience, is that you have a guarantee of content being high quality if it is being received well. Sure I could study for years and read all content of the sequences from various fields, but I can't read them all in one place without anything wasting my time in between.

So I don't think "this has already been figured out in book XX" implies that it isn't worth reading. Because I won't go out to read book XX, but I might read this post.

Frankly I think that most people have no business having confident beliefs about any controversial topics. It's a bit weird to argue what an average IQ person "should" believe, because, applying a metric like "what is the average IQ of people holding this belief" is not something they're likely to do. But it would probably yield better results than whatever algorithm they're using.

Your first sentence isn't really a sentence so I'm not sure what you were trying so say. I'm also not sure if you're talking about the same thing I was talkin... (read more)