Kaj_Sotala comments on SIAI’s Short-Term Research Program - Less Wrong

31 Post author: XiXiDu 24 June 2011 11:43AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (48)

Comments (48)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: Kaj_Sotala 25 June 2011 07:11:26AM * 6 points [-]

It's an exaggaration in that form, but a milder version seems pretty obvious to me. If you want to design a safe airplane, you need to know something about how to make a working airplane in the first place.

While there are certainly theoretical parts of FAI theory that you can make progress on even without knowing anything about AGI, there's probably a limit to how far you can get that way. For your speculations to be useful, you'll sooner or later need to know something about the design constraints. And they're not only constraints - they'll give you entirely new ideas and directions you wouldn't have considered otherwise.

It sounds nonsensical to claim that you could design safe airplanes without knowing anything about airplanes, that you could be a computer security expert without knowing anything about how software works, or that you could design a safe building without knowing anything about architecture. Why would it make any more sense to claim that you could design FAI without knowing AGI?

Comment author: Vladimir_Nesov 25 June 2011 12:59:46PM * 1 point [-]

In this analogy, the relevant concern maps for me to the notion of "safety" of airplanes. And we know what "safely" for airplanes is. It means people don't die. It's hard to make a proper analogy, since for all usual technology the moral questions are easy, and you are left with technical questions. But with FAI, we also need to do something about moral questions, on an entirely new level.

Comment author: Kaj_Sotala 25 June 2011 04:40:19PM 3 points [-]

I agree that solving FAI also involves solving non-technical, moral questions, and that considerable headway can probably be made on these without knowledge about AGI. I was only saying that there's a limit on how far you can get that way.

How far or near that limit is, I don't know. But I would think that there'd be something useful to be found from pure AGI earlier than one might naively expect. E.g. the Sequences draw on plenty of math/compsci related material, and I expect that likewise some applications/techniques from AGI will also be necessary for FAI.

Comment author: Will_Newsome comment score below threshold [+] (15 children)

Comment author: cousin_it 25 June 2011 01:40:38PM * 4 points [-]

I think you cannot figure out airplane safety before you build airplanes. For example, you'd have to be almost superhumanly smart to predict and beat flutter without experimenting. Or spin, as Kaj_Sotala pointed out. Or spatial disorientation. Even without going into details, can you complete the list of problems by yourself?

Comment author: Kaj_Sotala 25 June 2011 12:40:48PM * 4 points [-]

Nobody knowing how to build AGI hardly counts as "knowing tons already".

Some of the stuff about how to make safe airplanes is obvious. Stuff like how to design the plane so as to minimize the risk of spin isn't. If you disagree, please tell me how to build a maximally stall-safe airplane without looking at the Wikipedia link or otherwise looking it up.

Comment author: Will_Newsome comment score below threshold [+] (11 children)

Comment author: asr 26 June 2011 09:48:50AM 1 point [-]

I'd like evidence of the claim that humanity, or some subset thereof, "basically already knows how to build AGI". I would stipulate that we know how to do whole-brain emulation, giving adequate computational resources; is that what you had in mind?

Comment author: Will_Newsome comment score below threshold [+] (9 children)

Comment author: Kaj_Sotala 27 June 2011 11:10:58AM 6 points [-]

You are aware that exchanges such as this one will make most people think you're a troll, right?

Comment author: Will_Newsome comment score below threshold [+] (6 children)

Comment author: MixedNuts 27 June 2011 12:56:45PM 7 points [-]

When someone tells you a normally hidden consequence of what you're doing, you don't tell them off.
The norm is to give a full answer. If you don't, say you will elaborate later, give a summary, or say it's too long to explain. Don't unilaterally defect.
If you defect, you're telling people your time is more precious than theirs. This is rude.
In particular, you bother to post on LW in the first place. "Near-strangers deserve no information from me" is unfair.
We want to promote pointing out specific problems. A rant against memetics and signalling games in general is a bad answer to specific criticism.

Comment author: Kaj_Sotala 27 June 2011 01:38:21PM 4 points [-]

that a small amount of information when requested is significantly better than no information

That's assuming that the information is correct. It could also be wrong or misleading, in which case it would be better not to receive it. While "Do you mean whole brain emulation?" "No" doesn't fall into this category, claims like "we know how to build AGI" are definitely claims that could be wrong, and are indeed generally considered to be wrong.

Unless you provide a reasonable argument or reason for why we should believe such a claim, anyone maintaining any epistemic hygiene standards (or common sense, for that matter) will be forced to ignore it. Therefore such comments only serve as a distraction, providing no useful value but taking up space and attention.

If I were to comment on conversations and tell people that 2 + 2 = 5 and then refuse to provide any justification when asked, people would quite reasonably conclude that I was a troll, too.

Comment author: RichardKennaway 27 June 2011 02:37:09PM 3 points [-]

Anyway, I am aware that people do not seem to understand that a small amount of information when requested is significantly better than no information, and I am aware that there are no affordances that would allow people to take the 3 seconds of reflection that would be necessary to realize this after realizing that their implicit expectations about how much information they deserve to receive from near-anonymous people are probably not based on any sort of explicitly justified or explicitly reflected-upon analysis.

When I notice myself writing a sentence like that, I drop it and do something else. Maybe I would come back to it later, but not less than 24 hours later.

Comment author: Will_Newsome 29 June 2011 02:07:13PM 2 points [-]

I try to as well. Unfortunately I was distraught squared at the time and wasn't thinking at all clearly as far as social reasoning goes. My apologies to everyone for failing to keep my comments sufficiently constructive.

continue this thread »

Comment author: Multiheaded 31 December 2011 08:35:33PM * 0 points [-]

Comment author: Peterdjones 26 June 2011 12:44:01PM 1 point [-]

Could you expand?

Comment author: jsteinhardt 25 June 2011 12:39:16PM 4 points [-]

That is not in fact how to design safe airplanes. Airplanes are safe mainly because they don't crash. Making everything redundant in an effective way is quite nontrivial and probably depends on the specific design of the aircraft (if you disagree then provide a (rough) algorithm for how to make everything redundant in a way that actually achieves the safety levels of modern aircraft). More importantly, redundancy is not particularly helpful if we don't already know that the plane is guaranteed to fly properly in a wide range of operating conditions that are precisely enough quantified that we can determine in advance whether to ground a plane because e.g. the weather is too extreme.

Kaj_Sotala comments on SIAI’s Short-Term Research Program - Less Wrong

Article Navigation

Comments (48)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (48)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login