LESSWRONG
LW

All of Riteofwhey's Comments + Replies

Steelmaning AI risk critiques
Riteofwhey60

I would be somewhat more convinced that MIRI was up to it's mission if they could contribute to much simpler problems in prerequisite fields.

Reply
Steelmaning AI risk critiques
Riteofwhey20

Verification seems like a strictly simpler problem. If we can't prove properties for a web server, how are we going to do anything about a completely unspecified AI?

The AI take over scenarios I've head almost always involve some kind of hacking, because today hacking is easy. I don't see why that would necessarily be the case a decade from now. We could prove some operating system security guarantees for instance.

Reply
4gjm
Yes, verification is a strictly simpler problem, and one that's fairly thoroughly addressed by existing research -- which is why people working specifically on AI safety are paying attention to other things. (Maybe they should actually be working on doing verification better first, but that doesn't seem obviously a superior strategy.) Some AI takeover scenarios involve hacking (by the AI, of other systems). We might hope to make AI safer by making that harder, but that would require securing all the other important computer systems in the world. Even though making an AI safe is really hard, it may well be easier than that.
Steelmaning AI risk critiques
Riteofwhey60

Thanks for doing this. A lack of self criticism about AI risk is one of the reasons I don't take it too seriously.

I generally agree with http://su3su2u1.tumblr.com/ , but it may not be organized enough to be helpful.

As for MIRI specifically, I think you'd be much better served by mainstream software verification and cryptography research. I've never seen anyone address why that is not the case.

I have a bunch of disorganized notes about why I'm not convinced of AI risk, if you're interested I could share more.

Reply
4gjm
It's solving a different problem. Problem One: You know exactly what you want your software to do, at a level of detail sufficient to write the software, but you are concerned that you may introduce bugs in the implementation or that it may be fed bad data by a malicious third party, and that in that case terrible consequences will ensue. Problem Two: You know in a vague handwavy way what you want your software to do, but you yet don't know with enough precision to write the software. You are concerned that if you get this wrong, the software will do something subtly different from what you really wanted, and terrible consequences will ensue. Software verification and crypto address Problem One. AI safety is an instance of Problem Two, and potentially an exceptionally difficult one.