= 27d567bc2d48d9f40e9ccc20ea370b15)
Kaj_Sotala comments on SIAI’s Short-Term Research Program - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (48)
AFAIK, that was the stuff Goertzel was doing as Director of Research. Now that he isn't around anymore, those things were dropped.
Pretty much the whole "practical experimentation" angle is, again AFAIK, considered too unsafe by the people currently running things at SIAI. At least that's what I was told during my Visiting Fellow time.
I expect improving on state of the art in practical AI is also almost totally useless for figuring out a way towards FAI, so "unsafe" is almost beside the point (except that making things worse while not making them better is not a good plan).
How do you expect to prove anything about an FAI without even knowing what an AGI would look like? I don't think current AI researchers even have that great of an idea of what AGI will eventually look like...
Now improving on state of the art might not be helpful but being in a position where you could improve on state of the art would be; and the best way to make sure you are in such a position is to have actually done it at least once.
It will be (and look) the way we make it. And we should make it right, which requires first figuring out what that is.
An AGI is an extremely complex entity. You don't get to decide arbitrarily how to make it. If nothing else, there are fundamental computational limits on Bayesian inference that are not even well-understood yet. So if you were planning to make your FAI a Bayesian then you should probably at least be somewhat familiar with these issues, and of course working towards their resolution will help you better understand your constraints. I personally strongly suspect there are also fundamental computational limits on utility maximization, so if you were planning on making your FAI a utility maximizer then again this is probably a good thing to study. Maybe you don't consider this AGI research but the main approach to AGI that I consider feasible would benefit at least somewhat from such understanding.
In my opinion, provably friendly AI is hopeless to get to before someone else gets to AGI. The best thing one can hope for is (i) brain uploads come first, or (ii) a fairly transparent AGI design coupled with a good understanding of meta-ethics. This means that as far as I can see, if you want to reduce x-risk from UFAI then you should be doing one of the following:
Do you know where the "we have to have to work towards AGI before we can make progress on FAI" meme came from? (I'm not sure if that's a caricature of the position or what.)
It's an exaggaration in that form, but a milder version seems pretty obvious to me. If you want to design a safe airplane, you need to know something about how to make a working airplane in the first place.
While there are certainly theoretical parts of FAI theory that you can make progress on even without knowing anything about AGI, there's probably a limit to how far you can get that way. For your speculations to be useful, you'll sooner or later need to know something about the design constraints. And they're not only constraints - they'll give you entirely new ideas and directions you wouldn't have considered otherwise.
It sounds nonsensical to claim that you could design safe airplanes without knowing anything about airplanes, that you could be a computer security expert without knowing anything about how software works, or that you could design a safe building without knowing anything about architecture. Why would it make any more sense to claim that you could design FAI without knowing AGI?
In this analogy, the relevant concern maps for me to the notion of "safety" of airplanes. And we know what "safely" for airplanes is. It means people don't die. It's hard to make a proper analogy, since for all usual technology the moral questions are easy, and you are left with technical questions. But with FAI, we also need to do something about moral questions, on an entirely new level.
I agree that solving FAI also involves solving non-technical, moral questions, and that considerable headway can probably be made on these without knowledge about AGI. I was only saying that there's a limit on how far you can get that way.
How far or near that limit is, I don't know. But I would think that there'd be something useful to be found from pure AGI earlier than one might naively expect. E.g. the Sequences draw on plenty of math/compsci related material, and I expect that likewise some applications/techniques from AGI will also be necessary for FAI.