If you haven't already read about CEV yet, I'm pretty impressed. There are some failure modes that would crop up if you're not careful, but it's not far from that prima facie workable idea.
Never heard of CEV before, I might look into it later, but I don't have enough time to read it all right now. If its like what I suggested, the fitness function being to accuratley predict the users long-term and short-term goals, I was going to do that in an older AI project that never got finished.
Generally speaking, a smarter-than-human intelligence with strong goals wouldn't passively allow people with different goals to modify its goal system. After all, that would prevent it from achieving the goals it has.
Well once you create an artificial intelligence, then what? If you release the source code or the principles behind its design, anyone can build one with whatever goals they want. Your assuming that the only way another one could pop up is if the original was "hijacked" and pirated, but this probably won't be the case. I am currently working on building the simplest possible self improving system with someone else over the internet. Its for a currently in development higher-level programming language which will (hopefully :P) translate higher level instructions into source code, and learn from its mistakes which the users might point out. Since it is abstracted from the real world and confined to just matching input with output, there really isn't any danger in it taking over the world, although now that I think about it, it could theoretically write a better version of itself as a virus into an unsuspecting users program. Uh-oh, back to the drawing board :(
Since it is abstracted from the real world and confined to just matching input with output, there really isn't any danger in it taking over the world
You haven't heard of the AI Box Experiment yet, and that's just one failure mode.
Well once you create an artificial intelligence, then what?
If it's self-improving and smarter than human... then its goals get achieved. If you can tell that allowing other people to run their own versions of the AI could lead to disaster, then the AI can realize this as well, and act to prevent it.
IMO the most likely scena...
Now is the very last minute to apply for a Summer 2010 Visiting Fellowship. If you’ve been interested in SIAI for a while, but haven’t quite managed to make contact -- or if you’re just looking for a good way to spend a week or more of your summer -- drop us a line. See what an SIAI summer might do for you and the world.
(SIAI’s Visiting Fellow program brings volunteers to SIAI for anywhere from a week to three months, to learn, teach, and collaborate. Flights and room and board are covered. We’ve been rolling since June of 2009, with good success.)
Apply because:
Apply especially if:
(You don’t need all of the above; some is fine.)
Don’t be intimidated -- SIAI contains most of the smartest people I’ve ever met, but we’re also a very open community. Err on the side of sending in an application; then, at least we’ll know each other. (Applications for fall and beyond are also welcome; we’re taking Fellows on a rolling basis.)
If you’d like a better idea of what SIAI is, and what we’re aimed at, check out:
1. SIAI's Brief Introduction;
2. The Challenge projects;
3. Our 2009 accomplishments;
4. Videos from past Singularity Summits (the 2010 Summit will happen during this summer’s program, Aug 14-15 in SF; visiting Fellows will assist);
5. Comments from our last Call for Visiting Fellows; and/or
6. Bios of the 2009 Summer Fellows.
Or just drop me a line. Our application process is informal -- just send me an email at anna at singinst dot org with: (1) a resume/c.v. or similar information; and (2) a few sentences on why you’re applying. And we’ll figure out where to go from there.
Looking forward to hearing from you.