AIXI is able to be widely applicable because it tries to model every possible program that the universe could be running, and then it eventually starts finding programs that fit.
Driverless cars may start containing modeling things other than driving, and may even start trying to predict where their users are going to be, but I suspect that it would try and just track user habits or their smartphones, rather than trying to figure out their owner's economic and psychological incentives for going to different places.
Trying to build a car that's generally capable of driving and figuring out new things about driving might be dangerous, but there's plenty of useful features to give people before they get there.
Just wondering, is your intuition coming from the tighter tie to reality that a driverless car would have?
"It was terrible, officer ... my mother, she was so happy with her new automatic car! It seemed to anticipate her every need! Even when she forgot where she wanted to go, in her old age, the car would remember and take her there ... she had been so lonely ever since da' passed. I can't even fathom how the car got into her bedroom, or what it was, oh god, what it was ... doing to her! The car, it still ... it didn't know she was already ... all that blood ..."
I know people have talked about this in the past, but now seems like an important time for some practical brainstorming here. Hypothetical: the recent $15mm Series A funding of Vicarious by Good Ventures and Founders Fund sets off a wave of $450mm in funded AGI projects of approximately the same scope, over the next ten years. Let's estimate a third of that goes to paying for man-years of actual, low-level, basic AGI capabilities research. That's about 1500 man-years. Anything which can show something resembling progress can easily secure another few hundred man-years to continue making progress.
Now, if this scenario comes to pass, it seems like one of the worst-case scenarios -- if AGI is possible today, that's a lot of highly incentivized, funded research to make it happen, without strong safety incentives. It seems to depend on VCs realizing the high potential impact of an AGI project, and of the companies having access to good researchers.
The Hacker News thread suggests that some people (VCs included) probably already realize the high potential impact, without much consideration for safety:
Is there any way to reverse this trend in public perception? Is there any way to reduce the number of capable researchers? Are there any other angles of attack for this problem?
I'll admit to being very scared.