I agree with all of your summaries, so any readers should partially discount the many occurrences of "so far as I can tell." :)
How do you respond to the claim that the few researchers who are working directly on very general, Solomonoff-approximating AI systems are, when put together, dangerous?
Everything is somewhat dangerous; and relative risk is more important than absolute risk.
The entities that I think are most dangerous existential risks are inhumanly capable, distributed entities that hold many human lives and livelihoods hostage against direct assault and defend their continued existence ferociously, sometimes by disinformation or by promoting human irrationality. These entities are more like fault lines in a crystal or knots in string than they are like organisms. I'm thinking of various industries, the practice of advertising, perhaps capitalism, nationalism or some features of how they each are presently implemented - but the most dangerous entities are the ones that we don't currently recognize.
I believe this is closer to Bill Joy's pessimism, than EY's. It has everything to do with these sort of large, difficult-to-control social forces, and almost nothing to do with doubts about scientists' ability to control their experiments.
Studying AGI probably has some associated risks, but as a disruptive technology, it might be able to move some of those crystal flaws or knots out of the center of human society - and doing nothing, or simply allowing the invisible hand of the market to push you along, will not help.
A lot of people fear corporations. A lot of people fear governments. And not a few think religion is pernicious.
So none of them can be the ones we don't currently recognize.
I feel like I'm opening the Necronomicon, but besides religions, corporations, and governments, what else should I be worried about?
The forces that have been operating all along, that we don't perceive or name because they aren't close enough to any of the patterns our brains evolved to perceive. http://lesswrong.com/lw/10n/why_safety_is_not_safe/
The forces that have been operating all along, that we don't perceive or name because they aren't close enough to any of the patterns our brains evolved to perceive.
This belongs in a piece of Lovecraftian fiction.
I've contemplated writing something along those lines, but I haven't yet managed to come up with something that would be an entertaining story and still keep the point. Even Lovecraft, after all, only managed to write evocative fiction about an uncaring universe via the metaphor of intelligent monsters.
There was a short story (sorry, no cite) about humans picking up radio transmissions from an alien species which had solved the problem of war and, iirc, poverty, too. They were so prosperous they could afford to eat off of lead plates! But they were dying for reasons they couldn't understand.... IIRC, they died off before they could explain their social insight to humanity.
More subtly, I suspect that large scale mood disorders could affect the fate of cultures. Without fully invoking the mind-killer, if people stop caring enough to keep their society going, whether by not bothering with crucial details, or by building defection into their institutions, then there will be a long term effect.
I've heard of an old theory that sunspots could affect people by making them more irritable. (Again, no cite, and I don't think it matters if it was a real theory.) Anything that makes people generally more irritable increases the chance of war because the people at the top are more likely to create and react to slights and threats.
Large scale mood disorders would be in our blind spot, but not in a spooky or interesting way. They're just too big and too slow for us to notice, and besides, surely there's nothing wrong with our national character.
Don't mock me, I'm trying to stretch my imagination here.
What about something like shoes? The transition from barefooting to shoes-wearing norms might be prestige-driven (so despite the discomfort of switching TO shoes, it's prestigious) and the norm could be enforced by the discomfort of switching AWAY from shoes, once your feet are adjusted.
Generally, things that encourage one-way society-level transitions that look like "society is addicted to X" might be dangerous. The "things" in question (social practices, memes, consumer products) might look much wierder or more innocent than shoes.
Cultural rachetting, so that people forget they have options, is probably a problem in most cases. It's probably a good thing that people no longer think in terms of murderous feuds between families, though.
I do think rules rachetting-- that it's generally much easier to add rules than to get rid of them, is a very serious problem for human beings.
"Society is addicted to X?" Alcohol, tobacco, & caffeine come to mind. Of course, our present culture is trying hard to overcome its former tobacco addiction.
Addictive behaviors at the individual level are a problem, but a society seemingly unable to control itself could be caused by some other kind of irrationality at the individual level - or even something like individual "rational" self-interest.
Can you list some? If you mean AIXI-like systems, I see them as a new type of complexity theory used to help us understand the problem space, but not as a cognitive architecture that might be used in an AGI. Nothing using such an exhaustive approach could operate in the real world.
Sorry for the slow response. Solomonoff, until he died recently. Marcus Hutter is implementing an AIXI approximation last I heard. Eray Ozkural, implementing Solomonoff's ideas. Sergey Pankov, implementing an AIXI approximation.
he's getting Rete to run as an emergent consequence of the sum-product algorithm.
Say what? How does that work?
I was a bit disappointed, since (as far as I can tell) this work simply used Soar as an exotic programming language.
Pretty much all Soar work uses Soar as an exotic programming language. It's a production system. When I studied it, which admittedly was 18 years ago, it had no learning component beyond a mysterious, programmer-dependent "chunking". I would call it a cognitive infrastructure, that you can use to build a cognitive architecture.
Its claim to being a cognitive architecture relied (at least originally) largely on being able to model human performance time on different tasks. But any programming language will be able to do that, if the task naturally has some particular input-dependent runtime, and the task is described properly.
Soar has the problem that it's funded by the military, who want to use it in simulations eg JSAF; but they want untrained military personnel to be able to build things with it. They don't want a smarter AI as much as they want an easier-to-use AI.
I only saw the 15-minute version, and I didn't understand it, but Rosenbloom's papers, including an unpublished one about the "Bayesian Decision Cycle" are online: http://cs.usc.edu/~rosenblo/pubs.html
I was a bit disappointed, since (as far as I can tell) this work simply used Soar as an exotic programming language.
Every general-purpose AI tool converges on being a programming language.
The more it becomes a programming language, the less it has to do with AI.
I'm a bit confused as to what is Soar and how it works, but it does sound very interesting. Of course, trying to model the way the human mind works is the oppisite of what we should be trying to do. Immagine all of the short-comings of human reasoning highly exagerated in a computer simulation.
Re: "In particular, I want to understand why AI researchers are generally convinced that their experiments and research are fairly safe"
Stupid machines are the ones that kill people. Look at the roads. Machine intelligence will put an end that carnage.
This is a report from a LessWrong perspective, on the 30th Soar workshop. Soar is a cognitive architecture that has been in continuous development for nearly 30 years, and is in a direct line of descent from some of the earliest AI research (Simon's LT and GPS). Soar is interesting to LessWrong readers for two reasons:
Where I'm coming from: I'm a skeptic about EY/SIAI dogmas that AI research is more risky than software development, and that FAI research is not AI research, and has little to learn from the field of AI research. In particular, I want to understand why AI researchers are generally convinced that their experiments and research are fairly safe - I don't think that EY/SIAI are paying sufficient attention to these expert opinions.
Overall summary: John Laird and his group are smart, dedicated, and funded. Their theory and implementation moves forward slowly but continuously. There's no (visible) work being done on self-modifying, bootstrapping or approximately-universal (e.g. AIXItl) entities. There is some concern about how to build trustworthy and predictable AIs (for the military's ROE) - for example, Scott Wallace's research.
As far as I can tell, the Soar group's work is no more (or less) risky than narrow AI research or ostensibly non-AI software development. To be blunt - package managers like APT seem more risky than Soar, because the economic forces that push them to more capability and complexity are more difficult to control.
Impressions of (most of) the talks - they can be roughly categorized into three types.
I want to emphasize that these just my impressions (which are probably flawed - because of my inexperience I probably misunderstood important points), and the proceedings (that is, the slides that everyone used to talk with) will soon be available, so you can read them and form your own impressions.
There are three forks to my implicit safety case while developing. I'm not claiming this is a particularly good safety case or that developing Rogue-Soar was safe - just that it's what I have.
The first fork is that tasks vary in their difficulty (Pickering's "resistances"), and entities vary in their strength or capability. There's some domain-ish structure to entity's strengths (a mechanical engineering task will be easier for someone trained as a mechanical engineer than a chemist), and intention matters - difficult tasks are rarely accomplished unintentionally. I'm fairly weak, and my agent-in-progress was and is very, very weak. The chance that I or my agent solves a difficult task (self-improving AGI) unintentionally while writing Rogue-Soar is incredibly small, and comparable to the risk of my unintentionally solving self-improving AGI while working at my day job. This suggests that safer (not safe) AI development might involve: One, tracking ELO-like scores of people's strengths and task difficulties, and Two, tracking and incentivizing people's intentions.
The second fork is that even though I'm surprised sometimes while developing, the surprises are still confined to an envelope of possible behavior. The agent could crash, run forever, move in a straight line or take only one step when I was expecting it to wander randomly, but pressing "!" when I expected it to be confined to "hjkl" would be beyond this envelope. Of course, there are many nested envelopes, and excursions beyond the bounds of the narrowest are moderately frequent. This suggests that safer AI development might involve tracking these behavior envelopes (altogether they might form a behavior gradient), and the frequency and degree of excursions, and deciding whether development is generally under control - that is, acceptably risky compared to the alternatives.
The third fork is that the runaway takeoff arguments necessarily involve circularities and feedback. By structural inspection and by intention, if the AI is dealing with Rogue, and not learning, programming, or bootstrapping, then it's unlikely to undergo takeoff. This suggests that carefully documenting and watching for circularities and feedback may be helpful for safer AI research.