Thank you for your comments. I have included them in version 1.1 of the map, where I have swapped FRI and OpenAI/DeepMind, added Crystal Trilogy and corrected the spelling of Vernor Vinge.
I have taken the survey.
I think difference in date of birth (1922 vs ~1960) is less important than difference of date of publication (2003 vs ~2015).
On the Outside View, is criticism 12 years after publication more likely to be valid than criticism levelled immediately? I do not know. On one hand, science generally improves over time. On the other hand, if a particular work get the first criticism after many years, it could mean that the work is of higher quality.
I should clarify that I am referring to the section David Chapman calls: "Historical appendix: Where did the confusion come from?". I read it as a criticism of both Jaynes and his book.
No, I do know what Yudkowsky's argument is. Truth be told, I probably would be able to evaluate the arguments, but I have not considered it important. Should I look into it?
I care about whether "The Outside View" works as a technique for evaluating such controversies.
Yes! From the Outside View, this is exactly what I would expect substantial, well-researched criticism to look like. Appears very scientific, contains plenty of references, is peer-reviewed and published in "Journal of Statistical Physics" and has 29 citations.
Friedman and Shimonys criticism of MAXENT is in stark contrast to David Chapmans criticism of "Probability Theory".
Could you post a link to a criticism similar to David Chapman?
The primary criticism I could find was the errata. From the Outside View, the errata looks like a number of mathematically minded people found it to be worth their time to submit corrections. If they had thought that E. T. Jaynes was hopelessly confused, they would not have submitted corrections of this kind.
I don't think it's a good sign for a book if there isn't anybody to be found that criticizes it.
I think it is a good sign for a Mathematics book that there isn't anybody to be found that criticizes it except people with far inferior credentials.
Thank you for pointing this out. I did not do my background check far enough back in time. This substantially weakens my case.
I am still inclined to be skeptical, and I have found another red flag. As far as I can tell, E. T. Jaynes is generally very highly regarded, and the only person who is critical of his book is David Chapman. This is just from doing a couple of searches on the Internet.
There are many people studying logic and probability. I would expect some of them would find it worthwhile to comment on this topic if they agreed with David Chapman.
I do not know enough about logic to be able to evaluate the argument. But from the Outside View, I am inclined to be skeptical about David Chapman:
DAVID CHAPMAN
"Describing myself as a Buddhist, engineer, scientist, and businessman (...) and as a pop spiritual philosopher“
Web-book in progress: Meaningness
Tagline: Better ways of thinking, feeling, and acting—around problems of meaning and meaninglessness; self and society; ethics, purpose, and value.
EDWIN THOMPSON JAYNES
Professor of Physics at Washington University
Most cited works:
Information theory ...
My apologies for not being present. I did not put it into my calendar, and it slipped my mind. :(
You might also be interested in this article by Kaj Sotala: http://kajsotala.fi/2016/04/decisive-strategic-advantage-without-a-hard-takeoff/
Even though you are writing about the exact same subject, there is (as far as I can tell) no substantial overlap with the points you highlight. Kaj Sotala titled his blog post "(Part 1)" but never wrote a subsequent part.
Also, it looks like the last time slot is 2200 UTC. I can participate from 1900 and forward.
I will promote this in the AI Safety reading group tomorrow evening.
The title says 2017/6/27. Should it be 2017-05-27?
Good luck with meetup!
In the Skype-based reading group, we followed the "Ambitious" plan from MIRI's reading guide: https://intelligence.org/wp-content/uploads/2014/08/Superintelligence-Readers-Guide-early-version.pdf We liked the plan. Among other things, the guide recommended splitting chapter 9 into two parts, and that was good advice.
Starting from chapter 7, I made slides appropriate for a 30 minute summary: http://airca.dk/reading_group.htm
Be sure to check out the comments from the Lesswrong reading group by Katja Grace: http://lesswrong....
I think I agree with all your assertions :).
(Please forgive me for a nitpick: The opposite statement would be "Many humans have the ability to kill all humans AND AI Safety is a good priority". NOT (A IMPLIES B) is equivalent to A AND NOT B. )
There are no specific plans - at the end of each session we discuss briefly what we should read for next time. I expect it will remain a mostly non-technical reading group.
Do you think Leo Szilard would have had more success through through overt means (political campaigning to end the human race) or surreptitiously adding kilotons of cobalt to a device intended for use in a nuclear test? I think both strategies would be unsuccessful (p<0.001 conditional on Szilard wishing to kill all humans).
I fully accept the following proposition: IF many humans currently have the capability to kill all humans THEN worrying about long-term AI Safety is probably a bad priority. I strongly deny the antecedent.
I guess the two most plausible candidates would be Trump and Putin, and I believe they are exceedingly likely to leave survivors (p=0.9999).
The word 'sufficiently' makes your claim a tautology. A 'sufficiently' capable human is capable of anything, by definition.
Your claim that Leo Szilard probably could have wiped out the human race seems very far from the historical consensus.
Good idea. I will do so.
No, a Superintelligence is by definition capable of working out what a human wishes.
However, a Superintelligence designed to e.g. calculate digits of pi would not care about what a human wishes. It simply cares about calculating digits of pi.
I have taken the survey.
Congratulations!
My wife is also pregnant right now, and I strongly felt that I should include my unborn child in the count.
This interpretation makes a lot of sense. The term can describe events that have a lot of Knightian Uncertainty, which a "Black Swan" like UFAI certainly has.
You bring up a good point, whether it is useful to worry about UFAI.
To recap, my original query was about the claim that p(UFAI before 2116) is less than 1% due to UFAI being "vaguely magical". I am interested in figuring out what that means - is it a fair representation of the concept to say that p(Interstellar before 2116) is less than 1% because interstellar travel is "vaguely magical"?
What would be the relationship between "Requiring Advanced Technology" and "Vaguely Magical"? Clarke's third law is a straightfo...
Many things are far beyond our current abilities, such as interstellar space travel. We have no clear idea of how humanity will travel to the stars, but the subject is neither "vaguely magical", nor is it true that the sentence "humans will visit the stars" does not refer to anything.
I feel that it is an unfair characterization of the people who investigate AI risk to say that they claim it will happen by magic, and that they stop the investigation there. You could argue that their investigation is poor, but it is clear that they have worked a lot to investigate the processes that could lead to Unfriendly AI.
Like Unfriendly AI, algae blooms are events that behave very differently from events we normally encounter.
I fear that the analogies have lost a crucial element. OrphanWIlde considered Unfriendly AI "vaguely magical" in the post here. The algae bloom analogy also has very vague definitions, but the changes in population size of an algae bloom is a matter I would call "strongly non-magical".
I realize that you introduced the analogies to help make my argument precise.
Wow. It looks like light from James' spaceship can indeed reach us, even if light from us cannot reach the spaceship.
English is not my first language. I think I would put the accent on "reaches", but I am unsure what would be implied by having the accent on "super". I apologize for my failure to write clearly.
I now see the analogy with human reproduction. Could we stretch the analogy to claim 3, and call some increases in human numbers "super"?
The lowest estimate of the historical number of humans I have seen is from https://en.wikipedia.org/wiki/Population_bottleneck , claiming down to 2000 humans for 100.000 years. Human numbers will proba...
Intelligence, Artificial Intelligence and Recursive Self-improvement are likely poorly defined. But since we can point to concrete examples of all three, this is a problem in the map, not the territory. These things exist, and different versions of them will exist in the future.
Superintelligences do not exist, and it is an open question if they ever will. Bostrom defines superintelligences as "an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills."...
I meant claim number 3 to be a sharper version of your claim: The AI will meet constraints, impediments and roadblocks, but these are overcome, and the AI reaches superintelligence.
Could you explain the analogy with human reproduction?
Thank you. It is moderately clear to me from the link that James' thought-experiment is possible.
Do you know of a more authoritative description of the thought-experiment, preferably with numbers? It would be nice to have an equation where you give the speed of James' spaceship and the distance to it, and calculate if the required speed to catch it is above the speed of light.
Some of the smarter (large, naval) landmines are arguably both intelligent and unfriendly. Let us use the standard AI risk metric.
I feel that your sentence does refer to something: A hypothetical scenario. ("Godhood" should be replaced with "Superintelligence").
Is it correct that the sentence can be divided into these 4 claims?:
Do you mean th...
I've seen this claim many places, including in the Sequences, but I've never been able to track down an authoritative source. It seems false in classical physics, and I know little about relativity. Unfortunately, my Google-Fu is too weak to investigate. Can anyone help?
Could you elaborate on why you consider p(UFAI before 2116) < 0.01? I am genuinely interested.
It is an interesting way of looking at the maximal potential of AIs. It could be that Oracle Machines are possible in this universe, but an AI built by humans cannot self-improve to that point because of the bound you are describing.
I feel that the phrasing "we have reached the upper bound on complexity" and later "can rise many orders of magnitude" gives a potentially misleading intuition about how limiting this bound is. Do you agree that this bound does not prevent us from building "paperclipping" AIs?
I am tapping out of this thread.
It is possible to be extremely intelligent, and suffer from a delusion.
My thoughts exactly.
When I first heard it, it sounded to me like a headline from BuzzFeed: This one weird trick will literally solve all your problems!
Turns out that the trick is to create an IQ 20000 AI, and get it to help you.
(Obviously, Suspicious <> Wrong)
I've tried my hand at visualizing it:
http://i.imgur.com/VE0P8JY.png
This picture shows the very last instant that the shopkeeper can choose to reset Link.
There are a number of assumptions in my calculations, which might not be valid in the actual game. A key assumption is that arrows fly at 3 times walking speed.
The Shopkeeper will need to walk 1 tile north to reset Link. That requires the same amount of time as for an arrow to fly 3 tiles.
At T=0, Link starts moving north, and the arrow heading towards Link continues heading west.
At T=1, Link has mo
Thank you. That was exactly what I was after.
Hi,
I've read some of "Rationality: From AI to Zombies", and find myself worrying about unfriendly strong AI.
Reddit recently had an AMA with the OpenAI team, where "thegdb" seems to misunderstand the concerns. Another user, "AnvaMiba" provides 2 links (http://www.popsci.com/bill-gates-fears-ai-ai-researchers-know-better and http://fusion.net/story/54583/the-case-against-killer-robots-from-a-guy-actually-building-ai/) as examples of researchers not worried about unfriendly strong AI.
The arguments presented in the links above a...
I really like this visualization.
May I suggest another image, where the shopkeeper is in non-obvious danger:
To the left, the Shopkeeper is surrounded by ice-blocks, as in the images. All the way to the right, a monster is shooting arrows at Link, who is shooting arrows back at the monster. (The Gem-container is moved somewhere else.) Link, the Shopkeeper and the monster are on the same horizontal line. It looks like Link is about to heroically take an arrow that the monster aimed for the shopkeeper. The ice is still blocking, so the shopkeeper appears sa...
Thank you for explaining.