Google has AI safety protocol.

Citation?

I would also contemplate the scenario that the human species might turn out to be less impressive than it currently appears, and is actually a fairly typical example of a successful Earth species. Most achievements that distinguish humans from eg plankton are in the future (eg space industry), not the past or present.

This might sound strange. Arguments in favor of this perspective:

• Homo sapiens is not the greatest species in terms of population or total biomass.

• Homo sapiens is not the only species to make tools, use agriculture, build buildings, or adapt to a variety of terrestrial habitats.

• Homo sapiens is not the first species to have a catastrophic impact on the atmosphere.

Arguments against this perspective:

• The human economy is currently doubling in scale every couple decades.

• No species (probably) ever reached the edge of the atmosphere before Homo sapiens.

(To clarify, i think this question is far from settled. But i think the idea that Homo sapiens will be smaller-impact than expected is more likely than the scenario that historical gods are representations of unknown prosperous civilizations.)

While it is known that AI could be catastrophic, the only organisation (MIRI) which is doing most serios research on its prevention is underfunded. Providing finding to them could dramatically change probability of human survival, and we could estimate that 1 USD donated to them will save 10 human lives.

Is any of this true? "Most serious"? "Dramatically change probability of human survival"? 10 lives per $1?

My thoughts:

1. Google has (is) the biggest computer program = 3 bln lines of code

2. Google has world biggest database, including Youtube, 23andme, Gmail, Google books, all internet content

3. Google is the world biggest computer, which includes something like 1 per cent of total world computing power

4. Google did most impressive AI demonstartion recently that is win in Go.

5. Google is clearly interested in creating AI.

6. Google has AI safety protocol.

7. Google has money to buy needed parts, including people.

So it looks like Google is in winning position. How may be its main competitors? Military AIs in NSA. Other large companies.

Is that the canon explanation? I thought Skynet was acting out of self-preservation.

I think that most people already heard about the fact that AI could be catastrophic risk, and they already has their opinion about it.

In our circle that might be true but many people don't have an opinion that goes beyond terminator.

I've been thinking about what seems to be the standard LW pitch on AI risk. It goes like this: "Consider an AI that is given a goal by humans. Since 'convert the planet into computronium' is a subgoal of most goals, it does this and kills humanity."

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

Worse, the argument can then be made that this idea that an AI will interpret goals so literally without modelling a human mind constitutes an "autistic AI" and that only autistic people would assume that AI would be similarly autistic. I do not endorse this argument in any way, but I guess its still better to avoid arguments that signal low social skills, all other things being equal.

Is there any consensus on what the best 'elevator pitch' argument for AI risk is? Instead of focusing on any one failure mode, I would go with something like this:

"Most philosophers agree that there is no reason why superintelligence is not possible. Anything which is possible will eventually be achieved, and so will superintelligence, perhaps in the far future, perhaps in the next few decades. At some point, superintelligences will be as far above humans as we are above ants. I do not know what will happen at this point, but the only reference case we have is humans and ants, and if superintelligences decide that humans are an infestation, we will be exterminated."

Incidentally, this is the sort of thing I mean by painting LW style ideas as autistic (via David Pierce)

As far as we can tell, digital computers are still zombies. Our machines are becoming autistically intelligent, but not supersentient - nor even conscious. [...] Full-Spectrum Superintelligence entails: [...] social intelligence [...] a metric to distinguish the important from the trivial [...] a capacity to navigate, reason logically about, and solve problems in multiple state-spaces of consciousness [e.g. dreaming states (cf. lucid dreaming), waking consciousness, echolocatory competence, visual discrimination, synaesthesia in all its existing and potential guises, humour, introspection, the different realms of psychedelia [...] and finally "Autistic", pattern-matching, rule-following, mathematico-linguistic intelligence, i.e. the standard, mind-blind cognitive tool-kit scored by existing IQ tests. High-functioning "autistic" intelligence is indispensable to higher mathematics, computer science and the natural sciences. High-functioning autistic intelligence is necessary - but not sufficient - for a civilisation capable of advanced technology that can cure ageing and disease, systematically phase out the biology of suffering, and take us to the stars. And for programming artificial intelligence.

Sometimes David Pierce seems very smart. And sometimes he seems to imply that the ability to think logically while on psychedelic drugs is as important as 'autistic intelligence'. I don't think he thinks that autistic people are zombies that do not experience subjective experience, but that also does seem implied.

My favorite crazy unlikely idea about that is that the Paleocene-Eocene Thermal Maximum 50 megayears ago - a 200k year pulse of high CO2 levels and temperatures in which the CO2 was added over a timescale of less than 10k years (potentially much less) and had an isotopic composition consistent with having been liberated from biogenic deposits - could theoretically be explained by all the coal and oil deposits of Antarctica being burned followed by some positive feedbacks kicking in.

(Most land of Antarctica never having been investigated geologically in any detail at all due to being under kilometers of ice) (And Antarctica at that time being completely unglaciated and relatively temperate despite being where it is now by then) (And subsequent glaciation having scraped most of the surface clean of anything that was on it at the time)

We have an advantage in that we evolved in the tropics - you can take a tropical animal and keep it warm near the poles by wrapping it in clothes. It's much more difficult to take a cold-adapted polar animal and keep it alive in the tropics...

Worth noting:

• Rudimentary grain cultivation has been pushed back at least 23k years ago, well into the last glacial maximum and probably older: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0131422

• Cooked grain flour has been observed in Italy over 30k years ago, no information on the plants it came from.

Possibly indicating that the end of the last glaciation rather than new invention drove the more or less simultaneous large-scale agricultural transitions that occurred all across the old and new world ~10k years ago.

Good job with the main idea. However your speculation about past tech civilizations on Earth, artifacts preserved on moon etc. seems only half lucid.