EY could have such price if he invested more time in studying neural networks, but not in writing science fiction. Lesswrong is also full of clever minds which probably could be employed in any tiny AI project.

Several things I thought were interesting:

1. The commentator (on the Deepmind channel) calling out several of AlphaGo's moves as conservative. Essentially, it would play an additional stone to settle or augment some group that he wouldn't necessarily have played around. What I'm curious about is how much this reflects an attempt by AlphaGo to conserve computational resources. "I think move A is a 12 point swing, and move B is a 10 point swing, but move B narrows the search tree for future moves in a way that I think will net me at least 2 more points." (It wouldn't be verbalized like that, since it's not thinking verbally, but you can get this effect naturally from the tree search and position evaluator.)

2. Both players took a long time to play "obvious" moves. (Typically, by this I mean something like a response to a forced move.) 이 sometimes didn't--there were a handful of moves he played immediately after AlphaGo's move--but I was still surprised by the amount of thought that went into some of the moves. This may be typical for tournament play--I haven't watched any live before this.

3. AlphaGo's willingness to play aggressively and get involved in big fights with 이, and then not lose. I'm not sure that all the fights developed to AlphaGo's advantage, but evidently enough of them did by enough.

4. I somewhat regret 이 not playing the game out to the end; it would have been nice to know the actual score. (I'm sure estimates will be available soon, if not already.)

It is a lot easier to document that the Greeks had cutting-edge engineering than to prove that it was based on theoretical knowledge.

Greek aqueducts and post-Greek Roman aqueducts were much better than pre-Greek Roman aqueducts. The process of building them may not have been better, but the choice of what to build was more sophisticated. Before the Greeks they just had water run downhill, requiring tunnels and bridges, afterwards they also ran water uphill. So the Romans definitely learned something from the Greeks. Some people think that they must have understood something about water pressure to do this, which would count as science. But there is no record of how they did it, neither theory, nor rules of thumb developed by trial and error. It is a great mystery that the surviving books by Roman aqueduct engineers don't seem adequate for running the aqueducts, let alone building them.

(By "the Greeks" I mean the Hellenistic period of 300-150BC.)

A better documented connection between theory and application is that Archimedes wrote a book on the theory of simple machines and invented the screw pump. However, that history is also controversial.

Would you labels Google's project of AlphaGo "science" or "engineering"?

Was Roman engineering really based on Greek science? And by the way, what is Greek science? If I understand correctly, the most remarkable scientific contributions of the Greeks were formal geometry and astronomy, but empirical geometry, which was good enough for the practical engineering applications of the time, was already well developed since at least the Egyptians, and astronomy didn't really have practical applications.

Eventual diminishing returns, perhaps but probably long after it was smart enough to do what it wanted with Earth.

A drug that raised the IQ of human programmers would make the programmers better programmers. Also, intelligence is the ability to solve complex problems in complex environments so it does (tautologically) follow.

For almost any goal an AI had, the AI would make more progress towards this goal if it became smarter. As an AI became smarter it would become better at making itself smarter. This process continues. Imagine if it were possible to quickly make a copy of yourself that had a slightly different brain. You could then test the new self and see if it was an improvement. If it was you could make this new self the permanent you. You could do this to quickly become much, much smarter. An AI could do this.

But what if a general AI could generate specialized narrow AIs? That is something the human brain cannot do but an AGI could. Thus speed of general AI = speed of AI narrow + time to specialize.

"you can't prove that AI FOOM is impossible".

I don't agree.

I'm asking for references because I don't have them. it's a shame that the people who are able, ability-wise, to explain the flaws in the MIRI/FHI approach, actual AI researchers, aren't able, time-wise, to do so. It leads to MIRI's views dominating in a way that they should not. It's anomalous that a bunch of amateurs should become the de facto experts in a field, just because they have funding , publicity, and spare time.