Just because it is obvious to you doesn't mean that everybody immediately jumps to it. Me included and I like the statistics classes back then. Could you please point to the wheels?

Man, that sleeping beauty digression at the end. I'm so easily aggroed.

I wonder if I should write up a rant or a better attempt at exposition, against this "clearly the problem is underdetermined" position. The correct answer is somewhat difficult and all expositions I've seen (or written!) so far have had big shortcomings. But even if one didn't know the right answer, one should readily conclude that a principled answer exists. We assign things probabilities for fairly deep reasons, reasons undisturbed by trifles like the existence of an amnesia pill.

That aside, good talk :)

Argument for a powerful AI being unlikely - was this considered before?

One problem I see here is the "lone hero inventor" implicit assumption, namely that there are people optimizing things for their goals on their own and an AI could be extremely powerful at this. I would like to propose a different model.

This model would be that intelligence is primarily a social, communication skill, it is the skill to disassemble (understand, lat. intelligo), play with and reassemble ideas acquired from other people. Like literally what we are doing on this forum. It is conversational. The whole standing on the shoulder of giants thing, not the lone hero thing.

In this model, inventions are made by the whole of humankind, a network, where each brain is a node communicating slightly modified ideas to each other.

In such a network one 10000 IQ node does not get very powerful, it doesn't even make the network very powerful i.e. a friendly AI does not quickly solve mortality even with human help.

The primary reason I think such a model is correct that intelligence means thinking, we think in concepts, and concepts are not really nailed down but they are constantly modified through a social communication process. Atoms used to mean indivisible units, then they became divisible into little ping-pong balls, and then the model was updated into something entirely different by quantum physics, but is quantum physics based atom theory about the same atoms that were once thought to be indivisible or is this a different thing now? Is modern atomic theory still about atoms? What are we even mapping here and where does the map end and the territory begin?

So the point is human knowledge is increased by a social communication process where we keep throwing bleggs at each other, and keep redefining what bleggs and rubes mean now, keep juggling these concept, keep asking what you really mean under bleggs, and so on. Intelligence is this communication ability, it is to disassemble Joe's concept of bleggs and understand how it differs from Jane's concept of bleggs and maybe assemble a new concept that describes both bleggs.

Without this communication, what would be even intelligence? What would lone intelligence be? It is almost a contradictory term in itself. What would a brain alone in a universe intelligere i.e. understand if nothing would talk to it? Just tinker with matter somehow without any communication whatsoever? But even if we imagine such an "idiot inventor genius", some kind of a mega-plumber on steroids instead of an intellectual or academic it needs goals for that kind of tinkering with that material stuff, for that it needs concepts, and concepts come and evolve from a constant social ping-pong.

An AI would be yet another node in our network, and participate in this process of throwing blegg-concepts at each other probably far better than any human can, but still just a node.

But alas, I fear that Professor Riddle would not have found lasting happiness in Hogwarts."

"Why not? "

"Because I still would've been surrounded by idiots, and I wouldn't have been able to kill them," Professor Quirrell said mildly.

The solution seems obvious (albeit hard and dangerous): make the students smarter so they are no longer idiots.

I cannot be truly killed by any power known to me, and lossing Sstone will not sstop me from returning, nor sspare you or yourss my wrath. Any impetuous act you are contemplating cannot win the game for you, boy.

The last sentence was not said in parseltongue. Could it be that Quirrell used English because it is a lie, and he believes that there is something that Harry could do to win?

Tell us about your feed reader of choice.

I've been using Feedly since Google Reader went away, and has enough faults (buggy interface, terrible bookmarking, awkward phone app that needs to be online all the time) to motivate me towards a new one. Any recommendations?

Imagine we're looking at an obscure group called the Axiom of Choice which has the following policy agenda and weights them with the following relative importances:

• Anthropogenic global warming is real and dangerous: weight 20
• Jesus is the reason for the season: weight 30
• Farms should be collectively owned: weight 15
• Monopoly should be played with no house rules: weight 12
• The Beatles are overrated: weight 20
• Yogic flying is real: weight 3

If I'm on board with everything but the yogic flying, then I'd refer to myself as being 97% in agreement with them.

Now, of course, exact numbers of this sort are practically impossible to come up with. But with sufficient interaction with a group, you can get a sense of what they're about and what's really important to them, and come up with a fairly decent approximation. (Then of course you halve the distance between that and 100%, to add rhetorical weight to your discussion of your differences with them)

I take 10 000 units of Vit D each day. Partly because I'm a pasty nerd who never goes out and partly because large doses are anecdotally helpful for mood.

I take around 1.5mg of melatonin each night. Would have preferred 1 or less but it's too difficult to find them in smaller quantities so I make do with halving 3mg tablets. When I take them I find it significantly easier to get to sleep.

A good portion of LessWrong is unreadable for me as it is based on some kind of altruistic axiom. Personally, I care about myself, my immediate family and a few friends. I will feel a pang of suffering when I see people suffering but I do not feel that pang when I hear about people I don't know suffering, so I conclude that I don't care about other people beyond some abstract measure of proximity and their economic utility for me.

Agree. the road from creation of life to creation of any nervous system at all is an extremely long and fraught one.

Life on our planet has a very specific chemistry. It's possible that almost all possible chemistries limit complexity more than ours - leading to many planets of very simple organisms. Very large number of phyla on earth reach evolutionary dead ends both archae and bacteria are stuck as single cellular organisms, (or very simple aggregrates) - Plants cannot develop movement because of their cell walls, while insects cannot grow bigger because their lungs and exoskeletons do not scale upwards.

Genetics is an entire optimization layer underlying our own, neural one. I think the fact that it had to throw up an entire new, viable optimization layer represents a filter.