Thanks for this - I am try to be strict with my time at the moment so have not yet read HPMOR, so a nice succinct overview is useful.

I am a little curious about chapter 97 and getting a lawyer though.

My personal hypothesis is that all human preferences are transient preferences, produced as flashes of positive or negative affect towards some mental concept.

I like that. People do change preferences, a lot - there was that [not very accurate] quote saying (US Centric) along the lines of "If you under 25 and vote republic, you have no heart" "If you over 25 and vote liberal you have no brains"

The most difficult part of this is that people have ingrained beliefs and preferences that will make them unhappy if the other side is picked - rational or not, we cant pick preferences that make all people happy.

For example

Group1 prefers more taxes for social services and hates social injustice

Group2 hates higher taxes

So for Group1 to be happy the Tax_rate should be between 0.2 -> 0.4

but this makes Group2 unhappy as their preferred tax rate is between 0.05 -> 0.15

There is no value in tax_rate that makes all groups happy.

Even if this were solved with infiniate social services and zero tax rates the million other disparate preferences - whether rational or not would cause bigger issues (religeous | athiest, vi | emacs, etc)

Even without an AI, the current trend may well have a world where there is a blurring of real Football matches and simulations.

Certainly you can’t keep an AI safe by using such a model of football

I used to think that a detailed ontological mapping could provide a solution to keeping AI's safe but have slowly realized that it probably isn't likely to work overall. It would be interesting to test this though for small specifically defined domains (like a game of football) - it could work, or at least it would interesting to make a toy experiment to see how it could fail.

It would make sense to have a modular approach - do you have any suggestions on the types of modules the AI might consist of (excluding the engineering type ones like NLP, Machine vision, etc).

The type of modules an AI would consist of would depend on how it is actually implemented.

I tried to sign up but it fails with "not available in my location" error.

I am from Australia - is that a true error message or a glitch?

This in an interesting article, though necessarily abstract - how can we take this to implement an actual AI detector?

This could be the combination of research in the areas of antivirus software, network detection and intrusion, stock market agents along with some sort of intelligence honeypot (ie some construed event / data that exists for a split second that only an AI could detect and act on)

almost all agents are "enemies" to resource gathering agents, as they use up precious resources

If this were a friendly AI, shouldn't the utility function of the gathering agent take sharing / don't be greedy into account. Though as AI development advances we can't be sure that all governments / corporations / random researchers will put the necessary effort into making sure their intelligent agent is friendly.

See http://lesswrong.com/r/discussion/lw/ltf/false_thermodynamic_miracles/

I like this idea, but it could be subject to abuse - a brilliant engineering student may get a large upfront finance then suddenly change course and decide to be part time artist.

If we get to set the criteria of this man in the clouds, we can get useful behaviour. The main criticism of religion is that it's untrue, and that a dedicated observer will realise this. Here we know it to be untrue, but the AI cannot act on that knowledge (see my post on false thermodynamic miracles).

I think the idea of having additional agents B (and C) to act as a form of control is definitely worth pursuing, though I am not clear how it would be implemented.

Is 'w' just random noise added to the max value of u?

If so, would this just act as a limiter and eventually it would find a result close to the original max utility anyway once the random noise falls close to zero?