Understanding the actual abstract reasons for agents' decisions (such as decisions about agreeing with a given argument) seems to me a promising idea, I'm trying to make progress on that (agents' decisions don't need to be correct or well-defined on most inputs for the reasons behind their more well-defined behaviors to lead the way to figuring out what to do in other situations or what should be done where the agents err). Note that if you postulate an algorithm that makes use of humans as its elements, you'd still have the problems of failure modes, regret for bad design decisions and of the capability to answer humanly incomprehensible questions, and these problems need to be already solved before you start the thing up.

If our minds are deterministic computational machines within a universe without any objective value, all our goals are merely elaborate ways to make us feel content with our choices and a possibly inconsistent set of mental motivations. Attempting to model our psychology seems like the most efficient way to solve this problem.

Which problem? You need to define which action should AI choose, in whatever problem it's solving, including the problems that are not humanly comprehensible. This is naturally done in terms of actual humans with all their psychology (as the only available source of sufficiently detailed data about what we want), but it's not at all clear in what way you'd want to use (interpret) that human data.

"Attempting to model psychology" doesn't answer any questions. Assume you have a proof-theoretic oracle and a million functioning uploads living in a virtual world however structured, so that you can run any number of experiments involving them, restart these experiments, infer the properties of whole infinite collections of such experiments and so on. You still won't know how to even approach creating a FAI.

Unfortunately, if you think about it, "predicting how a person feels" isn't really helpful to anything, and doesn't contribute to the project of FAI at all (see Are wireheads happy? and The Hidden Complexity of Wishes, for example).

The same happens with other obvious ideas that you think up in the first 5 minutes of considering the problem, and which appear to argue that "research into nuts and bolts of AGI" is relevant for FAI. But on further reflection, it always turns out that these arguments don't hold any water.

The problem comes down the the question of understanding of what it is exactly you want FAI to do, not of how you'd manage to write an actual program that does that with reasonable efficiency. The horrible truth is that we don't have the slightest technical understanding of what it is we want.

The main mystery in FAI, as I currently see it, is how to define its goal. The question of efficient implementation comes after that and depending on that. There is no point in learning how to efficiently solve the problem you don't want to be solved. Hence the study of decision theory, which in turn benefits from understanding math.

See the "rationality and FAI" section, Eliezer's paper for a quick introduction, also stuff from sequences, for example complexity of value.

I suggest just getting some casual exercise or watching some good films and tv shows. They're full of emotionally motivating experiences.

I think there is a worrying tendency to promote puritan values on LW. I personally see no moral problem with procrastination, or even feeling bad every so often. I feel worried that I might not hit deadlines or experience some practical consequence from not working on a task but I wouldn't want to add moral guilt. I think if people lose sight of the pleasures in life they become nihilistic which in turn leads them to be selfish and cruel as an expression of their pain.

If you can feel good about yourself and recognise that the positive playful fun that can come with idle pleasures might actually be the point. They represent the one value system that does seem pretty sensible. If you can enjoy them, you can feel the emotional energy to be nice and supportive to others. I certainly don't want a friendly AI enforcing the morality of anti-procrastination, anti-unhealthy eating, anti-indulgence or any other form of self flagellating self improvement. Lets just be supportive of one another and try to have a good time.

I am sure these are interesting references for studying pure mathematics but do they contribute significantly to solving AI?

In particular, it is interesting that none of your references mention any existing research on AI. Are there any practical artificial intelligence problems that these mathematical ideas have directly contributed towards solving?

E.g. Vision, control, natural language processing, automated theorem proving?

While there is a lot of focus on specific, mathematically defined problems on LessWrong (usually based on some form of gambling), there seems to be very little discussion of the actual technical problems of GAI or a practical assessment of progress towards solving them. If this site is really devoted to rationality should we not at least define our problem and measure progress towards its solution. Otherwise we risk being merely a mathematical social club, or worse, a probability based religion?

I'm not sure of the merit of studying philosophy as opposed to just personally thinking about philosophical ideas. For me, the most profound pragmatic benefit has been to deeply alter my own psychology as a result of examining ideas like free-will and morality. I had a lot of unexamined assumptions and strongly felt conventions and taboos that I managed to overcome through examining my own feelings in a philosophical way. This is very different from the kind of learnt understanding that can be obtained by reading other peoples ideas. I think it is very common for people to parrot philosophical statements without them actually altering a persons behaviour in any significant way. In particular, I find a lot of classic philosophy unsatisfying as it seems much less relevant in our society. I think this is particularly the case if you adopt a science is enough to explain everything and the brain is a computer perspective. I'd love to read some philosophy that gave me a new and distinct perspective that might alter my priorities, and thus my behaviour, but I have yet to read such material. The closest I've encountered would be status anxiety which helpfully reframes classic philosophical statements concerning status and value in a modern context. This really brought home the extent to which our values are culturally specific. This helped me to pursue my life with my own priorities, as well as giving me the perspective to enable me to examine what I actually value. In that sense I consider these philosophical ideas to be much more practically useful to determining my life than any piece of technical or theoretical understanding.

Each time a question like this comes up it seems to get down voted as a bad question. I think it's a great question, just one for which there are no obviously satisfactory answers. Dennet's approach seems to be to say, if you just word things differently its all fine, nothing to see here. But to me this is a weird avoiding of the question.

We feel there is a difference between living things and inanimate ones. We believe that other people and some animals are feeling things that are similar to the feelings we have. Many people would find it absurd to think that devices or machines were feeling anything. Yet whatever computational model of our minds we create, it is hard to identify the point at which it starts to feel. It is easy to create a virtual character that appears to feel but most people doubt that it is doing any more than simulating feelings, similar to the inauthentic patterns of behaviour we form when we are acting or lying. I think one can imagine what life would feel like to be constantly acting, performing reasoned interactions without sincere emotion, if at heart we are computational why does all interaction not feel this way?

To me this distinction is what makes consciousness distinct and special. I think it is a fascinating consequence of a certain pattern of interacting systems. Implying that conscious feelings occur all over the place, perhaps every feedback system is feeling something.

My justification for this theory is an attempt to provide a simple explanation of the origin of conscious experience, based on a belief that explanations should be simple and lack special cases (I don't find the idea that human beings are fundamentally distinct from other structures particularly elegant).

There have been (and continue to be) many approaches to this, in fact the term Good old fashioned AI basically refers to this. It is very interesting that significant progress has not been made with this approach. This has led to greater use of statistical techniques, such as support vector machines or Bayesian networks. A basic difficulty with any approach to AI is that many techniques merely change the problem of learning, generalisation and problem solving into another form rather than solving it. E.g. Formal methods for software development move the problem from that of programming to that of formal specification. It's not clear that creating the specifications is any less error prone than the creation of the original program.

On the other hand there are enormous benefits to turning any knowledge into a form that a computer can manipulate. There has been some progress in doing this for mathematics. It is possibly one of the most depressing consequences of Godel's theorem that there has not been more work in this area. Writing proofs informally on paper doesn't get around Godel's theorem, it just makes the work harder to validate.

It's also a good challenge for those who feel that AI is just a matter of correctly applying Bayes theorem, as it is not clear how it could be applied to solve this problem.

Has anyone encountered a formal version of this? I.e. a site for the creation of formal logical arguments. Users can create axioms, assign their confidence to them and structure arguments using them. Users can then see the logical consequences of their beliefs. I think it would make a very interesting format for turning debate into a competitive game, whose results are rigorous, machine readable, arguments.