A high level post on its use would be very interesting.

I think my main criticism of the Bayes approach is that it leads to the kind of work you are suggesting i.e. have a person construct a model and then have a machine calculate its parameters.

I think that much of what we value in intelligent people is their ability to form the model themselves. By focusing on parameter updating we aren't developing the AI techniques necessary for intelligent behavior. In addition, because correct updating does not guarantee good performance (because the model properties dominate) then we will always have to judge methods based on experimental results.

Because we always come back to experimental results, whatever general AI strategy we develop its structure is more likely to be one that searches for new ways to learn (with bayesian model updating and SVMs as examples) and validates these strategies using experimental data (replicating the behaviour of the AI field as a whole).

I find it useful to think about how people solve problems and examine the huge gulf between specific learning techniques and these approaches. For example, to replicate a Bayesian AI researcher an AI needs to take a small amount of data, an incomplete informal model of the process that generates it (e.g. based on informal metaphors of physical processes the author is familiar with) and then find a way of formalising this informal model (so that its behaviour under all conditions can be calculated) and possibly doing some theorem proving to investigate properties of the model. They then apply potentially standard techniques to determine the models parameters and judge its worth based on experiment (potentially repeating the whole process if it doesn't work).

By focusing on Bayesian approaches we aren't developing techniques that can replicate these kinds of lateral and creative thinking behaviour. Saying there is only one valid form of inference is absurd because it doesn't address these problems.

I feel that trying to force our problems to suit our tools is unlikely to make much progress. For example, unless we can model (and therefore largely solve) all of the problems we want an AI to address we can't create a "Really Good Model".

Rather than manually developing formalisations of specific forms of similarity we need an algorithm to learn different types of similarity and then construct the formalisation itself (or not as I don't think we actually formalise our notions of similarity and yet can still solve problems).

Automated theorem proving is a good example where the problems are well defined yet unique, so any algorithm that can construct proofs needs to see meta patterns in other proofs and apply them. This brings home the difficulty of identifying what it means for things to be similar and also emphasises the incompleteness of a probabilistic approach: the proof that the AI is trying to construct has never been encountered before, in order for it to benefit from experience it needs to invent a type of similarity to map the current problem to the past.

Eh not impossible... just very improbable (in a given world) and certain across all worlds.

I would have thought the more conventional explanation is that the other versions are not actually you (just very like you). This sounds like the issue of only economists acting in the way that economists model people. I would suspect that only people who fixate on such matters would confuse a copy with themselves.

I suspect that people who are vulnerable to these ideas leading to suicide are in fact generally vulnerable to suicide. There are lots of better reasons to kill yourself that most people ignore. If you think you're at risk of this I recommend you seek therapy, thought experiments should not have such drastic effects on your actions.

Thanks for your reference it is good to get down to some more specific examples.

Most AI techniques are model based by necessity: it is not possible to generalise from samples unless the sample is used to inform the shape of a model which then determines the properties of other samples. In effect, AI is model fitting. Bayesian techniques are one scheme for updating a model from data. I call them incomplete because they leave a lot of the intelligence in the hands of the user.

For example, in the thesis reference the author designs a model of transformations on handwritten letters that (thanks to the authors intelligence) is similar to the set of transformations applied to numeric characters. The primary reason why the technique is effective is because the author has constructed a good transformation. The only way to determine if this is true is through experimentation, I doubt the bayesian updating is contributing significantly to the results, if another scheme such as an SVM was chosen I would expect it to produce similar recognition results.

The point is that the legitimacy or otherwise of the model parameter updating scheme is relatively insignificant in comparison to the difficulty in selecting a good model in the first place. As far as I am aware, as there are a potentially infinite set of models, Bayesian techniques cannot be applied to select between them, leaving the real intelligence being provided by the user in the form of the model. In contrast, SVMs are an attempt to construct experimentally useful models from samples and so are much closer to being intelligent in the sense of being able to produce good results with limited human interaction. However, neither technique addresses the fundamental difficulty of replicating the intelligence used by the author in creating the transformation in the first place. Fixating on a particular approach to model updating when model selection is not addressed is to miss the point, it may be meaningful for gambling problems but for real AI challenges the difference it makes appears to be irrelevant to actual performance.

I would love to discuss what the real challenges of GAI are and explore ways of addressing them, but often the posts on LW seem to focus on seemingly obscure game theory or gambling based problems which don't appear to be bringing us closer to a real solution. If the model selection problem can't be addressed then there is no way to guarantee that whatever we want an AI to value, it won't create an internal model that finds something similar (like paperclips) and decides to optimise for that instead.

Silently down voting criticism of Bayesian probability without justification is not helpful either.

From what I understand, in order to apply Bayesian approaches in practical situations it is necessary to make assumptions which have no formal justification, such as the distribution of priors or the local similarity of analogue measures (so that similar but not exact predictions can be informative). This changes the problem without necessarily solving it. In addition, it doesn't address the issue of AI problems not based on repeated experience, e.g. automated theorem proving. The advantage of statistical approaches such as SVMs is that they produce practically beneficial results with limited parameters. With parameter search techniques they can achieve fully automated predictions that often have good experimental results. Regardless of whether Bayesianism is the law of inference, if such approaches cannot be applied automatically they are fundamentally incomplete and only as valid as the assumptions they are used with. If Bayesian approaches carry a fundamental advantage over these techniques why is this not reflected in their practical performance on real world AI problems such as face recognition?

Oh and bring on the down votes you theory loving zealots :)

Thank you very much for your great reply. I'll look into all of the links. Your comments have really inspired me in my exploration of mathematics. They remind me of the aspect of academia I find most surprising. How it can so often be ideological, defensive and secretive whilst also supporting those who sincerely, openly and fearlessly pursue the truth.

Thank you, my main goal at the moment is to get a handle on statistical learning approaches and probability. I hope to read Jaynes's book and the nature of statistical learning theory once I have some time to devote to them. however I would love to find an overview of mathematics. Particularly one which focuses on practical applications or problems. One of the other posts mentioned the Princeton companion to Mathematics and that sounds like a good start. I think what I would like is to read something that could explain why different fields of mathematics were important, and how I would concretely benefit from understanding them.

At the moment I have a general unease about my partial mathematical blindness, I understand the main mathematical ideas underlying the work in my own field (computer vision) and I'm pretty happy with the subjects in numerical recipes and some optimisation theory. I'm fairly sure that I don't need to know more, but it bothers me that I don't. At the same time I don't want to spend a lot of time wading through proofs that are unlikely to ever be relevant to me. I have also yet to find a concrete example in AI where an engineering approach with some relatively simple applied maths has been substantially weaker than an approach that requires advanced mathematical techniques, making me suspect that mathematics is as it is because it appeals to those who like puzzles, rather than necessarily providing profound insight into a problem. Although I'd love to be proved wrong on that point.

So, assuming survival is important, a solution that maximises survival plus wireheading would seem to solve that problem. Of course it may well just delay the inevitable heat death ending but if we choose to make that important, then sure, we can optimise for survival as well. I'm not sure that gets around the issue that any solution we produce (with or without optimisation for survival) is merely an elaborate way of satisfying our desires (in this case including the desire to continue to exist) and thus all FAI solutions are a form of wireheading.

One frustration I find with mathematics is that it is rarely presented like other ideas. For example, few books seem to explain why something is being explained prior to the explanation. They don't start with a problem, outline its solution provide the solution and then summarise this process at the end. They present one 'interesting' proof after another requiring a lot of faith and patience from the reader. Likewise they rarely include grounded examples within the proofs so that the underlying meaning of the terms can be maintained. It is as if the field is constructed so that it is in the form of puzzles rather than providing a sincere attempt to communicate idea as clearly as possible. Another analogy would be programming without the comments.

A book like Numerical Recipies, or possibly Jaynes book on probability, is the closest I've found so far. Has anyone encountered similar books?

I'm not sure I understand the distinction between an answer that we would want and a wireheading solution. Are not all solutions wireheading with an elaborate process to satisfy our status concerns. I.e. is there a real difference between a world that satisfies what we want and directly altering what we want? If the wire in question happens to be an elaborate social order rather than a direct connection why is that different? What possible goal could we want pursued other than the one which we want?

Ok, so how about this work around.

The current approach is to have a number of human intelligences continue to explore this problem until they enter a mental state C (for convinced they have the answer to FAI). The next stage is to implement it.

We have no other route to knowledge other than to use our internal sense of being convinced. I.e. no oracle to tell us if we are right or not.

So what if we formally define what this mental state C consists of and then construct a GAI which provably pursues only the objective of creating this state. The advantage being that we now have a means of judging our progress because we have a formally defined measurable criteria for success. (In fact this process is a valuable goal regardless of the use of AI but it now makes it possible to use AI techniques to solve it).