Model selection is definitely one of the biggest conceptual problems in GAI right now (I would say that planning once you have a model is of comparable importance / difficulty). I think the way to solve this sort of problem is by having humans carefully pick a really good model (flexible enough to capture even unexpected situations while still structured enough to make useful predictions). Even with SVMs you are implicitly assuming some sort of structure on the data, because you usually transform your inputs into some higher-dimensional space consisting of what you see as useful features in the data.

Even though picking the model is the hard part, using Bayes by default seems like a good idea because it is the only general method I know of for combining all of my assumptions without having to make additional arbitrary choices about how everything should fit together. If there are other methods, I would be interested in learning about them.

What would the "really good model" for a GAI look like? Ideally it should capture our intuitive notions of what sorts of things go on in the world without imposing constraints that we don't want. Examples of these intuitions: superficially similar objects tend to come from the same generative process (so if A and B are similar in ways X and Y, and C is similar to both A and B in way X, then we would expect C to be similar to A and B in way Y, as well); temporal locality and spatial locality underly many types of causality (so if we are trying to infer an input-output relationship, it should be highly correlated over inputs that are close in space/time); and as a more concrete example, linear momentum tends to persist over short time scales. A lot of work has been done in the past decade on formalizing such intuitions, leading to nonparametric models such as Dirichlet processes and Gaussian processes. See for instance David Blei's class on Bayesian nonparametrics (http://www.cs.princeton.edu/courses/archive/fall07/cos597C/index.html) or Michael Jordan's tutorial on Dirichlet processes (http://www.cs.berkeley.edu/~jordan/papers/pearl-festschrift.pdf).

I'm beginning to think that a top-level post on how Bayes is actually used in machine learning would be helpful. Perhaps I will make on when I have a bit more time. Also, does anyone happen to know how to collapse URLs in posts (e.g. the equivalent of <a href=...>test </a> in HTML).

I'm a MWI cynic*, so here is my approach.

There is an interpretation of quantum mechanics called the many-world interpretation that rejects the wave-function collapse of the classical interpretation.

This interpretation leads some people to the belief that all possible alternative histories are real. Everything that can happen does happen. They assume that if you die in this world, you would also continue living on in another world. Since there must always be a chance you won't die, then there must be a world in which you live forever.

The problem with this belief is that at best it applies only for simple quantum systems, and generally not to events as large and complex as a person's death. To avoid death, a very large number of quantum level alternatives have to be simultaneously selected for. The probability of this simultaneous selection will be effectively zero.

In MWI terms this means that you die in all worlds, except the impossible ones.

Value the life you have and don't depend on quantum immortality.

* I like the MWI on lack of wave-function collapse and on quantum decoherence, but I don't think that the idea of separate worlds is necessary.

Bayesian approaches tend to be more powerful than other statistical techniques in situations where there is a relatively limited supply of data. This is because Bayesian approaches, due to being model-based, tend to have a richer structure that allows it to take advantage of more of the structure of the data; a second reason is because Bayes allows for the explicit integration of prior assumptions and is therefore usually a more aggressive form of inference than most frequentist methods.

I tried to find a good paper demonstrating this (called "learning from one example"), unfortunately I only came across this PhD thesis --- http://www.cs.umass.edu/~elm/papers/thesis.pdf , although there is certainly a lot of work being done on generalizing from one, or a small number of, examples.

I think the fundamental insight of Bayesianism is that Bayes' Theorem is the law of inference, not (just) a normative law but a descriptive law — that frequentist methods and other statistical algorithms that make no mention of Bayes aren't cleverly circumventing it, they're implicitly using it. Any time you use some data to generate a belief about some proposition, if you use a method whose output is systematically correlated with reality at all, then you are using Bayes, just with certain assumptions and simplifications mixed in.

The failing of frequentism is not in the specific methods it uses — it is perfectly true that we need simplified methods in order to do much useful inference — but in its claim of "objectivity" that really consists of treating its assumptions and simplifications as though they don't exist, and in its reliance on experimenters' intuition in deciding which methods should be used (considering that different methods make different assumptions that lead to different results). Frequentist methods aren't (all) bad, frequentist epistemology is.

If I remember correctly, it is perfectly possible to create Bayesian formulations of most frequentist methods; of course, they will often still talk about things that Bayesians don't usually care about, like P-values, but they will nevertheless reveal the deductively-valid Bayes-structure of the path from your data to that result. Revealing frequentist methods' hidden structure is important because it lets us understand why they work — when they do work — and it lets us predict when they won't be as useful.

Upvoted for a thoughtful comment.

1. I don't know anything about statistical learning theory.

2. I don't know what kinds of probability you're interested in learning, but would recommend Concrete Mathematics: A Foundation for Computer Science by Graham, Knuth and Patashnik and William Feller's two volume set An Introduction to Probability Theory and Its Applications.

3. I would second the recommendation of the Princeton Companion to Mathematics but would also warn it does not go into enough depth for one to get an accurate understanding of what many of the subjects discussed therein are about. This is understandable given space constraints.

4. The edifice of pure mathematics is vast and the number of people alive who could give a good overview of existing mathematics as a whole is tiny and possibly zero.

5. As a matter of practice, much of the information about how mathematicians learn and think about a given subject is never recorded. See this comment by SarahC and Bill Thurston's MathOverflow question Thinking and Explaining.

6. On average I've found reading math books that adopt a historical approach to the material therein to be considerably more useful than reading math books that adopt an axiomatic approach to the material therein.

7. Based on my (limited) impression of applied math, it's not uncommon for people to use advanced mathematical techniques to solve a practical problem because doing so makes for a good marketable story rather than because the advanced mathematical techniques are genuinely useful to analyzing the practical problem at hand.

8. There is an issue of a high noise-to-signal ratio in mathematics textbooks corresponding to the fact that many authors of textbooks don't have the depth of understanding of the creators of the theories that they're writing about and correspondingly do not emphasize the key points.

9. Concerning your suspicion that "mathematics is as it is because it appeals to those who like puzzles, rather than necessarily providing profound insight into a problem" - there's great variability among mathematicians here. Two essays which discuss dichotomies which are not identical to the one that you draw but which I think you'll find peripherally relevant are Timothy Gowers' The Two Cultures of Mathematics and Freeman Dyson's Birds and Frogs.

10. Those mathematicians who seek profound insight into problems often seek profound insight into problems within pure math rather than problems that arise in engineering.

11. Looking at your website, you might find it useful to check out the Brown University Pattern Theory Group. I don't have any subject matter knowledge of what they do, but the group includes David Mumford who is of extremely high caliber, having earned a Fields Medal in the 1970's for his work on algebraic geometry.

12. While I don't know enough to point you in the right direction to help you with your research, if you're interested in learning about pure math out of general intellectual curiosity then there are many books that I can recommend.

I agree with your remarks here and share your frustration. While books of the type that you're looking for are relatively uncommon; over the years I've amassed a list of ones that I've found very good. What subject(s) are you interested in learning? (N.B. There are large parts of math that I'm ignorant of - in particular I don't know almost anything about applied math and so may not be able to say anything useful - I just thought I'd ask in case I can help.)

is there a real difference between a world that satisfies what we want and directly altering what we want?

From an evolutionary point of view those things that manage to procreate will out compete those things that change themselves to not care about that and just wirehead.

So in non-singleton situations, alien encounters and any form of resource competition it matters whether you wirehead or not. Pleasure, in an evolved creature, can be seen as the giving (very poor) information on the map to the territory of future influence for the patterns that make up you.

One frustration I find with mathematics is that it is rarely presented like other ideas. For example, few books seem to explain why something is being explained prior to the explanation. They don't start with a problem, outline its solution provide the solution and then summarise this process at the end. They present one 'interesting' proof after another requiring a lot of faith and patience from the reader. Likewise they rarely include grounded examples within the proofs so that the underlying meaning of the terms can be maintained. It is as if the field is constructed so that it is in the form of puzzles rather than providing a sincere attempt to communicate idea as clearly as possible. Another analogy would be programming without the comments.

A book like Numerical Recipies, or possibly Jaynes book on probability, is the closest I've found so far. Has anyone encountered similar books?

Try to figure out what maximizes this estimate method. It won't be anything you'd want implemented, it will be a wireheading stimulus.

I'm not sure that there is a verbal description of a possible world that is also a wirehead stimulus for me. There might be, which might be enough to discount this method.

And questions about possible worlds involve quantifies of data that a mere human can't handle.

True.

Interesting, if I understand correctly the idea is to find a theoretically correct basis for deciding on a course of action given existing knowledge and then to make this calculation efficient and then direct towards a formally defined objective.

Yes, but there is only one top-level objective, to do the right thing, so one doesn't need to define an objective separately from the goal system itself (and improving state of knowledge is just another thing one can do to accomplish the goal, so again not a separate issue).

FAI really stands for a method of efficient production of goodness, as we would want it produced, and there are many landmines on this path, in particular humanity in its current form doesn't seem to be able to retain its optimization goal in the long run, and the same applies to most obvious hacks that don't have explicit notions of preference, such as upload societies. It's not just a question of speed, but also of ability to retain the original goal after quadrillions of incompletely understood self-modifications.