What Blueberry said. The page you linked just gives the standard program for solving Towers of Hanoi. What JamesAndrix was imagining was a program that comes up with that solution, given just the description of the problem -- i.e., what the human coder did.

The only way in which we can access an abstract mathematical fact, learn about its truth, is through some physical tools, such that we believe their behavior to be linked to the abstract statement.

In this case, we believe that if 5324 + 2326 = 7650, then adding 5324 apples and 2326 apples gives 7650 apples; and if 5324 + 2326 = 9650, then adding 5324 apples and 2326 apples gives 9650 apples; and if adding 5324 apples and 2326 apples gives 7650 apples, then 5324 + 2326 = 7650.

If instead of normal apples, we have magical apples that don't add, then we probably wouldn't believe that this particular abstract statement is related to the apples, and that observing the apples tells us something about the statement. On the other hand, we'd still need some tool to access the abstract fact in order to reason about it, perhaps a brain or weird-physics-based calculator if no simple objects comply. If such is not available, then we can't answer (or even ask) the question.

There was a thought related to this that I've been trying to untangle. Assuming the brain is computable, which seems to me to be quite a safe assumption, doesn't that imply that all of mathematics should be able to be framed computationally? It seems obvious to me that there should be some algorithm that could create mathematics by organizing a stream of input data in some fashion. After all, there is a (presumably) computable subsystem of Earth that does that, namely the mathematics community.

That said, in what light are we to consider "incomputable" mathematics? It seems like this is drawing a meaningless distinction given the computable universe hypothesis. Perhaps it can be thought of as a sort of counterfactual reasoning? This certainly seems to be the case with, say, accelerated Turing machines, the counterfactual being "what if there were no universal speed cap?"

Apologies if I'm not making sense, I've been awake for quite a while.

Since being introduced to Less Wrong and clarifying that 'truth' is a property of beliefs corresponding to how accurately they let you predict the world, I've separated 'validity' from 'truth'.

The syllogism "All cups are green; Socrates is a cup; therefore Socrates is green" is valid within the standard system of logic, but it doesn't correspond to anything meaningful. But the reason that we view logic as more than a curiosity is that we can use logic and true premises to reach true conclusions. Logic is useful because it produces true beliefs.

Some mathematical statements follow the rules of math; we call them valid, and they would be just as valid in any other universe. Math as a system is useful because (in our universe) we can use mathematical models to arrive at predictively accurate conclusions.

Bringing 'truth' into it is just confusing.

This has not been my experience in mathematics. There's lots and lots of theorems that I know where I don't have any physical counterpart for the claim.

You have your physical brain (I did mention this option in the comment). The "counterpart" can follow a syntactic description of the semantic fact, for example, which is how the axiomatic method normally works (but there are other options, the important thing is that you (correctly) believe the tool to be controlled by the abstract fact, as you believe your mathematical reasoning to be controlled by abstract facts).

Sean Carroll has a nice explanation of the general idea on his blog that falls somewhere between newscientist's uninformativeness and the usual layman-inaccessible arxiv article.

I'd like to see the inductive bias / max-ent stuff fleshed out with some math. Any pointers?

(I find new obvious things everywhere after the recent realization that any explicit consideration an agent knows is subject to whole agent's judgment, even "preference" or "logical correctness". This also explains a bit of our talking past each other in the other thread.)

I don't have much idea what you mean here. This seems important enough to write up as more than a parenthetical remark.

...and he also already had a talk at Oxford these days at FHI's Winter Intelligence Conference: http://www.fhi.ox.ac.uk/events_data/winter_conference

Videos should be available soon, and I must admit I'm a bit more eager to hear what he and others had to say at this gathering than what his forthcoming talk shall bring us...

Being in a similar position (also as far as aversion to moving to e.g. US is concerned), I decided to work part time (roughly 1/5 of the time of even less) in software industry and spend the remainder of the day studying relevant literature, leveling up etc. for working on the FAI problem. Since I'm not quite out of the university system yet, I'm also trying to build some connections with our AI lab staff and a few other interested people in the academia, but with no intention to actually join their show. It would eat away almost all my time, so I could work on some AI-ish bio-informatics software or something similarly irrelevant FAI-wise.

There are of course some benefits in joining the academia, as you mentioned, but it seems to me that you can reap quite a bit of them by just befriending an assistant professor or two.