This seems to be a common response - Tyrrell_McAllister said something similar:

I think that your distinction is really just the distinction between physics and mathematics.

I take that distinction as meaning that a precise maths statement isn't necessarily reflecting reality like physics does. That is not really my point.

For one thing, my point is about any applied maths, regardless of domain. That maths could be used in physics, biology, economics, engineering, computer science, or even the humanities.

But more importantly, my point concerns what you think the equations are about, and how you can be mistaken about that, even in physics.

The following might help clarify.

A successful test of a mathematical theory against reality means that it accurately describes some aspect of reality. But a successful test doesn't necessarily mean it accurately describes what you think it does.

People successfully tested the epicycles theory's predictions about the movement of the planets and the stars. They tended to think that this showed that the planets and stars were carried around on the specified configuration of rotating circles, but all it actually showed was that the points of light in the sky followed the paths the theory predicted.

They were committing a mind projection 'fallacy' - their eyes were looking at points of light but they were 'seeing' planets and stars embedded in spheres.

The way people interpreted those successful predictions made it very hard to criticise the epicycles theory.

Software programs for individuals.... prime association formation at a later time.... some short-term memory aid that works better than scratch paper

I have been obsessively researching this idea for several years. One of my conclusions is that an intelligence-amplification tool must be "incestuously" user-modifiable ("turtles all the way down", possessing what programming language designers call reflectivity) in order to be of any profound use, at least to me personally.

Or just biting the bullet and learning Mathematica to an expert level instead of complaining about its UI

About six months ago, I resolved to do exactly that. While I would not yet claim "black belt" competence in it, Mathematica has already enabled me to perform feats which I would not have previously dared to contemplate, despite having worked in Common Lisp. Mathematica is famously proprietary and the runtime is bog-slow, but the language and development environment are currently are in a class of their own (at least from the standpoint of exploratory programming in search of solutions to ultra-hard problems.)

While I would not yet claim "black belt" competence in it, Mathematica has already enabled me to perform feats which I would not have previously dared to contemplate, despite having worked in Common Lisp. Mathematica is famously proprietary and the runtime is bog-slow, but the development environment is currently in a class of its own (at least from the standpoint of exploratory programming in search of solutions to ultra-hard problems.)

Sounds cool, but this is not quite what I was aiming at.

I suppose the question is not whether it would be good, but rather how. Some quick brainstorming:

• I think people are "smarter" now then they were, say, pre-scientific-method. So there may be more trainable ways-of-thinking that we can learn (for example, "best practices" for qualitative Bayesianism)

• Software programs for individuals. Oh, maybe when you come across something you think is important while browsing the web you could highlight it and these things would be presented to you occasionally sort of like a "drill" to make sure you don't forget it, or prime association formation at a later time. Or some kind of software aid to "stack unwinding" so you don't go to sleep with 46 tabs open in your web browser. Or some short-term memory aid that works better than scratch paper. Or just biting the bullet and learning Mathematica to an expert level instead of complaining about its UI. Or taking a cutting-edge knowledge representation framework like Novamente's PLN and trying to enter stuff into it as an "active" note-taking system.

• Collaboration tools -- shared versions of the above ideas, or n-way telephone conversations, or freeform "chatroom"-style whiteboards or iteratively-refined debate thesis statements, or lesswrong.com

• Man-machine hybrids. Like having people act as the utility function or search-order-control of an automated search process.

Of course, neural prostheses may become possible at some point fairly soon. Specially-tailored virtual environments to aid in visualization (like of nanofactories), or other detailed and accurate scientific simulations allowing for quick exploration of ideas... "Do What I Mean" interfaces to CAD programs might be possible if we can get a handle on the functional properties of human cognitive machinery...

Or: "Physics is not Math"

Doesn't seem to deserve more mention than the creation of computing? Sure. But computing has already been created.

Not exactly, Thom. Roughly, for FAI you need precise self-modification. For precise self-modification, you need a precise theory of the intelligence doing the self-modification. To get to FAI you have to walk the road that leads to precise theories of intelligence - something like our present-day probability theory and decision theory, but more powerful and general and addressing issues these present theories don't.

Eurisko is the road of self-modification done in an imprecise way, ad-hoc, throwing together whatever works until it gets smart enough to FOOM. This is a path that leads to shattered planets, if it were followed far enough. No, I'm not saying that Eurisko in particular is far enough, I'm saying that it's a first step along that path, not the FAI path.

A lot of this probably comes down to:

Don’t assume – that you have a rich enough picture of yourself, a rich enough picture of the rest of reality, or that your ability to mentally trace through the consequences of actions comes anywhere near the richness of reality’s ability to do so.

Interesting exercise. After trying for a while I completely failed; I ended up with terms that are completely vague (e.g. "comfort"), and actually didn't even begin to scratch the surface of a real (hypothesized) utility function. If it exists it is either extremely complicated (too complicated to write down perhaps) or needs "scientific" breakthroughs to uncover its simple form.

The result was also laughably self-serving, more like "here's roughly what I'd like the result to be" than an accurate depiction of what I do.

The real heresy is that this result does not particularly frighten or upset me. I probably can't be a "rationalist" when my utility function doesn't place much weight on understanding my utility function.

Can you write your own utility fuinction or adopt the one you think you should have? Is that sort of wholesale tampering wise?

People on this site love to use fiction to illustrate their points, and a "biomoderate singularity managed by a superintelligent singleton" is very novel-friendly, so that's something!