## Exploration-Exploitation problems

Original post: http://bearlamp.com.au/exploration-exploitation-problems/

I have been working on the assumption that exploration-exploitation knowledge was just common. Unfortunately I did the smart thing of learning about them from a mathematician at a dojo in Melbourne, which means that no. Not everyone knows about it. I discovered that again today when I searched for a good quick explanation of the puzzle. With that in mind this is about **Exploration Exploitation**.

The classic Exploration-Exploitation problem in mathematics is the multi-armed bandit. Which is a slang term for a bank of slot machines. Where the player knows that each machine has a variable payoff and you have a limit number of attempts before you run out of money. You want to balance *trying out new machines* with unknown payoffs against exploiting the *knowledge you already have* from the earlier machines you tried.

When you first start on new bandits, you really don't know which will pay out and at what rates. So some exploration is necessary to know what your reward ratio in the territory will be. As your knowledge grows, you get to know which bandits are likely to pay, and which are not, and this later informs your choices as to where to place your dollars.

Mathematicians love a well specified problem like this because it allows us to make algorithm models of patterns that will return rewards or guarantee rewards given certain circumstances. (see also - the secretary problem which does similar. Where I showed how it applied to real life dating)

Some of the mathematical solutions to this problem look like:

**Epsilon greedy** - The best lever is selected for a proportion 1-ε of the trials, and a lever is selected at random (with uniform probability) for a proportion ε. A typical parameter value might be ε =0.1 but this can vary widely depending on circumstances.

**Epsilon-decreasing strategy**: Similar to the epsilon-greedy strategy, except that the value of ε decreases as the experiment progresses, resulting in highly exploratory behaviour at the start and highly exploitative behaviour at the finish.

Of course there are more strategies, and the context and nature of the problem matters. If the machines suddenly one day in the future all change, you might have a strategy that would prepare for potential scenarios like that. As you start shifting away from the hypothetical and towards real life your models need to increase complexity to cater to the details of the real world.

If this problem is more like real life (where we live and breathe), the possible variability of reality starts coming in to play more and more. In talking about this - I want to emphasise not the problem as interesting, but the solution of **<sometimes explore> and <sometimes exploit>** in specific ratios or for specific reasons. The mathematical solutions the the multi-armed bandit problem are used in such a way to take advantage of the balance between *not knowing enough* and *taking advantage of what you do know*.

What supercharges this solution and how it can be applied to real life is value of information.

Value of Information says that in relation to making a decision, what informs that decision is *worth something*. With expensive decisions, risky decisions, dangerous decisions, highly lucrative decisions, or particularly unknown decisions being more sure is important to think about.

VoI suggests that any decision that is worth money (or worth something) can have information that informs that decision. The value of information can add up to the value of the reward on correctly making the decision. Of course if you spend all the potential gains from the decision on getting the perfect information you lose the chance to make a profit. However usually a cheap (relative to the decision) piece of information exists that will inform the decision and assist.

How does this apply to exploration-exploitation?

The idea of VoI is well covered in the book, how to measure anything. While the book goes into detail and is really really excellent for applying to big decisions, the ideas can also be applied to our simple every day problems as well. With this in mind I propose a heuristic:

**You want to explore as much as to increase your information with regard to both the quality of the rest of the exploration and possible results and the expected returns on the existing knowledge.**

The next thing to supercharge our exploration-exploitation and VoI knowledge is Diminishing returns.

Diminishing returns on VoI is when you start out *not knowing anything at all*, and adding a little bit of knowledge goes a long way. As you keep adding more and more information the return on the extra knowledge has a diminishing value.

Worked example: Knowing the colour of the sky.

So you are blind and no one has ever told you what colour the sky is. You can't really be sure what colour the sky is but generally if you ask enough people the consensus should be a good enough way to conclude the answer.

So one guy gave you your first inkling of what the answer is. But can you really trust him?

Yea cool. Ten people. Probably getting sure of yourself now.

Really, what good is Two Thousand people after the first fifty? Especially if they all agree. There's got to be less value of the 2001st person telling you than there was the 3rd person telling you.

Going back to VoI, how valuable was the knowledge that the sky is blue? Probably not very valuable, and this isn't a great way to gather knowledge in the long run.

The great flaw with this is also if I asked you the question - "what colour is the sky?" you could probably hint as to a confident guess. If you are a well calibrated human, you already know a little bit of everything and the good news is that calibration is trainable.

With that in mind; if you want to play a calibration game there are plenty available on google.

The great thing about calibration is that it seems to apply across all your life, and all things that you estimate. Which is to say that *once you are calibrated, you are calibrated across domains*. This means that if you become good at it in one area, you become better at it in other areas. We're not quite talking about hitting the bullseye every time, but we are talking about being confident that the bullseye is over there in that direction. Which is essentially the ability to predict the future within a reasonable set of likelihoods.

Once you are calibrated, you can take calibration, use it to apply diminishing returns through VoI to supercharge your exploration exploitation. But we're not done. What if we add in Bayesian statistics? What if we can shape our predicted future and gradually update our beliefs based on tiny snippits of data that we gather over time and purposefully by thinking about VoI, and the diminishing returns of information.

I don't want to cover Bayes because people far smarter than me have covered it very well. If you are interested in learning bayes I would suggest heading to Arbital for their excellent guides.

But we're not done at bayes. This all comes down to the idea of trade-offs. Exploration VS exploitation is a trade off of {time/energy} vs expected reward.

A classic example of a trade-off is a story of Sharpening the Saw (from the book *7 habits of highly effective people*)

A woodcutter strained to saw down a tree. A young man who was watching asked “What are you doing?”

“Are you blind?” the woodcutter replied. “I’m cutting down this tree.”

The young man was unabashed. “You look exhausted! Take a break. Sharpen your saw.”

The woodcutter explained to the young man that he had been sawing for hours and did not have time to take a break.

The young man pushed back… “If you sharpen the saw, you would cut down the tree much faster.”

The woodcutter said “I don’t have time to sharpen the saw. Don’t you see I’m too busy?”

The thing about life and trade offs is that all of life is trade-offs between things you want to do and other things you want to do.

Exploration and exploitation is a trade off between the value of **what you know** and the value of **what you might know if you find out**.

**Try this:**

- Make a list of all the things you have done over the last 7 days. (Use your diary and rough time chunking)
- Sort them into exploration activities and exploitation activities.

Answer this: - Am I exploring enough? (on a scale of 1-10)
- Am I exploiting enough? (on a scale of 1-10)
- Have I turned down any exploring opportunities recently?
- Have I turned down any exploitation opportunities recently?
- How could I expand any exploring I am already doing?
- How could I expand any exploiting I am already doing?
- How could I do more exploring? How could I do less exploring?
- How could I do more exploiting? How could I do less exploiting?

There are two really important things to take away from the Exploration-Exploitation dichotomy:

- You probably make the most measurable and ongoing gains in the Exploitation phase. I mean - lets face it, these are long running goal-seeking behaviours like sticking to an exercise routine.
- The exploration might be seem more fun (find exciting and new hobbies) but are you sure that's what you want to be doing in regard to 1?

Meta: This is part 1 of a 4 part series.

This took in the order of 10-15 hours to finish because I was doing silly things like trying to fit 4 posts into 1 and stumbling over myself.

## How I use Anki to learn mathematics

# The rules I follow

## Decks.

## Kind of cards.

### Definition

### Theorem

#### Hypothesis

#### Multiple implications

This leads me to another problem. An hypothesis may have many implications. And a property may be implied by many hypothesis. Therefore, it is better to state:

#### Order of the words

### Example

# Learning

## Truth is holistic

You already know by now that truth is undefinable: by a famous result of Tarski, no formal system powerful enough (from now on, just system) can consistently talk about the truth of its own sentences.

You may however not know that Hamkins proved that truth is holistic.

Let me explain: while no system can talk about its own truth, it can nevertheless talk about the truth of its own *substructures*. For example, in every model of ZFC (the standard axioms of set theory) you can consistently define a model of standard arithmetics *and *a predicate that works as arithmetics' truth predicate. This can happen because ZFC is strictly more powerful than PA (the axioms of standard arithmetics).

Intuitively, one could think that if you have the same substructure in two different models, what they believe is the truth about that substructure is the same in both. Along this line, two models of ZFC ought to believe the same things about standard arithmetics.

However, it turns out this is not the case. Two different models extending ZFC may very well agree on which entities are standard natural numbers, and yet still disagree about which arithmetic sentences are true or false. For example, they could agree about the standard numbers, how the successor and addition operator works, and yet disagree on multiplication (corollary 7.1 in Hamkins' paper).

This means that when you can talk consistently about the truth of a model (that is, when you are in a more powerful formal system), that truth depends not only on the substructure, but on the entire structure you're immersed in. Figuratively speaking, local truth depends on global truth. Truth is holistic.

There's more: suppose that two model agree on the ontology of some common substructure. Suppose also that they agree about the truth predicate on that structure: they could still disagree about the meta-truths. Or the meta-meta-truths, etc., for all the ordinal levels of the definable truth predicates.

Another striking example from the same paper. There are two different extensions of set theory which agree on the structure of standard arithmetics and on the members of a subset A of natural numbers, and yet one thinks that A is first-order definable while the other thinks it's not (theorem 10).

Not even "being a model of ZFC" is an absolute property: there are two models which agree on an initial segment of the set hierarchy, and yet one thinks that the segment is a model of ZFC while the other proves that it's not (theorem 12).

Two concluding remarks: what I wrote was that there are different models which disagrees the truth of standard arithmetics, not that every different model has different arithmetic truths. Indeed, if two models have access one to the truth relation of the other, then they are bound to have the same truths. This is what happens for example when you prove absoluteness results in forcing.

I'm also remembered of de Blanc's ontological crises: changing ontology can screw with your utility function. It's interesting to note that updating (that is, changing model of reality) can change what you believe even if you don't change ontology.

## Consistent Plato

*A putative new idea for AI control; index here. **The idea here is somewhat underdeveloped.*

I've been suggesting AI designs that are not exactly expected-utility maximisers, while wondering if such designs could be stable under self improvement/subagent creation/general stuff. And also wondering what designs such agents would converge to if they did self-improve.

This post is just a short note warning people to make sure they are consistent when they use non-expected utility maximalisation, and to be clear which parts are Platonic and idealised and which parts are not.

Let's take the reduced impact AI design as an example. In this, the agent has a kind of utility:

v = u - R,

where u is a standard utility and R is a penalty term. The penalty term involved a probability estimate for what a putative "super AI" would estimate, based on some observations it could make. Thus it is better described as R(P), where P is the probability estimator of that "Super AI".

When calculating expected v, the agent will use its own probability estimator P'. Initially, P' is just a less powerful version of P, one that can, moreover, estimate properties about P.

How could this possibly go wrong? I'm glad you asked ^_^ The obvious risk is that the agent acts to replace P with Q, such that R(Q) is as low as possible.

This could come about if P referred to some actual physical algorithm, possibly one written down somewhere the AI could get at it. Then re-writing the algorithm P could increase v by reducing R(P).

It is somewhat more difficult if P and P' are defined in the same place (maybe P is just P' run for a lot longer). Then modifying P could run the risk of degrading the AI's overall accuracy in predictions. However, this may be a price the AI is willing to pay, if the R(P) penalty is severe enough. More likely, the AI could degrade the quality of P but not P' by introducing pathologies that would only affect the first.

Even if P itself is safe from modification, the AI can still hack the system. Recall that it does not calculate P, it estimates it, using P' - so effectively using P'(P). Thus modifications of P' can affect the AI's estimate of R(P). Even if P and P'(P) cannot be changed, the AI can still improve P' for predictions of u, while leaving P'(P) unchanged. Thus the AI may become super-super-intelligent at predicting facts about the world, while remaining (deliberately) only super-intelligent at predicting the penalty R(P). This could allow it to use its great intelligence to hack or trick the R(P) term.

The main requirement to avoid all this is that there is no "first-step divergence". That there never comes a time where the agent modifies itself in such a way that the initial relation between P and P' comes apart.

To avoid this, it might be best to treat both P and P' as different approximations to the same idealised platonic process (Kolmogorov complexity priors calculated by hypercomputation, possibly?). Then the two algorithms are actually strongly related (you can't really change them "independently"), and the AI is trying to minimise the true value of R(P), not the value it records. Self-modifications to P and P' will only work if this allows a *better *estimate of R(P), no necessarily a *lower *estimate.

Being inconsistent about what is platonic (in theory or in practice - some processes are essentially platonic because we can't affect them) and what isn't can lead to problems and unstable motivation systems.

## What's special about a fantastic outcome? Suggestions wanted.

I've been returning to my "reduced impact AI" approach, and currently working on some idea.

What I need is some ideas on features that might distinguish between an excellent FAI outcome, and a disaster. The more abstract and general the ideas, the better. Anyone got some suggestions? Don't worry about quality at this point, originality is more prized!

I'm looking for something generic that is easy to measure. At a crude level, if the only options were "papercliper" vs FAI, then we could distinguish those worlds by counting steel content.

So basically some more or less objective measure that has a higher proportion of good outcomes than the baseline.

## [LINK] 2014 Fields Medals and Nevanlinna Prize anounced

http://www.mathunion.org/general/prizes/2014

On August 13, 2014, at the opening ceremony of the [International Congress of Mathematicians](http://www.icm2014.org)) the Fields Medals, the Nevanlinna Prize and several other prizes were announced.

A full list of awardees with short citations:

Fields medals:

Artur Avila

is awarded a Fields Medal for his profound contributions to dynamical systems theory, which have changed the face of the field, using the powerful idea of renormalization as a unifying principle.

Quanta Magazine on Artur Avila

Manjul Bhargava

is awarded a Fields Medal for developing powerful new methods in the geometry of numbers, which he applied to count rings of small rank and to bound the average rank of elliptic curves.

Quanta Magazine on Manjul Bhargava

Martin Hairer

is awarded a Fields Medal for his outstanding contributions to the theory of stochastic partial differential equations, and in particular for the creation of a theory of regularity structures for such equations.

Quanta Magazine on Martin Hairer

Maryam Mirzakhani

is awarded the Fields Medal for her outstanding contributions to the dynamics and geometry of Riemann surfaces and their moduli spaces.

Quanta Magazine on Maryam Mirzakhani

Nevalinna prize:

Subhash Khot

is awarded the Nevanlinna Prize for his prescient definition of the “Unique Games” problem, and leading the effort to understand its complexity and its pivotal role in the study of efficient approximation of optimization problems; his work has led to breakthroughs in algorithmic design and approximation hardness, and to new exciting interactions between computational complexity, analysis and geometry.

Quanta Magazine on Subhash Khot

Gauss Prize:

Stanley Osher

is awarded the Gauss Prize for his influential contributions to several fields in applied mathematics, and for his far-ranging inventions that have changed our conception of physical, perceptual, and mathematical concepts, giving us new tools to apprehend the world.

Chern Medal Award:

Phillip Griffiths

is awarded the 2014 Chern Medal for his groundbreaking and transformative development of transcendental methods in complex geometry, particularly his seminal work in Hodge theory and periods of algebraic varieties.

Leelavati Prize:

Adrián Paenza

is awarded the Leelavati Prize for his decisive contributions to changing the mind of a whole country about the way it perceives mathematics in daily life, and in particular for his books, his TV programs, and his unique gift of enthusiasm and passion in communicating the beauty and joy of mathematics.

In addition to that, Georgia Benkart was announced as the 2014 ICM Emmy Noether lecturer.

It might be interesting to note a curious fact about the new group of Fields medalists:

each of them [is] a notable first for the Fields Medal: the first woman and the first Iranian, Maryam Mirzakhani; the first Canadian, Manjul Bhargava; Artur Avila, the first Brazilian; and Martin Hairer, the first Austrian to win a Fields Medal.

However, this unusual diversity of nationalities does not necessarily translate into a corresponding diversity of institutions, since (according to wikipedia) three out of four winners work in (or at least are affiliated with) universities that have already had awardees in the past.

Some notes on the works by Fields medalists can be found on Terence Tao's blog.

A related discussion on Hacker News.

## Reduced impact in practice: randomly sampling the future

*This was an idea developed along with Eliezer when he was at the FHI in November.*

In my post on "reduced impact AI" I talked about how you could estimate the impact of an AI by (hypothetically) observing a slice of the future. When Eliezer was here last November, we talked about how this could be cached out, and came up with a model.

Ignore the "output channel" from the original reduced impact post, and consider this situation:

We have two computers running in two similarly shielded rooms. One of them is a simple algorithm running through the digits of e (call it run-e), that was soon after deleted. Another is our dear old friend clippy, the AI motivated to maximise paperclips. Without removing clippy's paperclip preferences, we'd like to motivate it to have next to no impact on the world. How should we define "next to no impact"?

The obvious think is to compare it with run-e, and require it to have no more expected impact than that machine. How could this be measured? Well, the original idea was to put a stochastic process on clippy as it was turned on (inside the shielded room) and say that clippy has reduced impact if we can't find any extra evidence, from a vantage point 50 years in the future, that clippy was successfully turned on. Now put the stochastic same process on run-e and define:

Clippy has reduced impact if, from a vantage of 50 years into the future, we have no more evidence that clippy was turned on than we have of run-e being turned on.

## Examples in Mathematics

After reading Luke's interview with Scott Aaronson, I've decided to come back to an issue that's been bugging me.

Specifically, in the answer to Luke's question about object-level tactics, Scott says (under 3):

Sometimes, when you set out to prove some mathematical conjecture, your first instinct is just to throw an arsenal of theory at it. (..) Rather than looking for “general frameworks,” I look for easy special cases and simple sanity checks, for stuff I can try out using high-school algebra or maybe a five-line computer program, just to get a feel for the problem.

In a similar vein, there's the Halmos quote which has been heavily upvoted in the November Rationality Quotes:

A good stack of examples, as large as possible, is indispensable for a thorough understanding of any concept, and when I want to learn something new, I make it my first job to build one.

Every time I see an opinion expressing a similar sentiment, I can't help but contrast it with the opinions and practices of two wildly successful (very) theoretical mathematicians:

One striking characteristic of Grothendieck's mode of thinking is that it seemed to rely so little on examples. This can be seen in the legend of the so-called "Grothendieck prime". In a mathematical conversation, someone suggested to Grothendieck that they should consider a particular prime number. “You mean an actual number?” Grothendieck asked. The other person replied, yes, an actual prime number. Grothendieck suggested, “All right, take 57.” But Grothendieck must have known that 57 is not prime, right? Absolutely not, said David Mumford of Brown University. “He doesn’t think concretely." Consider by contrast the Indian mathematician Ramanujan, who was intimately familiar with properties of many numbers, some of them huge. That way of thinking represents a world antipodal to that of Grothendieck. "He never really worked on examples," Mumford observed. "I only understand things through examples and then gradually make them more abstract. I don't think it helped Grothendieck in the least to look at an example. He really got control of the situation by thinking of it in absolutely the most abstract way possible. It's just very strange. That's the way his mind worked."

(from Allyn Jackson's account of Grothendieck's life).

Saito: This is one typical point of your work. But I ﬁnd that in much of your work, by hearing one symptom you capture the central point of the problem and then give some general big framework. That’s my general impression of what you are doing.

Kontsevich: Yeah, I really don’t work on examples at such a level.

Saito: How can you work in that way?

Kontsevich: For myself sometimes I work on one or two examples, but...

Saito: You already keep some examples in mind, but still you construct theory.

Kontsevich: Yes. And generally I ﬁnd examples sometimes to be misleading. [Laughter]. Because often the properties of examples are too special, you cannot see general properties if you constantly work too much on concrete examples.

(from the IPMU interview).

Are they fooling themselves, or is there something to be learned? Perhaps it's possible to mention Gowers' Two Cultures in the answer.

###### P.S. First content post here, I would appreciate feedback.

## The Emergence of Math

In his latest post, Logical Pinpointing, Eliezer talks about the nature of math. I have my own views on the subject, but when writing about them, it became too long for a comment, so I'm posting it here in discussion.

I think it's important to clarify that I am not posting this under the guise that I am correct. I'm just posting my current view of things, which may or may not be correct, and which has not been assessed by others yet. So though I have pushed myself to write in a confident (and clear) tone, I am not actually confident in my views ... well, okay, I'm lying. I *am *confident in my views, but I know I *shouldn't* be, so I'm *trying* to pretend to myself that I'm posting this to have my mind changed, when in reality I'm posting it with the stupid expectation that you'll all magically be convinced. Thankfully, despite all that, I don't have a tendency to cling to beliefs which I have seen proven wrong, so* *I *will* change my mind when someone kicks my belief's ass. I won't go down without a fight, but once I'm dead, I'm dead.

Before I can share my views, I need to clarify a few concepts, and then I'll go about showing what I believe and why.

**Abstractions**

Abstract models are models in which some information is ignored. Take, for example, the abstract concept of the ball. I don't care if the ball is made of rubber or steel, nor do I care if it has a radius of 6.7cm or a metre, a ball is a ball. I couldn't care less about the underlying configuration of quarks in the ball, as long as it's a sphere.

Numbers are also abstractions. If I've got some apples, when I abstract this into a number, I don't care that they are apples. All I care about is how many there are. I can conveniently forget about the fact that it's apples, and about the concept of time, and the hole in my bag through which the apples might fall.

**Axiomatic Systems (for example, Peano Arithmetic)**

Edit: clarified this paragraph a bit.

I don't have much to say about these for now, except that they can all be reduced to physics. Given that mathematicians store axiomatic systems in their minds, and use them to prove things, they cannot help but be reducible to physical things, unless the mind itself is not physical. I think most LessWrongers, being reductionists, believe this, so I won't go into much more detail. I'll just say that this will be important later on.

**I Can Prove Things About Numbers Using Apples**

Let's say I have 2 apples. Then someone gives me 2 more. I now have 4. I have just shown that 2+2=4, assuming that apples behave like natural numbers (which they do).

But let's continue this hypothetical. As I put my 2 delicious new apples into my bag, one falls out through a hole in my bag. So if I count how many I have, I see 3. I had 2 to begin with, and 2 more were given to me. It seems I have just shown 2+2=3, if I can prove things about numbers using apples.

The problem lies in the information that I abstracted away during the conversion from apples to numbers. Because my conversion from apples to numbers failed to include information about the hole, my abstract model gave incorrect results. Like my predictions about balls might end up being incorrect if I don't take into account every quark that composes it. Upon observing that the apple had fallen through the hole, I would realize that an event which rendered my model erroneous had occurred, so I would abstract this new event into -1, which would fix the error: 2+2-1=3.

To summarize this section, I can "prove" things about numbers using apples, but because apples are not simple numbers (they have many properties which numbers don't), when I fail to take into account certain apple properties which will affect the number of apples I have, I will get incorrect results about the numbers.

**Apples vs Peano Arithmetic**

We know that Peano Arithmetic describes numbers very well. Numbers emerge in PA; we designed PA to do so. If PA described unicorns instead, it wouldn't be very useful. And if PA emerges from the laws of physics (we can see PA emerge in mathematicians' minds, and even on pieces of paper in the form of writing), then the numbers which emerge from PA emerge from the laws of physics. So there is nothing magical about PA. It's just a system of "rules" (physical processes) from which numbers emerge, like apples (I patched up the hole in my bag ;) ).

Of course, PA is much more convenient for proving things about numbers than apples. But they are inherently just physical processes from which I have decided to ignore most details, to focus only on the numbers. In my bag of 3 apples, if I ignore that it's apples there, I get the number 3. In SSS0, if I forget about the whole physical process giving emergence to PA, I am just left with 3.

So I can go from 3 apples to the number 3 by removing details, and from the number 3 to PA by adding in a couple of details. I can likewise go from PA to numbers, and then to apples.

**To Conclude, Predictions are "Proofs"**

From all this, I conclude that numbers** **are simply an abstraction which emerges in many places thanks to our uniform laws of physics, much like the abstract concept "ball". I also conclude that what we classify as a "prediction" is in fact a "proof". It's simply using the rules to find other truths about the object. If I predict the trajectory of a ball, I am using the rules behind balls to get more information about the ball. If I use PA or apples to prove something about numbers, I am using the rules behind PA (or apples) to prove something about the numbers which emerge from PA (or apples). Of course, the proof with PA (or apples) is much more general than the "proof" about the ball's trajectory, because numbers are much more abstract than balls, and so they emerge in more places.

So my response to this part of Eliezer's post:

Never mind the question of why the laws of physics are stable - why is logic stable?

Logic is stable for the same reasons the laws of physics are stable. Logic emerges from the laws of physics, and the laws of physics themselves are stable (or so it seems). In this way, I dissolve the question and mix it with the question why the laws of physics are stable -- a question which I don't know enough to attempt to answer.

Edit: I'm going to retract this and try to write a clearer post. I still have not seen arguments which have fully convinced me I am wrong, though I still have a bit to digest.

## Friendly AI and the limits of computational epistemology

Very soon, Eliezer is supposed to start posting a new sequence, on "Open Problems in Friendly AI". After several years in which its activities were dominated by the topic of human rationality, this ought to mark the beginning of a new phase for the Singularity Institute, one in which it is visibly working on artificial intelligence once again. If everything comes together, then it will now be a straight line from here to the end.

I foresee that, once the new sequence gets going, it won't be that easy to question the framework in terms of which the problems are posed. So I consider this my last opportunity for some time, to set out an alternative big picture. It's a framework in which all those rigorous mathematical and computational issues still need to be investigated, so a lot of "orthodox" ideas about Friendly AI should carry across. But the context is different, and it makes a difference.

Begin with the really big picture. What would it take to produce a friendly singularity? You need to find the true ontology, find the true morality, and win the intelligence race. For example, if your Friendly AI was to be an expected utility maximizer, it would need to model the world correctly ("true ontology"), value the world correctly ("true morality"), and it would need to outsmart its opponents ("win the intelligence race").

Now let's consider how SI will approach these goals.

The evidence says that the working ontological hypothesis of SI-associated researchers will be timeless many-worlds quantum mechanics, possibly embedded in a "Tegmark Level IV multiverse", with the auxiliary hypothesis that algorithms can "feel like something from inside" and that this is what conscious experience is.

The true morality is to be found by understanding the true decision procedure employed by human beings, and idealizing it according to criteria implicit in that procedure. That is, one would seek to understand conceptually the physical and cognitive causation at work in concrete human choices, both conscious and unconscious, with the expectation that there will be a crisp, complex, and specific answer to the question "why and how do humans make the choices that they do?" Undoubtedly there would be some biological variation, and there would also be significant elements of the "human decision procedure", as instantiated in any specific individual, which are set by experience and by culture, rather than by genetics. Nonetheless one expects that there is something like a specific *algorithm* or algorithm-template here, which is part of the standard *Homo sapiens* cognitive package and biological design; just another anatomical feature, particular to our species.

Having reconstructed this algorithm via scientific analysis of human genome, brain, and behavior, one would then idealize it using its own criteria. This algorithm defines the de-facto value system that human beings employ, but that is not necessarily the value system they would *wish* to employ; nonetheless, human self-dissatisfaction also arises from the use of this algorithm to judge ourselves. So it contains the seeds of its own improvement. The value system of a Friendly AI is to be obtained from the recursive self-improvement of the natural human decision procedure.

Finally, this is all for naught if seriously unfriendly AI appears first. It isn't good enough just to have the right goals, you must be able to carry them out. In the global race towards artificial general intelligence, SI might hope to "win" either by being the first to achieve AGI, or by having its prescriptions adopted by those who *do* first achieve AGI. They have some in-house competence regarding models of universal AI like AIXI, and they have many contacts in the world of AGI research, so they're at least *engaged* with this aspect of the problem.

Upon examining this tentative reconstruction of SI's game-plan, I find I have two major reservations. The big one, and the one most difficult to convey, concerns the ontological assumptions. In second place is what I see as an undue emphasis on the idea of outsourcing the methodological and design problems of FAI research to uploaded researchers and/or a proto-FAI which is simulating or modeling human researchers. This is supposed to be a way to finesse philosophical difficulties like "what is consciousness anyway"; you just simulate some humans until they agree that they have solved the problem. The reasoning goes that if the simulation is good enough, it will be just as good as if ordinary non-simulated humans solved it.

I also used to have a third major criticism, that the big SI focus on rationality outreach was a mistake; but it brought in a lot of new people, and in any case that phase is ending, with the creation of CFAR, a separate organization. So we are down to two basic criticisms.

First, "ontology". I do not think that SI intends to just program its AI with an apriori belief in the Everett multiverse, for two reasons. First, like anyone else, their ventures into AI will surely begin with programs that work within very limited and more down-to-earth ontological domains. Second, at least some of the AI's world-model ought to be obtained rationally. Scientific theories are supposed to be rationally justified, e.g. by their capacity to make successful predictions, and one would prefer that the AI's ontology results from the employment of its epistemology, rather than just being an axiom; not least because we want it to be able to question that ontology, should the evidence begin to count against it.

For this reason, although I have campaigned against many-worlds dogmatism on this site for several years, I'm not especially concerned about the possibility of SI producing an AI that is "dogmatic" in this way. For an AI to independently assess the merits of rival physical theories, the theories would need to be expressed with much more precision than they have been in LW's debates, and the disagreements about which theory is rationally favored would be replaced with objectively resolvable choices among exactly specified models.

The real problem, which is not just SI's problem, but a chronic and worsening problem of intellectual culture in the era of mathematically formalized science, is a dwindling of the ontological options to materialism, platonism, or an unstable combination of the two, and a similar restriction of *epistemology* to computation.

Any assertion that we need an ontology beyond materialism (or physicalism or naturalism) is liable to be immediately rejected by this audience, so I shall immediately explain what I mean. It's just the usual problem of "qualia". There are qualities which are part of reality - we know this because they are part of experience, and experience is part of reality - but which are not part of our physical description of reality. The problematic "belief in materialism" is actually the belief in the completeness of current materialist ontology, a belief which prevents people from seeing any need to consider radical or exotic solutions to the qualia problem. There is every reason to think that the world-picture arising from a correct solution to that problem will still be one in which you have "things with states" causally interacting with other "things with states", and a sensible materialist shouldn't find that objectionable.

What I mean by platonism, is an ontology which reifies mathematical or computational abstractions, and says that they are the stuff of reality. Thus assertions that reality is a computer program, or a Hilbert space. Once again, the qualia are absent; but in this case, instead of the deficient ontology being based on supposing that there is nothing but particles, it's based on supposing that there is nothing but the intellectual constructs used to model the world.

Although the abstract concept of a computer program (the abstractly conceived state machine which it instantiates) does not contain qualia, people often treat programs as having mind-like qualities, especially by imbuing them with semantics - the states of the program are conceived to be "about" something, just like thoughts are. And thus computation has been the way in which materialism has tried to restore the mind to a place in its ontology. This is the unstable combination of materialism and platonism to which I referred. It's unstable because it's not a real solution, though it can live unexamined for a long time in a person's belief system.

An ontology which genuinely contains qualia will nonetheless still contain "things with states" undergoing state transitions, so there will be state machines, and consequently, computational concepts will still be valid, they will still have a place in the description of reality. But the computational description is an abstraction; the ontological essence of the state plays no part in this description; only its causal role in the network of possible states matters for computation. The attempt to make computation the *foundation* of an ontology of mind is therefore proceeding in the wrong direction.

But here we run up against the hazards of computational *epistemology*, which is playing such a central role in artificial intelligence. Computational epistemology is good at identifying the minimal state machine which could have produced the data. But it cannot by itself tell you what those states are "like". It can only say that X was probably caused by a Y that was itself caused by Z.

Among the properties of human consciousness are knowledge that something exists, knowledge that consciousness exists, and a long string of other facts about the nature of what we experience. Even if an AI scientist employing a computational epistemology managed to produce a model of the world which correctly identified the causal relations between consciousness, its knowledge, and the objects of its knowledge, the AI scientist would not know that its X, Y, and Z refer to, say, "knowledge of existence", "experience of existence", and "existence". The same might be said of any successful analysis of qualia, knowledge of qualia, and how they fit into neurophysical causality.

It would be up to human beings - for example, the AI's programmers and handlers - to ensure that entities in the AI's causal model were given appropriate significance. And here we approach the second big problem, the enthusiasm for outsourcing the solution of hard problems of FAI design to the AI and/or to simulated human beings. The latter is a somewhat impractical idea anyway, but here I want to highlight the risk that the AI's *designers* will have false ontological beliefs about the nature of mind, which are then implemented apriori in the AI. That strikes me as far more likely than implanting a wrong apriori about physics; computational epistemology *can* discriminate usefully between different mathematical models of physics, because it can judge one state machine model as better than another, and current physical ontology is essentially one of interacting state machines. But as I have argued, not only must the true ontology be deeper than state-machine materialism, there is no way for an AI employing computational epistemology to bootstrap to a deeper ontology.

In a phrase: to use computational epistemology is to commit to state-machine materialism as your apriori ontology. And the problem with state-machine materialism is not that it models the world in terms of causal interactions between things-with-states; the problem is that it can't go any deeper than that, yet apparently we can. Something about the ontological constitution of consciousness makes it possible for us to experience existence, to have the concept of existence, to know that we are experiencing existence, and similarly for the experience of color, time, and all those other aspects of being that fit so uncomfortably into our scientific ontology.

It must be that the true *epistemology*, for a conscious being, is something more than computational epistemology. And maybe an AI can't bootstrap its way to knowing this expanded epistemology - because an AI doesn't *really* know or experience anything, only a consciousness, whether natural or artificial, does those things - but maybe a human being can. My own investigations suggest that the tradition of thought which made the most progress in this direction was the philosophical school known as transcendental phenomenology. But transcendental phenomenology is very unfashionable now, precisely because of apriori materialism. People don't see what "categorial intuition" or "adumbrations of givenness" or any of the other weird phenomenological concepts could possibly mean for an evolved Bayesian neural network; and they're right, there is no connection. But the idea that a human being is a state machine running on a distributed neural computation is just a hypothesis, and I would argue that it is a hypothesis in contradiction with so much of the *phenomenological* data, that we really ought to look for a more sophisticated refinement of the idea. Fortunately, 21st-century physics, if not yet neurobiology, can provide alternative hypotheses in which complexity of state originates from something other than concatenation of parts - for example, entanglement, or from topological structures in a field. In such ideas I believe we see a glimpse of the true ontology of mind, one which from the inside resembles the ontology of transcendental phenomenology; which in its mathematical, formal representation may involve structures like iterated Clifford algebras; and which in its biophysical context would appear to be describing a mass of entangled electrons in that hypothetical sweet spot, somewhere in the brain, where there's a mechanism to protect against decoherence.

Of course this is why I've talked about "monads" in the past, but my objective here is not to promote neo-monadology, that's something I need to take up with neuroscientists and biophysicists and quantum foundations people. What I wish to do here is to argue against the completeness of computational epistemology, and to caution against the rejection of phenomenological data just because it conflicts with state-machine materialism or computational epistemology. This is an argument and a warning that should be meaningful for anyone trying to make sense of their existence in the scientific cosmos, but it has a special significance for this arcane and idealistic enterprise called "friendly AI". My message for friendly AI researchers is not that computational epistemology is invalid, or that it's wrong to think about the mind as a state machine, just that all that isn't the full story. A monadic mind would be a state machine, but *ontologically* it would be different from the same state machine running on a network of a billion monads. You need to do the impossible one more time, and make your plans bearing in mind that the true ontology is something more than your current intellectual tools allow you to represent.

## Formalising cousin_it's bounded versions of Gödel's theorem

In this post, I'll try and put cousin_it's bounded version of Gödel's theorem on a rigorous footing. This will be logic-heavy, not programming heavy. The key unproved assumptions are listed at the bottom of the post.

Notes on terminology:

- T is our deductive system.
- ⊥ is a canonical contradiction in T. ¬A is defined to be A→⊥.
- T ⊢ A means that T proves the proposition A.
- T ⊢
_{j}A means that T proves A in a proof requiring no more than j symbols. - □ A is the statement "there exists a number that is an encoding in Gödel numbering of a valid proof in T of A".
- □
_{n}A is the statement "there exists a number that is an encoding in Gödel numbering of a valid proof in A with no more than n symbols in the proof".

The first needed fact is that there is a computable function g such that if T ⊢_{n} A, then T ⊢_{g(n)} □_{n} A. In informal language, this means that if T can prove A in j symbols, it can prove that it can do so, in g(j) symbols. This g will be the critical piece of the infrastructure.

Now, these arguments work with general definitions of proofs, but I like to put exact bounds where I can and move uncertainty to a single place. So I will put some restrictions on the language we use, and what counts as a proof. First, I will assume the symbols "(", ")", "j", and "=" are part of the language - ( and ) as parentheses, and j as a free variable.

I will require that proofs start and end with parenthesis. A proof need not state what statement it is proving! As long as there is a computable process that can check that "p is a proof of A", p need not mention A. This means that I can make the following assumptions:

- If p is a proof of A→B and q a proof of A, then (pq) is a proof of B (modus ponens).
- If p is a proof of A→B and q a proof of ¬B, then (pq) is a proof of ¬A (modus tollens).
- A proof that A↔B will be treated as a valid proof of A→B and B→A also.
- Well-formed sentences with free variables can also have proofs (as in "j=0 or j>0" and "j>3 → j
^{2}>9"). - If A and B have free variable j, and p is a proof that A→B, then (p(j=n)) is a proof that A(n)→B(n).

## The Quick Bayes Table

This is an effort to make Bayes' Theorem available to people without heavy math skills. It is possible that this has already been invented, because it is just a direct result of expanding something I read at Yudkowsky’s Intuitive Explanation of Bayes Theorem. In that case, excuse me for reinventing the wheel. Also, English is my second language.

When I read Yudkowsky’s Intuitive Explanation of Bayes Theorem, the notion of using decibels to measure the likelihood ratio of additional evidence struck me as extremely intuitive. But in the article, the notion was just a little footnote, and I wanted to check if this could be used to simplify the theorem.

It is harder to use logarithms than just using the Bayes Theorem the normal way, but I remembered that before modern calculators were made, mathematics carried around small tables of base 10 logarithms that saved them work in laborious multiplications and divisions, and I wondered if we could use the same in order to get quick approximations to Bayes' Theorem.

I calculated some numbers and produced this table in order to test my idea:

Decibels |
Probability |
Odds |

-30 |
0.1% |
1:1000 |

-24 |
0.4% |
1:251 |

-20 |
1% |
1:100 |

-18 |
1,5% |
1:63 |

-15 |
3% |
1:32 |

-12 |
6% |
1:16 |

-11 |
7% |
1:12.6 |

-10 |
9% |
1:10 |

-9 |
11% |
1:7.9 |

-8 |
14% |
1:6.3 |

-7 |
17% |
1:5 |

-6 |
20% |
1:4 |

-5 |
24% |
1:3.2 |

-4 |
28% |
1:2.5 |

-3 |
33% |
1:2 |

-2 |
38% |
1:1.6 |

-1 |
44% |
1:1.3 |

0 |
50% |
1:1 |

+1 |
56% |
1.3:1 |

+2 |
62% |
1.6:1 |

+3 |
67% |
2:1 |

+4 |
72% |
2.5:1 |

+5 |
76% |
3.2:1 |

+6 |
80% |
4:1 |

+7 |
83% |
5:1 |

+8 |
86% |
6.3:1 |

+9 |
89% |
7.9:1 |

+10 |
91% |
10:1 |

+11 |
93% |
12.6:1 |

+12 |
94% |
16:1 |

+15 |
97% |
32:1 |

+18 |
98.5% |
63:1 |

+20 |
99% |
100:1 |

+24 |
99.6% |
251:1 |

+30 |
99.9% |
1000:1 |

This table's values are approximate for easier use. The odds approximately double every 3 dB (The real odds are 1.995:1 in 3 dB) and are multiplied by 10 every 10 dB exactly.

In order to use this table, you must add the decibels results from the prior probability (Using the probability column) and the likelihood ratio (Using the ratio column) in order to get the approximated answer (Probability column of the decibel result). In case of doubt between two rows, choose the closest to 0.

For example, let's try to solve the problem in Yudkowsky’s article:

1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

1% prior gets us -20 dB in the table. For the likelihood ratio, 80% true positive versus 9.6% false positive is about a 8:1 ratio, +9 dB in the table. Adding both results, -20 dB + 9 dB = -11dB, and that translates into a 7% as the answer. The true answer is 7.9%, so this method managed to get close to the real answer with just a simple addition.

--

Yudkowsky says that the likelihood ratio doesn't tell the whole story about the possible results of a test, but I think we can use this method to get the rest of the story.

If you can get the positive likelihood ratio as the meaning of a positive result, then you can use the negative likelihood ratio as the meaning of the negative result just reworking the problem.

I'll use Yudkowsky's problem in order to explain myself. If 80% of women with breast cancer get positive mammographies, then 20% of them will get negative mammographies, and they will be false negatives. If 9.6% of women without breast cancer get positive mammographies, then 90.4% of them will get negative mammographies, true negatives.

The ratio between those two values will get us the meaning of a negative result: 20% false negative versus 90.4% true negative is between 1:4 and 1:5 ratio. We get the decibel value closest to 0, -6 dB. -20 dB - 6 dB = -26 dB. This value is between -24 dB and -30 dB, so the answer will be between 0.1% and 0.4%. The true answer is 0.2%, so it also works this way.

--

The positive likelihood ratio and the negative likelihood ratio are a good way of describing how a certain test adds additional data. We could describe the mammography test as a +9dB/-6dB test, and with only this information we know everything we need to know about the test. If the result is positive, it adds 9dB to the evidence, and if it is negative, it subtracts 6dB to it.

Simple and intuitive.

By the way, as decibels are used to measure physical quantities, not probabilities, I believe that renaming the unit would be appropriate in this case. What about DeciBayes?

## Trapping AIs via utility indifference

If you wanted to check whether your creations were prone to misbehaving, one way would be to leave a big juicy apple hanging tantalisingly from a very visible tree, turn your back for a while, and see what they get up to.

That would work if you were God and your creations were essentially children. But if the creator is us and the creation is a super-intelligent AI, such 'honey-pots' and unlikely to work. They are tricks, and we shouldn't rely on being able to trick a being more intelligent than ourselves - unless its intelligence doesn't matter.

We can use utility indifference to construct such traps. Utility indifference can be informally summarised as:

"There is a stochastic event X which can output 0 or 1. We wish the AI act as if it believed that X=0, with certainty. It's problematic for the AI to have incorrect beliefs, so the result is achieved through the definition of the AI's utility function."

Consider the following situation: a master AI that will create the source code of another, autonomous AI. The master AI is setup to produce the code of a Friendly AI, or a reduced impact AI - but we're not sure if the master AI is trustworthy. It would be good to go through the code very carefully before unleashing it (or erasing it). But if the master AI knows this (and we must assume it does), then it will either obfuscate the dangerous parts or put traps in it, making it risky for us to look too closely.

Then implement the following setup. The code travels through some stochastic event X, such that if X=0, the code is immediately transferred into a disciple AI, completely autonomous, which is at liberty to immediately modify its own code and erase any trace of what it was. If X=1, the code is instead copied to our hard drives for analysis. The AI is given some reason to keep the code short - maybe there is an increasing chance of the code getting erased the longer it is.

On top of this, we implement utility indifference: the master AI will act as if it believed X=0 with certainty, while we have setup X so that X=1 with near-certainty. Then the master AI will be motivated to produce the shortest code compatible with its utility function, in the 'knowledge' that this code will be copied into a powerful autonomous AI, and will never be seen by humans. It has no incentive to trap its code - quite the reverse, as it wants to keep it short.

And therefore we have all the time in the world to go through the code, looking for defection. Other ways of using utility indifference to trap AIs are no doubt possible, but this was the first setup that sprang to my mind.

## Second order logic, in first order set-theory: what gives?

*With thanks to Paul Christiano*

My previous post left one important issue unresolved. Second order logic needed to make use of set theory in order to work its magic, pin down a single copy of the reals and natural numbers, and so on. But set theory is a first order theory, with all the problems that this brings - multiple models, of uncontrollable sizes. How can these two facts be reconciled?

Quite simply, it turns out: for any given model of set theory, the uniqueness proofs still work. Hence the proper statement is:

- For any model of set theory, there is a unique model of reals and natural numbers obeying the second order axioms.

Often, different models of set theory will have the same model of the reals inside them; but not always. Countable models of set theory, for instance, will have a countable model of the reals. So models of the reals can be divided into three categories:

- The standard model of the reals, the unique field that obeys the second order axioms inside standard models of set theory (and some non-standard models of set theory as well).
- Non-standard models of the reals, that obey the second order axioms inside non-standard models of set theory.
- Non-standard models of the reals that obey the first order axioms, but do not obey the second order axioms in any model of set theory.

And similarly for the natural numbers.

## Seeking education

This will not be a long post; I have a simple question to ask: if you wanted to educate yourself to graduate level in mathematics, but didn't actually want to go to university, what would you do? I would ask for text-book recommendations, but I don't want to limit your responses (however, bear in mind that the wikipedia articles on, say, cardinality or well-ordering go over my head – they may skim my hairline, but over they go). Also bear in mind that while I personally have A-levels (British qualifications) in both Maths and Further Maths (which is to say, I know some calculus at least), there are probably plenty of people on lesswrong who don't and who desire the same information – so assume as much ignorance as you feel necessary (it's a shame, actually, that there isn't a sequence here on lesswrong for maths). What do you advise (if you think the query ill-defined, I would like to know that as well)?

## [LINK] The Hacker Shelf, free books.

Yes, this a repost from Hacker News, but I want to point out some books that are of LW-related interest.

The Hacker Shelf is a repository of freely available textbooks. Most of them are about computer programming or the business of computer programming, but there are a few that are perhaps interesting to the LW community. All of these were publicly available beforehand, but I'm linking to the aggregator in hopes that people can think of other freely available textbooks to submit there.

The site is in its beginning explosion phase; in the time it took to write this post, it doubled in size. If previous sites are any indication, it will crest in a month or so. People will probably lose interest after three months, and after a year the site will probably silently close shop.

## MacKay, Information Theory, Inference, and Learning Algorithms

I really wish I had an older version of this book; the newer one has been marred by a Cambridge UP ad on the upper margin of every page. Publishers ruin everything.

The book covers reasonably concisely the basics of information theory and Bayesian methods, with some game theory and coding theory (in the sense of data compression) thrown in on the side. The style takes after Knuth, but refrains from the latter's more encyclopedic tendencies. It's also the type of book that gives a lot of extra content in the exercises. It unfortunately assumes a decent amount of mathematical knowledge — linear algebra and calculus, but nothing you wouldn't find on the Khan Academy.

Hacker Shelf review, book website.

## Easley and Kleinberg, Networks, Crowds, and Markets

There's just a lot of stuff in this book, most of it of independent interest. The thread that ties the book together is graph theory, and with it they cover a great deal of game theory, voting theory, and economics. There are lots of graphs and pictures, and the writing style is pretty deliberate and slow-paced. The math is not very intense; all their probability spaces are discrete, so there's no calculus, and only a few touches of linear algebra.

Hacker Shelf review, book website.

## Gabriel, Patterns of Software

This is a more fluffy book about the practice of software engineering. It's rather old, but I'm linking to it anyway because I agree with the author's feeling that the software engineering discipline has more or less misunderstood Christopher Alexander's work on pattern languages. The author tends to ramble on. I think there's some good wisdom about programming practices and organizational management in general that one could abstract away from this book.

Hacker Shelf link, book website (scroll down).

## Nisan et. al., Algorithmic Game Theory

I hesitate to link this because the math level is exceptionally high, perhaps high enough that anyone who can read the book probably knows the better part of its contents already. But game/decision theory is near and dear to LW's heart, so perhaps someone will gather some utility from this book. There's an awful lot going on in it. A brief selection: a section on the relationship between game theory and cryptography, a section on computation in prediction markets, and a section analyzing the incentives of information security.

## Question on math in "A Technical Explanation of Technical Explanation"

"A Technical Explanation of Technical Explanation" (http://yudkowsky.net/rational/technical) defines a proper rule for a betting game as one where the payoff is maximized by betting an amount proportional to the probability of success.

The first example rule given is that the payoff is one minus the negative of the squared error, so for example if you make a bet of .3 on the winner, your payoff is 1-(1-.3)^2 = .51.

This doesn't seem like a good example. It works if there are only two options, but I don't think it works if there are three or more. For example, we imagine P(red) = .5, P(blue) = .2, P(green) = .3. If we place bets of .5, .2, and .3 respectively, the expected return I get is .6. (**edit**: Fixed a mistake pointed out by Douglas Knight.)

However, if I place bets of .51, .19, .3 the expected return is .60173. I have that the condition for maximization is

(1-R)P(R) = (1-B)P(B) = (1-G)P(G),

which I got by taking partial derivatives of the expectation and setting them equal. ("R" stands for the bet placed on red and "P(R)" for the probability of red, etc.) This is different than simply R=P(R), etc.

So does the article have a mistake, or do I, or did I miss part of the context?

## [LINK] What is it like to have an understanding of very advanced mathematics?

This, apparently:

You can answer many seemingly difficult questions quickly.

You are often confident that something is true long before you have an airtight proof for it (this happens especially often in geometry).

You are comfortable with feeling like you have no deep understanding of the problem you are studying.

Your intuitive thinking about a problem is productive and usefully structured, wasting little time on being aimlessly puzzled.

When trying to understand a new thing, you automatically focus on very simple examples that are easy to think about, and then you leverage intuition about the examples into more impressive insights.

...the biggest misconception that non-mathematicians have about how mathematicians think is that there is some mysterious mental faculty that is used to crack a problem all at once.

You go up in abstraction, "higher and higher". The main object of study yesterday becomes just an example or a tiny part of what you are considering today.

The particularly "abstract" or "technical" parts of many other subjects seem quite accessible because they boil down to maths you already know. You generally feel confident about your ability to learn most quantitative ideas and techniques.

You move easily between multiple seemingly very different ways of representing a problem.

Spoiled by the power of your best tools, you tend to shy away from messy calculations or long, case-by-case arguments unless they are absolutely unavoidable.

You develop a strong aesthetic preference for powerful and general ideas that connect hundreds of difficult questions, as opposed to resolutions of particular puzzles.

Understanding something abstract or proving that something is true becomes a task a lot like building something.

In listening to a seminar or while reading a paper, you don't get stuck as much as you used to in youth because you are good at modularizing a conceptual space and taking certain calculations or arguments you don't understand as "black boxes" and considering their implications anyway.

You are good at generating your own questions and your own clues in thinking about some new kind of abstraction.

You are easily annoyed by imprecision in talking about the quantitative or logical.

On the other hand, you are very comfortable with intentional imprecision or "hand waving" in areas you know, because you know how to fill in the details.

You are humble about your knowledge because you are aware of how weak maths is, and you are comfortable with the fact that you can say nothing intelligent about most problems.

## What mathematics to learn

There is, of course, Kahn Academy for fundamentals. We have already had a discussion on How to learn math.

What resources exist detailing *which* mathematics to learn in what order? What resources exist that explain the utility of different mathematical subfields for the purpose of directing studies?

## The Apparent Reality of Physics

Follow-up to: Syntacticism

I wrote:

The only objects that are real (in a Platonic sense) are formal systems (or rather, syntaxes). That is to say, my ontology is the set of formal systems. (This is not incompatible with the apparent reality of a physical universe).

In my experience, most people default^{1} to naïve physical realism: the belief that "matter and energy and stuff exist, and they follow the laws of physics". This view has two problems: how do you know stuff exists, and what makes it follow those laws?

To the first - one might point at a rock, and say "Look at that rock; see how it *exists* at me." But then we are relying on sensory experience; suppose the simulation hypothesis were true, then that sensory experience would be unchanged, but the rock wouldn't really *exist*, would it? Suppose instead that we are being simulated *twice*, on two different computers. Does the rock exist *twice as much*? Suppose that there are actually two copies of the Universe, physically existing. Is there any way this could in principle be distinguished from the case where only one copy exists? No; a manifest physical reality is observationally equivalent to N manifest physical realities, as well as to a single simulation or indeed N simulations. (This remains true if we set N=0.)

So a true description requires that the idea of *instantiation* should drop out of the model; we need to think in a way that treats all the above cases as identical, that justifiably puts them all in the same bucket. This we can do if we claim that *that-which-exists* is precisely the *mathematical structure* defining the physical laws and the index of our particular initial conditions (in a non-relativistic quantum universe that would be the Schrödinger equation and some particular wavefunction). Doing so then solves not only the first problem of naïve physical realism, but the second also, since trivially *solutions to those laws* must follow those laws.

But then why should we privilege our particular set of physical laws, when that too is just a source of indexical uncertainty? So we conclude that all possible mathematical structures have Platonic existence; there is no little XML tag attached to the mathematics of our own universe that states "this one exists, is physically manifest, is instantiated", and in this view of things such a tag is obviously superfluous; instantiation has dropped out of our model.

When an agent in universe-defined-by-structure-A simulates, or models, or thinks-about, universe-defined-by-structure-B, they do not 'cause universe B to come into existence'; there is no refcount attached to each structure, to tell the Grand Multiversal Garbage Collection Routine whether that structure is still needed. An agent in A simulating B is not a causal relation from A to B; instead it is a causal relation from B to A! B defines the fact-of-the-matter as to what the result of B's laws is, and the agent in A will (barring cosmic rays flipping bits) get the result defined by B.^{2}

So we are left with a Platonically existing multiverse of mathematical structures and solutions thereto, which can contain conscious agents to whom there will be every *appearance* of a manifest instantiated physical reality, yet no such physical reality exists. In the terminology of Max Tegmark (The Mathematical Universe) this position is the acceptance of the MUH but the rejection of the ERH (although the Mathematical Universe is an external reality, it's not an external *physical* reality).

Reducing all of applied mathematics and theoretical physics to a syntactic formal system is left as an exercise for the reader.

^{1}That is, when people who haven't thought about such things before do so for the first time, this is usually the first idea that suggests itself.

^{2}I haven't yet worked out what happens if a closed loop forms, but I think we can pull the same trick that turns formalism into syntacticism; or possibly, consider the whole system as a single mathematical structure which may have several stable states (indexical uncertainty) or no stable states (which I think can be resolved by 'loop unfolding', a process similar to that which turns the complex plane into a Riemann surface - but now I'm getting beyond the size of digression that fits in a footnote; a mathematical theory of causal relations between structures needs at least its own post, and at most its own *field*, to be worked out properly).

## Syntacticism

I've mentioned in comments a couple of times that I don't consider formal systems to talk about themselves, and that consequently Gödelian problems are irrelevant. So what am I actually on about?

It's generally accepted in mathematical logic that a formal system which embodies Peano Arithmetic (PA) is able to talk about itself, by means of Gödel numberings; statements and proofs within the system can be represented as positive integers, at which point "X is a valid proof in the system" becomes equivalent to an arithmetical statement about #X, the Gödel number representing X. This is then diagonalised to produce the Gödel sentence (roughly, g="There is no proof X such that the last line of X is g", and incompleteness follows. We can also do things like defining □ ("box") as the function from S to "There is a proof X in PA whose last line is S" (intuitively, □S says "S is provable in PA"). This then also lets us define the Löb sentence, and many other interesting things.

But how do we *know* that □S ⇔ there is a proof of S in PA? Only by applying some *meta-theory*. And how do we *know* that statements reached in the meta-theory of the form "thus-and-such is true of PA" are true of PA? Only by applying a meta-meta-theory. There is *no* a-priori justification for the claim that "A formal system is in principle capable of talking about other formal systems", which claim is used by the proof that PA can talk about itself. (If I remember correctly, to prove that □ does what we think it does, we have to appeal to second-order arithmetic; and how do we know second-order arithmetic applies to PA? Either by invoking third-order arithmetic to analyse second-order arithmetic, or by recourse to an informal system.)

Note also that the above is not a strange loop through the meta-level; we justify our claims about arithmetic_{n} by appeal to arithmetic_{n+1}, which is a separate thing; we never find ourselves back at arithmetic_{n}.

Thus the claim that formal systems can talk about themselves involves ill-founded recursion, what is sometimes called a "skyhook". While it may be a theorem of second-order arithmetic that "the strengthened finite Ramsey theorem is unprovable in PA", one cannot conclude from second-order arithmetic alone that the "PA" in that statement *refers to* PA. It is however provable in third-order arithmetic that "What second-order arithmetic calls "PA" is PA", but that hasn't gained us much - it only tells us that second- and third-order arithmetic call the same thing "PA", it doesn't tell us whether this "PA" is PA. Induct on the arithmetic hierarchy to reach the obvious conclusion. (Though note that none of this prevents the Paris-Harrington Theorem from being a theorem of n-th order arithmetic ∀n≥2)

What, then, is the motivation for the above? Well, it is a basic principle of my philosophy that the only objects that are real (in a Platonic sense) are formal systems (or rather, *syntaxes*). That is to say, my ontology is the ~~set~~class of formal systems. (This is not incompatible with the apparent reality of a physical universe; if this isn't obvious, I'll explain why in another post.) But if we allow these systems to have *semantics*, that is, we claim that there is such a thing as a "true statement", we start to have problems with completeness and consistency (namely, that we can't achieve the one and we can't prove the other, assuming PA). Tarski's undefinability theorem protects us from having to deal with systems which talk about truth in themselves (because they are necessarily inconsistent, assuming some basic properties), but if systems can talk about each other, and if systems can talk about provability within themselves (that is, if analogues to the □ function can be constructed), then nasty Gödelian things end up happening (most of which are, to a Platonist mathematician, *deeply unsatisfying*).

So instead we restrict the ontology to *syntactic systems devoid of any semantics*; the statement ""Foo" is true" is meaningless. There is a fact-of-the-matter as to whether a given statement can be reached in a given formal system, but that fact-of-the-matter cannot be meaningfully talked about in any formal system. This is a remarkably bare ontology (some consider it excessively so), but is at no risk from contradiction, inconsistency or paradox. For, what is "P∧¬P" but another, syntactic, sentence? Of course, applying a system which proves "P∧¬P" to the 'real world' is likely to be problematic, but the paradox or the inconsistency lies in the *application* of the system, and does not inhere in the system itself.

EDIT: I am actually aiming to get somewhere with this, it's not just for its own sake (although the ontological and epistemological status of mathematics *is* worth caring about for its own sake). In particular I want to set up a framework that lets me talk about Eliezer's "infinite set atheism", because I think he's asking a wrong question.

Followed up by: The Apparent Reality of Physics

## Dense Math Notation

I program, and am also presently working my way through some math books. I find that I often have to backtrack to look up pieces of notation like variables and operators. Unfortunately, this is very problematic. Greek and Latin letters give no indication of where they came from and are not usable search terms, even knowing the full context in which they appeared. Many authors have their own bits of idiosyncratic notation, often combinations of subscripting and line art generated by TeX macros. Since expanding equations out to their definitions is so difficult, I sometimes don't bother to investigate when one looks odd, which as you might expect leads to big trouble later when errors in understanding creep by.

This is not nearly as big a problem for me when reading code, however, because there every variable and nonstandard operator has a descriptive name wherever it's used, and documentation is never more than a few hotkeys away. The same thing could be done for math. Suppose you took a typical higher math book, and replaced every single-letter variable and operator with an appropriate identifier. For me, this would make it much more readable; I would gain a better understanding in less time. However, I don't know the effect size or how broadly this generalizes.

Do other people have this problem? Might this issue deter some people from studying math entirely? Has anyone tried the obvious controlled experiment? How about with an experimental group specifically of programmers?

## Inverse Speed

One must always invert.

I'm grateful to orthonormal for mentioning the following math problem, because it allowed me to have a significant confusion-dissolving insight (actually going on two, but I'll only discuss one in this post), as well as providing an example of how bad I am at math:

"[I]f you want to average 40 mph on a trip, and you averaged 20 mph for the first half of the route, how fast do you have to go on the second half of the route?"

When I read this, my first thought was "Huh? If you spend an hour going 20 mph and then spend another hour going 60 mph, you've just gone 80 miles in 2 hours -- for an average speed of 40 mph, just as desired. So what do you people mean it's impossible?"

As you can see, my confusion resulted from interpreting "half of the route" to refer to the total *time* of the journey, rather than the total *distance*.

This misinterpretation reveals something fundamental about how I (I know better by now than to say "we") think about *speed*.

In my mind, speed is a *mapping* from *times* to *distances*. The way to compare different speeds is by holding time constant and looking at the different distances traversed in that fixed time. (I know I'm not a total mutant in this regard, because even other people tend to visually represent speeds as little arrows of varying length, with greater lengths corresponding to higher speeds.)

In particular, I *don't* think of it as a mapping from *distances* to *times*. I don't find it natural to compare speeds by imagining a fixed distance corresponding to different travel times. Which explains why I find this problem so difficult, and other people's explanations so unilluminating: they tend to begin with something along the lines of "let d be the total distance traveled", upon which my brain experiences an error message that is perhaps best verbalized as something like "wait, what? Who said anything about a fixed distance? If speeds are varying, distances have to be varying, too!"

If speed is a mapping from times to distances, then the way that you add speeds together and multiply them by numbers (the operations involved in averaging) is by performing the same operations on corresponding distances. (This is an instance of the general definition in mathematics of addition of functions: (f+g)(x) = f(x)+g(x), and similarly for multiplication by numbers: (af)(x) = a*f(x).) In concrete terms, what this means is that in order to add 30 mph and 20 mph together, all you have to do is add 30 and 20 and then stick "mph" on the result. Likewise with averages: provided the times involved are the same, if your speeds are 20 mph and 60 mph, your average speed is 40 mph.

You cannot do these operations nearly so easily, however, if distance is being held fixed and time varying. Why not? Because if our mapping is from times to distances, then finding the time that corresponds to a given distance requires us to *invert* that mapping, and there's no easy way to invert the sum of two mappings (we can't for example just add the inverses of the mappings themselves). As a result, I find it difficult to understand the notion of "speed" while thinking of time as a dependent variable.

And that, at least for me, is why this problem is confusing: the statement doesn't contain a prominent warning saying "Attention! Whereas you normally think of speed as the being the (longness-of-)distance traveled in a given time, here you need to think of it as the (shortness-of-)time required to travel a given distance. In other words, the question is actually about *inverse speed*, even though it talks about 'speed'."

Only when I have "inverse speed" in my vocabulary, can I then solve the problem -- which, properly formulated, would read: "If you want your inverse speed for the whole trip to be 1/40 hpm, and your inverse speed for the first half is 1/20 hpm, how 'slow' (i.e. inversely-fast) do you have to go on the second half?"

**Solution**: *Now* it makes sense to begin with "let d be the total distance"! For inverse speed, unlike speed, accepts distances as inputs (and produces times as outputs). So, instead of distance = speed*time -- or, as I would rather have it, distance = speed(time) -- we have the formula time = speed^{-1}(distance). Just as the original formula converts questions about speed to questions about distance, this new formula conveniently converts our question about inverse speeds to a question about times: we'll find the time required for the whole journey, the time required for the first half, subtract to find the time required for the second half, then finally convert this back to an inverse speed.

So if d is the total distance, the total time required for the journey is (1/40)*d = d/40. The time required for the first half of the journey is (1/20)*(d/2) = d/40. So the time required for the second half is d/40 - d/40 = 0. Hence the inverse speed must be 0.

So we're being asked to travel a nonzero distance in zero time -- which happens to be an impossibility.

Problem solved.

Now, here's the interesting thing: I'll bet there are people reading this who (despite my best efforts) found the above explanation difficult to follow -- *and yet had no trouble solving the problem themselves*. And I'll bet there are probably also people who consider my explanation to be an example of belaboring the obvious.

I have a term for people in these categories: I call them "good at math". What unites them is the ability to produce correct solutions to problems like this without having to expend significant effort figuring out the sort of stuff I explained above.

If for any reason anyone is ever tempted to describe *me* as "good at math", I will invite them to reflect on the fact that an *explicit understanding* of the concept of "inverse speed" as described above (i.e. as a function that sends distances to times) was a *necessary prerequisite* for my being able to solve this problem, and then to consider that problems of this sort are customarily taught in middle- or high school, by middle- and high school teachers.

No indeed, I was not sorted into the tribe of "good at math".

I should find some sort of prize to award to anyone who can explain how to solve "mixing" problems in a manner I find comprehensible. (You know the type: how much of x% concentration do you add to your y% concentration to get z% concentration? *et similia*.)

## On Pi day, we eat pie; On Tau day, we eat Taoists?

I'd like to start by wishing everyone a Happy Pi day (even if for some of you it was yesterday).

Today, going about my usual Pi day celebration (which included pi(e) of the chocolate, cherry, apple, and movie variety), I stumbled across pi-protesters, who spoke of Tau-ism. For those who haven't heard, Tauist claim that Tau, represented by the Greek letter T, is the real circle constant. There's one proponent and his arguments here: http://tauday.com/

I read the article, saw the points that were made, and I've remained impartial (despite being a mathematics major). I can see Tau's usefulness; I can see why pi hasn't changed (and hardly would need to). So I decided to do something else: Present this claim to the LessWrong community, for those who are interested. What do you think?

## The Importance of Mathematics (Gowers)

For the past few days I've been pondering the question of how best to respond to paulfchristiano's recent posts and comments questioning the value of mathematical research. I don't think I can do it concisely, in a single post; bridging the inferential distance may require something more like a sequence of posts. I may end up writing such a sequence eventually, since it would involve ideas I've actually been wanting to write up for some time, and which are actually relevant to more than just the specific questions at issue here (whether society should sponsor mathematics, and given that it does, whether paulfchristiano or anyone else in the LW readership should pursue it).

However, as the preceding parenthetical hints at, I'm actually somewhat conflicted about whether I should even bother. Although I believe that mathematical research should be conducted by *somebody*, it's not at all clear to me that the discipline needs *more* people beyond those who already "get" its importance, and are out there doing it rather than writing skeptical posts like paulfchristiano's. It seems perfectly plausible to me that those who feel as paulfchristiano does should just leave the profession and do something else that feels more "important" to them. This is surely the best practical solution on an individual level for those who think they have a better idea than existing institutions of where the most promising research directions lie, at least until Hansonian prediction markets are (ever) implemented.

Nevertheless, for those interested in the society-level question of whether mathematics (as such) may be justifiably pursued by *anyone, *or any community of people, as a professional occupation (which is quite distinct from the question of whether e.g. paulfchristiano should personally pursue it), I recommend, at least as a start, grappling with the arguments put forward by the best mathematicians in their own words. I think this essay by Timothy Gowers (a Fields Medalist), titled "The Importance of Mathematics", is a good place to begin. I would particularly draw the attention of those like paulfchristiano, who think they have a good idea of which branches of mathematics are useful and which aren't, to the following passage, from pp.8-9 (unfortunately the illustrations are missing, but the point being made is pretty clear nonetheless):

So - mathematicians can tell their governments - if you cut funding to pure mathematical

research, you run the risk of losing out on unexpected benefits, which historically

have been by far the most important.

However, the miserly finance minister need not be convinced quite yet. It may be very

hard to identify positively the areas of mathematics likely to lead to practical benefits, but

that does not rule out the possibility of identifying negatively the areas that will quite

clearly be useless, or at least useless for the next two hundred years. In fact, the finance

minister does not even need to be certain that they will be useless. If a large area of

mathematics has only a one in ten thousand chance of producing economic benefit in the

next fifty years, then perhaps that at least could be cut.

You will not be surprised to hear me say that this policy would still be completely

misguided. A major reason, one that has been commented on many times and is implied

by the subtitle of this conference, "A Celebration of the Universality of Mathematical

Thought", is that mathematics is very interconnected, far more so than it appears on the

surface. The picture in the back of the finance minister's mind might be something like

Figure 4. According to this picture, mathematics is divided into several subdisciplines, of

varying degrees of practicality, and it is a simple matter to cut funding to the less practical

ones.A more realistic picture, though still outrageously simplified, is given in Figure 5.

(Just for the purposes of comparison, Figure 6 shows Figures 4 and 5 superimposed.) The

nodes of Figure 5 represent small areas of mathematical activity and the lines joining them

represent interrelationships between those areas. The small areas of activity form clusters

where there are more of these interrelationships, and these clusters can perhaps be thought

of as subdisciplines. However, the boundaries of these clusters are not precise, and many

of the interrelationships are between clusters rather than within them.

In particular, if mathematicians work on difficult practical problems, they do not do so

in isolation from the rest of mathematics. Rather, they bring to the problems several tools

- mathematical tricks, rules of thumb, theorems known to be useful (in the mathematical

sense), and so on. They do not know in advance which of these tools they will use, but they

hope that after they have thought hard about a problem they will realize what is needed to

solve it. If they are lucky, they can simply apply their existing expertise straightforwardly.

More often, they will have to adapt it to some extent(...)

Thus, a good way to think about mathematics as a whole is that it is a huge body of

knowledge, a bit like an encyclopaedia but with an enormous number of cross-references.

This knowledge is stored in books, papers, computers and the brains of thousands of

mathematicians round the world. It is not as convenient to look up a piece of mathematics

as it is to look up a word in an encyclopaedia, especially as it is not always easy to

specify exactly what it is that one wants to look up. Nevertheless, this "encyclopaedia" of

mathematics is an incredible resource. And just as, if one were to try to get rid of all the

entries in an encyclopaedia, or, to give a different comparison, all the books in a library,

that nobody ever looked up, the result would be a greatly impoverished encyclopaedia or

library, so, any attempt to purge mathematics of its less useful parts would almost certainly

be very damaging to the more useful parts as well.

## What Else Would I Do To Make a Living?

Response to: The Value of Theoretical Research

Reading paulfchristiano's article the other day, I realized that I had had many similar discussions with myself, and have been guilty of motivated stopping and poor answers to all of them.

However, one major roadblock in my pursuing better answers, is that I feel that I have been "locked in" to my current path.

I am currently a mathematics Ph.D. student. I did not have a minor. I don't have significant programming skills or employment experience. I know nothing about finance. I know a lot about mathematics.

Paul says:

There is a shortage of intelligent, rational people in pretty much every area of human activity. I would go so far as to claim this is the limiting input for most fields.

However, "most fields" is not a very good tool for narrowing my search space; I have spent my entire life in school, and I like having structures and schedules that tell me when I'm doing productive things and that I've progressed to certain stages. I'm not ready to drop out and do whatever, and I don't have a particular idea of what whatever might be.

On the other hand, I currently have a variety of resources available to me. For example, I have a steady income (a grad student stipend isn't much, but it's plenty for me to live on), and I have the ability to take undergraduate classes for free (though not the spare time at the moment.)

My current intent is to continue and finish my Ph.D., but to attempt to take classes in other subjects, such as linguistics, biology and chemistry, and computer science which might lead to other interesting career paths.

Has anybody else had a similar feeling of being "locked in"? How have you responded to it? For those who have studied mathematics, are you still? If you continued, what helped you make that decision? If you stopped, what about that? What did you end up doing? How did you decide on it?

## generalized n-categories?

This looks like an interesting, but a bit strange, old story. A bit similar to parts of an earlier posted essay by Gromov. However that may be, the Princeton IAS invited the author and so I'd like to know about how his concepts are intended to become implemented and applied: http://vbm-ehr.pagesperso-orange.fr/ChEh/articles/Baas%20paper.pdf

## Draft/wiki: Infinities and measuring infinite sets: A quick reference

**EDIT:** This is now on the Wiki as "Quick reference guide to the infinite". Do what you want with it.

It seems whenever anything involving infinities or measuring infinite sets comes up it generates a lot of confusion. So I thought I would write a quick guide to both to

- Address common confusions
- Act as a useful reference (perhaps this should be a wiki article? This would benefit from others being able to edit it; there's no "community wiki mode" on LW, huh?)
- Remind people that sometimes inventing a new sort of answer is necessary!

I am trying to keep this concise, in some cases substituting Wikipedia links for explanation, but I do want what I have written to be understandable enough and informative enough to answer the commonly occurring questions. Please let me know if you can detect a particular problem. I wrote this very quickly and expect it still needs quite a bit more work to be understandable to someone with very little math background.

I realize many people here are finitists of one stripe or another but this comes up often enough that this seems useful anyway. Apologies to any constructivists, but I am going to assume classical logic, because it's all I know, though I am pointing out explicitly any uses of choice. (For what this means and why anyone cares about this, see this comment.) Also as I intend this as a reference (is there *any* way we can make this editable?) some of this may be things that I do not actually know but merely have read.

Note that these are two separate topics, though they have a bit of overlap.

Primarily, though, my main intention is to put an end to the following, which I have seen here far too often:

**Myth #0: All infinities are infinite cardinals, and cardinality is the main method used to measure size of sets.**

The fact is that **"infinite" is a general term meaning "larger (in some sense) than any natural number"; different systems of infinite numbers get used depending on what is appropriate in context**. Furthermore, **there are many other methods of measuring sizes of sets, which sacrifice universality for higher resolution; cardinality is a very coarse-grained measure.**

## QFT, Homotopy Theory and AI?

What do you think about the new, exiting connections between QFT, Homotopy Theory and pattern recognition, proof verification and (maybe) AI systems? In view of the background of this forum's participants (selfreported in the survey mentioned a few days ago), I guess most of you follow those developments with some attention.

Concerning Homotopy Theory, there is a coming [special year](http://www.math.ias.edu/node/2610), you probably know Voevodsky's [recent intro lecture](http://www.channels.com/episodes/show/10793638/Vladimir-Voevodsky-Formal-Languages-partial-algebraic-theories-and-homotopy-category-), and [this](http://video.ias.edu/voevodsky-80th) even more popular one. Somewhat related are Y.I. Manin's remarks on the missing quotient structures (analogue to localized categories) in data structures and some of the ideas in Gromov's [essay](http://www.ihes.fr/~gromov/PDF/ergobrain.pdf).

Concerning ideas from QFT, [here](http://arxiv.org/abs/0904.4921) an example. I wonder what else concepts come from it?

BTW, whereas the public discussion focus on basic qm and on q-gravity questions, the really interesting and open issue is the special relativistic QFT: QM is just a canonical deformation of classical mechanics (and could have been found much earlier, most of the interpretation disputes just come from the confusion of mathematical properties with physics data), but Feynman integrals are despite half a century intense research mathematical unfounded. As Y.I. Manin called it in a recent interview, they are "an Eifel tower floating in the air". Only a strong platonian belief makes people tolerate that. I myself take them only serious because there is a clear platonic idea behind them and because number theoretic analoga work very well.