Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Metaphilosophical Mysteries

35 Post author: Wei_Dai 27 July 2010 12:55AM

Creating Friendly AI seems to require us humans to either solve most of the outstanding problems in philosophy, or to solve meta-philosophy (i.e., what is the nature of philosophy, how do we practice it, and how should we program an AI to do it?), and to do that in an amount of time measured in decades. I'm not optimistic about our chances of success, but out of these two approaches, the latter seems slightly easier, or at least less effort has already been spent on it. This post tries to take a small step in that direction, by asking a few questions that I think are worth investigating or keeping in the back of our minds, and generally raising awareness and interest in the topic.

The Unreasonable Effectiveness of Philosophy

It seems like human philosophy is more effective than it has any right to be. Why?

First I'll try to establish that there is a mystery to be solved. It might be surprising so see the words "effective" and "philosophy" together in the same sentence, but I claim that human beings have indeed made a non-negligible amount of philosophical progress. To cite one field that I'm especially familiar with, consider probability and decision theory, where we went from having no concept of probability, to studies involving gambles and expected value, to subjective probability, Bayesian updating, expected utility maximization, and the Turing-machine-based universal prior, to the recent realizations that EU maximization with Bayesian updating and the universal prior are both likely to be wrong or incomplete.

We might have expected that given we are products of evolution, the amount of our philosophical progress would be closer to zero. The reason for low expectations is that evolution is lazy and shortsighted. It couldn't possibly have "known" that we'd eventually need philosophical abilities to solve FAI. What kind of survival or reproductive advantage could these abilities have offered our foraging or farming ancestors?

From the example of utility maximizers, we also know that there are minds in the design space of minds that could be considered highly intelligent, but are incapable of doing philosophy. For example, a Bayesian expected utility maximizer programmed with a TM-based universal prior would not be able to realize that the prior is wrong. Nor would it be able to see that Bayesian updating is the wrong thing to do in some situations.

Why aren't we more like utility maximizers in our ability to do philosophy? I have some ideas for possible answers, but I'm not sure how to tell which is the right one:

  1. Philosophical ability is "almost" universal in mind space. Utility maximizers are a pathological example of an atypical mind.
  2. Evolution created philosophical ability as a side effect while selecting for something else.
  3. Philosophical ability is rare and not likely to be produced by evolution. There's no explanation for why we have it, other than dumb luck.

As you can see, progress is pretty limited so far, but I think this is at least a useful line of inquiry, a small crack in the problem that's worth trying to exploit. People used to wonder at the unreasonable effectiveness of mathematics in the natural sciences, especially in physics, and I think such wondering eventually contributed to the idea of the mathematical universe: if the world is made of mathematics, then it wouldn't be surprising that mathematics is, to quote Einstein, "appropriate to the objects of reality". I'm hoping that my question might eventually lead to a similar insight.

Objective Philosophical Truths?

Consider again the example of the wrongness of the universal prior and Bayesian updating. Assuming that they are indeed wrong, it seems that the wrongness must be objective truths, or in other words, it's not relative to how the human mind works, or has anything to do with any peculiarities of the human mind. Intuitively it seems obvious that if any other mind, such as a Bayesian expected utility maximizer, is incapable of perceiving the wrongness, that is not evidence of the subjectivity of these philosophical truths, but just evidence of the other mind being defective. But is this intuition correct? How do we tell?

In certain other areas of philosophy, for example ethics, objective truth either does not exist or is much harder to find. To state this in Eliezer's terms, in ethics we find it hard to do better than to identify "morality" with a huge blob of computation which is particular to human minds, but it appears that in decision theory "rationality" isn't similarly dependent on complex details unique to humanity. How to explain this? (Notice that "rationality" and "morality" otherwise share certain commonalities. They are both "ought" questions, and a utility maximizer wouldn't try to answer either of them or be persuaded by any answers we might come up with.)

These questions perhaps offer further entry points to try to attack the larger problem of understanding and mechanizing the process of philosophy. And finally, it seems worth noting that the number of people who have thought seriously about meta-philosophy is probably tiny, so it may be that there is a bunch of low-hanging fruit hiding just around the corner.

Comments (255)

Comment author: Larks 02 August 2010 11:19:07PM *  18 points [-]

The Unreasonable Effectiveness of Astronomy

It seems like human astronomy is more effective than it has any right to be. Why?

First I'll try to establish that there is a mystery to be solved. It might be surprising so see the words "effective" and "astronomy" together in the same sentence, but I claim that human beings have indeed made a non-negligible amount of astronomical progress. To cite one field that I'm especially familiar with, consider galaxies, where we went from having no concept of galaxies, to studies involving the milky way and other groups of light in the sky, to measuring their speed, location, age, and genesis, to the Einstein’s realizations that the flat universe and the Newtonian physics are both likely to be wrong or incomplete.

We might have expected that given we are products of evolution, the amount of our philosophical progress would be closer to zero. The reason for low expectations is that evolution is lazy and shortsighted. It couldn't possibly have "known" that we'd eventually need stargazing abilities to escape the planet. What kind of survival or reproductive advantage could these abilities have offered our foraging or farming ancestors?

From the example of my webcam, we also know that there are eyes in the design space of visual sensors that could be considered highly sensitive, but are incapable of making out distant stars. For example, a weasel is, apparently, incapable of making out more than a dim blurr. Nor would it be able to tell it was missing much, or have any reason to build telescopes.

Why aren't we more like CCTV in our ability to look at the stars? I have some ideas for possible answers, but I'm not sure how to tell which is the right one:

  1. Astronomic ability is "almost" universal in eye space. Low-quality or pathologically horizontal visual receptors are an example of an atypical mind.
  2. Evolution created stargazing ability as a side effect while selecting for the ability to see predators. This seems implausible; being able to see pretty lights in the sky would only serve to distract us.
  3. Stargazing ability is rare and not likely to be produced by evolution. There's no explanation for why we have it, other than dumb luck. This helps explain why there’s no sign of alien life yet; Stargazing is the great filter.
  4. We’re living in an ancestor simulation, which can only be run by species with the ability to escape their home planet, necessitating stargazing powers.

As you can see, progress is pretty limited so far, but I think this is at least a useful line of inquiry, a small crack in the problem that's worth trying to exploit. People used to wonder at the unreasonable effectiveness of philosophy, especially in probability, and I think such wondering eventually contributed to the idea of the philosophical universe if the world is made of philosophy, then it wouldn't be surprising that philosophy is, to para-quote Wei Dai, "appropriate to the objects of reality". I'm hoping that my question might eventually lead to a similar insight.

Comment author: jimrandomh 02 August 2010 11:46:05PM 0 points [-]

Ancient humans used the stars in the night sky as a compass to navigate by, so it would have been selected for. That said, I think (1), that astronomic ability is almost universal in eye space, is true. (At least for eyes that can see predators and obstacles in the dark, which much more strongly selected for than being able to navigate by starlight is.)

Comment author: Larks 03 August 2010 12:29:57AM 1 point [-]

I was going to suggest that, but didn't want to stretch the layout too much. How long ago did we start using the stars to navigate? I wouldn't imagine it would pre-date agriculture, and can't think evidence we could have to suggest our ancestors from before then used the stars to navigate.

Comment author: [deleted] 12 August 2012 10:35:21PM 0 points [-]

That said, I think (1), that astronomic ability is almost universal in eye space, is true. (At least for eyes that can see predators and obstacles in the dark, which much more strongly selected for than being able to navigate by starlight is.)

Huh, not sure. If I look at the sky with my left eye without wearing my glasses, I can barely see the stars, but I'm pretty sure I could see predators.

Comment author: thomblake 27 July 2010 01:59:02PM 13 points [-]

For those who might care, Wiley-Blackwell produces a journal called Metaphilosophy which is now 40 years old. It was founded by Terrell Ward Bynum (one of the major figures in computer ethics) and its current editor is Armen Marsoobian.

Comment author: cousin_it 27 July 2010 08:05:48AM *  10 points [-]

1: Philosophical ability is "almost" universal in mind space. Utility maximizers are a pathological example of an atypical mind.

I wouldn't spend much time thinking about this alternative, because it will probably be true for some ideas of "mind space" and false for others, and I don't believe we have enough information to describe the correct "mind space".

2: Evolution created philosophical ability as a side effect while selecting for something else.

Many people think the ability to argue and comprehend arguments arose due to runaway sexual selection for ability to manipulate and resist manipulation. I'm not sure how to test such an explanation.

Comment author: Cameron_Taylor 27 July 2010 08:39:44AM 0 points [-]

Some people also think the ability to argue and selectively not comprehend arguments arose due to runaway sexual selection for ability to manipulate and resist manipulation.

Comment author: Cameron_Taylor 28 July 2010 08:05:34AM -2 points [-]

This additional point is controversial even here?

Comment author: rwallace 27 July 2010 01:32:58PM 8 points [-]

Aside from more general issues that have been previously addressed (Friendly AI is a pipe dream, and the big advances on philosophical questions have for the most part been made by methods other than philosophy), a couple of specifics:

  1. We were selected for the ability to tell stories and win political arguments, and it seems to me that minds so selected, should be expected to be able to do philosophy, albeit not terribly well -- which is indeed the case.

  2. You criticize the universal prior because it would disagree with our intuition when presented with an alleged halting oracle (because the universal prior takes for granted that the universe is computable, whereas to human intuition this is an open question). On the one hand I have sympathy with your position, because while I would like to think the universe is computable, I also regard it as an open question. On the other hand... do you have any reason other than intuition to believe, in that scenario, our intuition would be right and the universal prior would be wrong?

Comment author: knb 01 August 2010 08:22:06AM 4 points [-]

the big advances on philosophical questions have for the most part been made by methods other than philosophy.

That is a really good point. Psychology, for instance, is old enough that much Psychological theory was developed before more rigorous empirical standards were developed. These early psych. theories were formulated primarily through philosophical methods (introspection, metaphorical/associative reasoning, etc.) They were very unsuccessful when they were eventually held up to empirical standards.

Comment author: CronoDAS 27 July 2010 07:48:32AM *  19 points [-]

One "interesting" thing about philosophy seems to be that as soon as a philosophical issue gets a definitive answer, it ceases to be part of philosophy and instead becomes either mathematics or science. For example, physical sciences were once "natural philosophy". Many social sciences were also once the domain of philosophy; economics, oddly enough, first developed as an offshoot of moral philosophy, and "philosophy of mind" predates the practice of psychology, cognitive science, neurobiology, and the badly-named "computer science" (which is really just a branch of mathematics).

Philosophy seems to be roughly equivalent to the study of confusing questions; when a question is no longer confusing, it stops being philosophy and instead becomes something else.

Comment author: Wei_Dai 27 July 2010 08:00:45AM 11 points [-]

One "interesting" thing about philosophy seems to be that as soon as a philosophical issue gets a definitive answer, it ceases to be part of philosophy and instead becomes either mathematics or science.

Agreed, and I think that accounts for the reputation philosophy has for not being productive. People see the confusion and slow progress in the fields that are still thought of as philosophy, and forget that philosophical progress is what allowed many fields to become mathematics or science.

Comment author: SilasBarta 27 July 2010 12:44:23PM 6 points [-]

Okay, but (for you, Wei_Dai, and anyone else), how about if you look at just the last 30 years, or 100, or 150? How many new, productive fields have been spun off of something recognized as philosophy?

Comment author: Perplexed 31 July 2010 10:24:10PM 5 points [-]

It depends how big you need a "field" to be. Much of philosophical logic split off to become mathematical logic (split complete by about 1930). That left the philosophers pondering things like entailment, tense logic, modality, epistemic logic, etc. But around 1960, Kripke put that stuff on a solid basis, and these kinds of logics are now an important topic in computer science. Certainly utility theory, decision theory, and subjective probability have only come over from philosophy (to econ, math, and AI) within the past 150 years. And there are still philosophers involved at the forefront of all these fields.

Comment author: JoshuaZ 27 July 2010 01:07:57PM *  11 points [-]

One issue that one runs into with your question is how one defines a new field being spun off. Some people have argued that biology didn't really split off from philosophy until the 1850s and 60s, especially with the work of Darwin and Wallace. This is a popular view among Kuhnians who mark a field as becoming science when it gains an overarching accepted paradigm. (However, one could argue that the field left philosophy before it entered science.)

The word "scientist" was first used in 1833, and prior to that "natural philosopher" was used. But certainly by the late 1700s, they were practicing what we could call science. So that argument fails even if one extends the date.

Economics is generally thought of having split off from philosophy when Adam Smith wrote The Wealth of Nations, and that's in the late 18th century. But arguably, merchantilist ideas were a form of economics that predated Smith and were separate from philosophy. And you could push the date farther up, pointing out that until fairly late most of the people thinking about economics are people like Bentham who we think of as philosophers.

Possibly the best example of an area that split off recently might be psychology. Wilhelm Wundt is sometimes regarded as the individual who split that off, doing actual controlled scientific experiments in the late 19th century. But there was research being done by scientists/biologists/natural philosophers much earlier in the 19th century, especially in regards to whether the nervous system was the source of cognition. Wikipedia claims that that work started as early as 1802 with Cabanis (this is surprising to me since I didn't realize he was that early). One could argue given all the subsequent Freudian and Jungian material that psychology didn't really split off from philosophy until that was removed from mainstream psychology which was in the 1960s and 70s. However, that seems like a weak argument.

Linguistics might be another example, but again, how you define the split matters. It also runs into the not tiny issue that much of linguistics spun off from issues of philology, a field already distinct from philosophy. But other areas of linguistics broke off later, and some people still seem to think of issues like Sapir-Whorf as philosophical questions.

So a lot of this seems to depend on definitions, but regardless of definitions it seems clear that no field has spun off in the last 30 years. Going back farther makes the question murkier, but a decent argument can be made that there has been no such spin off in the last 150 years.

Comment author: gwern 27 July 2010 05:26:26PM 7 points [-]

I think psychology is very strongly an example. You have only to read some old psychology textbooks. I read William James's Principles of Psychology (for a Wittgenstein course) from exactly a century ago, and it was a mix of extremely low-level unexplained experimental results and philosophical argumentation about minds and souls (James spending quite a bit of time attacking non-materialist views, of which there were no shortage of proponents). To point to some of the experiments decades earlier and say that it'd already split off is like pointing at Aristotle's biology work as the start of the split between natural philosophy and biology.

Comment author: amcknight 10 March 2012 12:31:32AM 1 point [-]

I would guess that these splits were generally not recognized as splits until much later when we had distinct bodies of work and then we can look back at the initial roots of the topic. This shows that there might be a bunch of roots of new fields present now that simply haven't grown large enough to be recognized yet.

Comment author: Blueberry 27 July 2010 06:20:28PM 1 point [-]

So a lot of this seems to depend on definitions, but regardless of definitions it seems clear that no field has spun off in the last 30 years.

Not even cognitive science? This blog seems to be in the process of splitting off philosophy of mind into cog sci and AI research.

Comment author: SilasBarta 27 July 2010 06:32:22PM *  3 points [-]

How similar is Eliezer Yudkowsky's research program to that which is commonly thought of as "philosophy"?

EY's work intersects with philosophy in the sense that he asks, "What cognitive architecture would make one have these philosophical discussions / intuitions?" But philosophy is not unique for him in this respect -- i.e., he would just as well ask, "What cognitive architecture would make one get this visual sensation that makes these things seem the most salient?"

Certainly, there are definitions, reasonable ones, for philosophy that cover what this site does, but the problem is that Wei_Dai hasn't defined what he means by "philosophy" here.

Comment author: cousin_it 27 July 2010 06:40:48PM *  5 points [-]

Sometime ago I was quite surprised to know that Kevin T. Kelly's work on Ockham's Razor, very rigorous and mathematical in nature, falls under "philosophy". Apparently modern philosophy can get quite awesome when it wants to.

(By the way, someone should really write an introductory LW post about this. I thought Johnicholas Hines would do it, but lately he seems to be missing.)

Comment author: thomblake 02 August 2010 07:19:49PM 2 points [-]

Typically, philosophers do whatever they want and label it 'philosophy', and will claim most positive historical figures as examples of 'philosophers'.

Symetrically, those who are skeptical of the value of philosophy will note that anyone who does anything useful couldn't possibly be doing philosophy, sometimes "by definition".

Comment author: jimrandomh 02 August 2010 07:38:01PM 3 points [-]

Typically, philosophers do whatever they want and label it 'philosophy', and will claim most positive historical figures as examples of 'philosophers'. Symetrically, those who are skeptical of the value of philosophy will note that anyone who does anything useful couldn't possibly be doing philosophy, sometimes "by definition".

Definitely true, and this suggests that the question of whether philosophy is good/bad/useful is fundamentally confused. One definition that I like is that philosophy is any academic study not otherwise classified. That explains why there are so many examples of fields starting out as philosophy, being given a classification and then not being philosophy any more. It also makes most attempts to say things about philosophy as a whole look rather silly. The only problem with this definition is that a few fields, like ethics, have classifications of their own but are too narrow to count as separate fields, so they're classified as subfields. Still, I think that this definition does a good enough job of dissolving silly questions that we can ignore a few special cases.

Comment author: IlyaShpitser 28 July 2010 07:35:42AM *  1 point [-]

Kelly's observation: inductive processes by necessity change their minds multiple times before arriving at the truth.

Kelly's proposal: inductive processes ought to minimize how often they change their minds before truth is reached. (There are some subtle issues here -- this proposal does not contradict "statistical efficiency" considerations, although it's hard to see why at first glance).

Comment author: SilasBarta 28 July 2010 05:32:40PM 0 points [-]

I don't think the work shown on that link would be regarded as typical philosophy -- it's more characteristic of computer science or statistics.

Comment author: RobinZ 28 July 2010 07:53:43PM 1 point [-]

What falls under the category of "typical philosophy", in your opinion?

Comment author: SilasBarta 28 July 2010 09:10:32PM *  1 point [-]

I didn't have a clear-cut definition in mind then -- I just thought that the Kelly link was far enough from being an edge case.

If I had to say, I would take a random selection of articles from the Stanford Encyclopedia of Philosophy, and that gives an idea of what typical philosphy is, as the term is normally used.

Comment author: Alexandros 27 July 2010 12:53:08PM 1 point [-]

I'll just note here the obvious parallel to AI, where everything useful that comes out of it gets reclassified as 'not-AI'.

Comment author: Blueberry 27 July 2010 06:28:52PM *  9 points [-]

While people say this sometimes, I don't think this is accurate. Most of the "AI" advances, as far as I know, haven't shed a lot of light on intelligence. They may have solved problems traditionally classified as AI, but that doesn't make the solutions AI; it means we were actually wrong about what the problems required. I'm thinking specifically of statistical natural language processing, which is essentially based on finding algorithms to analyze a corpus, and then using the results on novel text. It's a useful hack, and it does give good results, but it just tells us that those problems are less interesting than we thought.

Another example is chess and Go computing, where chess programs have gotten very very good just based on pruning and brute-force computation; the advances in chess computer ability were brought on by computing power, not some kind of AI advance. It's looking like the same will be true of Go programs in the next 10 to 20 years, based on Monte Carlo techniques, but this just means that chess and Go are less interesting games than we thought. You can't brute-force a traditional "AI" problem with a really fast computer and say that you've achieved AI.

Comment author: sharpneli 28 July 2010 09:09:52AM 6 points [-]

but it just tells us that those problems are less interesting than we thought.

Extrapolating from the trend it would not suprise me greatly if we'd eventually find out that intelligence in general is not as interesting as we thought.

When something is actually understood the problem suffers from rainbow effect "Oh it's just reflected light from water droplets, how boring and not interesting at all". It becomes a common thing thus boring for some. I, for one, think go and chess are much more interesting games now that we actually know how they are played, not just how to play.

Comment author: Blueberry 28 July 2010 09:17:30PM *  4 points [-]

My point was that go and chess are not actually understood. We don't actually know how they're played. There are hacks that allow programs to get good at those games without actually understanding the patterns involved, but recognizing the patterns involved is what humans actually find interesting about the games.

To clarify, "understanding chess" is a interesting problem. It turns out that "writing a program to be very good at chess" isn't, because it can be solved by brute force in an uninteresting way.

Another example: suppose computer program X and computer program Y are both capable of writing great novels, and human reviewers can't tell the difference between X's novels, Y's novels, and a human's. However, X uses statistical analysis at the word and sentence level to fill in a hard-coded "novel template," whereas Y creates characters, simulates their personality and emotions, and simulates interactions between them. Both have solved the (uninteresting) problem of writing great novels, but Y has solved the (interesting) problem of understanding how people write novels.

(ETA: I suspect that program X wouldn't actually be able to write great novels, and I suspect that writing great novels is therefore actually an interesting problem, but I could be wrong. People used to think that about chess.)

What's happened in AI research is that Y (which is actually AI) is too difficult, so people successfully solve problems the way program X (which is not AI) does. But don't let this confuse you into thinking that AI has been successful.

Comment author: Vladimir_M 28 July 2010 09:44:01PM *  18 points [-]

Blueberry:

My point was that go and chess are not actually understood. We don't actually know how they're played. There are hacks that allow programs to get good at those games without actually understanding the patterns involved, but recognizing the patterns involved is what humans actually find interesting about the games.

That's not really true. In the last two decades or so, there has been lots of progress in reverse-engineering of how chess masters think and incorporating that knowledge into chess engines. Of course, in some cases such knowledge is basically useless, so it's not pursued much. For example, there's no point in teaching computers the heuristics that humans use to recognize immediate tactical combinations where a brute force search would be impossible for humans, but a computer can perform it in a millisecond.

However, when it comes to long-term positional strategy, brute-force search is useless, no matter how fast, and until the mid-1990s, top grandmasters could still reliably beat computers by avoiding tactics and pursuing long-term strategic advantage. That's not possible any more, since computers actually can think strategically now. (This outcome was disappointing in a sense, since it basically turned out that the human grandmasters' extraordinary strategic abilities are much more due to recognizing a multitude of patterns learned from experience than flashes of brilliant insight.)

Even the relative importance of brute-force search capabilities has declined greatly. To take one example, the Deep Blue engines that famously matched Kasparov's ability in 1996 and 1997 relied on specialized hardware that enabled them to evaluate something like 100-200 million positions per second, while a few years later, the Fritz and Junior engines successfully drew against him even though their search capabilities were smaller by two orders of magnitude. In 2006, the world champion Kramnik was soundly defeated by an engine evaluating mere 8 million positions per second, which would have been unthinkable a decade earlier.

Comment author: Blueberry 29 July 2010 09:03:46PM 6 points [-]

Even the relative importance of brute-force search capabilities has declined greatly.

Thanks for updating me; I was indeed thinking of Deep Blue in the mid 90s. Good to know that chess programs are becoming more intelligent and less forceful.

(This outcome was disappointing in a sense, since it basically turned out that the human grandmasters' extraordinary strategic abilities are much more due to recognizing a multitude of patterns learned from experience than flashes of brilliant insight.)

This is what I would expect; a flash of brilliant insight is what recognizing a pattern feels like from the inside.

Comment author: sharpneli 29 July 2010 04:38:39PM 2 points [-]

but Y has solved the (interesting) problem of understanding how people write novels.

I think the whole point in AI research is to do something, not find out how humans do something. You personally might find psychology (How humans work) far more interesting than AI research (How to do things traditionally classified as 'intelligence' regardless of the actual method) but please don't generalize that notion and smack labels "uninteresting" into problems.

What's happened in AI research is that Y (which is actually AI) is too difficult, so people successfully solve problems the way program X (which is not AI) does. But don't let this confuse you into thinking that AI has been successful.

When mysterious things cease to be mysterious they'll tend to resemble the way "X".

Consider the advent of powered flight. By that line of argumentation one could write "We don't actually understand how flight works, There are hacks that allow machines to fly without actually understanding how birds fly." Or we could compare cars with legs and say that transportation is generally just a ugly uninteresting hack.

Comment author: Blueberry 29 July 2010 08:56:57PM -1 points [-]

I think the whole point in AI research is to do something, not find out how humans do something.

Depends on who's doing the research and why. You're right that companies that want to sell software care about solving the problem, which is why that type of approach is so common. On the other hand, I'm reluctant to call a mostly brute-forced solution "AI research", even if it's useful computer programming.

When mysterious things cease to be mysterious they'll tend to resemble the way "X".

No, I think you're missing my point. X is uninteresting not because it is no longer mysterious, but because it has no large-scale structure and patterns. We could consider another novel-writing program Z that writes novels in some other interesting and complicated way that's different than how humans do it, but still has a rich and detailed structure.

Continuing with the flight analogy: rockets, helicopters, planes, and birds all have interesting ways of flying, whereas the "brute force" approach to flight, throwing a rock really really hard, is not that interesting.

Another example: optical character recognition. One approach is to have a database of hundreds of different fonts, put a grid on each character from each font, and come up with a statistical measure that figures out how close the scanned image is to each stored character by looking at the pixels that they have in common. This works and produces useful software, but that approach doesn't actually care about the different letterforms and shapes involved with them. It doesn't recognize that structure, even though that's what the problem is about.

Comment author: drunkpotato 03 August 2010 01:08:53PM 2 points [-]

Arguably, OCR is about taking a small patch of an image and matching that to a finite set of candidate possible ground truths. OCR programs can do this sometimes better than most humans, if the only thing you look at is one distorted character.

OCR has traditionally been a difficult problem and there are some novel applications of statistics and heuristics used to solve it. But OCR is not what we actually care about: the problem is recognizing a document, or symbolically representing a sentence, and OCR is just one small problem we've carved out to help us deal with the larger problem.

Characters are important when they are part of words, and the structure of a document. They are important when they contribute to what the document means, beyond just the raw data of the image scan. Situating a character in the context of the word it's in, the sentence that word is in, and the context of the document (English novel, handwritten letter from the 18th century, hastily scribbled medical report from a German hospital in 1970's) is what allows a human to extrapolate what the character must be, even if the image of the original character is distorted beyond any algorithm's ability to recognize, or even obliterated entirely.

It's this effect of context which is hard to capture and encode into an OCR algorithm. This broader sense, of being able to recognize a character anywhere a human would, which is the end goal of the problem, is what my friends refer to as an AI-complete problem. (Apologies if this community also uses that phrase, I haven't yet seen it here on LW.)

To give a specific example, many doctors use the symbol "circle above a cross" to indicate female, which most people reading would understand. Why? We've seen that symbol before, perhaps many times, and understand what it means. If you've trained your OCR algorithm on the standard set of English alphanumeric characters, then it will attempt to match that symbol and come up with the wrong answer. If you've done unsupervised training of an OCR algorithm on a typical novel, magazine, and newspaper corpus, there is a good chance that the symbol for female does not appear as a cluster in its vector space.

In order to recognize that symbol as a distinct symbol that needs to be somehow represented in the output, an OCR algorithm would have to do unsupervised online learning as it's scanning documents in a new domain. Even then, I'm not sure how useful it would be, since the problem is not recognizing a given character. The problem is recognizing what that character should be given the context of the document you're scanning. The problem of OCR explodes into specializations of "OCR for novels, OCR for 18th century English letters, OCR for American hospitals", and even more.

If we want an OCR algorithm to output something more useful than [funky new character I found], and instead insert "female" into the text database, at some point we have to tell the algorithm about the character. There's not yet that I know of an OCR system that avoids this hard truth.

Comment author: NancyLebovitz 03 August 2010 01:26:59PM 0 points [-]

I like "AI-complete", though it wouldn't surprise me if general symbol recognition and interpretation is easier than natural language, whereas all NP-complete problems are equivalent.

Comment author: drunkpotato 03 August 2010 03:52:36PM 0 points [-]

I kept my initial comment technical, without delving into the philosophical aspects of it, but now I can ramble a bit.

I suspect that general symbol recognition and interpretation is AI-complete, because of these issues of context, world knowledge, and quasi-unsupervised online learning.

I believe there is a generalized learning algorithm (or set of algorithms) that use (at minimum) frequencies and in-built biological heuristics that we use to approach the world. In this view, natural language generation and understanding is one manifestation of this more general learning system (or constantly updating pattern recognition, if you like, though I think there may be more to it than simple recognition). Symbol recognition and interpretation is another.

"Recognition" and "interpretation" are themselves slippery words that hide the how and the what of what it is we do when we see a symbol. Computational linguists and psycholinguistics have done a good job of demonstrating that we know very little of what we're actually doing when we process visual and auditory input.

You are right that AI-complete probably hides finer levels of equivalency classes, wrapped up in the messy issue of what we mean by intelligence. Still, it's a handy shorthand to refer to problems that may require this more general learning facility, about which we understand very little.

Comment author: nazgulnarsil 28 July 2010 06:56:08AM -2 points [-]

"You can't brute-force a traditional "AI" problem with a really fast computer and say that you've achieved AI."

chinese room, etc.

Comment author: Blueberry 28 July 2010 09:22:49PM 3 points [-]

Elaborate? I'm familiar with Searle's Chinese Room thought experiment, but I'm not sure what your point is here.

Comment author: nazgulnarsil 03 August 2010 07:58:35AM 0 points [-]

much of what feels like deep reasoning from the inside has been revealed by experiment to be simple pattern recognition and completion.

Comment author: mattnewport 28 July 2010 09:47:34PM 0 points [-]

Much recent progress in problems traditionally considered to be 'AI' problems has come not from dramatic algorithmic breakthroughs or from new insights into the way human brains operate but from throwing lots of processing power at lots of data. It is possible that there are few grand 'secrets' to AI beyond this.

The way the human brain has developed suggests to me that human intelligence is not the result of evolution making a series of great algorithmic discoveries on the road to general intelligence but of refinements to certain fairly general purpose computational structures.

The 'secret' of human intelligence may be little more than wiring a bunch of sensors and effectors up to a bunch of computational capacity and dropping it in a complex environment. There may be no such thing as an 'interesting' AI problem by whatever definition you are using for 'interesting'.

Comment author: peterward 04 August 2010 01:06:09AM 0 points [-]

I agree with the general argument. I think (some) philosophy is an immature science, or predecessor to a science, and some is in reference to how to do things better, therefore subject to less stringent, but not fundamentally different, standards than science--political philosophy, say (assuming, counterfactually, political thinking were remotely rational). And of course a lot of philosophy is just nonsense--probably most of it. But economics can hardly be called a science. If anything, the "field" has experienced retrograde evolution since it stopped being part of philosophy.

Comment author: SilasBarta 27 July 2010 01:56:41AM *  15 points [-]

What is your definition of philosophy for this article?
Why is it a failing of a highly intelligent mind that it can't "do philosophy"?
Why would a Bayesian EU maximizer necessarily be unable to tell that a computable prior is wrong?
When is Bayesian updating the wrong thing to do?
What should I have learned from your link to Updateless Decision Theory that causes me to suspect that EU maximizing with Bayesian updating on a universal prior is wrong?
Doesn't rationality require identification of one's goals, therefore inheriting the full complexity of value of oneself?
What would count as an example of a metaphilosophical insight?

Comment author: Cameron_Taylor 27 July 2010 07:08:06AM 2 points [-]

What should I have learned from your link to Updateless Decision Theory that causes me to suspect that EU maximizing with Bayesian updating on a universal prior is wrong?

From what I can glean from the UDT descriptions it seems that UDT defines 'updating' to include things that I would prefer to describe as 'naive updating', 'updating wrong' or 'updating the wrong thing'.

Comment author: Vladimir_Nesov 28 July 2010 06:07:20PM 0 points [-]

Pray tell, what is the right thing to update?

Comment author: Nisan 27 July 2010 02:04:29PM 1 point [-]

Doesn't rationality require identification of one's goals, therefore inheriting the full complexity of value of oneself?

Seconded. We can certainly imagine an amoral agent that responds to rational argument — say, a paperclipper that can be convinced to one-box on Newcomb's problem. This gives rise to the illusion that rationality is somehow universal.

But in what sense is an EU-maximizer with a TM-based universal prior "wrong"? If it loses money when betting on a unary encoding of the Busy Beaver sequence, maybe we should conclude that making money isn't its goal.

If someone knows a way to extract goals from an arbitrary agent in a way that might reveal the agent to be irrational, I would like to hear it.

Comment author: Randaly 28 July 2010 08:00:34PM 1 point [-]

For instrumental rationality, yes; for epistemic rationality, no. If the reason the EU-maximizer loses money is because it believes that the encoding will be different than it actually is, then it is irrational.

Comment author: magfrump 27 July 2010 06:35:09AM 1 point [-]

At least as I understand his point about rationality being objective, I assume he means that "given a set of goals and possible decisions, the most effective decision is determined."

I don't really understand why this doesn't apply to morality as such, unless they aren't similar in the way he implies.

Comment author: XiXiDu 28 July 2010 06:05:32PM 0 points [-]

When is Bayesian updating the wrong thing to do?

I think, but I'm not sure... http://lesswrong.com/lw/nc/newcombs_problem_and_regret_of_rationality/

Comment author: SilasBarta 28 July 2010 06:18:38PM 2 points [-]

I'm familiar with that. As I understand it, EY does not say that Bayesian updating is suboptimal. If anything, he says the opposite, that standard rationality gives you the right answer.

Could you be more specific about where you believe that article claims Bayesian updating is the wrong thing to do?

Comment author: Vladimir_Nesov 28 July 2010 06:20:13PM 5 points [-]

Bayesian updating is the wrong thing to do in counterfactual mugging, and the reason TDT goes wrong on that problem is that it updates.

Comment author: red75 28 July 2010 07:20:02PM 0 points [-]

Does "extremal counterfactual mugging" where $100 is replaced by self-destruction of agent and $10000 is replaced by creation of 100 agent's copies (outside of agent's light cone) requires same answer as counterfactual mugging?

Comment author: SilasBarta 28 July 2010 06:23:51PM 0 points [-]

And this is an uncontroversial view here, which one can safely assert as a premise, as Wei_Dai did here?

Comment author: Vladimir_Nesov 28 July 2010 06:26:20PM 3 points [-]

I don't believe it's particularly controversial. There is a question of whether humans retain preference about counterfactual worlds, but decision-theoretically, not-updating in the usual sense is strictly superior, because you get to make decisions you otherwise wouldn't be able to.

Comment author: SilasBarta 28 July 2010 06:38:33PM *  0 points [-]

Okay, then let me try to trace back to the point where we disagree.

As I understand it:

1) Timeless Decision Theory tells you what to do, given your beliefs. Any belief updating would happen before you apply TDT then, so I don't understand how TDT would err in terms of doing a Bayesian update (and that update is wrong to do) -- the error is independent of TDT, as TDT (see below) shields your actions from making losing decisions on the basis of such a "bad" update.

2) TDT can be stated as, "When calculating EU for the outcomes of an action, you must instead weight each outcome's utility by the probability that this action would have led to it if your decision procedure were such that it outputs this action."

So, on counterfactual mugging, even if it reasons that it's not in the winning world, it reasons that its decision theory leads to the highest (TDT-calculated) EU by setting its action to a policy of paying out on losing, as then it can add the utility of the winning side into its EU.

Or does EY agree that TDT fails on CM? (I couldn't tell from the CM article.)

3) Edit: And even if this is a case of Bayesian updating failing, does that generalize to dropping it altogether?

Comment author: XiXiDu 28 July 2010 07:40:11PM 0 points [-]

I just answered due to a strong gut feeling. It's some time since I read that article.

But there's always a way to set up a particular situation (at least regarding thought-experiments) where the optimal strategy is by definition of the rules not to update on evidence. If I remember right it didn't matter if Omega left and so it couldn't remove money anymore, because it was capable of perfect prediction and only does bestow those who do not update, i.e. are irrational agents under most other 'real-life' circumstances.

Anyway, I just tried to point you to to something because nobody replied to that particular question yet. I even had to look up what 'TDT' stands for :-)

Sorry for bothering.

Comment author: Sniffnoy 27 July 2010 02:30:27AM *  4 points [-]

I don't understand the objection to the universal prior. Definition and computation are not the same thing. Yes, definition is subject to Berry's Paradox if you don't differentiate between definition, meta-definition, etc; but computation is not. In particular, what you list as "a short description" is only computable if P is, which a univeral prior won't be. (I would think the non-computability in itself would be more of a problem!)

Comment author: Wei_Dai 27 July 2010 02:38:55AM 0 points [-]

You probably have to read is induction unformalizable? before Berry's Paradox and universal distribution. (There's a link to the former in the latter, but I'm guessing you skipped it.)

Comment author: Sniffnoy 27 July 2010 07:57:34PM 3 points [-]

I may be repeating what Vladimir said here, but it seems to me your objection is basically "Oh shit! We can diagonalize!" (Which if we then collapse the levels can get us a Berry paradox, and others...)

So, yes, it follows that any system of description we can think of, there's some potential truth its corresponding "universal prior" (question - do those exist in general?) won't be able to infer. But the fact that this applies to any such system means we can't use it as a criterion to decide between them. At some point we have to just stop and say, "No, you are not allowed to refer to this concept in formulating descriptions." Maybe computability isn't the best one, but you don't seem to have actually given evidence that would support any other such system over it.

Or am I just missing something big here?

Comment author: Dre 28 July 2010 07:10:24AM 1 point [-]

The thing I got out of it was that human brain processes appear to be able to do something (assign a nonzero probability to a non-computable universe) that our current formalization of general induction cannot do and we can't really explain why this is.

Comment author: Tyrrell_McAllister 28 July 2010 05:14:40AM *  0 points [-]

I may be repeating what Vladimir said here, but it seems to me your objection is basically "Oh shit! We can diagonalize!"

...

Or am I just missing something big here?

I would also like to see this question addressed.

Comment author: Cyan 28 July 2010 03:43:14PM *  0 points [-]

Let me distill down what timtyler and Dre have written into a concisely stated question:

Premise 1: a human's brain implements a computable algorithm
Premise 2: a human can update on evidence and become convinced that a halting oracle exists
I'm not sure if it is true that: when exposed to evidence from an actually existing halting oracle, the posterior probability of an algorithm implementing the predictions of a Premise 2-type human will exceed that of an algorithm that assigns zero prior probability to an uncomputable universe
but if so, then -- Conclusion: an agent with a Solomonoff prior can become convinced that the universe contains a halting oracle.

And my question: did I do a stupid?

Comment author: jimrandomh 28 July 2010 04:36:38PM 3 points [-]

The last step doesn't look valid to me. After updating on the evidence, you have a human who thinks they've seen a halting oracle, and a Solomonoff agent who thinks he's seen a highly improbable event that doesn't involve a halting oracle. The fact that the human assigns a higher probability to the observations is unconvincing, because he could've just been extraordinarily lucky.

Also, there are entities that are impossible to distinguish from halting oracles using all the computational resources in the universe, which are not actually halting oracles. For example, a "can be proven to halt using a proof shorter than 3^^^3 bits" oracle has nonzero probability under the Solomonoff prior.

Comment author: Wei_Dai 28 July 2010 06:45:11PM *  2 points [-]

Also, there are entities that are impossible to distinguish from halting oracles using all the computational resources in the universe, which are not actually halting oracles. For example, a "can be proven to halt using a proof shorter than 3^^^3 bits" oracle has nonzero probability under the Solomonoff prior.

"proof shorter than 3^^^3 bits" means "proof shorter than 3^^^3 bits in some formal system S", right? Then I can write a program P that iterates through all possible proofs in S of length < 3^^^3 bits, and create a list of all TMs provable to terminate in less than 3^^^3 bits in S. Then P checks to see if P itself is contained in this list. If so, it goes into an infinite loop, otherwise it halts.

Now we know that if S is sound, then P halts AND can't be proven to halt using a proof shorter than 3^^^3 bits in S. What happens if we feed this P to your impostor oracle?

Comment author: jimrandomh 28 July 2010 09:04:38PM 0 points [-]

That works if you can guess S, or some S' that is more powerful than S. But might there be a formal system that couldn't be guessed this way? My first thought was to construct a parameterized system somehow, S(x) where S(x) can prove that S(y) halts when a trick like this is used; but that can be defeated by simply iterating over systems, if you figure out the parameterization. But suppose you tried a bunch of formal logics this way, and the oracle passed them all; how could you ever be sure you hadn't missed one? What about a proof system plus a tricky corner case detection heuristic that happens to cover all your programs?

Comment author: Cyan 28 July 2010 04:56:17PM 0 points [-]

The last step doesn't look valid to me.

The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent's posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.

Also, there are entities that are impossible to distinguish from halting oracles using all the computational resources in the universe, which are not actually halting oracles.

Given this, then contra Wei Dai, I don't know how any human attempting to internally implement Bayesian inference could possibly become convinced that a halting oracle exists.

Comment author: jimrandomh 28 July 2010 05:15:56PM 2 points [-]

The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent's posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.

You lost a level of indirection in there; computing the output of an algorithm does not mean believing that the output of that algorithm is true or even plausible. So the agent will correctly predict what the human will say, and believe that the human is mistaken.

Comment author: Cyan 28 July 2010 05:40:17PM 0 points [-]

The level of indirection isn't necessary: the Solomonoff agent's distribution is a weighted mixture of the outputs of all possible Turing machines, weighted according to the posterior probability of that Turing machine being the one that is generating the observations. Any Turing machine that predicts that the putative halting oracle gets one wrong on a particular trial gets downweighted to zero when that fails to occur.

Comment author: Wei_Dai 28 July 2010 08:06:29PM 0 points [-]

The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent's posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.

That looks like the same position that Eliezer took, and I think I already refuted it. Let me know if you've read the one-logic thread and found my argument wrong or unconvincing.

Comment author: Vladimir_Nesov 28 July 2010 09:14:41PM 0 points [-]

The idea is that universal prior is really about observation-predicting algorithms that agents run, and not about prediction of what will happen in the world. So, for any agent that runs a given anticipation-defining algorithm and rewards/punishes the universal prior-based agent according to it, we have an anticipation-computing program that will obtain higher and higher probability in the universal prior-based agent.

This by the way again highlights the distinction between what will actually happen, and what a person anticipates - predictions are about capturing the concept of anticipation, an aspect of how people think, and are not about what in fact can happen.

Comment author: timtyler 27 July 2010 07:31:15PM *  0 points [-]

You objection to the universal prior is: "what if Occam's razor breaks"?!?

Engineers would not normally be concerned with such things. Occam's razor has been fine for hundreds of years. If it ever breaks, we will update our probability expectations with whatever the new distribution is. Surely that is no big deal.

Original priors are not too important, anway. As soon as an agent is born, it is flooded with information from the world about the actual frequencies of things - and its original priors are quickly washed away and replaced with experience. If their experiences involve encounting the uncomputable, agents will simply update accordingly.

So: this topic seems like angels and pinheads - at least to me.

Comment author: Vladimir_Nesov 27 July 2010 10:26:16AM 3 points [-]

I haven't studied algorithmic probability literature in-depth, but it naively seems to me that one can straightforwardly extend the idea of universal probability to arbitrary logical languages, thus becoming able to assign plausibility to all mathematical structures. The same principle as with universal prior, but have a statement valued by the length of the shortest equivalent statement (from no non-logical axioms), and consequently a class of structures gets value from a statement describing it. This takes care of not noticing halting oracles and so on, you just need to let go of the standard theory/model of programs-only.

Comment author: Wei_Dai 27 July 2010 11:20:48AM 4 points [-]

Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski's undefinability theorem), so the universal prior based on L would assign it zero probability. A human would also think that such a device is unlikely, but not infinitely so. (I gave a version of this argument in is induction unformalizable?, which is linked to from Berry's Paradox and universal distribution. Did you read it?)

Comment author: Vladimir_Nesov 27 July 2010 11:29:29AM *  4 points [-]

Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski's undefinability theorem), so the universal prior based on L would assign it zero probability.

What do you mean by "decide the truth value"? Most statements aren't valid or unsatisfiable, there is no truth value for them. We are not assuming any models here, just assigning plausibility to (statement) elements of language's Lindenbaum algebra.

Such a device has no finite description in L (according to Tarski's undefinability theorem), so the universal prior based on L would assign it zero probability.

Whatever model you have in mind, it will be categorized on one side of each statement of the language. We are assigning plausibility to statements, and hence classes of structures, not individual structures (which are like individual points for a continuous distribution).

Comment author: cousin_it 27 July 2010 01:57:25PM *  9 points [-]

Vladimir, ever since I joined this site I've been hearing many interesting not-quite-formal ideas from you, and as my understanding grows I can parse more and more of what you say. But you always seem to move on to the next idea before finishing the last one. I think you should spend way more effort on transforming your ideas into actual theorems with proofs and publishing them online. Sharing "intuitions" only gets us so far.

I have much less trouble reading math papers from unfamiliar fields than reading your informal arguments, because your arguments rely on unstated background assumptions much more than you seem to realize. Properly preparing your results for publication, even if they don't get actually published somewhere peer-reviewed, should fix this problem.

Comment author: Vladimir_Nesov 27 July 2010 02:19:08PM 3 points [-]

I discuss things here because it's fun (and sometimes I learn useful lessons from expressing them here, in addition to my private notes), not because I consider it effective means of communication. The not-quite-formal ideas are most of the time in fact not-quite-formal, rather than informally communicated formal ideas (often because I don't understand the relevant math, a failure I'm working on). The dropped ideas are those I either found useless/meaningless/wrong or those that never came up in the discussion after some point.

Communicating informal ideas is too difficult, specifically because they assume tons of unstated background, background that you not only have to state, but convince people about. This is work both for the writer and for the reader. In addition, these informal ideas are not particularly valuable, which together with difficulty of communication makes the whole endeavor a waste of effort.

(At least on LW, common background gives a chance for some remarks to be understood, without that background having to be delivered explicitly.)

The plan is for all these hunches to eventually come together in a framework for decision theory, that should be transparently mathematical, and thus allow efficient little-hidden-background communication.

Comment author: Wei_Dai 28 July 2010 06:57:08AM *  1 point [-]

I'm afraid I still don't quite understand your idea. Can you explain it a bit more?

For example, suppose I come across a black box that takes a string as input and outputs a 0 or 1. What does your idea say is the probability that it's a halt-problem oracle, or a device that gives the truth value of statements in ZFC?

Or suppose I'm playing a game where I've been given a long string of bits and have to bet on the next one in the sequence. How do I use your idea to decide what to do?

(Feel free to pick your own examples if the above ones are not optimal for explaining your idea.)

Comment author: Vladimir_Nesov 28 July 2010 08:54:17AM 0 points [-]

What's ambiguous with the definition? For example, unsatisfiable statements will get about as big plausibility as the valid ones, and for theories that are not finitely axiomatizable, plausibility is not defined (so you can't ask about plausibility of some models, unless there is a categorical finite theory defining them). How to use this in decision-making is a special case of a more general open problem in ambient control.

Comment author: Wei_Dai 28 July 2010 09:15:21AM 0 points [-]

Part of what confuses me is that you said we're assigning plausibility to classes of structures, not individual structures, but it seems like we'd need to assign plausibility to individual structures in practice.

How to use this in decision-making is a special case of a more general open problem in ambient control.

Can't you give an example using a situation where Bayesian updating is non-problematic, and just show how we might use your idea for the prior with standard decision theory?

Comment author: Vladimir_Nesov 28 July 2010 09:45:10AM *  1 point [-]

If you can refer to an individual structure informally, then either there is a language that allows finitely describing it, or ability to refer to that structure is an illusion and in fact you are only referring to some bigger collection of things (property) of which the object you talk about is an element. If you can't refer to a structure, then you don't need plausibility for it.

Can't you give an example using a situation where Bayesian updating is non-problematic, and just show how we might use your idea for the prior with standard decision theory?

This is only helpful is something works with tricky mathematical structures, and in all cases that seems to need to be preference. For example, you'd prefer to make decisions that are (likely!) consistent with a given theory (make it hold), then it helps if your decision and that theory are expressed in the same setting (language), and you can make decisions under logical uncertainty if you use the universal prior on statements. Normally, decision theories don't consider such cases, so I'm not sure how to relate. Introducing observations will probably be a mistake too.

Comment author: Wei_Dai 28 July 2010 10:05:26AM 1 point [-]

either there is a language that allows finitely describing it

But if you fix a language L for your universal prior, then there will be a more powerful metalanguage L' that allows finitely describing some structure, which can't be finitely described in the base language, right? So don't we still have the problem of the universal prior not really being universal?

I can't parse the second part of your response. Will keep trying...

Comment author: Vladimir_Nesov 28 July 2010 10:08:24AM *  0 points [-]

But if you fix a language L for your universal prior, then there will be a more powerful metalanguage L' that allows finitely describing some structure, which can't be finitely described in the base language, right? So don't we still have the problem of the universal prior not really being universal?

It can still talk about all structures, but sometimes won't be able to point at a specific structure, only a class containing it. You only need a language expressive enough to describe everything preference refers to, and no more. (This seems to be the correct solution to ontology problem - describe preference as being about mathematical structures (more generally, concepts/theories), and ignore the question of the nature of reality.)

(Clarified the second part of the previous comment a bit.)

Comment author: Wei_Dai 29 July 2010 07:32:46PM *  0 points [-]

You only need a language expressive enough to describe everything preference refers to, and no more.

Why do you think that any logical language (of the sort we're currently familiar with) is sufficiently expressive for this purpose?

This seems to be the correct solution to ontology problem - describe preference as being about mathematical structures (more generally, concepts/theories), and ignore the question of the nature of reality.

I'm not sure. One way to think about it is whether the question "what is the right prior?" is more like "what is the right decision theory?" or more like "what is the right utility function?" In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I'm highly uncertain.

ETA: And sometimes I suspect even "what is the right utility function?" is really more like "what is the right decision theory?" than we currently believe. In other words there is objective morality after all, but we're currently just too stupid or philosophically incompetent to figure out what it is.

Comment author: Wei_Dai 27 July 2010 11:57:03AM 0 points [-]

Ok, I think I had misinterpreted your previous comment. I'll have to think over your idea.

Comment author: MichaelVassar 27 July 2010 05:02:54PM 0 points [-]

Maybe the human is a bad philosopher in this case and is simply wrong.

Comment author: timtyler 29 July 2010 07:43:28PM *  -2 points [-]

Re: "Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski's undefinability theorem), so the universal prior based on L would assign it zero probability."

It would never see the infinite description with the 0 probability, though - not enough space-time.

The evidence of the Oracle that the agent would get to see would be in the form of finite sensory inputs - and those would not be assigned zero probability. So: it could update on that evidence just fine - with no problems.

If the agent sees a tiny box with an Oracle inside it, that is just more finite sense-data about the state of the universe to update on - no problem - and no silly p=0 for an empirical observation.

Comment author: JustinShovelain 28 July 2010 09:06:39AM *  0 points [-]

Are you trying to express the idea of adding new fundamental "terms" to your language describing things like halting oracles and such? And then discounting their weight by the shortest statement of said term's properties expressed in the language that existed previously to including this additional "term?" If so, I agree that this is the natural way to extend priors out to handle arbitrary describable objects such as halting oracles.

Stated another way. You start with a language L. Let the definition of an esoteric mathematical object (say a halting oracle) E be D in the original language L. Then the prior probability of a program using that object is discounted by the description length of D. This gives us a prior over all "programs" containing arbitrary (describable) esoteric mathematical objects in their description.

I'm not yet sure how universal this approach is at allowing arbitrary esoteric mathematical objects (appealing to the Church-Turing thesis here would be assuming the conclusion) and am uncertain whether we can ignore the ones it cannot incorporate.

Comment author: Vladimir_Nesov 28 July 2010 09:51:22AM *  0 points [-]

Why "new terms"? If the language can finitely express a concept, my scheme gives that concept plausibility. Maybe this could be extended to lengths of programs that generate axioms for a given theory (even enumerable sets of axioms), rather than lengths of individual finite statements, but I guess that can be stated within some logical language just as well.

Comment author: JustinShovelain 28 July 2010 07:57:39PM 0 points [-]

By new "term" I meant to make the clear that this statement points to an operation that cannot be done with the original machine. Instead it calls this new module (say a halting oracle) that didn't exist originally.

Comment author: Vladimir_Nesov 28 July 2010 09:09:08PM 0 points [-]

What machine???

Comment author: Daniel_Burfoot 27 July 2010 04:54:58PM *  6 points [-]

unreasonable effectiveness of mathematics in the natural sciences, especially in physics

Note that with respect to the power of mathematics, it's as easy to view the cup as half-empty as half-full. Here's Jaynes on the issue:

Phenomena whose very existence is unknown to the vast majority of the human race (such as the diff erence in ultraviolet spectra of Iron and Nickel) can be explained in exhaustive mathematical detail but all of modern science is practically helpless when faced with the complications of such a commonplace fact as the growth of a blade of grass.

Comment author: phob 27 July 2010 05:01:23PM 10 points [-]

A priori, as intelligent beings, we expect the universe at our scale to be immensely complex, since it produced us. I don't view our inability to explain fully phenomena at our scale as unreasonable non-effectiveness.

Comment author: Baughn 28 July 2010 07:48:29PM *  3 points [-]

We should perhaps expect that, but I didn't actually do so until you mentioned it; not for the reasons you stated, at least not as quite that short and obvious a sentence.

Bravo.

Comment author: Mitchell_Porter 29 July 2010 08:31:42AM 2 points [-]

This statement from Jaynes sounds out of date. In the age of genomics and molecular biology, we can also go into exhaustive detail about the growth of a blade of grass.

Comment author: JoshuaZ 27 July 2010 02:22:41AM 6 points [-]

I'm not at all convinced that philosophy has been very successful. Indeed, the fact that there's nothing resembling a consensus among professional philosophers about almost anything you've described as achievements speaks pretty negatively to the success of philosophy. This contrasts strongly with the issue of mathematics where it seems that math has been deeply helpful for many different areas. For many branches of learning, the key to success has been to mathematicize the areas. In contrast, the more rigorous and reliable an area becomes generally the less it resembles what we generally call philosophy.

Comment author: MichaelVassar 27 July 2010 05:05:45PM 8 points [-]

I would just say that most professional 'philosophers' aren't doing 'philosophy' as I mean the term. Ditto professional 'scientists' and 'science'. Robin's data suggests that most MDs are incompetent. Mounds of data suggests the same of most financial professionals. Why not generalize?

I look at the history of philosophy, not at professional philosophers, if I want to find competent philosophy.

Comment author: SilasBarta 27 July 2010 05:50:11PM *  1 point [-]

Also, does the "professional philosopher community" have reality-grounded standards for what constitutes "good philosophy"? And could they say what the consequences would be of making such errors (relative to the current body of knowledge)?

Because without that, then being rejected by mainstream academic philosophy is no more worrisome than if you were criticized for not being up-to-date with the top theology, or not knowing which writers were truly "post-colonial".

From some authors, I get the impression that their standard is no more rigorous than, "what all my buddies in major philosophy departments agree with".

Comment author: Wei_Dai 27 July 2010 02:55:44AM 4 points [-]

Indeed, the fact that there's nothing resembling a consensus among professional philosophers about almost anything you've described as achievements [...]

Really? As far as I can tell, the consensus for Bayesian updating and expected utility maximization among professional philosophers is near total. Most of them haven't heard of UDT yet, but on Less Wrong and at SIAI there also seems to be a consensus that UDT is, if not quite right, at least on the right track.

For many branches of learning, the key to success has been to mathematicize the areas.

But how do you mathematicize an area, except by doing philosophy? I mean real world problems do not come to you in the form of equations to be solved, or algorithms to be run.

Comment author: CarlShulman 27 July 2010 04:22:18AM 7 points [-]

I run into a fair number of epistemologists who are not keen on describing beliefs in terms of probabilities and want to use binary "believe" vs "not believe" terms, or binary "justification." Bayesian updating and utility-maximization decision theory are pretty dominant among philosophers of probability and decision theorists, but not universal among philosophers.

Comment author: utilitymonster 27 July 2010 12:35:12PM 7 points [-]

I'm a philosophy grad student. While I agree that many epistemologists still think it is important to talk in terms of believe/not-believe and justified/non-justfied, I find relatively few epistemologists who reject the notion of credence or think that credences shouldn't be probabilities. Of those who think credences shouldn't be probability functions, most would not object to using a weaker system of imprecise probabilities (Reference: James M. Joyce (2005). How Probabilities Reflect Evidence. Philosophical Perspectives 19 (1):153–178). These people are still pretty much on team Bayesianism.

So, in a way, the Bayesian domination is pretty strong. In another way, it isn't: few debates in traditional epistemology have been translated in Bayesian terms and solved (though this would probably solve very many of them). And many epistemologists doubt that Bayesianism will be genuinely helpful with respect to their concerns.

Comment author: CarlShulman 27 July 2010 12:58:47PM *  0 points [-]

I mostly agree with this.

Comment author: thomblake 27 July 2010 01:49:12PM 8 points [-]

Really? As far as I can tell, the consensus for Bayesian updating and expected utility maximization among professional philosophers is near total. Most of them haven't heard of UDT yet, but on Less Wrong and at SIAI there also seems to be a consensus that UDT is, if not quite right, at least on the right track.

From my (anecdotal but varied) experience talking to professional philosophers about them, I'd (off-the-cuff) estimate 80% are not familiar with expected utility maximization (in the sense of multiplying the probability of outcome by the utility) or Bayesian updating, and of the rest, a significant portion think that the Bayesian approach to probability is wrong or nonsensical, or that "expected utility maximization" is obviously wrongheaded because it sounds like Utilitarianism.

Comment author: MichaelVassar 27 July 2010 05:06:48PM 7 points [-]

That matches my anecdotal and varied experience, and as we know, the singular of anecdote is 'update' and the plural is 'update more'.

Comment author: thomblake 28 July 2010 01:35:38PM 4 points [-]

That matches my anecdotal and varied experience, and as we know, the singular of anecdote is 'update' and the plural is 'update more'.

Should I quote you for this one, or was it someone else originally?

Comment author: timtyler 31 July 2010 09:28:03PM 1 point [-]

"Utilitarianism" is a term for a specific concept hogging a perfectly good name that could be used for something more general: utility-based decision making.

Comment author: SilasBarta 27 July 2010 01:04:40PM 5 points [-]

Just skim the Stanford Encyclopedia of Philosophy articles on probability and see how uncontroversial philosophers in general regard Bayesian inference. I think you'll see that they consider it problematic and controversial in general.

Comment author: JoshuaZ 27 July 2010 03:10:19AM *  6 points [-]

Really? As far as I can tell, the consensus for Bayesian updating and expected utility maximization among professional philosophers is near total.

According to the The PhilPapers Survey, 25.8% (ETA:Wrong number, 23.6% is the correct value. I quoted from the wrong entry) of surveyed philosophers were consequentialists of some form. That makes it hard to argue for a consensus about maximizing expected utility.

But how do you mathematicize an area, except by doing philosophy? I mean real world problems do not come to you in the form of equations to be solved, or algorithms to be run.

This seems to run into SilasBarta's inquiry above about what you mean by philosophy. I wouldn't for example think of the work of people like Galileo and Newton to be doing philosophy, but they took physics and put it on solid mathematical grounding. Similar remarks apply to Lavoisier or many people in other fields.

Comment author: utilitymonster 27 July 2010 12:24:06PM 6 points [-]

According to the The PhilPapers Survey, 25.8% of surveyed philosophers were consequentialists of some form. That makes it hard to argue for a consensus about maximizing expected utility.

There are a lot of philosophers who buy into maximizing expected utility, but aren't consequentialists. Proof: If you look at philosophers specializing in decision theory, 58% buy into consequentialism link. Of this group, the vast majority would go for something very close to expected utility maximization.

Part of this has to do with consequentialism not having a crisp definition that fits philosophers' intuitive usage. Some think consequentialism must be agent-neutral and get off the boat there (but could still be EU maximizers). Others have preferences that could (if made more coherent) satisfy the axioms of decision theory, but don't think that the utility function that represents those preferences also orders outcomes in terms of goodness. I.e., these people want to be EU maximizers, but don't want to maximize goodness (maybe they want to maximize some weighting of goodness and keeping their hands clean).

Comment author: JoshuaZ 27 July 2010 12:40:49PM *  1 point [-]

Valid point. The question asked was "Normative ethics: deontology, consequentialism, or virtue ethics?" (Note I actually quoted from the wrong entry above with the correct value as 23.6% but this makes little difference). It seems fair that the vast majority of deontologists and virtue ethicists are not EU maximizers. So, let's include everyone who picked consequentalist or "other" as an option. This should presumably overestimate the fraction which we care about for this purpose. That's a total of 55.9%, only slightly over half. Is that a consensus?

Comment author: NancyLebovitz 28 July 2010 01:18:38AM 2 points [-]

I don't think philosophy is unreasonably effective. It's at least plausible that we've got some ability to become conscious of relevant similarities. This ability is useful in a wide range of contexts. [1]

Once you have the ability, it's unsurprising if it's effective when an effort is made to apply it to broad commonalities of how the world and experience work.

[1] Even if we were mostly selected for talking each other into things, I don't know how conscious the process is for people who are naturally good at it. Anyone have information?

My hypothesis is that while there are elementary skills which are unconscious for those with natural skills, there are more complex problems that such people handle consciously.

Comment deleted 27 July 2010 07:23:42PM [-]
Comment author: Wei_Dai 28 July 2010 08:15:24AM *  1 point [-]

So the question is not "why don't we have any self-shadowing blind-spots", it is "why do we have a nontrivial set of non-self-shadowing blind spots?"

Agreed, but I think it's also, "why do we have fewer self-shadowing blind-spots than we might expect, given what we know about how evolution works?"

And while you're right that we can't be sure at this point that we have zero self-shadowing blind-spots (philosophical oversights that we'll never detect), I think there's a reasonable chance that's in fact the case. ETA: My argument for this belief is that one possible reason why we have fewer self-shadowing blind-spots than we might expect is that there is a single "ability to philosophize" that is sufficient (given enough raw intelligence) to overcome all such blind spots.

Comment author: Yvain 28 July 2010 08:18:13PM 3 points [-]

The opposite explanation also works: we use so many unrelated heuristics that there's no single area where they all fail simultaneously.

Comment author: Will_Sawin 03 August 2010 11:59:24PM 0 points [-]

If some of the heuristics are failing and some are succeeding, they are producing different results. Which process determines which results are correct? Should this be called "philosophical ability"?

(non-rhetorical questions)

Comment author: Yvain 05 August 2010 11:06:09PM 0 points [-]

It doesn't necessarily have to be cenralized. Some heuristics could have different weights than others, and stronger ones win out. Or there could be a reflective equilibrium among them.

...not that there's any evidence for any of this.

Comment deleted 27 July 2010 07:51:26PM *  [-]
Comment author: Wei_Dai 28 July 2010 08:15:44AM 2 points [-]

The timeless/updateless/acausal intuitions come from the human intuitions about pride, dignity, honor, etc, which were developed because humans interacted with other humans.

Umm... I first thought of the updateless idea while trying to figure out anthropic reasoning. I fail to see how that had anything to do with pride, dignity, honor, etc.

Comment author: daedalus2u 28 July 2010 02:06:26AM -2 points [-]

I think this is correct. Using my formulation, the Bayseian system is what I call a "theory of reality", the timeless one is the "theory of mind", which I see as the trade-off along the autism spectrum.

Comment author: Liron 27 July 2010 06:56:37AM *  2 points [-]

For example, a Bayesian expected utility maximizer programmed with a TM-based universal prior would not be able to realize that the prior is wrong.

What does it mean to "realize that a prior is wrong"? The mechanics of belief change in a Bayesian agent are fixed by the prior itself.

Nor would it be able to see that Bayesian updating is the wrong thing to do in some situations.

Bayesian updating is always the right thing to do. The only question is how to approximate a proper Bayesian update using efficient data structures and algorithms.

. . . it may be that there is a bunch of low-hanging fruit hiding just around the corner.

I would stay in the fruit tree metaphor and say they might be "hanging right over our heads".

Comment author: cousin_it 27 July 2010 08:17:24AM *  5 points [-]

A prior can be wrong if it assigns zero weight to the true state of the world. For example, if our universe does in fact contain halting problem oracles, the Bayesian superintelligence with a TM-based universal prior will never be able to believe that, no matter how many hard math problems get successfully solved by this weird black box. But a human would converge on the true belief pretty quickly. All this stuff, and more, is in Wei Dai's examples.

Comment author: Eliezer_Yudkowsky 27 July 2010 10:39:14AM 9 points [-]

AIXI with a TM-based universal prior will always produce predictions about the black box, and predictions about the rest of the universe based on what the black box says, that are just as good as any prediction the human can come up with. After all, the human is in there somewhere. If you think of AIXI as embodying all computable ways of predicting the universe, rather than all computable models of the universe, you may begin to see that's not quite as narrow as you thought.

Comment author: Wei_Dai 27 July 2010 10:54:25AM *  6 points [-]

Eliezer, that was your position in this thread, and I thought I had convinced you that it was wrong. If that's not the case, can you please re-read my argument (especially the last few posts in the thread) and let me know why you're not convinced?

Comment author: Eliezer_Yudkowsky 28 July 2010 08:33:04AM 2 points [-]

So... the part I found potentially convincing was that if you ran off a logical view of the world instead of a Solomonoff view (i.e., beliefs represented in e.g. higher-order logic instead of Turing machines) and lived in a hypercomputable world then it might be possible to make better decisions, although not better predictions of sensory experience, in some cases where you can infer by reasoning symbolically that EU(A) > EU(B), presuming that your utility function is itself reasoning over models of the world represented symbolically. On the other hand, cousin_it's original example still looks wrong.

Comment author: Wei_Dai 28 July 2010 09:08:53AM *  1 point [-]

not better predictions of sensory experience

You can make better predictions if you're allowed to write down your predictions symbolically, instead of using decimal numbers. (And why shouldn't that be allowed?)

ETA: I made this argument previously in the one-logic thread, in this post.

ETA 2: I think you can also make better (numerical) predictions of the form "this black box is a halting-problem oracle" although technically that isn't a prediction of sensory experience.

Comment author: Vladimir_Nesov 29 July 2010 08:32:26PM 0 points [-]

Why would you want to make any predictions at all? Predictions are not directly about value. It doesn't seem that there is a place for the human concept of prediction in a foundational decision theory.

Comment author: Wei_Dai 29 July 2010 08:41:06PM 1 point [-]

It doesn't seem that there is a place for the human concept of prediction in a foundational decision theory.

I think that's right. I was making the point about prediction because Eliezer still seems to believe that predictions of sensory experience is somehow fundamental, and I wanted to convince him that the universal prior is wrong even given that belief.

Comment author: Vladimir_Nesov 29 July 2010 08:44:59PM *  1 point [-]

Still, universal prior does seem to be a universal way of eliciting what the human concept of prediction (expectation, probability) is, to the limit of our ability to train such a device, for exactly the reasons Eliezer gives: whatever is the concept we use, it's in there, among the programs universal prior weights.

ETA: On the other hand, the concept thus reconstructed would be limited to talk about observations, and so won't be a general concept, while human expectation is probably more general than that, and you'd need a general logical language to capture it (and a language of unknown expressive power to capture it faithfully).

ETA2: Predictions might still be a necessary concept to express the decisions that agent makes, to connect formal statements with what the agent actually does, and so express what the agent actually does as formal statements. We might have to deal with reality because the initial implementation of FAI has to be constructed specifically in reality.

Comment author: Wei_Dai 29 July 2010 09:04:27PM *  1 point [-]

Umm... what about my argument that a human can represent their predictions symbolically like "P(next bit is 1)=i-th bit of BB(100)" instead of using numerals, and thereby do better than a Solomonoff predictor because the Solomonoff predictor can't incorporate this? Or in other words, the only reason the standard proofs of Solomonoff prediction's optimality go through is that they assume predictions are represented using numerals?

Comment author: timtyler 31 July 2010 09:33:02PM -1 points [-]

Surely predictions of sensory experience are pretty fundamental. To understand the consequences of your actions, you have to be able to make "what-if" predictions.

Comment author: timtyler 31 July 2010 09:30:59PM *  0 points [-]

Re: "It doesn't seem that there is a place for the human concept of prediction in a foundational decision theory."

You can hardly steer yourself effectively into the future if you don't have an understanding of the consequences of your actions.

Comment author: Vladimir_Nesov 01 August 2010 08:01:10AM *  0 points [-]

You can hardly steer yourself effectively into the future if you don't have an understanding of the consequences of your actions.

Yes, it might be necessary exactly for that purpose (though consequences don't reside just in the "future"), but I don't understand this well enough to decide either way.

Comment author: cousin_it 27 July 2010 11:05:04AM *  3 points [-]

Yes, the human is in there somewhere, but so are many other, incorrect predictors. To adopt their predictions as its own, AIXI neds to verify them somehow, but how? (I'm very confused here and may be missing something completely obvious.)

ETA: yeah, this is wrong, disregard this.

Comment author: cousin_it 29 July 2010 07:55:30PM *  3 points [-]

That took two days to parse, but now I understand how it works. You're right. I apologize to everyone for having defended an incorrect position.

My misconception seems to be popular, though. Maybe someone should write a toplevel post on the right way to think about the universal prior. Though seeing that some other people are even more hopelessly confused than me, and seem to struggle with the idea of "prior" per se, I'm not sure that introducing even more advanced topics would help.

Comment author: DefectiveAlgorithm 25 January 2014 01:21:15PM *  0 points [-]

I don't know much about Solomonoff induction, so I may be wrong about this, but is it not the case that the universal prior only takes into account computable functions which exactly output the sensory data? If that is the case, consider the following scenario:

We have a function F which takes an unbounded natural number N as input and is provably uncomputable for all valid inputs. We have a computable algorithm A which provably outputs lower and upper bounds for F for any valid input. Furthermore, it is provable that no computable algorithm can provably produce tighter bounds on F's output than A (regardless of N). We can see that A outputs the bounds for a closed interval in the set of real numbers. We know that all such intervals (for which the lower and upper bounds are not equal) are uncountable. Now imagine a physical hypercomputer which outputs F(0), then F(1), then F(2), etc. to infinity. No computable algorithm will be able to predict the next symbol output by this hypercomputer, but there will be computable minds capable of recognizing the pattern and so of using A to place stronger bounds on its predictions of future sensory experience than AIXI can.

EDIT: Actually, this scenario might be broken. Specifically, I'm not sure what it physically means to 'output' an uncomputable number, and I think that AIXI's problem dissolves if we limit ourselves to the computable (and thus countable) subsets of the output intervals.

Comment author: Vladimir_Nesov 27 July 2010 10:45:31AM *  0 points [-]

Is there a good exposition of this semantics (more generally, for algorithmic probability)?

Comment author: PhilGoetz 27 July 2010 07:59:02PM *  4 points [-]

For example, if our universe does in fact contain halting problem oracles, the Bayesian superintelligence with a TM-based universal prior will never be able to believe that.

I think this problem would vanish if you spelled out what "believe" means. The Bayesian superintelligence would quickly learn to trust the opinion of the halting problem oracle; therefore, it would "believe" it.

Comment author: timtyler 30 July 2010 05:02:29PM *  -2 points [-]

I am having a few problems in thinking of a sensible definition of "believe" in which the superintelligence would fail to believe what its evidence tells it is true. It would be especially obvious if the machine was very small. The superintelligence would just use Occcam's razor - and figure it out.

Of course, one could imagine a particularly stupid agent, that was too daft to do this - but then it would hardly be very much of a superintelligence.

Comment author: timtyler 27 July 2010 07:44:25PM *  -1 points [-]

P(true) = 0 - or p(false) = 1 - seem like trivial mistakes to avoid.

A "expected utility maximizer programmed with a TM-based universal prior" would surely not care very much if it was programmed with wrong priors after a while - since it would not be depending on the details of its priors much any more - due to having a big mountain of experience concerning what the actual expected frequency of events was. Its priors would be swamped by data - unless its priors were completely crazy.

The OP must be thinking of some different type of construct from me - and he doesn't seem to explain what it is.

Comment author: cousin_it 27 July 2010 07:52:02PM *  4 points [-]

P(true) = 0 or p(false) = 1 seem like trivial mistakes to avoid.

Unfortunately they aren't. A universal prior must enumerate all the ways a universe could possibly be. If your prior is based on Turing machines that compute universes, but our actual universe is uncomputable, you're screwed forever no matter what data comes in. Maybe the problem can be solved by a better universal prior, as Nesov suggests elsewhere in the thread, but as far as I understand it's an open problem right now.

ETA: pretty much this whole comment is wrong. The prior is over algorithms that generate sequences of sensory input, not over algorithms that define universes. This is an important distinction, sorry for missing it when I wrote this comment.

Comment author: PhilGoetz 27 July 2010 07:56:18PM 0 points [-]

Natural selection solves this problem.

Comment author: SilasBarta 27 July 2010 08:39:52PM 0 points [-]

A universal prior must enumerate all the ways a universe could possibly be. If your prior is based on Turing machines that compute universes, but our actual universe is uncomputable, you're screwed forever no matter what data comes in.

Being forced to use the nearest computable approximation to an uncomputable function does not make you screwed forever.

Comment author: cousin_it 27 July 2010 08:42:01PM *  1 point [-]

That depends on the uncomputable function. Some can make you very well screwed indeed. It's all there in Wei Dai's examples on everything-list and one-logic, I really wish people would read them, maybe we'd have an actual discussion then. Sorry for sounding harsh.

Comment author: SilasBarta 27 July 2010 08:49:08PM *  1 point [-]

That depends on the uncomputable function. Some can make you very well screwed indeed.

Right, but it's not necessarily true, or even likely, hence my point.

It's all there in Wei Dai's examples on everything-list and one-logic, I really wish people would read them, maybe we'd have an actual discussion then.

I did read the links, (including the link to the empty stub article!), and the google group discussions all seemed to end, from my brief perusing of them, with them coming to the consensus that Wei Dai hadn't established his provacative, counterintuitive point. (And some of the exchanges here show the same.)

At the very least, he should summarize the reasoning or examples, as per standard practice, so we know there's something to be gained from going to the links. This is especially true given that most readers had assumed that the opposite of Wei Dai's premises are true and uncontroversial.

Comment author: timtyler 27 July 2010 08:00:50PM *  -1 points [-]

To avoid such a trivial mistake, just follow the advice on:

http://lesswrong.com/lw/mp/0_and_1_are_not_probabilities/

Comment author: Blueberry 27 July 2010 09:05:56AM 2 points [-]

I would stay in the fruit tree metaphor and say they might be "hanging right over our heads".

Yeah, he really saw the light, but dropped the ball, when writing that stormy bag of mixed metaphors.

Comment author: Wei_Dai 28 July 2010 12:32:55AM *  0 points [-]

. . . it may be that there is a bunch of low-hanging fruit hiding just around the corner.

I would stay in the fruit tree metaphor and say they might be "hanging right over our heads".

Gee, that was obviously supposed to be a non-mixed metaphor about urban foraging. Yeah that's it. :)

Seriously, I thought about sticking with the fruit tree metaphor, but "hanging right over our heads" makes the problem sound too easy, so I decided to favor accuracy over literary elegance.

Comment author: nhamann 27 July 2010 06:46:27AM 2 points [-]

Do you think that the best way achieve solutions to meta-philosophy is to actually do philosophy? I ask because, like other posters, I'm skeptical of the magnitude of contributions from the field of philosophy with regard to philosophical insights. I'm definitely biased, being not nearly as familiar with philosophy as I am with science, but it seems to me that math and science do a great majority of the heavy lifting.

This is not to downplay the importance of philosophy in general, because I think Daniel Dennett is spot on when he says "[T]here is no such thing as philosophy-free; there is only science whose philosophical baggage is taken on board without examination." (I think there's a good argument that the entire symbolic AI program was a major philosophy-fail). It's really more a question of methodology.

The issue I see is that to giving a satisfactory answer to a question like "what is the nature of philosophy, and how do we use it?" likely involves a whole hell of a lot of neuroscience, psychology, linguistics, etc. research, and it seems unlikely that much can be gained by a philosophical approach to the question (which I assume involves surveying all the relevant scientific and philosophical literature, and then making reasoned arguments for why some aspect of philosophy use must be a certain way).

Comment author: Wei_Dai 27 July 2010 07:06:52AM 5 points [-]

Do you think that the best way achieve solutions to meta-philosophy is to actually do philosophy?

I don't know, but I think it's at least plausible that the answer is yes. This is one of those situations where we should probably take multiple approaches simultaneously.

The issue I see is that to giving a satisfactory answer to a question like "what is the nature of philosophy, and how do we use it?" likely involves a whole hell of a lot of neuroscience, psychology, linguistics, etc. research [...]

Maybe, but before von Neumann and Morganstern invented expected utility maximization, it might have seemed like we'd need a whole lot of neuroscience, psychology, linguistics, etc., to figure out decision theory, but that would have turned out to be wrong.

Comment author: nhamann 27 July 2010 07:32:19PM 0 points [-]

This is one of those situations where we should probably take multiple approaches simultaneously.

This is reasonable, and I agree.

Comment deleted 27 July 2010 07:18:06PM [-]
Comment author: Wei_Dai 27 July 2010 11:13:46PM 7 points [-]

Eliezer once complained that I wrote in an "obvious to Eliezer" style and should try to get beyond that. Well, I think what I'm doing is rational given my goals. Unlike Eliezer, whose plans depend on convincing a significant fraction of humanity that existential risk is something to take seriously and that his own approach for solving it (i.e., FAI) is correct, my current aims are mainly to answer certain confusing questions. I don't see much benefit in spending a lot of effort trying to get people to understand my ideas, or even to convince them that my problems should interest them, unless there's a reasonable chance they might contribute to the solution of those problems or point out where my ideas are wrong.

Or it might be that I'm just too lazy to write well and I'm rationalizing all this. :)

Comment author: SilasBarta 27 July 2010 08:12:53PM 5 points [-]

I'm surprised (and a tad disappointed) it got as high as 11! It casually assumes controversial, questionable premises and doesn't clearly define what its thesis is.

What exactly did you learn, and what are the answers to all my questions?

Comment author: cousin_it 27 July 2010 07:21:38PM *  2 points [-]

Me too. I usually reread Wei Dai's posts many times over months or even years, always finding new bits of insight that I missed the previous time.

Comment deleted 27 July 2010 07:27:48PM [-]
Comment author: PhilGoetz 27 July 2010 08:10:31PM *  3 points [-]

If "down-to-earth" means "demonstrating a connection with reality", then yes. There are some ideas here, but no definitions, examples, elaborations, or empirical support.

Comment author: PhilGoetz 27 July 2010 04:38:36PM *  0 points [-]

OP wrote:

It seems like human philosophy is more effective than it has any right to be. Why?

and I said, "What? Huh? Not!" Then OP wrote:

To cite one field that I'm especially familiar with, consider probability and decision theory, where we went from having no concept of probability, to studies involving gambles and expected value, to subjective probability, Bayesian updating, expected utility maximization, and the Turing-machine-based universal prior, to the recent realizations that EU maximization with Bayesian updating and the universal prior are both likely to be wrong or incomplete.

If you want to call math philosophy, then, yes, philosophy is effective. But then the post doesn't make any sense; the issues being raised don't apply. The opening claims philosophy is effective by pointing to math and economics and calling them philosophy. The rest of the post contrasts philosophy with math, and talks about how hard philosophy is, and how non-useful it appears to be; the only example provided is ethics.

I recommend Wei Dai try to rewrite the post, being more specific about what philosophy and meta-philosophy are (and what the main point of the post is), providing many more examples of "philosophy", and paying careful attention that they mean the same thing in all parts of the post.

And I'll say it again: Being a tribal forager is much more intellectually demanding than city-folk think it is.

Comment author: Blueberry 27 July 2010 06:15:39PM 1 point [-]

If you want to do math, you need some basic definitions, concepts, and motivations. Once philosophy has provided those, then you can start quantifying and proving theorems. See the history of economics, for instance.

Comment author: PhilGoetz 27 July 2010 08:00:49PM *  1 point [-]

Philosophy has never provided the basic definitions, concepts, or even motivations for math. The historical influence was the other way around: The successful use of math inspired the invention of philosophy as a rational discipline.

Comment author: Blueberry 27 July 2010 09:41:18PM 2 points [-]

Mathematical logic grew out of the philosophical analysis of arguments. Mathematically rigorous analysis and calculus grew out of the concepts of motion and speed used in understanding physics ("natural philosophy"), which itself grew out of philosophy. Probability and statistics, as applied to controlled studies, grew out of the philosophy leading to the scientific method.

Comment author: Mitchell_Porter 29 July 2010 09:26:22AM 0 points [-]

We can approach this question in a way which completely sidesteps the debate about whether the thinking which gets filed under the name of "philosophy" has been successful or unsuccessful, and whether philosophers are good or bad thinkers. Just try substituting "LessWrong" for "philosophy" throughout Wei's article; the problem is not substantially changed. Can the LessWrong worldview account for its own existence, and for the debate and discussion which occur here every day?

Comment author: ocr-fork 27 July 2010 10:58:04PM 0 points [-]

To cite one field that I'm especially familiar with, consider probability and decision theory, where we went from having no concept of probability, to studies involving gambles and expected value, to subjective probability, Bayesian updating, expected utility maximization, and the Turing-machine-based universal prior, to the recent realizations that EU maximization with Bayesian updating and the universal prior are both likely to be wrong or incomplete.

I don't see how bayesian utility maximizers lack the "philosophical abilities" to discover these ideas. Also, the last one is only half true. The "wrong" link is about decision theory paradoxes, but a bayesian utility maximizer would overcome these with practice.