AGI/FAI Theorist for Hire
I'm nearing the end of my employment at SIAI and looking for my next gig. If all else fails I will likely move back to the Bay area (I am currently in Japan) and take a job as a programmer somewhere. However, I would prefer to focus my attention directly on developing AGI and FAI theory. In addition to my current projects (described below), I can try to answer mathematical, philosophical, or other questions for a bit of cash. For some of my previous work, see my page on the arXiv.
In the past year I've been involved in two major projects at SIAI. Steve Rayhawk and I were asked to review existing AGI literature and produce estimates of development timelines for AGI. My work on this project got rather bogged down and proceeded slowly, although I did learn a lot and I've moved in the direction of predicting AGI soonish (5-20 years). After this I tried to produce an AGI technology demo for Google's AGI-11 conference. I was unable to finish my demo in time for the submission deadline, and shortly afterwards SIAI decided to let me go.
I have several projects that I would like to move forward with, and if I can get adequate funds (about $1000 per month to ensure my survival, or $2000 to live comfortably) I will be able to work on them.
Current project ideas:
- Continue development on my incomplete AGI project (optimally, technical details not to be published).
- Write a paper on AGI models that can be used as a basis for FAI research (similar to the way AIXI and its ilk are used now, but closer to reality than AIXI).
- Figure out how an AI can reason formally about using objects in its environment as tools for performing computations.
- I'm also interested in repurposing machine learning algorithms used for finding plausible hypotheses about data distributions into algorithms for finding action policies with high expected utility.
I'm open to suggestions for other topics. I don't consider myself an expert at empiricism, so I prefer to work in domains where I can reason formally. Some thing I'd be up for:
- If you have informal questions or concerns, I can try to think of formal mathematical questions that are similar.
- Once we're dealing with a mathematical question, I can try to answer it.
- If a question looks too hard for me to answer (as will often be the case), I can try to figure out exactly what is hard about it.
- I'm also interested in writing problem sets. If you want to learn about some weird domain that no textbook exists for, I'll try to figure out what some introductory problems in that domain would look like.
Prices for any of these services are negotiable. You can contact me here or at peter@spaceandgames.com.
Some rationality tweets
Will Newsome has suggested that I repost my tweets to LessWrong. With some trepidation, and after going through my tweets and categorizing them, I picked the ones that seemed the most rationality-oriented. I held some in reserve to keep the post short; those could be posted later in a separate post or in the comments here. I'd be happy to expand on anything here that requires clarity.
Epistemology
- Test your hypothesis on simple cases.
- Forming your own opinion is no more necessary than building your own furniture.
- The map is not the territory.
- Thoughts about useless things are not necessarily useless thoughts.
- One of the successes of the Enlightenment is the distinction between beliefs and preferences.
- One of the failures of the Enlightenment is the failure to distinguish whether this distinction is a belief or a preference.
- Not all entities comply with attempts to reason formally about them. For instance, a human who feels insulted may bite you.
Group Epistemology
- The best people enter fields that accurately measure their quality. Fields that measure quality poorly attract low quality.
- It is not unvirtuous to say that a set is nonempty without having any members of the set in mind.
- If one person makes multiple claims, this introduces a positive correlation between the claims.
- We seek a model of reality that is accurate even at the expense of flattery.
- It is no kindness to call someone a rationalist when they are not.
- Aumann-inspired agreement practices may be cargo cult Bayesianism.
- Godwin's Law is not really one of the rules of inference.
- Science before the mid-20th century was too small to look like a target.
- If scholars fail to notice the common sources of their inductive biases, bias will accumulate when they talk to each other.
- Some fields, e.g. behaviorism, address this problem by identifying sources of inductive bias and forbidding their use.
- Some fields avoid the accumulation of bias by uncritically accepting the biases of the founder. Adherents reason from there.
- If thinking about interesting things is addictive, then there's a pressure to ignore the existence of interesting things.
- Growth in a scientific field brings with it insularity, because internal progress measures scale faster than external measures.
Pseudolikelihood as a source of cognitive bias
Pseudolikelihood is method for approximating joint probability distributions. I'm bringing this up because I think something like this might be used in human cognition. If so, it would tend to produce overconfident estimates.
Say we have some joint distribution over X, Y, and Z, and we want to know about the probability of some particular vector (x, y, z). The pseudolikelihood estimate involves asking yourself how likely each piece of information is, given all of the other pieces of information. Then you multiply these together. So the pseudolikelihood of (x, y, z) is P(x|yz) P(y|xz) P(z|xy).
Not only is this wrong, but it gets more wrong as your system is bigger. By that I mean that a ratio of two pseudolikelihoods will tend towards 0 or infinity for big problems, even if the likelihoods are close to the same.
So how can we avoid this? A correct way to calculate a joint probability P(x,y,z) looks like P(x) P(y|x) P(z|xy). At each step we only condition on information "prior" to the thing we are asking about. My guess about how to do do this involves making your beliefs look more like a directed acyclic graph. Given two adjacent beliefs, you need to be clear on which is the "cause" and which is the "effect." The cause talks to the effect in terms of prior probabilities and the effect talks to the cause in terms of likelihoods.
Failure to do this could take the form of an undirected relationship (two beliefs are "related" without either belief being the cause or the effect), or loops in a directed graph. I don't actually think we want to get rid of undirected relationships entirely -- people do use them in machine learning -- but I can't see any good reason for keeping the latter.
An example of a causal loop would be if you thought of math as an abstraction from everyday reality, and then turned around and calculated prior probabilities of fundamental physical theories in terms of mathematical elegance. One way out is to declare yourself a mathematical Platonist. I'm not sure what the other way would look like.
Shock Levels are Point Estimates
This is a post from my blog, Space and Games. Michael Vassar has requested that I repost it here. I thought about revising it to remove the mind projection fallacy, but instead I left it in for you to find.
Eliezer Yudkowsky1999 famously categorized beliefs about the future into discrete "shock levels." Michael Anissimov later wrote a nice introduction to future shock levels. Higher shock levels correspond to belief in more powerful and radical technologies, and are considered more correct than lower shock levels. Careful thinking and exposure to ideas will tend to increase one’s shock level.
If this is really true, and I think it is, shock levels are an example of human insanity. If you ask me to estimate some quantity, and track how my estimates change over time, you should expect it to look like a random walk if I’m being rational. Certainly I can’t expect that my estimate will go up in the future. And yet shock levels mostly go up, not down.
Philadelphia LessWrong Meetup, December 16th
There will be a LessWrong meetup on Wednesday, December 16th (tomorrow). We're meeting at 7:15 PM at Kabul Restaurant at 106 Chestnut Street. Five people have confirmed so far.
The Domain of Your Utility Function
Unofficial Followup to: Fake Selfishness, Post Your Utility Function
A perception-determined utility function is one which is determined only by the perceptual signals your mind receives from the world; for instance, pleasure minus pain. A noninstance would be number of living humans. There's an argument in favor of perception-determined utility functions which goes like this: clearly, the state of your mind screens off the state of the outside world from your decisions. Therefore, the argument to your utility function is not a world-state, but a mind-state, and so, when choosing between outcomes, you can only judge between anticipated experiences, and not external consequences. If one says, "I would willingly die to save the lives of others," the other replies, "that is only because you anticipate great satisfaction in the moments before death - enough satisfaction to outweigh the rest of your life put together."
Let's call this dogma perceptually determined utility. PDU can be criticized on both descriptive and prescriptive grounds. On descriptive grounds, we may observe that it is psychologically unrealistic for a human to experience a lifetime's worth of satisfaction in a few moments. (I don't have a good reference for this, but) I suspect that our brains count pain and joy in something like unary, rather than using a place-value system, so it is not possible to count very high.
The argument I've outlined for PDU is prescriptive, however, so I'd like to refute it on such grounds. To see what's wrong with the argument, let's look at some diagrams. Here's a picture of you doing an expected utility calculation - using a perception-determined utility function such as pleasure minus pain.
Epistemic vs. Instrumental Rationality: Approximations
What is the probability that my apartment will be struck by a meteorite tomorrow? Based on the information I have, I might say something like 10-18. Now suppose I wanted to approximate that probability with a different number. Which is a better approximation: 0 or 1/2?
The answer depends on what we mean by "better," and this is a situation where epistemic (truthseeking) and instrumental (useful) rationality will disagree.
As an epistemic rationalist, I would say that 1/2 is a better approximation than 0, because the Kullback-Leibler Divergence is (about) 1 bit for the former, and infinity for the latter. This means that my expected Bayes Score drops by one bit if I use 1/2 instead of 10-18, but it drops to minus infinity if I use 0, and any probability conditional on a meteorite striking my apartment would be undefined; if a meteorite did indeed strike, I would instantly fall to the lowest layer of Bayesian hell. This is too horrible a fate to imagine, so I would have to go with a probability of 1/2.
As an instrumental rationalist, I would say that 0 is a better approximation than 1/2. Even if a meteorite does strike my apartment, I will suffer only a finite amount of harm. If I'm still alive, I won't lose all of my powers as a predictor, even if I assigned a probability of 0; I will simply rationalize some other explanation for the destruction of my apartment. Assigning a probability of 1/2 would force me to actually plan for the meteorite strike, perhaps by moving all of my stuff out of the apartment. This is a totally unreasonable price to pay, so I would have to go with a probability of 0.
I hope this can be a simple and uncontroversial example of the difference between epistemic and instrumental rationality. While the normative theory of probabilities is the same for any rationalist, the sorts of approximations a bounded rationalist would prefer can differ very much.
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)