My view about global rationality is similar to that the view of John Baez about individual risk-adversity. An individual should typically be cautious because the maximum downside (destruction of your brain) is huge even for day-to-day actions like crossing the street. In the same way, we have only one habitable planet and one intelligent species. If we (accidentally) destroy either we're boned. Especially when we don't know exactly what we're doing (as is the case with AI) caution should be the default approach, even if we were completely oblivious to the concept of a singularity.

that the most pressing issue is to increase the confidence into making decisions under extreme uncertainty or to reduce the uncerainty itself.

I disagree, it's not the most pressing issue. In a sufficiently complex system there are always going to be vectors we poorly understand. The problem here is that we have a global society where it becomes harder every year for a single part to fail independently of the rest. A disease or pathogen is sure to spread to all parts of the world, thanks to our infrastructure. Failure of the financial markets affect the entire world because the financial markets too are intertwined. Changes in the climate also affect the entire globe, not just the countries who pollute. An unfriendly AI cannot be contained either. Everywhere you look there are now single points of failure. The more connected our world becomes the more vulnerable we become to black swan events that rock the world. Therefore, the more cautious we have to be. The strategy we used in the past 100.000 years (blindly charge forward) got us where we are today but it isn't very good anymore. If we don't know exactly what we're doing we should make absolutely sure that all worst case scenarios affect only a small part of the world. If we can't make such guarantees then we should probably be even more reluctant to act at all. We must learn to walk before we can run.

Under extreme uncertainty we cannot err on the side of caution. We can reduce uncertainty somewhat (by improving our estimates) but there is no reason to assume we will take all significant factors into account. If you start out with a 0.001 probability of killing all of humanity there is no amount of analysis that can rationally lead to the conclusion "eh, whatever, let's just try it and see what happens", because the noise in our confidence will exceed a few parts in a million at the least, which is already an unacceptable level of risk. It took billions of years for evolution to get us to this point. We can now mess it up in the next 1000 years or so because we're in such a damn hurry. That'd be a shame.

Let G be a a grad student with an IQ of 130 and a background in logic/math/computing.

Probability: The quality of life of G will improve substantially as a consequence of reading the sequences.

Probability: Reading the sequences is a sound investment for G (compared to other activities)

Probability: If every person on the planet were trained in rationality (as far as IQ permits) humanity would allocate resources in a sane manner.

Given your argument, I'm a bit confused by why you assign such a high upper bound to P(Solipsism).

Probability: You are living in a simulation run by some sort of intelligence.

Probability: Other people exist independently of your own mind.

Probability: You are dreaming at this very moment. (Learning to dream lucidly is largely a matter of giving this a high probability and keeping it in mind, and updating on it when you encounter, for instance, people asking whether you're dreaming.)

Can you link to examples of irreducibly good academic writing?

6) Save some money and then move to Asia, South America or any other place with very low cost of living so you can do a few years of research full time.

I am fascinated by this idea in principle, but do you know anyone who has actually done it? I fear there are many nonobvious details that would derail the plan. Maybe we should create an LW outpost in Saigon or Bangalore or some other inexpensive place, since there are many people here who are excited about the idea of living inexpensively to free up more time for Big Thinking.

Will you be my life coach? You're amazing.

I have already transferred schools once, moving because there were no advisers in my area at school #1 (the one I had planned to work with became emeritus right as I joined). I like the school I am at now a lot more than I like computer vision. In fact, my main issue with my current situation is that it appears that no one can do fundamental research in computer vision: all of the major conferences require you to pander to shorter term commercial applications if you want to publish and I'd rather move to a new field than jump through those hoops.

I don't consider options 4, 5, 6, or 7 to be remotely realistic for me. I can't think of an Asian or South American countries to where I would be happy with the governments or the long distance from family and friends if I were to live there semi-permanently. Those considerations are at least as important to me as job considerations. I don't consider unemployment or extreme part-time work an option because I have other life goals, like traveling, home ownership, etc., that I want to financially support in addition to whatever career path I choose.

I appreciate your suggestions, but I have really thought about this a considerable amount. The post that I linked above has some more details about what thinking I have already done. I would really appreciate more targeted advice if you are interested. Given the climate for faculty jobs, what is the best way to try to achieve one? What are ways to do theoretical work / teaching at a university level for a living that are non-traditional?

As far as I can tell you identify two options: 1) continue doing the PhD you don't really enjoy 2) get a job you won't really enjoy.

Surely you have more options!

3) You can just do a PhD in theoretical computer vision at a different university.

4) You can work 2 days a week at a company and do your research at home for the remaining 4 days

5) Become unemployed and focus on your research full time

6) Save some money and then move to Asia, South America or any other place with very low cost of living so you can do a few years of research full time.

7) Join a startup company that is doing groundbreaking computer vision work

8) See if there is something else that you can be passionate about and do that.

Life's too short to do something you don't enjoy and you're now at a point in your life where the decisions you make are going to have real consequences. So do some soul searching and figure out what you really want and then figure out what you have to do to make it happen. That's life 101.

When you're spending the majority of your time doing something you don't really enjoy you have a big problem. This is the only life you have and it's easy to waste it 5 years at the time! Maybe your true dream is to work on the next Pixar movie, or to design special effects for the next CGI blockbuster! But if you aren't going to explore your options seriously you're not going to find out what you really want to do in life. If, on the other hand, you're absolutely sure you want to do theoretical computer vision research, then JUST DO THAT. There are thousands of universities with good computer vision departments. So unless you got a 1000 rejection letters on your desk you haven't even seriously explored your options yet.

(PS: Forget about doing research in the evenings after you get home from a day job. It doesn't work. Many people do this and then they figure out that after a full day's work you don't have the energy anymore to do really difficult stuff. Your lifestyle will change and you'll grow dependent on your job. Then as you get older you'll look back and call it a "silly dream" and wisely observe that you have to make compromises in life and that your ability to compromise on what you want makes you a responsible adult.)

(PPS: I'm trying to convey that being unhappy with your job should trigger "hair on fire" like panic.)

Just to clarify, does "irreducible" in (3) also mean that qualia are therefore extra-physical?

Not unless we are arguing over definitions. Tabooing the phrase "extra-physical", what Eliezer and Chalmers were arguing (or trying to argue) about is whether a superintelligent observer, with full knowledge of the physical state of a brain, would have the same level of certainty about the qualia that the brain experiences as it does about the physical configuration of the brain.

Actually, if they had phrased the debate in those terms it would have turned out better. I don't think that what they were arguing about was clearly defined by either party, which is why it has been necessary (in my humble opinion) for me to "repair" Eliezer's contribution.

So anyway, no it does not mean the same thing. I argue that qualia are not "extra-physical", because the observer does in fact have the same level of knowledge about the qualia as it does about the physical Universe. However, this only proves that qualia supervene upon physical brain states and does not demonstrate that qualia can ever be explained in terms of quarks (rather than "psycho-physical bridging laws" or some such idea).

It might be tempting to refer to (a degree of) belief in irreducibility of qualia as "non-physical", but for the purposes of this discussion it would confound things.

So doesn't that leave us with two options:

1) Evolution went from single cell prokaryotes to Homo Sapiens and somewhere during this period the universe went "plop" and irreducible qualia started appearing in some moderately advanced species.

2) Qualia are real and reducible in terms of quarks like everything else in the brain. As evolution produced better brains at some point it created a brain with a minor sense of qualia. Time passed. Brains got better and more introspective. In other words: qualia evolved (or "emerged") like our sense of smell, our eyesight and so forth.

I don't think that there's a good reason why you didn't describe qualia as "plopping" into existence in scenario 2 as well, or else in neither scenario. Since (with extreme likelihood) qualia supervene upon brain states whether they are irreducible or reducible, the existence of suitable brain states (whatever that condition may be) seems likely to be a continuous rather than discrete quality. "Dimmer" qualia giving way to "brighter" qualia, as it were, as more complex lifeforms evolve.

Note the similarity to Eliezer's post on the many worlds hypothesis here.