I figure morality as a topic is popular enough and important enough and related-to-rationality enough to deserve its own thread.

Questions, comments, rants, links, whatever are all welcome. If you're like me you've probably been aching to share your ten paragraph take on meta-ethics or whatever for about three uncountable eons now. Here's your chance.

I recommend reading Wikipedia's article on meta-ethics before jumping into the fray, if only to get familiar with the standard terminology. The standard terminology is often abused. This makes some people sad. Please don't make those people sad.

One of the sharpest and most important tools in the LessWrong cognitive toolkit is the idea of going meta, also called seeking whence or jumping out of the system, all terms crafted by Douglas Hofstadter. Though popularized by Hofstadter and repeatedly emphasized by Eliezer in posts like "Lost Purposes" and "Taboo Your Words", Wikipedia indicates that similar ideas have been around in philosophy since at least Anaximander in the form of the Principle of Sufficient Reason (PSR). I think it'd be only appropriate to seek whence this idea of seeking whence, taking a history of ideas perspective. I'd also like analyses of where the theme shows up and why it's appealing and so on, since again it seems pretty important to LessWrong epistemology. Topics that I'd like to see discussed are:

1. How conservation of probability in Bayesian probability theory and conservation of phase space volume in statistical mechanics are related—a summary of Eliezer's posts on the topic would be great.
2. How conservation of probability &c. are related to other physical/mathematical laws, e.g. Noether's theorem and quantum mechanics' continuity equation.
3. The history of the idea of conservation laws; whether the discovery of conservation laws was fueled by PSR-like philosophical-like concerns (e.g. Leibniz?), by lower level intuitive concerns, or other means.
4. How conservation of probability &c. are related to the idea of seeking whence [pdf] (e.g., "follow the improbability").
5. How the PSR relates to conservation of probability &c. and to seeking whence.
6. How going meta and seeking whence are related/equivalent.
7. Which philosophers have used something like the PSR (e.g. Spinoza, Leibniz) and which haven't; those who haven't, what their reasons were for not using it.
8. What kinds of conclusions are typically reached via the PSR or have historically been justified by the PSR, and whether those conclusions fit with LW's standard conclusions. If it disagrees with LW's standard conclusions, where does the PSR not apply or not apply as strongly; alternatively, why standard LW conclusions might be mistaken.
9. Whether Schopenhauer's four-fold division of the PSR makes sense. (Schopenhauer's a relatively LW-friendly continentalesque philosopher.) A summary of any criticisms of his four-fold division.
10. What makes the PSR, going meta, "JOOTS"-ing and seeking whence appealing, from a metaphysical, epistemological, pragmatic, and psychological perspective. What sorts of environments or problem sets select for it. (The Baldwin effect and similar phenomena might be relevant.)
11. What going meta / seeking whence looks like at different levels of organization; how one jumps out of systems at varying levels.
12. Eliezer's rule of derivative validity from CFAI and how it relates to the PSR; an analysis of how the (moral, or perhaps UDT-like decision-policy-centric) PSR might be relevant to Friendliness philosophy, e.g. as compared with CEV-like proposals [pdf].
13. How latent Platonic nodes in TDT [pdf] (p. 78) relate to the PSR.
14. A generalization of CFAI's causal validity semantics to timeless validity semantics in the spirit of the generalization of CDT to TDT, or perhaps even further generalizations of causal validity semantics in the spirit of Updateless Decision Theory or eXceptionless Decision Theory. (ETA: Whoops, Eliezer already discussed the acausal level, but seems to have only mentioned Platonic forms as an afterthought. Maybe ignore this bullet point.)
15. How the PSR and the rule of derivative validity relate to Robin Hanson's idea of pre-rationality and Wei Dai's questions about extending pre-rationality to include past selves' utility functions—whether this elucidates the relation between XDT and UDT.
16. Where Hofstadter picked up the idea of "going meta" and what led him to think it was important. What led Eliezer to rely on it so much and emphasize the importance of avoiding lost purposes.

Thank you for your cooperation and understanding. Don't worry, there won't be future posts like this, so you don't have to delete my LessWrong account, and anyway I could make another, and another.

But since you've dared to read this far:

Credibility. Should you maximize it, or minimize it? Have I made an error?

Discuss.

Don't be shallow, don't just consider the obvious points. Consider that I've thought about this for many, many hours, and that you don't have any privileged information. Whence our disagreement, if one exists?

Here.

Long story short, it's an attempt to justify the planetarium hypothesis as a solution to the Fermi paradox. The first half is a discussion of how it and things like it are relevant to the intended purview of the blog, and the second half is the meat of the post. You'll probably want to just eat the meat, which I think is relevant to the interests of many LessWrong folk.

The blog is Computational Theology. It's new. I'll be the primary poster, but others are sought. I'll likely introduce the blog and more completely describe it in its own discussion post when more posts are up, hopefully including a few from people besides me, and when the archive will give a more informative indication of what to expect from the blog. Despite theism's suspect reputation here at LessWrong I suspect many of the future posts will be of interest to this audience anyway, especially for those of you who take interest in discussion of the singularity. The blog will even occasionally touch on rationality proper. So you might want to store the fact of the blog's existence somewhere deep in the back of your head. A link to the blog's main page can be found on my LessWrong user page if you forget the url.

I'd appreciate it if comments about the substance of the post were made on the blog post itself, but if you want to discuss the content here on LessWrong then that's okay too. Any meta-level comments about presentation, typos, or the post's relevance to LessWrong, should probably be put as comments on this discussion post. Thanks all!

Are there any essays anywhere that go in depth about scenarios where AIs become somewhat recursive/general in that they can write functioning code to solve diverse problems, but the AI reflection problem remains unsolved and thus limits the depth of recursion attainable by the AIs? Let's provisionally call such general but reflection-limited AIs semi-general AIs, or SGAIs. SGAIs might be of roughly smart-animal-level intelligence, e.g. have rudimentary communication/negotiation abilities and some level of ability to formulate narrowish plans of the sort that don't leave them susceptible to Pascalian self-destruction or wireheading or the like.

At first blush, this scenario strikes me as Bad; AIs could take over all computers connected to the internet, totally messing stuff up as their goals/subgoals mutate and adapt to circumvent wireheading selection pressures, without being able to reach general intelligence. AIs might or might not cooperate with humans in such a scenario. I imagine any detailed existing literature on this subject would focus on computer security and intelligent computer "viruses"; does such literature exist, anywhere?

I have various questions about this scenario, including:

• How quickly should one expect temetic selective sweeps to reach ~99% fixation?
• To what extent should SGAIs be expected to cooperate with humans in such a scenario? Would SGAIs be able to make plans that involve exchange of currency, even if they don't understand what currency is or how exactly it works? What do humans have to offer SGAIs?
• How confident can we be that SGAIs will or won't have enough oomph to FOOM once they saturate and optimize/corrupt all existing computing hardware?
• Assuming such a scenario doesn't immediately lead to a FOOM scenario, how bad is it? To what extent is its badness contingent on the capability/willingness of SGAIs to play nice with humans?

• "Taking Ideas Seriously": Stylistically contemptible, skimpy on any useful details, contributes to norm of pressuring people into double binds that ultimately do more harm than good. I would prefer it if no one linked to or promoted "Taking Ideas Seriously"; superior alternatives include Anna Salamon's "Compartmentalization in epistemic and instrumental rationality", though I don't necessarily endorse that post either.
• "Virtue Ethics for Consequentialists": Stylistically contemptible, written in ignorance of much of the relevant philosophy and psychology literature, contributes to norm of rewarding people who confidently proselytize on subjects of which they do not possess a deep understanding. Thankfully nobody links to this post.

[Post redacted 'cuz I unfairly and carelessly misrepresented someone's views (Eliezer's). The messages of this post was: disbelief that aliens visit Earth in spaceships is a bad reason not to look into ufology. My apologies for this ugly post.]

Responding to this: http://lesswrong.com/r/discussion/lw/8ys/a_way_of_specifying_utility_functions_for_udt/

I had a similar idea a few months ago that highlights different aspects of the problem which I find confusing. In my version the UDT agent controls bits of Chaitin's constant instead of the universal prior directly, seeing as one of the programs that the oracle (which you can derive from Chaitin's omega) has to solve the halting problem for is the UDT agent's. But since the oracle for the oracle you get from Chaitin's constant depends on the latter oracle's bits, you seem to be able to ambiently control THE ENTIRE ARITHMETICAL HIERARCHY SAY WHAT!? That's the confusing part; isn't your one true oracle supposed to screen you off from higher oracles? Or is that only insofar as you can computably verify?

Anyway I like this theme of controlling computational contexts as it forms a tight loop between agent and environment, something currently lacking. Keep it up comrades!

For the last few months I've taken up the habit of explicitly predicting how much karma I'll get for each of my contributions on LW. I picked up the habit of doing so for Main posts back in the Visiting Fellows program, but I've found that doing it for comments is way more informative.

It forces you to build decent models of your audience and their social psychology, the game theoretic details of each particular situation, how information cascades should be expected to work, your overall memetic environment, etc. It also forces you to be reflective and to expand on your gut feeling of "people will upvote this a lot" or "people will downvote this a little bit"; it forces you to think through more specifically why you expect that, and how your contributions should be expected to shape the minds of your audience on average.

It also makes it easier to notice confusion. When one of my comments gets downvoted to -6 when I expected -3 then I know some part of my model is wrong; or, as is often the case, it will get voted back up to -3 within a few hours.

Having powerful intuitive models of social psychology is important for navigating disagreement. It helps you realize when people are agreeing or disagreeing for reasons they don't want to state explicitly, why they would find certain lines of argument more or less compelling, why they would feel justified in supporting or criticizing certain social norms, what underlying tensions they feel that cause them to respond in a certain way, etc, which is important for getting the maximum amount of evidence from your interactions. All the information in the world won't help you if you can't interpret it correctly.

Doing it well also makes you look cool. When I write from a social psychological perspective I get significantly more karma. And I can help people express things that they don't find easy to explicitly express, which is infinitely more important than karma. When you're taking into account not only people's words but the generators of people's words you get an automatic reflectivity bonus. Obviously, looking at their actual words is a prerequisite and is also an extremely important habit of sane communication.

Most importantly, gaining explicit knowledge of everyday social psychology is like explicitly understanding a huge portion of the world that you already knew. This is often a really fun experience.

There are a lot of subskills necessary to do this right, but maybe doing it wrong is also informative, if you keep trying.

One of the causes of schizophrenia might be the Borna disease virus. There are probably other communicable viruses that contribute to schizophrenia, seeing as how craziness is an attractor in mindspace and what not. This may or may not be a good reason to change some of your habits, but I figured I'd at least clue people in on a possible danger. I think there may be an abnormally high number of people around here that value their minds more than their lives.

Most people shouldn't worry about it, but those who are already at high risk might want to look further? It's unclear to me what the value of information is here.

Information on the subject is sort of difficult to come by. There's not much of it. The few papers I've read were interesting but many were many decades old. This paper acts as a decent intro to Borna-specific stuff: http://cmr.asm.org/cgi/content/full/16/3/534?view=long&pmid=12857781 . Priors are hard to get.

This is half-supposed to be a set-up to a joke about the "Borna rule" of dating crazy people but I can't think of a good punch-line.