I use text files. (.txt, because I hate waiting for a rich text editor to open, and I hate autocomplete for normal writing) It's the only way to be able to keep track of them. I sometimes write paper notes when I don't have a computer nearby, but I usually don't keep those notes. Sometimes if I think of something I absolutely have to remember as I'm dozing off to sleep, I'll enter it in my cell phone because I use that as an alarm clock and it's always close to my bed. But my cell phone's keyboard makes writing notes really slow, so I avoid it under normal circumstances.

I have several kinds of notes that I make. One is when I'm doing a hard intellectual task and I want to free up short-term memory, I will write things down as I think of them. I usually title this kind of file with the name of the task. For tasks too minor to remember just by a title like that, I just write something like " <project name> notes 2014-06-22".

I also write "where I left off <date>" notes, whenever I leave a programming project or something for a day (or sometimes even before I leave for lunch), because usually I will be forming ideas about how to fix problems as I'm fixing other problems, so I can save my future self some work by not forgetting them.

Omniscience and omnipotence are nice and simple, but "morally perfect" is a word that hides a lot of complexity. Complexity comparable to that of a human mind.

I think this is eminently arguable. Highly complex structures and heuristics can be generated by simpler principles, especially in complex environments. Humans don't currently know whether human decision processes (including processes describable as 'moral') are reflections of or are generated by elegant decision theories, or whether they "should" be. To my intuitions, morality and agency might be fundamentally simple, with 'moral' decision processes learned and executed according to a hypothetically simple mathematical model, and we can learn the structure and internal workings of such a model via the kind of research program outlined here. Of course, this may be a dead end, but I don't see how one could be so confident in its failure as to judge "moral perfection" to be of great complexity with high confidence.

Edit: in fact, now that I think of it this way, "universe which follows the rules of moral perfection by itself" wins over "universe which follows the rules of moral perfection because there is an ideal rational agent that makes it do so."

By hypothesis, "God" means actus purus, moral perfection; there is no reason to double count. The rules of moral perfection are found implicit in the definition of the ideal agent, the rules don't look like a laundry list of situation-specific decision algorithms. Of course humans need to cache lists of context-dependent rules, and so we get deontology and rule consequentialism; furthermore, it seems quite plausible that for various reasons we will never find a truly universal agent definition, and so will never have anything but a finite fragment of an understanding of an infinite agent. But it may be that there is enough reflection of such an agent in what we can find that "God" becomes a useful concept against which to compare our approximations.

I'm more worried that a term like "safe AGI" could provoke a response of "So you're trying to make sure that a system which is smarter than humans, and able to operate in arbitrary real-world environments, and able to invent new technologies to achieve its goals, will be safe? Let me save you some time and tell you right now that's impossible. Your research program is a pipe dream."

If someone has this reaction, then can't you just say "mission accomplished" and not worry about it too much? In any case, I think "AI safety" is probably the most beneficial to your goals. I would also not be too worried about AI researchers having a knee-jerk response to the term, for the same reasons you do.

Time spent doing any kind of work with a high skill cap.

Edit: Well, okay not any kind of work meeting that criterion, to preempt the obvious LessWrongian response. Any kind you can get paid for is closer to true.

One of my old CS teachers defended treating the environment as adversarial and knowing your source code, because of hackers. See median of 3 killers. (I'd link something, but besides a paper, I can't find a nice link explaining what they are in a small amount of googling).

I don't see why Yudkowsky makes superintelligence a requirement for this.

Also, it doesn't even have to be source code they have access to (which they could if it was open-source software anyway). There are such things as disassemblers and decompilers.

[Edit: removed implication that Yudkowsky thought source code was necessary]

Probably not. Of course, reading and posting to LW is more an opportunity to shirk effective altruism.

This is the kind of thing that when I take the outside view about my response, it looks bad. There is a scholarly paper refuting one of my strongly-held beliefs, a belief I arrived at due to armchair reasoning. And without reading it, or even trying to understand their argument indirectly, I'm going to brush it off as wrong. Merely based on the kind of bad argument (Bad philosophy doing all the work, wrapped in a little bit of correct math to prove some minor point once you've made the bad assumptions) I expect it to be, because this is what I think it would take to make a mathematical argument against my strongly-held belief, and because other people who share my strongly-held belief are saying that that's the mistake they make.

Still not wasting my time on this though.

Bruno de Finetti heard of [the author's empirical Bayes method for grading tests] and he wrote to me suggesting that the student should be encouraged to state their probability for each of the possible choices. The appropriate score should be a simple function of the probability distribution and the correct answer. An appropriate function would encourage students to reply with their actual distribution rather than attempt to bluff. I responded that it would be difficult to get third graders to list probabilities. He answered that we should give the students five gold stars and let them distribute the stars among the possible answers.

- Herman Chernoff (pg 34 of Past, Present, and Future of Statistical Science, available here)

we're human beings with the blood of a million savage years on our hands. But we can stop it. We can admit that we're killers, but we're not going to kill Today.

Captain James Tiberius Kirk dodging an appeal to nature and the "what the hell" effect, to optimize for consequences instead of virtue.

My impression is that they don't, because I haven't seen people who do this as low status. But they've all been people who are clearly high status anyway, due to their professional positions.

This is a bad template for reasoning about status in general, because of countersignaling.