= 27d567bc2d48d9f40e9ccc20ea370b15)
If you're curious about someone else's emotions or perspective, first, remember that there are two ways to encode knowledge of how someone else feels: by having a description of their feelings, or by empathizing and actually feeling them yourself. It is more costly --- in terms of emotional energy --- to empathize with someone, but if you care enough about them to afford them that cost, I think it's the way to go. You can ask them to help you understand how they feel, or help you to see things the way they do. If you succeed, they'll appreciate having someone who can share their perspective.
[LINK] General-audience documentary on cosmology, anthropics, and superintelligence
If you have friends or family you'd like to get thinking about cosmology and the like, this might be a nice documentary to stir up curiosity. Despite clearly being aimed at a general audience, I thought this documentary -- including interviews of Tegmark and Bostrom --- did a surprisingly good job of talking about the beginning of the universe and our place in it:
http://www.youtube.com/watch?v=oyH2D4-tzfM
Also, even though I've had all these thoughts before, it still makes me more emotionally motivated to live long enough to see scientific advances on these questions.
In response to
Macro, not Micro
My summary of this idea has been that life is a non-convex optimization problem. Hill-climbing will only get you to the top of the hill that you're on; getting to other hills requires periodic re-initializing. Existing non-convex optimization techniques are often heuristic rather than provably optimal, and when they are provable, they're slow.
skipping over the mechanisms for filtering good ideas from bad leaves me confused about the point of the post.
The point of the post is that most people, in most domains, should not trust that they are good at filtering good ideas from bad.
And the point of CFAR is to help people become better filtering good ideas from bad. It is plainly not to produce people who automatically believe the best verbal argument anyone presents to them without regard for what filters that argument has been through, or what incentives the Skilled Arguer might have to utter the Very Convincing Argument for X instead of the Very Very Convincing Argument for Y. And certainly not to have people ignore their instincts; e.g. CFAR constantly recommends Thinking Fast and Slow by Kahneman, and teaches exercises to extract more information from emotional and physical senses.
What if we also add a requirement that the FAI doesn't make anyone worse off in expected utility compared to no FAI? That seems reasonable, but conflicts the other axioms. For example, suppose there are two agents: A gets 1 util if 90% of the universe is converted into paperclips, 0 utils otherwise, and B gets 1 util if 90% of the universe is converted into staples, 0 utils otherwise. Without an FAI, they'll probably end up fighting each other for control of the universe, and let's say each has 30% chance of success. An FAI that doesn't make one of them worse off has to prefer a 50/50 lottery of the universe turning into either paperclips or staples to a certain outcome of either, but that violates VNM rationality.
And things get really confusing when we also consider issues of logical uncertainty and dynamical consistency.
What if we also add a requirement that the FAI doesn't make anyone worse off in expected utility compared to no FAI?
I don't think that seems reasonable at all, especially when some agents want to engage in massively negative-sum games with others (like those you describe), or have massively discrete utility functions that prevent them from compromising with others (like those you describe). I'm okay with some agents being worse off with the FAI, if that's the kind of agents they are.
Luckily, I think people, given time to reflect and grown and learn, are not like that, which is probably what made the idea seem reasonable to you.
Only a VNM-rational agent can have preferences in a coherent way, so if we're talking about aggregating people's preferences, I don't see any way to do it other than modeling people as having underlying VNM-rational preferences that fail to perfectly determine their decisions.
Non-VNM agents satisfying only axiom 1 have coherent preferences... they just don't mix well with probabilities.
My default answer to that is "all people alive at the time that the singularity occurs", although you pointed out a possible drawback to that (it incentivizes people to create more people with values similar to their own) in our previous discussion.
And also incentivizes people to kill people with values dissimilar to their own!
I don't think it would be terribly problematic. "People in the future should get exactly what we currently would want them to get if we were perfectly wise and knew their values and circumstances" seems like a pretty good rule. It is, after all, what we want.
Fair enough. Hmm.
Dumb solution: an FAI could have a sense of justice which downweights the utility function of people who are killing and/or procreating to game their representation in AI's utility function, or something like that do disincentivize it. (It's dumb because I don't know how to operationalize justice; maybe enough people would not cheat and want to punish the cheaters that the FAI would figure that out.)
Also, given what we mostly believe about moral progress, I think defining morality in terms of the CEV of all people who ever lived is probably okay... they'd probably learn to dislike slavery in the AI's simulation of them.
Thanks for writing this up!
So after reading that, I don't see how it could be true even in the sense described in the article without violating Well Foundation somehow, but what it literally says at the link is that every model of ZFC has an element which encodes a model of ZFC, not is a model of ZFC, which I suppose must make a difference somehow - in particular it must mean that we don't get A has an element B has an element C has an element D ... although I don't see yet why you couldn't construct that set using the model's model's model and so on. I am confused about this although the poster of the link certainly seems like a legitimate authority.
But yes, it's possible that the original paragraph is just false, and every model of ZFC contains a quoted model of ZFC. Maybe the pair-encoding of quoted models enables there to be an infinite descending sequence of submodels without there being an infinite descending sequence of ranks, the way that the even numbers can encode the numbers which contain the even numbers and so on indefinitely, and the reason why ZFC doesn't prove ZFC has a model is that some models have nonstandard axioms which the set modeling standard-ZFC doesn't entail. Anyone else want to weigh in on this before I edit? (PS upvote parent and great-grandparent.)
I don't see how it could be true even in the sense described in the article without violating Well Foundation somehow
Here's why I think you don't get a violation of the axiom of well-foundation from Joel's answer, starting from way-back-when-things-made-sense. If you want to skim and intuit the context, just read the bold parts.
1) Humans are born and see rocks and other objects. In their minds, a language forms for talking about objects, existence, and truth. When they say "rocks" in their head, sensory neurons associated with the presence of rocks fire. When they say "rocks exist", sensory neurons associated with "true" fire.
2) Eventually the humans get really excited and invent a system of rules for making cave drawings like "∃" and "x" and "∈" which they call ZFC, which asserts the existence of infinite sets. In particular, many of the humans interpret the cave drawing "∃" to mean "there exists". That is, many of the same neurons fire when they read "∃" as when they say "exists" to themselves. Some of the humans are careful not to necessarily believe the ZFC cave drawing, and imagine a guy named ZFC who is saying those things... "ZFC says there exists...".
3) Some humans find ways to write a string of ZFC cave drawings which, when interpreted --- when allowed to make human neurons fire --- in the usual way, mean to the humans that ZFC is consistent. Instead of writing out that string, I'll just write <ZFC is consistent> in place of it.
4) Some humans apply the ZFC rules to turn the ZFC axiom-cave-drawings and the cave drawing <ZFC is consistent> into a cave drawing that looks like this:
"∃ a set X and a relation e such that <(X,e) is a model of ZFC>"
where <(X,e) is a model of ZFC> is a string of ZFC cave drawings that means to the humans that (X,e) is a model of ZFC. That is, for each axiom A of ZFC, they produce another ZFC cave drawing A' where "∃y" is always replaced by "∃y∈X", and "∈" is always replaced by "e", and then derive that cave drawing from the cave drawing "<ZFC axioms> and <ZFC is consistent>" according to the ZFC rules.
Some cautious humans try not to believe that X really exists... only that ZFC and the consistency of ZFC imply that X exists. In fact if X did exist and ZFC meant what it usually does, then X would be infinite.
4) The humans derive another cave drawing from ZFC+<ZFC is consistent>:
"∃Y∈X and f∈X such that <(Y,f) is a model of ZFC>",
6) The humans derive yet another cave drawing,
"∃ZeY and geX such that <(Z,g) is a model of ZFC>".
Some of the humans, like me, think for a moment that Z∈Y∈X, and that if ZFC can prove this pattern continues then ZFC will assert the existence of an infinite regress of sets violating the axiom of well-foundation... but actually, we only have "ZeY∈X" ... ZFC only says that Z is related to Y by the extra-artificial e-relation that ZFC said existed on X.
I think that's why you don't get a contradiction of well-foundation.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Not justify: instead, explain. I understood that previously, handoflixue felt that status was dirty, but in understanding it has come to feel that it's just part of human nature (for most people, as the post points out).
I disagree. Justification is the act of explaining something in a way that makes it seem less dirty.