= 27d567bc2d48d9f40e9ccc20ea370b15)
Good point. May I ask, is "explicit utility function" standard terminology, and if yes, is there a good reference to it somewhere that explains it? It took me a long time until I realized the interesting difference between humans, who engage in moral philosophy and often can't tell you what their goals are, and my model of paperclippers. I also think that not understanding this difference is a big reason why people don't understand the orthagonality thesis.
In response to
The paperclip maximiser's perspective
Why does she care about music and sunsets? Why would she have scope insensitivity bias? She's programmed to care about the number, not the log, right? And if she was programmed to care about the log, she'd just care about the log, not be unable to appreciate the scope.
Not necessarily. You are assuming that she has an explicit utility function, but that need not be the case.
I predict the results to stay near to where they are (at N=20) however what this means for how we might better model people is unclear. (it might be reasonable to think this subset of population is in fact a collection of unusual thinkers but I would say its safe to assume that this is representative of most of the population in this case)
Do we need to start modelling people as more to ourselves (as we all seem to feel like we have unusual though processes) or less (as we might have unusual processes in different directions to each other)? would doing either make us more effective at life?
Honestly, I suspect that the average person models others after themselves even if they consider themselves to be unusual. So this poll probably shouldn't be used as evidence to shift how similarly we model others to ourselves, one way or another.
In response to
Experience of typical mind fallacy.
That was awesome - thank you for posting the poll! The results are quire intriguing (at N = 18, anyway - might change with more votes, I guess).
In response to
Self-verification
Your best bet would be to find some sort of channel for communicating with your future self that your adversary does not have access to. Other posters mentioned several such examples, with channels including: * keeping your long term memories (assuming that the memories couldn't be tampered with by the adversary) * Swallowing a message, getting it as a tattoo, etc. (assuming that the adversary can't force you to do that) * Using some sort of biometric lock (assuming that the adversary can't get a proper sample with causing detectable alternations to your blood chemistry which would be detectable in the sample) My personal addition: tell your friends/neighbors/the news the story. Unless your adversary can make you lie (or take your form or use mind magic or whatnot), these people can act as the channel you need.
If you don't have a channel of that sort, I believe you are out of luck. A formal proof eludes me at this time; I'll post again if I figure out out.
In response to
Could you tell me what's wrong with this?
You keep making a rookie mistake: trying to invent solutions without learning the subject matter first. Consider this: people just as smart as you (and me) have put in 100 times more effort trying to solve this issue professionally. What are the odds that you have found a solution they missed after gaining only a cursory familiarity with the topic?
If you still think that you can meaningfully contribute to the FAI research without learning the basics, note that smarter people have tried and failed. Those truly interested in making their contribution went on to learn the state of the art, the open problems and the common pitfalls.
If you want to contribute, start by studying (not just reading) the relevant papers on the MIRI web site and their summaries posted by So8res in Main earlier this year. And for Omega's sake, go read Bostrom's Superintelligence.
I think you are being a little too exacting here. True, most advances in well-studied fields are likely to be made by experts. That doesn't mean that non-experts should be barred from discussing the issue, for educational and entertainment purposes if nothing else.
That is not to say that there isn't a minimum level of subject-matter literacy required for an acceptable post, especially when the poster in question posts frequently. I imagine your point may be that Algon has not cleared that threshold (or is close to the line) - but your post seems to imply a MUCH higher threshold for posting.
In response to
Could you tell me what's wrong with this?
I'm not convinced that the solution you propose is in fact easier than solving FAI. The following problems occur to me:
1) How to we explain to the creator AI what an FAI is? 2) How do we allow the creator AI to learn "properly" without letting it self modify in ways that we would find objectionable? 3) In the case of an unfriendly creator AI, how do we stop it from "sabotaging" its work in a way that would make the resulting "FAI" be friendly to the creator AI and not to us?
In general, I feel like the approach you outline just passes the issue up one level, requiring us to make FAI friendly.
On the other hand, if you limit your idea to something somewhat less ambitious, e.g. how to we make a safe AI to solve [difficult mathematical problem in useful for FAI], then I think you may be right.
What if we assume a finite universe instead? Contrary to what the post we're discussing might suggest, this actually makes recurrence more reasonable. To show that every state of a finite universe recurs infinitely often, we only need to know one thing: that every state of the universe can be eventually reached from every other state.
Is this plausible? I'm not sure. The first objection that comes to mind is entropy: if entropy always increases, then we can never get back to where we started. But I seem to recall a claim that entropy is a statistical law: it's not that it cannot decrease, but that it is extremely unlikely to do so. Extremely low probabilities do not frighten us here: if the universe is finite, then all such probabilities can be lower-bounded by some extremely tiny constant, which will eventually be defeated by infinite time.
But if the universe is infinite, this does not work: not even if the universe is merely potentially infinite, by which I mean that it can grow to an arbitrarily large finite size. This is already enough for the Markov chain in question to have infinitely many states, and my intuition tells me that in such a case it is almost certainly transient.
You are absolutely correct. If the number of states of the universe is finite, then as long as any state is reachable from any other state, then every state will be reached arbitrarily often if you wait long enough.
In response to
On immortality
Mathematician here. I wanted to agree with @pianoforte611 - just because you have infinite time doesn't mean that every event will repeat over and over.
For those interested in some reading, the general question is basically the question of Transience in Markov Chains; I also have some examples. :)
Let us say that we have a particle moving along a line. In each unit of time, it moves a unit of distance either left or right, with probability 1/10 of the former and 9/10 of the latter. How often can we expect the particle to have returned to its starting point? Well, to return to the origin, we must have moved left and right an equal number of times. At odd times, this is impossible; at time 2n, the probability of this is (this is not difficult to derive, and a simple explanation is given here). Summing this over all n, we get that the expected number of returns is one in four - in other words, we have no guarantee of returning even once, much less an infinite number times!
If this example strikes you as somewhat asymmetric, worry not - if the point was moving in three dimensions instead of one (so it could up, down, forward, or back as well as left or right), then a weighing of 1/6 to each direction means that you won't return to the starting point infinitely often. If you don't like having a fixed origin, use two particles, and have them moving independently in 3 dimensions. They will meet after time zero with less-than-unit-probability (actually, the same probability as in the previous problem, since the problems are equivalent after you apply a transformation).
I hope this helps!
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
There is a not necessarily large, but definitely significant chance that developing machine intelligence compatible with human values may very well be the single most important thing that humans have or will ever do, and it seems very likely that economic forces will make strong machine intelligence happen soon, even if we're not ready for it.
So I have two questions about this: firstly, and this is probably my youthful inexperience talking (a big part of why I'm posting this here), but I see so many rationalists do so much awesome work on things like social justice, social work, medicine, and all kinds of poverty-focused effective altruism, but how can it be that the ultimate fate of humanity to either thrive beyond imagination or perish utterly may rest on our actions in this century, and yet people who recognize this possibility don't do everything they can to make it go the way we need it to? This sort of segues in to my second question, which is what is the most any person, more specifically, I can do for FAI? I'm still in high school, so there really isn't that much keeping me from devoting my life to helping the cause of making sure AI is friendly. What would that look like? I'm a village idiot by LW standards, and especially bad at math, so I don't think I'd be very useful on the "front lines" so to speak, but perhaps I could try to make a lot of money and do FAI-focused EA? I might be more socially oriented/socially capable than many here, perhaps I could try to raise awareness or lobby for legislation?
To address your first question: this has to do with scope insensitivity, hyperbolic discounting, and other related biases. To put it bluntly, most humans are actually pretty bad at maximizing expected utility. For example, when I first head about x-risk, my thought process was definitely not "humanity might be wiped out - that's IMPORTANT. I need to devote energy to this." It was more along the lines of "huh; That's interesting. Tragic, even. Oh well; moving on..."
Basically, we don't care much about what happens in the distant future, especially if it isn't guaranteed to happen. We also don't care much more about humanity than we do about ourselves plus our close ones. Plus we don't really care about things that don't feel immediate. And so on. Then end result is that most people's immediate problems are more important to them then x-risk, even if the latter might be by far the more essential according to utilitarian ethics.