= 27d567bc2d48d9f40e9ccc20ea370b15)
timtyler comments on Advice for AI makers - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (196)
Museums have some paperclips in them. You have to imagine future museums as dynamic things that recreate and help to visualise the past - as well as preserving artefacts.
If you were an intelligence only cared about the number of paperclips in the universe, you would not build a museum to the past, because you could make more paperclips with the resources needed to create such a museum.
This is not some clever, convoluted argument. This is the same as saying that if you make your computer execute
10: GOTO 20
20: GOTO 10
then it won't at any point realize the program is "stupid" and stop looping. You could even give the computer another program which is capable of proving that the first one is an infinite loop, but it won't care, because its goal is to execute the first program.
That's a different question - and one which is poorly specified:
If insufficient look-ahead is used, such an agent won't bother to remember its history - prefering instead the gratification of instant paperclips.
On the other hand, if you set the look-ahead further out, it will. That's because most intelligent agents are motivated to remember the past - since only by remembering the past can they predict the future.
Understanding the history of their own evolution may well help them to understand the possible forms of aliens - which might well help them avoid being obliterated by alien races (along with all the paper clips they have made so far). Important stuff - and well worth building a few museums over.
Remebering the past is thus actually an proximate goal for a wide range of agents. If you want to argue paperclip-loving agents won't build museums, you need to be much more specific about which paperclip-loving agents you are talking about - because some of them will.
Once you understand this you should be able to see what nonsense the "value is fragile" post is.
At this point, I'm only saying this to ensure you don't take any new LWers with you in your perennial folly, but your post has anthropomorphic optimism written all over it.
This has nothing to do with anthropomorphism or optimism - it is a common drive for intelligent agents to make records of their pasts - so that they can predict the consequences of their actions in the future.
Once information is lost, it is gone for good. If information might be valuable in the future, a wide range of agents will want to preserve it - to help them attain their future goals. These points do not seem particularly complicated.
I hope at least that you now realise that your "loop" analogy was wrong. You can't just argue that paperclipping agents will not have preserving the past in museums as a proximate goal - since their ultimate goal involves making paperclips. There is a clear mechanism by which preserving their past in museums might help them attain that goal in the long term.
A wide class of paperclipping agents who are not suffering from temporal myopia should attempt to conquer the universe before wasting precious time and resources with making any paperclips. Once the universe is securely in their hands - then they can get on with making paperclips. Otherwise they run a considerable risk of aliens - who have not been so distracted with useless trivia - eating them, and their paperclips. They will realise that they are in an alien race - and so they will run.
Did you make some huge transgression that I missed that is causing people to get together and downvote your comments?
Edit: My question has now been answered.
I haven't downvoted, but I assume it's because he's conflating 'sees the value in storing some kinds of information' with 'will build museums'. Museums don't seem to be particularly efficient forms of data-storage, to me.
Future "museums" may not look exactly like current ones - and sure - some information will be preserved in "libraries" - which may not look exactly like current ones either - and in other ways.
'Museum' and 'library' both imply, to me at least, that the data is being made available to people who might be interested in it. In the case of a paperclipper, that seems rather unlikely - why would it keep us around, instead of turning the planet into an uninhabitable supercomputer that can more quickly consider complex paperclip-maximization strategies? The information about what we were like might still exist, but probably in the form of the paperclipper's 'personal memory' - and more likely than not, it'd be tagged as 'exploitable weaknesses of squishy things' rather than 'good patterns to reproduce', which isn't very useful to us, to say the least.
I see. We have different connotations of the word, then. For me, a museum is just a place where objects of historical interest are stored.
When I talked about humans being "preserved mostly in history books and museums" - I was intending to conjour up an institution somewhat like the Jurassic park theme park. Or perhaps - looking further out - something like The Matrix. Not quite like the museum of natural history as it is today - but more like what it will turn into.
Regarding the utility of existence in a museum - it may be quite a bit better than not existing at all.
Regarding the reason for keeping objects of historical around - that is for much the same reason as we do today - to learn from them, and to preserve them for future generations to study. They may have better tools for analysing things with in the future. If the objects of study are destroyed, future tools will not be able to access them.
Not really, just lots of little ones involving the misuse of almost valid ideas. They get distracting.
You got voted down because you were rational. You went over some peoples heads.
These are popularity points, not rationality points.
That is something we worry about from time to time, but in this case I think the downvotes are justified. Tim Tyler has been repeating a particular form of techno-optimism for quite a while, which is fine; it's good to have contrarians around.
However, in the current thread, I don't think he's taking the critique seriously enough. It's been pointed out that he's essentially searching for reasons that even a Paperclipper would preserve everything of value to us, rather than just putting himself in Clippy's place and really asking for the most efficient way to maximize paperclips. (In particular, preserving the fine details of a civilization, let alone actual minds from it, is really too wasteful if your goal is to be prepared for a wide array of possible alien species.)
I feel (and apparently, so do others) that he's just replying with more arguments of the same kind as the ones we generally criticize, rather than finding other types of arguments or providing a case why anthropomorphic optimism doesn't apply here.
In any case, thanks for the laugh line:
My analysis of Tim Tyler in this thread isn't very positive, but his replies seem quite clear to me; I'm frustrated on the meta-level rather than the object-level.
The real dichotomy here is "maximising evaluation function" versus "maximising probability of positive evaluation function"
In paperclip making, or better, the game of Othello/Reversi, there are choices like this:
80% chance of winning 60-0, versus 90% chance of winning 33-31.
The first maximises the winning, and is similar to a paperclip maker consuming the entire universe. The second maximises the probability of succeeding, and is similar to a paperclip maker avoiding being annihilated by aliens or other unknown forces.
Mathematically, the first is similar to finding the shortest program in Kolmogorov Complexity, while the second is similar to integrating over programs.
So, friendly AI is surely of the second kind, while insane AI is of the first kind.
I don't think that a paperclip maximiser would "preserve everything of value to us" in the first place. What I actually said at the beginning was:
Not everything. Things are constantly being lost.
What I said here was:
We do, in fact, have detailed information about how much our own civilisation is prepared to spend on preserving its own history. We preserve many things which are millions of years old - and which take up far more resources than a human. For example, see how this museum dinosaur dwarfs the humans in the foreground. We have many such exhibits - and we are still a planet-bound civilisation. Our descendants seem likely to have access to much greater resources - and so may devote a larger quantity of absolute resources to museums.
So: that's the basis of my estimate. What is the basis of your estimate?
I agree with your criticism, but I doubt that good will come of replying to a comment like the one you're replying to here, I'm afraid.
That's pretty vague. Care to point to something specific?
A pathetic example, IMHO. Those were perfectly reasonable comments attempting to dispel a poster's inaccurate beliefs about the phenomenon in question.
Your use of "get together" brings to mind some sort of Less Wrong cabal who gathered to make a decision. This is of course the opposite of the truth, which is that each downvote is the result of someone reading the thread and deciding to downvote the comment. They're not necessarily uncorrelated, but "get together" is completely the wrong way to think about how these downvotes occur.
Actually, that's what I was meaning to evoke. I read his recent comments, and while I didn't agree with all of them, didn't find them to be in bad faith. I found it odd that so many of them would be at -3, and wondered if I missed something.
In seriousness, why would you deliberately evoke a hypothesis that you know is wildly unrealistic? Surely whatever the real reasons for the downvoting pattern are, they are relevant to your enquiry?
Perhaps "cabal who gathered to make a decision [to downvote]" is an overly ominous image.
However, we've seen cases where every one of someone's comments has been downvoted in a short span of time, which is clearly not the typical reason for a downvoting.
That's the kind of thing I was asking about.