= 27d567bc2d48d9f40e9ccc20ea370b15)
Will_Newsome comments on The curse of identity - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (296)
Meh. The goal of leading to sentient beings living, to people being happy, to individuals having the freedom to control their own lives, to minds exploring new territory instead of falling into infinite loops, to the universe having a richness and complexity to it that goes beyond pebble heaps, etc. has probably much more Kolmogorov complexity than the goal of maximizing the number of paperclips in the universe. If preferring the former is irrational, I am irrational and proud of it.
Oh, also "look at the optimization targets of the processes that created the process that is me" is a short program, much shorter than needed to specify paperclip maximization, though it's somewhat tricky because all that is modulo the symbol grounding problem. And that's only half a meta level up, you can make it more elegant (shorter) than that.
The shorter your encoded message, the longer the encryption / compression algorithm, until eventually the algorithm is the full raw unencoded message and the encoded message is a single null-valued signal that, when received, decodes into the full message as it is contained within the algorithm.
...isn't nearly as short or simple as it sounds. This becomes obvious once you try to replace those words with their associated meaning.
My point was that it's easier to program ("simpler") than "maximize paperclips", not that it's as simple as it sounds. (Nothing is as simple as it sounds, duh.)
I fail to see how coding a meta-algorithm to select optimal extrapolation and/or simulation algorithm in order for those chosen algorithms to determine the probable optimization target (which is even harder if you want a full PA proof) is even remotely in the same order of complexity as a machine learner that uses natural selection for algorithms that increase paperclip-count, which is one of the simplest paperclip maximizers I can think of.
It might not be possible to make such a machine learner into an AGI, which is what I had in mind—narrow AIs only have "goals" and "values" and so forth in an analogical sense. Cf. derived intentionality. If it is that easy to create such an AGI, then I think I'm wrong, e.g. maybe I'm thinking about the symbol grounding problem incorrectly. I still think that in the limit of intelligence/rationality, though, specifying goals like "maximize paperclips" becomes impossible, and this wouldn't be falsified if a zealous paperclip company were able to engineer a superintelligent paperclip maximizer that actually maximized paperclips in some plausibly commonsense fashion. In fact I can't actually think of a way to falsify my theory in practice—I guess you'd have to somehow physically show that the axioms of algorithmic information theory and maybe updateless-like decision theories are egregiously incoherent... or something.
(Also your meta-algorithm isn't quite what I had in mind—what I had in mind is a lot more theoretically elegant and doesn't involve weird vague things like "extrapolation"—but I don't think that's the primary source of our disagreement.)