= 27d567bc2d48d9f40e9ccc20ea370b15)
If Omega gets it right more than 99% of the time, then why would Alpha take 10-to-1 odds against Omega messing up?
It seems that both Taleb and Aaronde are talking about a much smaller scale change than things like space colonization and general nanotech.
Yeah, that was my impression. One of the things that's interesting about the article is that many of the technologies Taleb disparages already exist. He lists space colonies and flying motorcycles right along side mundane tennis shoes and video chat. So it's hard to tell when he's criticizing futurists for expecting certain new technologies, and when he's criticizing them for wanting those new technologies. When he says that he's going to take a cab driven by an immigrant, is he saying that robot cars won't arrive any time soon? Or that it wouldn't make a difference if they did? Or that it would be bad if they did? I think his point is a bit muddled.
One thing he gets right is that cool new technologies need not be revolutionary. Don't get me wrong; I take the possibility of truly transformative tech seriously, but futurists do overestimate technology for a simple reason. When imagining what life will be like with a given gadget, you focus on those parts of your life when you could use the gadget, and thus overestimate the positive effect of the gadget (This is also why people's kitchens get cluttered over time). For myself, I think that robot cars will be commonplace in ten years, and that will be friggin' awesome. But it won't transform our lives - it will be an incremental change. The flip side is that Taleb may underestimate the cumulative effect of many incremental changes.
In response to
Open Thread, December 1-15, 2012
Nassim Nicholas Taleb argues: The future will not be cool
http://www.salon.com/2012/12/01/nassim_nicholas_taleb_the_future_will_not_be_cool/
Taleb's characterization of "technothinkers" as cultural ignoramuses doesn't sound quite right to me, because they tend to read and assimilate the writings of learned (in the liberal arts sense) fantasy and science fiction writers. In this way they at least get some exposure to humane culture once removed, if they don't immerse themselves in it directly. J.R.R. Tolkien taught Anglo-Saxon language and literature at Oxford, for example.
And many of my cryonicist friends have studies of history, literature and other cultures under their belts, or in some cases the experience of actually living in non-Western cultures. Thomas Donaldson spent some time living with indigenous people in New Guinea, for example. Despite what Taleb thinks, in my experience, a passion for the past can exist comfortably with a desire to build, and live in, a better future.
I think the real difference between people like Taleb and the techno-optimists is that we think the present is cool. He brags about going to dinner in minimalist shoes, and eating food cooked over a fire, whereas I think it's awesome that I can heat things up instantly in a microwave oven, and do just about anything in meticulously engineered and perfectly fitted, yet cheaply mass-produced, running shoes without worrying about damaging my feet. I also like keyboards, and access to the accumulated knowledge of humanity from anywhere, and contact lenses. And I thought it was funny when he said that condoms were one of the most important new technologies, but aren't talked about much, as if to imply that condoms aren't cool. I think that condoms are cool! I remember when I first got condoms, and took one out to play with. After testing it a couple different ways, I thought: *how does anyone manage to break one of these!?" It's easy to extrapolate that no "cool" technology will exist in the future, if you don't acknowledge that any cool technology currently exists.
But I think Taleb's piece is valuable, because it illustrates what we are up against, as people trying to get others to take seriously the risks, and opportunities, presented by future technologies. Taleb seems very serious and respectable, precisely because he is so curmudgeonly and conservative, whereas we seem excitable and silly. And he's right that singularitarian types tend to overemphasize changes relative to everything that remains the same, and often conflate their predictions of the future with their desires for the future. I think that lesswrong is better than most in this regard, with spokespeople for SI taking care to point out that their singularity hypothesis does not predict accelerating change, and that the consequences of disruptive technology need not be good. Still, I wonder if there's any way to present a more respectable face to the public, without pretending that we don't believe what we do.
I can't think of a way of fitting a forest fire into this model either, which suggests it isn't useful to think of forest fires under this paradigm.
Forest fires are definitely OPs under my intuitive concept. They consistently select a subset of possible future (burnt forests). They're probably something like chemical energy minimizers; if I were to measure their efficacy, it would be something like number of carbon-based molecules turned into CO2. But the only reason we can come up with semi-formal measures like CO2 molecules or output on wires is because we're smart human-things. I want to figure out how to algorithmically measure it.
Isn't the crux of the decision-making process pretending that you could choose any of your options, even though, as a matter of fact, you will choose one?
Yes. But what does "could" mean? It doesn't mean that you they all have equal probability. If literally all you know is that there are n outputs, then giving them 1/n weight is correct. But we usually know more, like the fact that it's an AI, and it's unclear how to update on this.
Are you saying that an "AI" outputting random noise could do worse than an "AI" with optimization power measured at zero (i.e. zero intelligence)?
Absolutely. Like how random outputs of a car cause it to jerk around and hit things, whereas a zero-capability car just sits there. Also, we're averaging over all possible outputs with equal weights. Even if most outputs are neutral or harmless, there are usually more damaging outputs than good ones. It's generally easier to harm than destroy. The more powerful actuators the AI has, the most damage random outputs will do.
Thanks for all your comments!
Oops, looks like I was wrong about what you meant (ignore the edit). But yes, if you give a stupid thing lots of power you should expect bad outcomes. A car directed with zero intelligence is not a car sitting still, but precisely what you said was dangerous: a car having its controls blindly fiddled with. But if you just run a stupid program on a computer, it will never acquire power in the first place. Most decisions are neutral, unless they just happen to be plugged into something that has already been optimized to have large physical effects (like a bulldozer). Of those decisions that do have large effects, most will be destructive, but that's exactly what we should expect from a stupid optimization process acting on something that has already been finely honed by a smart optimization process.
what does "could" mean?
Good question. I think it has something to do with simply defining some set of actions to be your "options", and temporarily putting all your options on an equal footing, so that you end up with the one with the best consequences, rather than the one that seemed like the one you'd be most likely to choose. I don't think it even has much to do with probabilities, because then you run into self-fulfilling prophesies - doing what you predicted you'd do, thereby justifying the prediction.
In this case, we want to measure how good an agent did, relative to how it could have done. That is, how good were the consequences of the option it chose, relative to its other options. I don't see any reason to weight those options according to a probability distribution, unless you know what "half an option" means. And choosing a distribution poses huge problems. After all, we know the agent chose one of the options with probability 1.0, and all the others with probability 0.0.
Forest fires are definitely OPs under my intuitive concept. They consistently select a subset of possible future (burnt forests).
Well, you could just compare the rate of oxidation under a flame, to the average rate of oxidation of all surfaces (including those that happen to be on fire) within whichever reference class you prefer. (I think choosing a reference class (set of options) is just part of how you define the OP. And you just define the OP whichever way helps you understand the world best.)
Thanks for all your comments!
Is this actually helpful? I try to read up on the background for this stuff, but I never know if I'm just rehashing what's already been discussed, and if so, whether reviewing that here would be useful to anyone.
We considered random output as a baseline. It doesn't seem correct, to me.
1) You'd need a way to even specify the set of "output" of any possible OP. This seems hard to me because many OPs do not have clear boundaries or enumerable output channels, like forest fires or natural selection or car factories.
2) This is equal to a flat prior over your OPs outputs. You need some kind of specification for what possibilities are equally likely, and a justification thereof.
3) Even if we consider an AGI with well-defined output channels, it seems to me that random outputs are potentially very very very destructive, and therefore not the "default" or "status quo" against which we should measure.
I think the idea should be explored more, though.
1) You'd need a way to even specify the set of "output" of any possible OP. This seems hard to me because many OPs do not have clear boundaries or enumerable output channels, like forest fires or natural selection or car factories.
How do you define an optimization process without defining its output? If you want to think of natural selection as a force that organizes matter into self-replicators, then compare the reproductive ability of an organism to the reproductive ability of a random clump of matter, to find out how much natural selection has acted on it. If you want to think of it as a force that produces genomes, then compare an evolved genome to a random strand of DNA (up to some maximum length).
I can't think of a way of fitting a forest fire into this model either, which suggests it isn't useful to think of forest fires under this paradigm. But isn't that a good sign? If anything could be usefully modeled as an optimizer, wouldn't that hint that the concept is overly broad?
2) This is equal to a flat prior over your OPs outputs. You need some kind of specification for what possibilities are equally likely, and a justification thereof.
Why? Isn't the crux of the decision-making process pretending that you could choose any of your options, even though, as a matter of fact, you will choose one? I can see how you would run into some fuzziness if you tried to apply it to natural selection or even brains. But for the mathematical model, where the process selects from some abstract set of options, equal weighting seems appropriate. And this maps fairly straightforwardly onto an AI acting over a physical wire.
3) Even if we consider an AGI with well-defined output channels, it seems to me that random outputs are potentially very very very destructive, and therefore not the "default" or "status quo" against which we should measure.
(EDIT: D'oh! I just realized what you meant by random outputs being "destructive". You mean that if an AGI were to take its options to be "configurations of matter in the universe", then its baseline would be a randomly shuffled universe that was almost completely destroyed from our perspective. But I don't think this makes sense. Just because an AGI is smart enough to reorganize all matter in the universe doesn't mean that it makes sense for it to output decisions in that form. That would basically be a type error, just like if I were to decide "be in New York" instead of "drive to New York". The options the AGI has to choose from are outputs of a subroutine running inside of itself. So if it has a robot body, then the "default" or unoptimized output is random flailing about, or if it interacts through a text terminal, it would be printing random gibberish, most of which does nothing and leaves the configuration of the universe largely unchanged (and a few of which convince the programmer to give it access to the internet so it can take over the world.)).
<how do I strikeout?>Are you saying that an "AI" outputting random noise could do worse than an "AI" with optimization power measured at zero (i.e. zero intelligence)? Seems to me that, to reliably do worse than random, you would have to be trying to do badly. And you would have to be doing so with a strictly positive level of skill.</how do I strikeout?>
(Note: for a model of natural selection that might actually be usable in practice, suppose that we know a set X of mutations have occurred in a population over a given time, and that a subset of these X* have become fixed in the population (the rest have been weeded out). To calculate how "optimized" X* is, compare the reproductive fitness of the actual population to the average fitness of hypothetical populations which, instead of X*, had retained some random subset of the mutations from X (that is, selected with uniform probability from the power set of X). The measure of "reproductive fitness" could be as simple as population size.)
And as long as P(U($10)>U($5)) is large enough a priori, it will swamp out the difference.
Well, then why even update? (Or, more specifically, why assume that this is harmless normally, but an ace up your sleeve for a particular class of problems? You need to be able to reliably distinguish when this helps you and when this hurts you from the inside, which seems difficult.)
Because I can't prove that I'll go through this line of reasoning. Simulating my decision process as part of my decision would result in infinite recursion.
I'm not sure that I understand this; I'm under the impression that many TDT applications require that they be able to simulate themselves (and other TDT reasoners) this way.
Good questions. I don't know the answers. But like you say, UDT especially is basically defined circularly - where the agent's decision is a function of itself. Making this coherent is still an unsolved problem. So I was wondering if we could get around some of the paradoxes by giving up on certainty.
So paralized people aren't less intelligent just because "raise arm" isn't an option for them (That is, no combination of action potentials in their head will cause their arm to move).
Caveat: if someone is paralyzed because of damage to their brain, rather than to their peripheral nerves or muscles, then this is not true, which creates and undesirable dependency of the measured optimization power on the location of the cause of the disability. Despite this drawback, I like this formalization.
Erm, you probably want to use something like (EU* - EU[av]) / EU[av], where EU* is just the actual expected utility, and EU[av] is the average of the expected utilities of the counterfactual probability distributions over world states associated with each of the agents options.
No, that clearly makes no sense if EU[av] <= 0. If you want to divide by something to normalize the measured optimization power (so that multiplying the utility function by a constant doesn't change the optimization power), the standard deviation of the expected utilities of the counterfactual probability distributions over world states associated with each of the agent's options would be a better choice.
Caveat: if someone is paralyzed because of damage to their brain, rather than to their peripheral nerves or muscles, then this is not true,
That's why I specified that the you don't get penalized for disabilities that have nothing to do with the signals leaving your brain.
which creates and undesirable dependency of the measured optimization power on the location of the cause of the disability.
I disagree. I think that's kind of the point of defining "optimization power" as distinct from "power". A man in a prison cell isn't less intelligent just because he has less freedom.
No, that clearly makes no sense if EU[av] <= 0. If you want to divide by something to normalize the measured optimization power (so that multiplying the utility function by a constant doesn't change the optimization power), the standard deviation of the expected utilities of the counterfactual probability distributions over world states associated with each of the agent's options would be a better choice.
Great idea! I was really sloppy about that, realized at the last minute that taking a ratio was clearly wrong, and just wanted to make sure that you couldn't get different answers by scaling the utility function. I guess |EU[av]| does that, but now we can get different answers by shifting the utility function, which shouldn't matter either. Standard deviation is infinitely better.
A causal reasoner will compute about P(utility=U| do{action=$5}), which doesn't run into this trouble. This is the approach I recommend.
Probabilistic reasoning about actions that you will make is, to the best of my knowledge, not a seriously considered approach to making decisions outside of the context of mixed strategies in game theory, and even there it doesn't apply that strong, as you can see mixed strategies as putting forth a certain (but parameterized) action whose outcome is subject to uncertainty.
I don't think your sketch is correct for two reasons:
- The assumption that your action is utility-maximizing requires that you choose the best action, and so using it to justify your choice of action leads to circularity.
- Your argument hinges on P(U($10)>U($5)|A=$10) > P(U($5)>U($10)|A=$5), which seems like an odd statement to me. If you take the actions maximize utility assumption seriously, both of those are 1, and thus the first can't be higher than the second. If you view the actions as not at all informative about the preference probabilities, then you're just repeating your prior. If the action gives some information, there's no reason for the information to be symmetric- you can easily construct a 2x2 matrix example where the reverse inequality holds (that if we know they picked $5, they are more likely to prefer $5 to $10 than someone who picked $10 is to prefer $10 to $5, even though most people prefer $10 to $5.
What I am saying is that I don't assume that I maximize expected utility. I take the five-and-ten problem as a proof that an agent cannot be certain that it will make the optimal choice, while it is choosing, because this leads to a contradiction. But this doesn't mean that I can't use the evidence that a choice would represent, while choosing. In this case, I can tell that U($10) > U($5) directly, so conditioning on A=$10 or A=$5 is redundant. The point is that it doesn't cause the algorithm to blow up, as long I don't think my probability of maximizing utility is 0 or 1.
It's true that A=$5 could be stronger evidence for U($5)>U($10) than A=$10 is for U($10)>U($5). But there's no particular reason to think it would be. And as long as P(U($10)>U($5)) is large enough a priori, it will swamp out the difference. As long as making a choice is evidence for that being the optimal choice, only insofar as I am confident that I make the optimal choice in general, it will provide equally strong evidence for every choice, and cancel itself out. But in cases where a particular choice is evidence of good things for other reasons (like Newcomb's problem), taking this evidence into consideration can affect my decision.
So why can't I just use the knowledge that I'll go through this line of reasoning to prove that I will choose $10 and yield a contradiction? Because I can't prove that I'll go through this line of reasoning. Simulating my decision process as part of my decision would result in infinite recursion. Now, there may be a shortcut I could use to prove what my choice will be, but the very fact that this would yield a contradiction means that no such proof exists in a consistent formal system.
(BTW, I agree that CDT is the only decision theory that works in practice, as is. I'm only addressing one issue with the various timeless decision theories)
In response to
Mathematical Measures of Optimization Power
What I can't figure out is how to specify possible worldstates “in the absence of an OP”.
Can we just replace the optimizer's output with random noise? For example, if we have an AI running in a black box, that only acts on the rest of the universe through a 1-gigabit network connection, then we can assign a uniform probability distribution over every signal that could be transmitted over the connection over a given time (all 2^(10^9) possibilities per second), and the probability distribution of futures that yields is our distribution over worlds that "could have been". We could do the same thing with a human brain and, say, all combinations of action potentials that could be sent down the spinal cord over a given time. This is desirable, because it separates optimization power from physical power. So paralyzed people aren't less intelligent just because "raise arm" isn't an option for them (That is, no combination of action potentials in their head will cause their arm to move).
More formally, an agent is a function or program that has a range or datatype. The range/datatype is the set of what we would call the agent's options. So assume we can generate counterfactual outcomes for each option in the range, the same way your favorite decision theory does. Then we can take optimization power to be the difference between EU given what the agent actually does, and the average EU over all the counterfactuals.*
If the OP is some kind of black-box AI agent, it's easier to imagine this. But if the OP is evolution, or a forest fire, it's harder to imagine.
I'm not so sure. Choosing to talk about natural selection as an agent means defining an agent which values self-replication and outputs a replicator. So if you have a way of measuring how good a genome is at replicating, you could just subtract from that how good a random sequence of base-pairs is, on average, at replicating, to get a measure of how much natural selection has optimized that genome. Of course, you could do the same thing with an entire animal versus a random clump of matter, because the range of the agent is just part of the definition.
EDIT: * AlexMennen had a much better idea for normalizing this than I did ;)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
From the article:
This seems implausible, because humans learn almost everything we know over our lifetimes, from a starting instruction set smaller than existing pieces of software. If architecture really is overrated (meaning that existing architectures are already good enough) isn't it more likely that AIs learn so slowly now, simply because the computers aren't powerful enough yet?