Hypercomputation doesn't exist. There's no evidence for it - and nor will there ever be. It's an irrelevance that few care about. Solomonoff induction is right about this.
Also, competition between humans (with machines as tools) seems far more likely to kill people than a superintelligent runaway. However, it's (arguably) not so likely to kill everybody. MIRI appears to be focussing on the "killing everybody case". That is because - according to them - that is a really, really bad outcome.
The idea that losing 99% of humans would be acceptable losses may strike laymen as crazy. However, it might appeal to some of those in the top 1%. People like Peter Thiel, maybe.
Right. So, if we are playing the game of giving counter-intuitive technical meanings to ordinary English words, humans have thrived for millions of years - with their "UnFriendly" peers and their "UnFriendly" institutions. Evidently, "Friendliness" is not necessary for human flourishing.
"8 lives saved per dollar donated to the Machine Intelligence Research Institute. — Anna Salamon"
Nor does the fact that evolution 'failed' in its goals in all the people who voluntarily abstain from reproducing (and didn't, e.g., hugely benefit their siblings' reproductive chances in the process) imply that evolution is too weak and stupid to produce anything interesting or dangerous.
Failure is a necessary part of mapping out the area where success is possible.
Uploads first? It just seems silly to me.
The movie features a luddite group assassinating machine learning researchers - not a great meme to spread around IMHO :-(
Slightly interestingly, their actions backfire, and they accelerate what they seek to prevent.
Overall, I think I would have preferred Robopocalypse.
One other point I should make: this isn't just about "someone" being wrong. It's about an author frequently cited by people in the LessWrong community on an important issue being wrong.
Not experts on the topic of diet. I associated with members of the Calorie Restriction Society some time ago. Many of them were experts on diet. IIRC, Taubes was generally treated as a low-grade crackpot by those folk: barely better than Atkins.
To learn more about this, see "Scientific Induction in Probabilistic Mathematics", written up by Jeremy Hahn
This line:
Choose a random sentence from S, with the probability that O is chosen proportional to u(O) - 2^-length(O).
...looks like a subtraction operation to the reader. Perhaps use "i.e." instead.
The paper appears to be arguing against the applicability of the universal prior to mathematics.
However, why not just accept the universal prior - and then update on learning the laws of mathematics?
why did you bring up the 'society' topic in the first place?
A society leads to a structure with advantages of power and intelligence over individuals. It means that we'll always be able to restrain agents in test harnesses, for instance. It means that the designers will be smarter than the designed - via collective intelligence. If the the designers are smarter than the designed, maybe they'll be able to stop them from wireheading themselves.
...If wireheading is plausible, then it's equally plausible given an alien-fearing government, since wireheading
We can model induction in a monistic fashion pretty well - although at the moment the models are somewhat lacking in advanced inductive capacity/compression abilities. The models are good enough to be built and actually work.
Agents wireheading themselves or accidentally performing fatal experiments on themselves will probably be handled in much the same way that biology has handled it to date - e.g. by liberally sprinkling aversive sensors around the creature's brain. The argument that such approaches do not scale up is probably wrong - designers will al...
Naturalized induction is an open problem in Friendly Artificial Intelligence. The problem, in brief: Our current leading models of induction do not allow reasoners to treat their own computations as processes in the world.
I checked. These models of induction apparently allow reasoners to treat their own computations as modifiable processes:
This was discussed on Facebook. I'll copy-paste the entire conversation here, since it can only be viewed by people who have Facebook accounts.
Kaj Sotala: Re: Cartesian reasoning, it sounds like Orseau & Ring's work on creating a version of the AIXI formalism in which the agent is actually embedded in the world, instead of being separated from it, would be relevant. Is there any particular reason why it hasn't been mentioned?
Tsvi BT: [...] did you look at the paper Kaj posted above?
Luke Muehlhauser: [...] can you state this open problem using the notat...
Deutsch is interesting. He seems very close to the LW camp, and I think he's someone LWers should at least be familiar with.
Deutsch seems pretty clueless in the section quoted below. I don't see why students should be interested in what he has to say on this topic.
...It was a failure to recognise that what distinguishes human brains from all other physical systems is qualitatively different from all other functionalities, and cannot be specified in the way that all other attributes of computer programs can be. It cannot be programmed by any of the techn
There never was a bloggingheads - AFAIK. There is: Yudkowsky vs Hanson on the Intelligence Explosion - Jane Street Debate. However, I'd be surprised if Yudkowsky makes the same silly mistake as Deutsch. Yudkowsky knows some things about machine intelligence.
But in reality, only a tiny component of thinking is about prediction at all, let alone prediction of our sensory experiences.
My estimate is 80% prediction, with the rest evaluation and tree pruning.
He also says confusing things about induction being inadequate for creativity which I'm guessing he couldn't support well in this short essay (perhaps he explains better in his books).
He does - but it isn't pretty.
Here is my review of The Beginning of Infinity: Explanations That Transform the World.
I remember Eliezer making the same point in a bloggingheads video with Robin Hanson.
A Hanson/Yudkowsky bloggingheads?!? Methinks you are mistaken.
So:
To give an example of a survivalist, here's an individual who proposes that we should be highly prioritizing species-level survival:
As you say, this is not a typical human being - since Nick says he is highly concerned about others.
There are many other survivalists out there, many of whom are much more concerned with personal survival.
If you're dealing with creatures good enough at modeling the world to predict the future and transfer skills, then you're dealing with memetic factors as well as genetic. That's rather beyond the scope of natural selection as typically defined.
What?!? Natural selection applies to both genes and memes.
I suppose there are theoretical situations where that argument wouldn't apply
I don't think you presented a supporting argument. You referenced "typical" definitions of natural selection. I don't know of any definitions that exclude culture. H...
The question's more about what function's generating the fitness landscape you're looking at (using "fitness" now in the sense of "fitness function"). "Survival" isn't a bad way to characterize that fitness function -- more than adequate for eighth-grade science, for example. But it's a short-tern and highly specialized kind of survival [...]
Evolution is only as short-sighted as the creatures that compose its populations. If organisms can do better by predicting the future (and sometimes they can) then the whole process is...
Even to the extent that natural selection can be said to be care about anything, saying that survival is that thing is kind of misleading.
Well, I have gone into more details elsewhere.
It's perfectly normal for populations to hill-climb themselves into a local optimum and then get wiped out when it's invalidated by changing environmental conditions that a more basal but less specialized species would have been able to handle, for example.
Sure. Optimization involves going uphill - but you might be climbing a mountain that is sinking into the sea. How...
I'm pretty sure that we suck at prediction - compared to evaluation and tree-pruining. Prediction is where our machines need to improve the most.
search is not the same problem as prediction
It is when what you are predicting is the results of a search. Prediction covers searching.
It is interesting that his view of AI is apparently that of a prediction tool [...] rather than of a world optimizer.
If you can predict well enough, you can pass the Turing test - with a little training data.
If we're talking reference classes, I would cite the example that the first hominid species to develop human-level intelligence took over the world.
Note that humans haven't "taken over the world" in many senses of the phrase. We are massively outnumbered and out-massed by our own symbionts - and by other creatures.
Machine intelligence probably won't be a "secret" technology for long - due to the economic pressure to embed it.
While its true that things will go faster in the future, that applies about equally to all players - in a phenomenon commonly known as "internet time".
Doesn't someone have to hit the ball back for it to be "tennis"? If anyone does so, we can then compare reference classes - and see who has the better set. Are you suggesting this sort of thing is not productive? On what grounds?
As has been pointed out numerious times on lesswrong, history is not a very good guide for dealing with AI since it is likely to be a singular (if you'll excuse the pun) event in history. Perhaps the only other thing it can be compared with is life itself [...]
What, a new thinking technology? You can't be serious.
whoever builds the first AI can take over the world, which makes building AI the ultimate arms race.
As the Wikipedians often say, "citation needed". The first "AI" was built decades ago. It evidently failed to "take over the world". Possibly someday a machine will take over the world - but it may not be the first one built.
I didn't buy the alleged advantage of a noise free environment. We've known since von-Neumann's paper titled:
PROBABILISTIC LOGICS AND THE SYNTHESIS OF RELIABLE ORGANISMS FROM UNRELIABLE COMPONENTS
...that you can use unreliable computing components to perform reliable computation - with whatever level of precision and reliability that you like.
Plus the costs of attaining global synchrony and determinism are large and massively limit the performance of modern CPU cores. Parallel systems are the only way to attain large computing capacities - and you can'...
The point I was trying to make was more along the lines that choosing which parameters to model allows you to control the outcome you get. Those who want to recruit people to causes associated with preventing the coming robot apocalypse can selectively include competitive factors, and ignore factors leading to cooperation - in order to obtain their desired outcome.
Today, machines are instrumental in killing lots of people, but many of them also have features like air bags and bumpers, which show that the manufacturers and their customers are interested in...
People have predicted that corporations will be amoral, ruthless psychopaths too. This is what you get when you leave things like reputations out of your models.
Skimping on safety features can save you money. However, a reputation for privacy breaches, security problems and accidents doesn't do you much good. Why model the first effect while ignoring the second one? Oh yes: the axe that needs grinding.
There are techniques for managing reputation, and those techniques are also amoral. For example, a powerful psychopath caring about his reputation may use legal threats and/or assassination against people who want to report about his evil acts. Alternatively, he may spread false rumors about his competitors. He may pay or manipulate people to create a positive image of him.
Just because the reputation is used, it does not guarantee the results will be moral.
Reputational concerns apply to psychopaths too, and that's why not all of them turn violent. However it doesn't prevent all of them from turning violent.
Bayesian methods certainly require relative parsimony, in the sense that the model complexity needs to be small compared to the quantity of information being modeled.
Not really. Bayesian methods can model random noise. Then the model is of the same size as the data being modeled.
I often use simple models–because they are less effort to fit and, especially, to understand. But I don’t kid myself that they’re better than more complicated efforts!
Recommended reading: Boyd and Richerson's Simple Models of Complex Phenomena.
The reason the Solomonoff prior doesn't apply to social sciences is because knowing the area of applicability gives you more information.
That doesn't mean it doesn't apply! "Knowing the area of applicability" is just some information you can update on after starting with a prior.
Losing information isn't a crime. The virtues of simple models go beyond Occam's razor. Often, replacing a complex world with a complex model barely counts as progress - since complex models are hard to use and hard to understand.
Parsimony is good except when it loses information, but if you're losing information you're not being parsimonious correctly.
So: Hamilton's rule is not being parsimonious "correctly"?
Shane Legg prepared this graph.
It was enough to convince him that there was some super-exponential synergy:
There's also a broader point to be made about why evolution would've built humans to be able to benefit from better software in the first place, that involves the cognitive niche hypothesis.
I think we understand why humans are built like that. Slow-reproducing organisms often use rapidly-reproducing symbiotes to help them adapt to local environments. Humans using cultural symbionts to adapt to local regions of space-time is a special case of this general principle.
Instead of the cognitive niche, the cultural niche seems more relevant to humans.
On the other hand, I think the evolutionary heuristic casts doubt on the value of many other proposals for improving rationality. Many such proposals seem like things that, if they worked, humans could have evolved to do already. So why haven't we?
Most such things would have had to evolve by cultural evolution. Organic evolution makes our hardware, cultural evolution makes our software. Rationality is mostly software - evolution can't program such things in at the hardware level very easily.
Cultural evolution has only just got started. Education is sti...
I usually try to avoid the term "moral realism" - due to associated ambiguities - and abuse of the term "realism".
The thesis says:
more or less any level of intelligence could in principle be combined with more or less any final goal.
The "in principle" still allows for the possibility of a naturalistic view of morality grounding moral truths. For example, we could have the concept of: the morality that advanced evolutionary systems tend to converge on - despite the orthogonality thesis.
It doesn't say what is likely to happen. It says what might happen in principle. It's a big difference.
We're just saying that AGI is an incredibly powerful weapon, and FAI is incredibly difficult. As for "baseless", well... we've spent hundreds of pages arguing this view, and an even better 400-page summary of the arguments is forthcoming in Bostrom's Superintelligence book.
It's not mudslinging, it's Leo Szilard pointing out that nuclear chain reactions have huge destructive potential even if they could also be useful for power plants.
Machine intelligence is important. Who gets to build it using what methodology is also likely to have a signif...
It is true that there might not be all that much insight needed to get to AGI on top of the insight needed to build a chimpanzee. The problem that Deutsch is neglecting is that we have no idea about how to build a chimpanzee.
It isn't a testable hypothesis. Why would anyone attempt to assign probabilities to it?