Here is my attempt at a calculation. Disclaimer: this is based on googling. If you are actually knowledgeable in the subject, please step in and set me right.
There are 10^11 neurons in the human brain.
A neuron will fire about 200 times per second.
It should take a constant number of flops to decide whether a neuron will fire -- say 10 flops (no need to solve a differential equation, neural networks usually use some discrete heuristics for something like this)
I want a society of 10^6 orcs running for 10^6 years
As you suggest, lets let the simulation ru...
Still inability to realise what you are doing seems rather dangerous.
So far, all I've done is post a question on lesswrong :)
More seriously, I do regret it if I appeared unaware of the potential danger. I am of course aware of the possibility that experiments with AI might destroy humanity. Think of my post above a suggesting a possible approach to investigate -- perhaps one with some kinks as written (that is why I'm asking a question here) but (I think) with the possibility of one day having rigorous safety guarantees.
I think this calculation too conservative. The reason is (as I understand it) that neurons are governed by various differential equations, and simulating them accurately is a pain in the ass. We should instead assume that deciding whether a neuron will fire will take a constant number of flops.
I'll write another comment which attempts to redo your calculation with different assumptions.
It seems to me that by the time we can do that, we should have figured out a better way to create AI.
But will we have figured a way to reap the gains of AI safely for humanity?
but more than that, it knows that the real universe is capable of producing intelligent beings that chose this particular world to simulate.
Good point -- this undermines a lot of what I wrote in my update 1. For example, I have no idea if F = m d^3 x / dt would result in a world that is capable of producing intelligent beings.
I should at some point produce a version of the above post with this claim, and other questionable parenthetical remarks I made, deleted, or at least acknowledging that they require further argumentation; they are not necessary fo...
I guess I am willing to bite the bullet and say that, as long as entity X prefers existence to nonexistence, you have done it no harm by bringing it into being. I realize this generates a number of repulsive-sounding conclusions, e.g., it becomes ethical to create entities which will live, by our 21st century standards, horrific lives.
At least some of them will tell you they had rather not been born.
If one is willing to accept my reasoning above, I think one can take one more leap and say that statistically as long as the vast majority of these entities will prefer existing to never having been brought into being, we are in the clear.
The theorem you cite (provided I understood you correctly) does not preclude the possibility of checking whether a program written in a certain pre-specified format will have bugs. Bugs here are defined to be certain undesirable properties (e.g., looping forever, entering certain enumerated states, etc).
Baby versions of such tools (which automatically check whether your program will have certain properties from inspecting the code) already exist.
Nice idea...I wrote an update to the post suggesting what seemed to me to be a variation on your suggestion.
About program checking: I agree completely. I'm not very informed about the state of the art, but it is very plausible that what we know right now is not yet up to task.
I'm a bit skeptical about whether you can actually create a superintelligent AI by combining sped up humans like that,
Why not? You are pretty smart, and all you are is a combination of 10^11 or so very "dumb" neurons. Now imagine a "being" which is actually a very large number of human-level intelligences, all interacting...
is very, very difficult not to give a superintelligence any hints of how the physics of our world work.
I wrote a short update to the post which tries to answer this point.
Maybe they notice minor fluctuations in the speed of the simulation based on environmental changes to the hardware
I believe they should have no ability whatsoever to detect fluctuations in the speed of the simulation.
Consider how the world of world of warcraft appears to an orc inside the game. Can it tell the speed at which the hardware is running the game?
It can't. What it can ...
These are good points. Perhaps I should not have said "interact" but chosen a different word instead. Still, its ability to play us is limited since (i) we will be examining the records of the world after it is dead (ii) it has no opportunity to learn anything about us.
Edit: we might even make it impossible for it to game us in the following way. All records of the simulated world are automatically deleted upon completion -- except for a specific prime factorization we want to know.
This is a really bad argument for safety.
You are right, of ...
When talking about dealing and (non)interacting with real AIs, one is always talking about a future world with significant technological advances relative to our world today.
If we can formulate something as a question about math, physics, chemistry, biology, then we can potentially attack it with this scheme. These are definitely problems we really want to solve.
Its true that if we allow AIs more knowledge and more access to our world, they could potentially help us more -- but of course the number of things that can go wrong has to increase as well. Perhaps a compromise which sacrifices some of the potential while decreasing the possibilities that can go wrong is better.
Sometimes you're dealing with a domain where explicit reasoning provides the best evidence, sometimes with a domain where emotions provide the best evidence.
And how should you (rationally) decide which kind of domain you are in?
Answer: using reason, not emotions.
Example: if you notice that your emotions have been a good guide in understanding what other people are thinking in the past, you should trust them in the future. The decision to do this, however, is an application of inductive reasoning.
No. CFAR rationality is about aligning system I and system II. It's not about declaring system I outputs to be worthy of being ignored in favor of system II outputs.
I believe you are nitpicking here.
If your reason tells you 1+1=2 but your emotions tell you that 1+1=3, being rational means going with your reason. If your reason tells you that ghosts do not exist, you should believe this to be the case even if you really, really want there to be evidence of an afterlife.
CFAR may teach you techniques to align your emotions and reason, but this does not ...
Sure, you can work towards feeling more strongly about something, but I don't believe you'll ever be able match the emotional fervor the partisans feel -- I mean here the people who stew in their anger and embrace their emotions without reservations.
As a (rather extreme) example, consider Hitler. He was able to sway a great many people with what were appeals to anger and emotion (though I acknowledge there is much more to the phenomena of Hitler than this). Hypothetically, if you were a politician from the same era, say a rational one, and you understood that the way to persuade people is to tap into the public's sense of anger, I'm not sure you'd be able to match him.
Do the extend that it does require luck that simply means that it's important to have more people with rationality + competence + caring. If you have many people some will get lucky.
The "little bit of luck" in my post above was something of an understatement; actually, I'd suggest it requires a lot of luck (among many other things) to successfully change the world.
I think you might be pattern matching to straw-vulcan rationality, that's distinct from what CFAR wants to teach.
Not sure if I am, but I believe I am making a correct claim abo...
A very interesting and thought provoking post -- I especially like the Q & A format.
I want to quibble with one bit:
...How can I tell there aren't enough people out there, instead of supposing that we haven't yet figured out how to find and recruit them?
Basically, because it seems to me that if people had really huge amounts of epistemic rationality + competence + caring, they would already be impacting these problems. Their huge amounts of epistemic rationality and competence would allow them to find a path to high impact; and their caring would compe
For those people who insist, however, that the only thing that is important is that the theory agrees with experiment, I would like to make an imaginary discussion between a Mayan astronomer and his student...
These are the opening words of a ~1.5 minute monologue in one of Feynman's lectures; I won't transcribe the remainder but it can be viewed here.
Not sure...I think confidence, sales skills, and ability to believe and get passionate about BS can be very helpful in much of the business world.
Side-stepping the issue of whether rationalists actually "win" or "do not win" in the real world, I think a-priori there are some reasons to suspect that people who exhibit a high degree or rationality will not be among the most successful.
For example: people respond positively to confidence. When you make a sales pitch for your company/research project/whatever, people like to see you that you really believe in the idea. Often, you will win brownie points if you believe in whatever you are trying to sell with nearly evangelical fervor...
I'm very fond of this bit by Robin Hanson:
A wide range of topics come up when talking informally with others, and people tend to like you to express opinions on at least some substantial subset of those topics. They typically aren’t very happy if you explain that you just adopted the opinion of some standard expert source without reflection, and so we are encouraged to “think for ourselves” to generate such opinions.
I think the lumping of various disciplines into "science" is unhelpful in this context. It is reasonable to trust the results of the last round of experiments at the LHC far more than the occasional psychology paper that makes the news.
I've not seen this distinction made as starkly as I think it really needs to be made -- there is a lot of difference between physics and chemistry, where one can usually design experiments to test hypotheses; to geology and atmospheric science, where one mostly fits models to data that happens to be available; to psychology, where the results of experiments seem to be very inconsistent and publication bias is a major cause of false research results.
I agree with 99.999% of what you say in this comment. In particular, you are right that the parody only works in the sense of the first of your bulleted points.
My only counterpoint is that I think this is how almost every reader will understand it. My whole post is an invitation to to consider a hypothetical in which people say about strength what they now say about intelligence and race.
I confess that I have not read much of what has been written on the subject, so what I am about to say may be dreadfully naive.
A. One should separate the concept of effective altruism from the mode-of-operation of the various organizations which currently take it as their motto.
A.i. Can anyone seriously oppose effective altruism in principle? I find it difficult to imagine someone supporting ineffective altruism. Surely, we should let our charity be guided by evidence, randomized experiments, hard thinking about tradeoffs, etc etc.
A.ii. On the other han...
I'm not sure I understand your criticism. I don't mean this in a passive aggressive sense, I really do not understand it. It seems to me that "the stupid," so to speak, perfectly carries over between the parody and the "original."
A. Imagine I visit country X, where everyone seems to be very buff. Gyms are everywhere, the parks are full of people practicing weight-lifting, and I notice people carrying heavy objects with little visible effort. When I return home, I remark to a friend that people in X seem to be very strong.
My friend ...
First, I'm not so sure: if someone is actually inconsistent, then pointing out the inconsistency may be the better (more charitable?) thing to do rather than pretending the person had made the closest consistent argument.
For example: there are a lot of academics who attack reason itself as fundamentally racist, imperialistic, etc. They back this up with something that looks like an argument. I think they are simply being inconsistent and contradictory, rather than meaning something deep not apparent at first glance.
More importantly, I think your conjectur...
For what its worth, I have not downvoted any of your posts. Although we seem to be on opposite sides of this debate, I appreciate the thoughtful disagreement my post has received..
I would bet the opposite on #4, but that is beside the point. On #4 and #6, the point is that even if everything I wrote was completely correct -- e.g., if the scientific journals were actually full of papers to the effect that there is no such thing as a universal test of strength because people from different cultures lift things differently -- it would not imply there is no such thing as strength.
On #5, the statement that race is a social construct is implicit. Anyway, as I said in the comment above, there are a million similar statements that are being...
First, only some of the attacks I cited were brief and sketchy; others were lengthier. Second, I have cited a few such attacks due to time and space constraints, but in fact they exist in great profusion. My personal impression is that the popular discourse on intelligence and race is drowning in confused rhetoric along the lines of what I parodied.
Finally, I think the last possibility you cite is on point -- there are many, many people who are not thinking very clearly here. As I said, I think these people also have come to dominate the debate on this su...
A. I think at least some people do mean that concepts of intelligence and race are, in some sense, inherently meaningless.
When people say
"race does not exist because it is a social construct"
or that race does not exist because
"amount of variation within races is much larger than the amount of variation between races,"
I think it is being overly charitable to read that as saying
"race is not a scientifically precise concept that denotes intrinsic, context-independent characteristics."
B. Along the same lines, I believe I am...
See the reply I just wrote to gjm for an explanation of my motivations.
When I was writing this, I thought the intent to parody would be clear; surely no one could seriously suggest we have to strike strength from our dictionaries? I seem to have been way off on that. Perhaps that is a reflection on the internet culture at large, where these kinds of arguments are common enough not to raise any eyebrows.
Anyway, I went one step further and put "parody" in the title.
I was not trying to suggest that intelligence and strength are as alike as race and strength. Rather, I was motivated by the observation that there are a number of arguments floating around to the effect that,
A. Race doesn't exist
B. Intelligence doesn't exist.
and, actually, to a lesser extent,
C. Rationality doesn't exist (as a coherent notion).
The arguments for A,B,C are often dubious and tend to overlap heavily; I wanted to write something which would show how flawed those arguments are through a reductio ad absurdum.
To put it another way, even if str...
Hmm, on second thought, I added a [/parody] tag at the end of my post - just in case...
For what its worth, I have observed a certain reverence in the way great mathematicians are treated by their lesser-accomplished colleagues that can often border on the creepy. This is something specific to math, in that it seems to exist in other disciplines with lesser intensity.
But I agree, "dysfunctional" seems to be a more apt label than "cult." May I also add "fashion-prone?"
The links you give are extremely interesting, but, unless I am missing something, it seems that they fall short of justifying your earlier statement that math academia functions as a cult. I wonder if you would be willing to elaborate further on that?
Right, that is what i meant.