The issue arises specifically in the situation of recursive self-improvement: You can't prove self-consistency in mathematical frameworks of "sufficient complexity" (that is, containing the rules of arithmetic in a provable manner).
What this cashes out to is that, considering AI as a mathematical framework, and the next generation of AI (designed by the first) as a secondary mathematical framework - you can't actually prove that there are no contradictions in an umbrella mathematical framework that comprises both of them, if they are of "sufficient complex...
Alternatively - we communicate about the things that pose the most danger to us, in a manner intended to minimize that danger.
In a typical Level-4 society, people don't have a lot to fear from lions and they aren't in imminent danger of starvation. The bottom half of Maslow's hierarchy is pretty stable.
It's the social stuff where our needs run the risk of being unfulfilled; it is the social stuff that poses the most danger. So of course most of the communication that takes place is about social stuff, in manners intended to reinforce our own social status. This isn't a simulacra of reality - it is reality, and people suffer real harms for being insufficient to the task.
I think a substantial part of the issue here is the asymmetry created when one party is public, and one party is not.
Suppose a user is posting under their real name, John Doe, and another user is posted under a pseudonym, Azure_Pearls_172. An accusation by Azure against John can have real-world implications; an accusation by John against Azure is limited by the reach of the pseudonym. Azure can change their pseudonym, and leave the accusations behind; John cannot.
Doxxing can make a situation more symmetrical in this case. Whether or not i...
I am, unapologetically, a genius. (A lot of people here are.)
My experience of what it is like being a genius: I look at a problem and I know an answer. That's pretty much it. I'm not any faster at thinking than anybody else; I'd say I'm actually a somewhat slower thinker, but make up for it by having "larger" thoughts; most people seem to have fast multi-core processors, and I'm running a slightly slow graphics card. Depending on what you need done, I'm either many orders of magnitude better at it - or completely hopeless. It ...
Take a step back and try rereading what I wrote in a charitable light, because it appears you have completely misconstrued what I was saying.
A major part of the "cooperation" involved here is in being able to cooperate with yourself. In an environment with a well-mixed group of bots each employing differing strategies, and some kind of reproductive rule (if you have 100 utility, say, spawn a copy of yourself), Cooperate-bots are unlikely to be terribly prolific; they lose out against many other bots.
In such an environment, a strategem of defecting ag...
Evolution gave us "empathy for the other person", and evolution is a reasonable proxy for a perfectly selfish utility machine, which is probably good evidence that this might be an optimal solution to the game theory problem. (Note: Not -the- optimal solution, but -an- optimal solution, in an ecosystem of optimal solutions.)
Note that it is possible to deceive others by systematically adjusting predictions upward or downward to reflect how desirable it is that other people believe those predictions, in a way which preserves your score.
This is true even if you bucket your scores; say you're evaluating somebody's predictive scores. You see that when they assign a 60% probability to an event, that event occurs 60% of the time. This doesn't mean that any -specific- prediction they make of 60% probability will occur 60% of the time, however! They can balance out t...
...How does one correctly handle multi-agent dilemmas, in which you know the other agents follow the same decision theory? My implementation of "UDT" defects in a prisoner's dilemma against an agent that it knows is following the same decision procedure. More precisely: Alice and Bob follow the same decision procedure, and they both know it. Alice will choose between cooperate/defect, then Bob will choose between cooperate/defect without knowing what Alice picked, then the utility will be delivered. My "UDT" decision procedure reasons as follows for Alice: "i
The point there is that there is no contradiction because the informational content is different. "Which is the baseline" is up to the person writing the problem to answer. You've asserted that the baseline is A vs B; then you've added information that A is actually A1 and A2.
The issue here is entirely semantic ambiguity.
Observe what happens when we remove the semantic ambiguity:
You've been observing a looping computer program for a while, and have determined that it shows three videos. The first video portrays a coin showing tails. ...
If you have two options, A and B, 50% odds is maximal ignorance; you aren't saying they have equivalent odds of being true, you're saying you have no information by which to make an inference which is true.
If you then say we can split A into A1 and A2, you have added information to the problem. Like the Monty Hall problem, information can change the odds in unexpected ways!
There's no contradiction here - you have more information than when you originally assigned odds of 50/50. And the information you have added should, in real situations, info...
I tried potassium supplementation. The very first thing I noticed is that a significant portion of hunger was immediately converted into thirst; to be specific, where normally at time X I would be hungry, instead at time X I was thirsty instead. There was an immediate and overall reduction of calories in.
This suggests to me that I had a slight potassium deficiency which my body was compensating for by increasing the amount of food I was consuming.
Cursory research suggests potassium content in fresh foods has declined ~20% over the past century ...
Instead of further elaborations on my crackpot nonsense, something short:
I expect that there is some distance from a magnetic source between 10^5 meters and 10^7 meters at which there will be magnetic anomalies; in particular, there will be a phenomenon by which the apparent field strength drops much faster than expected and passes through zero into the negative (reversed polarity).
I specifically expect this to be somewhere in the vicinity of 10^6 meters, although the specific distance will vary with the mass of the object.
There should be a second magnetic...
Yes, but then it sounds like those who have no such altruistic desire are equally justified as those who do. An alternative view of obligation, one which works very well with utilitarianism, is to reject personal identity as a psychological illusion. In that case there is no special difference between "my" suffering and "your" suffering, and my desire to minimize one of these rationally requires me to minimize the other. Many pantheists take such a view of ethics, and I believe its quasi-official name is "open individualism".
I think this requires an as...
Where you see neutrality, he would see obligation.
In what sense is it an obligation? By what mechanism am I obligated? Do I get punished for not living up to it?
You use that word, but the only meaningful source of that obligation, as I see it, is the desire to be a good person. Good, not neutral.
I disagree, and I think that you are more of a relativist than you are letting on. Ethics should be able to teach us things that we didn't already know, perhaps even things that we didn't want to acknowledge.
This is a point of divergence, an...
Utility, as measured, is necessarily relative. By this I don't mean that it is theoretically impossible to have an objective measure of utility, only that it is practically impossible; in reality / in practice, we measure utility relative to a baseline. When calculating the utility of doing something nice for somebody, it is impractical to calculate their current utility, which would include the totality of their entire experience as summed in their current experience.
Rule utilitarianism operates in the same fashion much more straightforwardly,...
The tax should, in fact, cause some landlords / landowners to just abandon their land. This is a critical piece of Georgism; the idea that land is being underutilized, in particular as an investment which is expected to pay off in terms of higher land values / rents later, but also in terms of things like parking lots, where the current value of the use of the land may exceed the current taxes (which include only a portion of the value of the land and the improvements combined) while being lower than the Georgist taxes (which include the entire value...
Related: https://www.lesswrong.com/posts/57sq9qA3wurjres4K/ruling-out-everything-else
I do not think the linked post goes anywhere near far enough. In particular, it imagines that people share a common concept-space. The totality to which thought is arbitrary is, basically, complete.
I'm a crackpot.
Self-identifiably as so. Part of the reason I self-identify as a crackpot is to help create a kind of mental balance, a pushback against the internal pressure to dismiss people who don't accept my ideas: Hey, self, most people who have strong beliefs similar to or about the thing you have strong beliefs about are wrong, and the impulse to rage against the institution and people in it for failing to grasp the obvious and simple ideas you are trying to show them is exactly the wrong impulse.
The "embitterment" impulse can be quite strong; when ...
Why are you using what I presume is your real name here?
I'm not actually interested in whether or not it is your real name, mind; mostly I'd like to direct your attention to the fact that the choice of username was in fact a choice. That choice imparts information. By choosing the username that you did, you are, deliberately or not, engaging in a kind of signaling.
In particular, from a particular frame of reference, you are engaging in a particular kind of costly signaling, which may serve to elevate your relative local status, by tying any rep...
I get the impression, reading this and the way you and commenters classify people, that the magnitude of days is to some extent just equivalent to an evaluation of somebody's intellectual ability, and the internal complexity of their thoughts.
So if I said your article "Ruling Out Everything Else" is the 10-day version of a 10000-day idea, you might agree, or you might disagree, but I must observe that if you agree, it will be taken as a kind of intellectual humility, yes? And as we examine the notion of humility in this context, I think it should be ...
It isn't the thing that the KL divergence is measuring, it is an analogy for it. The KL divergence is measuring the amount of informational entropy; strictly speaking, zipping a file has no effect in those terms.
However, we can take those examples more or less intact and place them in informational-entropy terms; the third gets a little weird in the doing, however.
So, having an intuition for what the ZIP file does, the equivalent "examples":
Example 1: KLE(Reference optimizer output stage, ineffective optimizer output) is 0; KLE(Reference final stage,...
Stepping into a real-world example, consider a text file, and three cases, illustrating different things:
First case: Entirely ineffective ZIP compression, (some processes), effective ZIP compression. If we treat the ineffective ZIP compression as "the optimizer", then it is clear that some compression happens later in the sequence of processes; the number of bits of optimization increased. However, the existence or non-existence of the first ineffective ZIP compression has no effect on the number of bits of optimization, so maybe this isn't qui...
I notice that my ethics, my morality, my beliefs, differ in many ways from those of the past; I expect these things to differ in many ways from my own, in the future. I notice the relationship between these two concepts is reciprocal.
My grandfather talked to me, several times, about how he knew I had my own life, and that I wouldn't always want to spend a lot of time with him; he was explicitly giving me permission, I think, to do something that he himself regretted in his youth, but understood better with age. He was telling me to live unfette...
What does it look like, when the optimization power is turned up to 11 on something like the air conditioner problem?
I think it looks exactly like it does now; with a lot of people getting very upset that local optimization often looks un-optimized from the global perspective.
If I needed an air-conditioner for working in my attic space, which is well-insulated from my living space and much, much hotter than either my living space or the outside air in the summer, the single-vent model would be more efficient. Indeed, it is effectively combining the m...
You have a simplification in your "black swan awareness" column which I don't think it is appropriate to carry over; in particular you'd need to rewrite the equation entirely to deal with an anti-Taleb, who doesn't believe in black swans at all. (It also needs to deal with the issue of repricocity; if somebody doesn't hang out with you, you can't hang out with them.)
You probably end up with a circle, the size of which determines what trends Taleb will notice; for the size of the apparent circle used for the fan, I think Taleb will notice a slight dow...
Yes, it does depend on the selection model; my point was that the selection model you were using made the same predictions for everybody, not just Taleb. And yes, changing the selection model changes the results.
However, in both cases, you've chosen the selection model that supports your conclusions, whether intentionally or accidentally; in the post, you use a selection model that suggests Taleb would see a negative association. Here, in response to my observation that that selection model predicts -everybody- would see a negative association,...
So ... smart people are worse than average at the task of evaluating whether or not smart people are worse than average at some generic task which requires intellectual labor to perform, and in fact smart people should be expected to be better than average at some generic task which requires intellectual labor to perform?
Isn't the task of evaluating whether or not smart people are worse than average at some generic task which requires intellectual labor to perform, itself a task which requires intellectual labor to perform? So shouldn't we expect the...
Another term for this pattern of behavior is "the script"; this terminology, and the related narrative-oriented way of framing the behavior, seems particularly common as arising from LSD usage, dating back something like sixty years at this point to an individual whose name I can't quite recall.
In this framing, people see themselves as characters living out a story; the grayed-out options are simply those things that are out of character for them. Insofar as your character is "agent of chaos", as another commenter alludes to, you still have grayed-ou...
The topic question is "Why is Toby Ord's likelihood of human extinction due to AI so low?"
My response is that it isn't low; as a human-extinction event, that likelihood is very high.
You ask for a comparison to MIRI, but link to EY's commentary; EY implies a likelihood of human extinction of, basically, 100%. From a Bayesian updating perspective, 10% is closer to 50% than 100% is to 99%; Ord is basically in line with everybody else, it is EY who is entirely off the charts. So the question, why is Ord's number so low, is being raised in the conte...
Remember that mathematics is something we make up; mathematics isn't fundamental to, prior to, or indeed related to existence itself at all; mathematics is the process of formalizing rules and seeing what happens. You can invent whatever rules you want, although the interesting stuff generally doesn't really happen unless the rules are consistent / satisfiable with respect to one another.
The fact that mathematics happen to be useful in describing reality doesn't imply that reality is fundamentally mathematical, except in the sense that reality does s...
Suppose astronomers detect an asteroid, and suggest a 10% chance of it hitting the Earth on a near-pass in 2082. Would you regard this assessment of risk as optimistic, or pessimistic? How many resources would you dedicate to solving the problem?
My understanding is that 10% isn't actually that far removed from what many people who are deeply concerned about AI think (or, for that matter, people who aren't that concerned about AI think - it's quite remarkable how differently people can see that 10%); they just happen to think that a 10% chance o...
I think this is a useful abstraction.
But I think the word you're looking for is "god". In the "Bicameral Consciousness" sense - these egregores you refer to are gods that speak to us, whose words we know. There's another word, zeitgeist, that refers to something like the same thing.
If you look in your mind, you can find them; just look for what you think the gods would say, and they will say it. Pick a topic you care about. What would your enemy say about that topic? There's a god, right there, speaking to you.
Mind, in a sense...
If we consider the extra dimension(s) on which the amplitude of the wave function given to the Schrodinger Equation, the wave function instead defines a topology (or possibly another geometric object, depending on exactly what properties end up being invariant.)
If the topology can be evaluated over time by some alternative mathematical construct, that alternative mathematical construct may form the basis for a more powerful (in the sense of describing a wider range of potential phenomena) physics, because it should be constructable in such a way as to not ...
Suppose for a moment your washing machine is broken.
You have some options; you could ignore the problem. You could try to fix it yourself. You could call somebody to fix it. This isn't intended to be a comprehensive list of options, mind, these are cached thoughts.
Each of these options in turn produce new choices; what to do instead, what to try to do to fix it, who to call.
Let's suppose for a moment that you decide to call somebody. Who do you call? You could dial random numbers into your phone, but clearly that's not a great...
The point is that meaningful labor is increasingly "selection effort", the work involved in making a decision between multiple competing choices, and some starter thoughts about how society can be viewed once you notice the idea of making choices as meaningful labor (maybe even the only meaningful form of labor).
The idea of mapping binary strings to choices is a point that information is equivalent to a codification of a sequence of choices; that is, the process of making choices is in fact the process of creating information. For a choice between N ...
Suppose you have a list of choices a selection must be made from, and that the decision theory axioms of orderability and transitivity apply.
It should then be possible to construct a binary tree representing this list of choices, such that a choice can be represented as a binary string.
Likewise, a binary string, in a certain sense, represents a choice.
In this specific sense, what computers automate is the process of selection, of choice. Noticing this, and noticing that computers have automated away considerable amounts of "work", we must notice that...
Would you willingly go back in time and re-live your life from the beginning, with all the knowledge you have now? Say, knowing what stocks to purchase, what cryptocurrencies are worth buying and when, being able to breeze through education and skip ahead in life, and all the other advantages you would have?
If the answer to that is yes, then observe that this is exactly the same thing.
The point of this being that you don't actually think of past-you, present-you, and future-you as you in the same sense. You'll happily overwrite past-you with present-you, but you'd see it as a problem if future-you overwrote present-you, so far as to be equatable to dying.
What, exactly, does it even mean for "you" to exist for 100k years?
Is the "you" from yesterday "you"? Would you be comfortable with your conscious mind being replaced with the conscious mind of that entity? What about the "you" from tomorrow"? What about the "you" from 100k years in the future? If that's still "you", should it be a problem for your mind to be erased, and that mind to be written in its place?
What does it mean, for a thing to move?
First, what phenomenon are we even talking about? It's important to start here. I'm going to start somewhat cavalierly: Motion is a state of affairs in which, if we measure two variables, X and T, where X is the position on some arbitrary dimension relative to some arbitrary point using some arbitrary scale, and T is the position in "time" as measured by a clock (also arbitrary), we can observe that X varies with T.
Notice there are actually two distinct phenomena here: There is the fact that "X" changed, w...
Well, if my crackpot physics is right, it actually kind of reduces the probability I'd assign to the world I inhabit being "real". Seriously, the ideas aren't complicated, somebody else really should have noticed them by now.
But sure it makes predictions. There should be a repulsive force which can be detected when the distance between two objects is somewhere between the radius of the solar system and the radius of the smallest dwarf galaxy. I'd guess somewhere in the vicinity of 10^12 meters.
Also electrical field polarity should invert ...
Because it's expensive, slow, and orthogonal to the purpose the AI is actually trying to accomplish.
As a programmer, I take my complicated mirror models, try to figure out how to transform them into sets of numbers, try to figure out how to use one set of those numbers to create another set of those numbers. The mirror modeling is a cognitive step I have to take before I ever start programming an algorithm; it's helpful for creating algorithms, but useless for actually running them.
Programming languages are judged as helpful in part by how well they ...
Another crackpot physics thing:
My crackpot physics just got about 10% less crackpot. As it transpires, one of the -really weird- things in my physics, which I thought of as a negative dimension, already exists in mathematics - it's a Riemann Sphere. (Thank you, Pato!)
This "really weird" thing is kind of the underlying topology of the universe in my crackpot physics - I analogized the interaction between this topology and mass once to an infinite series of Matryoshka dolls, where every other doll is "inside out and backwards". Don't ask me... (read more)