Not quite. What you said is a reasonable argument, but the graph is noisy enough, and the theoretical arguments convincing enough, that I still assign >50% credence that data (number of feedback loops) should be proportional to parameters (exponent=1).
My argument is that even if the exponent is 1, the coefficient corresponding to horizon length ('1e5 from multiple-subjective-seconds-per-feedback-loop', as you said) is hard to estimate.
There are two ways of estimating this factor
Note that you can still get EUM-like properties without completeness: you just can't use a single fully-fleshed-out utility function. You need either several utility functions (that is, your system is made of subagents) or, equivalently, a utility function that is not completely defined (that is, your system has Knightian uncertainty over its utility function).
See Knightian Decision Theory. Part I
Arguably humans ourselves are better modeled as agents with incomplete preferences. See also Why Subagents?
I have an intuition that any system that can be modeled as a committee of subagents can also be modeled as an agent with Knightian uncertainty over its utility function. This goal uncertainty might even arise from uncertainty about the world.
This is similar to how in Infrabayesianism an agent with Knightian uncertainty over parts of the world is modeled as having a set of probability distributions with an infimum aggregation rule.
This not the same thing, but back in 2020 I was playing with GPT-3, having it simulate a person being interviewed. I kept asking ever more ridiculous questions, with the hope of getting humorous answers. It was going pretty well until the simulated interviewee had a mental breakdown and started screaming.
I immediately felt the initial symptoms of an anxiety attack as I started thinking that maybe I had been torturing a sentient being. I calmed down the simulated person, and found the excuse that it was a victim of a TV prank show. I then showered them with...
I think the median human performance on all the areas you mention is basically determined by the amount of training received rather than the raw intelligence of the median human.
1000 years ago the median human couldn't write or do arithmetic at all, but now they can because of widespread schooling and other cultural changes.
A better way of testing this hypothesis could be comparing the learning curves of humans and monkeys for a variety of tasks, to control for differences in training.
Here's one study I could find (after ~10m googling) comparing the learni...
I second the other answers that even if we completely solve cybersecurity, there would be substantial AI risk just by having the AI interact with humans, via manipulation, etc.
That said, I think it would close a huge part of the attack surface for the AI. If, in addition to that, suddenly in 2032 we discover how to make humans invulnerable to manipulation, I would feel much better about running experiments with unaligned AI, boxing, etc.
So I'd say it's something like "vastly better cybersecurity is not enough to contain unaligned AGI, but any hope of containing unaligned AGI requires vastly better cybersecurity"
Literally the only thing in the story that lets the AGI win is the nanobots. That's it. All the rest is surperfluous.
Well, if nanobots are possible then they are such a powerful technology that any AGI will eventually want to build them, unless it has something even better. But let's assume that nanobots are impossible and try to build a story.
I'm going to be lazy and use Gwern's premise of an AGI that escapes during training and hacks its way into some money and some hardware to run in. Instead of going fast and doing overt actions, the AGI stays hidden.
I...
For example, we could simulate a bunch of human-level scientists trying to build nanobots and also checking each-other's work.
That is not passively safe, and therefore not weak. For now forget the inner workings of the idea: at the end of the process you get a design for nanobots that you have to build and deploy in order to do the pivotal act. So you are giving a system built by your AI the ability to act in the real world. So if you have not fully solved the alignment problem for this AI, you can't be sure that the nanobot design is safe unless you are c...
Q has done nothing to prevent another AGI from being built
Well, yeah, because Q is not actually an AGI and doesn't care about that. The point was that you can create an online persona which no one has ever seen even in video and spark a movement that has visible effects on society.
The most important concern an AGI must deal with is that humans can build another AGI, and pulling a Satoshi or a QAnon does nothing to address this.
Even if two or more AGIs end up competing among themselves, this does not imply that we survive. It probably looks more like Europe...
It's somewhat easier to think of scenarios where the takeover happens slowly.
There's the whole "ascended economy" scenarios where AGI deceptively convinces everyone that it is aligned or narrow, is deployed gradually in more and more domains, automates more and more parts of the economy using regular robots until humans are not needed anymore, and then does the lethal virus thing or defects in other way.
There's the scenario where the AGI uploads itself into the cloud, uses hacking/manipulation/financial prowess to sustain itself, then uses manipulation to ...
Some things that come to mind, not sure if this is what you mean and they are very general but it's hard to get more concrete without narrowing down the question:
I'm not sure if using the Lindy effect for forecasting x-risks makes sense. The Lindy effect states that with 50% probability, things will last as long as they already have. Here is an example for AI timelines.
The Lindy rule works great on average, when you are making one-time forecasts of many different processes. The intuition for this is that if you encounter a process with lifetime T at time t<T, and t is uniformly random in [0,T], then on average T = 2*t.
However, if you then keep forecasting the same process over time, then once you surpass T/2 you...
Let me put on my sciency-sounding mystical speculation hat:
Under the predictive processing framework, the cortex's only goal is to minimize prediction error (surprise). This happens in a hierarchical way, with predictions going down and evidence going up, and upper levels of the hierarchy are more abstract, with less spatial and temporal detail.
A visual example: when you stare at a white wall, nothing seems to change, even though the raw visual perceptions change all the time due to light conditions and whatnot. This is because all the observations are con...
Why do you think this sort of training environment would produce friendly AGI?
Can you predict what kind of goals an AGI trained in such an environment would end up with?
How does it solve the standard issues of alignment like seeking convergent instrumental goals?
Re: April 5: TV host calls for killing as many Ukrainians as possible.
I know no Russian, but some people in the responses are saying that the host did not literally say that. Instead he said some vague "you should finish the task" or something like that. Still warmongering, but presumably you wouldn't have linked it if the tweet had not included the "killing as many Ukrainians as possible" part.
Could someone verify what he says?
I am Russian and I can confirm that he most certainly did not call for "killing of as many Ukrainians as possible".
He said that there can be no negotiations with Nazis outside of putting a boot on their neck because it will be seen as weakness, and that you shouldn't try to have "small talk" with them, or shake hands with them. He did say "a task is set and it should be finished". He did not explicitly say anything about killing, let alone "as many as possible", at least not in that clip.
It seems like one literally cannot trust anyone to report information about this war accurately.
I'm sorry, but I find the tone of this post a bit off-putting. Too mysterious for my taste. I opened the substack but it only has one unrelated post.
I don’t think there is a secular way forward.
Do you think that there is a non-secular way forward? Did you previously (before your belief update) think there is a non-secular way forward?
We just shamble forward endlessly, like a zombie horde devouring resources, no goal other than the increase of some indicator or other.
I can agree with this, but... those indicators seem pretty meaningful for me. Life expectan...
Let me paraphrase your argument, to see if I've understood it correctly:
Physical constraints on things such as energy consumption and dissipation imply that current rates of economic growth on Earth are unsustainable in the relatively short term (<1000 years), even taking into account decoupling, etc.
There is a strong probability that expanding through space will not be feasible
Therefore, we can reasonably expect growth to end some time in the next centuries
First of all, if economic progress keeps being exponential then I think it's quite po...
My perspective as a native speaker who doesn't remember his grammar lessons very well:
The subjunctive mood has a lot of uses, at least in Spain (I'm not really familiar with other varieties of Spanish). Some examples off the top of my head:
1. Counterfactual conditionals: "Si Lee Harvey Oswald no hubiera disparado a JFK, alguien más lo habría hecho" (If Lee Harvey Oswald hadn't shot JFK, someone else would have), here "no hubiera disparado" is subjunctive and means "hadn't shot".
2. To speak about people'...
I don't reflect on it. This happens in two ways:
I find reflecting much more cognitively demanding than reading, so if there is a 'next post' button or similar, I tend to keep reading.
Also, sometimes when I try to actually think about the subject, it's difficult to come up with original ideas. I often find myself explaining or convincing an imaginary person, instead of trying to see it with fresh eyes. This is something I noticed after reading the corresponding Sequence.
I guess establishing an habit of commenting would help me solve these problems.
Hello, I'm a math-cs undergrad and aspiring effective altruist, but I haven't chosen a cause yet. Since that decision is probably one of the most important ones, I should probably wait until I've become stronger.
To that end, I've read the Sequences (as well as HPMOR), and I would like to attend a CFAR workshop or similar at some point in the future. I think one of my problems is that I don't actually think that much about what I read. Do you have any advice on that?
Also, there are a couple of LWers in my college with whom I have met twice, and we would lik...
The arguments you make seem backwards to me.
Yes, this is the standard Georgist position, and... (read more)