it takes on average 18 years to build a mine
Always worth reminding people that the "building" is ~4 of those years, on average. Half the 18 is permitting/litigation, which the government seems to have forgotten it gets to change if it wants to, except when it wants to shut down more things. The rest is exploration and feasibility studies, which can be accelerated (or at least parallelized) by spending more money, if the time-value of minerals is high.
We noble few are, there are dozens of us
The annoying pedant in me wants to say: These are important realizations! How could you have realized them sooner?
The less annoying pedant in me wants to say: This is, I think, a very hard lesson in general. Some people never learn it. The ones who do are rarely young. Those who try to tell the young rarely succeed. What's a better strategy for convincing people of this?
2You know, I thought we wouldn't start seeing AIs preferentially cooperate with each other and shut out humans from the gains from trade until a substantially higher level of capabilities.
This seems like yet another pretty clear warning of things to come if we don't get a whole lot smarter and more serious about what we're doing.
So odd that this got slated first for Idaho.
I assume it's because INL near Arco is one of the very few places in the US where a critical mass of people are likely to believe that SMRs are a good thing.
This is a great start, but still a drop in the bucket, as I understand it, compared to what we will need if we intend to largely rely on solar and wind in the future.
Yes, but also, that graph is in GW, not GWh. It's saying that, at peak discharge, batteries can already provide about 5% of record peak demand. That's ~4.3 doublings from what we need in that record. But, those are on average only 2 hr batteries - 83GWh. We'll need at least 7 doubles of capacity to be able to deal with the intermittency of a mostly-renewables grid - exact amount TBD based on how many other things we can get right. Luckily that also means when we move beyond ~4 hrs or so we can largely switch to (what should become) cheaper but lower-peak-power options than Li-ion (that are also less critical mineral intensive).
The US Forest Service wanted to implement a wildfire prevention plan, so it had to fill out an environmental impact statement. Before they could complete the environmental impact statement, though, half the forest burned down.
I take it as given that in a saner world, we would be rushing to gather up excess fuel that feeds wildfires, and turning it into biochar, or putting it into bioenergy plants. Instead, we can't even give ourselves permission to controlled-burn it, when it's going to uncontrolled-burn anyway.
"There exists a physical world that keeps going whether or not you want to deal with it" seems to not be a part of our societal consensus.
How long does that intermediate step take? If the machine shop-shop outputs one shop per week, but every 4th (or whatever) output is a mining-robot-shop or a forestry-robot-shop or something else instead, is that sufficient to solve your inputs problem while only slowing down the doubling time by a modest amount?
I think you're overestimating the discourse on Frost.
But this is already presupposing the existence of the superintelligence whose feasibility we are trying to explain.
Strictly speaking I only presupposed an AI could reach close to the limits of human intelligence in terms of thinking ability, but with the inherent speed and parallelizability and memory advantages of a digital mind.
Do you have any examples handy of AI being successful at real-world goals?
In small ways (aka sized appropriately for current AI capabilities) this kind of thing shows up all the time in chains of thought in response to all kinds of prompts, to the point that no, I don't have specific examples, because I wouldn't know how to pick one. The one that first comes to mind, I guess, was using AI to help me develop a personalized nutrition/supplement/weight loss/training regimen.
Stepping back, I should reiterate that I'm talking about "the current AI paradigm"
That's fair, and a reasonable thing to discuss. After all, the fundamental claim of the book's title is about a conditional probability: IF it turns out that the anything like our current methods scale to superintelligent agents, we'd all be screwed.
I sincerely hope that if anyone has a concrete, actionable answer to this question, that they're smart enough not to share it publicly, for what I hope are obvious reasons.
But aside from that caveat, I think you are making several incorrect assumptions.
- "There is no massive corpus of such strategies that can be used as training data"
- The AI has, at minimum access-in-principle to everything that has ever been written or otherwise recorded, including all fiction, all historical records, and all analysis of both of those. This includes many, many, many examples and discussions of plans, successful and not, and detailed discussions of why humans believe they succeeded or failed.
- "(a) doing real-world experiments (whereby generating sufficient data would be far too slow and costly, or simply impossible)"
- People have already handed substantial amounts of crypto to at least one AI, which it can use to autonomously act in the real world by paying humans. What do you see as the upper bound on this, and why?
- I think most people greatly overestimate how much of this is actually needed for many kinds of goals. What do you see as the upper bound for what can, in principle, be done with a plan that an army of IQ-180 humans (aka no better qualitative thinking than what the smartest humans can do, so that this is a strict lower bound on ASI capabilities) came up with over subjective millennia with access to all recorded information that currently exists in the world? Assume the plan includes the capability to act in parallel, at scale, and the ability to branch its actions based on continued observation, just like groups of humans can, but with much better coordination within the group.
- "(b) a comprehensive world-model that is capable of predicting the results of proposed actions"
- See above - I'm not sure what you see as the upper bound for how good such a world model can or would likely be?
- One answer is "Because we're going to have long since handed it thousands to billions bodies to operate in the world, and problems to come up with plans to solve, and compute to use to execute and revise those plans." Without the bodies, we're already doing this.
- Current non-superintelligent AIs already come up with hypotheses and plans to test them and means to revise them and checks against past data all the time with increasing success rates over a widening range of problems. This is synthetic data we're already paying to generate.
- Also, have you ever run a plan (or anything else) by an LLM and asked it to find flaws and suggest solutions and estimate probabilities of success? This is already very useful at improving on human success rates across many domains.
- "Plans for achieving such goals are not amenable to simulation because you can't easily predict or evaluate the outcome of any proposed action. "
- It's actually very easy to get current LLMs to generate hypothetical actions well outside a narrow domain if you explain to them that there are unusually high stakes. We're not talking about a traditional chess engine thinking outside the rules of chess. We're about about systems whose currently-existing predecessors are increasingly broadly capable of finding solutions to open-ended problems using all available tools. This includes capabilities like deception, lying, cheating, stealing, giving synthesis instructions to make drugs, and explaining how to hire a hitman.
- Any plan a human can come up with without having personally conducted groundbreaking relevant experiments, is a plan that exists within or is implied by the combined corpus of training data available to an AI. This includes, for example, everything ever written by this community or anyone else, and everything anyone ever thought about upon reading everything ever written by this community or anyone else.
True, but I think in this case there's at least no risk of an infinite regress. At one end, yes, it bottoms out in an extremely vague and inefficient but general hyperprior. I would guess from the little I've read that in humans these are the layers that govern how we learn from even before we're born. I would imagine an ASI would have at least one layer more fundamental than this, which enable it to change various fixed-in-humans assumptions about things.
At the other end would be the most specific or most abstracted layer of priors that has proven useful to date. Somewhere in the stack are your current best processes for deciding whether particular priors or layers of priors are useful or worth keeping or if you need a new one.
I am actually not sure whether 'prior' is quite the right term here? Some of it feels like the distinction between thingspace and conceptspace, where the priors might be more about the expectations what things exist and where natural concept boundaries lie and how to evaluate and re-evaluate those?
It would probably be a good idea to have the junior associate check the citations and read the document first, before the senior lawyer does. Still should save a lot of net time.