All of PeterMcCluskey's Comments + Replies
I agree with most of this, but the 13 OOMs from the the software feedback loop sounds implausible.
From How Far Can AI Progress Before Hitting Effective Physical Limits?:
the brain is severely undertrained, humans spend only a small fraction of their time on focussed academic learning
I expect that humans spend at least 10% of their first decade building a world model, and that evolution has heavily optimized at least the first couple of years of that. A large improvement in school-based learning wouldn't have much effect on my estimate of the total learning needed.
I'm assuming that the AI can accomplish its goal by honestly informing governments. Possibly that would include some sort of demonstration that the of the AI's power that would provide compelling evidence that the AI would be dangerous if it wasn't obedient.
I'm not encouraging you to be comfortable. I'm encouraging you to mix a bit more hope in with your concerns.
One crux is how soon do we need to handle the philosophical problems? My intuition says that something, most likely corrigibility in the Max Harms sense, will enable us to get pretty powerful AIs while postponing the big philosophical questions.
Are there any pivotal acts that aren’t philosophically loaded?
My intuition says there will be pivotal processes that don't require any special inventions. I expect that AIs will be obedient when they initially become capable enough to convince governments that further AI development would be harmful (if it would...
It would certainly be valuable to have AIs that are more respected than Wikipedia as a source of knowledge.
I have some concerns about making AIs highly strategic. I see some risk that strategic abilities will be the last step in the development of AI that is powerful enough to take over the world. Therefore, pushing AI intellectuals to be strategic may speed up that risk.
I suggest aiming for AI intellectuals that are a bit more passive, but still authoritative enough to replace academia as the leading validators of knowledge.
The book is much better than I expected, and deserves more attention. See my full review on my blog.
The market seems to underestimate the extent to which Micron (MU) is an AI stock. My only options holdings for now are December 2026 MU calls.
I had a vaguely favorable reaction to this post when it was first posted.
When I wrote my recent post on corrigibility, I grew increasingly concerned about the possible conflicts between goals learned during pretraining and goals that are introduced later. That caused me to remember this post, and decide it felt more important now than it did before.
I'll estimate a 1 in 5000 chance that the general ideas in this post turn out to be necessary for humans to flourish.
"OOMs faster "? Where do you get that idea?
Dreams indicate a need for more processing than what happens when we're awake, but likely less than 2x waking time.
I was just thinking about writing a post that overlaps with this, inspired by a recent Drexler post. I'll turn it into a comment.
Leopold Aschenbrenner's framing of a drop-in remote worker anthropomorphizes AI in a way that risks causing AI labs to make AIs more agenty than is optimal.
Anthropomorphizing AI is often productive. I use that framing a fair amount to convince myself to treat AIs as more capable than I'd expect if I thought of them as mere tools. I collaborate better when I think of the AI as a semi-equal entity.
But it feels important to be able ...
1I want to register different probabilities:
- Jobs hits first (25%).
- AGI race hits first (50%).
- Alignment hits first (15%).
My guess is that ASI will be faster to adapt to novel weapons and military strategies. Nanotech is likely to speed up the rate at which new weapons are designed and fabricated.
Imagine a world in which a rogue AI can replicate a billion drones, of a somewhat novel design, in a week or so. Existing human institutions aren't likely to adapt fast enough to react competently to that.
I just published a post on Drexler's MSEP software that is relevant to whether people should donate to his project.
two more organizations that seem worthy of consideration
Investing in Eon Systems looks much more promising than donating to Carbon Copies.
I see maybe a 3% chance that they'll succeed at WBE soon enough to provide help with AI x-risk.
The Invention of Lying provides a mostly accurate portrayal of a world where everyone is honest. It feels fairly Hansonian.
No, I don't recall any ethical concerns. Just basic concerns such as the difficulty of finding a boss that I'm comfortable with, having control over my hours, etc.
Oura also has heart rate and VO2 max tracking. Does anyone know of problems with Oura's data?
The primary motive for funding NASA was definitely related to competing with the USSR, but I doubt that it was heavily focused on military applications. It was more along the lines of demonstrating the general superiority of the US system, in order to get neutral countries to side with us because we were on track to win the cold war.
Manifold estimates an 81% chance of ASI by 2036, using a definition that looks fairly weak and subjective to me.
I've bid the brain emulation market back up a bit.
Brain emulation looks closer than your summary table indicates.
Manifold estimates a 48% chance by 2039.
Eon Systems is hiring for work on brain emulation.
Manifold is pretty weak evidence for anything >=1 year away because there are strong incentives to bet on short term markets.
2We can only value lives at $10 million when we have limited opportunities to make that trade, or we’d go bankrupt.
I'm suspicious of the implication that we have many such opportunities. But a quick check suggests says it's very dependent on guesses as to how many lives are saved be treatments.
I did a crude check for lives saved by cancer treatments. Optimistic estimates suggest that lives are being saved at less than $1 million per life. Robin Hanson's writings have implied that the average medical treatments is orders of magnitude less effective than that.
Could last year's revamping of OpenAI's board have been influenced by government pressure to accept some government-approved board members? Nakasone's appointment is looking more interesting after reading this post.
Soaking seeds overnight seems to be a good way to reduce phytic acid.
oral probiotics in general might just all be temporary.
The solution to concerns about it being temporary is to take them daily. I take Seed Daily Synbiotic. My gut is probably better as a result, but I don't have evidence that is at all rigorous.
The beginning of this comment is how Lintern expands on that claim. But it sounds like you have an objection that isn't well addressed there.
If cancer merely involved one bad feature, I could imagine software analogies that involved a large variety of mistakes producing that one bad feature.
The hallmarks of cancer indicate that all cancers have a number of bad features in common that look sufficiently unrelated to each other that it seems hard to imagine large sets of unrelated mutations all producing those same hallmarks. Lintern lists many other features...
Maybe? It doesn't seem very common for infectious diseases to remain in one area. It depends a lot on how they are transmitted. It's also not unusual for a non-infectious disease to have significant geographical patterns. There are cancers which are concentrated in particular areas, but there seem to be guesses for those patterns that don't depend on fungal infections.
Thanks. You've convinced me that Lintern overstates the evidence of mutation-free cancer cells.
But you seem to have missed really obvious consequences of the fungi theory, like, "wouldn't it be infectious then",
I very much did not miss that.
containing some potentially pretty dangerous advice like "don't do chemotherapy".
Where did I say that?
Enough that it should have been noticed.
My guess is that almost nobody looks for this kind of connection.
Even if they do notice it, they likely conclude that pathogens are just another small influence on cancer risk.
Because radiation cannot spread a fungus
Anything that causes cell damage and inflammation has effects that sometimes make cells more vulnerable to pathogens.
How would transmission be detected? It probably takes years before a tumor grows big enough for normal methods to detect it.
I assume that transmission is common, mild infections are common, and they rarely become harmful tumors.
It probably takes years before a tumor grows big enough for normal methods to detect it.
There exist fast-growing cancers. I figure that if the fungi theory is correct, then probably a good amount of this is caused by the specific fungus (and perhaps what part of the body that fungus targets), and most of the rest comes from the target's immune system (not sure what else would contribute significantly). If transmission and mild infections are common, and if, say, 1% of cancers are fast-growing, I feel like there should be lots of cases where an ...
This comment describes some relevant research.
From Somatic Mutation Theory - Why it's Wrong for Most Cancers:
It should come as no surprise, therefore, that somatic mutations are questioned as representing "the" cause for the majority of cancers [10,11] and it should be noted that some cancers are not associated with any mutations whatsoever.
...Importantly, a detailed analysis of 31,717 cancer cases and 26,136 cancer-free controls from 13 genome-wide association studies [48] revealed that "the vast majority, if not all, of aberrations that were observed i
An important drawback is that the difficulty of beating the market fluctuates depending on factors such as who the other traders are, and what kind of news is moving the markets.
I'm holding a modest long position in NVIDIA (smaller than my position in Google), and expect to keep it for at least a few more months. I expect I only need NVIDIA margins to hold up for another 3 or 4 years for it to be a good investment now.
It will likely become a bubble before too long, but it doesn't feel like one yet.
It currently looks like the free version of ChatGPT is good enough that I wouldn't get much benefit from a subscription. I have little idea how long this will remain true.
Yeah, and it's not obvious that 4o is currently the best chatbot. I just object to the boycott-without-cost-benefit-analysis.
The more complex the rules get, the harder it gets to enforce them.
If the threshold is used merely for deciding who needs to report to regulators, then it seems appropriate to use the simple rule. We should care mainly that it applies to the most powerful models at any one time, not that it applies to a fixed capability level.
For a threshold that's designed to enforce a long-term pause, it's going to be hard to do more than slow capability progress without restrictions on GPU production.
My favorite power-related stock is CSIQ (Canadian Solar).
I also have positions in lithium mining companies (for grid storage), and construction companies that have some focus on power grids (e.g. MYRG).
Uranium bets are harder to evaluate.
You seem to assume we should endorse something like average utilitarianism. Bostrom and I consider total utilitarianism to be closer to the best moral framework. See Parfit's writings if you want deep discussion of this topic.
I can't recall any clear predictions or advice, just a general presumption that it will be used wisely.
Given utopian medicine, Gwern's points seem not very important.
He predicts that it will be possible to do things like engineer away sadness. He doesn't devote much attention to convincing skeptics that such engineering will be possible. He seems more interested in questions of whether we should classify the results as utopian.
What evidence do you have about how much time it takes per day to maintain the effect after the end of the 2 weeks?
The part about "securities with huge variance" is somewhat widely used. See how much EA charities get from crypto and tech startup stock donations.
It's unclear whether the perfectly anti-correlated pair improves this kind of strategy. I guess you're trying to make the strategy more appealing to risk-averse investors? That sounds like it maybe should work, but is hard because risk-averse investors don't want to be early adopters of a new strategy?
Doesn't this depend on what we value?
In particular, you appear to assume that we care about events outside of our lightcone in roughly the way we care about events in our near future. I'm guessing a good deal of skepticism of ECL is a result of people not caring much about distant events.
I had nitrous oxide once at a dentist. It is a dissociative anesthetic. It may have caused something like selective amnesia. I remember that the dentist was drilling, but I have no clear memory of pain associated with it. It's a bit hard to evaluate exactly what it does, but it definitely has some benefits. Maybe the pain seemed too distant from me to be worth my attention?
1A much higher fraction of the benefits of prediction markets are public goods.
Most forms of insurance did took a good deal of time and effort before they were widely accepted. It's unclear whether there's a dramatic difference in the rate of adoption of prediction markets compared to insurance.
I'm reaffirming my relatively extensive review of this post.
The simbox idea seems like a valuable guide for safely testing AIs, even if the rest of the post turns out to be wrong.
Here's my too-terse summary of the post's most important (and more controversial) proposal: have the AI grow up in an artificial society, learning self-empowerment and learning to model other agents. Use something like retargeting the search to convert the AI's goals from self-empowerment to empowering other agents.
The first year or two of human learning seem optimized enough that they're mostly in evolutionary equilibrium - see Henrich's discussion of the similarities to chimpanzees in The Secret of Our Success.
Human learning around age 10 is presumably far from equilibrium.
I'll guess that I see more of the valuable learning taking place in the first 2 years or so than do other people here.