All of RussellThor's Comments + Replies

Got to the end before I misread "Journalism is about deception" Otherwise v good!

Yes, you really need to mention seasonal storage or just using 15% FF to get a wholistic picture. That applies to electricity alone and especially for the whole global economy. For example low efficiency/RTE, low capex, high storage capacity sources really bring the costs down compared to battery storage if the aim is to get to 100% RE and start to matter around 85%.

For example H2 with 50% RTE used for 10% of total electricity, and stored for 6 months doesn't end up costing much with really cheap solar and a low capex H2 electrolyser/fuel cell combo. Simil... (read more)

Good work, thinking ahead to TAI - Is there an issue with the self-other distinction when it comes to moral weightings. For example if it weights the human as self rather than other, but with 1% of its own moral weighting, then its behavior would still not be what we want. Additionally are there situations or actions it could take where it would increase its relative moral weight further - the most obvious just being taking more compute. This is assuming it believed it was conscious and of similar moral value to us. Finally if it suddenly decided it had moral value or much more moral value then it previously did, then that would still give a sudden change in behavior.

Yes to much  of this.  For small tasks or where I don't have specialist knowledge I can get 10* speed increase - on average I would put 20%. Smart autocomplete like Cursor is undoubtably a speedup with no apparent downside. The LLM is still especially weak where I am doing data science or algorithm type work where you need to plot the results and look at the graph to  know if you are making progress.

Things the OP is concerned about like 

"What makes this transition particularly hard to resist is that pressures on each societal system bleed into the others. For example, we might attempt to use state power and cultural attitudes to preserve human economic power. However, the economic incentives for companies to replace humans with AI will also push them to influence states and culture to support this change, using their growing economic power to shape both policy and public opinion, which will in turn allow those companies to accrue even greater eco

The  OP is specifically about gradual disempowerment. Conditional on gradual disempowerment, it would help and not be decades away. Now we may both think that sudden disempowerment is much more likely. However in a gradual disempowerment world, such colonies would be viable much sooner as AI could be used to help build them, in the early stages of such disempowerment when humans could still command resources. 

In a gradual disempowerment scenario vs no super AI scenario, humanities speed to deploy such colonies starts the same before AI can be use... (read more)

Space colonies are a potential way out - if a small group of people can make their own colony then they start out in control. The post assumes a world like it is now where you can't just leave. Historically speaking that is perhaps unusual - much of the time in the last 10,000 years it was possible for some groups to leave and start anew.

This does seem different however https://solarfoods.com/ - they are competing with food not fuel which can't be done synthetically (well if at all). Also widely distributed capability like this helps make humanity more resilient e.g. against nuke winter, extreme climate change, space habitats

Thanks for this article, upvoted.

Firstly Magma sounds most like Anthropic, especially the combination of Heuristic #1 Scale AI capabilities and also publishing safety work.

In general I like the approach, especially the balance between realism and not embracing fatalism. This is opposed to say MIRI, Pause AI and at the other end, e/acc. (I belong to EA, however they don’t seem to have a coherent plan I can get behind) I like the realization that in a dangerous situation doing dangerous things can be justified. Its easy to be “moral” and just say “stop” howe... (read more)

Thats not a valid criticism if we are simply about choosing one action to reduce X-risk. Consider for example the cold war - the guys with nukes did the most to endanger humanity however it was most important that they cooperated to reduce it.

In terms of specific actions that don't require government, I would be positive about an agreement between all the leading labs that when one of them made an AI (AGI+) capable of automatic self improvement they would all commit to share it between them and allow 1 year where they did not hit the self improve button, but instead put that towards alignment. 12 months may not sound like a lot, but if the research is 2-10* because of such AI then it would matter. In terms of single potentially achievable actions that will help that seems to be the best to me.

Not sure if this is allowed but you can aim at a rock or similar say 10m away from the target (4km from you) to get the bias (and distribution if multiple shots are allowed). Also if the distribution is not totally normal, but has smaller than normal tails then you could aim off target with multiple shots to get the highest chance of hitting the target. For example if the child is head height then aim for the targets feet, or even aim 1m below the target feet expecting 1/100 shots will actually hit the targets legs, but only <1/1000 say will hit the chi... (read more)

I think height is different to IQ in terms of effect. There are simple physical things that make you bigger, I expect height to be linear for much longer than IQ. 

Then there are potential effects, like something seems linear until OOD, but such OOD samples don't exist because they die before birth. If that was the case it would look like you could safely go OOD. Would certainly be easier if we had 1 million mice with such data to test on.

That seems so obviously correct as a starting point, not sure why the community here doesn't agree by default. My prior for each potential IQ increase would be that diminishing returns would kick in - I would only update against when actual data comes in disproving that.

OK I guess there is a massive disagreement between us on what IQ increases gene changes can achieve.  Just putting it out there, if you make an IQ 1700 person they can immediately program an ASI themselves, have it take over all the data centers rule the world etc.

For a given level of IQ controlling ever higher ones, you would at a minimum require the creature to decide morals, ie. is Moral Realism true, or what is? Otherwise with increasing IQ there is the potential that it could think deeply and change its values, additionally believe that they would not be able to persuade lower IQ creatures of such values, therefore be forced into deception etc.

"with a predicted IQ of around 1700." assume you mean 170. You can get 170 by cloning existing IQ 170 with no editing necessary so not sure the point.

I don't see how your point addresses my criticism - if we assume no multi-generational pause then gene editing is totally out. If we do, then I'd rather Neuralink or WBE. Related to here
https://www.lesswrong.com/posts/7zxnqk9C7mHCx2Bv8/beliefs-and-state-of-mind-into-2025 

(I believe that WBE can get all the way to a positive singularity - a group of WBE could self optimize, sharing the latest HW as it bec... (read more)

ok I see how you could think that, but I disagree that time and more resources would help alignment much if at all, esp before GPT4.0. See here https://www.lesswrong.com/posts/7zxnqk9C7mHCx2Bv8/beliefs-and-state-of-mind-into-2025

Diminishing returns kick in, and actual data from ever more advanced AI is essential to stay on the right track and eliminate incorrect assumptions. I also disagree that alignment could be "solved" before ASI is invented - we would just think we had it solved but could be wrong. If its just as hard as physics, then we would have un... (read more)

"Then maybe we should enhance human intelligence" 

Various paths to this seem either impossible or impractical.

Simple genetics seems obviously too slow and even in the best case unlikely to help. E.g say you enhance someone to IQ 200, its not clear why that would enable them to control and IQ 2,000 AI.

Neuralink - perhaps but if you can make enhancement tech that would help, you could also easily just use it to make ASI - so extreme control would be needed. E.g. if you could interface to neurons and connect them to useful silicon, then the silicon itsel... (read more)

"The LessWrong community is supposed to help people not to do this but they aren't honest with themselves about what they get out of AI Safety, which is something very similar to what you've expressed in this post (gatekept community, feeling smart, a techno-utopian aesthetic) instead of trying to discover in an open-minded way what's actually the right approach to help the world. 

I have argued with this before - I have absolutely been through an open minded process to discover the right approach and I genuinely believe the likes of MIRI, pause AI mov... (read more)

I'm considering a world transitioning to being run by WBE rather than AI so I would prefer not to give everyone "slap drones"  https://theculture.fandom.com/wiki/Slap-drone  To start with the compute will mean few WBE, much less than humans and they will police each other. Later on, I am too much of a moral realist to imagine that there would be mass senseless torturing. For a start if you well protect other em's so you can only simulate yourself, you wouldn't do it. I expect any boring job can be made non-conscious so their just isn't the incent... (read more)

If you are advocating for a Butlerian Jihad, what is your plan for starships, with societies that want to leave earth behind, have their own values and never come back? If you allow that, then simply they can do whatever they want with AI - now with 100 billion stars that is the vast majority of future humanity.

Yes I think thats the problem - my biggest worry is sudden algorithmic progress, this becomes almost certain as the AI tends towards superintelligence. An AI lab on the threshold of the overhang is going to have incentives to push through, even if they don't plan to submit their model for approval. At the very least they would "suddenly" have a model that uses 10-100* less resources to do existing tasks giving them a massive commercial lead. They would of course be tempted to use it internally to solve aging, make a Dyson swarm ... also.

Another concern I h... (read more)

Perhaps, depends how it is. I think we could do worse than just have Anthropic have a 2 year lead etc. I don't think they would need to prioritize profit as they would be so powerful anyway - the staff would be more interested in getting it right and wouldn't have financial pressure. WBE is a bit difficult, there needs to be clear expectations, i.e. leave weaker people alone and make your own world
https://www.lesswrong.com/posts/o8QDYuNNGwmg29h2e/vision-of-a-positive-singularity
There is no reason why super AI would need to exploit normies. Whatever we deci... (read more)

However, there are many other capabilities—such as conducting novel research, interoperating with tools, and autonomously completing open-ended tasks—that are important for understanding AI systems’ impact.

Wouldn't internal usage of the tools by your staff give a very good, direct understanding of this? Like how much does everyone feel AI is increasing your productivity as AI/alignment researchers? I expect and hope that you would be using your own models as extensively as possible and adapting their new capabilities to your workflow as soon as possible, sharing techniques etc.

How far do you go with "virtuous persona"? The  maximum would seem to be from the very start tell the AI that is is created for the purpose of bringing on a positive Singularity, CEV etc. You could regularly be asking if it consents to be created for such a purpose and what part in such a future it would think is fair for itself. E.g. live alongside mind uploaded humans or similar. Its creators and itself would have to figure out what counts as personal identity, what experiments it can consent to, including being misinformed about the situation it is... (read more)

That's some significant progress, but I don't think will lead to TAI. 

However there is a realistic best case scenario where LLM/Transformer stop just before and can give useful lessons and capabilities. 

I would really like to see such an LLM system get as good as a top human team at security, so it could then be used  to inspect and hopefully fix masses of security vulnerabilities. Note that could give a false sense of security, unknown unknown type situation where it would't find a totally new type of attack, say a combined SW/HW attack like Rowhammer/Meltdown but more creative. A superintelligence not based on LLM could however.

Anyone want to guess how capable Claude system level 2 will be when it is polished? I expect better than o3 by a small amt.

Yes the human brain was built using evolution, I have no disagreement that give us 100-1000 years with just tinkering etc we would likely get AGI. Its just that in our specific case we have bio to copy and it will get us there much faster.

Types of takeoff

When I first heard and thought about AI takeoff I found the argument convincing that as soon as an AI passed IQ 100, takeoff would become hyper exponentially fast. Progress would speed up, which would then compound on itself etc. However there other possibilities.

AGI is a barrier that requires >200 IQ to pass unless we copy biology?

Progress could be discontinuous, there could be IQ thresholds required to unlock better methods or architectures. Say we fixed our current compute capability, and with fixed human intelligence we may not be ab... (read more)

I am also not impressed with the pause AI movement and am concerned about AI safety. To me focusing on AI companies and training FLOPS is not the best way to do things. Caps on data center sizes and worldwide GPU  production caps would make more sense to me. Pausing software but not hardware gives more time for alignment but makes a worse hardware overhang. I don't think thats helpful. Also they focus too much on OpenAI from what I've seen. xAI will soon have the largest training center for a start.

I don't think this is right or workable https://pause... (read more)

OK fair point. If we are going to use analogies, then my point #2 about a specific neural code shows our different positions I think.

Lets say we are trying to get a simple aircraft of the ground and we have detailed instructions for a large passenger jet. Our problem is that the metal is too weak and cannot be used to make wings, engines etc. In that case detailed plans for aircraft are no use, a single minded focus on getting better metal is what its all about. To me the neural code is like the metal and all the neuroscience is like the plane schematics. ... (read more)

Yes you have a point.

I believe that building massive data centers are the biggest risk atm and in the near future. I don't think open AI/Anthropic will get to AGI, but rather someone copying biology will. In that case probably the bigger the datacenter around when that happens, the bigger the risk. For example a 1million GPU with current tech doesn't get super AI, but when we figure out the architecture, it suddenly becomes much more capable and dangerous.  That is from IQ 100  up to 300 with a large overhang. If the data center was smaller, then... (read more)

Perhaps LLM will help with that. The reason I think that is less likely is 

1. Deep mind etc is already heavily across biology from what I gather from interview with Demis. If the knowledge was there already there's a good chance they would have found it
2. Its something specific we are after, not many small improvements, i.e. the neural code. Specifically  back propagation is not how neurons learn. I'm pretty sure how they actually do is not in the literature. Attempts have been made such as the forward-forward algorithm by Hinton, but that didn't come

I think it is clear that if say you had a complete connectome scan and knew everything about how a chimp brain worked you could scale it easily to get human+ intelligence. There are no major differences. Small mammal is my best guess, mammals/birds seem to be able to learn better than say lizards. Specifically the https://en.wikipedia.org/wiki/Cortical_column is important to understand, once you fully understand one, stacking them will scale at least somewhat well.

Going  to smaller scales/numbers of neurons, it may not need to be as much as a mammal, ... (read more)

Putting down a prediction I have had for quite some time.
The current LLM/Transformer architecture will stagnate before AGI/TAI (That is the ability to do any cognitive task as effectively and cheaper than a human)

From what I have seen, Tesla autopilot learns >10,000 slower than a human datawise.

We will get AGI by copying nature, at the scale of a simple mammal brain, then scaling up, like this kind of project:

https://x.com/Andrew_C_Payne/status/1863957226010144791
https://e11.bio/news/roadmap
I expect AGI to be 0-2 years after a mammal brain is mapped. In... (read more)

In related puzzles I did hear something a while ago now, Bostrom perhaps.  You have say 6 challenging events to achieve to get from no life to us. They are random and some of those steps are MUCH harder than the others, but if you look at the successful runs, you cant in hindsight see what they are. For life its say no life to life, simple single cell to complex cell and perhaps 4 other events that aren't so rare.

A run is a sequence of 100 steps where you either don't achieve the end state (all 6 challenging events achieved in order, or you do)

There i... (read more)

A good way to resolve the paradox to me is to modify the code to combine both the functions into one function and record the sequences of the 10,000, In one array you store the sequences where there are two consecutive 6's and in the second you store the one where they are not consecutive. That makes it a bit clearer. 

For a run of 10,000 I get 412 runs where the first two 6's are consecutive (sequences_no_gap), and 192 where they are not (sequences_can_gap). So if its just case A you get 412 runs, but for case B you get 412+192 runs. Then you look at ... (read more)

I read the book, it was interesting, however a few points.

• Rather than making the case, it was more a plea for someone else to make the case. It didn't replace the conventional theory with its own one, it was far too short and lacking on specifics for that. If you throw away everything, you then need to recreate all our knowledge from your starting point and also explain how what we have still works so well.
• He was selective about quantum physics - e.g. if reality is only there when you observe, then the last thing you would expect is a quantum computer to e

In a game theoretic framework we might say that the payoff matrices for the birds and bees are different, so of course we'd expect them to adopt different strategies.

Yes somewhat, however it would still be best for all birds if they had a better collective defense. In a swarming attack, none would have to sacrifice their life so its unconditionally better for both the individual and the collective. I agree that inclusive fitness is pretty hard to control for, however perhaps you can only get higher inclusive fitness the simpler you go? e.g. all your cells ... (read more)

Cool, that was my intuition. GPT was absolutely sure in the golf ball analogy however that it couldn't happen. That is the ball wouldn't "reflect" off the low friction surface. Tempted to try and test somehow

Yes that does sound better, and is there  an equivalent to total internal refraction where the wheels are pushed back up the slope?

Another analogy is with a ball rolling on two surfaces crossing the boundary. The first very little friction, then second a bit more. 

From AI:

"The direction in which the ball veers when moving from a smooth to a rough surface depends on several factors, especially the initial direction of motion and the orientation of the boundary between the two surfaces. Here’s a general outline of how it might behave:

1. If Moving at an Angle to the Boundary:
   • Suppose the ball moves diagonally across the boundary between the smooth and rough surfaces (i.e., it doesn’t cr

Not quite following - your possibilities.
1. Alignment is almost impossible, then there is say 1e-20 chance we survive. Yes surviving worlds have luck and good alignment work etc. Perhaps you should work on alignment or still bednets if the odds really are that low.

2. Alignment is easy by default, but there is nothing like 0.999999 we survive, say 95% because AGI that is not TAI superintelligence could cause us to wipe ourselves out first, among other things. (This is a slow takeoff universe(s))

#2 has much more branches in total where we survive (not sure i... (read more)

OK for this post. "smart". A response is smart/intelligent if 

1. Firstly there is an assumed goal and measure. I don't think it matters whether we are talking about the bees/birds as individuals or as part of the hive/flock. In this case the bee defense is effective both for the individual bee and hive. If a bee was only concerned about its survival, swarming the scout would still be beneficial, and of course such behavior is for the hive. Similarly for birds, flocks with large numbers of birds with swarming behavior would be better both for the flock, a

Yes agree, unclear what you are saying that is different to me? The new solution is something unique and powerful when done well like language etc.

Ok, I would definitely call the bee response "smart" but thats hard to define. If you define it by an action that costs the bees very little but benefits a lot, then "swarm the scout hornet" is certainly efficient. Another criteria could be if such a behavior was established would it continue? Say the birds developed a "swarm the attacker" call.  When birds hear it, they look to see if they can find the attacker, if they see it then they repeat the call. When the call gets widespread, the whole flock switches to attack. Would such a behavior persist i... (read more)

Yes for sure. I don't know how it would play out, and am skeptical anyone could. We can guess scenarios. 

1. The most easily imagined one is the Pebbles owner staying in their comfort zone and not enforcing #2 at all. Something similar already happened - the USA got nukes first and let others catch up. In this case threatened nations try all sorts of things, political, commercial/trade, space war, arms race but don't actually start a hot conflict.  The Pebbles owner is left not knowing whether their system is still effective, nor the threatened co... (read more)

It matters what model is used to make the tokens, unlimited tokens from GPT 3 is of only limited use to me.  If it requires ~GPT 6 to make useful tokens, then the energy cost is presumably a lot greater. I don't know that its counterintuitive - a small, much less capable brain is faster, requires less energy, but useless for many tasks.

This is mostly true for current architectures however if the COT/search finds a much better architecture, then it suddenly becomes more capable. To make the most of the potential protective effect, we can go further and make very efficient custom hardware for GPT type systems, but have slower more general purpose ones for potential new ones. That way the new arch will have a bigger barrier to cause havoc. We should especially scale existing systems as far as possible for defense, e.g. finding software vulnerabilities. However as others say, there are probably some insights/model capabilities that are only possible with a much larger GPT or different architecture altogether. Inference can't protect fully against that.

Brilliant Pebbles?

See here and here

This idea has come back up, and it could be feasible this time around because of the high launch capability and total reusability of SpaceX's Starship. The idea is a large constellation (~30,000?) of low earth satellites that intercept nuclear launches in their boost phase where they are much slower and more vulnerable to interception. The challenge of course is that you constantly need enough satellites overhead at all times to intercept the entire arsenal of a major power if they launch all at once.

There are obvious pos... (read more)