All of jacob_cannell's Comments + Replies

The effectiveness of weight sharing (and parameter compression in general) diminishes as you move the domain from physics (simple rules/patterns tiled over all of space/time) up to language/knowledge (downstream facts/knowledge that are far too costly to rederive from simulation).

BNNs cant really take advantage of weight sharing so much, so ANNs that are closer to physics should be much smaller parameter wise, for the same compute and capability. Which is what we observer for lower level sensor/motor modalities.

The single factor prime causative factor driving the explosive growth in AI demand/revenue is and always has been the exponential reduction in $/flop via moore's law, which simply is jevon's paradox manifested. With more compute everything is increasingly easy and obvious; even idiots can create AGI with enough compute.

Abilities/intelligence come almost entirely from pretraining, so all the situation awareness and scheming capability that current (and future similar) frontier models possess is thus also mostly present in the base model.

Yes, but for scheming, we care about whether the AI can self-locate itself as an AI using its knowledge. The fact that (at a minimum) sampling from the system is required for it to self-locate as an AI might make a big difference here.

So if your 'yes' above is agreeing that capabilities - including scheming - come mostly from pretr... (read more)

Training processes with varying (apparent) situational awareness

• 1:2.5 The AI seemingly isn't aware it is an AI except for a small fraction of training which isn't where much of the capabilities are coming from. For instance, the system is pretrained on next token prediction, our evidence strongly indicates that the system doesn't know it is an AI when doing next token prediction (which likely requires being confident that it isn't internally doing a substantial amount of general-purpose thinking about what to think about), and there is only a small RL proc

Input vs output tokens are both unique per agent history (prompt + output), so that differentiation doesn't matter for my core argument about the RAM constraint. If you have a model which needs 1TB of KV cache, and you aren't magically sharing that significantly between instances, then you'll need at least 1000 * 1TB of RAM to run 1000 inferences in parallel.

The 3x - 10x cost ratio model providers charge is an economic observation that tells us something about the current cost vs utility tradeoffs, but it's much complicated by oversimpliciation of the cur... (read more)

Not for transformers, for which training and inference are fundamentally different.

Transformer training parallelizes over time, but that isn't feasible for inference. So transformer inference backends have to parallelize over batch/space, just like RNNs, which is enormously less efficient in RAM and RAM bandwidth use.

So if you had a large attention model that uses say 1TB of KV cache (fast weights) and 1TB of slow weights, transformer training can often run near full efficiency, flop limited, parallelizing over time.

But similar full efficient transformer ... (read more)

Due to practical reasons, the compute requirements for training LLMs is several orders of magnitude larger than what is required for running a single inference instance.  In particular, a single NVIDIA H100 GPU can run inference at a throughput of about 2000 tokens/s, while Meta trained Llama3 70B on a GPU cluster^[1] of about 24,000 GPUs.  Assuming we require a performance of 40 tokens/s, the training cluster can run $\frac{2000}{40}\times 24000=1,200,000$ concurrent instances of the resulting 70B model.

I agree direction-ally with your headline, bu... (read more)

How is that even remotely relevant? Humans and AIs learn the same way, via language - and its not like this learning process fails just because language undersamples thoughts.

As the article points out, shared biological needs do not much deter the bear or chimpanzee from killing you. An AI could be perfectly human - the very opposite of alien - and far more dangerous than Hitler or Dhamer.

The article is well written but dangerously wrong in its core point. AI will be far more human than alien. But alignment/altruism is mostly orthogonal to human vs alien.

We are definitely not training AIs on human thoughts because language is an expression of thought, not thought itself.

Even if training on language was not equivalent to training on thoughts, that would also apply to humans.

But it also seems false in the same way that "we are definitely not training AI's on reality because image files are compressed sampled expressions of images, not reality itself" is false.

Approximate bayesian inference (ie DL) can infer the structure of a function through its outputs; the structure of the 3D world through images; and thoughts through language.

Premise 1: AGIs would be like a second advanced species on earth, more powerful than humans.

Distinct alien species arise only from distinct separated evolutionary histories. Your example of the aliens from Arrival are indeed a good (hypothetical) example of truly alien minds resulting from a completely independent evolutionary history on an alien world. Any commonalities between us and them would be solely the result of convergent evolutionary features. They would have completely different languages, cultures, etc.

AI is not alien at all, as we litera... (read more)

I also not sure of the relevance and not following the thread fully, but the summary of that experiment is that it takes some time (measured in nights of sleep which are rough equivalent of big batch training updates) for the newly sighted to develop vision, but less time than infants - presumably because the newly sighted already have full functioning sensor inference world models in another modality that can speed up learning through dense top down priors.

But its way way more than "grok it really fast with just a few examples" - training their new visual systems still takes non-trivial training data & time

I suspect that much of the appeal of shard theory is working through detailed explanations of model-free RL with general value function approximation for people who mostly think of AI in terms of planning/search/consequentialism.

But if you already come from a model-free RL value approx perspective, shard theory seems more natural.

Moment to moment decisions are made based on value-function bids, with little to no direct connection to reward or terminal values. The 'shards' are just what learned value-function approximating subcircuits look like in gory det... (read more)

TSMC 4N is a little over 1e10 transistors/cm^2 for GPUs and roughly 5e^-18 J switch energy assuming dense activity (little dark silicon). The practical transistor density limit with minimal few electron transistors is somewhere around ~5e11 trans/cm^2, but the minimal viable high speed switching energy is around ~2e^-18J. So there is another 1 to 2 OOM further density scaling, but less room for further switching energy reduction. Thus scaling past this point increasingly involves dark silicon or complex expensive cooling and thus diminishing returns eit... (read more)

Part of the issue is my post/comment was about moore's law (transistor density for mass produced nodes), which is a major input to but distinct from flops/$. As I mentioned somewhere, there is still some free optimization energy in extracting more flops/$ at the circuit level even if moore's law ends. Moore's law is very specifically about fab efficiency as measured in transistors/cm^2 for large chip runs - not the flops/$ habyrka wanted to bet on. Even when moore's law is over, I expect some continued progress in flops/$.

All that being said, nvidia's n... (read more)

I don't know who first said it, but the popular saying "Computer vision is the inverse of computer graphics" encompasses much of this viewpoint.

Computer graphics is the study/art of the approximation theory you mention and fairly well developed & understood in terms of how to best simulate worlds & observations in real-time from the perspective of an observer. But of course traditional graphics uses human-designed world models and algorithms.

Diffusion models provide a general framework for learning a generative model in the other direction - in pa... (read more)

Even if there is no acceptable way to share the data semi-anonymously outside of match group, the arguments for prediction markets still apply within match group. A well designed prediction market would still be a better way to distribute internal resources and rewards amongst competing data science teams within match group.

But I'm skeptical that the value of match group's private data is dominant even in the fully private data scenario. People who actually match and meetup with another user will probably have important inside view information inaccessib... (read more)

Certainly mood disorders like bipolar,depression,mania can have multiple causes - for examle simply doing too much dopaminergic simulants (cocaine, meth etc) can cause mania directly.

But the modern increased prevalence of mood disorders is best explained by a modern divergence from conditions in the ancestral environment, and sleep disorder due to electric lighting disrupting circadian rhythms is a good fit to the evidence.

The evidence for each of my main points is fairly substantial and now mainstream, the only part which isn't mainstream (yet) is the spe... (read more)

From my own study of mood disorders I generally agree with your valence theory of depression/mania.

However I believe the primary cause (at least for most people today) is disrupted sleep architecture.

To a first order approximation, the brain accumulates batch episodic training data during the day through indexing in the hippocampus (which is similar-ish to upper cortex, but more especially adapted to medium term memory & indexing). The brain's main episodic replay training then occurs during sleep, with alternation of several key phases (REM and sever... (read more)

Sure, but how often do the colonized end up better off for it, especially via trying to employ clever play-both-sides strategies?

I didn't say the colonized generally ended up better off, but outcomes did vary greatly. Just in the US the cherokees faired much better than say the Susquehannock and Pequot, and if you dig into that history it seems pretty likely that decisions on which colonizer(s) to ally with (british, french, dutch, later american etc) were important, even if not "clever play-both-sides strategies" (although I'd be surprised if that wasn't also tried somewhere at least once)

An idea sometimes floated around is to play them off against each other. If they're misaligned from humanity, they're likely mutually misaligned as well. We could put them in game-theoretic situations in which they're incentivized to defect against each other and instead cooperate with humans.

You are arguing against a strawman. The optimistic game-theoretic argument you should focus on is:

Misaligned AIs are - almost by definition - instrumental selfish power seeking agents (with random long term goals) and thus intrinsically misaligned with each other.... (read more)

Of course a massive advance is possible, but mostly just in terms of raw speed. The brain seems reasonably close to pareto efficiency in intelligence per watt for irreversible computers, but in the next decade or so I expect we'll close that gap as we move into more 'neuromorphic' or PIM computing (computation closer to memory). If we used the ~1e16w solar energy potential of just the Saraha desert that would support a population of trillions of brain-scale AIs or uploads running 1000x real-time.

especially as our NN can use stuff such as backprop,

The... (read more)

The paper which more directly supports the "made them smarter" claim seems to be this. I did somewhat anticipate this - "not much special about the primate brain other than ..", but was not previously aware of this particular line of research and certainly would not have predicted their claimed outcome as the most likely vs various obvious alternatives. Upvoted for the interesting link.

Specifically I would not have predicted that the graft of human glial cells would have simultaneously both 1.) outcompeted the native mouse glial cells, and 2.) resulted i... (read more)

Suffering, disease and mortality all have a common primary cause - our current substrate dependence. Transcending to a substrate-independent existence (ex uploading) also enables living more awesomely. Immortality without transcendence would indeed be impoverished in comparison.

Like, even if they 'inherit our culture' it could be a "Disneyland with no children"

My point was that even assuming our mind children are fully conscious 'moral patients', it's a consolation prize if the future can not help biological humans.

The AIs most capable of steering the future will naturally tend to have long planning horizons (low discount rates), and thus will tend to seek power(optionality). But this is just as true of fully aligned agents! In fact the optimal plans of aligned and unaligned agents will probably converge for a while - they will take the same/similar initial steps (this is just a straightforward result of instrumental convergence to empowerment). So we may not be able to distinguish between the two, they both will say and appear to do all the right things. Thus it... (read more)

But on your model, what is the universal learning machine learning, at runtime? ..

On my model, one of the things it is learning is cognitive algorithms. And different classes of training setups + scale + training data result in it learning different cognitive algorithms; algorithms that can implement qualitatively different functionality.

Yes.

And my claim is that some setups let the learning system learn a (holistic) general-intelligence algorithm.

I consider a ULM to already encompass general/universal intelligence in the sense that a properly sca... (read more)

My argument for the sharp discontinuity routes through the binary nature of general intelligence + an agency overhang, both of which could be hypothesized via non-evolution-based routes. Considerations about brain efficiency or Moore's law don't enter into it.

You claim later to agree with ULM (learning from scratch) over evolved-modularity, but the paragraph above and statements like this in your link:

The homo sapiens sapiens spent thousands of years hunter-gathering before starting up civilization, even after achieving modern brain size.

It would s

No, and that's a reasonable ask.

To a first approximation my futurism is time acceleration; so the risks are the typical risks sans AI, but the timescale is hyperexponential ala roodman. Even a more gradual takeoff would imply more risk to global stability on faster timescales than anything we've experience in history; the wrong AGI race winners could create various dystopias.

[Scaling law theories]

I'm not aware of these -- do you have any references?

Sure: here's a few: quantization model, scaling laws from the data manifold, and a statistical model.

True but misleading? Isn't the brain's "architectural prior" a heckuva lot more complex than the things used in DL?

The full specification of the DL system includes the microde, OS, etc. Likewise much of the brain complexity is in the smaller 'oldbrain' structures that are the equivalent of a base robot OS. The architectural prior I speak of is the complexity on top of that,... (read more)

Sure - I'm not saying no improvement is possible. I expect that the enhancements from adult gene editing should encompass most all of the brain tweaks you can get from drugs/diet. But those interventions will not convert an average brain into an Einstein.

The brain - or more specifically the brains of very intelligent people - are already very efficient, so I'm also just skeptical in general that there are many remaining small tweaks that take you past the current "very intelligent". Biological brains beyond the human limit are of course possible, but pr... (read more)

ANNs and BNNs operate on the same core principles; the scaling laws apply to both and IQ in either is a mostly function of net effective training compute and data quality.

How do you know this?

From study of DL and neuroscience of course. I've also written on this for LW in some reasonably well known posts: starting with The Brain as a Universal Learning Machine, and continuing in Brain Efficiency, and AI Timelines specifically see the Cultural Scaling Criticality section on the source of human intelligence, or the DL section of simboxes. Or you co... (read more)

We can roughly bin brain tissue into 3 developmental states:

1. juvenile: macro structure formation - brain expanding, neural tissue morphogenesis, migration, etc

2. maturing: micro synaptic structure formation, irreversible pruning and myelination

3. mature: fully myelinated, limited remaining plasticity

Maturation proceeds inside out with the regions closest to the world (lower sensory/motor) maturing first, proceeding up the processing hierarchy, and ending with maturation of the highest levels (some prefrontal, associative etc) around age ~20.

The human ... (read more)

Current AI is less sample efficient, but that is mostly irrelevant as the effective speed is 1000x to 10000x greater.

By the time current human infants finish ~30 year biological training, we'll by long past AGI and approaching singularity (in hyperexpoential models).

heritability of IQ increases with age (up until age 20, at least)

Straight forward result of how the brain learns. Cortical/cerebellar modules start out empty and mature inwards out - starting with the lowest sensory/motor levels closest to the world and proceeding up the hierarchy ending with the highest/deepest modules like prefrontal and associative cortex. Maturation is physically irreversible as it involves pruning most long-range connections and myelinating&strengthening the select few survivors. Your intelligence potential is constrained pr... (read more)

It would matter in a world without AI, but that is not the world we live in. Yes if you condition on some indefinite AI pause or something then perhaps, but that seems extremely unlikely. It takes about 30 years to train a new brain - so the next generation of humans won't reach their prime until around the singularity, long after AGI.

Though I do agree that a person with the genes of a genius for 2 years

Most genius is determined prenatally and during 'training' when cortex/cerebellum modules irreversibly mature, just as the capabilities of GPT4 are determined by the initial code and the training run.

Too slow too matter now due to the slow speed of neurons and bio learning combined with where we are in AI.

It does not. Despite the title of that section it is focused on adult expression factors. The post in general lacks a realistic mechanistic model of how tweaking genes affects intelligence.

genes are likely to have an effect if edited in adults: the size of the effect of a given gene at any given time is likely proportional to its level of expression

Is similar to expecting that a tweak to the hyperparams (learning rate) etc of trained GPT4 can boost its IQ (yes LLMs have their IQ or g factor). Most all variables that affect adult/trained performance do... (read more)

ANNs and BNNs operate on the same core principles; the scaling laws apply to both and IQ in either is a mostly function of net effective training compute and data quality. Genes determine a brain's architectural prior just as a small amount of python code determines an ANN's architectural prior, but the capabilities come only from scaling with compute and data (quantity and quality).

So you absolutely can not take datasets of gene-IQ correlations and assume those correlations would somehow transfer to gene interventions on adults (post training in DL lingo... (read more)

If it only requires a simple hack to existing public SOTA, many others will have already thought of said hack and you won't have any additional edge. Taboo superintelligence and think through more specifically what is actually required to outcompete the rest of the world.

Progress in DL is completely smooth as it is driven mostly from hardware and enormous number of compute-dependent small innovations (yes transformers were a small innovation on top of contemporary alternatives such as memory networks, NTMs etc and quite predictable in advance ).

Its easy to say something is "not that hard", but ridiculous to claim that when the something is build an AI that takes over the world. The hard part is building something more intelligent/capable than humanity, not anything else conditioned on that first step.

Robust to the "trusting trust" problem (i.e. the issue of "how do you know that the source code you received is what the other agent is actually running"). ''

This is the crux really, and I'm surprised that many LW's seem to believe the 'robust cooperation' research actually works sans a practical solution to 'trusting trust' (which I suspect doesn't actually exist), but in that sense it's in good company (diamonoid nanotech, rapid takeoff, etc)

It's not that hard to build an AI that kills everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should increase your P(doom) / shorten your timelines.

It's not that hard to build an AI that saves everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should decrease your P(doom) / shorten your timelines.

They do a value-handshake and kill everyone together.

Any two AIs unaligned with hum... (read more)

Corporations only exist within a legal governance infrastructure that permits incorporation and shapes externalities into internalities. Without such infrastructure you have warring tribes/gangs, not corporations.

The ways in which this shareholder value maximization has already seriously damaged the world and compromised the quality of human life are myriad and easily observable: pollution, climate change, and other such externalities. Companies' disregard for human suffering further enhances this comparison.

This is the naive leftist/marxist take. In... (read more)

Surely they must mean something like extropy/complexity? Maximizing disorder doesn't seem to fit their vibe.

Yes, but: whales and elephants have brains several times the size of humans, and they're yet to build an industrial civilization.

Size/capacity isn't all, but In terms of the capacity which actually matters (synaptic count, and upper cortical neuron count) - from what I recall elephants are at great ape cortical capacity, not human capacity. A few specific species of whales may be at or above human cortical neuron capacity but synaptic density was still somewhat unresolved last I looked.

Then, once a certain compute threshold was reached, it took a sha

Indeed, that's basically what happened in the human case: the distributed optimization process of evolution searched over training architectures, and eventually stumbled upon one that was able to bootstrap itself into taking off.

Actually I think the evidence is fairly conclusive that the human brain is a standard primate brain with the only change being nearly a few compute scale dials increased (the number of distinct gene changes is tiny - something like 12 from what I recall). There is really nothing special about the human brain other than 1.) 3x l... (read more)

I'd say this still applies even to non-LLM architectures like RL, which is the important part, but Jacob Cannell and 1a3orn will have to clarify.

We've basically known how to create AGI for at least a decade. AIXI outlines the 3 main components: a predictive world model, a planning engine, and a critic. The brain also clearly has these 3 main components, and even somewhat cleanly separated into modules - that's been clear for a while.

Transformers LLMs are pretty much exactly the type of generic minimal ULM arch I was pointing at in that post (I obvious... (read more)

I don't view ASI as substantially different than an upload economy.

I'm very confused about why you think that.

You ignored most of my explanation so I'll reiterate a bit differently. But first taboo the ASI fantasy.

• any good post-AGI future is one with uploading - humans will want this
• uploads will be very similar to AI, and become moreso as they transcend
• the resulting upload economy is one of many agents with different values
• the organizational structure of any pareto optimal multi-agent system is necessarily market-like
• it is a provable fact th