No - Coax cables are enormous in radius (EM wavelengths), and do not achieve much better than 1 eV / nm in practice. In the same waveguide radius you can you just remove the copper filler and go pure optical and then get significantly below 1 eV/nm anyway - so why even mention coax?

The only thing that was 'debunked' was in a tangent conversation that had no bearing on the main point (about nanoscale wire interconnect smaller than EM wavelength - which is irreversible and consumes close to 1 eV/nm in both brains and computers), and it was just my initial conception that coax cables could be modeled in simplification as relays like RC interconnect.

There are many complex tradeoffs between size, speed, energy, etc. Reversible and irreversible comms occupy different regions of that pareto surface. Reversible communication is isomorphic to transmitting particles - in practice always photons - and requires complex/large transmitter/receivers and photon sized waveguides etc. Irreversible communication is isomorphic to domino-based computing, and has the advantage - and cost - of full error correction/erasure at every cycle, and easier to guide down narrow and complex paths.

Sorry, explain again why floods of neurotransmitter molecules bopping around are ideally thermodynamically efficient? You're assuming that they're trying to do multiplication out to 8-bit precision using analog quantities? Why suppose the 8-bit precision?

I'm not assuming that, but its nonetheless useful as a benchmark for comparison. It helps illustrate that 1e5 eV is really not much - it just allows a single 8-bit analog mult for example.

Earlier in the thread I said:

Now most synapses are probably smaller/cheaper than 8-bit equiv, but most of the energy cost involved is in pushing data down irreversible dissipative wires (just as true in the brain as it is in a GPU). Now add in the additional costs of synaptic adjustment machinery for learning, cell maintenance tax, dendritic computation, etc

The synapse is clearly doing something somewhat more complex than just analog multiplication.

And in terms of communication costs (which are paid at the synaptic junction for the synapse -> dendrite -> soma path), that 1e5 eV is only enough to carry a reliable 1 bit signal only about ~100mm (1e5 nm) distance through irreversible nano/micro scale wires (the wire bit energy for axons/dendrites and modern cmos is about the same).

Reversible interconnect is much more complex - requires communicating through fully isolated particles over the wire distance, which is obviously much more practical for photons for various reasons, but they are very large etc. Many complex tradeoffs.

Imprecisely multiplying two analog numbers should not require 10^5 times the minimum bit energy in a well-designed computer.

Much depends on your exact definition of 'imprecisely'. But if we assume exactly 8-bit equivalent SNR, as I was using above, then you can lookup this question in the research literature and/or ask an LLM and the standard answer is in fact close to ~1e5 eV.

This multiplication op is a masking operation and not inherently reversible so it erases/destroys about 1/2 of the energy of the photonic input signal (100% if you multiply by 0, etc). So the min energy boils down to that required to represent an 8-bit number reliably as an analog signal (so for example you could convert a digital 8-bit signal to analog and back to digital losslessly all at the same standard sufficient 1eV reliability).

Analog signals effectively represent numbers as the 1st moment of a binomial distribution over carrier particles, and the information content is basically the entropy of a binomial over 1eV carriers which is ~0.5 log2(N) and thus N ~ 2^(2b) quanta for b bits of precision.

The energy to represent an analog signal doesn't depend much on the medium - whether you are using photons or electrons/ions. The advantage of the electronic medium is the much smaller practical device dimensions possible when using much heavier/denser particles as bit carriers: 1eV photons are micrometer scale, much larger than the smallest synapses/transistors/biodevices. The obvious advantage of photons is their much higher transmission speed: thus they are used for longer range interconnect (but mostly only for distances larger than the brain radius).

Not really - its vector matrix multiplication, not matrix matrix mult.

This is the answer, but

Essentially, the brain is massively underclocked because of design-space restrictions imposed by biology and evolution

The main restriction is power efficiency: the brain provides a great deal of intelligence for a budget of only ~20 watts. Spreading out that power budget over a very wide memory operating at very slow speed just turns out to be the most power efficient design (vs a very small memory running at very high speed), because memory > time.

Would have made much more sense (visually and otherwise) to show graphs in log space. Example: https://www.openphilanthropy.org/research/modeling-the-human-trajectory/

The effectiveness of weight sharing (and parameter compression in general) diminishes as you move the domain from physics (simple rules/patterns tiled over all of space/time) up to language/knowledge (downstream facts/knowledge that are far too costly to rederive from simulation).

BNNs cant really take advantage of weight sharing so much, so ANNs that are closer to physics should be much smaller parameter wise, for the same compute and capability. Which is what we observer for lower level sensor/motor modalities.

The single factor prime causative factor driving the explosive growth in AI demand/revenue is and always has been the exponential reduction in $/flop via moore's law, which simply is jevon's paradox manifested. With more compute everything is increasingly easy and obvious; even idiots can create AGI with enough compute.

Abilities/intelligence come almost entirely from pretraining, so all the situation awareness and scheming capability that current (and future similar) frontier models possess is thus also mostly present in the base model.

Yes, but for scheming, we care about whether the AI can self-locate itself as an AI using its knowledge. The fact that (at a minimum) sampling from the system is required for it to self-locate as an AI might make a big difference here.

So if your 'yes' above is agreeing that capabilities - including scheming - come mostly from pretraining, then I don't see how relevant it is whether or not that ability is actually used/executed much in pretraining, as the models we care about will go through post-training and I doubt you are arguing post-training will reliably remove scheming.

I also think it seems probably very hard to train a system capable of obsoleting top human experts which doesn't understand that it is an AI even if you're willing to take a big competitiveness hit.

Indeed but that is entirely the point - by construction!

Conceptually we have a recipe R (arch, algorithms, compute, etc), and a training dataset which we can parameterize by time cutoff T. Our objective (for safety research) is not to train a final agent, but instead to find a safe/good R with minimal capability penalty. All important results we care about vary with R independently of T, but competitiveness/dangerousness does vary strongly with T.

Take the same R but vary the time cutoff T of the training dataset: the dangerousness of the AI will depend heavily on T, but not the relative effectiveness of various configurations of R. That is simply a restatement of the ideal requirements for a safe experimental regime. Models/algos that work well for T of 1950 will also work for T of 2020 etc.

Training processes with varying (apparent) situational awareness

• 1:2.5 The AI seemingly isn't aware it is an AI except for a small fraction of training which isn't where much of the capabilities are coming from. For instance, the system is pretrained on next token prediction, our evidence strongly indicates that the system doesn't know it is an AI when doing next token prediction (which likely requires being confident that it isn't internally doing a substantial amount of general-purpose thinking about what to think about), and there is only a small RL process which isn't where much of the capabilities are coming from.

Abilities/intelligence come almost entirely from pretraining, so all the situation awareness and scheming capability that current (and future similar) frontier models possess is thus also mostly present in the base model.  The fact that you need to prompt them to summon out a situationally aware scheming agent doesn't seem like much of a barrier, and indeed strong frontier base models are so obviously misaligned/jail-breakable/dangerous that releasing them to the public is PR-harmful enough to motivate RLHF post training purely for selfish profit-motives.

> This implies that restricting when AIs become (saliently) aware that they are an AI could be a promising intervention, to the extent this is possible without greatly reducing competitiveness.

Who cares if it greatly reduces competitiveness in experimental training runs?

We need to figure out how to align superhuman models - models trained with > 1e25 efficient flops on the current internet/knowledge, which requires experimental iteration.  We probably won't get multiple iteration attempts for aligning SI 'in prod', so we need to iterate in simulation (what you now call 'model organisms').

We need to find alignment training methods that work even when the agent has superhuman intelligence/inference.  But 'superhuman' hear is relative - measured against our capabilities.  The straightforward easy way to accomplish this is training agents in simulations with much earlier knowledge cutoff dates, which isn't theoretically hard - just requires constructing augmented historical training datasets.  So you could train on a 10T+ token dataset of human writings/thoughts with cutoff 2010, or 1950, or 1700, etc.  These base models wouldn't be capable of simulating/summoning realistic situationally aware agents, their RL derived agents wouldn't be situationally sim-aware either, etc.