All of Andrew_Clough's Comments + Replies

Mechanisms like attention only seem analogous to a human's sensory memory.  Reasoning models have something like a working memory but even then I think we'd need something in embedding space to constitute a real working memory analog.  And having something like a short term memory could could help Claude avoid repeating the same mistakes.

This is, in some sense, very scary because when someone figures out how to train agent reasoning in embedded space there might be a very dramatic discontinuity in how well LLMs can act as agents.

The facts very strongly suggest that the board is not a monolithic entity.  Its inability to tell a sensible story about the reasons for Sam's firing might be due to such a single comprehensible story not existing but different board members having different motives that let them agree on the firing initially but ultimately not on a story that they could jointly endorse.  

Even in conventional programming it seems easier to ask about a famous person's parents than vice versa.  A name is an ambiguous pointer so if someone says "Tom Cruise" you'd generally just look for the most famous person of all the people who have that name and answer the question for that individual.  But to do the reverse you have to figure out that no "Mary Lee Pfeiffer" is famous enough on their own to be the target of the search and then go on to search through all the children of all the people named "Mary Lee Pfeiffer", notice that one is really famous, and then answer with that result.

I think that, in particular, protesting Meta releasing their models to the public is a lot less likely to go well than protesting, say, OpenAI developing their models.  Releasing models to the public seems virtuous on its face both to the general public and to many technologists.  Protesting that is going to draw attention to that specifically and so tend to paint the developers of more advanced models in a comparatively better light and their opponents in a comparatively worse light compared. 

Wouldn't the question be why epidemiologists don't study this?  As far as I'm aware virologists don't study the transmission of viruses between people any detail, or at least this would be an out-of-paradigm research project for any virologist investigating it.  I think the real question would be why epidemiologists don't investigate it.

I'm especially worried about the US going forward.  In general presidential republics don't have a good track record.  Luckily the US has benefited from strong notions of democratic legitimacy over most of its history.  Also, except at the very beginning, for a brief period around the birth of the Republican party, and recently with the introduction of open primaries US political parties haven't had strong ideological partisanship but were mostly very ideologically mixed with most partisanship being over spoils rather than ideology.  

In... (read more)

My impression has been that we mostly just have David Fravor's word that most of those independant lines of evidence exist.  Have there actually been interviews with, e.g., the ship radar operators where they describe seeing things that were only consistent with the UFO story?

I guess a better way of putting that is that R0 is fixed for a particular population but humans are composed of many different populations, just like there are other populations of different species a virus can also infect which might have their own R0s as well.

It'll tend to change with things population, social conventions, etc.  For the herd animal populations it was originally applied to you can pretty much ignore all of that but not for humans.  Especially for things like coronaviruses with a high k where R0 is driven by the fat tail of the distribution.  In a small village where most bat/human coronavirus crossovers tend to happen the village size limits how large a superspreader event can be.  Not so in a city.  And then you have things like Ebola spread being partially driven by funereal customs.

LLMs currently seem to lack any equivalent of the sort of global workspace that comprises our conscious minds, letting us silently plot many steps ahead and serialize our experiences into memories which preserve the most important parts of what we experience while dropping the rest.  I worry that this won't actually be all that hard to add with maybe a single conceptual breakthrough, leading to a stupendous augmentation of what is effectively the LLM's unconscious mind with a consciousness[1] and catapulting it from not being fully general to a genuine superintelligence very quickly.

[1] In the sense that you're unconscious during NREM sleep or unconscious of subliminal stimuli, not anything to do with qualia.

That's about where I am too.

How would horizontal transfer even work when you go from a prokariot with millions of base pairs of genetic material to a eukariot with billions of base pairs and all the complexity of RNA splicing?

It also applies to the stock market where buying an index fund that just invests in everything leads to fairly regular positive returns.

That's only for raw materials.  Actual production seems to have involved a fair amount of skilled labor per dose which probably dominates the $2 figure.  A commercial vaccine can automate that but then you've got lots of development costs as well as logistics and compliance to worry about.  I wouldn't expect that this vaccine is especially cheap.

One thing that makes this disease hard to reason about is the high k.  That is, most people infect a very small number of people but some people infect large numbers. When case numbers are less than a thousand we should't expect a strong signal regardless of the underlying infectivity of the new strain.  This is evidence against it being significant but I fear its not very strong evidence.

For me an important factor is that we have three different pints of data that suggest the new strain is more infectious.  First, it's rapidly replacing the existing strain in areas where it is preset.  Second, those areas are seeing surges of infections that don't occur in other areas.  Third, it seems like individuals infected with the new strain have 3 or 4 times the viral load of individuals with the previous strain - which would neatly explain higher transmissiblility. I'm going with an 85% chance that this is genuinely more transmissibl... (read more)

Do we have any good sense of the extent to which researchers from the Wuhan Institute of Virology are flying out across China to investigate novel pathogens or sites where novel pathogens might emerge?

On TWiV 595 they did an interview with a doctor who said he'd been able to get the survival rate of intubated patients up to 50% by using proning, though I don't recall them going into the details.

From my understanding of how the learned immune response works an infection has to grow to a certain size before it really starts to kick in. That would tend to suggest that, if a dose is enough to cause an infection in the first place, danger will remain roughly constant up to some inflection point before starting to become less dangerous.

Of course as doses go down the chance that the innate immune system cleans it up or none of the viruses manage to find an ACE2 receptor goes up. But in those cases you won't be training the immune system.

The people on This Week in Virology seemed convinced that the spike protein wasn't anything that had previously been seen and wasn't anything a human would design if they were working on creating a new virus.

SARS-Covid-2 doesn't look at all like a biological weapon. If they were dong experiments on trying to design a novel spike I don't think they'd do it in such an otherwise dangerous virus.

I can imagine that this virus infected someone in China, was brought to the lab for analysis then escape from the lab into Wuhan but that&a... (read more)

I think that if you consider that the chance of a threat to cause a given amount of disutility being valid is a function of the amount of disutility then the problem mostly goes away. That is, in my experience any threat to cause me X units of disutility where X is beyond some threshold is less than 1/10 as credible as a threat to cause me 1 unit of disutility. If someone threatened to kill another person unless I gave them $5000 I would be worried. If they threatened to kill 10 poeple I would be very slightly less worried. If they threatened to kill ... (read more)

I expect that many people who grew up to be scientists and mathematicians attempted to create famous proofs when they were young, but I also expect that for many engineers such as myself our youthful folly went more along the direction of perpetual motion machines. I'd actually like to see some research on what the correlations really are.