Quick Takes

Did anyone else just get a "lesswrong.com is asking to access other apps and services on your device" message from their browser when they opened the site? Anyone know what that's about?

A model to track: * AI gets increasingly good at X, so good that strong pressures arise to increasingly automate it, even if AI is still quite bad at some subtle but very important aspects of X. * More and more X-related work/activity gets AI-automated, including the parts where AI is still bad or fails frequently and non-gracefully. * * This is partly driven by the fact that it's just ~psychologically hard for a human to exert effortful X-related cognition when AI would be more efficient at 99% of it. * In cases where being good at [those aspects of X AI is still bad at][1] is critical, the quality of work deteriorates relative to the level before intense AI automation. 1. ^ I guess we could call them reverse salients?

I run a curated Discord for high agency people with Long COVID/ME (myalgic encephalomyelitis). The group includes tech founders, researchers, rationalists/EAs, etc. The focus is on troubleshooting each other's conditions actively, as well as creating a body of knowledge to bring back to the wider community in the form of writing, education, projects, companies, etc. Some know me as Liface, others as Liam Rosen - I have been in the Rationality community for over 10 years, previously in the Bay, now in New York City, and am the main moderator of r/slatestarcodex. I am a tech CEO whose former life abruptly ended last year. I am now bedbound with severe Long COVID/ME, and am dedicating 100% of my time and effort to helping myself and both everyone else around me figure out these complex conditions. If you, or anyone you know might be a good fit for this group, feel free to message me here, or email me at liamrosen@gmail.com.

I often see AIs [1] exhibit a (misaligned) drive to stop early: as in they make up not-very-sensible reasons why it's a good idea to stop working, disobey instructions to keep working unless some condition is met, and don't do a very thorough job (e.g. skipping significant chunks of the task) despite instructions that strongly say not to do this. I don't think they are "consciously" or saliently aware of this misalignment (but if you ask them, they'll often notice the behavior isn't desirable). [2] I see this most often in large, difficult tasks, especially if you don't decompose the task into smaller pieces and run one piece at a time. I think it's somewhat more common in tasks that aren't very precisely specified, but I also see this behavior in cases where the AI is just tasked with optimizing a metric (that it can easily evaluate), especially if the AI has (from its perspective) not made much progress for a little bit [3] . Why might this emerge? * Length penalties (or time penalties, or cost penalties) incentivize exiting/stopping earlier and this ended up converting into a misaligned/unreasonable drive (that overrides instruction following in practice). Length penalties wouldn't be an issue if the length penalties were aligned with the trade-offs I want the AI to follow [4] but in practice I'm often trying to get AIs to do long running autonomous tasks and to continue even when this yields relatively small gains (in metrics or apparent progress). I think in tasks close to the tasks I'm using AIs for it likely would have performed better in RL to stop earlier than I actually want (because training on shorter tasks is more competitive and often the additional progress I want the AI to make is harder to measure while the length penalty will bite). * The AI might have learned to stop before running out of context or compaction in training because compaction is bad for task completion. Thus, it learned to make up excuses to stop when nearing the point where co

if you want to know whether a given crypto project is a scam, there's a very reliable heuristic, which is to assume the answer is yes. but suppose you want to actually not discard the one occasional good idea. there is one very simple rule to determine if a crypto idea makes any sense whatsoever. crypto makes sense as a solution if you can't trust anyone. not the government, not some institution that is bound by laws, not a majority vote of some set of reasonable people. (but also you think civilization won't completely collapse, e.g the internet will keep working). removing the need to trust people/institutions is the only advantage of crypto. for this one advantage, we pay dearly in terms of everything else. crypto will always be slower, less efficient, less usable, less scalable than the obvious centralized solution. this is a completely self consistent threat model. but the set of things you can trust in this world is extremely tiny. you can't use anything tradfi, because those ultimately rely on visa/Mastercard and banks and such to not fuck you over. you can't live in a house you own unless you have the arms to defend your house, because your ownership of that house depends on the government having a record somewhere that you own that house, and its willingness to use police to defend your house. you can't really invest in anything except physical assets you can put in your house, because any stock is just a digital record at the DTCC. you can't really rely on any food or water from the outside world to be safe unless you are testing it yourself or producing it yourself. there are people who genuinely live in a world like this. especially in certain politically unstable countries. but this is a very small subset of people, and the vast majority of crypto projects will never ever be used by any of those people. (and there will almost always be a better way to help them.)

Daniel Ellsberg on the corrosive effect of knowing secret information I've never had a security clearance. But I have had private or secret rationalist information, and it's shocking how fast it corrupts my epistemics. Claiming to want comparative advantage I overweight the importance of the information. I flinch from imperiling my access to more secret information. Talking about a related area without divulging private information feels too hard so I don't bother.  I can fight all these effects, but that's a cost, and one I've learned to bill to the goal or person behind the secret information. 

On balance, I support banning/obstructing datacenter buildout. That said, I'm not actually sure whether that impacts the omnicide risk positively or negatively. I don't think the LLM paradigm is AGI-complete. I don't have utter confidence in that, but I think it's more likely than not. And in worlds in which it is non-AGI-complete, the labs' obsession with them is very helpful. They're wasting macroeconomic amounts of money on it, pouring most of our generation's AI-researcher talent into it, distracting funding from other AGI approaches. If LLMs then fail to usher in the Singularity, if it does turn out to be a bubble that pops (e. g., in 2030-2032, when the ability to aggressively scale compute is supposed to run out), this should cause another AI winter. AGI would become a decidedly unsexy thing to work on once more, in industry and probably in academia both. What would obstructing the LLM paradigm (via various compute limits) do? Well, it may cause the AGI megacorps to start looking into other directions now, while they still have massive amount of unspent manpower and capital. Which may lead to someone "succeeding" sooner.[1] Perhaps one shouldn't interrupt their enemy while they're making a mistake. Or perhaps that's not how the story goes. Perhaps: LLMs are a strong signal that AI is a massively powerful technology, a signal legible to many more people than theoretical arguments. They're attracting macroeconomic amounts of funding to AI, funneling a large fraction of our generation's talent towards working on AI, fueling inter-company and geopolitical AI-race dynamics. And while they cause other AGI approaches to be relatively neglected, they cause so much more absolute attention to be pointed at the AI industry that the amounts of funding/talent going into non-LLM AGI routes is still much greater than in the no-LLMs counterfactual. On top of that, even if LLMs aren't AGI-complete in themselves, they may still be useful enough to speed up SWE/research alo