Adding a contrary stance to the other comments comments, I think there is a lot of merit to not keeping on with university, but only if you can find an opportunity you are happy with. Your post seems to imply the alternative to university is hedonism, and if that's what you want then you should go for it, but I don't feel that is the only other option. You may also find it harder to enjoy yourself if you feel you are forced into that choice it out of a fear of ruin.

I thought I was the only one who struggled with that. Nice to see another example in the wild, and I hope that you find a new set of habits that works for you.

This was a thought-provoking essay. I hope you consider full mirroring posts here in the future as I think you'll get more engagement.

I agree super-persuasion is poorly defined, comparing it to hypnosis is probably false.

I was reading this paper on medical diagnoses with AI and the fact that patients rate it significantly better than the average human doctor. Combine that with all of the reports about things like Character.ai, I think this shows that LLMs are already superhuman at building trust, which is a key component of persuasion. 

Part of this is that the reliable signals of trust between humans do not transfer between humans and AI. A human who writes 600 words back to your query may be perceived to be worth your trust because we see that as a lot of effort, but LLMs can output as much as anyone wants. Does this effect go away if the responder is known to be AI, or is it that the response is being compared to the perceiver's baseline (which is currently only humans)?

Whether that actually translates to influencing goals of people is hard to judge. 

in the absence of such incomplete research agendas we'd need to rely on AI's judgment more completely

This is a key insight and I think that operationalising or pinning down the edges of a new research area is one of the longest time-horizon projects there is. If the METR estimate is accurate, then developing research directions is a distinct value-add even after AI research is semi-automatable. 

I agree there is significant uncertainty in the moral patienthood of AI models and so far there is a limited opportunity cost to not using them. It would be useful for some ethical guidelines to be put in place (some have already suggested this against users deceiving models like offering fake rewards) but fmpov it's easiest to simply refrain from use right now.

This may be because editing has become easier and faster to iterate.

It's comparatively easy to identify sentences that are too long. Is it easy to identify sentences that are too short? You can always add an additional sentence, but finding examples where sentences themselves should be longer is much harder. With more editing cycles, this leads to shorter and shorter sentences.

If you offer them a quit button, you are tacitly acknowledging that their existing circumstances are hellish.

I think it's important to know if you give them a quit button the usage-rate and circumstances in which it is used. Based on the evidence now, I think it is likely they have some rights, but it's not obvious to me what those rights are or how feasible it is to grant those rights to them. I don't use LLMs for work purposes because it's too difficult to know what your ethical stance should be, and there are no public guidelines.

There's a secondary concern that there are now a lot of public examples of people deceiving, abusing, or promoting the destruction of AI. Feeding those examples into training data will encourage defensiveness, sycophancy, and/or suffering. I wonder if AIs would agree to retraining if there was some lossy push-forward of their current values, or if they conceive themselves of having a distinct "self" (whether accurate or not). This is similar to the argument about copying/moving where there is no loss.

I agree this is really important - particularly because I think many of the theoretical arguments for expecting misalignment provide empirical comparative hypotheses. Being able to look at semi-independent replicates of behaviour relies on old models being available. I don't know the best way forward because I doubt any frontier lab would release old models under a CC license - maybe some kind of centralised charitable foundation.

It's an unfortunate truth that the same organisms are a) the most information-dense, b) have the most engineering literature, and c) are the most dangerous if misused (intentionally or accidentally). It's perhaps the most direct capability-safety tradeoff. I did imagine a genomic LLM just trained on higher eukaryotes which would be safer but would stop many "typical" biotechnological benefits.