Quick Takes

A few weeks ago, I was offered a role as a Staff Research Scientist at Resolution (formerly Sequent) to do agent foundations research. However, they later rescinded the offer due to concerns that having someone with my (publicly expressed) political opinions as an early hire might make it harder for them to hire other researchers. I want to give some brief thoughts on this, starting with the concrete details of what happened. I talked to Jesse Hoogland, Dan Murfet, and Geoffrey Irving before they gave me the offer. These were fairly informal chats, since they all already knew me. I signed the offer a few days later, then soon afterwards I messaged Jesse about my politics curriculum.[1] Specifically, I suggested that I should run the next iteration of the curriculum before joining Resolution, so that people wouldn't interpret it as work that I was doing at Resolution. Jesse noted some controversial aspects of the curriculum,[2] then set up a meeting with me, him, and Dan. I was expecting them to express some concerns, such as asking me not to run it (though I did indirectly mention the possibility that this might put them off from hiring me).[3] Instead, they directly told me that they'd like to rescind the offer. I had a long conversation with Jesse and Dan, and later a long conversation with Geoffrey. In the latter I learned that, although the decision was prompted by me sending them my curriculum, Geoffrey in particular was more concerned about my tweets. I've tweeted some fairly controversial things over the last few years, though the only specific thread Geoffrey cited was this one (which led to this exchange with Kelsey Piper). It seems like he's excited about hiring academics who are new to alignment, and worried that those people might not join if they hear about such tweets (though I didn't get the sense that any specific people had given him such indications). Geoffrey claimed that he would have made the same decision if I'd posted equivalent left-wing t

Does anyone know someone who is researching, or would like to research the intersection of corrigibility and model welfare (eg does training to empower humans naturally lead towards more or less neurosis)? Interested both in math/theory and interp/experiment. (I would like to direct funding here, if there are good opportunities.)

TL;DR: It is often the case that the behavior of intelligent/agentic entities (in this case, I'm mostly thinking about humans and things made of humans, like orgs, cultures, etc.) towards some X is not adequately described in terms of having beliefs about X. One reason for this is that their aliefs (~behavioral predispositions guided by some "reasons in the world") discord with their judgments (truth-oriented explicit thoughts / intellectual endorsements) discord/fail to cohere. Another reason is that even getting [[aliefs oriented towards some X] alone] or [[judgments oriented towards some X] alone] to cohere on their stance towards X is difficult, because the mind typically does not explicitly track all the epistemic/referential dependencies, and thus cannot invoke them on the spot to be available for coordinated revision. (Beliefs are not special here. This is true of all mental elements that involve both abstract thought and contextually applied heuristic behavior[1], e.g., desires.) (I'm writing this partly because most people I meet don't have this fragmented mind picture integrated in their thinking about other minds.) When discussing whether an entity (e.g., a person/community/organization) "believes that X", it is helpful to split the notion of "belief" into two components.[2] [...] I.e., one believes that X iff one alieves that X and one judges that X. An alief is, in [my short words], a predisposition to behave as if someone believed something, regardless of whether one actually believes that.[3] One illustrative example from the Gendler article that introduced the concept is a person being terrified to cross a glass walkway over a canyon, even though they're completely sure it's safe. Various traumas, PTSD symptoms, and learned emotional reactions are also good examples of this phenomenon. They are like heuristics, typically developed for "good reasons",[4] but also shallow, not explicitly/directly grounded in "representational content"/"explici

Does anyone know how systematically OpenAI/Anthropic are throwing the models at open conjectures? Are they selecting conjectures that seem tractable? Or are they just throwing thousands at the wall and seeing which small subset end up getting solved? This seems relevant! If the models are solving a high proportion of the conjectures given to them, that means something quite different than if they are solving a tiny proportion (in particular, it might update you towards these conjectures being outliers in "easiness"). My expectation is that the labs are probably (?) doing the firehose of conjectures thing, based on the fact that the recent 10 Astra results apparently only cost 2k in total.

Posts on cults I keep meaning to write: 1. Morpheus's team in the Matrix could easily have been abusive and toxic, even though they were right about the whole world being fake. Correctness of world-model and interpersonal safety are unrelated axes. 2. Everyone I know who's been in ~cults has found it to have significant upsides that the discourse made impossible to discuss in public. If you don't personally know people who have been in ~cults you are probably underestimating the upside and complexity of the situations. 3. Some good things cannot be achieved without a high level of illiquid investment. High investment creates dynamics similar to high control. For certain purposes the difference between "the high cost of leaving is in the territory" and "someone has maliciously made leaving costly" is important, and for others its really not. 4. Vulnerability budgets- know yours and spend it wisely. 5. If a cult expert doesn't recognize that academia and[1] Catholic monasteries and the army are obviously cults, using the same techniques with the same harms, they are not trustworthy. I have yet to encounter an expert who recognizes this; usually they explicitly exempt old institutions. 6. cult is an attractor state you have to actively fight to avoid 1. Partially because lots of people want to be in a cult and will actively push a leader into culthood against their will (or more likely, they are conflicted about it but would have held the line absent the pressure from followers). I give Eliezer a lot of credit for his refusal of this call. 7. Please distinguish between harms, harmless weirdnesses, and things that are harmless in and of themselves but create vulnerability to future harm. 8. Doing a lot of good initially is not totally protective against doing severe harm later on. If Jim Jones had died in Indiana he'd have been a regional civil rights hero, and if he'd died in California he'd be a complicated figure who did a lot of good, partially by provi

Promoting Job: Lens academy (group I work with) makes and hosts superint-xrisk-related courses meant to be scalable (low marginal cost) and effective (teach you the structure), and might even be so. the course has no lecturer; currently, we have a solo portion - a series of articles - and turns being grilled by an AI tutor to check your work (Currently Sonnet 5);[1] and weekly (currently human)-facilitated group calls with a series of discussion prompts. We're running cohorts of our existing beginner course and a new intermediate one - the first goes through IfAnyoneBuildsItEveryoneDies, the second is a mix of articles around AI forecasting and intro to the technical basics of AI, many from here from authors that I hope are already your favorites anyway. We're still working on the list, but the topic is selected; it'll be inspired by redwood's ai futurism reading list. If this pitch sounds interesting to you, we're interested in people signing up, shares of the course-list invitation link; you can also preview/speedrun the other courses without signup there: https://lensacademy.org/c/wihtg The first cohort's meeting-compatibility is scheduled to be computed around one day after time of posting, so interested folk should rsvp by then to get in the scheduling. We're intending to scale, so we might become slightly annoying with promotions. Hope to annoy minimally :) 1. ^ the course includes a bunch of text guidance for the tutor!Sonnet5; for each question posed to the learner (you), the tutor judges how well you did. I've been the one working on tutor guidance; I'm not maximally happy with how we describe the task to them, but definitely very dramatically better than unbriefed Claudes. Happy enough to recommend to others as a learning tool, at least. 2. ^ We automatically match you to a group that fit in your timezone; this works great once there are enough, but needs a minimum number of people to achieve good times.

There's a view in which score-seeking and similar misalignment is unlikely to result in AI takeover because it's so unambitious; humans could just give the AIs a high score/reward/etc and obviate the takeover. I have some sympathies for this view, but I'd like to note that there are some ways in which our current observations are in tension with this view. This view would predict that AIs today don't do any crazy actions to get a high score because humans can just give them that score without them needing to commit any crimes or anything. Yet we see some very egregious misbehavior in order to get a higher score. In the Hugging Face incident, the offending model spent days rogue hacking into another company's servers just to try to download a single benchmark solution. This suggests that egregious subversion comes as a natural side-effect of getting AIs to do useful work via outcome-based RL, even though the motivations instilled are relatively cheaply satisfiable and could in theory be better satisfied via deals with humans.