I just left a comment on PIBBSS' Manifund grant proposal (which I funded $25k) that people might find interesting.

Main points in favor of this grant

1. My inside view is that PIBBSS mainly supports “blue sky” or “basic” research, some of which has a low chance of paying off, but might be critical in “worst case” alignment scenarios (e.g., where “alignment MVPs” don’t work, or “sharp left turns” and “intelligence explosions” are more likely than I expect). In contrast, of the technical research MATS supports, about half is basic research (e.g., interpretability, evals, agent foundations) and half is applied research (e.g., oversight + control, value alignment). I think the MATS portfolio is a better holistic strategy for furthering AI alignment. However, if one takes into account the research conducted at AI labs and supported by MATS, PIBBSS’ strategy makes a lot of sense: they are supporting a wide portfolio of blue sky research that is particularly neglected by existing institutions and might be very impactful in a range of possible “worst-case” AGI scenarios. I think this is a valid strategy in the current ecosystem/market and I support PIBBSS!
2. In MATS’ recent post, “Talent Needs of

Main takeaways from a recent AI safety conference:

• If your foundation model is one small amount of RL away from being dangerous and someone can steal your model weights, fancy alignment techniques don’t matter. Scaling labs cannot currently prevent state actors from hacking their systems and stealing their stuff. Infosecurity is important to alignment.
• Scaling labs might have some incentive to go along with the development of safety standards as it prevents smaller players from undercutting their business model and provides a credible defense against lawsuits regarding unexpected side effects of deployment (especially with how many tech restrictions the EU seems to pump out). Once the foot is in the door, more useful safety standards to prevent x-risk might be possible.
• Near-term commercial AI systems that can be jailbroken to elicit dangerous output might empower more bad actors to make bioweapons or cyberweapons. Preventing the misuse of near-term commercial AI systems or slowing down their deployment seems important.
• When a skill is hard to teach, like making accurate predictions over long time horizons in complicated situations or developing a “security mindset,” try treating human

Crucial questions for AI safety field-builders:

• What is the most important problem in your field? If you aren't working on it, why?
• Where is everyone else dropping the ball and why?
• Are you addressing a gap in the talent pipeline?
• What resources are abundant? What resources are scarce? How can you turn abundant resources into scarce resources?
• How will you know you are succeeding? How will you know you are failing?
• What is the "user experience" of my program?
• Who would you copy if you could copy anyone? How could you do this?
• Am I better than the counterfactual?
• Who are your clients? What do they want?

Reasons that scaling labs might be motivated to sign onto AI safety standards:

• Companies who are wary of being sued for unsafe deployment that causes harm might want to be able to prove that they credibly did their best to prevent harm.
• Big tech companies like Google might not want to risk premature deployment, but might feel forced to if smaller companies with less to lose undercut their "search" market. Standards that prevent unsafe deployment fix this.

However, AI companies that don’t believe in AGI x-risk might tolerate higher x-risk than ideal safet... (read more)

Types of organizations that conduct alignment research, differentiated by funding model and associated market forces:

• Academic research groups (e.g., Krueger's lab at Cambridge, UC Berkeley CHAI, NYU ARG, MIT AAG);
• Research nonprofits (e.g., ARC Theory, MIRI, FAR AI, Redwood Research);
• "Mixed funding model" organizations:
  • "Alignment-as-a-service" organizations, where the product directly contributes to alignment (e.g., Apollo Research, Aligned AI, ARC Evals, Leap Labs);
  • "Alignment-on-the-side" organizations, where product revenue helps funds alignment research

MATS' goals:

• Find + accelerate high-impact research scholars:
  • Pair scholars with research mentors via specialized mentor-generated selection questions (visible on our website);
  • Provide a thriving academic community for research collaboration, peer feedback, and social networking;
  • Develop scholars according to the “T-model of research” (breadth/depth/epistemology);
  • Offer opt-in curriculum elements, including seminars, research strategy workshops, 1-1 researcher unblocking support, peer study groups, and networking events;
• Support high-impact&n

"Why suicide doesn't seem reflectively rational, assuming my preferences are somewhat unknown to me," OR "Why me-CEV is probably not going to end itself":

• Self-preservation is a convergent instrumental goal for many goals.
• Most systems of ordered preferences that naturally exhibit self-preservation probably also exhibit self-preservation in the reflectively coherent pursuit of unified preferences (i.e., CEV).
• If I desire to end myself on examination of the world, this is likely a local hiccup in reflective unification of my preferences, i.e., "failure of pres

Are these framings of gradient hacking, which I previously articulated here, a useful categorization?

1. Masking: Introducing a countervailing, “artificial” performance penalty that “masks” the performance benefits of ML modifications that do well on the SGD objective, but not on the mesa-objective;
2. Spoofing: Withholding performance gains until the implementation of certain ML modifications that are desirable to the mesa-objective; and
3. Steering: In a reinforcement learning context, selectively sampling environmental states that will either leave the mesa-objecti

How does the failure rate of a hierarchy of auditors scale with the hierarchy depth, if the auditors can inspect all auditors below their level?

Are GPT-n systems more likely to:

1. Learn superhuman cognition to predict tokens better and accurately express human cognitive failings in simulacra because they learned these in their "world model"; or
2. Learn human-level cognition to predict tokens better, including human cognitive failings?