AI Safety Camp connects new collaborators worldwide to discuss and decide on a concrete research proposal, gear up online as a team, and try their hand at AI safety research during intensive coworking sprints.

Fourteen teams formed at this year's virtual camp. Here are their original proposals. Below are summaries at the end of the camp. Each team presented their findings at our final online weekend together. Some published a paper or post since. Most are continuing work, so expect a few more detailed and refined write-ups down the line.

Do you have AI Safety research ideas that you would like others to work on? Is there a project you want to do and you want help finding a team to work with you? Apply here.

Want to fund organisers who had to freeze their salary? Email remmelt@aisafety.camp.

Uncertainty -> Soft Optimization

Research lead: Jeremy Gillen

Team Participants’ names: Benjamin Kolb, Simon Fischer, Juan Azcarreta Ortiz, Danilo Naiff, Cecilia Wood

Short write-up: For AISC 2023, our team looked into the theoretical foundations for soft optimization. Our goal at the beginning was to investigate variations of the original quantilizer algorithm, in particular by following intuitions that uncertainty about goals can motivate soft optimization. We ended up spending most of the time discussing the foundations and philosophy of agents, and exploring toy examples of Goodhart’s curse.

Our discussions centered on the form of knowledge about the utility function that an agent must have, such that expected utility maximization isn’t the correct procedure (from the designers perspective). With well calibrated beliefs about the true utility function, it’s always optimal to do EU maximization. However, there are situations where an agent is very sensitive to prior specification, and getting this wrong can have a large impact on the true utility produced by the agent. Several other unrelated threads were pursued, such as an algorithm with high probability of above-threshold utility, and the relationship between SGD and soft optimization.

Post: Soft-optimization, Bayes and Goodhart

Discussing and Crystallizing a Research Agenda Based on Positive Attractors & Inherently Interpretable Architectures

Research lead: Robert Kralisch

Team Participants’ names: Anton Zheltoukhov (TC), Sohaib Imran, Johnnie Pascalidis and David Liu

Short write-up: For AISC 2023, our team w...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments