benjaminko

Message

AISC team report: Soft-optimization, Bayes and Goodhart

This is a report on our work in AISC Virtual 2023. For AISC 2023, our team looked into the foundations of soft optimization. Our goal at the beginning was to investigate variations of the original quantilizer algorithm, in particular by following intuitions that uncertainty about goals can motivate soft optimization....

Jun 27, 2023•38

Message

32 karma

Member for 4 years

benjaminko — LessWrong

benjaminko

Message

benjaminko

AISC team report: Soft-optimization, Bayes and Goodhart

Jun 27, 2023•38

Message

32 karma

Member for 4 years

AISC team report: Soft-optimization, Bayes and Goodhart

Simon Fischer

Simon Fischer, benjaminko, jazcarretao, DFNaiff, Jeremy Gillen+ 0 more

Simon Fischer, benjaminko, jazcarretao, DFNaiff, Jeremy Gillen

This is a report on our work in AISC Virtual 2023.

For AISC 2023, our team looked into the foundations of soft optimization. Our goal at the beginning was to investigate variations of the original quantilizer algorithm, in particular by following intuitions that uncertainty about goals can motivate soft optimization. We ended up spending most of the time discussing the foundations and philosophy of agents, and exploring toy examples of Goodhart’s curse.

Our discussions centered on the form of knowledge about the utility function that an agent must have, such that expected utility maximization isn’t the correct procedure (from the designer's perspective). With well-calibrated beliefs about the true utility function, it’s always optimal to... (read 4279 more words →)

LESSWRONG
LW

LESSWRONG
LW

benjaminko

benjaminko

benjaminko

AISC team report: Soft-optimization, Bayes and Goodhart

benjaminko

benjaminko

benjaminko

AISC team report: Soft-optimization, Bayes and Goodhart