I like "Formal Ethics" or "AI Ethics". Including "Friendliness" in the name would probably be a mistake.

I was thinking something like "the Optimization Institute", or something else with the word "optimization" — since it's relevant to their research (and distinguishes their focus on precisely-understood optimization algorithms from other AGI projects that focus on often-nebulous and often-anthropomorphic views of intelligence), and doesn't have significant preexisting connotations, while still hopefully sounding a bit intriguing to people who haven't heard of it before.