Comment on Open Thread: June 2009

Tue, 02 Jun 2009 08:41:06 GMT

This might make a fun game actually.

Player One sets up a hypothetical AI (utility function + restraints), and Player Two, or the rest of the group, must find a situation where a super intelligence with that utility function would lead to a dystopia. Here's my entry:

1) Utility is obtained by 'votes'. Every person can either give +1 utility or -1 utility to the AI each day. To avoid issues of uncertainty over what constitutes a person, before the AI is turned on everyone is given a voting button. Additional voting buttons that may be created provide no utility.

2) Strongly discount future votes, using U(t) = 1 / 2^t for the new utility, where t is time in days.This way a constant vote for all time yields a finite utility.

3) The AI can not take any action until one year after it has been turned on, except for text outputs to satisfy (4)

4) The AI has an oracle component. Any voter can ask how their life on a particular future day will vary from the present day. The AI must answer to the best of its abilities.

Discuss