For example if I see a very good contest/offer I might think it too good to be true, and look for more information to alter my model and find the catch before taking the offer up.
How is this case different from any other decision? You compute the current probabilities for this is a fraud and this is an unusually good deal. You compute the cost of collecting more data in a specific fashion, and the probability distribution over possible futures containing a future version of you with better knowledge about this problem. You do the same for various alternative actions you could take instead of collecting more data right now, calculate expected long-run utility for each of the considered possible futures, and choose an action based on that information - either to prod the universe to give you more data about this, or doing something else.
I am glossing over all the interesting hard parts, of course. But still, is there anything fundamentally different about manipulating the expected state of knowledge of your future-self from manipulating any other part of reality?
Interesting question. Not quite what I was getting at. I hope you don't mind if I use a situation where extra processing can get you more information.
A normal decision theory can be represented as simple function from model to action. It should halt.
decisiontheory :: Model -> Action
Lets say you have a model that you can keep on expanding the consequences of and get a more accurate picture of what is going to happen, like playing chess with a variable amount of look ahead. What the system is looking for is a program that will recursively self improve an...
Last summer, 15 Less Wrongers, under the auspices of SIAI, gathered in a big house in Santa Clara (in the SF bay area), with whiteboards, existential risk-reducing projects, and the ambition to learn and do.
Now, the new and better version has arrived. We’re taking folks on a rolling basis to come join in our projects, learn and strategize with us, and consider long term life paths. Working with this crowd transformed my world; it felt like I was learning to think. I wouldn’t be surprised if it can transform yours.
A representative sample of current projects:
Interested, but not sure whether to apply?
Past experience indicates that more than one brilliant, capable person refrained from contacting SIAI, because they weren’t sure they were “good enough”. That kind of timidity destroys the world, by failing to save it. So if that’s your situation, send us an email. Let us be the one to say “no”. Glancing at an extra application is cheap, and losing out on a capable applicant is expensive.
And if you’re seriously interested in risk reduction but at a later time, or in another capacity -- send us an email anyway. Coordinated groups accomplish more than uncoordinated groups; and if you care about risk reduction, we want to know.
What we’re looking for
At bottom, we’re looking for anyone who:
Bonus points for any (you don’t need them all) of the following traits:
If you think this might be you, send a quick email to jasen@intelligence.org. Include:
Our application process is fairly informal, so send us a quick email as initial inquiry and we can decide whether or not to follow up with more application components.
As to logistics: we cover room, board, and, if you need it, airfare, but no other stipend.
Looking forward to hearing from you,
Anna
ETA (as of 3/25/10): We are still accepting applications, for summer and in general. Also, you may wish to check out http://www.singinst.org/grants/challenge#grantproposals for a list of some current projects.