All of davidkwca's Comments + Replies

The canonical problem in infinite ethics is to create a preference relation over [0,1]N which is in some sense "reasonable". That's what this approach does. For more background you can see this background article or my overview of the field.

Maybe not, if you can keep your solution to AGI secret and suppress it if it turns out that there's no way to solve the alignment problem in your framework.

Valid question. I think the way I framed it may over-sell the solution -- there'll still be some problems of reference-class choice. But, you should move toward gears to try and resolve that. The way in which policy-level thinking improves the reference-class tennis is that you're not stopping at reference-class reasoning. You're trying to map the input-output relations involved. If you think you're terrible at planning, but not in cases X, Y, and Z, what mechanisms do you think lie behind that? You don't have to know in order to use the reference classes, but if you are feeling uncertain about the validity of the reference classes, digging down into causal details is likely a pretty good way to disambiguate. For example, maybe you find that you're tempted to say something very critical of someone (accuse them of lying), but you notice that you're in a position where you are socially incentivised to do so (everyone is criticising this person and you feel like joining in). However, you alse value honesty, and don't want to accuse them of lying unfairly. You don't think you're being dishonest with yourself about it, but you can remember other situations where you've joined in an group criticism and later realized you were unfair due to the social momentum. I think the reference-class reaction is to just downgrade your certainty, and maybe have a policy of not speaking up in those kinds of situations. This isn't a bad reaction, but it can be seen as a sort of epistemic learned helplessness. "I've been irrational in this sort of situation, therefore I'm incapable of being rational in this sort of situation." You might end up generally uncomfortable with this sort of social situation and feeling like you don't know how to handle it well. So, another reaction, which might be better in the long-term, would be to take a look at your thinking process. "What makes me think they're a liar? Wow, I'm not working on much evidence here. There are all kinds of alternative expl

I'm not advocating what Hanson calls prediction-markets. I'm advocating a different setup that's described in the linked article. The core problem is that even if it would be possible in a perfect world to run evidence-based studies to gather the knowledge in the present system there are no economic incentives for anybody to run the required studies in a way that's likely to lead to effective clinical predictions. There's no accountability that pushes clinical trial design in a way that leads to clear clinical benefits. The incentives are mostly about overstating the effect of the intervention that's studied. Even if there would be a sincere attempt at running the required studies it would be much more expensive than the way we currently study interventions and that means we are likely to study less interventions and thereby slowing down innovation by making the invention of new interventions more costly.