I have an irrational preference

If your utility function weights you knowing things higher than most people's, that is not an irrationality.

It's "101"? I searched the regular internet to find out, but I got some yes's and some no's, which I suspect were just due to different definitions of intelligence.

It's controversial?? Has that stopped us before? When was it done to death?

I'm just confused, because if people downvote my stuff, they're probably trying to tell me something, and I don't know what it is. So I'm just curious.

Thanks. By the way, do you know why this question is getting downvoted?

I already figured that. The point of this question was to ask if there could possibly exist things that look indistinguishable from true alignment solutions (even to smart people), but that aren't actually alignment solutions. Do you think things like this could exist?
 

By the way, good luck with your plan. Seeing people actively go out and do actually meaningful work to save the world gives me hope for the future. Just try not to burn out. Smart people are more useful to humanity when their mental health is in good shape.

1. Yes, human intelligence augmentation sounds like a good idea.
2. There are all sorts of "strategies" (turn it off, raise it like a kid, disincentivize changing the environment, use a weaker AI to align it) that people come up with when they're new to the field of AI safety, but that are ineffective. And their ineffectiveness is only obvious and explainable by people who specifically know how AI behaves. Supposes there are strategies which ineffectiveness is only obvious and explainable by people who know way more about decisions and agents and optimal strategies and stuff than humanity has currently figured out thus far. (Analogy: A society who only know basic arithmetic could reasonably stumble upon and understand the Collatz conjecture; and yet, with all our mathematical development, we can't do anything to prove it. Just like we could reasonably stumble upon an "alignment solution" that we can't disprove that it would work, because that would take a much higher understanding of these kinds of situations.)
3. If the solution to alignment were simple, we would have found it by now. Humans are far from simple, human brains are far from simple, human behavior is far from simple. That there is one simple thing from which comes all of our values, or a simple way to derive such a thing, just seems unlikely.

Uh, this is a human. Humans find it much harder to rationalize away the suffering of other humans, compared to rationalizing animal suffering.

And the regular, average people in this future timeline consider stuff like this ethically okay?

hack reality via pure math

What - exactly - do you mean by that?

The above statement could be applied to a LOT of other posts too, not just this one.

How were these discovered? Slow, deliberate thinking, or someone trying some random thing to see what it does and suddenly the AI is a zillion times smarter?