I have felt the same fear that I am wasting my time talking to an extremely clever but disingenuous person. This is certainly no proof, but I will simply say that I assure you that I am not being disingenuous.
You use a lot of the words that people use when they talk about AGI around here. Perhaps you've heard of the Orthogonality Thesis?
From Bostrom's Superintelligence:
The orthogonality thesis
Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.
He also defines intelligence for the sake of explicating the aforementioned thesis:
Note that the orthogonality thesis speaks not of rationality or reason, but of intelligence. By “intelligence” we here mean something like skill at prediction, planning, and means–ends reasoning in general.
So, tending to be correct is the very definition of intelligence. Asking "Why are intelligent agents correct as opposed to incorrect?" is like asking "What makes a meter equivalent to the length of the path traveled by light in vacuum during a time interval of 1/299,792,458 of a second as opposed to some other length?"
I should also say that I would prefer it if you did not end this conversation out of frustration. I am having difficulty modeling your thoughts and I would like to have more information so that I can improve my model and resolve your confusion, as opposed to you thinking that everyone else is wrong or that you're wrong and you can't understand why. Each paraphrase of your thought process increases the probability that I'll be able to model it and explain why it is incorrect.
Two other people in this thread have pointed out that the value collapse into wireheading or something else is a known and unsolved problem and that the problems of an intelligence that optimizes for something assumes that the AI makes it through this in some unknown way. This suggests that I am not wrong, I'm just asking a question for which no one has an answer yet.
Fundamentally, my position is that given 1.) an AI is motivated by something 2.) That something is a component (or set of components) within the AI and 3.) The AI can modify that/those compone...
Part 1 was previously posted and it seemed that people likd it, so I figured that I should post part 2 - http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html