Isn't the far simpler and more likely scenario that you never have just one variable accounting for all of an outcome? If other variables are not perfectly correlated with the variable you are graphing you will get noise. Why is it surprising that that noise also exists in the most extreme points?

EDIT: misunderstood last few paragraphs.