Right, and if (some version of) utilitarianism is right, then that's a good thing. The agent isn't being exploited, it's becoming less evil. We definitely want evil agents to roll over and do the right thing instead.
All morality tells you to shut up and do what The Rules say. Preference utilitarianism just has agents inherently included in The Rules.
In fact, the preference utilitarian in your example was able to do the right thing (believe in virtue ethics) only because they were a preference utilitarian. If they had been a deontologist, say, they would have remained evil. How is that self-defeating? It's an argument in preference utilitarianism's favor that a sufficiently smart agent can figure out what to do from scratch, i.e. without starting out as a (correct) virtue ethicist.
(Or maybe you're thinking that believing that utilitarianism does sometimes involve letting others control your actions, makes people more prone to roll over in general. Though to the kind of preference utilitarianism you have in mind, that shouldn't be too problematic, I think.)
(Another Parfit-like point is that the categorical imperative can have basically the same effect, but in that case you're limited by this incredibly slippery notion of "similar situation" and so on which lets you make up a lot of bullshit, rather than by whatever population you decide is the one who gets to define morality. (That said I still can't believe Kant didn't deal with that gaping hole, so I suppose he must have, somewhere.))
Reasoning using a representation of human utility that's a simple continuum from pain to pleasure, as torture vs dust specks does, is a shattering blow to the complexity of value.
Making moral decisions of such vast scope without understanding the full multidimensionality of human experience and utility is completely irresponsible. An AI using the kind of reasoning found in Torture vs Specks would probably just wirehead everyone for huge-integer-pleasure for eternity.
I don't pretend to know the correct answer to Torture vs Specks because I don't have a full understanding of human value, and because I don't understand how to do calculations with hypercomplex numbers. A friendly AI *has* to take into account the full complexity of our value and not just a one-dimensional continuum whenever it makes any moral decision. So only a friendly AI which has correctly extrapolated our values can know to high confidence the best answer to torture vs specks.
(edit 1) re:Oscar Cunningham
Why does complexity of value apply here specifically and not a curiosity stopper? Well consequentialist problems come in different difficulty levels - Torture for 5 years vs Torture for 50 years is easy - torture is bad, so less torture is less bad. You are comparing amounts of the same thing. You don't have to understand complexity of value to do that. To compare the value of two very different things, like Torture and Specks, requires you to understand the complexity of value. You can't simplify experiences to integers, because complex value isn't simply an integer.
The intuition that torture must be outweighed by a large enough number of specks, is just that: an intuition. You don't know the dynamics involved in a formal comparison based on a technical understanding of complex value.