Preference of a given human is defined by their brain, and can be somewhat different from person to person, but not too much. There is nothing "objective" about this preference, but for each person there is one true preference that is their own, and same could be said for humanity as a whole, with the whole planet defining its preference, instead of just one brain. The focus on the brain isn't very accurate though, since environment plays its part as well.
I can't do justice to the centuries-old problem with a few words, but the idea is more or less this. Whatever the concept of "preference" means, when the human philosophers talk about it, their words are caused by something in the world: "preference" must be either a mechanism in their brain, a name of their confusion, or something else. It's not epiphenomenal. Searching for the "ought" in the world outside human minds is more or less a guaranteed failure, especially if the answer is expected to be found explicitly, as an exemplar of perfection rather than evidence about what perfection is, to be interpreted in nontrivial way. The history of failure to find an answer while looking in the wrong place doesn't prove that the answer is nowhere to be found, that there is now positive knowledge about the absence of the answer is the world.
Okay, so I'll ask again: why couldn't the humans real preference be to not sacrifice the children? Remember, you said:
You can't decide your preference, preference is not what you actually do, it is what you should do
You haven't really elucidated this. You're either pulling an ought out of nowhere, or you're saying "preference is what you should do if you want to win". In the latter case, you still haven't explained why giving up the children is winning, and not doing so is not winning.
And the link you gave doesn't help at all, since, if we're...
It's just possible that some lesswrong readers may be unfamiliar with Torchwood: It's a British sci-fi TV series, a spin-off from the more famous, and very long-running cult show Dr Who.
Two weeks ago Torchwood Series 3 aired. It took the form of a single story arc, over five days, shown in five parts on consecutive nights. What hopefully makes it interesting to rationalist lesswrong readers who are not (yet) Whovians was not only the space monsters (1) but also the show's determined and methodical exploration of an iterated Trolley Problem: in a process familiar to seasoned thought-experimenters the characters were tested with a dilemma followed by a succession of variations of increasing complexity, with their choices ascertained and the implications discussed and reckoned with.
An hypothetical, iterated rationalist dilemma... with space monsters... and monsters a great deal more scary - and messier - than Omega - what's not to like?
So, on the off chance that you missed it, and as a summer diversion from more academic lesswrong fare, I thought a brief description of how a familiar dilemma was handled on popular British TV this month, might be of passing interest (warning: spoilers follow)
The details of the scenario need not concern us too much here (and readers are warned not too expend too much mental energy exploring the various implausibilities, for want of distraction) but suffice to say that the 456, a race of evil aliens, landed on Earth and demanded that a certain number of children be turned over to them to suffer a horrible fate-worse-than-death or else we face the familiar prospects of all out attack and the likely destruction of mankind.
Resistance, it almost goes without saying, was futile.
The problems faced by the team could be roughly sorted into some themes
The Numbers dilemma - is it worth sacrificing any amount of children to save the rest?
The Quality dilemma: does it make any difference which children?
The choice dilemma: how should the sacrifical cohort be chosen?
The limits of human rationality: are there certain 'rational' decisions that are simply too much to expect a human being to be able to make?
Actually despite my jocular tone in the first paragraph I don't want to make too light of this series, as it was disturbing viewing.
Anyway: that being said: rationalist lesswrong community members may want to think dispassionately about the their answers before I reveal the conclusions that Russell T Davies (the writer) came to:
Numbers
Quality
Choice
This was handled by the politicians who considered two dimensions in the selection:
Rationality at the limit
On the question of 'how close' a straightforward evolutionary approach was used. Children of the decision-makers were safe, and grandchildren.
"And our nephews?" "Don't push it".
But the limits of rationality, it seems, are dependent upon gender: While it was recognised that no woman could be expected to agree to the rational sacrifice of her child, it was expected by some that men might have to, and in the end the main character - male - sacrificed a grandchild.
And that's it. Perhaps not a complete disposal of the trolley problem, but nevertheless an interesting excursion into the realms of philosphical dilemmas for a popular drama. Rationalism is a meme - pass it on.
(1) like many TV aliens: surprisingly able to construct spaceships without the benefit of an opposable thumb (2) Yes, that was actually the 1965 back-story.