We learn "values" from experience, not just "facts". (I'm putting scare-quotes here because I believe the fact/value dichotomy is often overblown.) This gives the person a place to stand which is neither gene nor meme nor simply a mixture of the two.
A huge amount of the value-related information that we get from our environment comes from other living entities attempting to manipulate us. Sometimes, they negotiate with us, or manipulate our sense data - rather than attempting to affect our values. However, sometimes they attempt to "hijack our brains" - and redirect our values towards their own ends, or those of their makers.
The biggest influences come from other humans, symbionts, pathogens and memes. Basically most goal directedness comes from other living, goal-directed systems - so genes and memes - though not necessarily your own genes and memes.
The next biggest source of human values comes from the theory of self-organising systems. The brain is probably the most important self-organising system involved. It mostly has desires that arise by virtue of it being a large reinforcement learning system. Essentially, the brain sometimes acts as though it wants its own reward signals - and it fulfills those desires by doing things like taking rewarding drugs. The brain was made by genes - but wireheading is not exactly what the genes want.
The next-most significant effect on human values is probably mistakes (e.g. sub-optimal adaptations).
Many humans delight in seeking out noble sources of value - probably for signalling reasons. They do not like hearing that genes and memes are primarily responsible for what they hold most dear - and the next biggest influences are probably wireheading and mistakes.
The next biggest source of human values comes from the theory of self-organising systems. The brain is probably the most important self-organising system involved. It mostly has desires that arise by virtue of it being a large reinforcement learning system.
That's the sort of thing I had in mind. Because our conceptual framework is learned from experience, what we learn to seek is not necessarily what our genes "want". Of course if you place a human being in "the ancestral environment" then you will get learned values that serve th...
The jacket text for Keith Stanovich's The Robot's Rebellion sums up the book well:
The book is an excellent introduction to the first stage of Yudkowskian philosophy: We are robots in a mechanistic universe running on a swiss army knife of cognitive modules. But at least we finally noticed we're robots, and we can use the skills of rationality to hop off our habit treadmills and pursue our values instead. These values are complex and often arbitrary, but we can use our reflective capacities to extrapolate our values based on "higher-order" desires, a desire for preference consistency, and other considerations. All this is argued for at length in Stanovich's book. The only thing missing is a discussion of what to do about all this when AI arrives.