We want our preferences to point to the territory, but they cannot do so in actual reality.

There's a distinction to be made between the fact that our knowledge about whether our preferences are satisfied is map-bound and the assertion that our preferences only take the map into account.