Most people don't have a systematic moral theory, where everything follows consistently from a small number of fundamental principles. Instead, they have a bundle of intuitions, emotions, principles, etc., which they use to varying extents in various situations. "More utilitarian" just means that they make moral judgments consistent with utilitarianism a larger percent of the time (which is consistent with having other terminal values besides well-being).
This study used several trolley-type problems, and found that libertarians were somewhat more likely to approve of flipping the switch or pushing the fat man (or throwing a person overboard from a lifeboat that is sinking because it has too many people on board). These dilemmas come from the utilitarianism vs. deontology debate, which suggests that libertarians might rely on cost-benefit reasoning more often and rights- or rules-based reasoning less often, but that is not necessarily true. Joshua Greene's take on these dilemmas is that the utilitarian response is based more on calculating reasoning (System 2) while the non-utilitarian response is based more on emotional aversions (System 1). So this result may just fit with the storyline that libertarians rely more on reasoning and less on emotions, with cost-benefit reasoning among the types of reasoning that they use more of.
An article from the Wall Street Journal. The original title might be slightly mind-killing for some people, but I found it moderately interesting especially considering that many LessWrongers formed part of the data set for the study the article talks about and a large fraction of us identified as libertarian on the last survey.
Inside the Cold, Calculating Libertarian Mind
The original paper.
Understanding Libertarian Morality: The Psychological Roots of an Individualist Ideology