Well I agree with this basically, and it reminds me of John Hasnas writing about customary legal systems. I find that when showing this to people I disagree with about ethics we usually end up in agreement:

In the absence of civil government, most people engage in productive activity in peaceful cooperation with their fellows. Some do not. A minority engages in predation, attempting to use violence to expropriate the labor or output of others. The existence of this predatory element renders insecure the persons and possessions of those engaged in production. Further, even among the productive portion of the population, disputes arise concerning broken agreements, questions of rightful possession, and actions that inadvertently result in personal injuries for which there is no antecedently established mechanism for resolution. In the state of nature, interpersonal conflicts that can lead to violence often arise.

What happens when they do? The existence of the predatory minority causes those engaged in productive activities to band together to institute measures for their collective security. Various methods of providing for mutual protection and for apprehending or discouraging aggressors are tried. Methods that do not provide adequate levels of security or that prove too costly are abandoned. More successful methods continue to be used. Eventually, methods that effectively discourage aggression while simultaneously minimizing the amount of retaliatory violence necessary to do so become institutionalized. Simultaneously, nonviolent alternatives for resolving interpersonal disputes among the productive members of the community are sought. Various methods are tried. Those that leave the parties unsatisfied and likely to resort again to violence are abandoned. Those that effectively resolve the disputes with the least disturbance to the peace of the community continue to be used and are accompanied by ever-increasing social pressure for disputants to employ them.

Over time, security arrangements and dispute settlement procedures that are well-enough adapted to social and material circumstances to reduce violence to generally acceptable levels become regularized. Members of the community learn what level of participation in or support for the security arrangements is required of them for the system to work and for them to receive its benefits. By rendering that level of participation or support, they come to feel entitled to the level of security the arrangements provide. After a time, they may come to speak in terms of their right to the protection of their persons and possessions against the type of depredation the security arrangements discourage, and eventually even of their rights to personal integrity and property. In addition, as the dispute settlement procedures resolve recurring forms of conflict in similar ways over time, knowledge of these resolutions becomes widely diffused and members of the community come to expect similar conflicts to be resolved in like manner. Accordingly, they alter their behavior toward other members of the community to conform to these expectations. In doing so, people begin to act in accordance with rules that identify when they must act in the interests of others (e.g., they may be required to use care to prevent their livestock from damaging their neighbors’ possessions) and when they may act exclusively in their own interests (e.g., they may be free to totally exclude their neighbors from using their possessions). To the extent that these incipient rules entitle individuals to act entirely in their own interests, individuals may come to speak in terms of their right to do so (e.g., of their right to the quiet enjoyment of their property).

In short, the inconveniences of the state of nature represent problems that human beings must overcome to lead happy and meaningful lives. In the absence of an established civil government to resolve these problems for them, human beings must do so for themselves. They do this not through coordinated collective action, but through a process of trial and error in which the members of the community address these problems in any number of ways, unsuccessful attempts to resolve them are discarded, and successful ones are repeated, copied by others, and eventually become widespread practices. As the members of the community conform their behavior to these practices, they begin to behave according to rules that specify the extent of their obligations to others, and, by implication, the extent to which they are free to act at their pleasure. Over time, these rules become invested with normative significance and the members of the community come to regard the ways in which the rules permit them to act at their pleasure as their rights. Thus, in the state of nature, rights evolve out of human beings’ efforts to address the inconveniences of that state. In the state of nature, rights are solved problems.

Ah, okay! We don't disagree then. Thanks for clearing that up!

ETA: Actually, with that clarification, I'd expect many others to agree as well - at least, it seems like what you mean by "custom" and what other posters have called "stuff people want you to do" coincide.