The work of Elinor Ostrom (2009 Nobel prize co-winner in economics) seems relevant. The Wikipedia page on her does a decent introduction. The relevant part of her work was in how societies use customs (other than market transactions) to regulate use of common resources. The relevant observation here is that the customs often seem strange and non-sensical, but they work. She summarized her findings, "A resource arrangement that works in practice can work in theory."

Similarly, the work of Peter Leeson on ordeals seems relevant. Ordeals were medieval methods of determining the outcome of what would today be a lawsuit. An example of an ordeal is (literally) trial by fire or trial by battle. Leeson shows how this facially strange and non-sensical custom actually served its purpose of dispensing justice. His research along these lines is surprising, unorthodox, and amusing.