Are you open to the idea of sailing right around this Scylla and Charybdis by discarding mass-participation democracy, or must the solution set be within the set of possible democracies?
Because if you are open to less-than-democratic solutions, restricting the voting franchise seems like a promising way forward. On this, try Jason Brennan's "Against Democracy", a critique of mass participation democracy, and an argument for a system where prospective voters must pass a knowledge test in order to vote.
Are there any real world examples of this? A knowledge test to vote of course sounds like an improvement. However there is the "what are the true facts" problem. Was the usa winning the vietnam war while it happened? Were we always at war with oceania?
You have the fundamental conflict that for the government to be held accountable and for errors to be recognized, the voters have to know they happened. But if the government writes the knowledge tests they can make it a precondition to vote be that you "know" they are doing a good job.
isn't increasing the competence of the voter akin to increasing the competence of the official, by proxy? I'm pattern matching this to yet another push-pull compromise between the ends of the spectrum, with a strong lean toward technocracy's side.
I'm assuming I'll have to read Brennan for his response to the criticism that it was tried in u.s. and made a lot of people very upset / is widely regarded as a bad move.
I agree with Gerald Monroe about the overall implementation problems even if you assume it wouldn't just be a proxy for race or class war (which ...
I disagree with how you split the two options. For example, the minorities can also be oppressed by technocracy (imagine university-educated Nazis planning the Final Solution), perhaps even more on average, because if they have 10% of the popular vote, it is probably closer to 0% among the technocracy.
(You may assume the technocracy is enlightened and benevolent, but if we go this way, why not simply assume an enlightened and benevolent dictator?)
An orthogonal approach is to consider the units of governance. Are the rules uniform for the whole country, or does each region decide for themselves? Regardless of whether the region uses the local populus, or the local technocrats... or could we go even more meta and assume that even the "populus or technocrats" question is answered differently in different regions?
An advantage of smaller regions is that you can try many things in paralell and see what works better. A disadvantage is that some regions might make very bad choices. (Greater variance means both much better and much worse outcomes than the average.) But the bad choice will not ruin the entire country. Some selfishness may be involved, for example regions in the middle of the country less willing to contribute to protection of borders. Also, there seems to be natural pressure towards centralization (see how USA evolved from "the federal government has only those powers specifically granted by the Constitution" to people acting as if the choice of President changes everything).
I think that these two categories correspond rather accurately to aristocracy (technocracy) and democracy (populism) in the terminology of classical political theory. As Aristotle lays out in the Politics, both structures of political organization have a way of being virtuous and beneficial, and a way of being perverted and self-destructive. In other words, both structures of political organization-the government of a few excellent people, and the government of every adult-can be excellent, as long as they attend to their particular tendencies towards perversion (arrogance and indifference of the aristocrats in Aristocracy, class resentments and enforced false-equality in a Republic).
Point being that any political structure needs an immune system, something to combat its inherent tendencies towards failure. I think this immune system necessarily includes what you will find in Aristotle's Ethics.
Aristotle's Ethics and Politics are in fact two parts of a single work. The Ethics is the first book and the Politics the second. The Ethics asks what is virtue and how it develops in an individual, and the Politics asks what kind of society can raise ethical individuals and how it comes about. They are excellent, profound, fundamental readings and they will certainly help you beef up your thinking on this question.
As for a specific social technology, may I venture: Something like a temple. A place where wise old men listen patiently to your nonsense and help you find your way to sense, and gather the courage and develop the subtle means that can help heal the senselessness around you.
I broadly see the situation as follows
populism is the failure mode that I can characterize as the two wolves and a sheep problem (voting on what's for dinner), adversarial dynamics between majorities and minorities are exacerbated.
technocracy is the failure mode where you get a bunch of wonks together to look for positive-sum solutions, maximize on behalf of the aggregate, etc., but you're fighting a losing battle to compress information for them (i.e. in the hayekian criticism of economic planning sense).
I guess they fall on opposite sides of a spectrum and I view actually-existing democracies as a constant push-pull, negotiating a sweet spot on the spectrum. I'm wondering if we can dissolve the problem with some truly galaxy-brained social technology.
What can I read to beef up my thinking about this?