empirical literature on what makes websites effective (which we've done a lot of now)

Can you share some of your sources?

"Effective" at what? Most websites aren't terribly concerned with aesthetic nuances -- they're concerned with things like monetization, channeling users into specific behaviours, avoiding user confusion, etc.

User interaction / interface design / usability / human-computer interface / etc. is a thriving field.

There is absolutely a collection of best practices, and most of it is in line with the research. Keep text to a minimum, have a clear visual hierarchy, create clear calls to action, if images have faces, make sure to follow the line of sight - these are all commonly accepted design practices, and are supported by the literature. Most good designers are familiar with color theory, etc.

Of course, most people don't hire a designer, or hire an awful one. This should be no surprise, as it's the same in most areas of human endeavors (how many people do you know who would go to a chiropractor for their back over a licensed doctor?).

However, there are large, well managed sites that you'll see breaking the accepted best practices left and right. The reason that you see this so frequently is that it's incredibly easy to run your own experiments online if you have enough traffic. There's websites that let you literally drag and drop to get an entirely new design,, and then will split test the pages just the right amount to get data while still optimizing to minimize lost conversions. Why would you use the best practices from theoretical "average website visitor" when you can use super easy software to figure out the best practices for your exact audience?

Finally, some of this stuff is anti-inductive. If you're trying to maximize attention, the best practice is whatever everybody else ISN'T doing. As soon as something becomes the accepted way to do things, it might be the wrong thing to do (although if youre trying to maximize usability, the opposite is true).

I don't perceive this as a problem unique to design. It's also a problem in education, business consulting, medicine, etc. Any time consumers don't have an easily available means of assessing product quality, there is little incentive for businesses to maximize effectiveness, so there will often be a huge gap between best practices and actual practices.

It sounds like you're mostly referring to UX. I share the impression that there's a lot of research in this area that rarely seems to be applied. One reason for that seems to be because people are lazy and/or too content with mediocrity. But the main reason seems to be just because it requires a lot of expertise. Which means either taking the time to train yourself, or pay others. Depending on your intent, this isn't always worth it (but I think people very much underestimate how "worth it" it is).

Keep in mind that not all targets are the same. Two examples:

http://craigslist.org - designed above all else to be useful. The aesthetic that matters here is 'useful'.

http://alteraeon.com - small static pages designed to work well with blind/visually impaired screen readers.

As far as I understand most websites don't optimize for aesthetics but for some business purpose.