Asset allocation matters too. Vanguard target retirement funds follow the conventional wisdom (more stocks when you're young, more bonds when you're older) and are pretty cheap. Plowing all new investments into a single target-date fund is good advice for most people*. I implemented a scheme to lower my expenses from 0.17% to 0.09%, but it was not worth the time, hassle, and tax complications. *People who should do something more complicated include retirees, who should strongly consider buying an annuity, and people who are saving to donate to charity.

The issue with an index fund that based on something like the SAP 500 is that the SAP 500 changes over time. If a company loses their SAP 500 all the index funds that are based on the SAP 500 dump their stocks on the market. On average that's not going to be a good trade. The same goes for the trade of buying the companies that just made it into the SAP 500. On average you are going to lose some money to hedge funds or investment banks who take the other side on those trades. In general you can expect that if you invest money into the stockmarket big powerful banks have some way to screw you. But they won't take all your money and index funds are still a good choice if you don't want to invest too much time thinking about investing.