However, regardless of all that, it seems to me that buying has some tremendous drawbacks, for which I can't see comparable upsides under any realistic circumstances.

Before I bought my house I ran the numbers and came to the same conclusion, that home ownership would not maximize my net worth and would increase certain types of risk. As a result I see home ownership as a luxury, not as an investment. I bought my house because I wanted it as a luxury and believed I could manage the risk.