That's a good point. I think some old hot water heaters might even be so full of small particles that they're hard to drain from the bottom, and you might need to get the water from the top.

However, I think most of the sediment would be insoluble in water, and can be avoided by letting the water settle for a few minutes. Any soluble particles would have long since dissolved, sitting in a bath of hot flowing water for years.