Your labeling of physicalism as an "upside-down" approach reminded me of this quote from Schopenhauer, which you would no doubt approve of:

Of all systems of philosophy which start from the object, the most consistent, and that which may be carried furthest, is simple materialism. It regards matter, and t with it time and space, as existing absolutely, and ignores the relation to the subject in which alone all this really exists. It then lays hold of the law of causality as a guiding principle or clue, regarding it as a self-existent order (or arrangement) of things, Veritas aeterna, and so fails to take account of the understanding, in which and for which alone causality is. It seeks the primary and most simple state of matter, and then tries to develop all the others from it; ascending from mere mechanism, to chemism, to polarity, to the vegetable and to the animal kingdom. And if we suppose this to have been done, the last link in the chain would be animal sensibility — that is knowledge — which would consequently now appear as a mere modification or state of matter produced by causality. Now if we had followed materialism thus far with clear ideas, when we reached its highest point we would suddenly be seized with a fit of the inextinguishable laughter of the Olympians. As if waking from a dream, we would all at once become aware that its final result — knowledge, which it reached so laboriously, was presupposed as the indispensable condition of its very starting-point, mere matter; and when we imagined that we thought matter, we really thought only the subject that perceives matter ; the eye that sees it, the hand that feels it, the understanding that knows it Thus the tremendous petitio principii reveals itself unexpectedly; for suddenly the last link is seen to be the starting-point, the chain a circle, and the materialist is like Baron Munchausen who, when swimming in water on horseback, drew the horse into the air with his legs.

I still think it is a confused philosophy, but it is a memorable and powerful passage.