Seems to me that you can dodge the Platonic implications (that Anathem was riffing on). You can talk about relations between objects, which depend on objects outside the mind of the speaker but have no independent physical existence in themselves; you need not only a shared referent but also some shared inference, but that's still quite achievable without needing to invoke some Form of, say, mathematical associativeness.