Just imagine if you were having a discussion with someone who said that the world is made of numbers. And you picked up a rock and said, so, this rock is made of numbers? And they said, sure. And you said, that's absurd. How could a rock be equal to 1+1, for example? They're completely different kinds of things. And they went off on a riff about how science has shown that all is number, and whenever you tried to point out the non-numerical aspects of reality, they'd just subsume that back into the all-is-number reductionism, and they'd stubbornly insist that, even if the rock was not equal to 1+1, it might be equal to some other numbers, and besides, what other sort of things could there be, besides numbers?
For me, the idea that is identical to some arrangement of particles in space is just like saying that 1+1 is a rock. The gulf between the nature of the allegedly identical entities is so great that the problem with the assertion ought to be obvious. In a sprinkling of point objects throughout space, where is the color? It's really that simple. It's just not there. It's not intrinsically there, anyway. You might propose that redness is a property of certain special configurations, but when you say that, you've embarked upon a form of dualism, property dualism. It's a dualism because on the one side, you have properties which are intrinsic to a geometrically defined situation, like distances and angles and shapes; and on the other side, you have properties which are logically independent of the geometric facts and have to be posited separately. For example, the existence of color experiences, or indeed any kind of experiences, in a brain.
In other words, the onus is on you to explain just what you think the connection is between arrangements of particles in space (e.g. a brain), and experiences of color. I have my own answer, but I want to hear yours first.
You won't find my answer interesting, but since you asked: I think experiences of color are among the states that particles in space can get into, just as the impulse to blink is a state particles in space can get into, just as a predisposition to generate meaningful English but not German sentences is a state that particles in space can get into, just as an appreciation for 17th-century Romanian literature is a state that particles in space can get into, just as a contagious head cold is a state that particles in space can get into. (Which is not to say ...
...at least not if you accept a certain line of anthropic argument.
Thomas Nagel famously challenged the philosophical world to come to terms with qualia in his essay "What is it Like to Be a Bat?". Bats, with sensory systems so completely different from those of humans, must have exotic bat qualia that we could never imagine. Even if we deduce all the physical principles behind echolocation, even if we could specify the movement of every atom in a bat's senses and nervous system that represents its knowledge of where an echolocated insect is, we still have no idea what it's like to feel a subjective echolocation quale.
Anthropic reasoning is the idea that you can reason conditioning on your own existence. For example, the Doomsday Argument says that you would be more likely to exist in the present day if the overall number of future humans was medium-sized instead of humongous, therefore since you exist in the present day, there must be only a medium-sized number of future humans, and the apocalypse must be nigh, for values of nigh equal to "within a few hundred years or so".
The Buddhists have a parable to motivate young seekers after enlightenment. They say - there are zillions upon zillions of insects, trillions upon trillions of lesser animals, and only a relative handful of human beings. For a reincarnating soul to be born as a human being, then, is a rare and precious gift, and an opportunity that should be seized with great enthusiasm, as it will be endless eons before it comes around again.
Whatever one thinks of reincarnation, the parable raises an interesting point. Considering the vast number of non-human animals compared to humans, the probability of being a human is vanishingly low. Therefore, chances are that if I could be an animal, I would be. This makes a strong anthropic argument that it is impossible for me to be an animal.
The phrase "for me to be an animal" may sound nonsensical, but "why am I me, rather than an animal?" is not obviously sillier than "why am I me, rather than a person from the far future?". If the doomsday argument is sufficient to prove that some catastrophe is preventing me from being one of a trillion spacefaring citizens of the colonized galaxy, this argument hints that something is preventing me from being one of a trillion bats or birds or insects.
And this could be that animals lack subjective experience. This would explain quite nicely why I'm not an animal: because you can't be an animal, any more than you can be a toaster. So Thomas Nagel can stop worrying about what it's like to be a bat, and the rest of us can eat veal and foie gras guilt-free.
But before we break out the dolphin sausages - this is a pretty weird conclusion. It suggests there's a qualitative and discontinuous difference between the nervous system of other beings and our own, not just in what capacities they have but in the way they cause experience. It should make dualists a little bit happier and materialists a little bit more confused (though it's far from knockout proof of either).
The most significant objection I can think of is that it is significant not that we are beings with experiences, but that we know we are beings with experiences and can self-identify as conscious - a distinction that applies only to humans and maybe to some species like apes and dolphins who are rare enough not to throw off the numbers. But why can't we use the reference class of conscious beings if we want to? One might as well consider it significant only that we are beings who make anthropic arguments, and imagine there will be no Doomsday but that anthropic reasoning will fall out of favor in a few decades.
But I still don't fully accept this argument, and I'd be pretty happy if someone could find a more substantial flaw in it.