Surely the size of the spider is often the most fundamental aspect? "Don't look now, but there's a huge brown spider behind you." If "big spider" were a species then it's inseparable (as benelliott points out) because it's a species name, but that's a different story. I wouldn't talk about a green enormous chameleon even if I knew that the chameleon was about to change colors without changing its size.

I think this work. To test it, imagine if there was a specific species of spider called "big spiders" and one of them was brown. I would then consider "brown big spider" to be more appropriate.

Marius makes a good point with the chameleon - Although when describing something as skinny/fat the color comes first (Red Faced Fat Man vs. Large/Huge Red-faced man)

Almost seems to me that we place words that categorize the object closest to the object - Brown Spider/Green Chameleon/Fat Man are all categories of those objects whereas a Big spider isn't as much of a category as it describes the size of the spider relative to other spiders in the same category.