I think you're really failing to grasp the content of the unique factorization theorem here. Firstly we don't think about factored numbers as products of primes up to permutation, we think of them as products of distinct prime powers (up to permutation, I suppose - but it's probably better here to just take a commutative viewpoint and not regard "up to permutation" as worth specifying). But more importantly, you need to take a multiary view of multiplication here, not a binary one. 1 is the empty product, so in particular, it is the product of no primes, or the product of each prime to the 0th power. That is its unique prime factorization. To take 1 as a prime would be like having bases for vector spaces include 0. Almost exactly like it - if we take the Z-module of positive rationals under multiplication, the set of primes forms a free basis; 1 is the zero element.
A second comment...
You've certainly convinced me that '1' should not be included in the set of things that are used to uniquely factor numbers. However, how I can I know if this set is the set of "primes"?
I guess I was thinking that the essence of primes was about their irreducibility/atomic-ness. The number 5 would be considered prime because you can't describe it multiplicatively in any way except by using the number 5. Using my preferred notion, the number 0 and the number -1 would also be "prime" (as Mr Hen guessed). Is there a different word for this concept?
Today I looked at the above illusion and thought, "Why do I keep thinking A and B are different colors? Obviously, something is wrong with how I am thinking about colors." I am being stupid when my I look at this illusion and I interpret the data in such a way to determine distinct colors. My expectations of reality and the information being transmitted and received are not lining up. If they were, the illusion wouldn't be an illusion.
The number 2 is prime; the number 6 is not. What about the number 1? Prime is defined as a natural number with exactly two divisors. 1 is an illusionary prime if you use a poor definition such as, "Prime is a number that is only divisible by itself and 1." Building on these bad assumptions could result in all sorts of weird results much like dividing by 0 can make it look like 2 = 1. What a tricky illusion!
An optical illusion is only bizarre if you are making a bad assumption about how your visual system is supposed to be working. It is a flaw in the Map, not the Territory. I should stop thinking that the visual system is reporting RGB style colors. It isn't. And, now that I know this, I am suddenly curious about what it is reporting. I have dropped a bad belief and am looking for a replacement. In this case, my visual system is distinguishing between something else entirely. Now that I have the right answer, this optical illusion should become as uninteresting as questioning whether 1 is prime. It should stop being weird, bizarre, and incredible. It merely highlights an obvious reality.
Addendum: This post was edited to fix a few problems and errors. If you are at all interested in more details behind the illusion presented here, there are a handful of excellent comments below.