I blame browser caching. In the process of testing I made several dozen requests without a Referer and all of them were 200 OK (and in the case of a couple dozen of those I double-checked that the file contents were identical with each other).

Edit: Actually, you can test this if your Apache install is set to log Referers to your access log (this is a common default). If so, you'll probably find that the 403 lines have Referers and the 200 lines don't. (You may also see some 304 Not Modified responses there, which would probably be either all without Referers or maybe a mixture.)