Basilisks seem to only affect people of a certain mental capacity able to understand and process them.

And, probably more importantly, without certain other mental capacities that allow them to handle information appropriately.