Because if there was a giant brain of physical matter hovering around Earth, we'd see it. If there was a giant brain of some undetectable non-matter stuff hovering around, then the fact that we can't see it doesn't mean a thing, one way or another.

It's still significantly less likely to be there than not, but rather more likely than if intelligence needs a brain-like matter substrate to work on.