The singularity at the center of a black hole is presumably equally illusory; general relativity simply breaks down there, giving nonsensical answers (infinities). Singularities are in the map, not the territory. (Am I the first to say that?)

You might be the first to say it that way although the idea has certainly been expressed before. However, we don't actually know this for sure. We don't what happens at the singularity in a black hole. It seems likely that are models really are giving something wrong, and there are a handful of suggestions about what might be actually happening, but none of them are completely mathematically satisfying, and the lack of observational evidence is also an issue. We might be able to answer this if we a) had a theory of quantum gravity or b) had some naked singularities we could observe. Neither looks very likely at the moment, with the first probably more promising.