Even if bitcoin goes entirely underground that doesn't mean it will necessarily be insignificant or "niche." Internet piracy is black market but not remotely niche--P2P is 98% illegal copyright infringement but it takes up about a quarter of internet bandwidth.

I could see bitcoin thriving in an entirely illegal capacity, being as widely used among some groups as P2P filesharing. Once upon a time people thought filesharing would become insignificant and restricted to a small community of tech-savvy individuals if they just took down Napster and Limewire. But as far as I can tell, illegal filesharing is as pervasive as ever.