I'll assume here that the speaker is neither culturally Japanese nor in Japan. Even under that assumption, being able to speak Japanese probably doesn't imply geekdom too strongly -- strictly speaking it's evidence for it, but most anime fans never learn the language past a few catchphrases, and it's not that uncommon a language in the West as non-Western European languages go.

However, using a Japanese phrase in a casual context implies quite a different set of things. You might be using it as tribal attire, which probably means you're an anime fan. You might expect it to carry meaning, which outside of a Japanese-run business, a language classroom, or a dojo probably implies that you're hanging out with anime fans. Or you might just be used to using it, which usually implies plenty of experience with one of the above.