I'd certainly call them much more significant to my identity than a e.g. my deltoid muscle, or some motor function parts of my brain.
It may be useful to recognize that this is a choice, rather than an innate principle of identity. The parts that speak are just modules, just like the parts that handle motor control. They can (and often do) run autonomously, and then the module that handles generating a coherent narrative stitches together an explanation of why you "decided" to cause whatever they happened to generate.
This sounds like a theory of identity as epiphenomenal homunculus. A module whose job is to sit there weaving a narrative, but which has no effect on anything outside itself (except to make the speech module utter its narrative from time to time). "Mr Volition", as Greg Egan calls it in one of his stories. Is that your view?
Another month has passed and here is a new rationality quotes thread. The usual rules are: