Well of course it worries people! Precisely the function of consciousness (at least in my current view) is to "paint a picture" of wholeness and continuity that enables self-reflective cognition. Problem is, any given system doesn't have the memory to store its whole self within its internal representational data-structures, so it has to abstract over itself rather imperfectly.

The problem is that we currently don't know the structure, so the discord between the continuous, whole, coherent internal feeling of the abstraction and the disjointed, sharp-edged, many-pieced truth we can empirically detect is really disturbing.

It will stop being disturbing about five minutes after we figure out what's actually going on, when everything will once again add up to normality.