I find this odd because it sounds like the exact opposite of the patternist view of identity

It's the lack of the patternist view of identity. I have no view of identity, so I disagree.

Would you not mind then if some process erased all of your memories?

It would be likely to cause problems, but beyond that, no. I don't see why losing your memory would be intrinsically bad.

I think the main thing I'm against is that any of this is fundamental enough to have any effect on anthropics. Erasing your memory and replacing it with someone else's who's still alive won't make it half as likely to be you, just because there's only a 50% chance of going from past him to you. Erasing your memory every day won't make it tens of thousands of times as likely to be one of them, on the basis that now you're tens of thousands of people.

You could, in principle, have memory mentioned in your utility function, but it's not like it's the end of the world if someone dies. I mean that in the sense that existance ceases for them or something like that. You could still consider it bad enough to warrant the phrase "it's like the end of the world".