I suppose the regret is a decent way out, although I would guess that you would need to experience true contrition over the murder or over the atrociousness of splitting your soul, rather than the mere attrition of noticing that you didn't like the consequences. At least, if you wanted to destroy the Horcrux using merely the touchy-feely power of love. There might be a way to magically enhance the power of attrition, but, again, it probably requires raw materials or living things or a wand or focused discipline, none of which are especially likely to be available.

There is a scene somewhere in Book 1 where Voldemort complains about his loss of psychological identity as he lusts after the unicorn blood or the Sorcerer's Stone or something like that. I forget if he was manipulating Quirrell or just boasting about how far he had risen or how desperate he was not to return to that state. Unfortunately, I can't find the exact quote because JKR has totally blocked GoogleBooks search, my book is elsewhere, and there are limits to my Potter-nerddom.

Also,

Riddle also reveals that he is Voldemort as a boy. He further explains that he learned from Ginny who Harry was and about his own deeds as Voldemort. ~Wikipedia, book 2