While I understand that the code/brain analogy is an analogy, I think you are significantly underplaying the dangers of doing this in a code base you do not understand.
Brain code doesn't crash, and the brain isn't capable of locking in a tight loop for very long; there are plenty of hardware-level safeguards that are vastly better than anything we've got in computers. Remember, too, that brains have to be able to program themselves, so the system is inherently both simple and robust.
In fact, brains weren't designed for conscious programming as such. What "mind hacking" essentially consists of is deliberately directing the brain to information that convinces it to make its own programming changes, in the same way that it normally updates its programming -- e.g. by noticing that something is no longer true, a mistake in classification has been made, etc. (The key being that these changes have to be accomplished at the "near" thinking level, which operates primarily on simple sensory/emotional patterns, rather than verbal abstractions.)
In a sense, to make a change at all, you have to convince the brain that what you are asking it to change to will produce better results than what it's already doing. (Again, in "near", sensory terms.) Otherwise, it won't "take" in the first place, or else it will revert to the old programming or generate new programming once you get it "in the field".
I don't mean you have to convince the person, btw; I mean you have to convince the brain. Meaning, you need to give it options that lead to a prediction of improved results in the specific context you're modifying. In a sense, it'd be like talking an AI into changing its source code; you have to convince it that the change is consistent with its existing high-level goals.
It isn't exactly like that, of course -- all these things are all just metaphors. There isn't really anything there to "convince", it's just that what you add into your memory won't become the preferred response unless it meets certain criteria, relative to the existing options.
Truth be told, though, most of my work tends to be deleting code, not adding it, anyway. Specifically, removing false predictions of danger, and thereby causing other response options to bump up in the priority queue for that context.
For example, suppose you have an expert system that has a rule like "give up because you're no good at it", and that rule has a higher priority than any of the rules for performing the actual task. If you go in and just delete that rule, you will have what looks like a miraculous cure: the system now starts working properly. Or, if it still has bugs, they get ironed out through the normal learning process, not by you hacking individual rules.
I suppose what I'm trying to say is that there isn't anything I'm doing that brains can't or don't already do on their own, given the right input. The only danger in that, is if you say, motivated yourself to do something dangerous without actually knowing how to do that thing safely. And people do that all the time anyway.
Followup to: Sunk Cost Fallacy
Related to: Rebelling Against Nature, Shut Up and Do the Impossible!
(expanded from my comment)
"The world is weary of the past—
O might it die or rest at last!"
— Percy Bysshe Shelley, from "Hellas"
Probability theory and decision theory push us in opposite directions. Induction demands that you cannot forget your past; the sunk cost fallacy demands that you must. Let me explain.
An important part of epistemic rationality is learning to be at home in a material universe. You are not a magical fount of originality and free will; you are a physical system: the same laws that bind the planets in their orbits, also bind you; the same sorts of regularities in these laws that govern the lives of rabbits or aphids, also govern human societies. Indeed, in the last analysis, free will as traditionally conceived is but a confusion—and bind and govern are misleading metaphors at best: what is bound as by ropes can be unbound with, say, a good knife; what is "bound" by "nature"—well, I can hardly finish the sentence, the phrasing being so absurd!
Epistemic rationality alone might be well enough for those of us who simply love truth (who love truthseeking, I mean; the truth itself is usually an abomination), but some of my friends tell me there should be some sort of payoff for all this work of inference. And indeed, there should be: if you know how something works, you might be able to make it work better. Enter intrumental rationality, the art of doing better. We all want to better, and we all believe that we can do better...
But we should also all know that beliefs require evidence.
Suppose you're an employer interviewing a jobseeker for a position you have open. Examining the jobseeker's application, you see that she was expelled from four schools, was fired from her last three jobs, and was convicted of two felonies. You ask, "Given your record, I regret having let you enter the building. Why on Earth should I hire you?"
And the jobseeker replies, "But all those transgressions are in the past. Sunk costs can't play into my decision theory—it would hardly be helping for me to go sulk in a gutter somewhere. I can only seek to maximize expected utility now, and right now that means working ever so hard for you, O dearest future boss! Tsuyoku naritai!"
And you say, "Why should I believe you?"
And then—oh, wait. Just a moment, I've gotten my notes mixed up—oh, dear. I've been telling this scenario all wrong. You're not the employer. You're the jobseeker.
Why should you believe yourself? You honestly swear that you're going to change, and this is great. But take the outside view. What good have these oaths done for all the other millions who have sworn them? You might very well be different, but in order to justifiably believe that you're different, you need to have some sort of evidence that you're different. It's not a special question; there has to be something about your brain that is different, whether or not you can easily communicate this evidence to others with present technology. What do you have besides the oath? Are you doing reasearch, trying new things, keeping track of results, genuinely searching at long last for something that will actually work?
For if you do succeed, it won't have been a miracle: you should be able to pin down at least approximately the causal factors that got you to where you are. And it has to be a plausible story. You won't really be able to say, "Well, I read all these blogposts about rationality, and that's why I'm such an amazing person now." Compare: "I read the Bible, and that's why I'm such an amazing person now." The words are different, but translated into math, is it really a different story? It could be. But if it is, you should be able to explain further; there has to be some coherent sequence of events that could take place in an material universe, a continuous path through spacetime that took you from there to here. If the blog helped, how specifically did it help? What did it cause you to do that you would not otherwise have done?
This could be more difficult than it now seems in your current ignorance: the more you know about the forces that determine you, the less room there is for magical hopes. When you have a really fantastic day, you're more likely to expect tomorrow to be like that as well if you don't know about regression towards the mean.
I'm not trying to induce despair with this post; really, I'm not. It is possible to do better; I myself am doing better than I was this time last year. I just think it's important to understand exactly what doing better really involves.
I feel bad blogging about rationality, given that I'm so horribly, ludicrously bad at it. I'm also horribly, ludicrously bad at writing. But it would hardly be helping for me to just shut up in despair—to go sulk in a gutter somewhere. I can only seek to maximize expected utility now, and for now, that apparently means writing the occasional blogpost. Tsuyoku—