Judging Aaron Swartz's decision-making capabilities in light of his suicide is a form of survivorship bias (pun not intended, I am terribly sorry). It's not at all clear that the outcome of his decisions was a predictable consequence of them.

On a societal level, it's both useful and inspiring to have some people take more risks than others (risk-taking people sometimes bring large benefits to society, risk-averse people get inspiration and some Bayesian evidence about what works from looking at what risk-taking people do). This is probably why whatever personality traits correlate with risk-taking are heritable and vary within populations, and it suggests the reasonable hypothesis that optimal group rationality doesn't necessarily mean that each individual in the group has optimal individual rationality.

I encountered a definition for the different concepts in one of my programming classes, while discussing AI (the trivial kind of AI in that case, not AGI), which has stuck with me:

Knowledge is the ability to more usefully apply data. Intelligence is the ability to more usefully apply knowledge. Wisdom is the ability to more usefully apply intelligence.

A key point is that each level ability becomes more generalized; knowledge applies to a wide range of data. Intelligence applies to a wide range of knowledge, and hence a very wide range of data. And wisdom applies to a wide range of intelligence, etc. A key point is that "more usefully" may indicate -not- working on a problem.

While hierarchical, each level is not necessarily dependent on the layers beneath it; one can be wise without intelligence, or intelligent without knowledge, or knowledgeable without data. More people have knowledge about quantum physics than have data about it; they can know the results of an experiment, and what it means, without looking at the data produced by that experiment. Whether or not this is a good thing depends on your perspective.

ETA: If this definition doesn't seem meaningful, try this instead, where data includes process definitions: Knowledge is data about data. Intelligence is data about data about data. Wisdom is data about data about data about data. You can have false knowledge, false intelligence, and false wisdom, just as much as you can have false data. Just as it is difficult to tell good data from bad without knowledge, it is difficult to tell good knowledge from bad without intelligence, and good intelligence from bad without wisdom. Experimentation in each case can also suffice, but is difficult.

To be succinct:

Wisdom is the development of good heuristics.

I've been thinking about this recently, noticeing that my more experienced coworkers have a good deal of expereince that tends to trump my intelligence and rationality in some cases.

I think you could relate this to the observation in machine learning that a mediocre algorthm trained on 10 times as much data is often superior to a more sophisticated one with less data.

As for appliability to FAI, it might generalize and account for the implications of this in the usual consequentialist fashion.

As far as I can tell, "wisdom" is just a word that refers to the sort of knowledge that (1) defines the person being described as high status, and (2) is the result of extensive experience. When I imagine someone as "wise", I think of the person looking rather eminent, and most likely sort of old simply because long stretches of experience require long stretches of living--that is being at least somewhat old.

So we know that many smart people make stupid (at least in retrospect) decisions. What these people seem to be lacking, at least at the moment they make a poor decision, is wisdom ("judicious application of knowledge").

Many people we would label as "smart" make decisions we end up labeling "stupid". This doesn't seem very remarkable. When I think of the word "smart", what comes to mind is a comparatively high mental ability in certain subjects, or someone who's demonstrated a comparatively high likelihood of coming to interesting insights, or getting good at something requiring strong mental ability, such as chess. Someone meeting that criteria making a decision we end up calling "stupid" seems no more interesting than someone we call "athletic" getting injured.

You're saying these people--those who we would be likely to label as "smart", yet sometimes make decisions we would likely call "stupid"--what they're missing is "wisdom". This makes it sound like 'wisdom' is some sort of component they're missing, as if this insight would put us on some sort of useful quest, analogous to being told that the way by which to open this box we want to open is "to find the key, which is somewhere in this house" (a clue).

Well, I would rephrase what you're saying as the completely unremarkable observation that someone we would likely call "smart", if they were to make a series of stupid decisions, we would probably be unlikely to call them "wise". This is a fact about how we employ English words, nothing more. Part of the meaning of "wise" seems to be consistency. Someone erratic, yet "smart", we would be unlikely to refer to by the word "wise". I don't see how this observation could generate any useful hypotheses pertaining to building FAI, or anything like that. As it doesn't seem to concern anything but definitions, the only application to FAI would, as far as I can tell, be one of suggesting which FAI to call "wise", and which to not--a rather uninteresting conversation indeed.

Clearly, if one created a human-level AI, one would want it to "choose wisely".

Here I just want to point out that although you transitioned to this sentence as if it was part of your general point, it should be mentioned that although the grammar may suggest that "wisely" in "choose wisely" is a conjugation of "wisdom" or "wise", it seems to be a slightly different word. 'Choosing wisely' just seems to be choosing based on calm, rational deliberation, like in telling someone to "choose wisely" one is suggesting they not be hasty. It doesn't seem to suggest anything pertaining to extensive experience, or anything like that, as the words "wise" and "wisdom" do.

Call me pedantic, but I'm just trying to show how slippy words can be, and the sort of care that's necessary to not get sucked into shuffling around words to no real purpose.

However, as human examples show, wisdom does not come for free with intelligence

You mean calling someone "smart" doesn't mean it would be tautological to call them "wise", as in the classic example of calling someone a "bachelor" meaning it would be a tautology to call them an "unmarried man"? Yeah, that much is obvious. Wisdom seems to suggest consistency, but plenty of people we call "intelligent" are rather erratic in certain respects, to no contradiction of that label. Again, I see no interesting insight here. We're still just discussing English-language conventions.

Actually, we usually don't trust intelligent people nearly as much as we trust wise ones (or appearing to be wise, at any rate).

Yeah, because consistency is a component of the common definition of "wise". We trust people we would consider consistent more than those we wouldn't label with that word.

For example, Aaron Swartz was clearly very smart, but was it wise of him to act they way he did, gambling on one big thing after another, without a clear sense of what is likely to happen and at what odds?

Again, there is an equivocation going on with this sort of transition. Although related in meaning, and sharing the same sequence of characters, the word "wise" in the question "was it wise of him" seems to be of a different meaning than the word "wise" in referring to Aaron as a "wise elder". The question "was it wise of him" seems no more than just asking whether it was a good idea, whereas the idea of being a "wise elder" seems to be about his experience, etc. Again the definitions are just being moved around in a word shuffle that doesn't seem to be getting us anywhere.

What algorithms/decision theories make someone wiser?

I don't know, ones that make them more consistent? Or ones that signal higher social status, or allow them to react more calmly when confronted with shocking situations? As with the rest of your post, you seem to be just asking questions about definitions, or making statements about how we use certain words. I can't seem to find any real, useful content in your post. It seems like no more than an exercise in messing around with definitions, masquerading as being in some way insightful.

Would someone implementing TDT or UDT to the best of their ability maximize their wisdom for a given intelligence/knowledge level?

I suspect part of what we ordinarily call "wisdom" involves having the right sort of utility function, which is not something that your decision theory can police. If someone were implementing TDT perfectly in order to fulfill their paperclip-maximizing desires, I doubt we would characterize them as wise.

It can be difficult to define Wisdom, but people generally recognize it when they encounter it.

In other words, it is an ill-defined concept that could mean many things, many of them possibly contradictory. Based on this non-definition I cannot see how you can draw so many conclusions. If I hand you an arbitrary agent, how can you measure its level of wisdom independent from its intelligence?