I am primarily referring to the unconscious drives underlying our actions, not our verbal goals. No matter what term I used to describe it, when I imagined myself doing very well in general relative to other people, spending every moment in focused and topical optimization, I was excited and driven to pursue the things I expected to make me like that. If I anticipated outcomes that did NOT involve me being that kind of person, there was far less unconscious drive to act.

Being hyper-competent was not a subgoal of programming or business, and if it were I would have your same critique. Being hyper-competent was a subgoal of having social success, having riches, being safe, a general assessment of "able to succeed even in difficult situations." Programming and business were rather what seemed consciously to be the best specific routes for achieving these things, but they involved not being the sort of hyper-competent person, and because I unconsciously desired that so much I was not, in practice, driven to pursue programming or business.

Similarly, unless you have already developed high competence in many concrete tasks, how would you recognise a mind that was a perfectly honed instrument for realizing your goals?

The term "perfectly honed instrument" is meant to convey an intuitive sense, not a technical description. But you would recognize such a person by them constantly engaging in what actually seemed to have the greatest marginal return on time, and probably by quickly developing unusually large amounts of skill.

Taboo "perfectly honed instrument", "hyper-competent" etc., and the goal dissolves.

Those terms refer to particular patterns of reality and not others - Bourne, rational!Quirrell, arguably rational!Dumbledore are all extensions of this intension. The average person is not.

Going up the pyramid of goals in such a context is an active hinderance, because the higher goals are harder to make operational.

By "going up the pyramid of goals" I'm referring to understanding more precisely the rules generating the particular, concrete situations we desire, and following a rule higher up on that pyramid. In other words, are there some things we could think of concretely, that once thinking of them, we realized were the real reason we had been motivated by something else was that we unconsciously anticipated it to lead to the first thing? This is something for each person to discover on their own, but it is something to discover.

"Beware affective death spirals on far-mode (sub)goals" or "Taboo specific terms in your goals to make them operationally useful" or possibly even "Check that your stated goals are not semantic stop-signs"

Are you doing those things already? Do they leave something left for you to desire in your rationality? These are all descriptions of much more surface-level techniques than what's being discussed here. This technique is about finding concrete things that make you think "hey, that's awesome, how can I get that?"

So, it seems to me that what you describe here is not moving up a hierarchy of goals, unless there are serious issues with the mechanisms used to generate subgoals. It seems like slogans more appropriate to avoiding the demonstrated failure mode are:

"Beware affective death spirals on far-mode (sub)goals" or "Taboo specific terms in your goals to make them operationally useful" or possibly even "Check that your stated goals are not semantic stop-signs"

As presented, you are claiming that:

I wanted to be a perfectly honed instrument for realizing my goals, similar to the hyper-competent characters in my favorite fictions

was generated as a subgoal of specific concrete goals (you mention programming and business). This seems to be a massive failure of planning. I would compare it to stating you would develop calculus to solve a constant speed distance-time problem, having never solved any of the latter sort of question. There is no shape to such a goal; to such an individual "calculus" is a term without content. Similarly, unless you have already developed high competence in many concrete tasks, how would you recognise a mind that was a perfectly honed instrument for realizing your goals? Taboo "perfectly honed instrument", "hyper-competent" etc., and the goal dissolves.

On the other hand, going up the pyramid of goals seems more likely to induce this error. Generally my high level goals are in farer modes and less concrete. Certainly "acquire awesome skills" is not something that I have generated as a subgoal of other goals; I have it as a generalisation of past methods of success, in the (inductive) belief that acquiring such skills will be useful in general. As subgoals to that I attempt general self improvement, for example learning to code in new languages or pushing other skillsets. Going up the pyramid of goals in such a context is an active hinderance, because the higher goals are harder to make operational.

Great post. The meet sounds awesome and touched on many things I'm interested in. I'd love to be on the same continent to attend these more regularly than once a year.

One possible typo: "This was followed by multiple passes for people to affiliated with any proposed topic".

Better directions to the JCR (with images) are here.

ETA: Also fixed the list of meetups to link there.

The foundational problem in your thesis is that you have grounded "rationality" as a normative "ought" on beliefs or actions. I dispute that assertion.

Rationality is more reasonably grounded as selecting actions so as to satisfy your explicit or implicit desires. There is no normative force to statements of the form "action X is not rational", unpacked as "If your values fall into {large set of human-like values}, then action X is not optimal, choosing for all similar situations where the algorithm you use is run".

There may or may not be general facts about what it is "rational" for "people" to do; it depends rather crucially on how consistent terminal values are across the set of "people". Neglecting trade with Clippy, it is (probably) not rational for humans to convert Jupiter to paperclips. Clippy might disagree.

It should be clear that rational actions are predicated on terminal values, and do not carry normative connotations. Given terminal values, your means of selecting actions may be rational or otherwise. Again, this is not normative; it may be suboptimal.

The thrust of your argument appears to be that: 1) Trolley problems are idealised 2) Idealisation can be a dark art rhetorical technique in discussion of the real world. 3) Boo trolley problems!

This is strange, this is the second comment that summarized an argument that I'm not actually making, and then argues against the made up summary.

My argument isn't against idealization - which would be an argument against any sort of generalized hypothetical and against the majority of fiction ever made.

No, my argument is that trolley problems do not map to reality very well, and thus, time spent on them is potentially conducive to sloppy thinking. The four problems I listed were perfect foresight, ignoring secondary effects, ignoring human nature, and constraining decisions to two options - these all lead to a lower quality of thinking than a better constructed question would.

There's a host of real world, realistic dilemmas you could use in place of a (flawed) trolley problem. Layoffs/redundancies to try to make a company more profitable or keep the ship running as is (like Jack Welch at GE), military problems like fighting a retreating defensive action, policing problems like profiling, what burden of proof in a courtroom, a doctor getting asked for performance enhancing drugs with potentially fatal consequences... there's plenty of real world, reality-based situations to use for dilemmas, and we would be better off for using them.

The thrust of your argument appears to be that: 1) Trolley problems are idealised 2) Idealisation can be a dark art rhetorical technique in discussion of the real world. 3) Boo trolley problems!

There are a number of issues.

First and foremost, reversed stupidity is not intelligence. Even if you are granted the substance of your criticisms of the activists position, this does not argue per se against trolley problems as dilemmas. The fact that they share features with a "Bad Thing" does not inherently make them bad.

Secondly, the whole point of considering trolley problems is to elucidate human nature and give some measure of training in cognition in stressful edge cases. The observation that humans freeze or behave inconsistently is important. This is why the trolley problems have to be trued in the sense that you object to - if they are not, many humans will avoid thinking about the ethical question being posed. In essence "I don't like your options, give me a more palatable one" is a fully general and utterly useless answer; it must be excluded.

Thirdly, your argument turns on the claim that merely admitting trolley problems as objects of thought somehow makes people more likely to accept dichotomies that "justify tyranny and oppression". This is risible. Even if the dichotomy is a false one, you surely should find one or the other branch preferable. It is perfectly admissible to say:

"I prefer this option (implicitly you presume that will be the taxation), but that if this argument is to be the basis for policy, then there are better alternatives foo, bar, etc., and that various important real world effects have been neglected."

Those familiar with the trolley problems and general philosophical dilemmas are more likely to be aware of the idealisations and voice these concerns cogently if idealisations are used in rhetoric or politics.

Fourthly, in terms of data, I would challenge you to find evidence suggesting that study of trolley problems leads to acceptance of tyranny. I would note (anecdotally) that communities where one can say "trolley problem" without needing to explain further seem to have a higher density of the libertarians and anarchists than the general population.

So in rough summary: 1) Your conclusion does not follow from the argument. 2)Trolley problems are idealised because if they aren't humans evade rather than engage. 3) Noting and calling out dark arts rhetoric is roughly orthogonal to thinking about trolley problems (conditional on thinking). 4) Citation needed wrt. increased tyranny in those who consider trolley problems.

Let's take the stock market as an example. The stock market prices are in principle predictable, only not from the data itself but from additional data taken from the newspapers or other sources. How does the CRM apply if the data does not in itself contain the neccessary information?

Let's say I have a theory that cutting production costs will increase stock prices in relation to the amount of cost cut and the prominence of the company and the level of fear of a crash on the stock market and the level of a "bad news indicator" that is a weighted sum of bad press for the company in the past. How would I test my theory with CRM?

Credence isn't constrained to be in [0,1]???

It seems to me that you are working very hard to justify your solution. It's a solution by argument/intuition. Why don't you just do the math?

The experimenters fix 2 unique constants, k1,k2, each in {1,2,..,20}, sedate you, roll a D20 and flip a coin. If the coin comes up tails, they will wake you on days k1 and k2. If the coin comes up heads and the D20 that comes up is in {k1,k2}, they will wake you on day 1.

I just used Bayes rule. W is an awakening. We want to know P(H|W), because the question is about her subjective probability when (if) she is woken up.

To get P(H|W), we need the following:

P(W|H)=2/20 (if heads, wake up if D20 landed on k1 or k2)

P(H)=1/2 (fair coin)

P(W|T)=1 (if tails, woken up regardless of result of coin flip)

P(T)=1/2 (fair coin)

Using Bayes rule, we get:

P(H|W)=(2/20)(1/2) / [(2/20)(1/2)+(1)*(1/2)] = 1/11

With your approach, you avoid directly applying Bayes' theorem, and you argue that it's ok for credence to be outside of [0,1]. This suggests to me that you are trying to derive a solution that matches your intuition. My suggestion is to let the math speak, and then to figure out why your intuition is wrong.

Continuity problem is that the 1/2 answer is independent of the ratio of expected number of wakings in the two branches of the experiment

Why is this a problem? I'm perfectly comfortable with that property. Since you really just have one random variable in each arm. You can call them different days of the week, but with no new information they are all just the same thing

By D do you mean W?

What's happened is closer to E(H|D) = E(D|H) E(H) / E(D), over one run of the experiment, and this yields 1/3 immediately.

Is this how you came up with the 1/3 solution? If so, I think it requires more explanation. Such as what D is precisely.