Yes, formalizing Friendliness is not the sort of thing you'd want one person doing. I agree. I don't consider that "philosophy", and it's the sort of thing other FAI team members would have to be able to check. We probably want at least one high-grade actual cryptographer.

Of the others, the nonperson predicate and the moral-progress parts are the main ones where it'd be unusually hard to solve and then tell that it had been solved correctly. I would expect both of those to be factorable-out, though - that all or most of the solution could just be published outright. (Albeit recent experience with trolls makes me think that no insight enabling conscious simulations should ever be published; people would write suffering conscious simulations and run them just to show off... how confident they were that the consciousness theory was wrong, or something. I have a newfound understanding of the utter... do-anything-ness of trolls. This potentially makes it hard to publicly check some parts of the reasoning behind a nonperson predicate.) Anthropic reasoning / "reality fluid" is the sort of thing I'd expect to be really obvious in retrospect once solved. R1 and R2 should be both obvious in retrospect, and publishable.

I have hopes that an upcoming post on the Lob Problem will offer a much more concrete picture of what some parts of the innards of FAI development and formalizing look like.

A tree falls in a forest with no-one to hear it. Does it make a sound?

Order-dependence and butterfly effects - knew about this and had it in mind when I wrote CEV, I think it should be in the text.

Counterfactual Mugging - check, I don't think I was calling TDT a complete solution before then but the Counterfactual Mugging was a class of possibilities I hadn't considered. (It does seem related to Parfit's Hitchhiker which I knew was a problem.)

Solomonoff Induction - again, I think you may be overestimating how much weight I put on that in the first place. It's not a workable AI answer for at least two obvious reasons I'm pretty sure I knew about from almost-day-one, (a) it's uncomputable and (b) it can't handle utility functions over the environment. However, your particular contributions about halting-oracles-shouldn't-be-unimaginable did indeed influence me in toward my current notion of second-order logical natural induction over possible models of axioms in which you could be embedded. Albeit I stand by my old reply that Solomonoff Induction would encompass any computable predictions or learning you could do about halting oracles in the environment. (The problem of porting yourself onto any environmental object is something I already knew AIXI would fail at.)

X being a surprisingly simple counterargument. Not a new one, necessarily, just one that the text either fails to address or quietly dismisses without actually refuting it.

An annoying amount of philosophy is guys sitting in armchairs going "but what if contradictory things can be true? Didn't think of that, didja?"

EDIT:

For example:

Swampman, original formulation:

"it can't recognize anything, because it never cognized anything in the first place."

(Based on his glass marble thought experiment, Davidson defines "recognize" causally; that is, whatever caused the idea of the marble is what he means by "the marble", even if it has changed shape or been replaced with a copy. A nonstandard but useful definition, which implies we can be mistaken about what we mean/recognize/etc.)

Swampman is, of course, essentially a Boltzmann brain; he was formed by chance into a form that happens to be conscious. But more than that, he has been formed into a copy of someone else. Now, if he was formed entirely by chance, then an observer should not consider his beliefs to be correct, any more than a random number generator can reliably predict the weather. In this sense the concept of "recognize" is working perfectly; the author is just being mean.

However, the reader of course knows that Swampman is causally connected to Davidson, via the author's fiat; thus our intuitions in this case are closer to a clone than a true Boltzmann brain. And if a character in the thought experiment were to notice that Swampman was a perfect copy (which seems reasonable) they would have sufficient evidence postulate the author. Or at least some other god-level power screwing with them. In this sense Swampman is causally connected to Davidson - who really did "cognize" his friends - and thus genuinely "recognizes" them."

And, of course, an unfortunate amount of philisophy magically disappears when you start playing taboo and otherwise avoiding arguments over definitions.

SECOND EDIT:

OK, I can't reply to comments here due to the karma toll. If anyone's interested, I could post the example on an open thread for criticism?

EDIT: there goes another conversation. Thank you karma toll.

You made a claim that I - and, you yourself acknowledged, most LWers - had a low prior for.

i know. But you are not supposed to downvote for disagreement.

I'm not downvoting for disagreement, I'm downvoting for absurd claims without any damn evidence. If you had provided, say, an example of a LW user who is better at philosophy - as opposed to a terminology quibble - then I would not have downvoted even if I didn't think it was sufficient.

Well, I was impressed by LW, so there's naturally going to be some correlation.

Level meaning how good he seems at evaluating philosophical arguments, identifying flaws, and mostly avoiding the flaws that seem especially common in philosophy.

And i would say he isnot good at coming to clear conclusions,

I think you misspelled "correct" there.

examing all the relevant arguments,

I always considered phiosophers far to willing to spend ages dealing with some BS position that could be demolished in seconds if they were trying to win and not just "do philosophy" for as long as possible, but whatever.

aviding known errors,

If you have spotted an error, I advise you to point it out. Not just make vuage statements about how there are totally loads of errors.

avoiding hubris,

Truly, the gods shall punish him for his arrogance.

using standard terminology, etc.

I suppose standard terminology might help people critique his work, at that. Still, it seems a minor flaw TBH.

One piece of evidence, which, as I noted, is at best extremely weak and misleadingly labelled.

I fail to see what was misleading.

You called it EY "thanking real philosophers for fixing his arguments" which, in the context of you saying that there were better philosophers on LW, rather implies that their skill at philosophy allowed them to identify a mistake well known to, well, experts in the field; not them interpreting his unclear terms charitably, which while a good thing is hardly relevant.

So? Philosophy is mostly arguing, and arguing more clearly is arguing better...is better philosophy. Are you saying the link was irrelevant or what?

I think you accidentally quoted your self instead of me there.

"Rhetorical" is a rather rtheorical way of saying "clear". EY can claim to have a Wonderful Solution in his head that no-one can undestand, but thre is no reason anyone else shouild buy that. To solve a probllem is to publish a solution that persuades others, for all practical purposes.

Um, no. To solve a problem is to find the correct solution, which may or may not require persuading others that it is correct, or indeed that it is incorrect or that they shouldn't eat tuna.

But clarity has nothing to do with "persuasion" in the advertiser's or politician's sense. Clarity is not a varnish that makes bad arguments look good: clarification is a major part of the substance of philosophical solution.

Considering how unclear to a layman much philosophy is, I assume you mean that "clarity" is helping experts to find any mistakes you may have made. This is important, and indeed a separate discipline from what we might call the Dark Arts. Eliezer could probablydo with some more of this, yeah. OTOH, philosophers generally seem pretty bad at this, so I'm not sure if Eliezer is unusually bad. And it's certainly not the only or even main thing philosophers are supposed to be good at.

But not, I suspect, simply because you're contradicting the consensus. You have made sweeping claims, not considered likely by most LWers, regarding, among other things, users on this site (which we can all see,) yet you failed to provide examples; your lack of examples is at best unhelpful and at worst ... well dishonest or symptomatic of massive bias, whichever of those is "worst".

I argue my points.

Did you quote the wrong thing here too? I'm not sure how this link is relevant.

You did not provide any evidence for this claim.

ETA:

What would evidence that someone is bad at something look like?

Evidence that someone is unusually bad would look like lots of people being better. So would evidence that "A number of better-than-EY philosophers have visited this site, sometimes displaying their skills in discussions with EY."

If you don't know it other than by name, that's not my problem. It was straightforward mathematics.

When did I claim no one at SI held your views on anthropics? And I really don't think anthropics could be called straightforward by anyone.

There are pathological, intuitively confusing cases such as the number example; reading a number is incredibly selective for it being that number, so the update, in fact, does pull the probability up.

Congratulations, you understood the point. Similarly, decent arguments are highly diagnostic of philosophical ability because most people couldn't make an even half-assed argument if they sat down and spent all day at it; by LW standards, most philosophy grads can't find their asses either, and that's a very selective filter as well (philosophy majors are the highest-scoring group on the GRE for both verbal and writing sections, and are around 4 or 5 for the math section below physics & mathematics as one would expect). Making an argument that doesn't instantly fail is sadly so rare that just seeing one moves you a long way towards '1 in a million'.

Yudkowsky is not exactly Ayn Rand level popular, is he? If that's what you're after, pick anyone more famous than Yudkowsky and you're done. Easy.

I never said that fame scaled smoothly with importance. If I had to put the cutoff where fame stops adding additional evidence, I think I'd put it somewhere upwards of a Wikipedia article.

Look back up, Muehlhauser has stated that it is a tough task choosing someone of Yudkowsky's level of philosophical ability.

Sure. You're fishing from a limited pool to begin with: there aren't many professional philosophers these days, their numbers are probably shrinking as humanities programs get pressured. To put some numbers in perspective: the annual East coast meeting of the American Philosophical Association (APA) is the biggest single gathering of philosophers (tenured professors, associates, adjuncts, grad students, etc) in the world as far as I know. It numbers ~2000 attendants. Making things even more difficult, if I were one of them, I doubt I would spend much time on MIRI/FHI-related issues even if I were a true believer: it'd be way too risky for my already extremely precarious career. (Recruiting-wise, it might be best just to try to find computer science people and have them try their hand at philosophy; there's a lot of them, they're almost as smart in general, they have direct familiarity with a lot of the issues, they'll have the right intuitions about things like 'computers really are just machines that do what the programs say', and funding is a lot easier for them.)

By the way there's a pattern, various Ayn Rands and Keith Ranieres and Ron Hubbards and other self improvement gurus slash philosophers slash world saviours are popular philosophers among non-philosophers but not recognized by other philosophers.

I've actually never heard of Keith Raniere despite growing up in NY and visiting RPI; Wikipedia doesn't do a good job of describing what's so bad about it... ("Expensive brainwashing"? Brainwashing doesn't work, that's why cults have annual attrition rates in the double-digits.)

Anyway; yes, I would agree that the previous points also increase the chance EY would fall into that category of frauds. After all, such frauds are also pretty rare, so it's hardly impossible for evidence to increase our beliefs both that EY is a good philosopher and also such a fraud.]

(An example: houses catching on fire are rare. Houses not on fire with red spotlights around them are also rare. If I see in the sky above a house in the woods flickering red light, this is consistent with both the house being on fire and them having set up spotlights for a party; and my beliefs in the possibility of a fire and the possibility of spot spotlights will both increase quite a bit even though they're mutually exclusive scenarios.)