All of bryjnar's Comments + Replies

Often the effect of being blinded is that you take suboptimal actions. As you pointed out in your example, if you see the problem then all sorts of cheap ways to reduce the harmful impact occur to you. So perhaps one way of getting to the issue could be to point at that: "I know you care about my feelings, and it wouldn't have made this meeting any less effective to have had it more privately, so I'm surprised that you didn't"?

Wireheading traps.

An agent is "wireheading" if it is taking an action that a) provides it with enormous amounts of utility for little effort, b) is trivial or seemingly unrelated to its "main" utility function or goals.

People have discussed the possibility of an AI wireheading as a problem for getting it to do what you want, "what if it just works out a way to set your metric to +ve infinity and then goes to sleep satisfied?".

But we can also use this as a guard-rail.

A "wireheading trap" is an action which a) is very hard for an AI to do below a level of ca... (read more)

We have trained ML systems to play games, what if we trained one to play a simplified version of the "I'm an AI in human society" game?

Have a population of agents with preferences, the AI is given some poorly specified goal, it has the ability to expand its capabilities etc. You might expect to observe things like a "treacherous turn".

If we could do that it would be quite the scary headline "Researchers simulate the future with AI and it kills us all". Not proof, but perhaps viral and persuasive.

I think I would argue that harm/care isn't obviously deontological. Many of the others are indeed about the performance of the action, but I think arguably harm/care is actually about the harm. There isn't an extra term for "and this was done by X".

That might just be me foisting my consequentialist intuitions on people, though.

"What if there's an arms race / race to the bottom in persuasiveness, and you have to pick up all the symmetrical weapons others use and then use asymmetrical weapons on top of those?"

Doesn't this question apply to other cases of symmetric/asymmetric weapons just as much?

I think the argument is that you want to try and avoid the arms race by getting everyone to agree to stick to symmetrical weapons because they believe it'll benefit them (because they're right). This may not work if they don't actually believe they&#x... (read more)

The point that the Law needs to be simple and local so that humans can cope with it is also true of other domains. And this throws up an important constraint for people designing systems that humans are supposed to interact with: you must make it possible to reason simply and locally about them.

This comes up in programming (to a man with a nail everything looks like a hammer): good programming practice emphasises splitting programs up into small components that can be reasoned about in isolation. Modularity, compositionality, abstraction, etc. aside from t... (read more)

I am reminded of Guided by the Beauty of our Weapons. Specifically, it seems like we want to encourage forms of rhetoric that are disproportionately persuasive when deployed by someone who is in fact right.

Something like "make the structure of your argument clear" is probably good (since it will make bad arguments look bad), "use vivid examples" is unclear (can draw people's attention to the crux of your argument, or distract from it), "tone and posture" are probably bad (because the effect is symmetrical).

So a good test ... (read more)

Yes, this is very annoying.

I found Kevin Simmler's observation that an apology is a status lowering to be very helpful. In particular, it gives you a good way to tell if you made an apology properly - do you feel lower status?

I think that even if you take the advice in this post you can make non-apologies if you don't manage to make yourself lower your own status. Bits of the script that are therefore important:

• Being honest about the explanation, especially if it's embarassing.
• Emphasise explanations that attribute agency to you - "I just didn't think about i

This is a great point. I think this can also lead to cognitive dissonance: if you can predict that doing X will give you a small chance of doing Y, then in some sense it's already in your choice set and you've got the regret. But if you can stick your fingers in your ears enough and pretend that X isn't possible, then that saves you from the regret.

Possible values of X: moving, starting a company, ending a relationship. Scary big decisions in general.

Something that confused me for a bit: people use regret-minimization to handle exporation-ex... (read more)

I've read it shallowly, and I think it's generally good. I think I'll have some more comments after I've thought about it a bit more. I'm surprised either by the lack of previous quantitative models, or the lack of reference to them (which is unsurprising if they don't exist!). Is there really nothing prior to this?

I would dearly, dearly love to be able to use the fairly-standard Markdown footnote extension.

Fantastic post, I think this is right on the money.

Many more Newcomblike scenarios simply don't feel like decision problems: people present ideas to us in specific ways (depending upon their model of how we make choices) and most of us don't fret about how others would have presented us with different opportunities if we had acted in different ways.

I think this is a big deal. Part of the problem is that the decision point (if there was anything so firm) is often quite temporally distant from the point at which the payoff happens. The time when you &quo... (read more)

You cannot possibly gain new knowledge about physics by doing moral philosophy.

This seems untrue. If you have high credence in the two premisses:

• If X were a correct physical theory, then Y.
• Not Y.

then that should decrease your credence in X. It doesn't matter whether Y is a proposition about the behaviour of gases or about moral philosophy (although the implication is likely to be weaker in the latter case).

Constructivist logic works great if you interpret it as saying which statements can be proven, or computed, but I would say it doesn't hold up when interpreted as showing which statements are true (given your axioms). It's therefore not really appropriate for mathematics, unless you want to look at mathematics in the light of its computational or proof-theoretic properties.

Dialethists requires paraconsistent logic, as you have to be able to reason in the presence of contradictions, but paraconsitent logic can be used to model other things than truth. For example, constructive logic is often given the semantics of showing what statements can be proven, rather than what statements are true. There are similar interpretations for paraconsistent logic.

OTOH, if you think that paraconsistent logic is the correct logic for truth, then you probably do have to be a dialethist.

That's pretty weird, considering that so-called "sophisticated" consequentialist theories (where you can say something like: although in this instance it would be better for me to do X than Y, overall it would be better to have a disposition to do Y than X, so I shall have such a disposition) have been a huge area of discussion recently. And yes, it's bloody obvious and it's a scandal it took so long for these kinds of ideas to get into contemporary philosophy.

Perhaps the prof meant that such a consequentialist account appears to tell you to foll... (read more)

I agree that "right for the wrong reasons" is an indictment of your epsitemic process: it says that you made a prediction that turned out correctly, but that actually you just got lucky. What is important for making future predictions is being able to pick the option that is most likely, since "being lucky" is not a repeatable strategy.

The moral for making better decisions is that we should not praise people who predict prima facie unlikely outcomes -- without presenting a strong rationale for doing so -- but who then happen to be corre... (read more)

Great post! I wish Harsanyi's papers were better known amongst philosophers.

Mainstream philosophy translation: moral concepts rigidly designate certain natural properties. However, precisely which properties these are was originally fixed by certain contingent facts about the world we live in and human history.

Hence the whole "If the world had been different, then what is denoted by "morality" would have been different, but those actions would still be immoral (given what "morality" actually denotes)" thing.

This position is sometimes referred to as "sythetic ethical naturalism".

I'm still worried about the word "model". You talk about models of second-order logic, but what is a model of second-order logic? Classically speaking, it's a set, and you do talk about ZF proving the existence of models of SOL. But if we need to use set theory to reason about the semantic properties of SOL, then are we not then working within a first-order set theory? And hence we're vulnerable to unexpected "models" of that set theory affecting the theorems we prove about SOL within it.

It seems like you're treating "model" a... (read more)

It's like the opposite of considering the Least Convenient Possible World; the Most Convenient Possible World! Where everything on my side turns out as well as possible, and everything on yours turns out as badly as possible.

I'm pretty sure that the idea of the previous two paragraphs has been talked about before, but I can't find where.

It's pretty commonly discussed in the philosophical literature on utilitarianism.

I think most of this worrying is dissolved by better philosophy of mathematics.

Infinte sets can be proven to exist in ZF, that's just a consequence of the Axiom of Infinity. Drop the axiom, and you can't prove them to exist. You're perfectly welcome to work in ZF-Infinity if you like, but most mathematicians find ZF to be more interesting and more useful. I think the mistake is to think that one of these is the "true" axiomatization of set theory, and therefore there is a fact of the matter over whether "infinite sets exist". There are ... (read more)

This definitely seems to be a post-metaethics post: that is, it assumes something like the dominant EY-style metaethics around here (esp the bit about "intrinsic moral uncertainty"). That's fine, but it does mean that the discussion of moral uncertainty may not dovetail with the way other people talk about it.

For example, I think many people would gloss the problem of moral uncertainty as being unsure of which moral theory is true, perhaps suggesting that you can have a credence over moral theories much like you can over any other statement you a... (read more)

Oh, I see. Sorry, I misinterpreted you as being sceptical about the normal usage of "purpose". And nope, I can't give a taboo'd account of it: indeed, I think it's quite right that it's a confused concept - it's just that it's a confused concept not a confused use of a normal concept.

I'd claim that there is a distinct concept of "purpose" that people use that doesn't entail an agent with that purpose. It may be a pretty unhelpful concept, but it's one that people use. It may also have arisen as a result of people mixing up the more sound concept of purpose.

I think you're underestimating people who worry about "ultimate purpose". You say they "don't even understand the context", as opposed to people who "understand the full context of the concept". I'm not sure whether you're just being a linguist... (read more)

"What's the point of that curious tool in your shed?"

"Oh, it's for clearing weeds."

The purpose of the tool is to clear weeds. This is pretty underdetermined: if I used it to pick my teeth then there would be a sense in which the purpose of the tool was to act as a toothpick, and a sense in which I was using it for a purporse unintended by its creator, say.

Importantly, this isn't supposed to be a magically objective property of the object, no Aristotelian forms here! It's just a feature of how people use or intend to use the object.

+1 nitpickiness.

And Eliezer makes the same mistake in the linked article too ;) Not that it exactly matters!

If we're naming fallacies, then I would say that this post commits the following:

The Linguistic Consistency Fallacy: claiming, implicitly or otherwise, that a word must be used in the same way in all instances.

A word doesn't always mean the same thing even if it looks the same. People who worry about the purpose of life aren't going to be immediately reassured once you point out that they're just missing one of the relata. "Oh, silly me, of course, it's a three-place relation everywhere else, so of course I was just confused when I was using it here&q... (read more)

By semantics I mean your notion of what's true. All I'm saying is that if you think that you can prove everything that's true, you probably have a an overly weak notion of truth. This isn't necessarily a problem that needs to be "fixed" by being really smart.

Also, I'm not saying that our notion of proof is too weak! Looking back I can see how you might have got the impression that I thought we ought to switch to a system that allows infinite proofs, but I don't. For one thing, it wouldn't be much use, and secondly I'm not even sure if there even is such a proof system for SOL that is complete.

Absolutely, but it's one that happens in a different system. That can be relevant. And I quite agree: that still leaves some things that are unknowable even by supersmart AI. Is that surprising? Were you expecting an AI to be able to know everything (even in principle)?

They explicitly don't address that:

Second, it might seem that this approach to determining Personal CEV will require a reasonable level of accuracy in simulation. If so, there might be concerns about the creation of, and responsibility to, potential moral agents.

Ooookay. The whole "loop" thing feels like a leaky abstraction to me. If you had to do that much work to explain the loopiness (which I'm still not sold on) and why it's a problem, perhaps saying it's "loopy" isn't adding much.

This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person

I think I may still be misunderstanding you, but this seems wrong. The whole point is that even if you're on some kind of weird drugs that make you think that drinking bleach would be great, the ide... (read more)

It is!? Does anyone know a proof of Compactness that doesn't use completeness as a lemma?

Yes. Or, at least, I did once! That's the way we proved it the logic course I did. The proof is a lot harder. But considering that the implication from Completeness is pretty trivial, that's not saying much.

Great post! It's really nice to see some engagement with modern philosophy :)

I do wonder slightly how useful this particular topic is, though. CEV and Ideal Avisor theories are about quite different things. Furthermore, since Ideal Advisor theories are working very much with ideals, the "advisors" they consider are usually supposed to be very much like actual humans. CEV, on the other hand, is precisely supposed to be an effective approximation, and so it would seem surprising if it were to actually proceed by modelling a large number of instance... (read more)

This comment reads to me like: "Haha, I think there are problems with your argument, but I'm not going to tell you what they are, I'm just going to hint obliquely in a way that makes me look clever."

If you actually do have issues with Sobel's arguments, do you think you could actually say what they are?

A lot of what you've said sounds like you're just reiterating what Luke says quite clearly near the beginning: Ideal Advisor theories are "metaphysical", and CEV is epistemic, i.e. Ideal Advisor theories are usually trying to give an account of what is good, whereas, as you say, CEV is just about trying to find a good effective approximation to the good. In that sense, this article is comparing apples to oranges. But the point is that some criticisms may carry over.

[EDIT: this comment is pretty off the mark, given that I appear to be unable to read the first sentence of comments I'm replying to. "historical context" facepalm]

I absolutely agree that this will help people stop being confused about Godel's theorem, I just don't know why EY does it in this particular post.

Do you have any basis for this claim?

Nope, it's pure polemic ;) Intuitively I feel like it's a realism/instrumentalism issue: claiming that the only things which are true are provable feels like collapsing the true and the knowable. In this case the decision is about which tool to use, but using a tool like first-order logic that has these weird properties seems suspicious.

Oh yeah - brain fail ;)

the compactness theorem is equivalent to the ultrafilter lemma, which in turn is essentially equivalent to the statement that Arrow's impossibility theorem is false if the number of voters is allowed to be infinite.

Well, I can confirm that I think that that's super cool!

the compactness theorem is independent of ZF

As wuncidunci says, that's only true if you allow uncountable languages. I can't think of many cases off the top of my head where you would really want that... countable is usually enough.

Also: more evidence that the higher model theory of first-order logic is highly dependent on set theory!

I think it's worth addressing that kind of argument because it is fairly well known. Penrose, for example, makes a huge deal over it. Although mostly I think of Penrose as a case study in how being a great mathematician doesn't make you a great philosopher, he's still fairly visible.

Exactly (I'm assuming by subset you mean non-strict subset). Crucially, a non-standard model may not have all the bijections you'd expect it to, which is where EY comes at it from.

Sure. So you're not going to be able to prove (and hence know) some true statements. You might be able to do some meta-reasoning about your logic to figure some of these out, although quite how that's supposed to work without requiring the context of set theory again, I'm not really sure.

This post doesn't really say anything interesting about retributive justice at all. It sounds like what's actually bugging you is the question of national "sovereignty". Plus, you Godwined yourself. Between these things, you give a pretty bad impression. Perhaps if you reposted it with a less flamebaity example and a title like "Is there any ethical reason to respect national sovereignty?" or something you might fare better.

A few things.

a) I'm a little confused by the discussion of Cantor's argument. As I understand it, the argument is valid in first-order logic, it's just that the conclusion may have different semantics in different models. That is, the statement "the set X is uncountable" is cashed out in terms of set theory, and so if you have a non-standard model of set theory, then that statement may have non-standard sematics.

This is all made horrendously confusing by the fact that when we do model theory we tend to model our domains using sets. So even in a n... (read more)

Right. So, as I said, you are counselling that "anthropics" is practically not a problem, as even if there is a sense of "expect" in which it would be correct to expect the Boltzmann-brain scenario, this is not worth worrying about because it will not affect our decisions.

That's a perfectly reasonable thing to say, but it's not actually addressing the question of getting anthropics right, and it's misleading to present it as such. You're just saying that we shouldn't care about this particular bit of anthropics. Doesn't mean that I wouldn't be correct (or not) to expect my impending dissolution.