The horrifying importance of domain knowledge

NancyLebovitz

There are some long lists of false beliefs that programmers hold. isn't because programmers are especially likely to be more wrong than anyone else, it's just that programming offers a better opportunity than most people get to find out how incomplete their model of the world is.

I'm posting about this here, not just because this information has a decent chance of being both entertaining and useful, but because LWers try to figure things out from relatively simple principles-- who knows what simplifying assumptions might be tripping us up?

The classic (and I think the first) was about names. There have been a few more lists created since then.

Time. And time zones. Crowd-sourced time errors.

Addresses. Possibly more about addresses. I haven't compared the lists.

Gender. This is so short I assume it's seriously incomplete.

Networks. Weirdly, there is no list of falsehoods programmers believe about html (or at least a fast search didn't turn anything up). Don't trust the words in the url.

Distributed computing Build systems.

Poem about character conversion.

I got started on the subject because of this about testing your code, which was posted by Andrew Ducker.

LWers try to figure things out from relatively simple principles

If you only do this you are going to fail. This is why Aristotle got physics so badly wrong, right? Reasoning has to be a cyclical process of compressing complex realities into sets of simple principles and only then are you licensed to decompress those principles into making statements about reality.

3NancyLebovitz11y

Thank you. That was there (plus some extras-- see the article now) in my original post at lj, but somehow it got lost as I was bullying the html to work here.

Addressing common misconceptions is worthwhile, and lists are a good way to do it. You can find similar lists in the academic literature for many different subjects, e.g., here's an article about common misconceptions in thermodynamics. I've also mentioned math books listing counterexamples before. Many counterexamples address common misconceptions.

I like to create Anki cards for common misconceptions, to make sure I don't makes these mistakes myself. One issue with this is whether correcting specific cases causes my brain to recognize a more general case.... (read more)

So while many of these false beliefs are worth noting, it's worth thinking why programmers make these mistakes in practice. While it might be the case that names can include numbers, it's probably also going to be the case that the majority of numbers get into names via user error. Depending on the purpose of your database, it might be more valuable to avoid user error, than avoid a minority of users being excluded.

The reason I mention this is a lot of things in life are a study in trade offs. Quantum mechanics isn't very good at describing how big things work, and classical mechanics isn't very good at describing how small things work.

Weirdly, there is no list of falsehoods programmers believe about html (or at least a fast search didn't turn anything up).

A lot of programmers believe they can parse HTML with regular expressions.

7DanArmak11y

A lot of programmers believe they can parse HTML at all. Go read the official W3C parser algorithm, I'll wait. First thing you'll notice is that there is no formal grammar - the spec is of the actual parser state machine. Then you notice each past-and-present HTML version has its own parser algorithm spec, and there is no official documentation on the differences between them, never mind rationale. Then you realize that HTML5 is now a "living spec", so the parser algorithm at that link occasionally changes, and past versions and changelogs are deliberately not published... HTML is a parseable format like PHP is a programming language. There is no spec, there is only whatever bugs and quirks a particular browser version happens to contain. (Oh, you thought browsers actually follow any of those published W3C specs? HAHAHAHAHA sob.)

-2eternal_neophyte11y

HTML is indeed a turd of a standard.

3fubarobfusco11y

Hey, can we try not invoking the blasphemous things that dangle from the staked corpses o̭̙̥͚̘͍̠f dead universes?

2DanArmak11y

If I were feeling snarky I'd add, "a lot of people believe they are programmers." (To adapt a quotation, "Anyone can code. That doesn't mean anyone should.")

From Falsehoods Programmers Believe About Names:

anything someone tells you is their name is — by definition — an appropriate identifier for them.

There should be a list of false things people coming from common law jurisdictions believe about how choice of identity works on the rest of the globe.

3NancyLebovitz11y

I'm not sure what's intended by "appropriate" there-- it might not be so much a claim about the law as a claim that it's a name the person wants to use and you shouldn't argue with them about it. Even then, impersonation is an issue..

Sometimes a user puts something in a "name" field that they do not actually intend to be used to identify themselves.

They may be trying to get that string displayed to other users in a highlighted fashion. If someone puts "Wal-Mart Sucks" in the name field on a blog comment, it isn't because they seriously want to be identified by the surname of Sucks. They're just saying that Wal-Mart sucks, in a dramatic way.

They may be trying to break the system in one way or another. If someone puts their name as "Robert'; drop table students; --" then depending on the social and technical context they might be giving themselves a clever alias; or they might be trying to attack the database.

3NancyLebovitz11y

All fair enough. There's also the possibility of accidentally entering wrong characters-- I assume this is unlikely since people should know how to type their names, but people have to type their names so much that even a low chance of fumble-fingers is going to occur now and then.

0TrE11y

Or their mom might be a hacker. Incidentally, there are many cases where I don't care about my username at all and have to come up with something. I'd find it acceptable if they'd just give me a number and a password, or let me register just with a password (perhaps provided by them?), maybe plus e-mail.

I think it's worth noting that, yes, if you want your database of names/addresses/times/etc. to be fully robust, you need to essentially represent these items as unconstrained strings of arbitrary length (including zero).

However, in practice, most likely you're not building a fully robust database. For example, you are not solving the problem of, "how can I fully represent all of the marvelous variety of human names and addresses ?", but rather, "how can I maximize the changes that the packages my company is shipping to customers will actual... (read more)

2Lumifer11y

As a nitpick -- yes, he can. In the third world it's not uncommon to NOT have a working system of usual addresses and locations are typically specified as town -- local landmark -- directions from that local landmark.

0Bugmaster11y

Right, I was thinking in the context of our Western society. But in the third world, as you said, the opposite is true: an address like "123 Main St., Sometown Somecountry" simply does not work. So it is still not the case that you need to implement a fully general address database that covers all possible cases; you only need to cover the cases that you personally care about.

-3VoiceOfRa11y

Of course, you're much less likely to be shipping a package to those kinds of third world countries, and even if you did, you'd have trouble making sure it gets delivered to its destination for other reasons.

Also, there is a place where the sun rises in the west and sets in the east.

(Gur fha evfrf orpnhfr bs gur Rnegu'f ebgngvba, ohg rira va gur nofrapr bs gur Rnegu'f ebgngvba vg jbhyq evfr naq frg bapr n lrne orpnhfr bs ubj gur Rnegu'f nkvf cbvagf. Vs lbh ner pybfr rabhtu gb gur Abegu Cbyr, gur ebgngvba unf artyvtvoyr pbagevohgvba gb gur fha'f evfvat naq frggvat naq gur fha evfrf naq frgf cerggl zhpu bayl orpnhfr bs gur nkvf. Tb gb gur Abegu Cbyr, frr jurer gur fha evfrf, naq tb njnl sebz gur cbyr n fubeg qvfgnapr va n qverpgvba fhpu gung guvf cbvag vf gb gur jrfg.)

A problem of believing something wrong could also be a problem of classification or problem of language. There is so much lost in translation and as far as a programmer is concerned they deal with a very formal language. So they are not as good at informal languages and nuances.

I'm not a programmer, so my view may be a little bit skewed.. but this seems like a list of things that may or may not be correct (and I'd like to put things in a continuum, or percentages, or some other more practically observable other than a list) but it doesn't really get any further than that. How substantial many of the claims are?

I honestly doubt "programmers" get all these wrong. I'm not going to link to a post (I've seen some people replying with a post as if it's a substitute for actually saying what's wrong) or even say that "prog... (read more)

5NancyLebovitz11y

You may be pointing out a problem with English. "Programmers" could simply mean more than one programmer, but there's an implication that all programmers have those problems or perhaps that the problems are pervasive. I get the impression that these are mistakes that someone has seen (or made) at least once. Some of them may mostly be made by programmers who are beginners. I agree that it would be nice to get percentages, even if there are large error bars. I think the main point of these lists is to warn programmers that they may need much more specific knowledge than they think they need. For what it's worth, the commenters on those lists are programmers, and I'm not seeing comments which say "That never happens!".

3ChristianKl11y

If I ask most programmer or people of other domains whether all months have more than 27 days I think most of them would say, of course all months have more than 27 days. That if wednesday is the second of a month the following friday out to be the 4th of the months. You actually need to know about the strangeness of september 1752 to know that there this one month in the calender who had less days. It had no days between the 2nd and the 14th. Children get taught in school that minutes have 60 seconds and not that there a body that decides 18 months in advance whether the minute is supposed to have 59, 60, 61 or 62 seconds.

0Gunslinger11y

The september 1752 example sounds like something you'll find on a trivia show. It's not really such a good example. It's the exception rather than the rule. When I read this I feel like I'm back in elementary school being the detail obsessed nerd. I can't say anything about the minute example but seeing the trend is to take some obscure occurrence, pointing fingers and saying "how can you not know that?" and looking like a special snowflake to every regular person. In practical terms, what are the merits of all those examples? Going back to the lists, some of them are probably bad design[1], like one example that a backup is a string of numbers 053901011991.html so let's not focus on them. [1] What constitutes "bad design" may vary; some people could probably easily filter through many files like that using ls. Some people prefer minimalism, others don't feel compelled to use their processing power so sparingly, and would rather get the job done more quickly. (It seems like this has some time implications. If you have cleaner code, you can work with it more easily in the future, if you just want a task done and forget about it) So if I were to describe "bad design" in a way that holds some water, I would say that it hurts productivity.

3ChristianKl11y

The whole point of the list is that there are exceptions to rules that most people consider to be true in all cases. If you program systems than you get bugs because of corner cases that you don't anticipate. You need domain knowledge to know all the corner cases. Leap seconds manage to crash real world computer systems because their designers didn't handle them properly. You don't want any software that has a calendar to crash simply because a user asks the system to show september 1752.

1Lumifer11y

Actually, the proper solution is to practice defensive programming, not trust the user input, and be generous with sanity checks. Failing gracefully is much easier when your software knows it's not in Kansas any more.

0HungryHobo11y

Which sounds nice right up until a production system shuts itself down gracefully a few hours before a daylights saving time switch purely because of tests which turn out to be more picky than the actual thing they're supposed to be protecting. Multi-byte characters can do surprising things to scripts designed to truncate logs written by someone who didn't take into account the maximum size of characters and chinese production server names. A lot of a programmers day can end up being related to fixing bugs due to incorrect assumptions or failing to take edge cases into account and knowing lots of edge cases and handling a reasonable portion of them right away is far better than making the most restrictive possible assumptions off the bat. You don't want to end up running into the Y2Gay problem: http://qntm.org/gay

0Lumifer11y

Do you know that "the system" can handle that input fine? If you do, why did you sanity check reject it? Sanity checks are just code -- you certainly can write bad ones. So? That post argues via mind-numbingly stupid strawmen (or should that be strawschemas?). Yes, you should try to be not stupid, most of the time, to the best of your ability. I agree :-/

-1Jiro11y

This is no different from asking someone the fastest route to get to the store and being told "go a mile down that road and take a left" even though the person didn't check to see if the road was temporarily blocked. If you ask someone directions, they're probably not going to add "unless the road isn't temporarily blocked" and "unless a meteor hit the store last night and I didn't know about it yet" and "unless there's a quantum fluctuation that will move all your molecules right next to the store".

0ChristianKl11y

In many cases Google Maps has disclaimers like that. In programming you usually do care more about edge cases then you care in daily life.

0Lumifer11y

Of course not. Many hackers are programmers, few programmers are hackers. People fitting esr's description are rare and cubicle peons with a keyboard and a screen are legion.

Could you turn the tracking links into direct ones?

you fail to distinguish between the question of whether to let people self identify themselves and whether people in hormone therapy really change their gender.

Well, I may or may not be mindkilled but I'm certainly confused. The only mention I made of hormone therapy was to suggest that in the special context of prisoners wanting to move to a different prison on account of gender transition, being on hormone therapy might be a useful criterion for being demonstrably serious. (Not for being actually male/female. The point is that it is evidence of the sincerity of the prisoner's claim, which we need because in this situation insincere claims might otherwise be a problem.)

So far as I can see,

the debate whether or not an XY transsexual with no penis, a vagina and big boobs is female or male

simply hasn't come up in this discussion, or at least not the bit of it you and I have been engaging in. Accordingly, I have no idea how anything I've written can suggest that I'm failing to distinguish that debate from any other debate. I also don't think the question was ever quite

whether or not the act of identifying as female means that your gender is female.

although obvious that isn't a mile from what Fluttershy was saying.

So it looks to me as if there's a failure of communication here. Let me attempt to say clearly and explicitly what I think the issues are. My apologies for the length of what follows.

Underlying question #1: under what circumstances, if any, is someone who is anatomically/chromosomally/hormonally male "really" female?
- I'm not sure this is the sort of question that has a definite answer. There are multiple somewhat defensible ways to use words like "male" and "female" and the answer could be different for them (even without bringing transgender into it; intersex conditions are pretty complicated).
Underlying question #2: how should we use terms like "male" and "female" in cases where there's divergence between those features and others such as the person's (expressed and/or internal) "gender identity"?
- My own opinion is that we get a healthier and happier society by usually taking "gender identity" as having priority over anatomy, chromosomes, etc., and accordingly that if someone identifies as (say) male then in most contexts we should treat that person as male.
- There are plenty of exceptions. Your example of prison might be one. So might competitive sports. So, for sure, might medical care.
Fluttershy's proposal was to answer question #2 with "according to expressed gender identity".
- Fluttershy didn't say anything about difficult cases where there's ground to suspect that expressed and internal gender identity might differ (e.g., someone trying to exploit the system), where taking someone at their word might be more than averagely dangerous (e.g., when the person in question is a violent criminal), etc.
- One possibility is that Fluttershy simply believes we should always, unconditionally, take everyone at their word on this issue. I think that is unlikely; in any case, if so then I disagree. (See above.)
- Another possibility is that Fluttershy was proposing a general policy, which might need modification in unusual cases (e.g., prison), but stated it briefly without all the caveats that it might need.
- My money is firmly on the second of these, and in particular I am fairly sure Fluttershy was not saying that the mere act of saying "I am a woman" makes someone a woman.
Your response, though, was to point to an unusual case as if it demonstrated the badness of Fluttershy's proposal.
- If you were taking Fluttershy to be claiming that we should always, unconditionally, take everyone at their word, then I think that is a needlessly uncharitable interpretation, enough so that I will gently suggest you consider that I might not be the only person in this discussion vulnerable to politics.
I pointed out that "hard cases make bad law"; that the fact that a policy of always taking people at their word might work badly when the people in question are serious criminals is little objection to a more reasonable policy of generally taking people at their word, while being more cautious in some special situations -- like prisons.
I made some concrete proposals for safeguards one might apply in prisons. As you say, those safeguards mean that we would not be simply and naively taking prisoners at their word as soon as they say "I'm a woman". They do, however, go a considerable way towards accommodating transgender prisoners.
- That doesn't mean that I regard the proposal with the safeguards as giving a sensible criterion in less dangerous situations. I don't.
One remark about the context for all this. Fluttershy was responding to a comment from VoiceOfRa which takes it for granted that what's going on with transgender people is (and I quote) "delusions and hallucinations", on a par with someone who believes himself to be simultaneously Jesus and John Lennon.
- I suggest that it is not reasonable to take Fluttershy's comment as meaning that the whole debate here is between "a person's gender is whatever they have most recently said it is" and everything else, any more than it is to take VoiceOfRa's comment as meaning that the whole debate here is between "a person's gender is determined by their chromosomes and nothing else" (or whatever, in fact, VoR's position is; if he's made that clear, then I haven't seen it). One comment was a response to the other. And my comment was attempting to address both.

that you simply want to be pro-LGBT instead of engaging with detailed arguments that distinguish different questions.

Well, you're entitled to your impression. I confess I'm unsure how you got it. In particular, I don't see that you made any such detailed arguments; in fact, the principal point of my original comment was that you failed to distinguish between "what should we do in general?" and "what should we do for the more difficult and dangerous case of imprisoned criminals?".

Wow, this discussion really took off! I mainly replied to VoiceOfRa's comment in order to encourage people to be nice to trans people in general, rather than to advocate any highly specific and well-thought-out position.

7Lumifer11y

I think the situation is a going to become bit more difficult if we confine out attention to the cases where gender/sex matters. Just declaring yourself to be a male/female/lizard overlord/banana is nothing particularly exciting and is likely to lead to shrugs or, at most, some eye-rolling. But once you demand some rights as a member of that particular group, the situation becomes much muddier. As you point out, there are "plenty of exceptions" to the my-gender/sex-is-whatever-I-say approach. The interesting question is whether most circumstances where gender/sex matters turn out to be such "exceptions".

LESSWRONG
LW

LESSWRONG
LW

24

The horrifying importance of domain knowledge

24

24