You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

fubarobfusco comments on Everybody's talking about machine ethics - Less Wrong Discussion

14 Post author: sbenthall 17 September 2014 05:20PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (16)

You are viewing a single comment's thread.

Comment author: fubarobfusco 18 September 2014 01:30:28AM *  9 points [-]

One of boyd's examples is a pretty straightforward feedback loop, recognizable to anyone with the slightest degree of systems engineering:

Consider, for example, what’s happening with policing practices, especially as computational systems allow precincts to distribute their officers “fairly.” In many jurisdictions, more officers are placed into areas that are deemed “high risk.” This is deemed to be appropriate at a societal level. And yet, people don’t think about the incentive structures of policing, especially in communities where the law is expected to clear so many warrants and do so many arrests per month. When they’re stationed in algorithmically determined “high risk” communities, they arrest in those communities, thereby reinforcing the algorithms’ assumptions.

This system — putting more crime-detecting police officers (who have a nontrivial false-positive rate) in areas that are currently considered "high crime", and shifting them out of areas currently considered "low crime" — diverges under many sets of initial conditions and incentive structures. You don't even have to posit racism or classism to get these effects (although those may contribute to failing to recognize them as a problem); under the right (wrong) conditions, as t → ∞, the noise (that is, the error in the original believed distribution of crime) dominates the signal.

The ninth of Robert Peel's principles of ethical policing is surprisingly relevant: "To recognise always that the test of police efficiency is the absence of crime and disorder, and not the visible evidence of police action in dealing with them." [1]