All of Peter Hase's Comments + Replies

Nice work! Two good points from the paper:

  • "Works should evaluate how their techniques perform on randomly or adversarially sampled tasks"
  • "...highlights a need for techniques that allow a user to discover failures that may not be in a typical dataset or easy to think of in advance. This represents one of the unique potential benefits of interpretability methods compared to other ways of evaluating models such as test performance"

But the biggest obstacle is probably just operational capacity.

I see. I know the team has its limits and has already been in a lot of work to propping up AF/LW, which is generally appreciated!

I think I am most confused what you mean by "access to the discourse".

I mean the ability to freely participate in discussion, by means of directly posting and commenting on threads where the discussion is occurring. Sorry for not making this clearer. I should have more clearly distinguished this from the ability to read the discussion, and the ability to partic... (read more)

5habryka
Sorry, I was suggesting a system in which instead of first posting to LW via the LW interface, you just directly submit to the AIAF, without ever having to think about or go to LW. Then, there is a submission queue that is only visible to some moderators of the AIAF that decides whether your content shows up on both LW and the AIAF, or on neither. This would make it more similar to classical moderated comment-systems. I think a system like this would be clearer to users, since it's relatively common on the internet, but would also have the problems I described. One specific problem with having a submission + admin-review system is that the user has to invest a lot of resources into writing a post, and then only after they invested all of those resources do they get to know whether they get any benefit from what they produced and whether their content (which they might have spent dozens of hours writing) is accepted. This is I think one of the primary things that creates a lot of resentment, and when I talk to people considering publishing in various journals, this is often one of the primary reasons they cite for not doing so. When designing systems like this, I try to think of ways in which we can give the user feedback at the earliest level of investment, and make incremental benefit available as early as possible. The current system is designed that even if your post doesn't get promoted to the AIAF, you will likely still get some feedback and benefit from having it on LW. And also, it tries to set expectations that getting a post onto the AIAF is more like a bonus, and the immediate level of reward to expect for the average user, is what you get from posting on LW, which in my experience from user-interviews causes people to publish earlier and faster and get more feedback before getting really invested, in a way that I think results in less resentment overall if it doesn't get promoted.  I do think some people see very little reward in posting to LW instead

Thanks for the informative reply! This clarifies a lot about the forum to me, and I'm glad you found the post helpful in some ways.

Let me also add on to some of the points above.

Use the Intercom button in the bottom right corner!

This is good to know about! I simply never knew that was a chat button, and I guess Owen and our mod intermediary didn't know about it since it didn't come up? I bet we could have saved a lot of trouble if we'd first talked through this a few months ago.

In particular, the application is not something we respond to The right t

... (read more)
habryka180

Second, I understand that LW fulfilled the crowdsourcing role here naturally at the inception of the AF. I think it's worth revisiting whether it best fulfills this role going forward. It seems like we both have PR concerns, for instance. Ignoring engineering and UI considerations for a second, why not split the AF into two channels: a channel for current members and a channel for new posters (where content from the New channel kept distinct from the main channel)? This way content promotion for AF is managed by AF members visiting AF and not by AF members

... (read more)
habryka160

This is good to know about! I simply never knew that was a chat button, and I guess Owen and our mod intermediary didn't know about it since it didn't come up? I bet we could have saved a lot of trouble if we'd first talked through this a few months ago.

The mod intermediary (Ben Pace) definitely knew, and Owen has also messaged us quite a few times on Intercom in the past, so not sure what exactly went wrong here. Looking at the chat logs, I think it was mostly a failure of process by the LW team, caused by FB messenger being such an informal chat-tool tha... (read more)