Daniel_Burfoot comments on Mathematical simplicity bias and exponential functions - Less Wrong

12 Post author: taw 26 August 2009 06:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (82)

You are viewing a single comment's thread. Show more comments above.

Comment author: Daniel_Burfoot 27 August 2009 02:41:15PM 2 points [-]

It seems totally reasonable to start with a simple model and add complexity necessary to explain the phenomenon.

Careful. It is reasonable to add complexity if the complexity is justified by increased explanatory power on a sufficiently large quantity of data. If you attempt to use a complex model to explain a small amount of data, you will end up overfitting the data. Note that this leaves us in a somewhat unpleasant situation: if there is a complex phenomenon regarding which we can obtain only small amounts of data, we may be forced to accept that the phenomenon simply cannot be understood.

Comment author: MichaelBishop 27 August 2009 05:27:18PM 2 points [-]

Yes, this is exactly the point I was getting at when I wrote: "Of course it is best to test the model on new data."