Things to look for in an experiment

Sometimes the outcome of an experiment is a very unexpected result for the researcher. This happens often in running imperial results on economic theories. Peter Kennedy lists the source of such problems and examples in his paper, “Oh No! I Got the Wrong Sign! What Should I Do?” by Peter Kennedy. Here is a summary of the most important things that a researcher should look for when running an experiment.

• real versus nominal: using real interest rate rather than nominal interest rate
• defining learning: “researchers regressed learning, measured as the difference between posttest and pretest scores, on the pretest score (as a measure of student ability) and other explanatory variables, obtaining a negative sign on pretest”. For example “the true specification may be that the posttest score depends on the pretest score with a coefficient less than unity. Subtracting pretest from both sides of this relationship produces a negative coefficient on pretest in the relationship connecting the score difference to the pretest score.”
• forward looking: for example having making predictions speculation when there is a forward looking in the data

I have to reiterate the quote that peter brought in his paper from Thaler’s (2000, 139) remark that “Some economists seem to feel that data-driven theory is, somehow, unscientific. Of course, just the opposite is true.”

Hoover (1995, 243) “ . . . data mining is misunderstood, and once it is properly understood, it is seen to be no sin at all.” Needless to say that the experiment design and researcher’s approach in arriving at a conclusion play a key role in making statistical analysis and data mining methods a useless endeavor or a powerful tool.