A presentation of research from Shingo Goto (Associate Professor of Finance, College of Business)
Abstract:
Researchers in many scientific disciplines are increasingly concerned about the probability of making erroneous decisions after testing a large number of hypotheses. Both false discoveries of spurious effects (false positives; Type I errors) and false omissions of true effects (false negatives; Type II errors) are inevitable outcomes of statistical tests. But a serious concern arises when researchers misuse or abuse a single-hypothesis testing rule, e.g. “rejecting the null hypothesis at the 5% significance level,” to report only positive results after testing multiple hypotheses”—a practice called the “p-hacking.” About a decade behind medical research, the multiple-testing problem now receives strong attention in many social science disciplines, where numerous researchers are hunting for predictive relationships in the jungle of Big Data.
P-hacking and HARKing (Hypothesizing After the Results are Known) are quite unavoidable in our profession and it is worthwhile discussing how we can measure and control their effects in scientific reporting. From practitioners’ standpoint, suppressing the risk of false discoveries (e.g. due to p-hacking) may increase missed discoveries (false negatives). In a recent study, Shingo and his co-author employ a data-mining strategy to generate over 15,000 financial (accounting) ratios for each US public firm and examine if these ratios help predict the firm’s subsequent stock returns both in-sample and out-of-sample. Along the way, they find that about 44% (or 28%) of the “significant” findings at the 5% (or 1%) significance level are actually false. Meanwhile, they find a substantial number of false negatives, too.
This presentation will draw on recent research in empirical asset pricing (a field of financial economics), but Shingo will strive to make the presentation as non-financy as possible. Shingo will focus on general methodologies of analyzing multiple-testing problems and controlling false discovery rates. Some of these discussions should also be applicable to empirical research in many social science disciplines beyond financial economics.