On Thursday-Friday the conference on theoretical linguistics in Estonia took place. “Theoretical” not so much in the sense of “formal”, but general discussions on the theory and what that has to do with the language sciences. This event takes place every four years, and gives a place to talk about theory for people thinking about theory in the language sciences. This time the topic was the impact of the quantitative turn on language theory, so I figured that someone should also talk about the replication crisis and took this job for myself.
This is partly inspired by the SMLP summer school on statistics in Potsdam. In addition to very involved and high quality teaching of statistics in various shapes and forms, I came back with two new ideas: 1) that worrying about problems with stats does not have to be for a few freaks, but this all can be pretty obvious and convincing when you show it with a few visualizations and interactive tools; 2) - articulated by people who know much better - worrying about proper conduct and replicability is moving towards the mainstream, while ideas can often be old, the social momentum in worrying about this is growing. Based on the show of hands at TheoLingEst 2017 during the talk in the room, around half of the ~40 people in the room were aware of the issues with replicability.
Thus, I took as my purpose to try to give such a demo to linguists with various backgrounds. The examples that I found most easy to show were p-hacking (and the simplicity of this) within the “garden of forking paths”, and data peeking or optional stopping when collecting data. Both are on the other hand quite relevant and easy to miss issues when doing your first data analysis. In fact, in linguistics these issues are practically far from solved, both for experimental studies and corpus studies, as the solutions of cross-validation or pre-registering are used in only a few of the studies. It is difficult to talk about these issues without becoming an evangelist for science openness, so I added a bit of that too. Particularly, I liked Daniel Lakens’ speculations on how by 2070, the earlier lack of openness in data will seem unbelievable. Nicely put by him on this slide:
To do real science we need team science. Which means .... sharing both ideas & data! Seems sensible but surprising number of people still afraid of being scooped #futureofscience with @lakens pic.twitter.com/ocRRqrQ2Ge
— Caro Rowland (@CaroRowland) November 10, 2017
All of these topics matter a lot due to limited resources in science, and should do so especially more in even smaller communities (e.g. local communities of linguists, such as the one in Estonia). I’ve added the points and links I found most useful for such a demo here just in case. Perhaps even better than a demo, Maki Naro has made a nice comic on this here - Repeat after me.
fishing expedition'' orp-hacking’’ and the research hypothesis was posited ahead of time. (Unpublished.)I’ve also uploaded the slides if they might be useful. They are (mostly) in Estonian.