Reflections of an open science convert 3: More challenges to maintaining open research practices
(This is part 3 of a 3-part series. Part 1 can be found here and Part 2 here)
In part 2 of this blog series, I wrote about some challenges to open science practices that might be addressed by (a change of) policies at academic journals. This may solve many, but not all problems. A final and crucial challenge to open science practices is that they imply a different perspective on ownership. People’s focus should shift from “my research” to “research in general”. Taking the first perspective has traditionally provided the gateway to promotion, funding, and, at least for some, media-exposure and fame. Taking the second perspective, however, refers to contributing to the body of knowledge in one’s field: that is, a higher cause, for which the visibility of individual researchers is irrelevant. I believe that changing from an individual to a collective perspective will be necessary, yet will be the biggest challenge for all researchers.
“changing from an individual to a collective perspective will be (…) the biggest challenge for all researchers”
For example, projects in the Open Science Framework (OSF) are private by default, that is, until the research team explicitly chooses to go public. My co-workers and I keep preregistrations under embargo, preferably until submitting the paper. Judging from the number of private projects on the OSF, we are not the only ones. From a “my research” perspective, it is hard to overcome the reflexive fear of being scooped: if someone else sees your project and writes about your idea first, then they will get the credit for it. However, from a “research in general” perspective this should not be a concern: if something is important it should be part of the literature, regardless of who puts it there1. Ownership should not matter – yet it does . I have come across researchers who identify themselves with their particular research method to such extent that they dismiss null-findings of every independent study employing “their” method. I have even found myself in the situation that a straightforward request for study materials with the intent to conduct a direct replication study escalated into a dispute about our team’s ability to carry out the study. Such an exaggerated sense of ownership is detrimental to progress.
Despite the challenges mentioned here and in my previous blogpost, I feel strongly about my commitment to incorporate open science into my research and teaching. To be sure, the area I am working in is changing. I am happy to see colleagues reflecting on their research practices, publications calling for a change of practices in the field of Clinical Psychology (e.g., Tackett et al., 2017), relevant titles in the list of journals adopting Registered Reports and initiatives bringing together clinical psychology researchers with an interest in open science practices. However, as long as incentives in academia focus on the individual level, progress across all areas is likely to be slow. In our department, as in many institutes, tenure-track promotion depends on how many completed products (e.g., publications, supervised PhD graduations, grants) people can list on their resumes within a limited time-period. It is clear that such a policy promotes (misguided) attributions of ownership as well as strategic behavior, such as investing minimally in preregistrations. As (clinical) psychologists we know it is very unlikely that people will be able to change incentivized behavior on their own. Therefore, reward structures need to change at the level of the university. Rewarding collective effort is crucial for a successful shift towards a “research in general” perspective on ownership.
“I feel strongly about my commitment to incorporate open science into my research and teaching”
There is another benefit of rewarding teamwork rather than individual effort. Statistical illiteracy among researchers is one of the reasons why psychology as a whole is in trouble. The solution is thought to lie in educating people about statistics. Of course, furthering a basic understanding is important, but investing in statistical skills at an individual level is not the ultimate solution. There is a limit to what individual researchers can do. For example, I have things to say about psychotrauma and memory that I think are pretty important for clinical theory as well as practice. To nourish this expertise, I need to keep up with the developments in several large areas of the literature. Given my teaching and other work obligations, I lack the time to immerse myself in advanced statistical methods. There are colleagues in our department who are definitely much more specialised in that area. However, those colleagues have to work under the same incentives as I do. Because they need to build their own CV, they need to limit the time they spend on consultation, and on collaborations that are not directly to their benefit. In addition, our department has only a few statistical specialists relative to the number of researchers in other (applied) areas. Ideally, every research question is addressed by a team including at least one applied statistician (Dahy, 2019). In contrast to theoretical statisticians who work predominantly on developing statistical theory, applied statisticians work on projects in close long-term collaboration with researchers in a particular content area and are appointed at the level of the research group.
“Adopting these changes in large numbers may go hand in hand with a change of the incentive structures in the scientific system.”
With this blogpost series I hope to have contributed to an understanding in students and early career colleagues of why the problems in the field have persisted for so long. I hope to have contributed to the growing awareness in my more senior colleagues that a change in attitude is imperative (see also Wessel & Niemeyer, 2019). As researchers, we must recognize problematic research practices. As authors, we need to be transparent in writing up our studies. As reviewers, we should refrain from suggesting post hoc hypotheses or analyses. Adopting these changes in large numbers may go hand in hand with a change of the incentive structures in the scientific system. Journals could enforce preregistration and the sharing of data and materials. Policymakers could enforce open science practices by, for example, rewarding preregistrations, facilitating collective rather than individual efforts, and appointing more (applied) statisticians. I am looking forward to seeing the field growing in the coming years.
(1) Daniël Lakens made a related point in a TEDx talk with the eloquent title “If not me, then someone else; But if not us, then no one.”
References:
Dahy, D.L. (March 10, 2019). Statistical reform. Medium.
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., … Shrout, P. E. (2017). It’s time to broaden the replicability conversation: Thoughts for and from Clinical Psychological Science. Perspectives on Psychological Science, 12(5), 742–756. doi:10.1177/1745691617690042
Wessel, I., & Niemeyer, H. (accepted pending minor revision). We need to change our attitude and journals can help. European Journal of Psychotraumatology. Preprint available at https://psyarxiv.com/xa3kr/
Note:
Image by Simon Hammond, licenced under CC BY 2.0
Ineke Wessel (Twitter: @InekeWessel) received her PhD degree from Maastricht University. She studies (emotional) autobiographical memory. Her research interests include the involvement of memory in the origins and maintenance of psychopathology and the malleability of emotional memories themselves, including false / recovered memories. Her work applies to clinical psychology (e.g. Memory processes in Posttraumatic Stress Disorder), as well as forensic psychology (eyewitness memory). Relatively recently she became fascinated with the question of what the current replication crisis in psychology may mean for clinical psychology.
Websites
http://www.rug.nl/staff/j.p.wessel/
https://orcid.org/0000-0001-6312-376X
