The Dangerous Comfort of a Hundred Replications

Second-year Psychology students participating in the University Honours College follow a workshop on Blogging Science, in which they learn to communicate science to the general public by means of informing, giving an opinion, and relating science to issues in society. A selection of these blog posts is published on Mindwise. Today’s post is by Johannes Jamroszczyk.

---

 

In the face of the replication crisis in psychology, we often seek comfort in those studies that have been replicated successfully over and over again. However, after a recent study I worked on, I realized something about the power of replications over our scientific thinking, which has changed how I look at effects that have been replicated many times.

First, let me introduce the phenomenon we studied, which has led to this insight. When studying visual perception, one quickly learns that the brain does not uniformly process all visual stimuli. A particular and well-established example of this is the Oblique Effect, which describes our preferences for horizontal and vertical orientations. If you grew up in anything close to a modern urban environment, science will, with high probability, find this phenomenon in your visual perception as well. The dominance of this orientation preferences in people from modern cities, opposed to people that grew up surrounded by tribal, and predominantly circular architecture, suggests that visual experience plays an important role in the arise of this effect (Annis & Frost, 1973). Ernst Mach (1861) was the first researcher who documented how changes in horizontally and vertically oriented lines were easier to identify than in diagonal lines. The name ‘oblique effect’, which has by now become widely known in visual perception research, was established by Stuart Appelle in 1972. It has since been replicated in many experiments and different contexts, and was even discovered to exist in urban birds such as pigeons (Donis, 1999). The mountain of successfully replicated studies on this effect has since grown (e.g., Heeley et al.,1997; Meng & Qian, 2015).

 

The oblique effect has been replicated in many experiments and different contexts, and was even discovered to exist in urban birds such as pigeons.

 

In the face of the replication crisis in psychological research, this seems like good news. Our visual perception is just more accurate when working with vertical and horizontal stimuli than diagonal ones, and experiments have shown repeatedly. The story could end here, case closed. But it doesn’t. After all this research on the effect, experimenters could not manage to rule out that the visual environment’s cues during the experiment may affect how we perform on the actual experimental task. Let me explain this in more detail.

In visual perception research, the visible environment around the experimental stimuli is tough to control for. It is nearly impossible to eliminate all visual elements except for the experimental stimulus. Even the motion of our head or blinking of the eyes may cause confounding variations in how we perceive something.

Most of the previous experiments on the oblique effect lacked environmental control. They were all done in environments that offered the participants numerous orientation references such as vertical and horizontal edges of paper, and computer screen frames. The presence of cardinally oriented objects in the environment could create a form of neuronal summation effect, or in other words, an amplification of similar oriented stimuli in the actual experiment.

So does the oblique effect even exist? The many replications of the oblique effect have caused a very one-sided view, and it might be valuable to take a different perspective. With new technologies, better-controlled experiments on visual perception have become possible, and we should not be afraid to address the shortcomings of all the past replications.

In this case, we used Virtual Reality headsets to gain better control over the references that we wanted to display during the experiment. We sampled first-year Psychology students via a student online pool. We then invited them to a lab at the Heymans institute, where we let them identify changes in subsequently displayed striped patches (Figure 1) using an Oculus Quest virtual reality display and a computer mouse (Figure 2).

 

 

 

 

 

 

 

 

 

 

Our participants did not know until the debriefing that, for each third of the sample, we added differently oriented virtual frames around the experiment, similar to what the structure of a computer screen with different tilts would look like (Figure 3). After the magic of data analysis met the collected data, we saw something remarkable: The orientation of the frame and the orientation of the experimental stimulus were interacting with each other. Specifically, we found that when the frame and the stimulus shared the same orientation, the participants’ performances for that particular stimulus orientation became better. Moreover, this was true for both horizontal and oblique orientations. This suggests that external visual cues, over and above the orientation of the stimulus, may actually play a significant role in the occurrence of the oblique effect.

 

We should stay curious about possible alternative explanations of effects, even –or especially– when they have been reproduced a thousand times.

 

Let that sink in for a second. An effect observed for more than 150 years and replicated numerous times seems to depend on more than just the orientation of the experimental stimulus; the orientations of shapes found in the visible environment during the experiment seem to matter as well. Not only is this a fascinating finding for visual perception research, but it also teaches all of us something essential about our understanding of replicated studies. Modern psychological research is very focused on clearing up the ambiguity caused by the replication crisis. Researchers are aiming to find out which effects are reliable and which are not. However, this focus on replicability can limit the way we are viewing studies that have been successfully replicated many times. The accumulation of apparent evidence can create a strong sense of confirmation in our beliefs, soften our scientific scrutiny, and cast a shadow on possible moderators.

This brings me back to the beginning of this post. When I say I look differently at successful replications, I am not trying to be pessimistic or even cynical about the usefulness of replicating studies. In fact, by sharing this experience with you, I am trying to emphasize the very contrary. In the replication crisis, every experiment whose effect is supported by yet another replication acts as a beacon in the dark, helping to keep up our spirits about what psychological research is doing in the first place. But it should not stop us from being critical, even about the findings that have accumulated loads of evidence. Millions of lurking factors out there might mediate, moderate, or create the apparent relations between the variables and concepts we measure every day. We should stay curious about possible alternative explanations of effects, even –or especially– when they have been reproduced a thousand times. As new directions of thinking and ways of measuring come about and the depth of critical thinking evolves, we should, once in a while, turn around and see if those new accomplishments and ideas help us to identify limitations in our previous ways of thinking.

 

 

References

Annis, R. C., & Frost, B. (1973). Human visual ecology and orientation anisotropies inacuity. Retrieved from Science, 182(4113), 729–731. http://www.scopus.com/inward/record.url?eid=2-s2.0-0015900221&partnerID=tZOtx3y1

Appelle, S. (1972). Perception and discrimination as a function of stimulus orientation: the “oblique effect” in man and animals. Psychological Bulletin, 78(4), 266–278.

Donis, F. J. (1999). The oblique effect in pigeons (Columba livia). Journal of Comparative Psychology, 113(2), 107–115. https://doi-org.proxy-ub.rug.nl/10.1037/0735-7036.113.2.107

Heeley, D., Buchanan-Smith, H., Cromwell, J., & Wright, J. (1997). The oblique effect in orientation acuity. Vision Research, 37(2), 235-242. doi:10.1016/S0042-6989(96)00097-1

Mach, E. (1861 ). Über das Sehen von Lagen und Winkeln durch die Bewegung des Auges. Sitzungsberichte der Kaiserlichen Akademie der Wissenschaften, 43(2), 215–224.

Meng, X., & Qian, N. (2005). The oblique effect depends on perceived, rather than physical, orientation and direction. Vision Research, 45(27), 3402–3413. https://doi.org/10.1016/j.visres.2005.05.016

 

Image credit: all images by Johannes Jamroszczyk.