How Do Researchers Actually Prove Things in Software Engineering?

Ever heard someone say “it’s scientifically proven” in a tech discussion and wondered what that actually means? Today let’s pull back the curtain on how software engineering research really works. Spoiler alert: it’s harder than you think, and that’s because of us – the human factor.

The Evidence Game

First thing you need to know: in software engineering, we rarely “prove” things. Instead, we gather evidence. Eduardo explains why:

Eduardo: “It’s really hard to say that oh this… Maybe in some very specific case yes, but in some cases like using this kind of stuff improves developer productivity. So what we can do is to get evidence regarding this.”

Think about it. You could run an experiment comparing developers using Tool A versus Tool B. But does that prove Tool A is better? Not really. Different people, different projects, different contexts – they all matter. What works for one team might not work for another.

Why Software Engineering Research is Tricky

Here’s the challenge: we’re not studying predictable physical systems. We’re studying people writing code. As Eduardo puts it:

Eduardo: “In software engineering people is an important factor. You can have the greatest framework in the world. If the people that are trying to use don’t know how to use, then it will not work.”

Even Andreas, one of the podcast viewers, pointed out that this makes software engineering research more like psychology or medical studies than physics. And he’s right! That’s why many research methods in software engineering actually come from human sciences.

The Research Toolkit

So how do researchers gather this evidence? Eduardo describes multiple approaches:

Quantitative Studies

These involve surveys with large numbers of people to get statistically significant results. You can spot patterns and trends across many developers.

Eduardo: “We can get like surveys with a huge number of people so we can get some more statistically… this will be a more quantitative study.”

Qualitative Studies

These go deeper with fewer people through interviews. You get rich insights but from a smaller group.

Eduardo: “We can also interview and be more deep with a lower number of professionals. So this one will be more qualitative, and then we will try to get some insights about that.”

The Secret Sauce: Triangulation

The real magic happens when you combine different methods:

Eduardo: “Usually we try to mix stuff and to get this evidence from different perspectives. So maybe we can get more simple information from a lot of people and more in-depth information from a small number of people, and then try to triangulate the findings.”

Real-World Challenges

One interesting aspect is bias management. Students often participate in research studies, but researchers account for this:

Eduardo: “We ask some information especially about the experience you have and the background you have. So for example in a more quantitative study we can do some statistical tests to see how much that experience influenced that on some answer.”

When researchers analyze data, they consider who participated. If it was mostly students, the conclusions explicitly state “these results are for this type of population.” To generalize, you’d need people from other backgrounds and regions.

The Cultural Factor

Here’s something many people don’t consider: even code readability can be culturally influenced! Eduardo mentions:

Eduardo: “I even think that culturally it changes from someone in western compared to someone in other parts of the world.”

This is why good research papers always describe their study population and acknowledge limitations.

From Lab to Practice

Researchers use various data collection methods:
– Online surveys for broad reach
– In-person case studies for depth
– Interviews (online or in-person) for detailed insights
– Code analysis and metrics
– Experiments with real developers

The key is using whatever opportunities are available to gather evidence that can help us understand something better.

Why This Matters to You

You might think, “I’m not doing research, why should I care?” Here’s why:

When you’re evaluating a new framework, library, or architectural pattern, are you thinking systematically about the evidence? Or are you just going with what’s popular?

Professional master’s students with 10+ years of experience who do research work report back that the systematic thinking changes how they approach decisions. Instead of going with gut feeling, they think about ways to actually evaluate things before making big decisions.

Eduardo: “This more analytic things, even if you are not going to work with research, I think this way of thinking can help you a lot even if you are going to the market.”

The Bottom Line

Software engineering research isn’t about proving absolute truths. It’s about gathering evidence from multiple angles to build a better understanding of what works, when it works, and why it might not work in other contexts.

Next time you see research results, look for:
– How many people were studied?
– What methods were used?
– Were multiple perspectives considered?
– What limitations are mentioned?

Understanding how research works makes you a better consumer of technical information and a more critical thinker about technology choices.

How do you evaluate new technologies in your team? Do you gather evidence systematically, or is it more of a gut-feeling decision?