Supporting Requirements-Engineering Research That Industry Needs: The NaPiRE Initiative

FORTY YEARS AFTER requirements engineering (RE) was first acknowledged as an independent discipline, a strong and (more or less) self-confident research community has emerged and contributed a variety of tools, methods, and approaches. Today, the common view is that “RE as a discipline is stable and respected,” as Sarah Gregory pointed out in the previous instalment of this department.¹

However, it’s also evident that after 40 years, conducting the research that industry needs is still a challenge. “Research that industry needs” means research that solves industrial problems practitioners face. But do we really understand those problems? Here, I elaborate on this research challenge and outline the NaPiRE (Naming the Pain in Requirements Engineering) initiative, which aims to tackle it. But first, let’s examine the actual state of RE research and practice.

RE Is Challenging—in Practice and Research

Nobody can refute RE’s importance and its challenges. As an interconnected discipline, much depends on its success because many decisions—and problems—in software development projects are rooted in it. Given RE’s critical and challenging nature, it shouldn’t be surprising that project failures are still too often caused by insufficient RE.

So why are we still struggling with RE after 40 years? Do we lack proper scientific contributions that aim to solve practitioners’ problems, or do we lack proper knowledge and technology transfer between research and practice? Probably both.

For practitioners, it’s often (made) difficult to share insights into their RE, given its criticality to their company’s competitive environment. For researchers, it’s generally very difficult to obtain clear pictures of RE in practice. The discipline is characterized largely by uncertainty and human factors such as experiences, expertise, fears, beliefs, and even politics.² These factors affect the choice of methods, approaches, and tools in individual practical contexts. What might work well for one project might be completely alien to the next project’s needs and culture.³ This renders it difficult to empirically investigate the discipline. In the end, RE is a highly individual means to an end—as individual as the people working on problems and the problems they work on.

The State of RE Research

It’s not surprising that the biggest challenge in RE research is to provide proper empirical figures that demonstrate particular success factors. However, those factors are critical determinants of what works in practice and what doesn’t.⁴ Consequences are that the state of empirical evidence in RE is particularly weak and that much of everyday industrial practices is governed by conventional wisdom rather than empirical evidence.

Other symptoms of the disconnect between research and practice⁵ are that we researchers keep proposing solutions to problems we barely understand and that these solutions either remain unknown in practice or—worse—are blindly adopted without a proper understanding of their potential benefits and shortcomings. The rather low practical impact of many of our contributions reinforces this disconnect. The dominant presence of goal-oriented RE approaches in research and their absence in practice could well be one example of the disconnect.⁶

Although the practical relevance of research certainly shouldn’t be the sole measure of success, we still need to break this vicious cycle and foster more problem-driven RE research. To this end, RE researchers need to close the knowledge gap regarding the plethora of situations practitioners face in their projects, to eventually pave the road for the industrial adoption of our contributions.

We Need Proper RE Theories

Motivated by the overall situation and the need for a stronger body of RE knowledge, Stefan Wagner (University of Stuttgart) and I proposed the NaPiRE initiative in 2012 in a workshop at the annual meeting of the International Software Engineering Research Network (isern.iese.de/Portal). Our overall objective was to establish a holistic theory of RE practices and problems in industry to help guide research along specific industrial contexts and needs.

To that end, we proposed running NaPiRE as a biennial replicated family of internationally distributed surveys in which each run helps extend an initial theory and make it more robust over time. We further proposed disclosing all data and resulting publications to the public to make our results transparent and reproducible and to help other researchers ground their work in empirical data.

At first, our idea received a lukewarm reception, but now we number 58 researchers worldwide, conducting the third replication in 25 countries. NaPiRE’s success has strengthened our confidence in a shared vision, and the initiative has become an effort by the community, for the community.

NaPiRE: Problems Are Everywhere!

Here, I present selected results from the second survey replication, conducted in 2014 and 2015, with responses from 228 companies in 10 countries. I concentrate on a specific set of problems our respondents are experiencing; for further details, see “Naming the Pain in Requirements Engineering: Contemporary Problems, Causes, and Effects in Practice.”⁷ My goal is to give you an overview of the diverse problems and the importance of better understanding them.

Figure 1 summarizes the survey respondents’ top 21 RE problems and those problems’ frequency. The chart further shows the problems’ criticality by indicating the number of projects in which they led to project failure. The most frequent problem was incomplete requirements, followed by communication flaws between the project team and customer, underspecified requirements, and moving targets.

Figure 1. Key requirements-engineering problems and their criticality, indicated by the number of projects in which they led to project failure.⁷

At the same time, moving targets, although ranked as the fourth-most-frequent problem, become the top-priority problem when we consider only the project failure ratios. Furthermore, when considering all the problems, someone might argue that many of them could be eliminated by clear RE process models or artifact templates—that is, blueprints for how to specify requirements or which terminology to rely on. Interestingly, many of the respondents did use such templates. To better understand the reasons for the problems and propose fruitful solutions, we need a deeper understanding of the problems.

A Closer Look at the Problems

The NaPiRE data captures not only practices and problems but also the problems’ root causes and consequences, going beyond the simple case of project failure. Figure 2 visualizes an example of such analysis of communication flaws between the project team and customer. The figure is a probabilistic cause-and-effect diagram⁸ (also called an Ishikawa diagram) in which the percentages represent the share of given answers. The closer phenomena are to the central horizontal line, the higher their probability of occurrence in projects.

Figure 2. Causes and effects of communication flaws between the project team and customer. The left side shows the causes; the right side shows the effects. The closer phenomena are to the central horizontal line, the higher their probability of occurrence in projects. The percentages represent the share of the survey respondents’ answers.

The left side of Figure 2 shows the problem’s root causes. The right side shows the problem’s effects, including, for instance, poor product quality caused by missing or incorrect features. In particular, the diagram shows that the top five root causes don’t indicate a lack of guidance from templates; rather, they indicate organizational and social factors (for example, language barriers, missing direct communication with customer, or missing customer engagement). One natural follow-up question is whether projects applying agile practices avoid these issues given that agile practices aim to support communication between all the involved parties.

By blocking the data according to the software process models used, we can select all the responses from the practitioners who applied agile practices. Table 1 summarizes the outcome; it lists the top five problems and groups them by company size. Interestingly, those companies still experienced communication flaws between the project team and customer. Even more surprisingly, they encountered additional problems that agile practices aim to solve.

As you can see, many of the challenging project situations that agile practices claim to address, such as moving targets, still manifest themselves as severe problems. Space limitations prevent us from further reasoning about the particularities of the respondents’ contexts, but the key takeaway of this small example is already evident. The claims we often associate with specific solutions, such as agile practices, and our expectations when they’re applied universally, aren’t in tune with today’s industrial reality. We all would benefit from a better understanding of the industrial reality to align our research with the problems in practice.

As we mentioned before, we’re running the third replication of NaPiRE to further explore the state of the RE practice, focusing on knowledge gaps such as those I illustrated in this article. We hope this improves the community’s understanding of what’s really going on out there, and it will help us steer our research in a problem-driven manner. I cordially invite researchers to join this or similar initiatives, and I invite practitioners to support us with their answers to do their part to produce the research that industry needs. For details on NaPiRE and how to contribute, visit www.re-survey.org.