Skip to main content

Home

Evidence-Based Software Engineering

Welcome to the web site for Evidence-Based Software Engineering (EBSE). This website provides an introduction for researchers, practitioners and teachers in computing, software engineering and IT who have heard about EBSE and want to know more about it. The website explains the basic concepts and processes, as well as providing some resources, together with links to sources of other, more detailed, material.

Whether you are interested in EBSE as a potential user or gatherer of evidence, or may even be considering commissioning others to undertake the task, we hope that you will find our description of EBSE sufficiently interesting and useful to read further!

So, what is EBSE?

EBSE is concerned with using findings from empirical research (based on observation and experimentation) to determine what software engineering practices, tools and standards work, and the situations when and where they work.

Originating in 2004, EBSE was inspired by the success of evidence-based medicine (EBM), as employed in clinical medicine and healthcare. EBM arose because recommendations from experts were not always supported by research and there was evidence that some recommended clinical practices could actually endanger patients. In the case of software engineering, the 80’s and 90’s saw a plethora of methods and tools recommended by consultants and experts that were largely a matter of opinion, usually expressed as experience papers, and not supported by any reliable evidence. EBSE was seen as a means of addressing that problem. It involved adapting evidence-based practices to meet the rather different characteristics of software engineering, and the consequences that these characteristics have for empirical studies performed on software engineering topics.

EBSE aims to provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision-making processes involved in the development and maintenance of software.

How does EBSE work?

EBSE is defined as a five-stage process involving the following steps:

  1. Converting the need for information (such as about development and maintenance methods, management procedures etc.) into an answerable question.
  2. Tracking down the best evidence with which to answer that question.
  3. Critically appraising that evidence for its validity (closeness to the truth), impact (size of the effect), and applicability (usefulness in software development practice).
  4. Integrating the critical appraisal with our software engineering expertise and with our stakeholders’ values and circumstances.
  5. Evaluating our effectiveness and efficiency in executing steps 1-4 and seeking ways to improve them both for next time.

A core tool of the evidence-based approach is the systematic review, which usually supports steps 1-4 of the EBSE process. However, in the case of systematic reviews commissioned by an external organisation, the first EBSE step may be performed by the commissioning organisation.

What’s a Systematic Review?

A systematic review is a specific form of secondary study, and the material provided by this web site is concerned with how to conduct, report and use the various forms of secondary study used in software engineering.

A systematic review aims to identify and summarize all relevant high-quality empirical studies that are related to the question being addressed (referred to as the primary studies), in a way that ensures that its conclusions are fair and trustworthy.

The systematic review process is intended to avoid human errors due to tiredness or fatigue, such as misunderstanding or misinterpreting an empirical study report, or wrongly transcribing information from an empirical report; as well as errors arising from personal bias, such as giving greater weight to results that agree with our own opinions.

What questions do Systematic Reviews answer?

There are three broad categories of systematic review, each of which addresses a different sort of question.

  • Mapping Studies. These are relatively ‘lightweight’ reviews which aim to answer questions of the form “what studies have investigated…?”. A good mapping study will not only aim to identify relevant studies, but also to provide some form of analysis, perhaps based upon categories (types of empirical study, types of participant, size of task etc.). They can provide a useful overview of a topic area and can help to determine if a more detailed study would be worthwhile.
  • Qualitative Systematic Reviews. A qualitative systematic review provides a deeper level of knowledge than a mapping study, and typically is used to aggregate information from studies that report the benefits, risks, and stakeholder attitudes involved in using SE technologies and to prioritise the most important issues these identify. They can be based on aggregating results from thematic analysis of case studies and/or studies reporting industry opinion surveys. Such a review can provide useful guidance on issues to consider when adopting new technologies, including identifying potential barriers.
  • Quantitative Systematic Reviews. As might be expected, a quantitative systematic review aggregates quantitative findings, typically provided as statistical values, and where possible, these aim to provide some form of numerical information using statistical techniques such as meta-analysis. An example of the outcomes to be expected of such a review might be to provide a ranking of a set of techniques in terms of their effectiveness.

Which forms of systematic review are most commonly used in EBSE?

The findings of many systematic reviews covering a range of software engineering issues have been published since 2004. Most published systematic reviews are either mapping studies or qualitative systematic reviews (sometimes something of a mix of both). This is because the human-centric nature of SE, with its dependence upon individual skills, makes it difficult to conduct the sort of studies that can contribute to a quantitative systematic review.

The EBSE web site

This web site is maintained by researchers who are interested in EBSE (see ‘About Us’) and its purpose is to act as a resource for members of the wider Software Engineering community. Everyone is welcome to use the material it provides. The site is intended to provide material that will be useful for practitioners, researchers, teachers, students and policy-makers, particularly those who might wish to make evidence-informed decisions related to software engineering activities. A schematic view of the resources provided on the site is shown below.

Much of our material is concerned with providing ‘how-to’ guidance about performing and reporting the findings from systematic reviews. More detailed information about the role of EBSE relevant to different stakeholders is presented in the sub-pages below.

This page was updated in October 2023.