The future of evaluation is mixed

Welcome

Funded by the British Academy, the COMbINE network was established in 2015 by a group of academics from the research evaluation field who were keen to implement a culture of high quality mixed-methodology into the field. The field of research evaluation has been traditionally associated with quantitative methodological approaches such as bibliometrics, network analysis and econometrics.

However, recent research policy developments conceptualising research excellence in terms of both societal and scientific outputs, the importance of open access, and the effectiveness of incentives for influencing desirable research behaviour, has necessitated that the field adopt broader, more qualitative approaches in order to address these new questions.

The task of research and innovation policy has broadened in scope, encompassing not just economic goals, but also the solution of key societal challenges. This is reflected in the Europe 2020 Strategy of promoting smart, sustainable and inclusive growth (European Commission 2010). At the same time, the research and innovation system has become more complex, with knowledge creation and innovation increasingly the result of co-creative processes involving a variety of actors.

Research and innovation policies can be characterized as drawing on a “policy mix” (Flanagan et al., 2011; Kern, 2012; Magro and Wilson, 2013) to address socio-technical transitions. This included “transformative innovation policy”, which essentially requires taking stock of different stances among relevant actors (researchers, business, citizens) and how they are aligned in relation to societal challenges (Steward, 2012; Veugelers, 2012). Existing innovation policies and the design of specific instruments can be seen as the result of policy-making processes that involve theory-based rationales, actors and governance structures.

A sound evidence base is needed to understand these complex processes; however, current tools, datasets and indicators are not equipped to meet the demand of broadened policy goals or to capture the engagement of different types of actors in research and innovation policy processes.

Therefore, this arm of COMbINE will explore complementary methodologies that will aid in investigating and understanding the design, execution, and evaluation of policies and governance structures. Examples of useful approaches will be presented and discussed, in particular combined quantitative and qualitative methods. In addition, the potential of datasources, visualisation tools and input from actors at all stages will be further developed.

“Methods should be mixed in a way that has complementary strengths and non-overlapping weaknesses. It involved the recognition that all methods have their limitations as well as their strengths” Johnson & Turner (2003)

No other topic seems to divide academic departments and disciplines more than the choice of research methodology. However, mixed methodology is more than negotiated compromise between qualitative and quantitative methods and making the decision of how to integrate and combine methodological angles, to build upon the strength of each is not always straightforward.

Mixed methods can be extremely powerful, but it is not always the best choice for your research.

Despite being quite young, the theories underpinning mixed methodology are already well developed, and extend to considerations of mixed methods as more than just a triangulation tool (Denzin, 1970). Formal design considerations have already been developed by Morse (1991), and further developed in a number of studies (O’Cathain et al 2008; Sale et al, 2002; Howe 2004; Freshwater, 2007). Little has been done to develop these frameworks and the understandings they can contribute in the research evaluation field. This arm of the COMbINE network aims to address this shortfall.

This arm of the COMbINE network aims to equip researchers with the necessary skills to make informed choices about the choice, and implementation of mixed or multiple methodologies for their research projects. Here, we draw upon the already significantly developed quantitative techniques in the research evaluation field, and use them to develop frameworks that include the benefits qualitative and mixed methodologies have to offer.

We offer here a series of seminars, formal course learning, and teaching materials that can be used to enhance the teaching of mixed methods both within the evaluation field, and beyond.

Plans for formal courses, conducted both in residence and virtually are in currently in progress. We would encourage you to monitor this site for announcements of these courses.