During the year 2017, the debate about the design of the next Framework Programme has evolved quite rapidly, notably following the presentation of the Interim Evaluation Report
for Horizon 2020. At this relatively early phase of the discussion the focus is not yet on detailed contents and concrete figures, but rather on the broad concepts and key objectives for the
European Research and Innovation Programme after 2020.
Against this background it seems timely and useful to shed some light at three keywords dominating the current debate. While this paper is having a closer look at the role of “impact” as a guiding principle for FP9, THINK Piece 4/2017 has a closer look at “missions” in research, and THINK Piece 5/2017 presents some reflections on “innovation” in this context.
The current statements from the European Commission promote very prominently the idea that the next Framework Programme should be designed to have greater impact, and go as far as
stating that impact is even the key feature for the next FP.
At first sight, this announcement goes along with common sense, as it is indeed a reasonable expectation that a public intervention should have an impact – otherwise it would be meaningless and a waste of money. It is hence not surprising that the big chorus of FP beneficiaries gave a very warm welcome to these intentions and seem to fully go along with this new priority. As a result, there has been very little public debate on the role of impact as a yardstick for the design of FP9. This is somewhat surprising, as a closer look reveals that things might not be as obvious as the broad public consensus suggests.
1. What is the impact of an FP?
Framework Programmes are multi-annual multi-billion multi-disciplinary multi-national endeavours, so the footprint of several thousand projects involving several ten thousand
people is tremendous. The consequences of FP activities can be found in many different facets of our individual, societal, economic and political life.
A few examples might help to get a better feeling for this enormous complexity:
But there is of course also a negative impact of the FP, most visibly through the fact that money spent on the FP prevents public funding on other desirable issues …
Impact is thus a vast concept. This is positive, as it allows bringing together this wide range of effects under a common roof. At the same time, impact is per se not a very operational concept, and general notions like “more impact” or “better impact” are not particularly meaningful without further explicit interpretations.
2. How to measure the impact of an FP?
Enormous efforts have been made over recent years to gain a better understanding of the many impacts of research, notably through more systematic and analytical accounting systems
and the use of data mining and big data. This means that today we have a much better view on impacts of the FP than ever before.
Unfortunately, this progress is hampered by many profound problems which still prevent us from getting a complete and reliable picture. Two unsolved issues are mentioned here just to illustrate the difficulties:
We know today much more about many impacts of the FP than in the past, but we must acknowledge that a wide range of impacts still escapes from our radar – and that aggregating the diversity of impacts into something like a quantifiable impact figure looks still largely like a mission impossible.
3. What are the time lines for assessing impact?
Not surprisingly, the different dimensions of the FP impact do not occur simultaneously, they rather spread over a long time. This can be illustrated by a closer look at the situation for Horizon 2020, the current FP running from 2014 to 2020:
The sequence in the political decision-making process for the Framework Programmes is such that the interim evaluation of the previous FP becomes in fact the most influential assessment. Against the timeline described above it is by no means a criticism of the Lamy Report on the Interim Evaluation of Horizon 2020 to state that as of today we know in fact very little about the full impact of this programme.
4. Which parts of previous FPs had the biggest impact?
Since the Framework Programmes cover such a large – and in fact ever more enlarging… - range of fields and activities, it would be very helpful, and in a sense crucial, for the
design of a new FP based on “impact” as key factor to get a clear picture on how the different programme lines have scored so far in this respect.
Such information, however, is completely missing – not only for the FP, but also for all other large national or international research programmes.
The European Commission has put over recent years a lot of emphasis on “Better Regulation” and as part of this approach an ex-ante impact assessment has become mandatory for all activity lines. These assessments include a cost – benefit analysis which means that an activity should only be considered if the expected benefits (a proxy for impact…) would exceed the costs incurred. As these calculations often use different methodologies, numerous assumptions, weak data and sometimes large econometric modelling, they are useful as a first filter for a “go – no go” decision, but would not allow comparing and even rank different activities.
The straight answer to the basic question is thus: we simply don’t know.
5. How to increase the impact of the next FP?
While increasing the impact of the next FP is a very laudable aim, the analysis in this paper so far does not suggest any obvious strategies to get there – as the evidence we have
at our disposal is simply too limited and too weak to draw reliable conclusions.
Also, the European Commission has not yet come up with any clear statement on how to implement this in FP9. Reading between the lines of the most recent “Overarching Strategic Document” to the Horizon 2020 Work Programmes 2018 to 20201 hints at two possible avenues, mentioned in this specific context and possibly highlighted as elements of a future FP9 design:
The new set of work programmes provides even more detailed indications on the expected impact of the planned activities, and applicants are expected to respond in full to these in their proposals.
While it is always positive to raise the awareness for this important issue, there is on the other hand the doubt whether the impacts imagined in the work programme are realistic ones – and whether there could be other, possibly even more important ones, which might be neglected.
“If we talk a lot about it, things will get better” is a frequent behaviour in everyday life, and we all know about instances where this proved to work – but also about many cases where it failed completely. From a more systemic point of view there is so far no evidence available that there is in fact a positive correlation between more detailed expected impacted impact statements in work programmes and proposals and a more substantive real impact from the subsequent projects.
Big projects – big impact?
Another possible approach, which one could read between the lines of the Strategic Document, is a kind of renewed “big is beautiful” logic. This is a very relevant issue, as in
Horizon 2020 there is a range in the funding per project from less than 5.000€ to almost 500 million€. It is obvious that large projects will produce more and most of the time more visible
impacts than smaller ones. It is important, however, that small does not necessarily mean “low”, and that the great diversity of smaller activities might well generate in sum larger impacts than
a single “big ticket”.
As long as there is a complete lack of any evidence on this issue, such as an indication of “impact per € FP funding”, it seems problematic to argue in favour of larger activities based on a “boosting impact” rhetoric. One might want to go this direction for political reasons, but this should be clearly communicated as such.
6. Some preliminary conclusions
This paper argues that impact, one of the key objectives of the next FP, is an extremely huge and complex concept which largely lacks a coherent operational meaning. Available
evidence is scarce, and even non-existent for several key questions. In other words, the current situation could be described as “We want to focus on impact, but we do not know much about
In this context it is surprising that there is no public debate, and all major stakeholders seem to be so convinced of generating impact, that nobody dares to question the process.
There is, however, a big risk that the “focus on impact” argument could be used to promote other agendas, and to hide political priority settings behind the seemingly uncontroversial strive to “boost impact”.
For the emerging debate it might be useful to retain two simple messages:
Version 1.0 – 06.12.2017 - Feedback