Two-stage calls –

 A real improvement in times of low success rates?

Download PDF

The relatively low success rates for applicants in most parts of Horizon 2020 have been heavily and unanimously criticized by the stakeholders. In response, the European Commission introduced more generally a proposal evaluation in two stages, in order to ease the burden of unsuccessful applicants during the first stage. This approach received a very positive feedback from the scientific communities.

This paper presents a simple model to analyse the likely consequences of such a change and shows that, despite the general enthusiasm about the new approach, the net effects are rather nuanced. To make this approach a success, a very careful implementation will be required.

0. Intro

Success rates, defined as the ratio between proposals submitted and proposals funded, have dropped quite substantially across Horizon 2020: While success rates in FP7 were around 20%, the figure for Horizon 2020 is so far around 15%.

Since apparently most other features of Horizon 2020 run reasonably well, the issue of success rates has become the focal point of public criticism. Lobby groups and policy makers complain about the high amount of effort (and notably money...) that is finally wasted in the 85% of unsuccessful proposals. While the most popular “solution” to this problem – an increase in the number of funded proposals – remains wishful thinking in times of tight budgets, the European Commission has reacted to the growing unease by announcing the more general use of two-stage calls.

This approach is generally welcomed by the scientific community, as it promises to cut the effort required by the applicants considerably through much simpler and easier proposals at the first stage.

The following analysis based on a simple model will show that things are a bit more complicated as was reflected so far in the public debate – and that a careful implementation is needed to safeguard the expected efficiency gains and to avoid unwarranted distributional effects.

1.      A simple model

The rather trivial baseline scenario for a single-stage call reads

TE = AE * NA

TE        Total Effort

AE        Average Effort per application

 NA       Number of applications

The “total effort” in terms of time, money and other resources for all applications in a given call is determined by the number of applications multiplied with the average effort for each application. Throughout this paper the concept of “effort” is used, as the more tangible concept of “costs” might hint at concrete figures, which this paper is unable to present.

For the single-stage call, the starting assumption is that there were 100 applications, requiring on average an effort of 10, so that the total effort required for all applications under this call is 1000.

Box 1: Baseline scenario for a single stage evaluation

Box 1 illustrates the scenario for a 15% success rate. The total coloured surface in the box indicates the overall effort invested by all applicants.

Things get slightly more complex for the two-stages call, where the formula used reads as follows:

TE = (AE1*NA1) + (AE2*NF*SR)

 TE        Total Effort

 AE1      Average Effort per first round application

 NA1     Number of first round applications

 AE2     Average Effort for a second round application

 NF       Number of proposals to be funded

 AR     Admission Ratio (Applications admitted to the second round in relation to the number of   proposals to be funded)

The total effort for such a call is calculated as the sum of efforts required for the first round and for the second round:

•  At the initial stage the effort required is again determined by the number of applications at the first round multiplied with the average effort for each first round application.

• The effort required for the second stage depends on the average effort for each application submitted to the second stage, the number of proposals likely to be funded, and the ratio by which the number of applications admitted to the second stage is higher than the number of proposals funded.

 The baseline scenario for the two-stages call is using the following assumptions:

·         AE1 = 2

The average effort for a proposal outline is estimated at 20% of the effort required for a full proposal.

·         NA1 = 100

 The number of applications in the first round is unchanged compared to a single-stage call

·         AE2 = 10

The average effort for preparing a full proposal at the second stage is equal to the one for preparing a proposal in a single-stage call

·         NF = 15

This assumption is in line with current success rates in Horizon 2020.

·         AR = 3

The number of proposals admitted to the second stage should be three times higher than the number of projects likely to be funded.

Table 1 in Annex provides some (fictive) calculations for illustration purposes. The figures used are mere assumptions and not based on any empirical evidence. The aim here is not to come up with any kind of measurement, but just to provide some ideas about the impact of changes in the different variables.

Box 2 illustrates the data from the corresponding column in Table 1, showing that the Total Effort for a two-stage call under the assumptions made above would be 650, so roughly one third less than for a single stage call. This would represent a real progress in terms of saving monetary and intellectual resources and justify a clear recommendation for the implementation of two-stage calls.

Box 2: Baseline scenario for a two-stages evaluation

In Box 2 and all following boxes the rectangular “black box” represents the total effort required in a single-stage call. The coloured surface represents the efforts required by participants in a two-stage call. For as long as the “white” surface inside the box is greater than coloured surfaces outside that box, the effort required in a two-stage calls is lower.

But as always in mathematics – and in real life ... -, everything depends on the assumptions made. The following four chapters present for each of the variables in the model a detailed analysis on plausible variations and likely consequences.

2.      Three pages, five pages, twenty pages? –

  Variations in the average effort for a first round application

The main reason for the attractiveness of the two-stage calls is the massive reduction in the effort needed to prepare a proposal outline in the first round as compared to the efforts required for a fully developed proposal in a single-stage call.

Obviously the savings are most significant if first round applications consist of short texts of several pages only. On the other hand, the need to carry out a substantial and well grounded proposal evaluation puts a natural limit to ideas for radical simplification, as it seems not feasible to base such an evaluation process on very short abstracts.

If no stringent boundaries are implemented, the tendency among applicants might be to come up with more substantial documents at the first round, just to underline the seriousness of the application and to increase their chances with the evaluation panel. Similarly, if the Commission comes up with a too detailed catalogue of issues to be addressed by applicants in the first round already, the average effort required is also likely to increase.

 Box 3: Higher effort required for first stage proposals

Box 3 (based on the corresponding column in Table1) illustrates the impact of such a tendency: If the average effort for a first round application is not 20% of a full proposal, but rather 40%, the net savings in terms of efforts required for a two-stages call would be reduced by more than half, and the net effect would be in the order of 15% only, which can no longer be regarded as a decisive progress.

3.      Elasticity of demand? – Variations in the number of first round applications

The massive or at least considerable reduction of the average effort required at the first stage means that applicants have to pay a much lower price for the participation in a call. Contrary to the assumption made in the baseline scenario, where the number of applications remains stable, economic theory would suggest that with a reduced price one would observe an increase in demand. The degree of induced change is described by the concept of “elasticity of demand”, which varies on classical markets and might also vary across different parts of the Framework Programme.

One could expect that in areas with a limited number of potential applicants, such as probably Fusion research or aeronautics, a massive cut in the application costs will have no dramatic effect on the number of applications, as there are high barriers to entry for new candidates and almost all members of the relevant scientific communities are already involved. But there are also major fields in Horizon 2020, such as Social Sciences and Humanities, but probably also Life Sciences, where a supposed reduction in the effort required by the factor five will mobilise additional applicants from a large pool of experts so far not involved.

Box 4: Increased number of proposals in the first round

Box 4 (based on the corresponding column in Table 1) illustrates the impact of this elasticity of demand: If the number of applications in the first round is increasing by just 50% (after a “price cut” by the factor five...), the net savings in terms of efforts required for a two-stage call would be reduced, so that the remaining net effect would be in the order of 25% only.

4.      Nice evaluators? – Variations in the admission ratio for the second round

There seems to be some general consensus that for a solid evaluation at the second stage one should admit three times as many applications as projects could be funded. This is regarded as a good compromise between the need to have a broad range of good applications to select the truly outstanding ones, and the wish to limit the burden linked to a second round proposal for a too great number of applicants.

In the baseline scenario, this means that for 15 proposals to be funded 45 out of the 100 proposals submitted at the first stage should be admitted to the second round. This is a situation where first stage evaluators operate so to speak “in the comfort zone”, as they could give a green light for the second round to 45% of the proposals.

Things change, however, quite drastically if the number of proposals submitted increases, as argued in the previous chapter. With 150 proposals at the first stage, evaluators were forced to work with a success rate of just 30% for the admission to the second stage. Many evaluators will feel uncomfortable with this situation, and there will be a kind of natural tendency to lower the pressure by increasing the number of proposals admitted to the final round. Unfortunately though, “nice” evaluators will result in an increase in the overall effort required for the call.

Box 5: Increased number of proposal admitted to the second stage

Box 5 (based on the corresponding column in Table 1) illustrates the impact of “nice” evaluators as a consequence of a high elasticity of demand: If the number of applications in the first round is increasing by 50%, and evaluators in the first round give a green light for 60 proposals (instead of 45 in the baseline scenario), the net effect of the two-stages evaluation would shrink to just 10% of the initial total effort.

5.      Quit or double? – Variations in the average effort for a second round application

The final variable to be analysed is the average effort for a proposal at the second stage. In the baseline scenario it is assumed that this is equal to the effort for a proposal in a single stage procedure.

Here again, basic economic theory suggests that applicants might again change their behaviour, as the willingness to invest in a proposal might be substantially higher, if the chance of success is no longer 15% (as in single stage calls), but rather 33% (as in the final round of a two-stages evaluation).

Box 6: Increased efforts at the final stage

Box 6 (based on the corresponding column in Table 1) illustrates the impact of such increased efforts at the second stage: Just by an increase of 20% in the average effort for a proposal in the second round, the overall net effect of introducing a two stages evaluation would be reduced by more than 25%.

6.      Adding-up of unwarranted effects

Whereas in the previous chapters for didactic purposes changes in behaviour were analysed in isolation for each of the variables, a more realistic scenario is the one where the different effects do actually cumulate.

It is surprising to see, how relatively minor changes at the different parameters do add up in the end to put the overall efficiency at risk.

Box 7: Adding-up of unwarranted effects

Box 7 (based on the corresponding column in Table 1) illustrates such a scenario where the different effects add-up:

•  The proposals at the first round require not just 20% of the effort of a full proposal, but rather 30%
•  The number of applicants at the first stage increases by 30%
•  Evaluators are “nice” and allow 40% of proposal to go to the second stage
• Applicants at the final stage put on average 10% more effort in their proposals

All these relatively small changes taken together lead at the end to a situation where the total effort required by a two-stage call is almost identical to the efforts required for a single stage call.

7.      Good for the bad ones, bad for the good ones? – Distribution of efforts among participants

The simple model presented here does not only allow to analyse possible effects on the total effort required by all applicants, it permits also to analyse the distribution of these efforts across different groups of participants.

Not surprisingly, participants who are not successful in the first round are substantially better off than in a single stage call. On the other hand, all applicants admitted to the second round will need to invest more efforts.

In the numerical examples presented above, the “savings” for participants not successful in the first round (in red colour in the different boxes above) are in the order of 70 to 80%, whereas the relative “loss” for the participants in the second stage ranges from 20 to 40% (always compared to the scenario in a single call).

The result of this analysis is somewhat at odds with the political priority to stimulate excellence: Excellent and good consortia will have to invest (may-be even considerably) more, whereas the lower basic effort required might attract even more applicants which are “far from excellence”.

8.      Less effort, more happiness?

The examples presented above illustrated that there is after all a fair chance to cut the overall effort required from call participants through two-stage calls.

It is a somewhat different question, whether this would also lead to higher degree of “satisfaction” among the participants. Unfortunately, the picture is not as a clear and positive as one would have hoped for:

• Depending on the elasticity of demand (see chapter 3), the number of proposals submitted in the first round is very likely to be substantially higher than in a single stage call. Since the total number of projects funded will remain unchanged, this results in a substantial increase in the number of unsuccessful applicants.
• In parallel, the success rate will drop further – at least as long as one defines the success rate in the classical way as the ratio between proposal submitted (in the first round ...) and proposals funded.
•  Successful applicants are unlikely to complain (despite increased efforts required), and there will be probably less complaints from totally unsuccessful applicants (as these invested considerably less efforts). Applicants admitted to the second round, but finally not successful, represent an important minority of applicants (around 30 to 40%) which is very likely to complain, as they had to bear substantially higher efforts without any tangible reward.

9.      Some simple recommendations

The purpose of this paper is not to argue against two-stage calls. The analysis carried out confirms that there are good chances to reach substantial cuts in the overall effort required.

The analysis also highlighted, however, that the success of this implementation will depend on a number of crucial factors, which would need a very careful ex-ante analysis and close monitoring.

The following recommendations seem to be of particular relevance:

•  If first stage applications become too “heavy”, the net advantage of a two-stage procedure fades away. Requirements at this stage should focus just on issues relevant for the admission to the second round – any further add-ons will reduce the potential efficiency gains to a large extent.
• The elasticity of demand is probably rather different from one research area to another. Although it is impossible to anticipate exactly the future number of applications, it is obvious that in areas with a high number of potential applicants the introduction of a two-stage procedure might lead to a massive increase in proposals, which puts at risks the efficiency gains initially hoped for.
• Evaluators need to be briefed clearly and should be made aware of the potentially negative consequences of a too “nice” attitude in the first round.
• Commission guidelines for the second stage proposals should not only specify the required content, but also present examples of “useless” efforts. Evaluators should also be briefed accordingly to fight a culture of “more is always better” and to avoid a tendency for putting more and more efforts in such proposals. An obvious example for such a costly, but rather irrelevant effort is the development of a logo at the proposal stage...
•  The rather awkward effect of putting more burdens on strong applicants and fewer burdens on comparatively weak applicants has some explosive potential. While a limited extra effort might be acceptable to most applicants at the second round, it is important to respect these (probably rather narrow) limits, as going beyond these might mean that the frustration among the best participants might gain extra ground.
•  Since the final outcome of a change in the call structure is by no means clear, it seems important not to become dogmatic about this issue and to monitor the next calls very carefully. Based on the analysis presented above, it is likely that two-stage calls will reduce the overall effort required in many cases, but there will be also situations where the classical single-stage calls remain preferable.

Annex:

Table 1: Data used for analyzing the effects of different variables

Version 1.0 - 08.11.2016 - Thanks for your feedback