Mind the counterfactual

As we've argued before, whenever anyone makes a claim about the cost or benefits of the EU, look out for the counterfactual - i.e what would be the case if the UK weren't an EU member.

The classical "3 million jobs would be lost if the UK left the EU" claim is one of the most conspicuous examples of a rogue statistic without any credible counterfactual attached to it.

Of course 3 million jobs wouldn't be lost over night if the UK left the EU, because all trade with Europe wouldn't suddenly disappear (even North Korea trades with the rest of the world). But, as we also have argued, Better Off Outers who believe a world of unilateral trade and pareto optimal regulation would ensue on the day after the UK left the EU are also making huge leaps of faith. 

On that note, UKIP’s Tim Congdon takes issue with something we said in a blog last week in his latest group email. When highlighting problems with the CBI’s recent cost-benefit analysis we were a bit loose in our critique of Congdon's study measuring regulatory costs, pointing out similarities with UKIP’s own analysis (right) in terms of the counterfactual.

We argued that Better Off Outers have:

"Mastered the art of measuring EU costs with no reference to a counterfactual (see [Tim Congdon’s How much does the European Union cost Britain?] – for a fine example). To a large extent, the CBI study falls into the same trap."

To be fair to Congdon he has described a counterfactual in his UKIP paper:

"The UK’s departure from the EU would allow us to adopt unilateral free trade, which economists tend to regard as the best policy."

And in his email:

"In my study I estimated the costs of EU regulation and protectionism assuming that the EU would carry on pretty much as it had been doing for the last 20/30 years, and that the UK outside the EU would be lightly-regulated and pro-free trade."

So, he does provide a counterfactual - apologies for saying otherwise. There is still a problem however: it's a completely unrealistic counterfactual. The UK would not turn into to Hong Kong on Day 2. However much one would like to see all the costs of the EU to evaporate and for the UK to adopt unilateral free trade, with corresponding unilateral liberalisation on the EU's part - covering everything from rules of origin for UK-exported cars to "passports" (if you can trade in one country you can trade in all) for UK-based fund managers. Do you really trust Whitehall and Westminster that much? Also, under article 50, the Mediterranean bloc could stop EU trade liberalisation for a very long time. Call us pessimistic but that’s life.

Bringing the point back round to cost benefit studies in general, contrary to what Congdon suggests in his email, we believe determining the counterfactual for the UK outside the EU is actually pretty difficult (see our report 'Trading Places' for a full discussion of this), and providing a quantitative assessment of it is even harder. As the Channel 4 fact check blog highlights, this difficulty has lead to a very wide range of estimates on the costs and benefits of EU membership.

In any case, our wider point still stands, that studies of this type are full of caveats which hamper their effectiveness, no matter how tempting putting a single number on the costs/benefits of EU membership sounds.

Is the CBI right to claim the net benefit of the EU to the UK is 4-5% of GDP?

Is the CBI right to claim
the EU benefits UK GDP by 4-5%?

As we discussed in our previous post, the CBI's report on Europe, published today, makes for interesting and thought-provoking reading. However, the CBI’s estimate of the net benefits of EU membership isn’t the strongest part of the report.

The CBI puts the net benefit at between £62bn and £78bn, or between 4% and 5% of UK GDP. This, it claims, is equivalent to £3,000 per household or £1,225 per individual. Credit to the CBI for trying to inject some hard numbers into the debate. However, putting a single figure on the costs and benefits of such a complex arrangement is notoriously difficult – as we ourselves know too well. The CBI does readily admit that in its report, and to be fair, very clearly qualifies its figure.

 

Still, there are at least three problems with this figure, which in combination means it should be taken with a huge pinch of salt.

 

Arbitrary extrapolation based on a highly limited literature review: Ultimately, the figure is taken from a literature review of previous estimates of the benefits. On average, the literature surveyed puts the net benefit of EU membership at between 2%  and 3% of GDP. The review covers only five pieces of literature with the most recent one being from 2008. This is a very small pool of literature to draw from. More critically, the CBI goes onto assert,

“Since these studies are not mutually exclusive…it is not unreasonable to infer that the net benefit arising from EU membership is somewhat higher than 2–3%, perhaps in the region of 4–5% as a conservative estimate.”

It’s widely accepted that EU membership comes with unquantifiable benefits, but the CBI takes a massive leap of faith here. It seems to suggest that the net benefits of different aspects can be tallied up given that they are not mutually exclusive – and it may well be that dynamic effects triggered by, say, EU market access mean net benefits are often underestimated. However, curiously, the CBI provides no proper explanation or evidence for why it settled on 4% to 5%, leaving us guessing where this extra net benefit actually comes from. Having discussed this with the CBI, it’s clear that they have aggregated the net benefits of various aspects of the EU from different studies.

 

While there is logic in this approach, given the diverse nature of the studies it is tricky to simply add parts of the up, assuming that they work the same together as they do in isolation. After all, there is a reason why these studies have struggled to produce a clear figure for all the areas themselves. Once this approach was chosen more detail should have been included in the CBI report, even if it meant adding a statistical or economic annex (given the hugely sensitive nature of the EU cost-benefit debate).

 

No proper counterfactual is given: What are these net benefits measured against? Does the CBI assume that the UK outside the EU would be left with no trade deal and no single market access? If not, then the net benefit is presumably lower than the 4% to 5% identified. This is the classical shortcoming of most EU cost-benefit studies. In the CBI study, the problem is exacerbated by the fact that the various pieces of literature that it draws from will themselves have diverse counterfactuals.

It’s all the more surprising given that the CBI rightly goes to great lengths to discuss the counterfactuals when it comes to the costs of EU membership, arguing for example that some EU regulations would remain even if the UK left the EU given that they would be domestically needed or pursued by international groups  (which we agree with).

 

Why the same rigour is not applied to its calculation of the net benefits of the EU is not clear. UKIP-types have mastered the art of measuring EU costs with no reference to a counterfactual (see here for a fine example). To a large extent, the CBI study falls into the same trap.

 

Benefits aren’t evenly distributed: Lastly, in an understandable attempt to present an easily digestible figure, the CBI converts its percentage number into pounds and presents it as an evenly distributed benefit over households and individuals. Of course, this is unlikely to be the real benefit felt by households or individuals, with any benefit distributed unevenly and in ways which are nearly impossible to measure.