To be fair to the DWP, ex ante assessment of the impacts of policy change is difficult. Especially when the impacts rely upon behavioural effects that are unknown and unknowable in advance. So when it modelled the savings from the implementation of the so-called bedroom tax it was always going to represent a broad estimate. The DWP quite rightly conducted sensitivity analysis to see how robust its conclusions were to different assumptions on key variables.

However, once the policy is in place real data will begin to emerge and ex ante assessments can be recalibrated. That is what Rebecca Tunstall of the Centre for Housing Research at York has attempted. Her report, published yesterday, has generated considerable political and media interest.

The brief report is well worth reading, for several reasons.

It is interesting because the starting point was a Freedom of Information request, submitted to gain access to the DWP’s model. The DWP responded with the skeleton of the model and its outputs rather than the inner workings, so the researchers had to reverse engineer the model to work out how the financial impacts of the benefit changes had been calculated. I’m not sure I can think of another housing project that has had to start from FoI. But I may just have missed it.

The report then considers the assumptions used by DWP and notes that some are prima facie, and even without bringing new data into the picture, rather odd. The proportion of housing benefit-dependent households in the social rented sector used in some of the scenarios was well out of line with anything likely to be observed in reality. But perhaps the most significant and questionable assumption is that the policy change would deliver the projected savings of £480m in 2013/14 only if it is assumed that no tenants moved as a result of cuts to their housing benefit. Whatever the proportion of tenants who move as a result of the policy turns out to be, it certainly won’t be zero.

And, as Tunstall doesn’t quite phrase it, the slightly questionable assumptions are biased in the same direction. They lead to a higher estimate of projected savings.

The same picture emerges when real data is substituted for assumptions in the DWP model. It turns out that, in the first four or five months of policy implementation, tenants are not behaving in the ways that the model assumed. And the differences are all such as to reduce the likely savings from the reduction in housing benefit. Tunstall suggests that, simply by making this adjustment to the model, a quarter to a third of the projected savings disappear.

Finally, Tunstall lists a range of other costs and consequences that a holistic assessment of the policy should take into account. In many respects these are not news. They are points that have been made repeatedly by a number of commentators, including me, both in the run up to policy implementation and beyond.

My gut feeling is that if these additional costs are appropriately weighed in the balance then the overall ‘savings’ from underoccupancy penalty will start to look rather like a net cost.

In Parliament today Nick Clegg referred to ongoing DWP-commissioned independent research into the impact of the housing benefit changes. He expressed some concern about the possibility of adverse impacts in very specific cases, if not about the direction of policy overall. In this respect he is somewhat at odds with his own party policy.

But Esther McVey, the new Welfare minister, was quick to dismiss the CHP research as based upon too small a sample to be taken seriously. The research was based upon a total of 127,494 homes in England, which represents 2.6% of the UK social housing stock. I’m not sure what Ms McVey’s qualifications are as a social researcher, but I’d say that was a very reasonable sample upon which to start saying intelligent things about what’s going on. Clearly, more observations are typically desirable – and the CHP research is suitably cautious on the point – but there is nothing in the nature of the sample to suggest that the results are implausible. There is nothing to counter the argument that this research provides a better understanding of the likely impacts of the policy than the original DWP impact assessments. We’ll have to see whether the DWP evaluation will move things even further forward.

At a more general level, Tunstall’s report is a good example of what genuinely independent commentary and analysis can bring to policy debate. It may, of course, be inconvenient to policymakers seeking to put a particular spin on the situation. But that’s as it should be.

Tunstall does not pass judgement on the policy, but does pass judgement on the quality of the analysis used to justify the policy. The report concludes that “there appear to be serious shortcomings in the DWP model used to project savings”. That seems to me to be entirely fair comment.

Image: Plymouth by Steve Cadman via flickr.com under Creative Commons