3.2 Methodology: Between Quantitative and Qualitative Approaches

While case study research was the standard method in the first two waves of research, quantitative analyses have become more and more popular in the third wave. The analytical advantage of these quantitative approaches is their broad empirical basis, which usually spans several decades and policy areas as well as all or almost all member states. This has to be regarded as an asset in itself, since case study research, while often times being able to establish interesting causal relationships, has problems in proving that the patterns identified are actually representative for other cases. At the same time, however, quantitative studies often struggle with finding appropriate indicators to measure the concepts that are considered theoretically relevant and with identifying causal relationships behind the correlations they find. As indicated by the rather inconclusive results described above, quantitative EU implementation research seems to be ridden with similar problems. The main difficulties appear to be associated with the data employed to measure the dependent variable. Those studies that use the transposition information reported in the Commission’s annual reports and the transposition measures reported in Celex and other databases have the first major drawback that these data are restricted to the legal phase of transposition. But even if we accept completely disregarding issues of enforcement and application, these data have a second major problem: they do not give any indication of the correctness of what has been officially notified by the member states. Cases where transposition is being notified on time are thus treated as if all (legal) requirements had been met dutifully no matter how incomplete or incorrect the notified laws may be. This method thus seriously underestimates the actual size (and probably also the shape) of the transposition deficit.

That this is a major drawback can even be shown by using other statistical data: A quick look at the official Commission data on infringement proceedings against non-compliant member states reveals that non-transposition is only part of the story. Among the 517 directive-related infringement proceedings that were transferred to the European Court of Justice between 2002 and 2004, only about 60 per cent concerned cases of non-notification (own calculation based on CEC 2005 : Annex II). Scholars who look at transposition rates and notification data only thus turn a blind eye to the remaining 40 per cent of the cases. This alone should raise serious doubts as to the appropriateness of this kind of data.

This brings us to the second type of quantitative studies, which uses infringement data. These data are certainly less distorted than the official transposition data, as they include at least some of the actual cases of incorrect or insufficient transposition. However, in-depth empirical case studies have clearly demonstrated that these data are also far from perfect. Due to a serious lack of resources, the Commission is only able to systematically detect and pursue cases of late notification, while many cases of inaccurate transposition – and even more so cases of insufficient enforcement or wrongful application – slip past its attention. Thus, Falkner et al. (2005 : 204-205) conclude that the Commission’s infringement data only represent the “tip of the iceberg”, which does “not necessarily say much about the size or the shape of those parts that remain below the waterline”.

One way out of this unpleasant situation was suggested by Mastenbroek (2005 : 1113). She makes the case for a combination of both statistical and qualitative methods, for example by conducting qualitative case studies to scrutinise the findings derived from statistical analyses. However, this would not solve the central problem. A few additional case studies will not suffice to actually scrutinise the results of quantitative studies that are based on data of rather poor quality. Instead of starting with the weaknesses of statistical analyses and trying to eradicate them by qualitative studies, we could also proceed the other way round. Collaborative research projects, such as the ones carried out by Siedentopf and Ziller (1988 ) or Falkner et al. (2005 ), have successfully demonstrated that qualitative research does not have to be confined to a small-n setting. Unless the data problems associated with quantitative studies have been eliminated, trying to carry out more of these medium-n qualitative studies seems to be at least as worthwhile as the option of supplementing quantitative analyses with a few additional case studies. This is especially true if we want to learn more about enforcement and application, an area where no quantitative data are at hand at all.