However, modular invariance is neither sufficient, nor necessary for a CFT to be consistent.
Not sufficient, because there are more consistency conditions to be checked -- for instance, that the spectrum closes under fusion. And not necessary, because a CFT can be perfectly consistent on the sphere, and not make sense on the torus -- but, apart from string theorists, who really needs the torus? In spite of these limitations, modular invariance in rational CFT has been a very useful tool for classifying possible models, including such complicated theories as Gannon's heterotic WZW models.
The question is whether modular invariance can be a comparably useful tool in non-rational CFTs -- by which I mean CFTs whose spectrums contain a continuous infinity of representations of the symmetry algebra. The problems which arise in non-rational CFTs are:
- the torus partition function is infinite,
- in certain regularizations, it depends on a density of states, a not necessarily fundamental or well-defined quantity,
- reconstructing the spectrum from the torus partition function is difficult if not impossible.
Of course we should begin with the case of the simplest non-trivial non-rational CFT -- Liouville theory.
The case of Liouville theory
Problems 1 and 2 were neatly avoided in the proof of modular invariance of Liouville theory, by considering the torus one-point function instead of the partition function. The one-point function is perfectly finite, and does not depend on any density of states. So, adding one field on the torus could be considered as an elegant regularization of the partition function. Notice that the regulator is the momentum of the added field -- the one-point function does not depend on its position.
The disadvantage of considering the one-point function is the complexity of the calculations. Instead of the characters which appear when computing partition functions, we now have one-point conformal blocks. The computation was done in Liouville theory using a non-trivial relation between such one-point blocks and four-point blocks on the sphere, which reduced modular invariance to crossing symmetry of a sphere four-point function. In addition, the calculation makes use of the three-point structure constant -- in the case of Liouville theory, this is given by the DOZZ formula.
Such calculations of torus one-point functions can therefore be used for proving the consistency of a non-rational CFT on higher genus Riemann surfaces. But this can only be done once the CFT has been solved on the sphere. Such calculations are therefore not helpful when it comes to doing a basic consistency check of the spectrum, and even less for exploring the space of possible CFTs.
The case of the 2d black hole
The 2d black hole is described by a CFT which is more complicated and also more interesting than Liouville theory. Some effort has been devoted to investigating its torus partition function, but as far as I understand its modular properties are not yet fully understood. Actually, the modular properties of the characters of the corresponding supercoset are surprisingly intricate, and in particular exhibit a holomorphic anomaly. So these objects have interesting mathematical properties. However the physical motivations for studying them are less compelling, since the spectrum of the model has been known for a long time, and there is little doubt that it is correct.
The problem with extended symmetries
In non-rational CFTs with only Virasoro symmetry, reconstructing the spectrum from the torus partition function is a priori not completely hopeless: the spectrum is encoded in the density of states, which is a function of one variable (namely the conformal dimension), while the torus partition function is a function of the modulus. Of course there are difficulties: possible multiplicities of representations are hard if not impossible to detect, integration domains are ambiguous as they can be moved by shifting contours, and the density of states is not really a fundamental quantity, as it depends on field normalizations.
However, in non-rational CFTs with extended symmetries, reconstructing the spectrum is a priori hopeless, because representations depend on several variables, whereas the torus partition function is still a function of one variable. The only way out is to find a generalization of the torus partition function which depends on several variables while still having good modular properties. There is no reason for this to generally exist: the good modular properties of the torus partition function ultimately comes from the geometrical realization of the Virasoro algebra as the algebra of conformal transformations, while extended symmetry algebras have no such realization. Nevertheless, this exists in the case of affine Lie algebras -- the symmetry algebras of WZW models, where exponentials of currents can be inserted in the partition function.
In the case of conformal Toda theories, such a generalization of the torus partition function is not known to exist. The symmetry algebra of such theories are W algebras, and it would be natural to insert exponentials of W currents in the torus partition function. But the resulting object does not have nice properties under modular transformations.
Exploring simple modular invariants
There was recently a brave attempt to explore the space of possible non-rational CFTs using modular invariance. The basic idea is to construct manifestly modular invariant objects, and to try to interpret them as torus partition functions. And this seems to lead to non-trivial constraints on the underlying CFTs if they exist. However, not a single example of the modular invariant objects is the partition function of a known CFT. And the proposed spectrums are in general not closed under fusion. Therefore, more work would be needed for fully understanding the proposed modular invariants.
In non-rational CFTs, the efforts to study torus partition functions of known CFTs on the one hand, and to classify possible modular invariants on the other hand, have not yet made contact. Moreover, the study of modular invariance, while it has given rise to beautiful mathematical objects, has not produced much physical insight into non-rational CFTs. This may be due to a lack of effort and to the difficulty of the subject. However, the study of non-rational CFTs has been making much progress using a variety of other techniques. And when it comes to performing a quick consistency check on a proposed spectrum, closure under fusion is not much more difficult to use, and not much less constraining, than modular invariance.
So, it could be that the experience with rational CFT is misleading, and that the torus partition function is not a particularly useful object in non-rational CFT.