In practice, 5.25" drives equalled or exceeded the capacity of 8" drives when 5.25" floppy drives started using HD media.
The Evolution of 5.25" Drive Systems
There are only two different recording media that were ever commonly used in floppy drives: the original media (often called "DD") and the later HD media. Note that when it comes to floppy diskette drive systems the term "density" is somewhat confusing because it's used not only for media (i.e., the composition of the recording surface) but also radial track density (e.g. 48 or 96 tracks per inch (tpi): a property of the drive, not the media) and the encoding of bits into flux transitions (a property of the controller, not the drive or the media).
Over time, physical capacity was expanded in various ways. The following roughly traces the evolution 5.25" diskette systems. The sizes are for popular formats but could easily vary by 20% or more in either direction, depending on sector size and other factors. (These answers mention a variety of formatting changes used to put more data onto a floppy using the same physical density specifications.)
- The original single-sided single-density systems used original media, 40 tracks (48 tpi) and FM encoding for a capacity of around 90 KB/diskette. (Some early drives could address only 35 tracks.)
- Single-sided double-density (SSDD) systems used original media, 40 tracks and MFM encoding for a capacity of around 160 KB/diskette.
- Double-sided double-density (DSDD or sometimes "2D") systems obviously doubled the above to about 320 KB/diskette.
- The move to 80-track (96 tpi) drives, sometimes called "quad-density" or "2DD" doubled that again to 720 KB. But note that the double- to quad-density increase was via a completely different method than the single- to double-density increase, via a change to the drive head rather than a change to how the controller wrote data to the diskette.
- HD drives used a different media that approximately doubled the linear recording density, and they started out as 80-track drives, so they stored about 1.2 to 1.44 MB/diskette. For reasons I won't get into the most popular 5.25" format was 1.2 MB (less than double) even though 3.5" systems (which are essentially slightly miniaturised 5.25" systems) exactly doubled it to 1.44 MB.
8" vs. 5.25" Drive Systems
8" drive systems in common use followed the first three steps of this progression, but at the same recording density offered both more tracks (77 instead of 40) and more sectors per track (typically 26 × 256 byte sectors versus 16 × 256 byte sectors for 5.25"). Thus, in one common format a DSDD 8" diskette stored around 2×77×26×256/1024 = 1000 KB whereas a similarly formatted 5.25" diskette stored 2×40×16×256/1024 = 320 KB. The most common format soon changed slightly to increase the sizes to 1.2 MB for an 8" diskette and 360 KB for a 5.25" diskette.
5.25" drives started their progress towards catching up when drives with narrower heads running at 96 tpi were introduced, bringing their standard capacity to 640 or 720 KB. But this was still considerably less than 8" drives. It was when HD media was introduced that the standard 1.2 MB 2HD 5.25" format approximately equaled the capacity of the 8" DSDD format.¹²
Capacity was immediately extended beyond this for special cases such as Windows installation diskettes; a change in formatting allowed 1.5-1.8 MB on an HD diskette. But the same formatting tricks (mainly, larger sectors for less overhead) were possible on 8" drive systems, though I don't know if they were ever used.
¹ In the case of the NEC PC-9801 series, it exactly equaled the 8" format because NEC for whatever reason decided that they would use exactly the 8" format: 77 tracks (even though the drive supported 80), the same number and size of sectors and, strangely enough, even running the drive at the standard 8" rate of 360 RPM rather than the 300 RPM at which 5.25" drives normally run.
² The 3.5" HD diskettes stored 1.44 MB instead of 1.2 MB because they used 18 instead of 15 sectors per track. They were capable of this because, as part of the process of "shrinking" the 5.25" diskette to 3.5" whilst maintaining exact format compatibility they had to increase the coercivity to compensate for the smaller surface area. DD/HD coercivity for 5.25" media was 300/600 Oersteds; for 3.5" media it was 660/720 Oe.)
Further Development
8" drives were perfectly capable of improving in the same way as the 5.25" drives did, moving from 48 to 96 tpi and DD to HD media. Indeed, someone had obviously manufactured such drives because Maxell manufactured media for them. But by the time these improvements were available 8" drives had become relatively expensive and inconvenient compared to even 5.25" drives, not to mention 3.5" drives, and this is probably the reason why 96 tpi HD 8" drive systems were very rare. But if you don't take this rarity into account, 5.25" drives never caught up with 8" drives in storage capacity.
3.5" drive system capacity continued to increase, though HD media was the last system that received truly widespread acceptance. 3.5" ED drives ("Extended Density," with a typical formatted capacity of 2.88 MB) introduced new ED media and perpendicular recording and were fairly widely available, though they never achieved the widespread acceptance of HD drive systems. By this point 5.25" drive system development, like 8" drive systems before it, had fallen off, but even so there was a 5.25" 2.44 MB drive system (probably also using ED media) standardised enough to be available from at least two manufacturers.
Eventually laser-servo designs bumped 3.5" capacity up to 20 MB and beyond. The only format here that achieved any significant popularity were the LS-120 and LS-240 SuperDisk formats (120 and 240 MB, respectively). I don't know how difficult it would have been to transfer laser-servo technology to 5.25" or 8" systems.
Further Information
A good source of many fascinating details about many kinds of floppy drive systems is Herb Johnson's Tech information on floppy disks drives and media web pages, which provided many of the references above. (But I may be biased since I am, in a small way, a contributor to those pages.)