As an essential part of nanoparticle-on-mirror (NPoM) plasmonic nanocavities, metal mirrors play an important role not only in determining the optical response of the nanocavities but also their application performance. Here the effect of mirror quality on the optical response of nanosphere-on-mirror (NSoM) and nanocube-on-mirror (NCoM) nanocavities is experimentally studied. Polycrystalline sputtered gold films (SGFs), template-stripped gold films (TSGFs), and single-crystalline gold microflakes (GMFs) are investigated and compared. Due to the great improvement in the surface roughness that can minimize fluctuations in the gap morphology, NSoM and NCoM nanocavities formed on smooth TSGFs and GMFs have a better cavity-to-cavity homogeneity in the scattering spectrum than those formed on comparably rougher SGFs. In addition, there is an obvious change in the spectral positions of the resonance modes of NSoM and NCoM nanocavities formed on SGFs due to the variation in the gap morphology, which is reproduced very well by theoretical calculations based on measured dielectric functions of the gold films. Finally, due to the reduction in electron scattering losses from SGFs to TSGFs and GMFs, an increase in the quality factors and scattering intensities of the resonance modes is observed for nanocavities formed on the corresponding films.