We reconstruct the optical response of nanostructures of increasing complexity by fitting interferometric time-resolved photoemission electron microscopy (PEEM) data from an ultrashort (21 fs) laser excitation source with different harmonic oscillator-based models. Due to its high spatial resolution of ~40 nm, PEEM is a true near-field imaging system and enables in normal incidence mode a mapping of plasmon polaritons and an intuitive interpretation of the plasmonic behaviour. Using an actively stabilized Mach–Zehnder interferometer, we record two-pulse correlation signals with 50 as time resolution that contain information about the temporal plasmon polariton evolution. Spectral amplitude and phase of excited plasmon polaritons are extracted from the recorded phase-resolved interferometric two-pulse correlation traces. We show that the optical response of a plasmon polariton generated at a gold nanoparticle can be reconstructed from the interferometric two-pulse correlation signal using a single harmonic oscillator model. In contrast, for a corrugated silver surface, a system with increased plasmonic complexity, in general an unambiguous reconstruction of the local optical response based on coupled and uncoupled harmonic oscillators, fails. Whereas for certain local responses different models can be discriminated, this is impossible for other positions. Multidimensional spectroscopy offers a possibility to overcome this limitation.