We report on the continuous measurement of fluorescence lifetimes at low light levels. Fluorescence photons following pulsed excitation generate a pulse sequence with exponentially distributed amplitudes and interphoton times at the output of a time-to-amplitude converter. This sequence is turned into a continuous step function and is time averaged with an adjustable bandwidth. For a single-exponential decay, our approach yields identical results as would be obtained from fitting fluorescence decays, while being a real-time technique. The proposed technique performs especially well at low count rates. We demonstrate the applicability of the method at the example of confocal fluorescence lifetime imaging of single molecules.