Magnetic particle imaging is a new modality, which allows the determination of the spatial distribution of magnetic nanoparticles in-vivo. A standard approach for magnetic particle image reconstruction employs a so-called system matrix. The system matrix is typically acquired by a calibration measurement, whereby the system response at numerous positions in the field-of-view is measured. Due to the measurement process, the system matrix contains noise, which affects the reconstruction of the particle distribution. In this paper, the special structure of the system matrix is exploited for noise reduction by applying frequency domain filters. It is shown that image reconstruction with the denoised system matrix yields an improved resolution and a better signal-to-noise ratio.