We developed a highly sensitive measurement system of AC magnetic fields for magnetic nanoparticle imaging (MPI). First, we developed a detection system using pickup coils made of cooled Cu wire and HTS tape. The pickup coils were connected to a resonant capacitor in order to enhance the signal voltage generated in the pickup coils. The magnetic field noise at the resonant frequency was as low as 90 fT/Hz1/2 and 13 fT/Hz1/2 for the Cu and HTS coils, respectively. Next, we demonstrated the detection of nanoparticles using third-harmonic signal generated by the nonlinear magnetization of nanoparticles. An excitation field having a frequency of 1 kHz and an amplitude of 1.6 mT was applied to the magnetic particles and the third harmonic signal at 3 kHz was detected in order to reduce the interference from the excitation field. We demonstrated the detection of 0.1 mg of magnetic nanoparticles. We obtained a clear contour map of the magnetic field from the particles, and could detect the particles located as far as 100 mm under the pickup coil.