Unless the audio just happens to be at zero volts at the point you switch, there will be some voltage on the relay - after all, AC voltage spends very little of its time close to zero. And if the little metal springy contacts bounce at all (they probably do) then that voltage gets switched on and off rapidly with each bounce. That makes a pop.
I think expecting relays to be silent is a bit unrealistic. Some of them are quiet, but silent? Not really.
BTW, that's a completely different mechanism of making a pop than the usual thing you get in pedals, where you've got a bias voltage to avoid, although the fundamental problem of springy metal contacts remains.