I'd take some suggestions from here and try to fix the ticking this way:
First try it with a battery. (make a little battery-to-DC jack adapter) If there's no ticking, try the filtering suggestions. If the ticking remains even when battery powered, you don't need to add filtering to the power supply, but something else.
I'd try grounding the output when bypassed. That might work...
If all else fails, sacrifice the LED indicator and use that third pole to stop the oscillator when in bypass. You don't really need the LED for that effect, right?
