There are several sources of what sounds like distortion on a circuit like that.
One, of course, is the dynamic limitations of FETs. You may want to consider reducing the gain of the op-amp a bit since that might be pushing the FETs in unpleasant ways.
Another source of apparent distortion, which isn't really distortion comes from envelope ripple. The envelope follower in any compressor attempts to balance out responsiveness, in terms of tracking instantaneous changes in level, with smoothness. Maximizing one will invariably impede the other so they aim for a compromise.
When the tracking of the envelope follower is not smooth enough, the little variations introduced as the note trails off result in small fluctuations in sweep of the control element (in this case a FET) that sound like a burr added to the note and very much like distortion. This is more true of designs that use a half-wave rectifier circuit (which both the OS and Dr Q do) than those using a full-wave rectifier, and also more true of designs that use a speedy FET or transistor control element (like the DQ and OS) than those using a more sluggish optical control element.
Just for the heck of it, increase the value of the 4.7uf capacitor to ground just after the diode. You can either tack on another cap (e.g., 4.7 or even 10uf) or replace the one there. If the "distortion" appears to be reduced with only that change, then that is your problem. You can also consider increasing the value of the 100k resistor to 330k or even 470k to extend decay time a bit more. A slower decay will generally increase the apparent smoothness of a simple half-wave rectifier circuit like that in the OS.
