I tried to replace R10 from 12k to 18k, but it didn't bring any improvement to Up/Down action. After that, I tried a Filter switch, as proposed by Mark, and it brought some changes, although I like it the most when switched to 3n3. However, I'll leave it as on my schematic.
Switching that switch produces a thump, so any advice how to avoid that is welcome!
Next, I started experimenting with C12 value. Putting 1u there made sweeping shorter and also brought bigger "dead" zone at Up/Down pot CW side (and perhaps some at CCW), so it seems that this capacitor is the main problem. At 1u, dead zone is 22k big, compared to 3k6 with 10u.