DIYstompboxes.com

It took a while to get this up, but I think the results are worth it. You can see it at http://www.youtube.com/watch?v=RrufhPHi6m8 (http://www.youtube.com/watch?v=RrufhPHi6m8)

You all know I'm a fan of blind tests to eliminate the biases we all have. This is the first blind tasting test of effects that I know of. The audience was friendly , but blind; that means that they were willing to come to a blind test as a favor, but that they did not know which pedal they were listening to during the test. They were not paid, except for the snacks and sodas we provided, plus any entertainment they got from watching and voting.

We set up a number of categories of pedals and did a blind test of the sound using a live guitarist playing the same riffs for each pedal; same guitar, same amp, same player for each pedal, and multiple live samples of each pedal to listen to.

The Visual Sound pedals did well, as we had hoped they would. There are some surprises in there as well. Your ears may hear things differently, of course, because everyone's taste is potentially different. But I think this is a good (and fair - we worked hard at that!) illustration of a number of things about judging pedal sounds. I think you may be interested in this set of videos for a number of reasons. See what you think.

We are having DVDs made.

Did the players know which pedal they were playing? I think that there are subtle effects that can occur to the way a player plays depending on his/her expectations. Also, who handled settings on the pedals?

Ahhhhh....that DVD. :icon_biggrin:

That's a whole helluva lot of work, both for the player and the rater.

You should get Bob to talk to Gordon Logan at Vanderbilt (if he hasn't already).  Gord and I go wayyyyyy back to when he and his wife first arrived at McGill from University of Alberta.  Besides being an acknowledged world-class cognitive scientist with a distinguished career and very active laboratory, Gord is also a pretty decent guitar player, avid collector of vintage gear (I'm pretty sure that George Gruhn has bought some of his many pets from the proceeds of sales to Gordon), and from what he tells me still gigging intermittently.  Gord would hip Bob to the "perfect" blind test....not that these aren't well done already.

Quote from: Ben N on February 23, 2009, 12:59:44 PM
Did the players know which pedal they were playing? I think that there are subtle effects that can occur to the way a player plays depending on his/her expectations. Also, who handled settings on the pedals?

Good questions, and really pertinent ones for a fair test.

There was only one player, primarily to keep player variation down. While it is conceivable that Zac could have ferreted out what pedal was what, we tried to eliminate that, and for just the reasons you mention. From where Zac stood it was difficult to see what pedal was being used, and there were so many that locating the active one would have been difficult. We also did a lot of back and forth between pedals to ensure that there were enough samples that everyone listening could get a firm idea whether they liked Pedal #1 or Pedal #2 better. I believe a lot of that has been edited out, just because it's incredibly tedious to watch all that if you're not participating. Zac was also cautioned to do his best job of making the riffs the same. In addition, we ran Zac through a prelim session where we watched for Zac acting differently on some pedals versus others. We did catch him leaning back slightly on some pedals versus others as he played, and coached him to stand the same, play the same, keep facial expressions the same, and not to try to see what was being tested. The pedal order being tested got fairly random too, as the method was A-B comparison, where the audience voted whether Pedal #1 or Pedal #2 was better. The (to them, at the time) less-good pedal was eliminated, and another pedal in the same group (like overdrive, or distortion)  was subbed in to compare to the winner of the first comparison. Which pedal was #1 and #2 got changed a lot, and it was difficult to follow if you were not personally the one pushing the buttons. I was in the audience and knew what was happening, and it was hard for me to follow in my head. So, yes, it is conceivable that the player's response to the sound changed the way he played. But we tried really hard to wash that out.

As for knob settings, yes, that has a huge effect on the sound. Not only do the settings have an effect, the fact is that pots typically have a +/-20% tolerance on their base value. A "100K" pot may be 80K to 120K and the maker would ship it out just fine. In that kind of setup, to get rid of that variation, we had a pre-shootout shootout where we and the studio crew sat in the audience position and voted on knob settings ahead of time. We did the A-B comparisons to deliberately make all of the pedals in a class (e.g. distortion or compression) sound the same. We reasoned that it's easy to make them sound different, just diddle the knobs. And if you want to "prove" that one is better than another, you can diddle the knobs to make it come out that way for some settings rather than others. But if you deliberately try to make them the same, it reduces the variations. One subtlety is that sheer output level was tested for the same. In two setups where even identical pedals are tested blind, people will tend to think of the slightly louder one as "clearer" or "cleaner" or "better". So we and the studio crew tried to wash out the level variations ahead of time as well.  Pedal settings was a group issue, where we tried (- blind!) to get them to same level, same sound as much as possible.

Quote from: Mark Hammer on February 23, 2009, 01:10:04 PM
Ahhhhh....that DVD. :icon_biggrin:

That's a whole helluva lot of work, both for the player and the rater.

You should get Bob to talk to Gordon Logan at Vanderbilt (if he hasn't already).  Gord and I go wayyyyyy back to when he and his wife first arrived at McGill from University of Alberta.  Besides being an acknowledged world-class cognitive scientist with a distinguished career and very active laboratory, Gord is also a pretty decent guitar player, avid collector of vintage gear (I'm pretty sure that George Gruhn has bought some of his many pets from the proceeds of sales to Gordon), and from what he tells me still gigging intermittently.  Gord would hip Bob to the "perfect" blind test....not that these aren't well done already.

Yeah, that DVD.  :icon_biggrin:  I didn't realize that they would put it on YouTube. I'll get you a hard copy anyway.

I suspect that Bob might be interested in talking to Gordon. We do know that these are not theoretically perfect blind tests. The bottom line is that they were as close to a theoretically perfect blind test as we had time and money to do and still have some results that would be interesting to see, which was also important to us. Not much use in having a great set of results that no one will see because it's so boring to look at. As you're well away, controlling extraneous variation to isolate the thing you're testing is a huge effort in tests like this. Or it should be, if you're after an honest test.

Were the pedals hooked up in series or were they each totally isolated with a TB looper or something?

Great video though, loving it!


Hahahahahah the drummer in the Op-Amp vid!  :D :D

The tricky part of that comparison is that the pedals are deliberately set up to sound as identical as possible.  While that accomplishes the goal of assessing whether it is possible for 4-6 pedals to achieve the same single end-point (and obviously it is), the fact remains that any of those pedals is not necessarily going to consistently provide the same range of settings as any other. So, A and B might be identical twins for tone when controls x, y, and z are set for their mid-points, but B has more gain at max, or less treble push than A at max tone, and so on.

I'm certainly not trying to paint the shootout as biased in any manner.  It isn't.  Not in the playing, and not in the blind presentation.  But when Bob holds up the various pedals and says "This is the Fnortenizer, that costs around $250, and this is the Magic Hunga-dunga that sells for around $450", the listener/viewer/rater is then supposed to ask themselves "Well if they sound the same, why the hell am I spending all that money on pedal X?".  The answer to the question may well be because X has more range on all of its control parameters, and is capable of copping not only what pedal Y can do, but an awful lot more.  It's a credit to Bob's integrity that he doesn't judge or milk that, but at the same time, the naive viewer has to remember that they are listening to an assortment of pedals trying to achieve a singular sonic objective and not any broader range than that.  If the goal they aim for is yours too, then clearly if you can nail it in amore convenient and dependable package for $100 less, great.  As Prince says, "There's joy in repetition", but judgments about a pedal ought to be based on what it can do for you, in total.  Otherwise, pedals start to become the equivalent of cheap keyboards that have a few canned sounds and insist that you leave your creativity and sense of adventure in the closet at home.

Again, this is not a harsh criticism.  Rather, it is a challenge that pretty much any blind test faces: "How do I fairly demonstrate, in blind fashion, the range of things these two or more devices can do?  How do I demonstrate both areas of overlap AND nonoverlap without the individual rater monkeying around in hands-on fashion and identifying the device (and consequently being influenced by brand knowledge)?".  Certainly this shootout comes as close to ideal as any I've seen, and comes a LOT closer to the ideal than simply comparing youtube or tonefrenzy soundclips.  But there is still a lot missing.  I think it is useful to the whole industry to figure out ways to get around that conundrum.

Still kudos for even attempting, and for going public with it. :icon_biggrin:

Great work. I especially like the Op Amp comparison.  ;D

I do agree - range matters. Also, a big part of the Mekkano-Anime DethBlaster may be how it sounds when all the knobs are full up/down/12/9/etc. and if a user is looking for that given sound, knowing that it can also be adjusted to some other tone setting may be completely extraneous. I didn't fully appreciate how hard controlling the variables is and I knew a smattering about testing going into it. I'm hoping that the response we get to this will let us do some more advanced testing to make public.

Very cool, thanks for doing that!!

But you should have used a true bypass loop for each pedal, as buffers before or after a pedal definately change the tone of the adjacent pedals. For example, running two TS9s in a row, the 1st one will sound different than the 2nd one. (kurtlives was hinting at this). I saw him pressing the pedals' buttons directly so it seems that all were connected.

Also,  as Mark mentioned, the Klon is no better than a TS at those settings but the Klon at the clean boost setting does things no TS clone match.

have fun!

Perhaps the sequel will put the various effects in "blank" boxes and leave the player to it.  The boxes could be labeled "volume, tone," etc, but otherwise no distinguishing markings.

It is an industry joke that when psychologists who specialize in perception (the study of psychophysics) do their doctoral research work, the "sample" usually consists of the Ph.D. student, their advisor and the student's girlfriend or boyfriend.  Why?  Because to do solid blind assessment of "Is example 1 more X than example 2?" over the full range of possible values and combinations requires just hundreds and hundreds and hundreds of training/testing trials, and those 3 people are the only ones that would put up with that crap without being paid. :icon_lol:   This includes presenting the same thing as both example 1 and 2, in order to calculate false alarm rates.  So, setting a Route 808 as both pedal A and B and asking people "which one do you like better?".  So, like I say, even though this is an admirable attempt, compared to what those who study human perception as science (as opposed to market research) normally do, there's a big gap still to be bridged.

It is also worth noting that such blind testing really only starts to become pertinent as the market gets cluttered with both medium-to-large-scale production pedals of every sort of national provenance (all of which can be purchased on-line from the same we've-got-everything-you-could-ever-want retailers) that all really DO make a point of trying to farm the same little strip of land.  While it makes perfect sense now, this sort of exercise would have been rather silly to contemplate in 1979.

P.S.: I used to have a Mekkano-Anime DethBlaster  I changed the input cap to plastic film and the first few resistors to metal oxide.  Sounds MUCH better.  More Deth, more Blasting.  But that titanium chassis?.....ugh!!  Couldn't drill holes for extra toggles to save my life.  Just HAD to rehouse it in a nice black plastic Hammond box from Radio Shack. :icon_wink:

I started falling asleep in the overdrive vid- they all sounded so similar. I'm not an overdrive guy I guess...

The distortion vid was fun. You could really hear the differences. Box of Rock as well as the Hyde sounded great.

My faves so far are the op amp and true bypass "mythbusters" though. They are both very effective. Hopefully a few more light bulbs will go on over people's heads when they see these.

Good stuff R.G.! Tell Bob to keep it up! :icon_wink:

As RG points out earlier in this thread, there are component tolerances to deal with.  The mistake that anyone doing pedal-to-pedal comparisons makes far too much of the time is that, whether blind-rated or not, there can often be unit-to-unit variations that impact on the final tone.  So, if you like pedal A more than pedal B, is that because pedal A will always sound that way, relative to pedal B, or is this just a "lucky unit"?  Translating this into statistical terms, how much variance in rater evaluations is attributable to the model/brand, and how much is attributable to the individual unit?

Really, that is simply another version of what happens with on-line soundclips, or with tryouts at a friend's house or music store.  Occasionally one hears something under circumstances which favour A over B, or which misrepresent the overall average value/quality/tone of A or B.  The mistake is to confuse what is true of an exemplar with what is true of the signal/stimulus under ANY circumstance.  That is also one of the reasons why one normally controls for false-alarm/hit rates.  If I'm tired or distracted or whatever, I could easily say that A is less whatever than B, even though A and B are actually identical.  Similarly, if pedal A has some unique constellation of component values, relative to some other exemplar of pedal A, I might prefer A to B because A gets shown off in some manner which, though not always true of A, still makes it seems "better" than the exemplar of B.

Again, these are not criticisms, but simple realities in the measurement of human perception and peferences as a function of the objective physical realities being experienced.  It's a tough gig, whether you're a pedal manufacturer or a doctoral student.

Interesting vid and yet more pedals presented for our examination.

Any shortcomings in the comparisons I can think of have already been addressed, buffers interacting, and some pedals perhaps not having their best attributes demonstrated and the inevitable trade off between rigorous A/B procedure and having the auditors gnaw their legs off to escape. But the comparison is what it is and it was produced by a manufacturer with products in play.

Thanks.

Very cool, indeed.  I too liked the op-amp vid. 

Kinda like being at the optometrists  ;D

Cool set of videos. Can you set up something like this where people are playing thru different length guitar cables or a stompbox with one type of capacitor vs a different type of capacitor in a second otherwise identical circuit? I'd like to see how many people can really hear the differences between these things like they think they can.  :icon_twisted:

Some of that stuff is already done in the demo CD that comes with Dave Hunter's recent book on guitar effects pedals.

I totally enjoyed this shootout, what i noticed in both the overdrive and dist vids was that most people were more impressed with the more bassy sounds (including me). The one point  i would make about this is that in a full band situation the results could be more positive towards the trebly sounds.

I am massively impressed with the noise floor in the hyde pedal when it is in series with another pedal.

Quote from: Ice-9 on February 23, 2009, 03:15:51 PM
I totally enjoyed this shootout, what i noticed in both the overdrive and dist vids was that most people were more impressed with the more bassy sounds (including me). The one point  i would make about this is that in a full band situation the results could be more positive towards the trebly sounds.

I am massively impressed with the noise floor in the hyde pedal when it is in series with another pedal.

I regularly take the bassier, ballsier distortions over the thinner sounding ones every time.  :icon_wink:

Good example of gathering information and putting out exposure for your products at the same time.
I enjoyed it !

Great and very ambitious project! Very carefully planned and executed. A/B testing is very difficult to do right, and you really took it as far as practically possible here.

As a comparison, there was a show on Swedish TV recently where they did A/B tests between cheap/expensive instruments and had some celebrities try to tell the difference. Now guess what - one of Swedens most seasoned guitarrists couldn't tell the difference between a vintage '56 Les Paul and a cheap $100 clone. How is that possible? Well, the actual test had a guitarist with very moderate talent slam each instrument for about 10 seconds through a Peavey 5150 with really high gain, drenched in delay and reverb...

Quote from: Ice-9 on February 23, 2009, 03:15:51 PM
I totally enjoyed this shootout, what i noticed in both the overdrive and dist vids was that most people were more impressed with the more bassy sounds (including me). The one point  i would make about this is that in a full band situation the results could be more positive towards the trebly sounds.

I am massively impressed with the noise floor in the hyde pedal when it is in series with another pedal.

I totally agree with this! You hit the nail on the head.  ;D

Also, I would use a system, where the voting could happen anonymous.

Maybe, it was better if the name of the pedal was on the screen for us. It's hard to follow.

Overall an interesting and fun demonstration. In the field of overdrives all came across as remarkably unremarkable; that is, all sounded very similar. Among the more dramatic presentations were the Visual Sounds`s buffering qualities on the Myth busters true bypass section and the impressive low noise floor when in series with another high-gain pedal.

10,000.000 hits stomp switch sounds good too. The guy who tested it must have blisters on his bunions.

Quote... you should have used a true bypass loop for each pedal, as buffers before or after a pedal definately change the tone of the adjacent pedals. For example, running two TS9s in a row, the 1st one will sound different than the 2nd one. (kurtlives was hinting at this). I saw him pressing the pedals' buttons directly so it seems that all were connected.

It certainly affects pedals that do not have a high input impedance. Pedals with an input impedance over some threshold tend to be less affected. Then there's the issue of inter-pedal capacitance in the wiring and connections. Buffered versus true bypass affects tone? You bet it does. That's the point, and one of the points of the true bypass mythbuster. True bypass lets your pedal input and output get loaded with all the cable either ahead of or behind it, as well as the buffer changing things, as some people claim. I think it becomes a point of discussion, and even better testing, as some have noticed here, whether it's the buffer affecting tone or the buffer UN-affecting tone compared to true bypass. Then on top of that there is pedal sensitivity too. I'd like to do that test someday.

QuoteI totally enjoyed this shootout, what i noticed in both the overdrive and dist vids was that most people were more impressed with the more bassy sounds (including me). The one point  i would make about this is that in a full band situation the results could be more positive towards the trebly sounds.

That's probably true, and I suspect it's a built-in bias of guitar players. When we play alone, we like full sounding guitars. When we play in a band, we're looking for something to cut through the mix and be heard, and that's often more treble heavy. I have caught myself at this several times in my own playing, such as it is.

On the other hand, all of the pedals being tested had more available treble in the tone pots to go. None of them were running at full-treble setting, so all of them could be zooped up that way.

QuoteAlso, I would use a system, where the voting could happen anonymous.

Yeah, that's one of the things that we just couldn't manage within the range of the available time and money. Something like a personal voting pad for each audience member, or interviewing them all one at a time (eeek! Run the test sixty times??)

QuoteMaybe, it was better if the name of the pedal was on the screen for us. It's hard to follow.

It was hard to follow if you were there, too. That's one reason I think the test tended toward unbiased in that respect. It would have been hard to memorize a sound and look for it to vote for it to nobble the results. You really did have to think about did the most recent tone sound better than the last one.

Quote from: GREEN FUZ on February 23, 2009, 04:38:34 PM
10,000.000 hits stomp switch sounds good too. The guy who tested it must have blisters on his bunions.

Yeah. The guy who had to do that lost a bet with us...  :icon_biggrin: 
Actually, we use an ALPS brand switch that ALPS says is good to 10M operations. We figure they know more about it that we do.

So where in the pedal order did you put the Visual pedals?

Hmmm... I realize that I don't know that detail. I could ask Bob where they were. Bob was the guy working the board, and would be most familiar with it.

I was focusing so hard on the external part - the blind, coaching the guitar player not to favor or disfavor one, stuff like that, I left the board and operation to Bob and so I don't have a clear picture of which pedal was where compared to the others.

Should I have controlled or randomized that?

RG, you know I am normally very critical especially with these comparison things. Having said that I think you did a pretty fair and decent job. I have experience of years with commercial pedals and my only criticism is not with the guys here or the public response, but sadly the larger companies feeling threatened might feel they need to come down on you for this. Again it is really sad, as I know you did your level best to make the whole thing as unbiased as possible. The good news is other companies might not be threatened yet, but as you continue to succeed a response might be in the offing. Anyway well done RG and I commend you for your efforts. :icon_smile:

thank you, it really is a valuable resource, and I'm sure I will refer back to it many times.

hmmmm....besides being skeptical of these type of commercial ads, i also come from the same school of thought as analogmike... after watching the youtube clips, it did appear that the visual pedal was second in every chain of pedals if counting from guitar. I could be wrong. Placement surely would have an impact on tone, specially since i saw it running directly after a buffered boss pedal, etc. I know your boss knows this and remember some of your amazing insights into this, so that doesnt seem like something that would be overlooked. Randomization of that experimenting parameter would help the results, as would not letting the riffmaster see which pedal he was using, which were well within his peripheral sight...that is clear from watching the video. Also a show of hands is not very scientific either...you see guys looking around for affirmation before they vote...how about filling out some surveys and then tallying at the end...and how can you extrapolate winning results if you are going head to head one at a time? did he just do visual vs. A, visual vs. B, etc and declare Visual the winner if it beat the majority of them in a head to head or did it only have to beat one pedal head to head? He even double checks one of the results when it appears the Visual pedal might have barely lost...man this is psychology 101. Lastly, is the one that beat Visual in any given test the overall winner for that category? And the thing about 10,000, 0000 hits on the footswitch...come on people, this is a rating given when tested from a machine that taps the switch repeatedly in succesion in a completely "unhuman" way...those crapp blue toggle switches are rated for like a million latches, and i can guarantee you they would never last that long under actual playing conditions. I would focus more on the ease of replacement and less mechanical parts, which are both positives.

I am very critical when it comes to these type of things (ala my degree). Why not do it right the first time? This wouldnt even pass the very basic criteria for a valid scientific experiment with statistically significant results...i hate to be the total %^&*ah here but let's call a spade a spade. This is an informercial for Visual with all due respect to RG. We have all seen this type of "blind" experiment with soda, food dehydrators, magic cleaners, leg hair removing wax, you name it.  It's a ploy to sell pedals while simultaneously giving a "shout out" to the very pedals that were the inspiration for the Visual pedals in the first place. Obviously the CEO is a decent man, but it's equally obvious he's a business man too.

But it definitely did teach me how similar some of these pedals sound which was very cool. For that alone it was valuable. Especially the overdrives. Also, I have played Visual pedals and am impressed with the way they sound, which is no surprise. Thanks for the link. Cool post nonetheless.

Quote from: reverbie on February 24, 2009, 05:24:07 AM
But it definitely did teach me how similar some of these pedals sound which was very cool. For that alone it was valuable. Especially the overdrives. Also, I have played Visual pedals and am impressed with the way they sound, which is no surprise. Thanks for the link. Cool post nonetheless.

I was just blown away by how discriminating professional musicians were ... hell all the overdrives sounded the same (and just as good as each other) to me. 

I agree with some of the previous discussion that some fx may have unique "distinguishing features" that others don't share, which won't necessarily show up in a comparison like this. But regardless, I think as general "consumer education", these videos are really a great service (with the understanding that Visual Sound has a vested interest in you choosing their product, of course).

But IMO the two "mythbuster" vids were the most effective. They did a great job of demonstrating the fallacy of deifying one narrow aspect or piece-part of a circuit like this. If nothing else, these are systems of components. Everything depends on everything else.

It's great to see the education taking place. Especially in a place like Nashville, one area in the U.S. anyway where a lot of the intense hype seems to be generated.

  Well I did notice a telltale Workhorse hubcap on the wall, was the speaker in the wall?, hard to tell the shape of what the hubcap [there's a better name, I'm having fun with...it looks like a hubcap and that's the only best way I can describe so you know exactly what I'm talking about]...
 and that the bass response of the amplification was quite smooth, yet well defined.
 Not exactly typical, more like what I'd choose when I'm choosin' though.

Quote from: DougH on February 24, 2009, 08:06:22 AM
I agree with some of the previous discussion that some fx may have unique "distinguishing features" that others don't share, which won't necessarily show up in a comparison like this. But regardless, I think as general "consumer education", these videos are really a great service (with the understanding that Visual Sound has a vested interest in you choosing their product, of course).

But IMO the two "mythbuster" vids were the most effective. They did a great job of demonstrating the fallacy of deifying one narrow aspect or piece-part of a circuit like this. If nothing else, these are systems of components. Everything depends on everything else.

It's great to see the education taking place. Especially in a place like Nashville, one area in the U.S. anyway where a lot of the intense hype seems to be generated.

Yeah, the mythbuster videos were great. It's funny how people in general so readily buy into a bunch of hogwash like a JRC4558 opamp being better or worse than an RC4558 opamp. They're the same thing! :icon_lol: But I already knew that.  :icon_wink:

Quote from: DougH on February 24, 2009, 08:06:22 AM
But IMO the two "mythbuster" vids were the most effective. They did a great job of demonstrating the fallacy of deifying one narrow aspect or piece-part of a circuit like this. If nothing else, these are systems of components. Everything depends on everything else.

I watched the IC test (coudn't bear to watch the bypass vid, conclusion is predetermined). Of course 4558 chips will sound about the same. I don't know what that video amplifier chip was (probably VERY similar to a 4558 but higher slew rate for higher frequencies?).
But why not test the TA75558 chip which is the one that everyone removes from the ts9?

I can tell the difference betweem the JRC2043 chip and a 4558 chip even at a noisy guitar show, and so did everyone
who listened when I demod it back in '94 at the first Classic American Guitar Show in NY. If you can't hear that, other hobbies beckon ;)

Have fun!

Perhaps instead of coaching the guitarist to not visually cue people about his taste you could give him a stool to sit on and have him face away from the audience (and the pedals) when he's playing.

Also, while it wouldn't be feasible for how you were doing this test, it might be cool to mic the amp and give everyone an identical set of headphones, I imagine that a lot of those pedals are going to sound better in the front row than in the back row.

I think it would be cool if you got together with some other manufacturers and did another one of these at one of the big guitar shows or other industry events and had an impartial team (maybe guys from industry magazines or retired designers or psych students/music instructors at the local college) running the experiment.  I think it would be fun to attend one of these.

QuoteI can tell the difference betweem the JRC2043 chip and a 4558 chip even at a noisy guitar show,[...]

I hear differences between TI4558 and JRC4558's (and TL072's, LF353's, etc) as well in my tube driver circuit anyway. The question is, are they spellbinding tonal nirvana differences or are they noise level?  The differences are not equivalent to the hype generated or golden tone promised by changing one component.

People tend to believe what they feel they need to in order to rationalize their last purchase...

Quote from: DougH on February 24, 2009, 10:03:54 AM

QuoteI can tell the difference betweem the JRC2043 chip and a 4558 chip even at a noisy guitar show,[...]

I hear differences between TI4558 and JRC4558's (and TL072's, LF353's, etc) as well in my tube driver circuit anyway. The question is, are they spellbinding tonal nirvana differences or are they noise level?  The differences are not equivalent to the hype generated or golden tone promised by changing one component.

Good question, the 2043 and 4558 have about the same amount of noise. (For those of you who are not TS homos, the 2043 was used in many original 1980s TS9 pedals).  Try a 2043, I think you will agree it might be good for garage rock sounds, or something like the FACES sound, as it has a very gritty, slightly nasty tone. The 4558 is a lot smoother and better for blues, classic rock, jazz etc.

What I meant to convey in my post, is that the outcome of the opamp mythbuster is not the definitive conclusion, as they failed to test chips that I know would sound different in the VS OD pedal (meaning, NOT all op amps sound the same).

Quote from: DougH on February 24, 2009, 08:06:22 AM

It's great to see the education taking place. Especially in a place like Nashville, one area in the U.S. anyway where a lot of the intense hype seems to be generated.

You hit the nail right on the head-I'm in Nashville and can attest to that. There's a lot of "trendy" users here, too. For example, in my circle of musicians I was the first cat to build and gig with my own bypass boxes- basically got laughed at. Then when those same people caught up to the concept, it was all about who has brand-a or brand-b looper on their board (AND they started "educating" me about it!). I reacted by taking mine off and using as many cheap, buffered pedals as I could (for a while, at least).

I figure anywhere there is a big recording scene, lots of studio musicians, performers, etc, the hype will be spun up into high gear. :icon_wink:

I think the tests show that not only can all the overdrives be made to sound similar, but that the rest of the gear is so important. Almost any overdrive into a slightly distorted quality vintage or boutique tube amp can make great sounds. A Landgraff or Klon into a Behringer, not so much. I always notice when I watch YouTube videos of pedal tests by ProGuitarShop that it sounds like the test amp has been perfectly set to make the pedal sound as good as it possibly can. They make a point of saying they always use the same amp, solo65, whatever that is, for all the tests but you can tell when he plays the bypass sound that the SETTINGS of the amp are not the same for each video. So a lot of iffy pedals sound just like a great tube amp because, wait for it, they are pushing an already breaking up TUBE AMP with the bass turned way up.
I'm reminded of when I played in my last band. My rig was a Gibson 335 style Ibanez into a Traynor BassMaster. That amp is very JTM45 sounding and I played into a single 12" sealed cab I made myself. It was nearly the size of a 2-12" cab. Big crunchy sound. On the foor I had a tuner, wah, delay, boss eq, and 808 tubescreamer. The eq was mostly flat and only used for lead boost on the bridge pickup. The tube screamer was set for max volume and minimum distortion. I only used it for lead boost on the neck pickup. I usually was on the bridge pickup. And usually neither pedal was on. People would compliment me on my tone all the time and go,"ahhhh TS808!"   :P
Regardless, I am impressed by the low noise of the Hyde and think that there are good reasons to build or buy quality pedals. Great tests!

The videos are interesting to watch. They tell me a couple of things: Pedals sometimes sound different. Pedals sometimes sound the same. Different people like different things.

Regarding the scientific side - it would be a shame to reduce music into science.

Anti-parallel series diodes noisegate somewhere along the signalpath towards the end  :icon_wink: ?

I liked all the videos. Cool stuff, and kudos to all the people over at Visual Sound ;)
I really like the new nifty features on those new Visual Sound stompboxes. I particularly liked the Son of Hyde. If memory serves me right, it's based around the Marshall Shredmaster, isn't it?

Also, I'm curious whether the preferred pedals would be the same if they had been tested in a different order....

With all due respect to reverbie:

Quote from: reverbie on February 24, 2009, 05:24:07 AM
hmmmm....besides being skeptical of these type of commercial ads, i also come from the same school of thought as analogmike... after watching the youtube clips, it did appear that the visual pedal was second in every chain of pedals if counting from guitar. I could be wrong. Placement surely would have an impact on tone, specially since i saw it running directly after a buffered boss pedal, etc.

Skepticism is good. It is in fact skepticism that led to us doing this. But you are correct - we did have dogs in the hunt, we admit that up front, and we did a lot of backstage work to try to eliminate that as an issue.

You bring up an issue that may need tested, or at least eliminated by the design of a future experiment. By the way, as Mark can tell you, just the design of experiments is a technical specialty, all on its own. I have some familiarity with it, but am by no means a technical specialist in this field.

Does placement in the chain of similar devices, on its own, change sound enough to change the results of such an experiment?
Clearly, I don't know, because (a) I didn't think of that ahead of time and (b) I don't know the order of the effects in the test for sure well enough. I think this issue is worthy of a test all its own, designed properly. Maybe another way to look at this is I can assure you that there was no attempt to nobble the test results by careful, crafty placement of Visual Sound pedals second in line every time (second in line being the magic, perfect place to be?!?).

As an interesting sidelight, the first or second position in the test is the toughest position to be in, because the winner of that matchup must be individually compared against every other pedal in the lineup to be selected overall. Succeeding positions are in fewer matchups. So being in the first pair means being tested directly against every other pedal to be overall winner. First and second pedals have a rougher test - no drawing a bye into the semi-finals as in sports competitions.

Quote
I know your boss knows this and remember some of your amazing insights into this, so that doesnt seem like something that would be overlooked.

I've never either heard him say anything on the issue of second-in-the-chain being better, and never heard it mentioned by others. It is possible that he could have known it and never said anything over the ten-plus years I've known him. But I'm clueless about my own amazing insights into this. Can you point me to some of that? I apologize if I'm seeming dense or deliberately coy - I'm not. Maybe I don't understand what you mean by "knows this" (i.e. what "this" is, other than being second in line.)

QuoteRandomization of that experimenting parameter would help the results, as would not letting the riffmaster see which pedal he was using, which were well within his peripheral sight...that is clear from watching the video.

OK, I'll go for that, as I've already said. All I can say is that he was instructed not to shade the results and that I was watching for that. This is one of those things that was not made impossible by the design of the experiment, other than trying to play fair, which was happening; of course, someone (this is not accusative, I do not mean this as an ad hominem attack) who did not like the results will not be able to accept the "playing fair" thing. It wasn't perfect, as I've said before - just all we could do within the time and money available.

QuoteAlso a show of hands is not very scientific either...you see guys looking around for affirmation before they vote...how about filling out some surveys and then tallying at the end...and how can you extrapolate winning results if you are going head to head one at a time?

Agreed. Audience-to-audience interaction is an issue. A perfect test would have done the entire mess one person in the studio at a time, as I've already said. Beyond that, sixty people is not a very large sample size, Mark's notes about the advisor, student, and girlfriend aside. That's not perfect either.
This can only be taken as an indicator. However, the resulting head-to-head looking was fairly done, because the audience could look from one to the other for all pedals equally while voting.  It could have worked against any pedal as well as for it. It's a confounding issue (as they say in the design-of-experiments biz) instead of evidence of favoritism.

Quotedid he just do visual vs. A, visual vs. B, etc and declare Visual the winner if it beat the majority of them in a head to head or did it only have to beat one pedal head to head?

It was single elimination. Two pedals of unknown brand are compared, a vote is taken. The loser is eliminated. Next pedal is compared to the winner of the first comparison. The nominal "best" pedal is compared to all of the others with at most one level of indirection. So if the first or second pedal tried was eventually the best, it was compared against every single pedal. If the third pedal was eventually best, it was compared to all the other pedals and only indirectly to the loser of the first pairing. Basically, a pedal only gets judged second best once. This is a valid way of picking "best" (presuming the method of judging one versus the next is accepted as valid), but it does not produce a one to N ranking, and does not produce an N by N matrix of every-to-every comparisons. In particular, no, it was not Visual vs A, Visual vs B, etc. The Visual Sound pedal also would be eliminated on any single loss.

Quote
He even double checks one of the results when it appears the Visual pedal might have barely lost...man this is psychology 101.

Can you point out which section of video that is? I'd like to review it.

Several votes were close - which indicates only that the performance of the pedals was close, equal numbers of people liking one versus the other better, and that abstention was allowed. I'd like to see whether Bob's talk to the audience about ties was on the video there. Bob did explain just this issue, that in fact there may be ties, but that we did, for purposes of eventually selecting one as best, need to pick one, and that where that happens, he'd pester for abstainers to weigh in.

There's one other issue of preconceived results going on here. You say "He even double checks one of the results when it appears the Visual pedal might have barely lost". I'd like to see (a) if it is obvious to the audience that a Visual Sound pedal was in contention when the results were double checked (b) if the Visual Sound pedal was in fact the purported loser before the recheck and (c )whether this was the only instance of a re-check. Your implicit assumption in saying this is that Bob is manipulating the results by recalling votes until the audience gets it the way he wants it. As you say, Psychology 101. Assumptions like that need testing.

You don't know Bob, so I'm sure your skepticism is based on other owners of effects companies. And that's a reasonable basis for this. However, if you did know Bob, and had participated in the run up to this, you'd know that Bob would rather just write off the money to do the test than manipulate the results. But you have only my word for that, and that's suspect too since I work for Visual Sound. So once again, I'm down to - I was there, it was done as fairly as we could, within the constraints.

You don't mention the most straightforward version of faking the results - running the tests and simply declaring the winner to be who you want. A truly unscrupulous person would just do that. Why fool with letting the guitarist favor one pedal by playing better for some pedals versus others, place all your pedals second in line, or re-call votes until you get what you like? Faking is easy. Doing it honestly is hard.

QuoteLastly, is the one that beat Visual in any given test the overall winner for that category?

No, see the discussion of the testing method. The Visual Sound pedal could have been eliminated at any test. There was no attempt to simply compare Visual Sound to each other pedal, then declare it the winner. The Visual Sound pedal, if it was really second in line every time (which I don't know, see above) would have to be compared to every other pedal directly to be the overall winner. A Visual Sound pedal would be eliminated the first time it lost a vote, just like any other pedal in the contest.

Quote
And the thing about 10,000, 0000 hits on the footswitch...come on people, this is a rating given when tested from a machine that taps the switch repeatedly in succesion in a completely "unhuman" way...those crapp blue toggle switches are rated for like a million latches, and i can guarantee you they would never last that long under actual playing conditions. I would focus more on the ease of replacement and less mechanical parts, which are both positives.

You're correct about the testing method. A machine did the test, I'd guess. As I said, I didn't do the testing of the switch. ALPS specifies that in their product literature. Do I expect 10M operations? Of course not. Do I expect it would last ONE million from a switch rated for ten million, if anyone ever used a pedal that much? Yes. I think that's a valid engineering and human factors thing to do. Do you agree?

One million operations is one operation per second for 11.6 days, working 24 hours a day. Or, for a four hour gig, pressing it twice per three minute song, 6250 gigs, five a week (you're on tour) gives 25 years of five gigs a week for 50 weeks a year. That is, much more than anyone including professional touring musicians will ever give the pedal.

On the toggle switches. Those are indeed rated for much fewer operations, largely because toggle switches rated for 10M are simply not available in sizes that fit. However, I suspect (- but cannot prove  :icon_lol:  ) that no player ever flips the toggle switches twice per song for every song in a gig. Of course, we don't claim the toggles are rated for ten million operations, either.

We did try to make the mechanical plunger simple and easy to replace, as well as non-critical. The actual electrical switch, if it ever needs replacement, is currently $1.13 in ones from Mouser (and is a stocked Mouser part) and can be replaced by just soldering a new one into the small PCB which holds the tactile switch.

Thanks - making that cheap and easy to replace was deliberate.

Quote
I am very critical when it comes to these type of things (ala my degree).

As you should be - see my comments on skepticism above.

QuoteWhy not do it right the first time?

As I addressed - time and money. Would you like to participate in funding the perfect test?

QuoteThis wouldnt even pass the very basic criteria for a valid scientific experiment with statistically significant results...

No such claim was made, was it? In point of fact, I believe I've done a lot of self evaluation already in explaining that there were several areas where it could have been done better in the sense of a perfectly designed experiment.

Statistically valid is yet another criteria. That's an easy thing to toss in, and I warned Bob ahead of time that we'd hear about sixty people watching and voting not being "statistically valid". People hear "statistically valid" and think "yeah, yeah, statistically valid" but they have no particular concept of the things that go into making a statistically valid test. I'll leave it at "statistically valid" being a much, much bigger test than was done. We did decide that some information was better than no information, which was the other alternative.

Quotei hate to be the total %^&*ah here but let's call a spade a spade.

As a personal observation, I find that the words "i hate to be..." can often be interpreted as "it gives me no displeasure to announce that ..."  :icon_lol:

QuoteThis is an informercial for Visual with all due respect to RG. We have all seen this type of "blind" experiment with soda, food dehydrators, magic cleaners, leg hair removing wax, you name it.  It's a ploy to sell pedals while simultaneously giving a "shout out" to the very pedals that were the inspiration for the Visual pedals in the first place. Obviously the CEO is a decent man, but it's equally obvious he's a business man too.

And this gets right down to the issue of belief. If you believe that it's all smoke and mirrors, you're going to disbelieve the results, no matter what the test, test methodology, etc. And that's OK. Believe what you want. Of course we're happy to see that we did well. (As I've already stated here... )Would we make this public if we came out on the bottom of this? Duhh...

On the other hand, did we fake the results, either directly (by simply announcing winners irrespective of the voting) or by subtly and cleverly manipulating the pedals, the audience, the guitarist, the playing, by maladjusting the competitors ahead of time (hmmm, one you missed!), by modify the competitors ahead of time (there's another), by having a "sucks" button that Bob could press with his toe on the floor (ayie! another), by modifying half the chairs in the audience to vibrate when the "right" pedal was in play, or by any of the several million other ways it could have been fudged? No. That can be believed or not.

I would not submit this to a peer reviewed scientific journal (hmmm... the Journal of Scientific Sound Evaluation?) as a perfectly designed experiment. Is it pure and utter balderdash and selfserving lala like some audio and guitar advertising. It is not. But that's an item of belief or not. I have to tell you that some research I've heard about in peer reviewed scholarly journals as the absolute, well-designed-experiment, statistically-significant results have turned out after a short while to have simply been made up. Whatcha gonna do when the guys in the lab coats fake their data?

Quote
But it definitely did teach me how similar some of these pedals sound which was very cool. For that alone it was valuable. Especially the overdrives. Also, I have played Visual pedals and am impressed with the way they sound, which is no surprise. Thanks for the link. Cool post nonetheless.

Nonetheless.  :icon_biggrin:
We expected most of this kind of evaluation of the effort, and some that you didn't mention as well.
In all seriousness, I would welcome your help in designing an unassailable test methodology for pedal sounds. 

I think this was a great test.  And whilst I listen to all the complaints about people saying: "It's not a fair test" blah blah blah.  I cant help but think that a lot of people missed the purpose of the test.  Which seems to be to debunk a few common myths and show the world the quality of product that Visual Sound produces.  The test is even set up to have Visual Sound products potentially fail against their counterpart pedals!  I dont see many if any companies doing that

A lot of you guys sound like you want to file a report against Visual Sound with the Stompbox Police.   Visual Sound was not obligated to provide a fair test, they werent even obligated to provide a test.  But they did, and very bravely I might add. 

We can all sit here and argue about pedal placement in the chain, cognitive studies etc.  But I think that would really be overkill for a test on this scale.

Great stuff.  Need more of it.  It's easy to poke holes in any testing procedure unless the budget is nearly limitless.  I already thought most overdrives and distortions sounded the same, EXCEPT FOR THE EQ.  That is probably the easiest difference for the ears to hear between devices that have gone thru a recording chain.  Even sitting with a breadboard on my amp and switching out op amps is hard to hear a difference, although you can.  Switching SI diodes to LED's or Ge's is easy to hear.  Changing the treble roll off cap is easy to hear as is the bass roll off cap.  Adding a notch filter and varying the notch is easy to hear, or messing with a BMP tone control.  Things like "feel" sustain, asymmetrical distortion, etc. are much more subtle and harder to pick up with a mic and reproduce with electro mechanical speakers.  You can make a Dist+ or Tube Screamer into a 100 different pedals just by messing with the caps that contribute to the EQ and/or the clipping diodes.  Way more AUDIBLE than the op amp, buffers, etc.   :icon_cool:

I've only seen the op-amp one so far, and greatly enjoyed it. Bob (?) did seem to be cheerleading a little - understandable given that the audience seemed to be flagging a bit.

An excellent video though - if money ever permits it I'd love to see more. I must remember to see the overdrive one.

Just a couple of technical points.

1) The order of presentation CAN matter.  If I "like" #2 more than #1 (or #1 more than #2), when the one I "like more" now becomes identified as #1 for the next comparison, there can be a bias to hear the one I "know I liked" as better.  Again, not a harsh criticism, just noting unintended factors that can influence a sequential comparison such as that used.

2) When psychophysicists (i.e., the student, girlfriend and advisor) do their comparisons, holistic "liking" judgments are not used that often.  More commonly, the perceiver/rater is asked to rate which of two stimuli is "more" of some particular dimension.  That dimension need not be an objective one, but it's better if it is a single one.  So, hearing two pedals and asked to provide a blind rating of which is "smoother" is reasonable.  And while I normally wince at the mention, I'd even accept "Which has better note definition?".  I have to find the article again, but there was a paper I dfound online from a guy who was a McGill music prof for a bit, and another guy from Japan, doing "semantic differential" ( http://en.wikipedia.org/wiki/Semantic_differential ) ratings of distortion pedals.

Quote from: Mark Hammer on February 24, 2009, 04:31:20 PM
Just a couple of technical points.

1) The order of presentation CAN matter.  If I "like" #2 more than #1 (or #1 more than #2), when the one I "like more" now becomes identified as #1 for the next comparison, there can be a bias to hear the one I "know I liked" as better.  Again, not a harsh criticism, just noting unintended factors that can influence a sequential comparison such as that used.

Makes sense. Always presenting the preferred selection as #1 in #1 or #2 choices would be obvious; noting a preference for #2 and always presenting #2 as the same unit would be slightly more devious.  :icon_eek:

And I need to note that I don't know whether Bob always kept the previously selected winner as its previous number (#1 or #2) or whether it became the new #1 (or #2 either), or if it was random. I suspect that random would be a more valid test.

Quote
2) When psychophysicists (i.e., the student, girlfriend and advisor) do their comparisons, holistic "liking" judgments are not used that often.  More commonly, the perceiver/rater is asked to rate which of two stimuli is "more" of some particular dimension.  That dimension need not be an objective one, but it's better if it is a single one.  So, hearing two pedals and asked to provide a blind rating of which is "smoother" is reasonable.  And while I normally wince at the mention, I'd even accept "Which has better note definition?".  I have to find the article again, but there was a paper I dfound online from a guy who was a McGill music prof for a bit, and another guy from Japan, doing "semantic differential" ( http://en.wikipedia.org/wiki/Semantic_differential ) ratings of distortion pedals.

That makes sense as well.

(I grinned to myself as I started to type this, knowing that you knew that I'd be typing it. :icon_biggrin: ) Could you help me suggest some suitable dimensions? I can think of a few, but just coming up with sensible dimensions is a tricky thing. For instance, "harshness" and "shrillness" are kind of close to "brilliantness" and "clarity", the difference being primarily the consonance or discordance of the harmonic content, maybe. Asking which is shriller versus asking which one has the most brilliant, clear tone might be one subtle way to influence outcomes as well. Maybe this has to be done like the telephone polling stuff without which a USA president can't make any decision; ask the same thing several slightly different ways to try to size up the response. Kind of like asking in a poll whether the pollee thought George W. Bush was a threat or a menace.  :icon_lol:

I would like to collect up these kind of things for inclusion in any further such testing. This kind of thing is a way we can all contribute to the formal state of the art in the effects biz, I think. Coming up with good practice for testing is a real step forward. Even better is coming up with some idea what you lose when it's less that perfect - like having all the participants (subjects? victims?  :icon_eek:) see the presentations at the same time rather than individually, and some means of doing private voting that's possible without collecting and counting paper ballots, etc. Or whether perfection is all or nothing for such tests.

Youse guys with some psych training, help me out here. What's good, what's better, what's perfect, and where is it that we have to get or it's worthless to try?

Could you help me suggest some suitable dimensions?

"which pedal has more haunting mids?"

Great videos. I was very impressed by the son of hyde floor noise.

Quote from: analogmike on February 24, 2009, 04:59:19 PM
Could you help me suggest some suitable dimensions?
"which pedal has more haunting mids?"

Haunting mids.!?

Kewl. I'll write that one down.  :icon_biggrin:

Quote from: R.G. on February 24, 2009, 04:55:41 PM
Youse guys with some psych training, help me out here. What's good, what's better, what's perfect, and where is it that we have to get or it's worthless to try?

It's worthless to try. But still worth it. I enjoyed the videos very much, because they show a lot of pedals under the same conditions. It gives a rare opportunity to hear the difference between some pedals under certain cicrumstances, but of course not all. And they also show that people like different things, to repeat myself. And to elaborate - you can do these kinds of tests and make sure they follow scientific standards all you want - different people will still like different things. That's the whole point of music. Some pedals are clearly better than others, but you simply can't extend that into saying that "this pedal is the best", which I'm sure you know.

It all comes down to taste. What suits people. And no such test can prove anything in that matter. In my opinion, even trying to apply the same scientific laws that the technology of the pedals follow, to judging the quality of the sounds they produce is a faulty strategy.

Note that a preliminary session like this for gathering information, put more questions on the table, discussing what are valuable questions, how to make a better test, etc. etc. is a valuable experience.  This is an engineering trial and not a "as scientific as possible" one (I think).  Doing an experiment like this is good for beginning to identify what could be issues that should be investigated with closer attention.  This test could not be scientific because it is not done by an independent examiner.  That told, it is not that it does not have any value.  On the contrary, by filming, taking notes, identifying problematic situations, the examiners can beguine to identify what are the needs of guitarists.  In other words it can be seen as a tool for gathering conceptual inputs.  So, if Visual Sound might want to do a survey or use an other tool for orienting their conception methodology, there will be some base material to construct on.  It is a necessity for a company to produce documentation, select and archive them to guide the designers work.  Some examples: they could, in the future, do: some interviews with musicians, doing some focus groups with different group of persons that are related in a way with the stompbox industry, market studies, etc.  All this to identify what are different needs and way to furfill them conveniently (base needs, performance needs, and innovation needs, etc.).

These informations, at a moment or an other will have to be analysed (and this is not necessarily a scientific process) with different tools, (graphical, tables, statistics ,discussions, etc.) as to put more structure in all the knowledge that has been gathered.  All qualitative and quantitative inputs are to be considered at first because they all might be points that will have to be considered in the overall quality of the product (and also it's quality image).  Example: If to put a X mythical chip is an option to be considered even if there is no scientific truth that it is better, the option should be considered, because that is what a lot of clients « want » and thus it is a selling point even if the company knows that there are perhaps no big differences comparing to other options.  But the company will know what their decision will be founded on, it will not be an arbitrary choice.

What I am not sure is the way that the shoutout was presented, I am not sure if it is a good idea to collect information, putting out exposure for the product and selling the merits of the company products at the same time.  I really wonder if there are not some pervert situations that might pop-up.  Example: when I watched the clips I did not like to see the boss making publicity for it's products at the end of the clips, while at the same time saying that he was not favoritizing Visual Sound's stuff.  There is nothing contradictory in that way of doing it, but it produce a suspicious feeling that can lead to bad thinking.  Also, the fact that it is not a scientific trial but a "public" shoutout done by the manufacturer: I am not convinced if it is a really good idea.  All this is all linked to the way we perceived the clips and what are our beliefs; nothing to do with what were the real intentions of the ones who have done the experiment.  Insisting to show that we are good guys will not always produce the desired results.  What most people believe is that the industry is there to make some money, not to reveal the truth.  If they are willing to reveal some truth it is that they are willing to make some money with it.  And there are a lot of manufacturers that give false truth.  So doubt will always be there somehow.  That is why I am not sure (in the real sense of it, not the "pejorative" sense) of the effect of those clips on the "general market".

The final thoughts when I watched it was : There seems to be a lot of goals here, what is the « real » conclusion or purpose of this trial finally.  What am I in front of.  It's obviously related with some commercial will...  These guys are playing two roles at the same time.  Can it really be sincere ?

My view is that some stuff should not be there in the clips.  I think that what the boss of Visual Sound is putting out at the end of the clips is a bit « too much ».  It doesn't add to the credibility of the company (or the clips).  Things like how silent the stompboxes are, should be in an other videos.  I have the impression that those videos are provoking mixed feelings that could have been avoided.  I also have the feeling that because we all know R.G. (and it is him who is presenting the experiment on DIYstompboxes), we tend to forget those points and because we know that he is a helpful guy and we are accustomed to his sincerity, we see the clips with an other eye than those who go directly on Youtube to see them.

Hope my comments will be of some help (and from my perspective I liked them, of course...)

@frank_p
Thoughtful and reasonable commentary.

A perfect test would have done the entire mess one person in the studio at a time
  Or a buzzer for each participant to keep individual responses secret to eliminate 'crowd voting'.
  Of course then it isn't 'readable responses' to the camera either. Raising of hands broadcasts responses completely transparently in a way that is immediately obvious as unadulteratable to the viewer.
 

When I was in grad school, I got interested in the chemical senses: smell and taste.  One of the reasons why we tend to know much more about hearing and vision than about smell and taste is because sounds and sights disappear as soon as you stop presenting them, so you can keep presenting another one and another and another.  In contrast, smells and tastes get lodged in the very organs used to sense them.  Little molecules of flavour find their way into those little potholes known as tastebuds.  Aromatic molecules get stuck in the mucus membranes.  Which means that in the space of an hour, you could present a couple of tastes for comparison maybe 40-60 times.  Presenting colour images and asking people to press a switch as fast as they can to indicate if the two samples are same or different can be done with at least 10x as many comparisons accomplished in the same time period.  Long story short: you can get much more research accomplished in the same amount of time for sight and sound than you can for taste and smell.

For taste research, at least, it is a standard practice to not only wait a little while between stimuli, but also to use a "palate cleanser" of distilled water; the idea being that you want a taste to be experienced in comparison to nothing, and not in comparison to what's left over from the last thing in your mouth.  Note that tasting things involves swishing them around in your mouth (and I think that happens because one has to search out available "unoccupied" taste receptors to maximize the sensation).  While there is certainly no shortage of sound receptors in a pedal comparison, the sonic phenomenon being evaluated requires something analogous to "swishing the sound around".  In other words, the effect is not perceived immediately, like a splash of colour or a teaspoon of salt in a gallon of water, or a brief spray of perfume.  It is something that is perceived over time.  Presenting it over time (in our case, some 5-10 seconds or so of playing) starts to become analogous to the prolonged exposure that naturally occurs with tastes and smells, so maybe an "auditory palate cleanser" is also needed.

In our pedal context, it seems to me that the palate cleanser ought to be a clean tone.  The question is, how does it get used?  One way is certainly to have some clean playing in between pairs of pedals.  But should a palate cleanser be used between pedals up for comparison?  Perhaps, but perhaps it should be something different than 5 seconds of clean strumming.

Or maybe, there should be a palate cleanser but NO gap or a very brief clean period between any two pedals.  I'm sure RG could design some little PIC-controlled switching thing that would let Zac or whomever play away, Bob or whomever hits a switch to get it rolling, and the circuit systematically cycles between clean and some random order of A or B.  So, for example, A-clean-B-clean-B-clean-A-clean-A-clean-B-clean-A.  The person/s listening then decide whether there is an audible difference in some designated dimension between them.  If it's a room full of people, you either rent those "instant polling" gadgets my senior management is so fond of at organization-wide annual meetings, or else you spend a couple of bucks and make yourself a multiplexed set of yes/no switches (a momentary 3-position toggle that returns to centre-off is perfect) that can feed a parallel port on a laptop.  Nobody sees hands, and nobody has to count.  The auto-switcher/randomizer can either keep the device being auditioned a secret (for "Is there a difference or not?"), or else can enable an indicator LED for those instances where you want people to indicate whether A or B has more of the designated dimension.

Maybe I'm getting too complicated now. :icon_wink:

I've suddenly had an idea pop into my head for what can bring the test further, although the chances of it happening is close to nothing.

Get the manufacturers of each pedals or their selected representative involved.

1. Each Manufacturer/Representative sets their pedal settings to what they feel will be liked by the majority of players. Their "best" setting if you will. They are allowed to audition everything else to be used in the actual experiment to aid in this setting. That is to say, the guitarist, the guitar, his choice of pickup, the riff he'll be playing, the amp, etc. Of course, a neutral judge will need to set all their volume levels to be perceived the same. Alternatively a decibel meter could be used if we want to be more scientific.

We know each pedal is capable of a wide range of sounds. But some pedals have a "sweet spot" or a certain setting which it is famous for, or that the manufacturer is proud of. The analogy is a chef is also capable of a whole range of dishes but is only allowed to showcase one when it comes to competitive judging. He brings the best he can muster and that he feels the judges will like.

2. Each Manufacturer/Representative sets up their pedal in the way they feel it should be best placed. Basically: Does it need want a buffered pedal in front/behind it? Is it expecting a high/low impedance input? Let it be entirely up to them. If they feel the need for a buffer anywhere, let them place a standardized buffer where they choose.

This is to eliminate the "oh my pedal sounds best 2nd in a chain" or "my pedal is expecting an active guitar" theories or such. I would say an identical Boss pedal for the use of every buffer would be a fairly accurate method, Boss being one of the most commonly used effect.

3. Some switching method will have to be created where Guitar -> common cable -> manufacturer's chain -> common cable -> amp.

Likely this will involve a long true bypass switcher plus equal lengths of cables to each manufacturer's chain.

4. Because the switching method which will likely involve a true bypass switcher, we can now do a double blind test, where even the person doing the switching will not know which pedal is being switched in. The person setting up the chains obviously should not be involved in the execution, nor interact with anyone else in the experiment for the duration of the experiment.


That's for the general idea. The details I'm sure can be carefully thought about and learnt from the experience of the VS shootout, which I enjoyed greatly  :icon_biggrin:

If they really wanted a true test they would put the guts in generic sealed enclosures with plain knobs and let the test group have them for a week.

I'm surprised anyone took that comparison seriously  :icon_confused:

RG i will take you up on your offer if you are serious and design you an easy straightforward credible experiment testing your brands pedals against keeley, fulltone, klon, and whoever else  :icon_lol:  It would be my honor.

FrankP..."These guys are playing two roles at the same time.  Can it really be sincere ?"...Conflict-of-Interest.

Absolutely not in my opinion....if the intention of the video was an unbiased shootout, there wouldnt be the advert for Visual at the beginning and end. The burden of proof is on the company that posts a "blind pedal shootout" as an advert for their own pedals. Surely Visual knew that some people would be skeptical. Like I already said, why not hire an independant thrid party experimenter like a million other companies do? I liked the videos and don't know Bob's intention but considering that none of these are even original circuits in the first place, I'm gonna lean towards my gut skepticism here.

What was the motivation in the first place? Why do you have to compare your pedals against only the chosen ones...klon, fulltone, keeley....maybe it has something to do with the fact that their customers are in the same market as Visual customers and you want their business? There is nothing wrong with that, just saying.

companies hire independent third parties for credible experiments and even hide their sponsoring identities knowing that scrutiny would surely follow.

RG..."As a personal observation, I find that the words "i hate to be..." can often be interpreted as "it gives me no displeasure to announce that ..."

no actually its my way of saying "if no one else is gonna kick the elephant in the room, then i guess i will"

...  "I also have the feeling that because we all know R.G. (and it is him who is presenting the experiment on DIYstompboxes), we tend to forget those points and because we know that he is a helpful guy and we are accustomed to his sincerity, we see the clips with an other eye than those who go directly on Youtube to see them"

brilliant. thank you.

Quote from: reverbie on February 24, 2009, 10:35:15 PM
FrankP..."These guys are playing two roles at the same time.  Can it really be sincere ?"...
Absolutely not.

Could you prove that assertion, please?  :icon_biggrin:

Quote from: reverbie on February 24, 2009, 10:35:15 PM
RG i will take you up on your offer if you are serious and design you an easy straightforward credible experiment testing your brands pedals against keeley, fulltone, klon, and whoever else  :icon_lol:  It would be my honor.

Consider yourself honored, and get started.  :icon_biggrin:

Quote
FrankP..."These guys are playing two roles at the same time.  Can it really be sincere ?"...Conflict-of-Interest.
Absolutely not in my opinion....

Thank you. I felt pretty certain given the level of intelligence you've shown before that you wouldn't start doing the dance on that one.  :icon_biggrin:

Quote...I liked the videos and don't know Bob's intention but considering that none of these are even original circuits in the first place, I'm gonna lean towards my gut skepticism here.

That's fine.  :icon_biggrin:
I have no issue with that forumlation of what you said at all. It does carry different semantics, as I know you know.  :icon_biggrin:

Quote
RG..."As a personal observation, I find that the words "i hate to be..." can often be interpreted as "it gives me no displeasure to announce that ..."

no actually its my way of saying "if no one else is gonna kick the elephant in the room, then i guess i will"

Kicking elephants is always fun.  :icon_biggrin:
I trust that you did notice that I said "personal observation" and "can be", not "that's what he said". I'm sure that, in the best traditions of English as practiced by our former president (lower case intended) William Jefferson Clinton, it means exactly what you meant it to mean, not more or less.  :icon_biggrin:

This series of videos didn't strike me as purporting to be the ultimate non-biased shoot out for all pedals ever made.  It struck me as being a pretty decent selection of well known/well respected pedals being listened to in a reasonably objective manner.  As such I think I heard some excellent sounds and felt that the Visual Sound pedals could really hold their own or even excel.  In this light, I didn't see it as being out of line to demonstrate the quietness and impressive bypass system of VS pedals.  No one's life is going to be ruined if they view a demonstration like this and are swayed to buy a VS pedal without having heard every other pedal done under double blind rigorously scientific studies.  If there were claims made that this was the ultimate unbiased listening test of effects pedals, I would have been skeptical, but I liked it for what it was: a good chance to hear some well made pedals in direct a/b tests with other pedals without an excess of hype.  So I say thanks and well done.

Scott

So what happened to the rest of the chorus pedal shootout? I would really like to see a comparison of the other pedals too, not just the final two contenders. I don't always agree with the audience and don't find the comparison useful unless I actually get to hear the other pedals involved in the shootout with my own ears. I don't understand why this video was chopped, while the others show the entire process. Is there a chance that we may get to see the full chorus pedal shootout on youtube in the future?

Quote from: reverbie on February 24, 2009, 10:35:15 PM
...  "I also have the feeling that because we all know R.G. (and it is him who is presenting the experiment on DIYstompboxes), we tend to forget those points and because we know that he is a helpful guy and we are accustomed to his sincerity, we see the clips with an other eye than those who go directly on Youtube to see them"

What can I say? Sure it may seem that way, but in my case I actually like these pedals. Honestly, a purely gut reaction, devoid of familiarity. Consciuosly removing my own prejudice from the mix.

I guess it all comes down to epistemology. When designing such a test, one automatically assumes that what is tested actually is testable, comparable and as a result of that, possible to objectivily say "this is better than that". That may not be true. The quality of sound is constructed by human perception, and thus dependent on all the external factors that colour that perception. One could for example say that all attempts to create perfect test conditions are doomed to fail. For example, some people in this thread have argued that one should remove factors like the player knowing what pedal he's playing, making him stand still so that the audience won't judge the pedals out of how much fun the player is having. But why remove all the elements that makes us enjoy sound? Music is a social phenomenon, and removing all the humanity from a test would in my opinion make the test worse. In the end, you end up with at test that tells how good a pedal performs under perferct test conditions, where the science behind the test decides what those conditions should be, not the living and breathing humans that are supposed to enjoy it. Wouldn't that be kind of dull? Who on earth would get something out of that? It's like distilled water.

I'm not trying to recreate the positivism debate here. But I think there are some basic insights to be had from the people who argumented against absolute objectivity and argued in favor of qualitative insights instead.

Therefore - the videos are fun to watch, and the mythbuster videos are the best because they have as a goal to falsify some hypotheses. The others are nice and fun, and good marketing of nice products. As for the scientific part, it would be wise to just stop there and not go any further along that path.

My 2 cents.

QuoteYou can make a Dist+ or Tube Screamer into a 100 different pedals just by messing with the caps that contribute to the EQ and/or the clipping diodes.

I'll go a step further and say that EQ is the heart and soul of any distortion pedal- period.

All this assym/sym/diode-type/clipping-method/opamp stuff is "tomayto/tomotto" stuff AFAIC. When designing, pick your favorite distortion method. Which one? Who cares... But pay very close attention to the EQ. I firmly believe that the main reason the TS is so popular is due to the EQ.

There's a lot of philo-psychobabble in this thread that goes over my head. But I think these videos are really intended to be more "demonstrations" than "tests". If that makes you feel better, I would suggest adopting that perspective of it. In the end, the subject matter is completely subjective. Trying to "objectify" it or arrive at a "more better" "legitimate" scientific test of it is really a waste of time AFAIC.

What they do accomplish is they show a typical group of "tone fest" or "tone party" attendees that things are not always what they seem, or are led to believe. Gear site forumites who feel they have to establish a group consensus before any purchase decision would do good to see these. If nothing else, the videos encourage people to try things and make up their own minds before following any established "convention", i.e., think for themselves. That's a good and healthy thing, and the vids provide a good service. Whether they are or aren't a "legit" scientific test or etc is beside the point IMO.

  Fair ?  There's no 'fair' in stompboxings.
  There's 'Fare':
  get along , succeed <how did you fare on your exam?>
  Every 'stompbox group' I've ever met and had a sound opinion about, had, at the very least:
  a Guitar [wood or other, source pickup, etc,].
  Cables [shielded and 'good' hopefully].
  Amplifier of some sort.
  Speaker [and cabinet]
  Without all that stuff, stompboxes only make very small to small clicking noises, any one of the other necessary 'members' can help make things great or totally mess everything over.
  What these pedals did in the test is what they did in that particular case scenario, many of the players I know don't have very large, relatively quiet rooms in which to try pedals and amps out.
  As far as I'm concerned, they all passed the most important test.
  Any great pedal can be a terrible pedal, just like the next one.
  To a large degree it depends on what it depends on:...the myriad-ic plethora of interactive 'other items' they depend on to stink or beautify, then, how that is precieved.
  Information gathered to make stompbox choices may be best applied when used in conjunction with your own personal logic [I know mine needs routine testing, tuning or reworking] and discretion [tends to come in various forms with the logic].
  Here's a piece of info that could be gathered from analyzing the film and other data:
  Ears, like eyes, perceptions, amps, speakers, cables, guitars, stompboxes and rooms, all share a common possibility: They can be very different from one to the next. Add in 'control knobs' and 'perceptions'..these and other variances can increase exponentially.
  Interesting to see the slight to more extreme Eq changes change the audience reactions.

Quote from: DougH on February 25, 2009, 08:03:27 AM

QuoteYou can make a Dist+ or Tube Screamer into a 100 different pedals just by messing with the caps that contribute to the EQ and/or the clipping diodes.

I'll go a step further and say that EQ is the heart and soul of any distortion pedal- period.

All this assym/sym/diode-type/clipping-method/opamp stuff is "tomayto/tomotto" stuff AFAIC. When designing, pick your favorite distortion method. Which one? Who cares... But pay very close attention to the EQ. I firmly believe that the main reason the TS is so popular is due to the EQ.

I agree. I just made a Electra/Trotsky varient purpose built for an Epiphone Valve JR head. It has a lot of low cut so it is a dirty treble boost. The input cap, diode array and tone rolloff cap were all tweaked for this amp only. It sounds ok through my other amps, typical simple dirtbox. But through the Valve JR it's rather ridiculous how good it sounds.

Mark, did you talk about JUST NOTICABLE DIFFERENCES?  I didn't have time to read the whole thread yet.

Doug, a new marketing method.  The Stompbox party.  Sort of like make up, pots and pans, sex toys, and lingerie.   :icon_cool:

I use and voice my distortions at really low bedroom (I mean Laboratory) volume levels.  Would they be voiced the same at Madison Square Garden levels, probably not.  I use a dark 20 year old Peavey (PV) combo amp with a 12 in speaker.  Would I voice the distortions the same way for a Raging Marshal Stack (RMS), probably not.  Would I prefer one distortion with my '69 Melody Maker with the Seymour Duncan Full Shred pick up, a different one with my Strat, and another with my PRS or Les Paul, yes.

Would one pedal sound better for one situation than another, probably so.  Would you prefer one and I another, probably so.     :icon_cool:

Right, and I think EQ is more than just how you get it to work in different situations. IMO, EQ is the pedal. IOW, these are all EQ pedals with distortion.

It's not possible to please everyone with a series of tests like that. Most people will probably automatically assume that the testers are biasing things towards their own products and are doing some sort of smoke and mirrors kind of thing.

I still want an A/B test where people are hearing different guitar cables of the same length, everything from a $10 cable to a ridiculously overpriced "oxygen-free" $100 cable. I'd like to see if anyone can really tell a difference. I have my doubts.

And the same for opamps of different types. I still contend that the average musician isn't going to be able to tell much of a difference between a JRC4558 and a KA358 (or whatever other kind of opamp you want to put in the blank). I still maintain that what most people think they are hearing has already been shaped by what they have already been told or read about somewhere. For me, the truest test would be to get a room full of non-musicians to hear this stuff and see what their opinion is - not audiophiles or music afficianados, either. Just plain people that listen to whatever is on their car radio. They will not be biased one way or the other when it comes to this stuff.

It would also be fun to see the different response from different "groups" of guitarists that classify themselves in different styles.  Like bluesers, classic rockers, metalickers and "progressive post-modernist avantgarde pseudo-punk not wannabe a rocker...".

Quote from: frank_p on February 25, 2009, 12:58:35 PM

It would also be fun to see the different response from different "groups" of guitarists that classify themselves in different styles.  Like bluesers, classic rockers, metalickers and "progressive post-modernist avantgarde pseudo-punk not wannabe a rocker...".

Yeah, that would also be interesting.

Quote from: DougH on February 25, 2009, 08:22:14 AM
There's a lot of philo-psychobabble in this thread that goes over my head. But I think these videos are really intended to be more "demonstrations" than "tests". If that makes you feel better, I would suggest adopting that perspective of it. In the end, the subject matter is completely subjective. Trying to "objectify" it or arrive at a "more better" "legitimate" scientific test of it is really a waste of time AFAIC.

What they do accomplish is they show a typical group of "tone fest" or "tone party" attendees that things are not always what they seem, or are led to believe. Gear site forumites who feel they have to establish a group consensus before any purchase decision would do good to see these. If nothing else, the videos encourage people to try things and make up their own minds before following any established "convention", i.e., think for themselves. That's a good and healthy thing, and the vids provide a good service. Whether they are or aren't a "legit" scientific test or etc is beside the point IMO.

Ditto  :).  This was not a laboratory test to find the end all supreme king of guitar pedal tone.  It was simply to showcase that Visual Sound pedals are comparable to some of the mojo pedals out there.  I think they accomplished that very well.  Whether or not people's opinions were swayed by those around them seems irrelevant since there seem to be quite a few guitarists out there who buy gear because the people around them have convinced them to.  Visual Sound made no claims that their pedals were superior to Klon or Fulltone or anyone else's but they did show us that they have a couple cool tricks up there sleeve (noise suppression, buffer) while still sounding great.  I think everyone's reading a little too much into the "test" aspect of it.  Just my $.02

I think this pedal demonstration is a good thing. But honestly consider a "test" to be nearly impossible, only a comparison of different flavors and tastes.

Here's the major issue.

In this shootout, all pedals were compared while set to similar tones, or as similar as possible.  Which pedal was the one that was set to its sweet spot to which the others were matched? If I were setting it up then it would be my pedal. Not for any nefarious reason, but because my overdrive sounds exactly the way I sincerely believe an overdrive should sound. It is also the one I am most familiar with and could most easily get a great baseline sound from. It's reasonable to expect anyone else would do it much the same way for the same reasons.  Who set up the unaffected sound of the amp? There are some players and, of course, pedals that are best with a very clean undriven amp tone, others depend on the amp being slightly cranked. I expect the setup for the comparison to reflect those biases.

Please don't anyone read this as an accusation that the test was in any way rigged or unfair. It simply is what it is. A comparison of a lot of different pedals under the same conditions. I'm just pointing out that those conditions cannot have been optimal for all pedals.

A more perfect comparison might be with each pedal set to do what it does best into an amp best dialed in for it. This could be an hour or more of dialing in as in preparing to record a track on an album. Nobody would want to sit through that and it wouldn't be any fun. It'd be work that a studio engineer or producer would charge money for. It would also result in so many differences in result that comparison would be more difficult. We get back to the fact that there are so many good pedals that the differences are mostly a question of taste rather than of better or worse.

For me the take-away is that there are some really great pedals and the Visual Sound stuff is among them. Oh, and as a long time hater of almost every chorus I've ever heard the VS actually sounds good. Excellent work.

QuoteA more perfect comparison might be with each pedal set to do what it does best into an amp best dialed in for it.

This could be done if you had multiple amp setups which are preset before the show starts.

I've preached for years that this, to me, is what is best about great soundclips. They show each pedal in its best light, the way they were intended to be used. Who cares how they compare? Comparisons, esp direct through soundcards and etc put me to sleep after about 2 seconds. Overdrive shootouts bore me to tears. I want to hear intention and potential. What is it capable of? Absolute comparison is impossible so why bother.

The amp used was a Visual Sound amp if I recognised it correctly.

What sort of amp is it ( forgive my ignorance ) , a tube amp or a solid state one ?

So far I haven't found information on the amp or did I miss it ?

My point is certain pedals don't respond good to a solid state amp while others don't mind so much.

And the whole chain is important : a good sounding pedal on one amp can sound not so good or downright bad on another amp , it all has to do with the voicing of the channel used ihmo. 

Who is gonna  shed some light ?

Alf 

Quote from: alfafalfa on February 25, 2009, 02:54:03 PM
The amp used was a Visual Sound amp if I recognised it correctly.

What sort of amp is it ( forgive my ignorance ) , a tube amp or a solid state one ?

So far I haven't found information on the amp or did I miss it ?

My point is certain pedals don't respond good to a solid state amp while others don't mind so much.

And the whole chain is important : a good sounding pedal on one amp can sound not so good or downright bad on another amp , it all has to do with the voicing of the channel used ihmo. 

Who is gonna  shed some light ?

Alf 

It's a tube amp made by visual sound.  I think it is now discontinued.

Quote from: R.G. on February 24, 2009, 10:39:50 PM

Quote from: reverbie on February 24, 2009, 10:35:15 PM
FrankP..."These guys are playing two roles at the same time.  Can it really be sincere ?"...
Absolutely not.

Could you prove that assertion, please?  :icon_biggrin:

I'm reminded of a sweet little sequence in the current season of "24" where the president asks the Jack Bauer character "How do I know that I can trust you?", and Bauer replies "With all due respect, Madame President, ask around."

As for the usefulness of pursuing the "ideal" comparison/shootout method, I think there is some usefulness with respect to effects development, and with respect to effects mythbusting, but I think I agree that pursuing methods to find out which pedal is better is a waste of time. 

I tried to consider which of those two good reasons is the more critical, but I can't make up my mind.  If one strives to improve an effect in some manner, that intended improvement should matter in an audible and expected way, and blind testing can confirm it, and sometimes even define the limits of improvement.  For example, just exactly what sort of modulation is preferred by phaser users at different speeds?  Worth asking if you ask me.  Conversely, perhaps it is important to test whether assumptions about what contributes to "note definition" are accurate, just to be sure one is not franticly pursuing designs or components that really have less impact than one thinks.

Quote from: DougH on February 25, 2009, 08:22:14 AM
There's a lot of philo-psychobabble in this thread that goes over my head. But I think these videos are really intended to be more "demonstrations" than "tests". If that makes you feel better, I would suggest adopting that perspective of it. In the end, the subject matter is completely subjective. Trying to "objectify" it or arrive at a "more better" "legitimate" scientific test of it is really a waste of time AFAIC.

What they do accomplish is they show a typical group of "tone fest" or "tone party" attendees that things are not always what they seem, or are led to believe. Gear site forumites who feel they have to establish a group consensus before any purchase decision would do good to see these. If nothing else, the videos encourage people to try things and make up their own minds before following any established "convention", i.e., think for themselves. That's a good and healthy thing, and the vids provide a good service. Whether they are or aren't a "legit" scientific test or etc is beside the point IMO.

Thank you. That's probably a much better way to look at this.

Quote from: deaconque on February 25, 2009, 01:08:55 PM
Ditto  :).  This was not a laboratory test to find the end all supreme king of guitar pedal tone.  It was simply to showcase that Visual Sound pedals are comparable to some of the mojo pedals out there.  I think they accomplished that very well.  Whether or not people's opinions were swayed by those around them seems irrelevant since there seem to be quite a few guitarists out there who buy gear because the people around them have convinced them to.  Visual Sound made no claims that their pedals were superior to Klon or Fulltone or anyone else's but they did show us that they have a couple cool tricks up there sleeve (noise suppression, buffer) while still sounding great.  I think everyone's reading a little too much into the "test" aspect of it.  Just my $.02

Thank you. That's probably a much better way to look at this.

Quote from: DougH on February 25, 2009, 02:29:17 PM
... Absolute comparison is impossible so why bother.

I think I'm rapidly coming around to this viewpoint.  :icon_lol:

Quote from: alfafalfa on February 25, 2009, 02:54:03 PM
The amp used was a Visual Sound amp if I recognised it correctly.
What sort of amp is it ( forgive my ignorance ) , a tube amp or a solid state one ?
So far I haven't found information on the amp or did I miss it ?
My point is certain pedals don't respond good to a solid state amp while others don't mind so much.
And the whole chain is important : a good sounding pedal on one amp can sound not so good or downright bad on another amp , it all has to do with the voicing of the channel used ihmo. 
Who is gonna  shed some light ?

I'm guessing that's going to be up to me.  :icon_biggrin:
It's a Visual Sound Stallion, a 60W 2x12 combo, with the inside speakers disconnected and sitting on an experimental 4x12 cab. The 4x12 is a prototype of a cab we were going to build, but have delayed. It's 4x Celestion Seventy80s in a cab that's very much like every other 4x12 on the market, excepting only for having the hubcap treble diffusers on the speakers to spread the treble around. The audio path is three 21AX7s driving 2x 6L6 in an entirely conventional arrangement. The only thing novel about it is that it is designed to be effect friendly by having no channel switching, master volume, overdrive channel, etc. etc. The amp just barely hits breakup with the volume at 10 and a single coil strat with volume at 10 driving it. It's a tube amp optimized for hearing what your guitar or effects sound like, but still retaining tube amp characteristics in the clean range. It's the most transparent (ugh! I hate myself for using that word, but it's accurate in this case) tube amp I know of at volume, excepting perhaps some early Fenders from the day when they tried to get amps to do just this. Voicing is entirely conventional. Except for the sound level, any early Fender amp would have been very much like this through the speakers.

I'm sure we'll get some replies saying that some of the pedals tested only sound good through Marshalls or AC30s or BoostAgoGo amps set to 4/5/3/5 on MV  because that would put those pedals at a disadvantage, so that shows the test was nobbled. :icon_rolleyes:
We used our amp because (a) we had one handy and (b) it's designed to show off effects of all kinds well, letting you hear the effect, not the effect as blenderized by the amp.

Quote from: Mark Hammer on February 25, 2009, 03:57:55 PM
As for the usefulness of pursuing the "ideal" comparison/shootout method, I think there is some usefulness with respect to effects development, and with respect to effects mythbusting, but I think I agree that pursuing methods to find out which pedal is better is a waste of time. 

Trust you to have something accurate and to the point to say!  :icon_biggrin: Maybe a perfect comparision process isn't possible, for reasons having nothing to do with experimental method.

I'm going to rely on reverbie to come up with a good, solid, methodology which would be scientifically and statistically valid. And meanwhile, I think I'll go spend a few moments thinking up an UNfair, NONobjective, entirely self-serving method of testing pedals.  Bwahh-ha -ha!
:icon_lol:

Thanks a bunch for doing this!!  I really enjoyed the level of professionalism and the time and money it must have taken to put this together.  It was fun to see all of these pedals next to each other. 

I have to admit that I have a bias in that I really, really don't like focus groups.  I've done them a few times for quick money.  I find them to be pretty comical in a non-intentional sort of way.  They are like a combination of crass market capitalism and that young high school teacher you had who tried really hard to be the "cool" teacher from 'Dead Poets Society.'  But I digress... I also have to object to the "science" of this experiment.  I only watched the overdrive and distortion comparisons but I noticed that the Visual Sounds pedals were last in line (not including the turd that is the Boss metal zone).  While this didn't garauntee anything, it certainly put the pedals in the most probable position to "win" the "competition."  Also, in the overdrive set, all of the pedals sounded 99% the same except that the Visual Sounds one was noticeable louder.  Most people hear louder as "better."  I'm also not sure how the opamp chip myth was busted since it seemed most people were able to tell that there was some difference.  The focus group leader led the group to think that the difference was "small" (admitidly very small) but a combination of small things could be medium things, and anyway I'm not sure what the myth is in the first place? 

   

It`s obvious the amp was tweaked to make the Visual Sound pedals sound best. I bet all the attendees received free Visual Sound T-shirts + beer and biscuits too. If you slow down some of the sequences you can clearly see Bob winking and giving the thumbs up whenever the Visual Sounds pedal is played, in a blatant attempt to influence the audience.*







* Where`s the irony emoticon gone.

The audience look glassy eyed to me.  Either someone put something in the mince pies or they were all hypnotised beforehand.[/David Ike]

 If Danelectro threw together a blind "which is better" shootout video comparing their cool cat pedals to their cloned boutique ones, would anybody object or be skeptical? Would that be appropriate? And what's the difference?

If RG worked for F-tone and they did a "blind" shootout against an Orman booster, would anyone question it? What's the difference here? Presentation? Why even make it a matter of who is "best" or "wins" unless you are trying to sell something???


"50 of Nashvilles Top Musicians Have Decided. Now It's Your Turn".  Bwa ha ha!

I guess everyone is more interested in arguing on and on for pages about the validity of the process than answering my legitimate and simple question about the chorus pedal shootout. It seems the only way to get my question heard is to attack the validity Visual sounds method, but since I don't really care about that, I'll just ask my simple question again:

Why was the full chorus pedal shootout not included on youtube? I am very interested in seeing the comparisons with the other chorus pedals. I don't really care if the other pedals were eliminated by the audience, because I rarely agree with the audience anyway. The only way I find these shootouts useful is if I get to hear the blind tests for myself, so why wasn't the full chorus pedal shootout included like all of the other others? Might there be a chance that it will be posted in the future? Thanks.

Also, thank you Visual Sounds for doing these tests. They are very informative.

Quote from: Projectile on February 25, 2009, 08:01:20 PM
Why was the full chorus pedal shootout not included on youtube? I am very interested in seeing the comparisons with the other chorus pedals.

From what RG indicated, what is posted is a) a bit of a surprise to him, and b) part of what will be on a DVD.

As I have posted on numerous occasions, chorus pedals are one of those cases where the core technology itself (BBDs) virtually demands some degree of uniformity in design. Having just emptied the dishwasher, I'll use a pertinent analogy: How many ways can you design a knife?  It has to have a sharp edge that is long enough to be useful yet not so long as to be either dangerous, heavy, or floppy.  It has to be rigid enough, and it has to have enough of a handle to grip that you can actually hold it securely and apply pressure.  Everything after that is cosmetic.  Same deal with chorus pedals.  You HAVE to have a clock generator.  You HAVE to have some anti-aliasing/anti-clock-noise lowpass filtering.  You HAVE to have a splitter and mixer stage.  You HAVE to have an LFO.  Everything after that is simply cosmetic choices in clock range (and that tends to account for a LOT of the seeming sonic difference between different brands), LFO speed range, lowpass filtering parameters, noise control, etc., and of course, the controls implemented.  Not surprising at all that so many sound so close to each other.

Quote from: Mark Hammer on February 25, 2009, 08:25:35 PM

Quote from: Projectile on February 25, 2009, 08:01:20 PM
Why was the full chorus pedal shootout not included on youtube? I am very interested in seeing the comparisons with the other chorus pedals.

From what RG indicated, what is posted is a) a bit of a surprise to him, and b) part of what will be on a DVD.

As I have posted on numerous occasions, chorus pedals are one of those cases where the core technology itself (BBDs) virtually demands some degree of uniformity in design. Having just emptied the dishwasher, I'll use a pertinent analogy: How many ways can you design a knife?  It has to have a sharp edge that is long enough to be useful yet not so long as to be either dangerous, heavy, or floppy.  It has to be rigid enough, and it has to have enough of a handle to grip that you can actually hold it securely and apply pressure.  Everything after that is cosmetic.  Same deal with chorus pedals.  You HAVE to have a clock generator.  You HAVE to have some anti-aliasing/anti-clock-noise lowpass filtering.  You HAVE to have a splitter and mixer stage.  You HAVE to have an LFO.  Everything after that is simply cosmetic choices in clock range (and that tends to account for a LOT of the seeming sonic difference between different brands), LFO speed range, lowpass filtering parameters, noise control, etc., and of course, the controls implemented.  Not surprising at all that so many sound so close to each other.

Absolutely agree.  I've yet to hear a chorus pedal that made me say "Wow, that's different!".  After a certain position on the speed or rate knob they all become too wobbly and unusable anyway, so the range isn't even a factor IMO.

  I saw the chorus segment on Youtube.
 
 

Quote from: petemoore on February 25, 2009, 10:28:17 PM
  I saw the chorus segment on Youtube.
 
 

Where? The only segment I could find just showed the final two pedals in the shootout, but there is a comment that explains that there were more pedals involved that were eliminated. The video does not show the entire shootout and just comes across as an advertisement for the VS H2O. I don't understand why this shootout was chopped when the others were so excellently displayed.

  There was some talk and the comparison soon focused between the TC and the V2.   
  They sounded great and pretty darn close, Chorus-wise.

I just watched all the videos last night.  I found them really interesting.

One thing I always say is - can the audience tell the difference?  I think the player, the feel, the response to picking (etc) creates subtle differences between quality gear that will mostly be lost to any audience. Roomful of musicians included!  this confirmed that a bit for me.

I'm one of those guys who has come to love good buffers.  But I also notice certain pedals (overdrives etc) respond slightly (read: I'm anal) differently to a high impedance signal than a buffered signal. I put a buffer as early in the signal as my ears will let me.

So I was bummed about the in-line buffers too, but what can you do...without them, then you can say pedal 5 in line is seeing more capacitance than pedal 1, etc.  There's no eliminating all the variables for naysayers.

Nevertheless, I liked how the similarities between the 808 style overdrives were mostly negligible.  I heard some differences, but agree they were so slight to the listener.  the VS route 808 clearly seemed to have more 'bass.' It was the clearest difference among them.  Keep in mind though, these are all basically the same circuit.  But I agree that the hype for subtle differences is often way too overpriced!   :)  Glad I know how to solder now.

While the distortions all sounded different, I thought the BoR sounded the nicest and classiest, clearly. I couldn't believe they guys chose the modded RAT over it. But I guess I've always been a fan of 'less' mid scoop than the average guy. The Hyde was a good second, and the best 'modern' sound (less mids) but that is personal preference.

The video didn't note that the Hyde has 'noise reduction' circuitry.  It's a gate, right RG?  Otherwise you'd think the lack of noise was a revolutionary thing...all high quality gain pedals are going to amplify noise.  The Hyde actively combats it while the other's dont...I'd liked to have heard the noise gate off, or an explaination that that is why it was dead silent, if nothing else.  :icon_mrgreen:

I was surprised by the audience saying the choruses sounded the same.  To me the TC's settings sounded faster and more warbling.  No one else seemed to think so.  ???  But they're both great pedals, clearly.

The op-amp thing--I was disapointed by this, while likewise I smiled. I find that one of the biggest differences when I've switched op-amps is in relation to the way the pedal responds to input 'dynamics' and recovery from clipping.  Strumming a full G on chord with humbuckers, i think yeah, op amp differences and pedal differences between 808 will be basically negligible.

My op-amp experience: Before I started building, I heard op amp swaps would affect tone a bit.  I have a Barber Small Fry, and ordered a $3-4 op amp.  When I got it, I a/bed them.  I heard a difference in the way my low E string 'broke up' and overall clipping.  I actually WANTED to like the one I bought, but though the stock one sounded better.  I was sad.  No listener would hear it, but I could feel it, and I knew I wasted $4. I decided to trust Dave Barber's choice chip over everyone swearing that some other chip would be an improvement - also because I figured it sounded plenty good already.  At least now I have other uses for that $4 chip.  :icon_mrgreen:

So I really don't think all op amps sound the same, as this video may lead most to conclude (I think, incorrectly). But I also don't think they are the end all be all to a pedal...subtle differences only add up in large numbers.  A good life principle: don't major on the minors.       :)

In the end, I will definitely be adding VS pedals to my short list of gear to recommend to guys who want good stombox tone at good prices and don't want the hype and tone chasing I've been going through for years..

QuoteWhile the distortions all sounded different, I thought the BoR sounded the nicest and classiest, clearly. I couldn't believe they guys chose the modded RAT over it.

I didn't care for the RAT much myself. But there was another one, can't remember which one, that just sounded horrible to my ears. It sounded like it wasn't biased correctly or something.

There are a LOT of things that make an audible difference, or CAN make such a difference.  The only problem is that those differences only exist in audible fashion under very specific circumstances.  If you are an adamant user of a particular setup and settings, and a particular factor makes a consistent difference for YOU, groovy.  But there is little guarantee that it will necessarily make a difference for anyone else.

I'm reminded of all the frenzy that exists/existed over pickup phase-switching, coil-tapping, series/parallel coils, yadda, yadda.  All of that CAN make an interesting tonal difference in a pristine clean context, but plug the instrument into distortion X, crank the amp gain until the speakers in your 4x12 cab break up, and all that subtlety simply doesn't exist anymore.  You can switch from neck with the tone off to bridge with the tone full up, you can push your wah forward OR pull your heel back, and you still can't hear anything different.

So, for my part, it's not that there IS no difference when people swap this for that, but rather it's whether a hypothetical difference makes a practical difference for me.  The tree fell in the forest, and 20 years later we can see the darn thing lying on the ground (and maybe even chop it up into some great slab bodies or simply necks), but if I wasn't there to hear it, it never made a sound.  Conversely, make yourself a big pot of chili, lace it with a fistful of scotch bonnets or habaneros, and an equal fistful of garlic, then ask yourself if it made a damn bit of difference whether you used sea salt or regular iodized salt from a box.  As I am so fond of saying: context is everything.

Am re-watching the videos after reading a lot of these comments.  The text notes that have been added on the videos help a lot!

If you have a problem with the fact that the person doing the test is the owner of VS, I understand.  Maybe they can get someone completely neutral like "gearmandude" to do it next time (i like his shootouts).   

I personally have some hearing loss in one ear, which makes midrange tones sound even honkier to me than usual--so I have never been a big TS guy.  My tastes are a little different than a lot of you.  I clearly am no golden ear, but I generally agreed with the decisions made in most of the shootouts.  But, I also know I would tweak the pedals differently, so I was really only being shown a jumping off point.

I think this experiment is fine for what it is -- a simple blind shootout.  It is not a scientific test to determine which pedal is the best.  "Best" doesn't really apply in situations where taste or lack thereof is involved.  You could treat it like wine tasting -- not comparing them head to head, but each one rated on its own, and then tallying the votes, but that wouldn't work either.

Quote from: reverbie on February 25, 2009, 06:15:52 PM
If Danelectro threw together a blind "which is better" shootout video comparing their cool cat pedals to their cloned boutique ones, would anybody object or be skeptical? Would that be appropriate? And what's the difference?

If RG worked for F-tone and they did a "blind" shootout against an Orman booster, would anyone question it? What's the difference here? Presentation? Why even make it a matter of who is "best" or "wins" unless you are trying to sell something???

Not really sure what your point is, but I'd like to see those two demonstrations myself. I think it would interesting. I really have no emotional/ethical/etc feelings about this whatsoever though, i.e. "I don't have a dog in this fight".

I think I would rather hear all these pedals in these contexts than to never hear them at all.  Honestly, I dislike going to music stores to try out gear because I can't hear any subtleties when the dude three amps down in bringing down ceiling tiles with the dual rectifier and I don't build circuits that I don't already have some idea what they sound like.  Honestly, I have never played a RAT, TS, etc. and so I think that at least I get to hear some of what they can do as opposed to not hear anything.  There is always going to be bias, such is the nature of perception, but I like that these videos have been posted and think that, understood and used in the correct manner, they can provide at least some useful information.

jumping in quite late... but I really like the vids and comparisons. Good stuff!

Great job, and a real myth buster  ;D

Most of the distortion pedals I have tried and i would say that was a pretty fair comparison. Bob was also running the risk to fail..............

I'm sorry to say my approach is way more simplistic. At least in my humble experience, there are pedals that sound good, and normally if they are good , they are under most circumstances . I don't care about true bypass . Proof is that the Son of Hyde (TO MY EARS) was the best out of the bunch, closely followed by the BOR , This also showed that no matter how much you mod a DS1, it will still sound like a DS1.

The opamp shootout was another good one. I have spent a full evening with no less than 10 different opamps, and the only one where I could hear a sliiiight difference against the 4558 , was a 4559.

What would have been even better is a cable comparison :).

I'm sorry to say this again. OD's are very simple circuits, and I'm still wondering how can anybody ask 300 USD for a TS clone. Several builders offer top notch TS clones with top quality compnents for little more than 100 USD.

What would have been even better is a cable comparison
  They didn't spend much time on it, but I did, for any testing, choose very connective cabling. That's what they did.
  Between the HQ 25' cable test [not only show of hands but the sound through the monitors here even showed it] demonstrating what cable capacitence does to a guitar signal and how people find it enjoyable/displeaing..
  ..and the bypass demonstrations..
  A reasonable conclusion can be drawn that cable capacitence is a common enough that most of either have some of that or a buffer to drive cable lengths.

This is a very decent presentation, and a good service to the customer base.  I do, however, have to take issue with the op amp mythbuster.  Strumming an open G chord is not the way to tell the differences between them, and yes, there are unquestionable differences. 

I finally watched a few of the videos.  It looked like the player could possibly see the effects over his left shoulder and the people could see the presenters face.

The effect demos were more of solo test to me, because it depends on the room and other players and instruments and volume and...

I think the videos were good.  I liked how you could hear the sound change with the distortions.  To me some sounded "farther" away and other "closer" and some "thinner" and others "fuller"  Each of the sounds can help adjust the total band/CD sound.

So to me there was no better just different (two distortions I did not understand what the sound was designed for but that's a taste issue)

Looked like a lot of work on all the peoples parts

The TC chorus pedal demo made me think Cure with it's settings

I thought the audience was hilarious.

They were falling asleep left and right and whenever they called a vote they all looked at each other, raised their hands halfway, took them down if nobody was doing the same, etc. Typical, I guess. And then the drummer! Ahahahaa...

But that should have a large influence on your voting. I would have suggested that an electronic voting system, to actually get real results. Crowd behavior is too influential.

One thing that you cannot judge by listening is how a pedal reacts to the player. I have always felt that for me, the response can be just as important as the overall tone. If the pedal doesn't react the way you want it to (dynamics, tone changes), then it doesn't stay on my board.

That being said, there's no doubt that the Visual sound pedals do sound good. My friend uses it and he sounds very good. For him, the best sounds are of the tube screamer variety.

Quote from: aron on May 29, 2009, 08:37:51 PM
One thing that you cannot judge by listening is how a pedal reacts to the player. I have always felt that for me, the response can be just as important as the overall tone. If the pedal doesn't react the way you want it to (dynamics, tone changes), then it doesn't stay on my board.

That being said, there's no doubt that the Visual sound pedals do sound good. My friend uses it and he sounds very good. For him, the best sounds are of the tube screamer variety.

I agree

Quote from: aron on May 29, 2009, 08:37:51 PM
One thing that you cannot judge by listening is how a pedal reacts to the player. I have always felt that for me, the response can be just as important as the overall tone. If the pedal doesn't react the way you want it to (dynamics, tone changes), then it doesn't stay on my board.

That being said, there's no doubt that the Visual sound pedals do sound good. My friend uses it and he sounds very good. For him, the best sounds are of the tube screamer variety.

I also agree.

Quote from: aron on May 29, 2009, 08:37:51 PM
One thing that you cannot judge by listening is how a pedal reacts to the player. I have always felt that for me, the response can be just as important as the overall tone. If the pedal doesn't react the way you want it to (dynamics, tone changes), then it doesn't stay on my board.

That being said, there's no doubt that the Visual sound pedals do sound good. My friend uses it and he sounds very good. For him, the best sounds are of the tube screamer variety.

I agree.

I was thinking of that two days ago.  Looking a car passing is not the same as driving one.  And doing a painting is not the same as watching one.  Altrough "art" is made to be seen and heard, there is no doubdt that the experience of the artist is completely different then the one of the public.  And when the artist become the public he may not repond to the same criterias.