A MUCH NEEDED TALK - For Both Life Sciences And Life
Found this to be interesting professionally, as well as applicable to so many other things that we discuss here on NTers.
Reliability , from our leaders, from the media, from ourselves; why is there such a dearth of it in todays world?
A MUCH NEEDED TALK
I was invited to give a talk this fall on the wave of irreproducibility washing over the life science research landscape. I have to admit, though, it was with a range of emotions that I considered accepting this offer.
On one hand, I'm always interested in interacting with BioTechniques readers and authors, or any scientist working at the lab bench for that matter. Finding out more information on the techniques and methods driving science today, and the needs for the future, is something I have a true passion for and the reason that I do what I do on a daily basis. On the other hand, it is truly sad to me that there would even need to be a session at this conference devoted to a topic like irreproducibility.
As I contemplated whether I would (or even could) give such a talk, I started thinking about some possible topics. To me, true fraud is somewhat outside the scope of this discussion. While it can be argued that fraud is on the rise, leading to the increasing frequency in irreproducible results published today, there is no way to tell for sure if deliberate misconduct is on the upsurge. Today, we have sophisticated software to evaluate figures and images for manipulation, not to mention the internet, which enables everyone with a computer and some free time to examine articles for plagiarism or some other form of malfeasance. So, is there more fraud today, or are we simply better equipped to detect it? I don't know.
Most irreproducibility, I would guess, is due to other factors. Experiments are performed well, results are interpreted as thoroughly as possible, but in the end no one can replicate the findings. This is a situation that should not only interest all life scientists but scare them as well. Imagine that you do everything properly, check your controls, and report your methods as clearly as possible. But that antibody you used, was not as reliable as you thought. The validation was poor, and it is not specific for your protein of interest. Or that cell line did not behave as you expected. These are real problems, real issues for today's scientists. But such problems cannot be dealt with in a vacuum; solutions require multiple sources of input (researchers, suppliers, journals, and funding agencies).
Surprisingly, the more I think about this invitation, the less sad and confused I am. My talk should be only one facet of a larger discussion on the topic of irreproducibility. And as more researchers and administrators publicly address the issue, the life science community will be the beneficiaryincreasing experimental rigor and creating standards might act to eliminate the fountain of irreproducibility. In the end, small talks can be strong starting points.
Tags
Who is online
62 visitors
I have run across this in geology, as well, which is a physical science. Some results can't be reproduced to be the exact same in separate experiments. Primarily because there are boocoos of other factors involved over which the scientist has no control.
Using the best equipment available, can a pumping test be run the same way twice? One would think it would be possible-- the aquifer is going to yield what the aquifer is capable of yielding, and no more. Right?
Yet, outside factors also play a part. Is the air pressure rising or falling? When was the last rainfall? Is the river rising? Did it rain, up-watershed from your well point? Are there any other withdrawal points, or recharge points within a 5 mile radius of your well that are operating? Is the generator powering the pump the same used before? Is it operating at maximum efficiency? What kind of gas did you use the first time? Has the gas formula changed? Is the pump the exact same pump? Are you using the same orifice to check the pumping rate? Has it remained steady? At what exact time was the measurement of the water depth in the well taken? Was the same probe used to measure during the two tests?
And then, the human factor-- What time of day is it? (Night measurements are usually slightly more inaccurate, because it is dark out there in the middle of nowhere.) Is the measurement point, (like the edge of the pipe) clearly marked? Does your helper know which side of the pipe to hold the probe line on? What is the temperature? (Cold weather measurements are more likely to be taken quickly, so that you can go back sooner and sit in the warm truck.) Is farmer Brown using his septic system field more because he is doing laundry? Is farmer Johnson irrigating his tomatoes 3 miles away? Did farmer Jones install a groundwater heat pump system?
All this sounds silly, but can really affect your results. Let's say that at 1650 minutes into the test, the water level is 27.85 feet. Comparing results, you realize that you don't have a measurement at exactly 1650 minutes into your test, but at 1675 minutes. Is your measurement at 1675 minutes in line between the measurements of the other test?
I welcome this, but I also realize that not all experiments CAN be duplicated exactly. Life, is what it is, determined and capricious. All scientists aim to limit the human factor, but not all experiments are able to be performed the exact same way twice.
Analysis in the lab should be reproducible, whether positive or negative (both being just as valuable to the goal of the experiment, which is finding out which hypothesis work, and which don't). In strict tech terms, if a reliable piece of information (like a QC reagent) becomes unreliable, then all other testing on that instrument, or by that methodology , is suspect until the conflict has been resolved.
In this case I believe that the author is discussing reported findings that have been published, that cannot be backed-up by a reliable source, namely, reproducing the observed phenomena. In essence I believe he asking, why is there so much BS being published as truth?
I guess in my field, we are seldom in the laboratory. Even the petrologists have ranges of values for kinds of rocks-- i.e. greywacke has at least 15% of the rock composed of matrix material, and we learned that it had a certain percentage of quartz grains, but that isn't in the Wikipedia entry. Thank God that has apparently been updated... It is really difficult to look at a rock and estimate how much material is between the grains of the rock...
In the lab, you can limit outside factors, but in the field, you can't. You always have to take into consideration what else is going on.
I agree that there seems to be a lot of bunkum being produced as scientific evidence, (Raspberry Ketone: Fat burner in a bottle). In my field, its been this way for years and years. The oil and gas guys call it, "The lure of the closed contour." Meaning that when you draw a map, close the contours so that it looks like a dome-- an oil and gas production site. That data doesn't fit? Throw it away!
Cherry picking is bad practice in any discipline . Some disciplines utlize it anyway . One of the most infamous examples is in climatology [which is still too unsophisticated to properly be called a science ] .
BTW , some of the worst examples of unreproducible experiments are in the social sciences . In fact there was an article showing that MOST of that discipline's results are unreproducible .
I think so, too-- yet I also know that some of the well logs used in the studies were drilled by cable tool rigs, which are marked with chalk every 5'. The deeper the hole, the longer the cable stretches, so no one really knows where one is, unless the well is gamma logged... You have to weigh all the factors.
At least in the water industry, the logs may not be perfectly accurate, they're usually close. I'll never forget drilling into gray lime, Shyrock lime, gray lime, black shale, someone else's lime. (change of formation, they didn't know what the name was...)
I don't know enough about hydrology or geology to respond to your points . But I will agree that "close enough" is usually a good practice to get useful approximations ...
The Journal of Irreproducible Results
I used to see articles from this occasionally, haven't seen anything in a while.
I posted a link to a podcast from SETI in the Climate Change group. I'm not sure the podcast is still up. In that show, one of the scientists stated that perhaps 50% of published scientific papers were wrong. Not because of fraud, but because of studies done incorrectly.
I remember my first year of organic chemistry was tough. Not because we couldn't follow the experiments or didn't know what to do, mostly it was because of poorly cleaned and maintained equipment by the student assistants. Hard to get the expected results when the beakers and test tubes are contaminated from the start.
Geology sort of follows the concept of the Corps of Engineers: Measure it with a micrometer, mark it with a grease pencil, cut it with an axe.
A geologic formation is not like a road bed. Formations change both laterally and vertically, and not in even layers, or by anything one can see. One uses averages over distances to describe the long term behavior of wells. As Steve says below, it's hard to get accurate results when the test tubes and beakers aren't cleaned properly. It's also hard to get accurate results outside in the field.
Next time you find a stream bed, notice the floor of the stream. There are piles of sand and rocks here, rounded pebbles there, sharp rocks and fine sand there... And just make it much larger, and buried. It's like trying to drill into a 50 story building, with 4 holes, and find the janitor's closet.
There are way too many factors to get the same result each time.
Thanks for your response Dowser, nice getting a point of view I've not worked closely with. I can't imagine the problems we would have if we tried to run our lab outside...OMG way too many variables! Then again the field-labs that the military have are awesome.
Close enough is a good explanation though. I mean, QC in the medical field is quite harsh, especially the technical side of it. Reproducing QC results are stringent; yet, are still statistical analysis in the end. So variable limitations are observed, and managed to be useful as well as workable.
So when you read about "discoveries" or "proof" that cannot be proven byreproducibility, I wonder about these publications, and cannot fathom why it has become vogue to publish undetermined hypothesis as fact.
Thanks for the link Steve, good stuff!
:~)
Here is some geology from the panhandle of Texas.
And here's some weather :
beautiful! I love the stripes!
Not so fond of tornados...
I think it is criminal to publish data that hasn't been vetted. But, that's me. No wonder we are so confused about the food, etc. that is "good" for us.
Thanks for a great article! I wish you and I could go out into the field sometime, and you could see the process... It used to be a lot of fun! And of course was endlessly challenging.
That would be very cool; I've never observed hydrology up-close before!
When You get to close, it can be hard and wet at the same time.... :D
LOL, Brolly!
I have always enjoyed it!