It is easy to do bad science. There's a lot of it around. Many of the so-called science stories we read in the press are based on bad science (or no science at all). They catch the eye because they often claim startling breakthroughs and discoveries, usually about cancer, or ageing, or pain relief, or weight control.
Genuine scientific research is painstaking and slow. Many tiny steps are taken in order to gradually increase the sum of knowledge. Research is often repetitive and tedious, yet it is always intellectually stimulating and can occasionally be wildly exciting,
To perform good science, the researcher must, as far as possible, try to minimise human bias. This is especially difficult when doing studies involving human beings. Volunteers who take part in research studies tend to be people who are concerned about their health, so there is a bias before the experiment has even begun. Ideally there should be balanced representatives of gender, ethnic group and age range, but this is not always possible.
When conducting a drug trial very elaborate procedures need to be followed. Let us assume drug "X" is being promoted as an anti-depressant. How can this be tested?
The first thing is to obtain as many volunteers as possible to participate in the trial. The more people tested, the more reliable the results are likely to be, since this reduces some of the variation due to chance.
The next thing is to divide these volunteers randomly into two groups. This will involve using random number tables or computer generated random numbers. It is not good enough for the researcher to do it manually.
Next, the drug is given to one group and a "placebo" given to the other. The placebo is physically identical to the medication being tested, except it does not contain any active ingredients. The drugs and placebos must be administered in a "double blind" environment. That is, neither the patient nor the researcher knows whether they are handling the drug or the placebo.
Similarly, when the effects of the treatment are measured, no-one knows who received what. The truth of which pill the participants actually received is only revealed after the results have been analysed using appropriate statistical techniques.
Finally the research must be written as a scientific research paper which describes in detail the methods used, the results obtained, the analysis done and conclusions made. Sufficient detail should be given so that other research teams can duplicate the experiment to validate or refute the findings.
The research paper is then sent to a reputable scientific journal for publication. The editor of the journal sends copies of the paper to some other independent scientists who work in the same scientific field. Their job is to study the paper carefully, to see if the science is good, to criticise and recommend whether the work is worthy of publication. This process is called "peer review" since the research work is being evaluated by scientists who are "peers" - that is, contemporaries and in the same field as the researchers.
If the reports are favourable, the research paper is published. If not, the editor may send it back to the authors with (anonymous) reviewers' comments for revision, or the paper may even be rejected outright.
All scientific research should be subjected to peer review. It encourages researchers to meet high standards and prevents the spread of rumour, crazy claims, and personal views. Research which has not been through the process has had no such "stamp of quality." It might be nothing more than pure charlatanism. (Tip: Next time you read anything which claims wonderful health benefits for a particular herb, food or drink, check to see if the author mentions who did the research and where it was published).
This is not to say that the peer review process is faultless. There have been cases where papers have been rejected simply because a reviewer disagreed with the findings. (Not wanting to believe that any research could have contradicted their own pet theory, the reviewer rejected the paper.) However a good editor should be able to recognise these incidents.
Some journals are now published on the web and research papers are evaluated through various types of "Open Peer Review" where opinions from a wider range of experts can be obtained.
Such a review system tries to ensure the highest quality of scientific research. However, big business has money, and money has the power to corrupt. Ben Goldacre in his book "Bad Science" highlights how drug companies, for example, have managed to bias research in their favour. In 2008 a group of academics examined all the trials of a certain group of anti-depressant drugs called SSRIs (Selective serotonin reuptake inhibitors), such as Prozac, which had been listed and registered with the US Federal Drugs Agency.
They found that 23 of the 74 trials (31%) had never been published in the medical literature. Almost all of the trials with positive results found their way into the medical journals. Nearly all of the trials with negative or questionable results were either not published at all or were published in a way that tried to portray the results with a positive spin. (It's not clear exactly how the negative trials were buried. Drug companies may have suppressed them, authors may not have submitted them, or journals may have decided not to publish them.).
This selective bias no doubt enhanced the medical profile of SSRIs and their use by doctors. The danger is that this selective publication may have led doctors to make inappropriate decisions when prescribing medicaton for their patients.
Through media sensationalism bad science can be made to look good and good science can be made to look bad. The only people who can ensure that the public gets an accurate picture are the scientists themselves. They are gradually making their voices heard through internet resources such as the New York Academy of Sciences' "Science in the City" podcasts, on-line scientific journals and science blogs which publicise press releases from research institutions.
Research scientists need to do much more to keep the public directly and properly informed. They must uphold high standards of research, and they must be cautious in their dealing with the press, who are expert twisters of words and meaning.
Knowledge is a public good and should be widely dispersed. It should not be hidden away or compromised for the sake of publicity, self aggrandisement or money.
by Christine Betterton Jones - BSc. (Zoology), PhD (Parasitology)
Bibliography
1. Bad Science Ben Goldacre, 2008. ISBN 978-0-00728487-0 Harper Collins 370pp
2. The Depressing News about Anti-depressants Sharon Begley, Newsweek Jan 29 2010 on http://www.newsweek.com/id/232781/page/1
3. Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy
Erick H. Turner, M.D., Annette M. Matthews, M.D., Eftihia Linardatos, B.S., Robert A. Tell, L.C.S.W., and Robert Rosenthal, Ph.D. The New England Journal of Medicine Volume 358:252-260 January 17, 2008 on http://content.nejm.org/cgi/content/full/358/3/252
4. Science for the Public - the New York Academy of Sciences http://www.nyas.org/WhatWeDo/SciencetheCity.aspx
5. Science Blog http://www.scienceblog.com
