Selective Reporting of Methodology
GUIDELINE 21: Authors have an ethical obligation to report all aspects of the study that may impact the independent replicability of their research.
Replication of others’ research is one of the hallmarks of the scientific enterprise. As such, scientists and scholars have a responsibility to inform others about the specific procedures used in their research. This information is typically found in the methods section of a research paper. The primary purpose of the methods section is to provide other researchers with sufficient details about the study so that in the event that anyone wishes to replicate the study, they will have enough information to do so. For example, we identify the subjects of our study (e.g., select clinical population, specific species of animals) and provide important details about characteristics of the sample, such as how subjects were recruited, that are relevant to the kinds of variables that are being manipulated and measured.
The Methods section also contains description of instrumentation or other observational techniques that are used in carrying out a study. Whether data were collected using sophisticated machinery, such as a positron emission tomography or via a simple questionnaire, scientists must describe these materials with enough detail to allow other researchers to conduct the study.
Perhaps the most important part of a Methods section is the description of the actual procedure that was used to carry out the study. Here, investigators must explain in clear language the series of steps that were used to establish, observe, or manipulate the independent variables. They must offer a complete description of the testing conditions and all of the other necessary details that would allow an independent investigator to carry out the same study again. Any essential details that are inadvertently omitted from this section may lead others to carry out replication attempts that will be doomed to failure, resulting in a waste of valuable time and resources. A more serious offense occurs when an author intentionally leaves out an important detail about the procedure or a crucial event that altered the conditions of the study. There are several reasons why some authors will knowingly leave important details out of a research report. Perhaps an extraneous variable was introduced into the study while it was in progress leading to biased results. For the sake of expediency, rather than discarding the biased results and starting all over again, an investigator may inappropriately leave that major detail out of the report. The important point here is that authors have an obligation to describe all of the important aspects of the research conducted, even if some of those details reflect poorly on his or her abilities.
Because of the concern that some investigators may at times omit important details of the methodology used, guidelines have been formulated to help authors write better research reports. For example, for reports describing randomized control trials authors are advised to consult Moher, Schultz, and Altman’s (2001) Consort statement, which is a set of guidelines designed to improve the quality of such reports.