Conceptual Errors in Scientific Inquiry

In the field of social sciences, for something to be considered scientifically valid, it must undergo rigorous testing using what is known as the scientific method. Simply put, this means that the theory, premise, technique, etc. being investigated has been evaluated by
well-established procedures, found to be replicable under various situations, and then the research is published in accepted scientific journals or presented at scientific forums. In its most simple form, a hypothesis is formed based on a theory or concept, then tested
with an experiment to see if its predictions can be objectively confirmed. The data provided by the experiment is analyzed and the results published so that they can be reviewed by other scientists and researchers.

The process is streamlined by what is known as literature reviews. Many research projects start with a historical review of prior research that supports or detracts from a specific line of inquiry. This can be used to bypass certain elements that have already been tested so that research can be conducted on new elements that can be deduced from the previously proved (or disproved) material. Typically, the “value” of the research being reported is largely dependent on the type of journal being cited, the best being peer reviewed journals. This is because it is assumed that the material has been carefully scrutinized by other experts in the field before it is published, eliminating, for example, possible bias or flawed experimental procedures.

In addition, some researchers publish what are known as summary reviews, compilations of related published research with summaries based on the outcomes of the individual research documents. Again, these summaries are often cited to expedite future research,
to advance or repudiate certain lines of inquiry, and to support arguments for or against certain theories, premises, techniques, etc.

This process is not immune from error, however. For example, if an investigator chooses to study a hypothesis based on a flawed understanding of the premise or technique being studied, it is possible to produce flawed, inaccurate, or false data. This can then be
compounded if that data is published in an accepted scientific journal, because it may thereafter be used to support or deny future research, hypotheses or conclusions.

This is, in fact, what has been happening for a number of years. A recent article in The Economist reports, “Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis,” and “Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 ‘landmark’ studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk.” (From
http://www.economist.com/news/leaders/21588069-scientific-research-has-changedworld-now-it-needs-change-itself-how-science-goes-wrong accessed 2-2-2014.)

In his probing article, 35 Years Revisited: Conceptual Errors in Scientific Inquiry, a Case Study (http://nlpwiki.org/35-years-revisited-3), Richard M. Gray explores the problem in great detail. Gray’s case study is the field of Neuro-Linguistic Programming (NLP), a relatively new approach to communication and personal development. The focus of his article centers on two recurring problems he has identified with critiques of NLP. As Gray writes, “First, bad research is often promulgated as the received wisdom. Second is the tendency to uncritically rely upon the definitions, interpretations and conclusions made in published research, despite misunderstandings in the material and even
misrepresentation of reported findings.” He goes on to state that this has resulted in “…academic critiques claiming that NLP lacks a theory and that it has been disproven by extensive body of research mostly dating from the 1980s.”

Gray takes a very close look at the assumptions and definitions used by researchers who did not adequately understand the field to begin with. For example, the assumption that “…preferred representation system (PRS) was some kind of theoretical foundation upon which the rest of NLP depended,” resulted in numerous and repeated studies that focused on a teaching tool rather than on some basic tenet of NLP. PRS refers to the use of one of the senses (such as sight) over the others (hearing, feeling, tasting, smelling) to encode, remember and/or understand an experience. Because the original researchers misunderstood the concept and future researchers used their flawed definition of PRS to construct additional hypotheses and studies, research repeatedly attempted to prove that each individual used one and only one sense repeatedly over the other senses.

Another example of how inadequate training and understanding of NLP concepts resulted in false research results is demonstrated in numerous studies of eye accessing cues (EACs). NLP describes how individuals may in certain contexts demonstrate a pattern of
eye movements that are linked to visual, feeling, auditory and other experiences. These cues are often linked with predicates – certain words – that presuppose one of the senses, such as see, hear, or feel. However, as Gray emphasizes, without adequate training in
both how to elicit and observe these EACs, researchers repeatedly produced results that failed to verify their existence. These negative findings were again promulgated in scientific journals.

One final area discussed by Gray is some of the NLP research on rapport. Gray states, “As is often done in poorly designed research, the term is defined using a standard definition, a dictionary definition or a definition that is current in the researcher’s subdiscipline.” He goes on to note that, as used by NLP practitioners, the standard definition of rapport is inadequate to encompass the meaning espoused by NLP which encompasses much more behavior and physiology than the standard definition. Gray points out that, “Unless the phenomenon under question is tested as defined by its proponents, the test is invalid.” True to this statement, findings looking at the emergence of rapport were generally not supportive of the tenets being studied.

Gray fully explores the fallibility of the early research on NLP and highlights several serious problems including “…the assumed ubiquity of the PRS; its presumed centrality to NLP; the presumption that the indicators, eye accessing cues and predicates, always agree as to the conscious perceptual system; and that rapport implies something other than a set of observable changes in interpersonal responses.” One of the long-term negative impacts of these flawed studies has been a significant reduction in the number of research studies attempted, and more significantly, funded. He concludes with several recommendations to improve the future of all research, such as requiring “…that
responsible journals … institute editorial policies requiring the recruitment of experts from the relevant fields, with special attention given to obtaining qualified reviewers for those fields that are new or outside of the main stream.”

The results of the problems Gray reported have seriously eroded NLP’s acceptance in mainstream psychology and other fields such as education and coaching where it has been very successfully applied. One major source of the promulgation of the inaccurate research is in the entry on Neuro-linguistic programming found in Wikipedia. Following the same erroneous path as others, Wikipedia states, “Reviews of empirical research find that NLP's core tenets are poorly supported. The balance of scientific evidence reveals NLP to be a largely discredited pseudoscience. Scientific reviews show it contains numerous factual errors, and fails to produce the results asserted by proponents.” (From http://en.wikipedia.org/wiki/Neuro-linguistic_programming accessed 2-2-2014.)

The sources cited by the Wikipedia article are exactly those that Gray reviewed and demonstrated how they are flawed, inaccurate and biased. The Wikipedia article is a perfect example of flawed research being used as the basis for the evaluation of a whole field. This is crucial, because the reach of its readership and the severity of its negative portrayal of NLP make it difficult to attract the types of researchers who can conduct valid research as well as the funds to pay for such projects.

To address this issue, Frank Bourke, PhD., a clinical and research psychologist, founded the nonprofit NLP Research and Recognition Project
(http://www.researchandrecognition.org/) to advance the science of Neuro-Linguistic Programming and bring the research up to a scientific standard necessary for more widespread implementation. An offshoot of that project, NLP Wiki (http://nlpwiki.org/wiki) has been created to offset the misleading Wikipedia article. Contributions from a wide array of internationally recognized professionals in the field of
NLP have been compiled and are presented on this site as well as a comprehensive presentation of research in the field. It is hoped that as more people access these sites and become informed as to the true validity of NLP approaches, the greater scientific community will also see the value in pursuing research in this field.

 

[BACK TO MAIN ARTICLE PAGE]