"'How did this happen? How did over 30 papers get published? Not because the
scientists who wrote the papers were stupid. Not because they were lying.
But because they were deceiving themselves.'"
Another likely explanation would be that, when a scientist first discovers a
new phenomenon, he defines the characteristics of that phenomenon and the
circumstances under which it appears. That way, when other scientists try
to repeat his experiments, if they get results with the same characteristics
under the same circumstances, then they will believe they have successfully
reproduced the original experiments. Most of the time this method works
very well; unfortunately, some sources of background noise can also produce
specific, repeatable patterns that may be defined as real phenomena. Also
unfortunately once people start thinking in terms of specific definitions,
it can often be difficult for them to think outside these definitions. Even
critics of the new phenomenon usually work within the definitions rather
than challenge them. After all, definitions are so important in science
that you cannot do science if you don't understand the definitions. It is a
rare intelligence that can argue within the definitions, but think beyond
them as well.
Then there is the problem that most scientists have not the time, money or
manpower to repeat a published project, so they simply accept it as true (if
they are convinced by the paper that it is true), then use it as a basis for
developing new projects. Because of that, they are generally going to try
to interpret their results within the parameters set by the original
research. If their results appear to contradict the original research, they
usually question their own technique, rather than the original research.
Only if they get a result that simply cannot be reconciled with the original
data would they begin to question it. This can actually be done more
quickly and easily than thinking beyond the definitions, but for awhile at
least it can be very tempting to believe the original research than your own
results.
These two things could also lead to the result lamented in the the paragraph
quoted by Pim without being caused by self-deception.
Kevin L. O'Brien