peer review

“Phone-A-Friend?” or The Value of Private Peer Review

The popular TV game show “Who Wants to be Millionaire?” allowed contestants to seek “Lifelines” such as “Phone-A-Friend” or “Ask-the-Audience” in order to help them correctly answer trivia questions.  This rule most often provided the contestant valuable input and increased the player’s odds of success. (Ask-the-Audience seemed to offer the correct answer surprisingly often – an example of the value of crowd-sourcing). Like in “Millionaire,” technical researchers and other academics are often required to cross-check their research to increase the odds that it is reliable through the venerated institution of peer review. Improving upon the game show example, peer review generally involves a researcher in a specific field submitting his or her findings to a panel of respected experts in the same field – an important validation step.

It’s well known that peer review provides critical checks on scientific or technical claims.  As David Faigman puts it, “In fact, [the process is] one that, while competing to uncover new truths, cooperates in the general goal of advancing knowledge.  If done well, which usually involves some level of anonymous – i.e., “blind” – evaluation, [many technical findings or claims can] be provided with objective and detached evaluations of the methods and principles that underlie the … opinion...". Of course, peer review is not without its critiques. It does not have a perfect record of uncovering flaws. The solution, however, does not lie in abandoning it altogether. Indeed, many new innovations suggest that more peer review is necessary to fully vet research findings.  

In academia, peer review is used in the course of disclosing research in journals. The question then arises - if peer review is a necessary check on researcher’s findings pre-publication, can the process have valuable use cases beyond this important yet narrow application?

The answer is yes.

Many private businesses making risk-based decisions rely on expert analysis, due diligence, and/or conclusions as a fundamental input.  Decisions relying on the strongest and most accurate of these inputs will, in general, carry less risk and yield more successful outcomes.  Conversely, when expert’s inputs are based upon faulty methods and principles, the resulting decisions will surely be suboptimal; as they say “garbage in, garbage out.” Peer Review can have tremendous value for private business by improving upon the accuracy of key input analysis. The process can also have value as a sharpened predictive tool whereby decision makers who might rely upon inaccurate or un-vetted analysis can avoid bad choices. In these applications, improved decision-making is the goal, not simply publication.

There are many examples of using “private peer review” to enhance business decision-making.

Venture Capital / Institutional Investors

Venture capitalists and large institutional investment vehicles invest in private early stage companies that often contain technological risks. The accuracy of studies and research to prove efficacy, causation, etc. on these components (often done internally and not reviewed even by regulators) can make or break the success of a potential investment. Although these firms usually perform due diligence in-house or with a team of trusted advisors, there is some evidence that the system doesn’t work all that well.

The dramatic rise and fall of private novel blood testing company Theranos serves as an example here. While proving a counterfactual is difficult, Theranos’ unique process was never peer reviewed or published. Presumably this occurred because the technology was a trade secret. However, if the peer review could have been done confidentially without disclosure, this likely could have helped investors ask the right questions to better understand the technological risk. The key here is that top researchers are often needed to perform thorough due diligence on complex claims or technologies. Private peer review can provide this essential feedback.

The Legal System

Our adversarial system pits two parties with conflicting claims against each other often in front of a gatekeeper (judge) and a fact-finder (jury) to adjudicate the matter.  Lawsuits involving virtually every technical issue (accounting to zoology) bring experts to write reports supporting each side’s argument.  However, each side inevitably finds the expert that supports his or her opinion. The challenge is for non-expert judges to ferret out which expert is mainstream and/or reliable. The task is usually difficult if not impossible. Where judges let un-vetted and unreliable expert opinions to be determinative, litigation results can produce bad precedent. Bad science creates bad law.

On the other hand, litigators can use confidential peer review to strengthen their own expert’s draft report and to challenge the opposing side’s final report if appropriate. Shouldn’t best methods and practices be checked by top experts to ensure proper conclusions?

Many other actors in the legal sphere can utilize such private peer review. Mediators and arbitrators can similarly benefit from blind peer review.  Judges can independently make use of peer review to assist in ruling on whether an expert’s underlying principals and methodologies are reliable.  Litigation funding firms can employ peer review to conduct due diligence on cases for potential investment in a manner similar to institutional investors above (with litigation claims the investable asset). The ultimate point is that when two parties are stuck in a “battle of the experts,” private peer review provides a valuable source of information to break the tie.

Many More….

There are many other industries that produce important expert reports/analyses/conclusions that can be made stronger and more accurate with the benefit of private peer review – corporate technical reports (particularly in self-regulated industries), think-tank technical papers, insurance claims evaluation, corporate marketing materials making use of technical claims, etc.

In sum, peer review can have great value whether on a TV game show or, more importantly, where risk-based decisions contain key technical opinions whose accuracy can have dramatic effects on business or legal outcomes. In research, peer reviewed findings are made public. Private peer review allows for a confidential analysis using the same process used by academia. The key is that there is no public disclosure – the findings strictly are used to increase the firm's competitive advantage and decision-making process.

Comments (0)

    Leave a comment