On January 21, 2016, Professor David Faigman submitted an amicus brief regarding City of Pomona v. SQM North America and the improper admission of SQM North America's groundwater expert. See our recent post about the brief.

The Atacama desert is extremely arid. These conditions favor perchlorate accumulation. The isotopic makeup of the perchlorate is likely unique to the area.

Overview

Our case study of City of Pomona v. SQM North America illustrates peer review as a novel must-have tool for litigation involving expert testimony. Plaintiff City of Pomona's expert Neil Sturchio was excluded from testifying about the source of groundwater contamination. The district court stated that the methods used were not generally accepted, were not retestable, and reached too far in the conclusions reached from the data collected - all violations under Daubert. Our neutral peer reviewers found, post-hoc, that the methods utilized by Sturchio were accepted, tested, and capable of drawing strong conclusions. The Ninth Circuit ultimately reversed the exclusion. JuriLytics peer review would have saved Pomona tens of thousands of dollars in appeals fees had the district court been able to hear from neutral scientists in the field.

Case Facts

The City of Pomona noticed that levels of perchlorate in it's water supply were above the state-mandated limits in 2007. In bearing the costs to remediate the contamination, the city sued SQM North America, whom the city alleged was the main importer of the perchlorate via Champion nitrate fertilizer sourced from the Atacama desert in Chile. The city hired an expert, Dr. Neil Sturchio, to investigate the source of the contamination. Using a methodology known as “stable isotope analysis,” he determined that the dominant source of perchlorate was the Atacama desert.

District Court Excludes Sturchio

Before trial, the district court held an evidentiary hearing under Daubert, and excluded Neil Sturchio. The court found several issues with the Expert Report and testimony:

  • General Acceptance - "...the procedures used by Dr. Sturchio for isotopic analysis of perchlorate have not been generally accepted by the scientific community. In fact, the Department of Defense’s 'Guidance Manual for Forensic Analysis of Perchlorate in Groundwater using Chlorine and Oxygen Isotopic Analysis' states that 'the techniques and Quality Assurance/Quality Control (QA/QC) parameters are still being refined, and there are no USEPA-certified methods for CSIA of organic or inorganic compounds.'"
  • Testability - "The Procedures used in this case are further flawed in that they have not been tested by other laboratories and are not subject to retesting given the failure to take dual samples."
  • Reference Database - "Dr. Sturchio's database is too limited in order for him to reliably comment on the exclusiveness of the location of the potential source of perchlorate in Pomona's water with an acceptable rate of error."

Ninth Circuit Reverses

Under appeal, the Ninth Circuit reversed. "[B]ecause the district court abused its discretion by not allowing a jury to resolve contested but otherwise admissible expert testimony, we reverse the district court's order of exclusion...and remand for trial." In the opinion, the three main issues the district court found were discussed at length:

  • General Acceptance
    "scientific methods that are subject to 'further testing and refinement' may be generally accepted and sufficiently reliable. There are 'no certainties in science'..."

    "The controlling standards published in the Guidance Manual are subject to further evolution. A 'disagreement over, not an absence of, controlling standards' is not a basis to exclude expert testimony. The methods described in the Guidance Manual are the product of 12 peer-reviewed publications on stable isotope analysis of perchlorate. The Guidance Manual is a product of interlaboratory collaboration that began before the initiation of this litigation."
  • Testability
    The question is whether an expert's methodology can be “challenged in some objective sense, or whether it is instead simply a subjective, conclusory approach that cannot reasonably be assessed for reliability...”

    "...The district court's conclusion was erroneous for three reasons: (1) other laboratories have tested the methodologies from the Guidance Manual used by Dr. Sturchio; (2) Dr. Sturchio's procedures are subject to retesting by another laboratory; and (3) challenges to the results obtained by using the techniques from the Guidance Manual go to the weight of the evidence and are a question for the fact finder, not the trial court..."
  • Reference Database
    "Dr. Sturchio's analysis [showed] that the 'dominant source of perchlorate in the Pomona groundwater is from Atacama (Chile)' and that the samples contained “minor amounts of perchlorate from other non-Atacama sources including synthetic and/or indigenous natural sources. 'Dr. Aravena's expert report cautioned that “not all the potential perchlorate sources have been characterized.' Dr. Sturchio, however, responded to Dr. Aravena's contention by arguing that Dr. Aravena's opinion was based on disclosures and quotations from old and outdated publications. Dr. Sturchio also explained that when the Pomona study was conducted, synthetic and Atacama sources of perchlorate were well known and well characterized. At most, this battle among experts merely shows that Dr. Sturchio may not know the isotopic composition of every source of perchlorate in the world with a certainty. Under Daubert, however, such a 'certainty' is not required, thus making this an invalid basis to exclude expert testimony."

Peer Review

Perchlorate (ClO4-) has several isotopes that have different proportions depending on geochemical processes that vary based on geography.

To understand the scientific issues surrounding this case, JuriLytics commissioned a peer review of Sturchio's Report. The reviewers focused on admissibility questions under Rule 702: (1) Is the testimony is based on sufficient facts or data? (2) Is the testimony the product of reliable principles and methods? (3) Has the expert reliably applied the principles and methods to the facts of the case?

Ultimately, the judge will make the decision of what facts are subject to admissibility considerations. However, our peer review is guided by a simple principle to determine whether the issue is under the gatekeeper's purview: is the issue a general scientific issue, or is it case-specific to this particular situation? As an example of this distinction, in DNA profiling, the question "How much degradation of DNA evidence is acceptable to produce reliable results?" would be a question for the gatekeeper, whereas the question "How degraded was the DNA sample in this case?" is a matter for the fact-finder to weigh.

Our case study utilized a double-blind review process to remove bias from the analysis. We first compiled a suggested list of reviewers based on publication and citation metrics within the field of "stable isotope geochemistry." We removed candidates that we found had co-published with Sturchio or had previously been excluded in court. These suggestions were then forwarded to Dr. Qing-zhu Yin, an associate editor of Geochimica et Cosmochimica Acta, a top Geochemistry journal. He was not aware of the identity of JuriLytics' client. Dr. Yin then nominated 20 potential reviewers either from JuriLytics' suggested list or through his own knowledge of the field. We then contacted the Reviewers in the order specified by the editor. After detailed conflict evaluation, we found three Reviewers for Dr. Sturchio's report: Dr. Zachary Sharp (Google Scholar), Dr. Howard Spero (Google Scholar), and Dr. Karen Johannesson (Google Scholar)

The comments of the three reviewers are highlighted below. The comments are categorized based on the three criteria that the district court used to exclude Sturchio:

District Court Opinion

"...the procedures used by Dr. Sturchio for isotopic analysis of perchlorate have not been generally accepted by the scientific community. In fact, the Department of Defense’s 'Guidance Manual for Forensic Analysis of Perchlorate in Groundwater using Chlorine and Oxygen Isotopic Analysis' states that 'the techniques and Quality Assurance/Quality Control (QA/QC) parameters are still being refined, and there are no USEPA-certified methods for CSIA of organic or inorganic compounds.'"

Johannesson
Johannesson

"...other stable isotope and environmental geoscientists have thoroughly and quantitatively vetted the methods and techniques employed to quantify the stable isotope ratios of perchlorate that are used in the Expert Report. "

"...the specific use of δ37Cl, δ18O, and Δ17O to fingerprint and trace the origin of perchlorate in environmental samples is less common in the scientific literature than the use of these isotopes to trace the origin and movement of water through Earth systems. Many, although not all, of the scientists involved in such investigations have worked with the author of the Expert Report in some capacity, either through direct collaboration, or more commonly, by adopting the protocol outlined in the Expert Report that has been previously gone through extensive peer-review as demonstrated by the numerous publications published by the author of the Expert Report and his colleagues from the USGS and DOE. This indicates to me, in conjunction with the numerous peer reviewed scientific papers published by the author of the Expert Report, that the parameters and techniques employed in the Expert Report are well accepted by the stable isotope geochemical community in general, and geochemists who study the stable chlorine and oxygen isotope ratios of perchlorate, in particular."

"...the Expert Report carefully documents the quality assurance and quality control (QA/QC) criteria that were followed during sampling and analysis that are crucial for ensuring high-quality data. The measurement precision of the stable isotope ratio analyses were all better than ± 0.5 ‰, which is always smaller than the individual sample symbols plotted on the δ37Cl vs. δ18O and Δ17O vs. δ18O cross-plots/discrimination diagrams (e.g., Fig. 7 of the Expert Report)."


Sharp
Sharp

"...the Sturchio lab is one of the finest in the world. The study outlines strict protocols that include tying the data into international standards. This means that any other laboratory using the same standardization would come to the same conclusion. The method of error propagation are well described."

"...The methodologies used in this work are state-of-the-art. Sturchio has been making these difficult measurements for decades. His work is considered first-rate."


Spero
Spero

"...The use of oxygen and chlorine isotope geochemistry to chemically fingerprint the origin/sources of perchlorate, is a relatively recent advance in isotope geochemistry, and was developed primarily in the laboratory of Neil Sturchio. From reading Dr. Sturchio’s papers, I conclude that his extraction methodologies yield purified perchlorate gas that has the necessary purity (>99%) to produce accurate δ18O, δ37Cl and Δ17O data on an IRMS system. The sample purification procedure is based on experimental data in peer-reviewed scientific publications and I do not see any issues with the procedures. "

District Court Opinion

"The Procedures used in this case are further flawed in that they have not been tested by other laboratories and are not subject to retesting given the failure to take dual samples."

Johannesson
Johannesson

"...it seems prudent that one approach that could be undertaken to strengthen the results and conclusions of the Expert Report would be to send duplicates of blind samples of groundwater from the site in question in Pomona, California, to the author of the Expert Report’s laboratory and another laboratory that routinely makes the same types of measurements."

"...The intra laboratory precision of the stable isotope ratios analyses are documented and of high quality. One weakness is not having a standard reference material (e.g., USGS-37 and USGS-38) with agreed upon δ37Cl ratios, which can slightly hamper direct evaluation measurement accuracy."

"...the δ37Cl and perhaps Δ17O ratios of the reference materials are not standardized. That is, because few laboratories appear to use USGS-37 and USGS-38 for δ37Cl and Δ17O, the precise ratios of these stable isotope is not well constrained, or at least not certified by an organization such as NIST. Nonetheless, the laboratory employed for the Expert Report followed a standard and well-accepted approach to this problem that is employed by most stable isotope laboratories for many measurement. Specifically, by comparison of the measured δ37Cl for USGS-37 and USGS-38 to an internal reference standard of methyl chloride gas, for which δ37Cl is very well known as it is measured during every analysis conducted in the laboratory, they can produce reliable and reproducible stable isotope ratios for the two external standards, USGS-37 and USGS-38. Therefore, such an approach allows the laboratory to quantitatively evaluate the accuracy of their measurement. Therefore, high accuracy of the results reported in the Expert Report is expected following this standard approach."


Sharp
Sharp

"...In some studies, uncertainties in measurements can lead to false conclusions. All analyses have uncertainties. In this study, even if we assume ridiculously large uncertainties, the same conclusions are obtained. The analytical facilities of Neil Sturchio are respected as some of the finest (if not the finest) in the world. The conclusions are robust."

"...the Sturchio lab is one of the finest in the world. The study outlines strict protocols that include tying the data into international standards. This means that any other laboratory using the same standardization would come to the same conclusion. The method of error propagation are well described."


Spero
Spero

"...Precision and accuracy for IRMS systems are laboratory specific and vary depending on the gases being analyzed and the amount of sample available for analysis. Table 1 in the Expert Report, presents the reproducibility of measurements on standards and represent the ‘best’ precision one can obtain for unknown samples. Accuracy is determined through analyses of samples with standards of known values, and subsequent correction of the measured vs actual values for those standards. The corrections obtained for standards are then applied to sample values. The analyses of standards for δ37Cl and Δ17O show that the precision of these measurements is comparable to data obtained in other published studies in the peer review literature. However, the analytical precision for δ18O analyses (~0.5‰) is approximately 5 times higher than precision I could find in the published literature (~0.1‰). However, I note that data in Table 4 – Estimated percentages of perchlorate sources – are based on δ37Cl and Δ17O data."

District Court Opinion

"Dr. Sturchio's database is too limited in order for him to reliably comment on the exclusiveness of the location of the potential source of perchlorate in Pomona's water with an acceptable rate of error."

Johannesson
Johannesson

"...the conditions necessary to produce the stable isotope ratios reported for these Atacama deposits are exceedingly rare as the Atacama Desert is by far the driest desert on the planet, and perhaps the highest in terms of elevation. I suspect that one of the only other places in the solar system with similar conditions that we know about is Mars. It is also important to underscore that the perchlorate in the Pomona groundwaters is very different than another synthetic perchlorate samples analyzed and very different from perchlorate formed naturally in the desert southwest of the USA. It seems highly unlikely, although as a scientist not necessarily impossible, that the perchlorate in Pomona groundwaters analyzed for the Expert Report originated from any other source than the nitrate-rich deposits of the Atacama desert."


Sharp
Sharp

"...The case for a predominantly Atacama source is very strong. It is unusual to have three isotopes systems that can be applied to a source study. In this case, all three support Atacama as the primary perchlorate source."

"...All synthetic perchlorate has a Δ17O value of zero, as pointed out in the report, so a predominantly synthetic source can be ruled-out."

"...Another, yet unknown exotic source could theoretically explain the Pomona data, but would require a number of unlikely conditions. First, the δ37Cl value of the source would have to be the same as Atacama. This limits the possible sources. This 'other' source would also have had to be commercially available for fertilizer. I have not looked in detail for possible natural nitrate fertilizers, but the options are quite limited. An extremely arid source is needed. The Qaidam Basin, China has been mined since 1989 and extensive mining operations exist in the Dead Sea (Fertilizer Manual. 1998. edited by UN Industrial Development Organization, International Fertilizer Development Center). I am not aware of any Cl isotope data from either of these deposits. They could have identical isotope compositions as the Atacama deposits, but this would be unlikely."


Spero
Spero

"...The database that Sturchio has amassed with respect to fertilizer sources is meager and not complete by any means. Without information about the particular type of fertilizer used in an area (e.g. records of where farmers purchased their fertilizer and what the sources of that fertilizer had been), a researcher would need to know the Cl, O and 17O excess geochemistry of the top fertilizers available for use by farmers in order to confidently state that the Atacama fertilizer is the primary source of the aquifer perchlorate. Once the top 95% or so primary fertilizers had been characterized geochemically, one could potentially eliminate the possibility that another fertilizer could have a chemical fingerprint that was the same as the Atacama perchlorate geochemistry with a high degree of confidence (I have no idea how many different fertilizer sources exist/were used in the area)."

"...it is important to note that the 17O excess chemical signature indicates that the ‘natural’ Atacama fertilizer is composed of some oxygen that underwent upper atmosphere photolytic reactions whereas the synthetic and SHP perchlorate displays no such oxygen component. This parameter is sufficient to separate these sources with nearly absolute confidence."

"...It is ‘possible' that another region of the world could have fertilizer source with a similar geochemical signature for Cl, 18O and Δ17O excess. Without analyses of fertilizers from other regions, one cannot exclude this possibility. "

Takeaways

Peer review of Sturchio's report affirmed his principles and methods. This neutral peer review, if made available to the judge in a Daubert evaluation, would have prevented his exclusion. Tens of thousands of dollars in appeals costs could have been saved by a cost-effective peer review of the science in this case.

The district court excluded Sturchio's opinions in part based on the premises that the methods are not generally accepted and that no retesting of the results was possible. Yet, all three reviewers stated that the methods were well-accepted in the field and that his lab was state-of-the-art.

In terms of testability, the Sturchio lab used external calibration standards to peg the accuracy of the measurements. These standards allow an apple-to-apple comparison of his results to any other lab in the world. As Dr. Sharp said, "The study outlines strict protocols that include tying the data into international standards. This means that any other laboratory using the same standardization would come to the same conclusion." Testability was not a concern for Sturchio's work.

With regard to the database, two reviewers agreed that although one cannot rule out alternative sources from the Atacama, various other factors indicate that alternative sources are remote possibilities. Sharp mentions two remote possibilities, the Qaidam Basin and the Dead Sea, but ultimately discounts them. Johannesson is also skeptical about another source with similar conditions to the Atacama: "the Atacama Desert is by far the driest desert on the planet, and perhaps the highest in terms of elevation. I suspect that one of the only other places in the solar system with similar conditions that we know about is Mars." Spero cannot discount other possibilities for sourcing the perchlorate. However, he suggests that if the top fertilizers used in Pomona can be characterized, other possible sources can be eliminated. All reviewers agree that man-made sources of perchlorate can be eliminated.

The database used by Sturchio, ultimately, did not support an absolute and categorical conclusion that the Atacama was the source of perchlorate. Yet, such a conclusion was not necessary for the City of Pomona to present a convincing case. The reviews suggest an alternative litigation strategy for Pomona. Sturchio's identification that the Pomona perchlorate is consistent with Atacama perchlorate is significant in and of itself. Coupled with documentation that the bulk of fertilizer used in Pomona was sourced from Atacama, the case would have been quite convincing without compromising the "analytical gap" between the data and the conclusions Sturchio reached.

In this case study, JuriLytics was able to affirm Sturchio's methods and principles, and could have saved Pomona enormous resources spent towards appeal. Even better, if peer review was used for Sturchio's draft reports, such reviews may have saved Sturchio from drawing unnecessarily strong conclusions when a more integrated litigation strategy would have ultimately resulted in a better case.

Documents

Expert Report of Neil Sturchio

Peer Review of Karen Johannesson

Peer Review of Zachary Sharp

Peer Review of Howard Spero