Fingerprint comparison: error, repeated decisions and agreement between examiners
Fingerprint comparison has several measurable limits, and the measurements concern different things. Accuracy describes whether a conclusion agrees with the known source of two impressions. Repeatability concerns the same examiner reaching the same decision again. Reproducibility concerns agreement between examiners. A partial or distorted mark can create difficulty at each stage, but one percentage cannot stand for all three. The original experiments distinguish source errors from changes of mind and from differences over whether enough detail exists for a conclusion.
The National Forensic Science Technology Center describes fingerprint comparison through analysis, comparison, evaluation and verification, or ACE-V. Analysis includes deciding whether the mark has usable information. Evaluation can produce identification, exclusion or an inconclusive result. An inconclusive result may reflect poor quality, insufficient detail or noncomparable areas; it is a decision category, not automatically an error. Computer searches supply potential candidates, while examiners assess the impressions. Verification adds another examiner’s assessment. These distinctions matter when interpreting an experiment that measures individual decisions rather than an entire laboratory workflow.
Ulery, Hicklin, Buscaglia and Roberts’s 2011 study involved 169 practicing latent-print examiners and a pool of 744 latent-and-reference image pairs. Each examiner received approximately 100 pairs. The pool contained 520 same-source pairs and 224 different-source pairs; repeated assignments produced many more decisions than distinct pairs. Participants first assessed the latent mark’s value and then, when appropriate, compared it with the reference. The researchers knew which impressions shared a source. Those known origins made it possible to classify false identifications and false exclusions independently of examiner agreement.
Six false identifications occurred in 4,083 different-source comparisons whose latent marks had been judged of value for individualization. Five examiners made those six errors. The paper reports the rate, rounded to one decimal place, as 0.1%. Its 7.5% false-exclusion rate concerns same-source comparisons judged of value for individualization, a different denominator and a different kind of error. Neither percentage represents the proportion of all criminal cases that contain faulty fingerprint evidence. Decisions that a mark had no value, inconclusive comparisons and determinations about source occupy distinct categories in the study’s design.
The fingerprint comparison experiment deliberately included challenging images, including different-source candidates from database searches. Its reference impressions contained more poor-quality examples than would be typical of the FBI database. Participants knew they were being tested, and they could not return to earlier comparisons. The authors caution that individual decisions do not measure the complete operational process, which may include verification and quality assurance. Other participants’ independent assessments detected every false identification in this dataset, an observation about these comparisons rather than a guarantee that review always catches an error.
The related 2012 study examined repeatability by retesting 72 of the original examiners after approximately seven months. Each received 25 previously seen pairs. Selection was stratified: it included randomly selected comparisons and deliberately reassigned earlier errors. Examiners were not told that they had seen the pairs before. They repeated 89.1% of initial identification decisions and 90.1% of initial exclusion decisions; most changes went to inconclusive. Those figures describe retention of decision categories on retest. They are not accuracy rates, and the selected retest is not a fresh random sample of ordinary cases.
Reproducibility in the same research concerned different examiners assessing the same material. The 2012 paper reports that at least 90% agreement was reached on 73% of same-source pairs when distinguishing identification from inconclusive, and on 56% of different-source pairs when distinguishing exclusion from inconclusive. Those percentages describe the share of image pairs meeting an agreement threshold. They do not mean that 73% or 56% of decisions were accurate. Much disagreement concerned the sufficiency of information for a categorical conclusion, with difficult images producing less consistency than easier ones.
Even the apparent consistency of a fingerprint comparison depends on the categories being counted. For latent-value decisions, the 2012 study reports 89.7% repeatability when grouping responses as suitable for identification or not suitable. Separating the latter group into no value and value for exclusion only reduced repeatability to 84.6%. The impressions had not changed; the classification had become finer. This is why agreement about whether to identify a source must be distinguished from agreement about the mark’s basic usefulness, and why an exclusion-only category cannot silently disappear from a numerical comparison.
NIST and the National Institute of Justice’s separate human-factors review addressed the wider system around fingerprint comparison. NIST’s 2012 announcement describes recommendations for documentation detailed enough for another examiner to assess the work, laboratory reporting rules and clearer testimony. It also identifies training, fatigue, stress, lighting and quality control as possible influences. These are proposed safeguards and sources of risk, not another measured error rate. Together, the records locate the limitation in interpreting and communicating the available ridge detail: a source conclusion, an inconclusive result and an agreement statistic each have a different evidential meaning.
- National Forensic Science Technology Center, Fingerprint Analysis: How It Is Done — www.forensicsciencesimplified.org
- Ulery, Hicklin, Buscaglia and Roberts, Accuracy and Reliability of Forensic Latent Fingerprint Decisions (2011) — NIST-hosted original paper — www.nist.gov
- Ulery, Hicklin, Buscaglia and Roberts, Repeatability and Reproducibility of Decisions by Latent Fingerprint Examiners (2012) — PLOS ONE — journals.plos.org
- NIST, Experts Recommend Measures to Reduce Human Error in Fingerprint Analysis, February 21, 2012 — www.nist.gov