Conformance Testing

Use the conformance test suite to verify ISO/IEC 29794-4 compliance

NFIQ2 includes a conformance test suite to verify that your build correctly implements ISO/IEC 29794-4. This guide shows how to use the test suite and interpret results.

Overview

The conformance test suite validates that NFIQ2 produces consistent, specification-compliant results by:

1. Running NFIQ2 on a standardized dataset
2. Comparing your output against expected reference values
3. Identifying any discrepancies

The conformance dataset images are subject to a license agreement and must be downloaded separately from NIST.

Test Suite Components

The conformance test suite is located in the conformance/ directory:

conformance/
├── README.md                                      # Documentation
├── diff.py                                        # Comparison script
├── conformance_expected_output-v2.3.0.csv        # Reference output
└── conformance_expected_output-v2.3.0-mapped.csv # Reference with 0-100 mapping

Files Included

conformance_expected_output-v2.3.0.csv

Expected output from NFIQ2 v2.3.0 when run against the conformance dataset. Contains:

• Unified quality scores
• All quality metric values (from -v flag)
• Actionable feedback (from -a flag)

conformance_expected_output-v2.3.0-mapped.csv

Expected output with all values mapped to 0-100 scale (using -b flag).

diff.py

Python script to compare two CSV outputs from NFIQ2 ≥2.1. Requires Python 3 and Pandas ≥1.1.0.

Prerequisites

1. Download Conformance Dataset

   Download the conformance dataset images from the NIST website.

   The dataset contains fingerprint images specifically designed for conformance testing.

2. Install Python Dependencies

   Install required Python packages:

   pip install pandas>=1.1.0
   

   Or with conda:

   conda install pandas>=1.1.0
   
3. Build NFIQ2

   Build NFIQ2 with the CLI tool enabled:

   mkdir build && cd build
   cmake .. -DBUILD_NFIQ2_CLI=ON
   cmake --build . --config Release
   

Running Conformance Tests

Step 1: Generate Your Output

Run NFIQ2 on the conformance dataset to generate your output CSV:

# Navigate to your build directory
cd build/bin

# Run NFIQ2 on all conformance images
./nfiq2 -i /path/to/conformance/images/*.png \
        -m /path/to/nist_plain_tir-ink.txt \
        -v -a \
        -o my_conformance_output.csv

Flags explained:

• -i: Input fingerprint images (conformance dataset)
• -m: Model file (random forest parameters)
• -v: Include quality metric values
• -a: Include actionable feedback
• -o: Output CSV file

For mapped output (0-100 scale), add the -b flag and compare against conformance_expected_output-v2.3.0-mapped.csv.

Step 2: Compare Against Reference Output

Use the diff.py script to compare your output with the expected reference:

cd /path/to/nfiq2/conformance

python3 diff.py \
    conformance_expected_output-v2.3.0.csv \
    /path/to/my_conformance_output.csv

Understanding Results

Conformant Output

If your build is conformant, you’ll see no output:

$ python3 diff.py conformance_expected_output-v2.3.0.csv my_output.csv
$ echo $?
0

Exit code 0 indicates perfect conformance.

Non-Conformant Output

If there are differences, they will be printed:

$ python3 diff.py conformance_expected_output-v2.3.0.csv my_output.csv
"filename","column",csv1_value,csv2_value,difference
"image001.png","QualityScore",85,84,-1.0
"image002.png","FDA_Bin10_Mean",12.34,12.35,0.01
"image003.png","FingerJetFX_MinutiaeCount",42,41,-1.0

Exit code 1 indicates differences were found.

diff.py Usage

Basic Usage

python3 diff.py <reference.csv> <your_output.csv>

Command-Line Options

python3 diff.py [options] <reference.csv> <your_output.csv>

reference.csv · string · required

Path to the reference CSV (e.g., conformance_expected_output-v2.3.0.csv)

your_output.csv · string · required

Path to your generated NFIQ2 CSV output

-o · string

Write differences to a file instead of stdout

python3 diff.py reference.csv my_output.csv -o differences.csv

-s · flag

Silent mode: only print True/False to indicate if files match

python3 diff.py reference.csv my_output.csv -s
# Output: False (if differences found)
#     or: True (if identical)

Examples

Standard Comparison

# Compare outputs and display differences
python3 diff.py \
    conformance_expected_output-v2.3.0.csv \
    my_conformance_output.csv

Validated Columns

The diff.py script validates the following columns:

Basic Metrics

• Filename: Image filename
• FingerCode: Finger position code
• QualityScore: Unified quality score (1-5)
• OptionalError: Error messages if any

Quality Checks

• Quantized: Image quantization check
• Resampled: Image resampling check
• EmptyImageOrContrastTooLow: Contrast validation
• UniformImage: Uniformity check
• FingerprintImageWithMinutiae: Minutiae presence
• SufficientFingerprintForeground: Foreground adequacy

Frequency Domain Analysis (FDA)

• FDA_Bin10_0 through FDA_Bin10_9: Frequency bins
• FDA_Bin10_Mean: Mean FDA value
• FDA_Bin10_StdDev: FDA standard deviation

Minutiae Features

• FingerJetFX_MinutiaeCount: Total minutiae count
• FingerJetFX_MinCount_COMMinRect200x200: Center minutiae
• FJFXPos_Mu_MinutiaeQuality_2: Minutiae quality
• FJFXPos_OCL_MinutiaeQuality_80: High-quality minutiae

Orientation & Ridge Features

• OCL_Bin10_*: Orientation certainty level bins
• OF_Bin10_*: Orientation flow bins
• LCS_Bin10_*: Local clarity score bins
• RVUP_Bin10_*: Ridge valley uniformity bins
• OrientationMap_ROIFilter_CoherenceRel
• OrientationMap_ROIFilter_CoherenceSum

Other Measures

• ImgProcROIArea_Mean: ROI area mean
• MMB: Minutiae-based measure
• Mu: Mean intensity

Interpreting Differences

Acceptable Differences

Minor floating-point differences (< 0.001) may occur due to:

• Different compiler optimizations
• Platform-specific math libraries
• Rounding in intermediate calculations

Unacceptable Differences

Large differences or incorrect quality scores indicate:

• Build configuration issues
• Incorrect model file
• Modified source code
• Platform-specific bugs

If you encounter significant differences, verify:

1. Using the correct model file
2. Build configuration matches reference
3. No modifications to source code
4. Conformance dataset is correct version

Automated Testing

Integrate conformance testing into your CI/CD pipeline:

Shell Script

#!/bin/bash
set -e

# Run NFIQ2 on conformance dataset
./nfiq2 -i conformance_images/*.png \
        -m nist_plain_tir-ink.txt \
        -v -a \
        -o test_output.csv

# Compare with reference
if python3 conformance/diff.py \
    conformance/conformance_expected_output-v2.3.0.csv \
    test_output.csv -s; then
    echo "✓ Conformance test PASSED"
    exit 0
else
    echo "✗ Conformance test FAILED"
    python3 conformance/diff.py \
        conformance/conformance_expected_output-v2.3.0.csv \
        test_output.csv \
        -o differences.csv
    echo "Differences written to differences.csv"
    exit 1
fi

Troubleshooting

Missing Required Columns

Error: The following required columns are missing in <file>

Solution: Ensure you ran NFIQ2 with both -v and -a flags:

./nfiq2 -i images/*.png -m model.txt -v -a -o output.csv

Row Count Mismatch

Error: The number of rows in the CSV files differ

Solution: Verify you’re using the complete conformance dataset with all images.

Unmatched Filenames

Error: Some "Filename" keys are not present in both files

Solution: Check that:

• File paths in both CSVs use consistent separators
• All conformance images were processed
• No extra images were included

Pandas Version Error

Error: ModuleNotFoundError: No module named 'pandas'

Solution: Install Pandas:

pip install pandas>=1.1.0

Best Practices

Test Early

Run conformance tests immediately after building to catch issues early

Automate Testing

Integrate conformance tests into your CI/CD pipeline

Document Differences

If intentional changes cause differences, document them clearly

Version Control

Keep conformance outputs in version control to track changes

Next Steps

Computing Scores

Learn how to compute quality scores in your application

Quality Measures

Understand the quality measures being validated