IBSI 2 Phase 3 Compliance: Reproducibility

Overview

IBSI 2 Phase 3 focuses on reproducibility across different software implementations. Unlike Phases 1 and 2, there are no consensus reference values. Instead, participant results are compared against each other to measure overlap.

This page documents Pictologics' agreement with 9 other teams on a multimodal (CT, MRI, PET) soft-tissue sarcoma dataset of 51 patients.

How to Run the Benchmarks

1. Download the Data

• IBSI 2 Phase 3 Data: Please refer to the IBSI GitHub repository for all data download instructions.

Organize the data as follows: - data/ibsi2/data/validation/ct/*.nii.gz - data/ibsi2/data/validation/pet/*.nii.gz - data/ibsi2/data/validation/mri/*.nii.gz - data/ibsi2/data/validation/masks/*.nii.gz

2. Run Validation Programmatically

from pictologics import RadiomicsPipeline

# Example: Run Mean filter (ID 2) on a specific patient/modality
image_path = "data/ibsi2/data/validation/ct/STS_001_image.nii.gz"
mask_path = "data/ibsi2/data/validation/masks/STS_001_CT_mask.nii.gz"

pipeline = RadiomicsPipeline()

# Define IBSI 2 Phase 3 CT Preprocessing
preprocess_steps = [
    {"step": "resample", "params": {
        "new_spacing": (1.0, 1.0, 1.0),
        "interpolation": "cubic",
        "mask_interpolation": "linear",
        "mask_threshold": 0.5
    }},
    {"step": "round_intensities", "params": {}},
    {"step": "resegment", "params": {"range_min": -200, "range_max": 200}}
]

# Add Mean Filter (ID 2) and Feature Extraction
config = preprocess_steps + [
    {"step": "filter", "params": {"type": "mean", "support": 3}},
    {"step": "extract_features", "params": {"families": ["intensity"]}}
]

pipeline.add_config("phase3_demo", config)
results = pipeline.run(image_path, mask_path, config_names=["phase3_demo"])
print(results["phase3_demo"])

Phase 3 Results

Summary: Processed 153 scans (51 patients × 3 modalities), compared against 9 teams.

Tolerance Breakdown (% of Feature Range)

A value matches when its difference from the team's value is within 1, 5 or 10 % of the team's range for that modality, filter and feature (the maximum minus the minimum of the team's values over the 51 patients). Our value is first rounded to the team's number of decimals, as in the IBSI tolerance tests. A zero range needs an exact match.

Agreement by Configuration

The heatmap below shows the percentage of values within 10 % of the team's range (for the modality, filter and feature) for each configuration/team combination. Point at a cell for its values.

Per-Patient Agreement

The heatmap below shows per-patient agreement with each team across all filter configurations. Point at a cell for the scan and its values.

Mismatch Details

Mismatches shown below are cases where the error exceeds 10% of the team's range for the modality, filter and feature.

Configuration Legend

Simoncelli Boundary

The IBSI 2 manual's phase 3 table lists the mirror boundary for all filters. For the Simoncelli filters (IDs 8 and 9), Pictologics uses the periodic boundary of its FFT filters, as in IBSI 2 phases 1 and 2, because it gives the better match: over all teams, 58.1 % of the Simoncelli values are within 1 % of the team's range with the periodic boundary and 56.0 % with the mirror boundary; the periodic boundary matches 4 of the 7 teams at least as well. Share of values within 1 % of the range:

CERR

195 of 16524 values differ by more than 10 % of the range. The 10 largest, relative to the range:

Cardiff University

281 of 24786 values differ by more than 10 % of the range. The 10 largest, relative to the range:

King's College London

166 of 23868 values differ by more than 10 % of the range. The 10 largest, relative to the range:

NCT Dresden

128 of 24786 values differ by more than 10 % of the range. The 10 largest, relative to the range:

Qurit SERA

3161 of 24751 values differ by more than 10 % of the range. The 10 largest, relative to the range:

UCSF

14683 of 24786 values differ by more than 10 % of the range. The 10 largest, relative to the range:

USZ

12 of 16524 values differ by more than 10 % of the range. The 10 largest, relative to the range:

UdeS

4425 of 24786 values differ by more than 10 % of the range. The 10 largest, relative to the range:

Veneto Institute of Oncology

2501 of 24786 values differ by more than 10 % of the range. The 10 largest, relative to the range: