Medical Image De-Identification Benchmark Challenge

Providing data for an image deID benchmark is challenging because actual PHI/PII cannot be publicly released, and gold standard deID answers keys for synthetic PHI/PII datasets are not available. Most deID algorithms have been developed in house using private datasets containing PHI/PII and evaluated through manual inspection by experts. This approach limits who can develop these algorithms to those who have access to such datasets. Developers without these in-house resources can benefit from realistic DICOM datasets that do not contain actual PHI/PII but rather synthetic facsimiles of PHI/PII, together with an automated mechanism for evaluating their results. Through a separate project we created the MIDI Dataset, a large portion of which was used in the MIDI-B Challenge. The MIDI Dataset is a large and diverse set of clinical DICOM images from multiple modalities and submitting sites, sourced from original already de-identified images in The Cancer Imaging Archive (TCIA) (also available in the National Cancer Institute Imaging Data Commons (IDC)), then infused with synthetic identifiers (PHI/PII). Imaging modalities represented in the MIDI Dataset include computed radiography (CR), magnetic resonance (MR), computed tomography (CT), positron emission tomography (PET), digital X-Ray (DX), structured report (SR), mammography (MG), and ultrasound (US). In the MIDI-B Challenge, most images are from the MR, CT, and PT modalities. Details of the design and construction of the MIDI Dataset and the ground truth answer keys are described elsewhere (Rutherford et al., 2021b). Large portions of the MIDI dataset were used for the MIDI-B challenge and partitioned into validation and test sets which are outlined in Rutherford et al, included in this special issue.

The MIDI-B Challenge consisted of three phases of Training (3 months), Validation (1 month), and Test (1 week). The challenge was implemented in the Synapse platform (Sage Bionetwork). Participants were required to register for the challenge. All operations for the MIDI-B challenge, including access to validation and test data and submission of results were handled through the Synapse platform. Participants were allowed to download the appropriate dataset for each phase of the challenge through the Synapse platform.

The MIDI-B Challenge participants were expected to train their models on their own (in house) DICOM images. However, to familiarize participants with the dataset used for validation and testing, they were pointed to a small subset of the MIDI dataset (before and after deID) available through The Cancer Imaging Archive (TCIA) and Imaging Data Commons (IDC) (Rutherford et al., 2021b).

During the validation phase, DICOM images from 216 patients of the MIDI dataset were available to the participants as the validation dataset. The validation dataset contained a total of 23,921 instances, from 241 studies and 280 series. The distribution of the validation dataset modalities is shown in Table 1.

To advance to the test phase, each participating team was required to submit a short manuscript describing their algorithm and results for publication in the MICCAI associated proceedings in the Springer Lecture Notes in Computer Science. The manuscripts were reviewed by members of the challenge organizing committee and comments were provided to the authors. During the test phase, DICOM images from 322 patients of the MIDI dataset were available to the participants as the test dataset. The test dataset contained a total 29,660 instances, from 364 studies and 428 series. The test dataset has the same image modalities as the validation dataset. The distribution of the test dataset modalities is shown in Table 2.

The answer keys for the MIDI dataset (divided into validation and test datasets) are each a database of correct deID actions taken with respect to the value of various DICOM data elements. Each answer key is used to evaluate DICOM deID. They contain information on the tag, tag_name, value, action category, action_text, answer_category, etc. The key includes eight actions to be performed on the DICOM file header and two actions to be performed on the pixel data. The header actions are <date_shifted>, <patid_consistent>, <tag_retained>, <text_notnull>, <text_removed>, <text_retained>, <uid_changed>, and <uid_consistent>. The pixel data actions are <pixels_hidden> and <pixels_retained>. The action definitions used in the MIDI-B Challenge are listed below in Table 3.

In the validation phase and test phase, participants were required to submit two mapping files, patient ID and Unique Identifier (UID), along with de-identified images to the submission portal. The mapping files allowed correspondence between before and after files to be established. The two mapping files are table lists in CSV format, indicating the linkage of the patient IDs or UIDs before and after deID. The MIDI-B validation script (Rutherford et al., 2021b) evaluated each submission using the appropriate answer key, mapping files, and DICOM images.

We evaluated the de-ID results against various externally defined requirements, including the HIPAA Privacy Rule’s ’safe harbor method’ (18 elements) (U.S. Department of Health and Human Services, 2025), the DICOM PS3.16 Basic Confidentiality Profile National Electrical Manufacturers Association (2025) (NEMA). and TCIA’s best practices (The Cancer Imaging Archive, 2025), as proxies for the real-world risk of re-identification, which is indeterminable.

For each action type, we list the description and the scoring in Table 3. The instance-based accuracy, was computed as the percentage of the correctly executed actions across all instances including partial credit as shown in Table 3.

There were two measurement strategies used to assess the performance of the submissions: series-based and instance-based methods. The series-based method calculates the action accuracy as a percentage at the series level, while the instance-based method calculates the action accuracy as a percentage at the instance level. For the series-based method, an error for an action in any instance of the series counts as a complete error for that action in the series (no partial credit was given as it is at the instance level). Series-based scoring was used for final rankings in MIDI-B to normalize across modalities with very different numbers of instances per series (such as multiple slices in one series that share similar characteristics). The assumption was that the DICOM data elements and burned-in pixel text would be the same across different instances of the same series and hence have the same PHI/PII actions. This assumption was only partially true. There was no expectation that the series-based and instance-based scoring would produce the exact same rankings but that the rankings would probably be similar, as turned out to be the case. To be more informative to participants during the test phase, we also provided instance-based results.

Our evaluation method generated detailed Scoring and Discrepancy Reports, as described in the Appendix (A.1.1, A.1.2, A.1.3, and A.2). The Scoring Report consisted of three sheets: Scoring, Actions, and Categories. The Scoring sheet provided the number of correctly executed actions against the total number of required actions, which was used in computing the accuracy of each algorithm (Table A1). The Actions sheet summarized the number of errors, correct actions, and totals for each action (Table A2). The Categories sheet enumerated the number of errors, correct actions, and totals for each type and subcategory (Table A3). The Discrepancy Report, focused on pixel deID, listed the differences between the de-identified images and the corresponding Answer key (Table B1).

A total of 108 attempts from 17 individuals were submitted to the Validation Submission Portal. The performance for all submissions in the validation phase is shown in Figure 2. The series-based performance across all submissions was $96.49\%\pm 4.78\%$.

Each team was allowed to submit only once during the test phase. The series-based performance for all teams is shown in Table 4, and the series-based performance across all teams was $99.53\%\pm 0.60\%$.

The series-based method captured the unique action errors for each series but did not evaluate the performance at the instance level. Therefore, we also computed the accuracy using the instance-based method, as shown in Table 4. The instance-based performance across all teams was $99.09\%\pm 1.15\%$. The rankings based on the instance level are not exactly the same as the rankings based on the series level. Additionally, we performed the Analysis of Variance (ANOVA) to assess significant differences between the performances of the series-based and instance-based methods. The ANOVA test revealed a significant difference with a $p-value$ of $0.04458$. As mentioned earlier, the final rankings for the validation and test phases of the MIDI-B Challenge were based on the series-based method.

We summarize the detailed action errors based on the series-level scoring report for all teams inTable 5. The bold numbers indicate the best performance across all teams for each action type. Overall, most teams performed well in actions related to <date_shifted>, <patid_consisitent>, <pixels_hidden>, <uid_changed>, and <uid_consistent>, but not all did well for other actions. Notably, no team achieved the best performance across all action categories. In Appendix A.3, we also provide informative performance for the instance-base result in Table C1. Errors per type for both series-based and instance-based methods are summarized in Table C2 and Table C3.

No team performed perfectly or scored the best across all action types. There was more consistent performance in some action types than others and this was mostly true for action types on which most teams scored well. We provide our analysis of action-type-specific issues below.

The three action types on which most submitters performed perfectly or nearly perfectly were the <patid_consistent>, the <uid_changed>, and <uid_consistent> actions. They are measuring very similar PHI/PII deID steps. Basically, unique identifiers (UIDs) that are PHI/PII need to be replaced consistently (the same old UID with the same new UID) throughout the set of DICOM files in order to preserve referential integrity. This should be a fairly straightforward coding technique, usually involving hashing or table lookups. Any errors should be simple coding errors and algorithmic shortcomings.

The <date_shifted> action also requires a straightforward algorithmic substitution. Where dates in DICOM date data elements for a given patient are all consistently shifted by some amount to retain the temporal relevance between different imaging studies for the same patient. The amount to shift is arbitrary and should be different for each patient to reduce the risk of reverse engineering the shift amount. The validation script used for the MIDI-B challenge checked that dates were changed but not whether the dates were shifted consistently for the same patient or differently for different patients. No teams scored perfectly for this action type. We see two possible reasons for this. The date fields in the fixed DICOM data elements are not consistently formatted and some of the dates being checked by the validation script contain free text data.

For the <tag_retained> action only two teams scored perfectly, and no teams scored perfectly for the <text_notnull> action. These action types are measuring DICOM standard compliance rather than PHI/PII removal. A DICOM file needs to remain DICOM compliant to be maximally useful for downstream processing/analysis. The validation script is therefore penalizing each noncompliant data element. Unfortunately, the MIDI dataset used as input for the MIDI-B challenge is not itself perfectly DICOM compliant because of the imperfect original data or synthetic data generation. So, while the intent was to penalize only PHI/PII removal that was handled improperly resulting in DICOM noncompliant data elements (i.e., made no worse by deID), the validation script actually penalized all noncompliance, including pre-existing issues. The submitters were not explicitly tasked with making noncompliant DICOM files compliant and many did not. All submitters were essentially penalized the same based on this factor.

The most challenging action types were <text_removed> and <text_retained>. They are measuring PHI/PII removal from free text DICOM data elements. The <text_removed> action measures whether PHI/PII that should be removed is completely removed. The <text_retained> action measures how much text that is not PHI/PII is removed by mistake (i.e., is undesirable to remove for preservation of research utility). There is often an algorithmic tradeoff between aggressively labeling text as PHI/PII to reduce false negatives at the expense of increasing false positives. Team 02 had the fewest combined errors for these two action categories and the best balance between the two action type errors. Most errors in the <text_removed> and the <text_retained> actions were due to retaining/removing text or only partially removing/retaining it. For example, in some submissions, the text in Series Description (0008, 103e) still existed for the <text_removed>. The Study Description (0008,1030) text ”<BREASTˆROUTINE for MASS for 311-25-3722>” was incorrectly trimmed to “<BREASTˆROUTINE for 311-25-3722>” for the <text_retained>. In these cases, the improper operations led to a penalty of partial score for the actions.

Only two teams successfully completed the <tag_retained> action. For example, the value of Acquisition Number (0020, 0012) was deleted by some teams.

For <text_notnull> action, none of the submissions were entirely successful for all DICOM images. For example, there was no value for Image Type (0008, 0008) in some submissions. The existence of free text made the action more challenging.

Optical character recognition (OCR) techniques and tools were widely used by MIDI-B participants to de-identify the burned-in PHI/PII from the pixel images. In the action of <pixels_hidden>, four teams performed well. The errors occurred due to false positives or false negatives in the deID process of burned-in text. However, it is difficult to conclude the performances because only a very small number of pixel images with burned-in PHI/PII were present in the MIDI dataset and evaluated. For the <pixels_retained> action, half of the submissions failed to perform correctly. While nearly all teams achieve $98+\%$ accuracy in the total score, there are still many errors in pixel image deID, including false positives (FP), and false negatives (FN). Figure 3 demonstrates several instances of improper pixel image deID. The top-left image shows the original image. The top-right image depicts the post-deID image, which includes false negatives (the PHI/PII information remains in the top red box) and false positives (some white blocks appear in the bottom red box). The bottom-left image shows a different post-deID of the same image with false negatives, where the date of birth still appears in the red box. Finally, the bottom-right image demonstrates over-removal because of the false positives, where the anatomical structure image has been altered beyond the removal of PHI/PII information.

Improper pixel image deID may be partial, such as the retention of partial PHI/PII information or the removal of non-PHI/PII information. In the top-row of Figure 4, the deID process result contains false negatives. For example, only the last name is de-identified in the example, while the first name remains. In the bottom-row of Figure 4, the text ’SEMI-UPRIGHT’ at the position of the bottom red box has been misrecognized as PHI/PII information and then incorrectly redacted.

As mentioned, a limitation of the current MIDI-B dataset is the small number of images with PHI/PII data burned into the pixels. This limitation arose partly from the effort required to construct such images and partly from the effort needed to evaluate them for deID. As a result, the scoring greatly underweighted the action of <pixels_hidden>.

The MIDI-B challenge was the first of its kind, and as such, during the course of the challenge and afterward, we learned many lessons and noted areas that could be improved in the future.

The first lesson was how best to evaluate the action accuracy. We decided to calculate accuracy by dividing the total number of correct actions by the total number of actions treating all action types equally. This was problematic in two ways. It weights all types equally even if some types of actions/errors are more impactful for either PHI/PII retention or utility preservation. Deciding on a weighting scheme for each type is not obvious or impartial so we chose not to assign any arbitrary scaling.

The second lesson was the impact of the implicit weighting that occurs based on the number of occurrences of each action type in the dataset. Assuming that each action type is important, then performance for that action type should have an equivalent impact on the final score. Our scoring approach assigned more weight to types which have more actions. This was partially addressed by providing detailed scoring breakdowns by type to the participants. A simple correction for the final score is to normalize the score by actions per type to give equal weight to each type, then the scoring function would be $Accuracy=\frac{1}{N}\sum_{i=1}^{N}\frac{C_{i}}{S_{i}}$, where $N$ is the number of action types, $C_{i}$ and $S_{i}$ are the sum of scores for all actions and the total number of the $i^{th}$ actions types, respectively, as shown in Table 6. For all teams the normalized series-based scores are lower than unnormalized series-based scores used in the MIDI-B Challenge. This implies that all teams did better on the action types which were more prevalent versus the less prevalent action types. Alternatively, the accuracy formula could incorporate a scaling of the perceived importance of each type. With this approach, after normalizing by number of actions in a type, each type could also be assigned an arbitrary weight depending on its perceived importance, with more critical types receiving higher weights. The resulting scoring function becomes $Accuracy=\sum_{i=1}^{N}\left(\frac{C_{i}}{S_{i}}\times\omega_{i}\right)$, where $C_{i}$, $S_{i}$, and $\omega_{i}$ are the sum of scores, the total number, and the corresponding weight of the $i^{th}$ action type, respectively.

The second lesson learned pertained to the selection of the measurement method. As discussed in the section on measurement metrics, we chose the series-based method for the challenge. This method highlights the uniqueness of errors at the series level, allowing us to assess performance by focusing on overarching patterns and trends across a series of actions. While this method has its advantages, it also has limitations. Specifically, it may not effectively capture all errors that occur at the instance level. Errors that are isolated or specific to individual instances could be overlooked or under-weighted, potentially leading to an incomplete understanding of the system’s overall accuracy. This trade-off underscores the importance of carefully considering the goals and scope of the evaluation when selecting a measurement method.