Entity Mapping — Brainstorming Scenarios

A catalogue of real and realistic mapping challenges to stress-test the CanonicalMapper design. Each scenario includes a concrete example; solutions are intentionally omitted.

1. Model Doesn't Support a Common Dataset Entity

The model lacks a recognizer for entity types that appear frequently in the dataset.

Real example (Notebook 4): Default Presidio Analyzer has no ORGANIZATION recognizer. The synth dataset contains 200+ ORG annotations — all silently missed unless manually remapped.

Issue type produced: DATASET_ONLY (INFO) — non-blocking. Entities that the model never predicts are flagged so you're aware, but they don't block get_mapped_results_dataframe(). The false negatives are still counted in recall.

2. Massive Model Entity Set vs. Small Dataset

The model emits thousands of fine-grained labels; the dataset uses ≤10 broad categories.

Real example (Notebook 5): The OpenMed HuggingFace model predicts PERSON, LOCATION, ORGANIZATION, etc., while the synth dataset labels city, country, street_address, state, county, coordinate, postcode as separate entities.

Issue type produced: COLLISION_AMBIGUOUS (WARNING) — blocking. When a depth-2 ancestor like LOCATION is seen in predictions but the canonical surface is at depth 3 (computed by majority vote from the dataset annotations), the mapper can't determine which depth-3 entity to project onto. The user must call map({'LOCATION': 'LOC'}) (or another appropriate target) to resolve.

Projection rules in action: If instead the dataset uses depth-2 labels and the canonical surface locks at depth 2, then fine-grained model labels like STREET_ADDRESS auto-collapse to LOCATION as COLLISION_TRIVIAL (INFO, non-blocking).

3. Semantic Ambiguity Requiring Human Judgment

Two entity types from different branches are semantically plausible for the same text span.

Real example (Notebook 4): NRP (Nationality/Religious/Political group) auto-maps to NATIONALITY, which sits in the DEMOGRAPHIC branch. The synth dataset treats "Tunisian", "French" as location-adjacent. The user must override with mapper.map({'NRP': 'NATIONALITY'}) or another target.

4. Same Dataset Label, Different Granularity Across Samples

The dataset uses a single label for spans that would naturally split into different canonical entities.

Real example: The synth dataset tags the entire address block as STREET_ADDRESS, but a model trained on CoNLL or OntoNotes might predict LOC for the city portion and GPE for the country portion only.

5. Country-Prefixed Labels from Regulatory Datasets

Datasets built for compliance (GDPR, HIPAA, PCI-DSS) use country-prefixed entity names that may or may not decompose cleanly.

Identification tiers: Country-prefix matching uses two tiers — COUNTRY (demonym or ISO code with remaining alias lookup) and COUNTRY_FALLBACK (strip leading country code and try again). Both fire before fuzzy matching.

6. Overlapping / Nested Entities

The model and dataset disagree on span boundaries because one tags nested entities and the other tags flat ones.

7. Label Collision Across Hierarchy Branches

The same string alias appears under multiple canonical entities in the hierarchy.

Issue types produced: - COLLISION_CROSS_BRANCH (WARNING) — blocking. Raised when a label resolves to a canonical entity that has co-occurring labels on the same tokens mapping to a different hierarchy branch. Must be resolved with map() before extracting results. - COLLISION_AMBIGUOUS (WARNING) — blocking. Raised when a depth-2 ancestor maps to multiple depth-3 entities on the canonical surface (the top co-occurring candidate is shown in overlap_counts). Use map({'LABEL': 'CANONICAL'}) to pick the right one.

8. Model Predicts Entities the Dataset Doesn't Annotate (Prediction-Only)

The model finds PII types the dataset creators never labeled — every detection is a false positive.

Real example (Notebook 5): After mapping, several Presidio predictions had no dataset counterpart.

Issue type produced: PREDICTION_ONLY (WARNING) — blocking. These labels inflate precision with false positives. You have three resolution options: 1. Suppress — mapper.map({'CREDIT_CARD': None}) excludes the label from evaluation entirely 2. Remap — mapper.map({'CREDIT_CARD': 'FINANCIAL'}) counts detections against the FINANCIAL annotation set 3. Keep as FP — if you want these counted as false positives deliberately, this isn't directly supported; suppression is the recommended path

9. Dataset Has Entities the Model Can Never Produce (Annotation-Only)

The dataset annotates entity types that the model's architecture or recognizer set can't detect.

Unlike prediction-only, the mapper currently has no ANNOTATION_ONLY issue type to flag this.

10. Depth-Level Mismatch Within a Single Evaluation

The dataset and model operate at different hierarchy depths, producing ancestor–descendant pairs in the same result row.

Real example (Notebook 5): Dataset labels city, street_address, postcode (depth 3), model predicts LOCATION (depth 2).

How the new API handles this: The canonical depth is computed automatically by majority vote from the dataset annotations. If the dataset is predominantly depth-3, the canonical surface is depth-3 and depth-2 model predictions trigger COLLISION_AMBIGUOUS (WARNING). Resolve with map({'LOCATION': 'LOC'}) to pick the right depth-3 target.

11. BIO/IOB Tag Scheme Artifacts

The model outputs BIO-tagged labels; the mapper must strip them correctly.

12. Fuzzy Matching False Positives

Short or similar entity names get incorrectly matched by fuzzy resolution.

13. Multi-Language / Multi-Script Entity Names

Datasets with non-English PII use native terms or transliterated labels.

The current hierarchy aliases are English-only. Any non-English label goes straight to UNRESOLVED.

14. Conflicting Mapping Decisions at Different Evaluation Depths

A mapping override that makes sense at depth 2 becomes wrong at depth 3.

15. Composite / Multi-Entity Spans

A single text span semantically contains multiple entity types, but only one label is assigned.

16. Temporal Ambiguity: Dates, Ages, and Durations

Numeric and temporal spans are inherently ambiguous across multiple entity types.

17. Suppressed Entities That Still Have Annotations

An entity is suppressed (→ None) because the model can't detect it, but the dataset has annotations. Those ground-truth spans silently vanish from the evaluation.

Real example (Notebook 5): EDUCATION_LEVEL, OCCUPATION, and LICENSE_PLATE are pre-mapped to None. After suppression the dataset annotations for those entities are excluded from the evaluation — this is intentional, but be aware that recall can appear inflated. The mapper does not produce a blocking issue for this; it's the user's explicit choice.

18. One-to-Many: A Single Model Entity Maps to Multiple Dataset Entities

The model predicts one label for spans that the dataset splits across categories.

At coarse depth they merge fine. At fine depth, the model can't distinguish — every prediction is technically correct but imprecise.

19. Cross-Domain Dataset Merge

A user combines two datasets from different domains, each with its own labeling convention.

The mapper sees identical labels and merges them — but the underlying entities aren't equivalent.

20. Model Confidence Threshold Effects on Mapping

Entity mapping is evaluated after the model has already made decisions based on a score threshold — changing that threshold shifts which entities even appear.

Real example: Notebook 4 uses threshold 0.4, Notebook 5 uses 0.3. The lower threshold in NB5 introduces AGE predictions (score 0.01 base) that only survive with context boosting — these are absent at 0.4.

Summary Matrix