4 Evaluate Presidio Analyzer

Evaluate Presidio Analyzer using the Presidio Evaluator framework

This notebook demonstrates how to evaluate a Presidio instance using the presidio-evaluator framework.

Steps: 1. Load dataset 2. Dataset statistics 3. Define the Presidio Analyzer 4. Run predictions 5. Review and adjust entity mapping 6. Evaluate 7. Results and error analysis

For an example with a custom Presidio instance, see notebook 5.

# install presidio evaluator via pip if not yet installed

#!pip install presidio-evaluator

import json
from collections import Counter
from pathlib import Path
from pprint import pprint
import pandas as pd
from presidio_analyzer import AnalyzerEngine

from presidio_evaluator import InputSample
from presidio_evaluator.entity_mapping import CanonicalMapper, IncompleteMapping
from presidio_evaluator.evaluation import ModelError, Plotter, SpanEvaluator
from presidio_evaluator.experiment_tracking import get_experiment_tracker
from presidio_evaluator.models import PresidioAnalyzerWrapper

pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.set_option('display.max_colwidth', None)

1. Load dataset from file

dataset_name = "synth_dataset_v2.json"
dataset = InputSample.read_dataset_json(Path(Path.cwd().parent, "data", dataset_name))
print(len(dataset))

tokenizing input:   0%|          | 1/1500 [00:00<04:44,  5.27it/s]

loading model en_core_web_sm


tokenizing input: 100%|██████████| 1500/1500 [00:04<00:00, 326.80it/s]

1500

This dataset was auto generated. See more info here Synthetic data generation.

def get_entity_counts(dataset: list[InputSample]) -> Counter:
    """Return a dictionary with counter per entity type."""
    entity_counter = Counter()
    for sample in dataset:
        for tag in sample.tags:
            entity_counter[tag] += 1
    return entity_counter

2. Simple dataset statistics

entity_counts = get_entity_counts(dataset)
print("Count per entity:")
pprint(entity_counts.most_common(), compact=True)

print(
    "\nMin and max number of tokens in dataset: "
    f"Min: {min([len(sample.tokens) for sample in dataset])}, "
    f"Max: {max([len(sample.tokens) for sample in dataset])}"
)

print(
    f"Min and max sentence length in dataset: "
    f"Min: {min([len(sample.full_text) for sample in dataset])}, "
    f"Max: {max([len(sample.full_text) for sample in dataset])}"
)

print("\nExample InputSample:")
print(dataset[0])

Count per entity:
[('O', 19626), ('STREET_ADDRESS', 3071), ('PERSON', 1369), ('GPE', 521),
 ('ORGANIZATION', 504), ('PHONE_NUMBER', 350), ('DATE_TIME', 219),
 ('TITLE', 142), ('CREDIT_CARD', 136), ('US_SSN', 80), ('AGE', 74), ('NRP', 55),
 ('ZIP_CODE', 50), ('EMAIL_ADDRESS', 49), ('DOMAIN_NAME', 37),
 ('IP_ADDRESS', 22), ('IBAN_CODE', 21), ('US_DRIVER_LICENSE', 9)]

Min and max number of tokens in dataset: Min: 3, Max: 78
Min and max sentence length in dataset: Min: 9, Max: 407

Example InputSample:
Full text: The address of Persint is 6750 Koskikatu 25 Apt. 864
Artilleros
, CO
 Uruguay 64677
Spans: [Span(type: STREET_ADDRESS, value: 6750 Koskikatu 25 Apt. 864
Artilleros
, CO
 Uruguay 64677, char_span: [26: 83]), Span(type: ORGANIZATION, value: Persint, char_span: [15: 22])]

3. Define the Presidio Analyzer

Using Presidio with default parameters (not recommended for production). For a customised example see notebook 5.

Note: The dataset may use different entity labels than Presidio (e.g. STREET_ADDRESS vs LOCATION). We'll align them in section 5 after running predictions.

# Loading the vanilla Analyzer Engine, with the default NER model.
analyzer_engine = AnalyzerEngine(default_score_threshold=0.4)

pprint("Supported entities for English:")
pprint(analyzer_engine.get_supported_entities("en"), compact=True)

print("\nLoaded recognizers for English:")
pprint(
    [rec.name for rec in analyzer_engine.registry.get_recognizers("en", all_fields=True)],
    compact=True,
)

print("\nLoaded NER models:")
pprint(analyzer_engine.nlp_engine.models)

'Supported entities for English:'
['CREDIT_CARD', 'MAC_ADDRESS', 'EMAIL_ADDRESS', 'NRP', 'IBAN_CODE', 'PERSON',
 'US_SSN', 'CRYPTO', 'IP_ADDRESS', 'DATE_TIME', 'PHONE_NUMBER', 'LOCATION',
 'US_ITIN', 'US_BANK_NUMBER', 'US_PASSPORT', 'URL', 'UK_NHS',
 'US_DRIVER_LICENSE', 'MEDICAL_LICENSE']

Loaded recognizers for English:
['CreditCardRecognizer', 'UsBankRecognizer', 'UsLicenseRecognizer',
 'UsItinRecognizer', 'UsPassportRecognizer', 'UsSsnRecognizer', 'NhsRecognizer',
 'CryptoRecognizer', 'DateRecognizer', 'EmailRecognizer', 'IbanRecognizer',
 'IpRecognizer', 'MedicalLicenseRecognizer', 'MacAddressRecognizer',
 'PhoneRecognizer', 'UrlRecognizer', 'SpacyRecognizer']

Loaded NER models:
[{'lang_code': 'en', 'model_name': 'en_core_web_lg'}]

# Wrap the analyzer for evaluation and analysis
wrapped_analyzer = PresidioAnalyzerWrapper(analyzer_engine=analyzer_engine)

# Set up the experiment tracker and log model + dataset params
experiment = get_experiment_tracker()
params = {"dataset_name": dataset_name, "model_name": wrapped_analyzer.name}
params.update(wrapped_analyzer.to_log())
experiment.log_parameters(params)
experiment.log_dataset_hash(dataset)

4. Run predictions

Run the model on the full dataset. Returns a 5-column DataFrame (sentence_id, token, annotation, prediction, start_indices).

%%time
results_df = wrapped_analyzer.predict_dataset(dataset)
results_df.head()

CPU times: user 5.63 s, sys: 142 ms, total: 5.77 s
Wall time: 5.84 s

	token	annotation	prediction	start_indices
0	The	O	O	0
1	address	O	O	4
2	of	O	O	12
3	Persint	ORGANIZATION	O	15
4	is	O	O	23

5. Review entity mapping

The entity mapping process auto-resolves labels using a hierarchical entity tree. The CanonicalMapper class analyzes the results (predictions and annotations), and checks if there are any gaps towards the evaluation process.
If you call render_html() to inspect issues, you'd see if there are any gaps that prevent you from completing the evaluation successfully. Make sure you address all the issues by using:
mapper.map({"LABEL": "CANONICAL"}) to remap or
mapper.map({"LABEL": None}) to suppress.
For more information about entity mapping, see Notebook 6 - Entity Mapping.

mapper = CanonicalMapper()
mapper.analyze(results_df)
mapper.render_html()

Entity Mapping Audit 19 labels identified

✓ Ready for evaluation

Unresolved: 0Cross-branch: 0Pred-only: 0Dataset-only: 1Same-branch: 2

1. Blocking issuesnone

Labels that could not be mapped automatically (ERROR). Resolve these before calling mapper.get_mapped_results_dataframe().

✓ No blocking issues. All labels are fully resolved.

2. Warnings1 item(s)

Non-blocking warnings (WARNING). These won't prevent evaluation but may affect results.

WARNING ORGANIZATION

The dataset annotates this entity type, but the model never predicts it. There are no predictions to evaluate, so recall will be zero for this label. Consider whether the model is expected to detect this entity type.

3. Annotation labels17 label(s) from your dataset

The entity labels in your ground-truth annotations. Non-blocking issues (INFO) are shown inline.

Label	Resolved as	Tokens
STREET_ADDRESS	`ADDRESS`	3071
PERSON	`PERSON`	1369
GPE	`GPE`	521
ORGANIZATION	`ORGANIZATION`	504
PHONE_NUMBER	`PHONE_NUMBER`	350
DATE_TIME	`DATE_TIME`	219
TITLE	`TITLE`	142
CREDIT_CARD	`FINANCIAL`	136
US_SSN	`SSN`	80
AGE	`AGE`	74
NRP	`NATIONALITY`	55
ZIP_CODE	`ADDRESS`	50
EMAIL_ADDRESS	`EMAIL_ADDRESS`	49
DOMAIN_NAME	`DOMAIN`	37
IP_ADDRESS	`IP_ADDRESS`	22
IBAN_CODE	`FINANCIAL`	21
US_DRIVER_LICENSE	`DRIVER_LICENSE`	9

4. Prediction labels12 label(s) from the model

The entity labels your model outputs. Non-blocking issues (INFO) are shown inline.

Label	Resolved as	Tokens
PERSON	`PERSON`	1552
DATE_TIME	`DATE_TIME`	978
LOCATION	`LOCATION`	605
PHONE_NUMBER	`PHONE_NUMBER`	176
CREDIT_CARD	`FINANCIAL`	105
US_SSN	`SSN`	80
NRP	`NATIONALITY`	63
EMAIL_ADDRESS	`EMAIL_ADDRESS`	49
URL	`URL`	37
IBAN_CODE	`FINANCIAL`	21
IP_ADDRESS	`IP_ADDRESS`	21
US_DRIVER_LICENSE	`DRIVER_LICENSE`	4

mapped_results = mapper.get_mapped_results_dataframe()

# Log entity mappings for experiment tracking
experiment.log_parameter('entity_mappings', json.dumps(mapper.get_mapping()))

6. Evaluate

evaluator = SpanEvaluator(iou_threshold=0.75)
f_beta = 2  # F-beta parameter used throughout evaluation

skip words not provided, using default skip words. If you want the evaluation to not use skip words, pass skip_words=[]

hierarchical_results = evaluator.calculate_hierarchical_scores(
    mapped_results, 
    beta=f_beta
    )
binary_results = hierarchical_results["binary"]
branch_results = hierarchical_results["branch"]
detailed_results = hierarchical_results["detailed"]


# A. Binary PII/O metrics
print("Level: binary (PII vs O):")
pprint({
    "Precision": round(binary_results.pii_precision, 3),
    "Recall":    round(binary_results.pii_recall, 3),
    f"F{f_beta}":  round(binary_results.pii_f, 3),
})

Level: binary (PII vs O):
{'F2': 0.661, 'Precision': 0.733, 'Recall': 0.646}

# B. Branch-level per-entity scores (PERSON, LOCATION, DATE_TIME, …)
branch_plotter = Plotter(
    results=branch_results,
    model_name=wrapped_analyzer.name,
    display_mode="interactive",
    beta=f_beta,
)
branch_plotter.plot_scores(include_pii_aggregate=False, annotation_entities_only=True)

# C. Detailed-level per-entity scores (NAME, GPE, PHONE_NUMBER, …)
detailed_plotter = Plotter(
    results=detailed_results,
    model_name=wrapped_analyzer.name,
    display_mode="interactive",
    beta=f_beta,
)
detailed_plotter.plot_scores(include_pii_aggregate=False, annotation_entities_only=True)

7. Confusion matrix

# D. Branch-level confusion matrix (only branch entities that appear in the data)
entities, confmatrix = branch_results.to_confusion_matrix()
branch_plotter.plot_confusion_matrix(entities=entities, confmatrix=confmatrix)

8. Error analysis

Now let's look into results to understand what's behind the metrics we're getting. Note that evaluation is never perfect. Some things to consider: 1. There's often a mismatch between the annotated span and the predicted span, which isn't necessarily a mistake. For example: <Southern France> compared with Southern <France>. In the second text, the word Southern was not annotated/predicted as part of the entity, but that's not necessarily an error. 1. The synthetic dataset used here isn't representative of a real dataset. Consider using more realistic datasets for evaluation

branch_plotter.plot_most_common_tokens()

8a. False positives

Most common false positive tokens:

ModelError.most_common_fp_tokens(branch_results.model_errors)

Most common false positive tokens:
[('8', 15),
 ('sunday labor', 13),
 ('greek', 10),
 ('rościsław dudek dobrosław tomaszewski owens duran kathrine filemonsen dds '
  'yuito hirai abby holloway irena bílá',
  9),
 ('margaret s. stouffer kristen rocher panjiva riku andou jeremy hartmann '
  'ástríður steinarsdóttir impi nummelin',
  9),
 ('60s', 8),
 ('greenlander', 8),
 ('couple', 7),
 ('10th', 7),
 ('cyprus', 7)]
---------------
Example sentence with each FP token:
    - 8 + years (`8` pred as DATE_TIME)
    - the last Sunday before Labor Day (`sunday labor` pred as DATE_TIME)
    - Greek (`greek` pred as DEMOGRAPHIC)
    - Rościsław Dudek Dobrosław Tomaszewski Owens Duran Kathrine Filemonsen DDS Yuito Hirai Abby Holloway Irena Bílá (`rościsław dudek dobrosław tomaszewski owens duran kathrine filemonsen dds yuito hirai abby holloway irena bílá` pred as PERSON)
    - Margaret S. Stouffer Kristen Rocher Panjiva Riku Andou Jeremy Hartmann Ástríður Steinarsdóttir Impi Nummelin (`margaret s. stouffer kristen rocher panjiva riku andou jeremy hartmann ástríður steinarsdóttir impi nummelin` pred as PERSON)
    - the 60s (`60s` pred as DATE_TIME)
    - Greenlander (`greenlander` pred as PERSON)
    - a couple of months (`couple` pred as DATE_TIME)
    - 10th year (`10th` pred as DATE_TIME)
    - Cyprus (`cyprus` pred as LOCATION)





[('8', 15),
 ('sunday labor', 13),
 ('greek', 10),
 ('rościsław dudek dobrosław tomaszewski owens duran kathrine filemonsen dds yuito hirai abby holloway irena bílá',
  9),
 ('margaret s. stouffer kristen rocher panjiva riku andou jeremy hartmann ástríður steinarsdóttir impi nummelin',
  9),
 ('60s', 8),
 ('greenlander', 8),
 ('couple', 7),
 ('10th', 7),
 ('cyprus', 7)]

More FP analysis

# Get false positives for PERSON entity
fps_df = ModelError.get_fps_dataframe(branch_results.model_errors, entity="PERSON")
fps_df[["full_text", "token", "annotation", "prediction"]].head(20)

	full_text	token	annotation	prediction
0	J Gelencsér ג€	j gelencsér ג€	O	PERSON
1	Answer:"Tube Snake Boogie	answer:"tube snake boogie	O	PERSON
2	Faina D. Yefremova 's	faina d. yefremova	O	PERSON
3	Sara Schwarz \n	sara schwarz \n	O	PERSON
4	Rua	rua	O	PERSON
5	Verdafero Eklund Michael Hodge	verdafero eklund michael hodge	O	PERSON
6	Verdafero Eklund Michael Hodge	verdafero eklund michael hodge	O	PERSON
7	Király u.	király u.	O	PERSON
8	Kaczmarek Bonifacy Kaczmarek	kaczmarek bonifacy kaczmarek	O	PERSON
9	Kaczmarek Bonifacy Kaczmarek	kaczmarek bonifacy kaczmarek	O	PERSON
10	Kaczmarek Bonifacy Kaczmarek	kaczmarek bonifacy kaczmarek	O	PERSON
11	Yefimov	yefimov	O	PERSON
12	Janka M. Szász	janka m. szász	O	PERSON
13	Jožef Albin Jožef Albin København	jožef albin jožef albin københavn	O	PERSON
14	Jožef Albin Jožef Albin København	jožef albin jožef albin københavn	O	PERSON
15	Flateyri	flateyri	O	PERSON
16	Glyn St	glyn	O	PERSON
17	Marie Langrová	marie langrová	O	PERSON
18	Kyle Kuefer	kyle kuefer	O	PERSON
19	Davis Reynolds	davis reynolds	O	PERSON

8b. False negatives (FN)

Most common false negative examples (should be predicted as entity but wasn't):

ModelError.most_common_fn_tokens(branch_results.model_errors, n=15)

Most common false negative tokens:
[('greenlander', 11),
 ('64', 7),
 ('46', 6),
 ('61', 6),
 ('47', 6),
 ('65', 6),
 ('54', 5),
 ('21', 4),
 ('78', 4),
 ('63', 4),
 ('35', 4),
 ('40', 4),
 ('22', 4),
 ('36', 4),
 ('42', 4)]
---------------
Example sentence with each FN token:
    - Greenlander (`greenlander` annotated as LOCATION)
    - 64 (`64` annotated as DEMOGRAPHIC)
    - 46 (`46` annotated as DEMOGRAPHIC)
    - 61 (`61` annotated as DEMOGRAPHIC)
    - 47 (`47` annotated as DEMOGRAPHIC)
    - 65 (`65` annotated as DEMOGRAPHIC)
    - 54 (`54` annotated as DEMOGRAPHIC)
    - 21 (`21` annotated as DEMOGRAPHIC)
    - 78 (`78` annotated as DEMOGRAPHIC)
    - 63 (`63` annotated as DEMOGRAPHIC)
    - 35 (`35` annotated as DEMOGRAPHIC)
    - 40 (`40` annotated as DEMOGRAPHIC)
    - 22 (`22` annotated as DEMOGRAPHIC)
    - 36 (`36` annotated as DEMOGRAPHIC)
    - 42 (`42` annotated as DEMOGRAPHIC)





[('greenlander', 11),
 ('64', 7),
 ('46', 6),
 ('61', 6),
 ('47', 6),
 ('65', 6),
 ('54', 5),
 ('21', 4),
 ('78', 4),
 ('63', 4),
 ('35', 4),
 ('40', 4),
 ('22', 4),
 ('36', 4),
 ('42', 4)]

More FN analysis

# Get false negatives for PHONE_NUMBER entity
fns_df = ModelError.get_fns_dataframe(branch_results.model_errors, entity="CONTACT")

fns_df[["full_text", "token", "annotation", "prediction"]].head(20)

	full_text	token	annotation	prediction
0	60 - 56 - 85 - 91 SzaszJanka@cuvox.de	60 56 85 91 szaszjanka@cuvox.de	CONTACT	O
1	( 37 ) 788 - 063 063 966	37 788 063 063 966	CONTACT	O
2	0490 75 40 81	0490 75 40 81	CONTACT	O
3	0494 92 82 32	0494 92 82 32	CONTACT	O
4	699 956 915	699 956 915	CONTACT	O
5	( 99 ) 645 - 791	99 645 791	CONTACT	O
6	467 3395 JulietaSouzaAlmeida@cuvox.de	467 3395 julietasouzaalmeida@cuvox.de	CONTACT	O
7	78 651 450	78 651 450	CONTACT	O
8	416 60 039 +46 ( 0)8 928 571 38 +1 - 984 - 182 - 0190	416 60 039 +46 0)8 928 571 38 +1 984 182 0190	CONTACT	O
9	0688 872 49 99	0688 872 49 99	CONTACT	O
10	9472 7916	9472 7916	CONTACT	O
11	21 284 698 2548 JamiePratt@jourrapide.com	21 284 698 2548 jamiepratt@jourrapide.com	CONTACT	O
12	21 253 109 8211 208 815	21 253 109 8211 208 815	CONTACT	O
13	083 564 9312	083 564 9312	CONTACT	O
14	01.84.17.61.18	01.84.17.61.18	CONTACT	O
15	451 5986	451 5986	CONTACT	O
16	99 668472	99 668472	CONTACT	O
17	358 0594 ErikBaader@teleworm.us	358 0594 erikbaader@teleworm.us	CONTACT	O
18	0393 1144137 leonemazzi@rhyta.com	0393 1144137 leonemazzi@rhyta.com	CONTACT	O
19	72 128 827	72 128 827	CONTACT	O