Skip to content

3 Split by pattern number

Train/Test/Validation split of input samples.

This notebook shows how train/test/split is being made on a List[InputSample]

This is different from the normal split since we don't want sentences generated from the same pattern to be in more than one set. (Applicable only if the dataset was generated from templates)

# install presidio via pip if not yet installed

#!pip install presidio-analyzer
#!pip install presidio-evaluator

from datetime import date

from presidio_evaluator import InputSample
from presidio_evaluator.validation import save_to_json, split_dataset

%reload_ext autoreload

Load full dataset

all_samples = InputSample.read_dataset_json("../data/synth_dataset_v2.json")
print(len(all_samples))

Split to train/test/dev

TRAIN_TEST_VAL_RATIOS = [0.7, 0.2, 0.1]

train, test, validation = split_dataset(all_samples, TRAIN_TEST_VAL_RATIOS)

Train/Test only (no validation)

# TRAIN_TEST_RATIOS = [0.7,0.3]
# train,test = split_dataset(all_sampleTRAIN_TEST_RATIOSEST_RATIOS)

Save the different sets to files

DATE_DATE = date.today().strftime("%b-%d-%Y")

save_to_json(train, f"../data/train_{DATE_DATE}.json")
save_to_json(test, f"../data/test_{DATE_DATE}.json")
save_to_json(validation, f"../data/validation_{DATE_DATE}.json")

print(len(train))

print(len(test))

print(len(validation))

assert len(train) + len(test) + len(validation) == len(all_samples)