Testing in Machine Learning

• If this is your first time working with data, then this also might be the first time when you are looking at such a large data set maybe 100k lines(or more) of json records. In order to read and understand the data set, you might want to familiarize yourself with Jupyter notebooks and pandas library.

# To view the test data, columns and values of records in the Jupyter notebook and see the output of each command.

import pandas as pd

test_df = pd.read_json('path/to/raw_data.jsonl', lines=True)

test_df.head()

• You can refer the pandas cheat sheet here.
• Now, that you are comfortable using the pandas library and would like to move to writing some data validation tests. Great expectations library helps you to write pre train tests with some data validations.

From the Great Expectation documentation, here is what it says:

With Great Expectations, you can assert what you expect from the data you load and transform, and catch data issues quickly — Expectations are basically unit tests for your data.

In brief, the kind of data validations you might like to add:

– expect_column_to_exist

– expect_column_values_to_be_unique

– expect_column_values_to_not_be_null

– expect_column_values_to_match_regex

– you can refer the full list of expectations here.

A basic script with minimal validations using Great Expectations library might look like:

#validations_get_dataset.py

from pathlib import Path

import pandas as pd

import great_expectations as ge

DATA=’/path/to/test_data’

assert Path(DATA).exists()

raw_data = pd.read_json(f’{DATA}/raw/raw_data.jsonl’, lines=True)

print(‘\n — Columns from raw data: — \n’, raw_data.columns)

ge_raw = ge.from_pandas(raw_data)

expected_columns = [

‘description’,

‘serial_number’,

‘Product’]
result = ge_raw.expect_table_columns_to_match_set(expected_columns,     exact_match = True, result_format={‘result_format’: ‘BASIC’}, include_config=True, catch_exceptions=True)

assert result[‘success’]

To add the pre-train tests to your pipeline, you might want to understand the existing ML pipeline, in my case I was introduced to dvc.yaml which consists of all stages involved in a ML pipeline specified in a yaml format.

In short, a dvc.yaml is a file that contains all the individual stages of a machine learning pipeline starting from data ingestion to training to evaluation. For better understanding, please read more about DVC and dvc.yaml.

How to add data validation stages to the existing dvc.yaml?

#dvc.yaml

#first stage: data ingestion, second stage: data validationstages:

 get_dataset:

   cmd: wget /path/to/raw_data.jsonl -P ./data/raw

   outs:

   - ./data/raw/raw_test_data.jsonl

 validate_get_dataset:

   cmd: python ./data_validations/validations_get_dataset.py

   deps:

     - ./data/raw/raw_test_data.jsonl

     - ./data_validations/validations_get_dataset.py

:

woah! You just finished adding pre-train tests to your ML pipeline. The next time your pipeline is run for retraining the model, your data validation checks will be in place to make sure that the training data is as expected.

Post train tests require a deeper understanding of a few concepts, which are summarized in brief(in order) below:

• There are three types of Post train tests: Minimum functionality tests, Invariance tests and Directional Expectation tests. You can read more about these here.
• In order to get started with Minimum functionality tests and be able to check for the consistency of the results, we need to introduce ourselves to the confusion matrix and the classification report(recall, precision and f1 score).
• Once you have a decent understanding of each of these metrics, you might want to proceed to get these numbers for your data. In Machine Learning, we usually split our data set into two subsets: training data and testing data which we call as Test/Train split.
• Next up we want to see the numbers as a part of the classification report, for which you might want to read up here.
• Every time you train the model, you can make sure that the numbers do not drop. These tests can be included as a part of the ML pipeline the same way as pre-train tests.
• For more visual representation of the classification report, you can use the library matplotib.

I hope the information here helps you in getting started with Testing in Machine Learning. However, I would be interested to know how you test data/ML pipeline in your project? Let me know in the comments.

PS: I also write about life experiences in general and share travel stories on YouTube.