When you build and deploy a prediction model in Graphext, you'll be able to inspect how it was built and how it performed using the Models panel of your project.

Inside you'll find three tabs - General - Training - Result - each containing information about your model. This information helps you unpick how your model was built, its strengths and weaknesses as well as how it performed in specific areas. But there are a lot of technical terms at play that are important to get your head around first. We'll walk through them here ...

‍

‍

To build a prediction model, choose Models → Train & Predict as your analysis type. Read up on building prediction models in Graphext in our docs article here.

‍

A Brief Introduction

Test & Train

Before a prediction model can be used to predict your target variable, it first needs to be trained. Training here means using data to learn which values of the model's parameters generate predictions that best match known labels (i.e. the values of the target variable).

In Graphext, we use all labelled rows in your dataset to train the model (i.e. rows where the target variable doesn't have missing values). The whole story is somewhat more complex, however. If we did indeed use all data in the training process, we would create a model that is very good at predicting known data, but we wouldn't have any way to ensure it can also predict data it hasn't seen before.

‍

“It takes characteristics of the individual as input and provides a predictive score as output. The higher the score, the more likely it is that the individual will exhibit the predicted behaviour.”

- Eric Siegel

‍

For that reason, the training procedure can be configured such as to use parts of the labelled data to train the model, and reserve other parts of the data to evaluate the model's performance on new data. These are usually referred to as train and test splits of the data. The performance of the model on the test split(s) provides an estimate of the performance you can expect when using the model to predict new data later on.

‍

We wrote more about training models in this docs article.

‍

Why Evaluate Models

Evaluating a prediction model helps us understand how good the model is at predicting the target variable given some data it hasn't seen before. Using accuracy scores and other performance metrics, we get a sense of whether the model is able to make correct or inaccurate predictions. Not only this, but through evaluation of a model, we can understand how to improve it by changing its factors - (i.e. the features, or input variables used) - or parameters.

Note that our recipes and flows allow you to use evaluation metrics to automatically find the best parameters of the model, a process usually called model tuning - or hyperparameter optimization. For example, when training a decision tree, instead of specifying how big (deep) the tree is allowed to be, you can use tuning to automatically find the best depth automatically.

‍

See these articles for more details.

• Scikit Learn | Tuning the hyper-parameters of an estimator
• Wikipedia | Hyperparameter optimization

‍

3 Tabs: General - Training - Result

Inside the Models panel of your project, there are 3 tabs; General - Training - Result. Each tab contains information about different aspects of your model. Open and close these tabs by clicking on them.

‍

General

The General tab contains meta-information about the model alongside details of the open-source code used to develop the model. You'll also find a description of the model and notes from our team of data scientists and engineers regarding its use.

‍

Training

The Training tab documents the parameters and methodology used to train the model (e.g. how its performance was evaluated and if or how the model was tuned). The parameters will vary depending on the type of model you are using. A decision tree will have different parameters than linear regression.

The Validation field explains how, and how many times, the labelled data was split into train and test sets, to fit the model's parameters and evaluate its performance respectively.

‍

A decision tree will have different parameters than linear regression.

‍

If the model was tuned, a corresponding field will indicate which parameters were optimized, and which range of the values was explored to find the best model. Finally, the training tab summarizes which variables were used as factors (features) and target (labels), along with their data type. When using the model to predict new data, the new data must have variables with the same name and type as indicated here.

‍

Result

The Result tab reveals information about the performance of your model. As well as the parameters used by your final model (important if these were found automatically by tuning), you'll find metrics with which to evaluate its accuracy and performance.

The metrics shown here will be the ones you selected when you trained the model, or which Graphext has selected for you automatically as appropriate for the prediction problem - for instance, classification or regression.

‍

In the Results tab you'll find the parameters used by your final model as well as metrics with which to evaluate its accuracy and performance.

‍

The metrics in the train section measure the model's performance on data it has seen during training (easy, but informative only in so far as a low score here indicates that the selected model is too weak to learn from the provided data, or that the data doesn't contain sufficient information about the target variable). The metrics in the test section report performance of the model on data not seen during training, which is an estimate of the future performance of your model.

Of particular interest here, usually, is the spread between training and test errors. Good performance on the training data, but poor performance on test data may indicate that the model has been overfitted, i.e. the model has started to "memorize" irrelevant details of the training data, instead of focusing on more general structures in the data that would allow it to predict new data equally well. If this is the case, consider using the tuning options to let the model select parameters that don't lead to overfitting. Alternatively, changing the model's parameters yourself, you could try to restrict it to be less "powerful" (e.g. making a decision tree smaller).

If the model performs poorly on both training and test data, in contrast, it may mean that the model has been underfitted. In this case, you may consider again using tuning to find parameters that make the model more "powerful" (e.g. exploring a greater range of decision tree sizes), manually changing the parameters with the same goal, or providing more data for the model to learn from.

‍

General

Inside the General tab, you'll find information detailing the technology used to develop the model as well as basic information and notes describing its function and how it works.

‍

Name

The name of the kind of model trained.

‍

Path

How to refer to your model in recipes when using it again for prediction tasks.

‍

Task

The intended function or purpose of the model.

‍

Library

A link to the documentation and open-source code used to develop the model.

‍

Date Trained

The date that the model was initially created. This will often be the same as the date you created the project.

‍

Description

A more detailed explanation of the development of the model and the mechanics behind it.

‍

Notes

Tips from the data science team at Graphext on using the model as well as their thoughts on the performance of this type of model.

‍

Training

Inside the Training tab, you'll find the parameters used to train the model alongside details of it's target and factors.

‍

Parameters

Parameters control the way that a model performs. You customize a model's parameters by answering questions when you setup the model using the Graphext project setup wizard. The values you set for each parameter get passed into the model when you execute your project.

The parameters passed to a model depend on the type of model used. Graphext currently supports two model types for each prediction task.

‍

Classification

• Catboost Classifier
• Logistic Regression

Regression

• Catboost Regressor
• Linear Regression

To understand more about the parameters of your model - how to use them and what they mean - check out our documentation here.

‍

Validation

The Validation field explains how, and how many times, the labelled data was split into train and test sets, to fit the model's parameters and evaluate its performance respectively.

‍

Data

The information displayed about your data inside the Training tab shows the factors and target used by your model.

‍

rows

The number of rows in your dataset.

‍

features

The variables passed as factors to your model. These are used to calculate relationships to your target variable.

‍

target

The variable that your model has made predictions about. The relationships of a target to a model's factors define how a model calculates a prediction.

‍

Result

Inside the Result tab, you'll find information on evaluating the performance of your model. Metrics to do this are presented here alongside the calculated importance of each of your model's factors.

‍

Parameters

To understand more about the parameters of your model - how to use them and what they mean - check out our documentation here.

‍

Metrics

The metrics shown in the results tab represent evaluations of the model's predictive performance. All metrics are displayed as a number between 0 and 1. Higher numbers represent more accurate (better) performances.

The number that follows the  +/-  beside a metric represents the standard deviation of that metric.

‍

Test

accuracy

A model's test accuracy is a number indicating the number of classifications that a model correctly predicted divided by the total number of predictions made - when considering the test dataset.

‍

Train

accuracy

A model's training accuracy is a number indicating the number of classifications that a model correctly predicted divided by the total number of predictions made - when considering the training dataset.

‍

recall_weighted

Recall is the ability of the classifier to find all the positive samples. It is calculated using the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives.

‍

precision_weighted

Precision is the ability of the classifier not to label as positive a sample that is negative. It is calculated using the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives.

‍

f1_weighted

The F-Score is another measure of a model's accuracy. It is defined as the harmonic mean of the model's recall and precision metrics.

‍

Feature Importance

Feature importance is a score given to each factor used to calculate a model's predictions. It represents the 'usefulness' of a feature in supporting the model in making predictions. Values are assigned between 0 and 1 again with higher values indicating stronger feature importance.

‍