Gradient boosted
trees with XGBoost
CREDIT RIS K MODELIN G IN P YTH ON
Michael Crabtree
Data Scientist, Ford Motor Company
Decision trees
Creates predictions similar to logistic regression
Not structured like a regression
CREDIT RISK MODELING IN PYTHON
Decision trees for loan status
Simple decision tree for predicting loan_status probability of default
CREDIT RISK MODELING IN PYTHON
Decision tree impact
Loan True loan status Pred. Loan Status Loan payoff value Selling Value Gain/Loss
1 0 1 $1,500 $250 -$1,250
2 0 1 $1,200 $250 -$950
CREDIT RISK MODELING IN PYTHON
A forest of trees
XGBoost uses many simplistic trees (ensemble)
Each tree will be slightly better than a coin toss
CREDIT RISK MODELING IN PYTHON
Creating and training trees
Part of the xgboost Python package, called xgb here
Trains with .fit() just like the logistic regression model
# Create a logistic regression model
clf_logistic = LogisticRegression()
# Train the logistic regression
clf_logistic.fit(X_train, np.ravel(y_train))
# Create a gradient boosted tree model
clf_gbt = xgb.XGBClassifier()
# Train the gradient boosted tree
clf_gbt.fit(X_train,np.ravel(y_train))
CREDIT RISK MODELING IN PYTHON
Default predictions with XGBoost
Predicts with both .predict() and .predict_proba()
.predict_proba() produces a value between 0 and 1
.predict() produces a 1 or 0 for loan_status
# Predict probabilities of default
gbt_preds_prob = clf_gbt.predict_proba(X_test)
# Predict loan_status as a 1 or 0
gbt_preds = clf_gbt.predict(X_test)
# gbt_preds_prob
array([[0.059, 0.940], [0.121, 0.989]])
# gbt_preds
array([1, 1, 0...])
CREDIT RISK MODELING IN PYTHON
Hyperparameters of gradient boosted trees
Hyperparameters: model parameters (settings) that cannot be learned from data
Some common hyperparameters for gradient boosted trees
learning_rate : smaller values make each step more conservative
max_depth : sets how deep each tree can go, larger means more complex
xgb.XGBClassifier(learning_rate = 0.2,
max_depth = 4)
CREDIT RISK MODELING IN PYTHON
Let's practice!
CREDIT RIS K MODELIN G IN P YTH ON
Column selection for
credit risk
CREDIT RIS K MODELIN G IN P YTH ON
Michael Crabtree
Data Scientist, Ford Motor Company
Choosing speci c columns
We've been using all columns for predictions
# Selects a few specific columns
X_multi = cr_loan_prep[['loan_int_rate','person_emp_length']]
# Selects all data except loan_status
X = cr_loan_prep.drop('loan_status', axis = 1)
How you can tell how important each column is
Logistic Regression: column coef cients
Gradient Boosted Trees: ?
CREDIT RISK MODELING IN PYTHON
Column importances
Use the .get_booster() and .get_score() methods
Weight: the number of times the column appears in all trees
# Train the model
clf_gbt.fit(X_train,np.ravel(y_train))
# Print the feature importances
clf_gbt.get_booster().get_score(importance_type = 'weight')
{'person_home_ownership_RENT': 1, 'person_home_ownership_OWN': 2}
CREDIT RISK MODELING IN PYTHON
Column importance interpretation
# Column importances from importance_type = 'weight'
{'person_home_ownership_RENT': 1, 'person_home_ownership_OWN': 2}
CREDIT RISK MODELING IN PYTHON
Plotting column importances
Use the plot_importance() function
xgb.plot_importance(clf_gbt, importance_type = 'weight')
{'person_income': 315, 'loan_int_rate': 195, 'loan_percent_income': 146}
CREDIT RISK MODELING IN PYTHON
Choosing training columns
Column importance is used to sometimes decide which columns to use for training
Different sets affect the performance of the models
Model Model Default
Columns Importances
Accuracy Recall
loan_int_rate, person_emp_length (100, 100) 0.81 0.67
loan_int_rate, person_emp_length,
(98, 70, 5) 0.84 0.52
loan_percent_income
CREDIT RISK MODELING IN PYTHON
F1 scoring for models
Thinking about accuracy and recall for different column groups is time consuming
F1 score is a single metric used to look at both accuracy and recall
Shows up as a part of the classification_report()
CREDIT RISK MODELING IN PYTHON
Let's practice!
CREDIT RIS K MODELIN G IN P YTH ON
Cross validation for
credit models
CREDIT RIS K MODELIN G IN P YTH ON
Michael Crabtree
Data Scientist, Ford Motor Company
Cross validation basics
Used to train and test the model in a way that simulates using the model on new data
Segments training data into different pieces to estimate future performance
Uses DMatrix , an internal structure optimized for XGBoost
Early stopping tells cross validation to stop after a scoring metric has not improved after a number of
iterations
CREDIT RISK MODELING IN PYTHON
How cross validation works
Processes parts of training data as (called folds) and tests against unused part
Final testing against the actual test set
1 https://scikit 2 learn.org/stable/modules/cross_validation.html
CREDIT RISK MODELING IN PYTHON
Setting up cross validation within XGBoost
# Set the number of folds
n_folds = 2
# Set early stopping number
early_stop = 5
# Set any specific parameters for cross validation
params = {'objective': 'binary:logistic',
'seed': 99, 'eval_metric':'auc'}
'binary':'logistic' is used to specify classi cation for loan_status
'eval_metric':'auc' tells XGBoost to score the model's performance on AUC
CREDIT RISK MODELING IN PYTHON
Using cross validation within XGBoost
# Restructure the train data for xgboost
DTrain = xgb.DMatrix(X_train, label = y_train)
# Perform cross validation
xgb.cv(params, DTrain, num_boost_round = 5, nfold=n_folds,
early_stopping_rounds=early_stop)
DMatrix() creates a special object for xgboost optimized for training
CREDIT RISK MODELING IN PYTHON
The results of cross validation
Creates a data frame of the values from the cross validation
CREDIT RISK MODELING IN PYTHON
Cross validation scoring
Uses cross validation and scoring metrics with cross_val_score() function in scikit-learn
# Import the module
from sklearn.model_selection import cross_val_score
# Create a gbt model
xg = xgb.XGBClassifier(learning_rate = 0.4, max_depth = 10)
# Use cross valudation and accuracy scores 5 consecutive times
cross_val_score(gbt, X_train, y_train, cv = 5)
array([0.92748092, 0.92575308, 0.93975392, 0.93378608, 0.93336163])
CREDIT RISK MODELING IN PYTHON
Let's practice!
CREDIT RIS K MODELIN G IN P YTH ON
Class imbalance in
loan data
CREDIT RIS K MODELIN G IN P YTH ON
Michael Crabtree
Data Scientist, Ford Motor Company
Not enough defaults in the data
The values of loan_status are the classes
Non-default: 0
Default: 1
y_train['loan_status'].value_counts()
loan_status Training Data Count Percentage of Total
0 13,798 78%
1 3,877 22%
CREDIT RISK MODELING IN PYTHON
Model loss function
Gradient Boosted Trees in xgboost use a loss function of log-loss
The goal is to minimize this value
True loan status Predicted probability Log Loss
1 0.1 2.3
0 0.9 2.3
An inaccurately predicted default has more negative nancial impact
CREDIT RISK MODELING IN PYTHON
The cost of imbalance
A false negative (default predicted as non-default) is much more costly
Person Loan Amount Potential Pro t Predicted Status Actual Status Losses
A $1,000 $10 Default Non-Default -$10
B $1,000 $10 Non-Default Default -$1,000
Log-loss for the model is the same for both, our actual losses is not
CREDIT RISK MODELING IN PYTHON
Causes of imbalance
Data problems
Credit data was not sampled correctly
Data storage problems
Business processes:
Measures already in place to not accept probable defaults
Probable defaults are quickly sold to other rms
Behavioral factors:
Normally, people do not default on their loans
The less often they default, the higher their credit rating
CREDIT RISK MODELING IN PYTHON
Dealing with class imbalance
Several ways to deal with class imbalance in data
Method Pros Cons
Gather more data Increases number of defaults Percentage of defaults may not change
Penalize models Increases recall for defaults Model requires more tuning and maintenance
Sample data differently Least technical adjustment Fewer defaults in data
CREDIT RISK MODELING IN PYTHON
Undersampling strategy
Combine smaller random sample of non-defaults with defaults
CREDIT RISK MODELING IN PYTHON
Combining the split data sets
Test and training set must be put back together
Create two new sets based on actual loan_status
# Concat the training sets
X_y_train = pd.concat([X_train.reset_index(drop = True),
y_train.reset_index(drop = True)], axis = 1)
# Get the counts of defaults and non-defaults
count_nondefault, count_default = X_y_train['loan_status'].value_counts()
# Separate nondefaults and defaults
nondefaults = X_y_train[X_y_train['loan_status'] == 0]
defaults = X_y_train[X_y_train['loan_status'] == 1]
CREDIT RISK MODELING IN PYTHON
Undersampling the non-defaults
Randomly sample data set of non-defaults
Concatenate with data set of defaults
# Undersample the non-defaults using sample() in pandas
nondefaults_under = nondefaults.sample(count_default)
# Concat the undersampled non-defaults with the defaults
X_y_train_under = pd.concat([nondefaults_under.reset_index(drop = True),
defaults.reset_index(drop = True)], axis=0)
CREDIT RISK MODELING IN PYTHON
Let's practice!
CREDIT RIS K MODELIN G IN P YTH ON
