Scribd Logo
Upload

Welcome to Scribd!

  • 50 views

    Chapter 3

    Uploaded by

    david Abotsitse
    The document discusses techniques for scaling data and using K-nearest neighbors (KNN) regression for machine learning models applied to finance data in Python. It covers removing weekday features, using scikit-learn's scaler to standardize features, and making histograms to visualize the effects of scaling. It then discusses implementing a basic neural network using Keras and evaluating its performance on the data. Finally, it covers techniques for addressing overfitting like dropout and ensembling multiple models to improve predictions.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Chapter 3

    Uploaded by

    david Abotsitse
    50 views58 pages
    The document discusses techniques for scaling data and using K-nearest neighbors (KNN) regression for machine learning models applied to finance data in Python. It covers removing weekday features, using scikit-learn's scaler to standardize features, and making histograms to visualize the effects of scaling. It then discusses implementing a basic neural network using Keras and evaluating its performance on the data. Finally, it covers techniques for addressing overfitting like dropout and ensembling multiple models to improve predictions.

    Original Description:

    hjk

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document discusses techniques for scaling data and using K-nearest neighbors (KNN) regression for machine learning models applied to finance data in Python. It covers removing weekday features, using scikit-learn's scaler to standardize features, and making histograms to visualize the effects of scaling. It then discusses implementing a basic neural network using Keras and evaluating its performance on the data. Finally, it covers techniques for addressing overfitting like dropout and ensembling multiple models to improve predictions.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    50 views58 pages

    Chapter 3

    Uploaded by

    david Abotsitse
    The document discusses techniques for scaling data and using K-nearest neighbors (KNN) regression for machine learning models applied to finance data in Python. It covers removing weekday features, using scikit-learn's scaler to standardize features, and making histograms to visualize the effects of scaling. It then discusses implementing a basic neural network using Keras and evaluating its performance on the data. Finally, it covers techniques for addressing overfitting like dropout and ensembling multiple models to improve predictions.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    You are on page 1of 58

    Scaling data and

    KNN Regression
    MACHINE LEARNING FOR FINANCE IN PYTHON

    Nathan George
    Data Science Professor
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Feature selection: remove weekdays
    print(feature_names) print(feature_names[:-4])

    ['10d_close_pct', ['10d_close_pct',
    '14-day SMA', '14-day SMA',
    '14-day RSI', '14-day RSI',
    '200-day SMA', '200-day SMA',
    '200-day RSI', '200-day RSI',
    'Adj_Volume_1d_change', 'Adj_Volume_1d_change',
    'Adj_Volume_1d_change_SMA', 'Adj_Volume_1d_change_SMA']
    'weekday_1',
    'weekday_2',
    'weekday_3',
    'weekday_4']

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Remove weekdays
    train_features = train_features.iloc[:, :-4]
    test_features = test_features.iloc[:, :-4]

    MACHINE LEARNING FOR FINANCE IN PYTHON


    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Scaling options
    Scaling options:

    min-max

    standardization

    median-MAD

    map to arbitrary function (e.g. sigmoid, tanh)

    MACHINE LEARNING FOR FINANCE IN PYTHON


    MACHINE LEARNING FOR FINANCE IN PYTHON
    sklearn's scaler
    from sklearn.preprocessing import scaler

    sc = scaler()
    scaled_train_features = sc.fit_transform(train_features)
    scaled_test_features = sc.transform(test_features)

    MACHINE LEARNING FOR FINANCE IN PYTHON


    MACHINE LEARNING FOR FINANCE IN PYTHON
    Making subplots
    # create figure and list containing axes
    f, ax = plt.subplots(nrows=2, ncols=1)
    # plot histograms of before and after scaling
    train_features.iloc[:, 2].hist(ax=ax[0])
    ax[1].hist(scaled_train_features[:, 2])
    plt.show()

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Scale data and use
    KNN!
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Neural Networks
    MACHINE LEARNING FOR FINANCE IN PYTHON

    Nathan George
    Data Science Professor
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Neural networks have potential
    Neural nets have:

    non-linearity

    variable interactions

    customizability

    MACHINE LEARNING FOR FINANCE IN PYTHON


    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Implementing a neural net with keras
    from keras.models import Sequential
    from keras.layers import Dense

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Implementing a neural net with keras
    from keras.models import Sequential
    from keras.layers import Dense

    model = Sequential()
    model.add(Dense(50,
    input_dim=scaled_train_features.shape[1],
    activation='relu'))
    model.add(Dense(10, activation='relu'))
    model.add(Dense(1, activation='linear'))

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Fitting the model
    model.compile(optimizer='adam', loss='mse')
    history = model.fit(scaled_train_features,
    train_targets,
    epochs=50)

    MACHINE LEARNING FOR FINANCE IN PYTHON


    plt.plot(history.history['loss'])
    plt.title('loss:' + str(round(history.history['loss'][-1], 6)))
    plt.xlabel('epoch')
    plt.ylabel('loss')
    plt.show()

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Checking out performance
    from sklearn.metrics import r2_score

    # calculate R^2 score


    train_preds = model.predict(scaled_train_features)
    print(r2_score(train_targets, train_preds))

    0.4771387560719418

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Plot performance
    # plot predictions vs actual
    plt.scatter(train_preds, train_targets)
    plt.xlabel('predictions')
    plt.ylabel('actual')
    plt.show()

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Make a neural net!
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Custom loss
    functions
    MACHINE LEARNING FOR FINANCE IN PYTHON

    Nathan George
    Data Science Professor
    MACHINE LEARNING FOR FINANCE IN PYTHON
    MSE with directional penalty
    If prediction and target direction match:

    ∑(y − y^)2
    If not:

    ∑(y − y^)2 ∗ penalty

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Implementing custom loss functions
    import tensorflow as tf

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Creating a function
    import tensorflow as tf
    # create loss function
    def mean_squared_error(y_true, y_pred):

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Mean squared error loss
    import tensorflow as tf

    # create loss function


    def mean_squared_error(y_true, y_pred):
    loss = tf.square(y_true - y_pred)
    return tf.reduce_mean(loss, axis=-1)

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Add custom loss to keras
    import tensorflow as tf

    # create loss function


    def mean_squared_error(y_true, y_pred):
    loss = tf.square(y_true - y_pred)
    return tf.reduce_mean(loss, axis=-1)
    # enable use of loss with keras
    import keras.losses
    keras.losses.mean_squared_error = mean_squared_error

    # fit the model with our mse loss function


    model.compile(optimizer='adam', loss=mean_squared_error)
    history = model.fit(scaled_train_features, train_targets, epochs=50)

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Checking for correct direction
    tf.less(y_true * y_pred, 0)

    Correct direction:

    neg * neg = pos

    pos * pos = pos

    Wrong direction:

    neg * pos = neg

    pos * neg = neg

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Using tf.where()
    # create loss function
    def sign_penalty(y_true, y_pred):
    penalty = 10.
    loss = tf.where(tf.less(y_true * y_pred, 0),
    penalty * tf.square(y_true - y_pred),
    tf.square(y_true - y_pred))

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Tying it together
    # create loss function
    def sign_penalty(y_true, y_pred):
    penalty = 100.
    loss = tf.where(tf.less(y_true * y_pred, 0),
    penalty * tf.square(y_true - y_pred),
    tf.square(y_true - y_pred))

    return tf.reduce_mean(loss, axis=-1)


    # enable use of loss with keras
    keras.losses.sign_penalty = sign_penalty

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Using the custom loss
    # create the model
    model = Sequential()
    model.add(Dense(50,
    input_dim=scaled_train_features.shape[1],
    activation='relu'))
    model.add(Dense(10, activation='relu'))
    model.add(Dense(1, activation='linear'))

    # fit the model with our custom 'sign_penalty' loss function


    model.compile(optimizer='adam', loss=sign_penalty)
    history = model.fit(scaled_train_features, train_targets, epochs=50)

    MACHINE LEARNING FOR FINANCE IN PYTHON


    The bow-tie shape
    train_preds = model.predict(scaled_train_features)
    # scatter the predictions vs actual
    plt.scatter(train_preds, train_targets)
    plt.xlabel('predictions')
    plt.ylabel('actual')
    plt.show()

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Create your own
    loss function!
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Overfitting and
    ensembling
    MACHINE LEARNING FOR FINANCE IN PYTHON

    Nathan George
    Data Science Professor
    MACHINE LEARNING FOR FINANCE IN PYTHON
    Simplify your model

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Neural network options
    Options to combat over ing:

    Decrease number of nodes

    Use L1/L2 regulariation

    Dropout

    Autoencoder architecture

    Early stopping

    Adding noise to data

    Max norm constraints

    Ensembling

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Dropout

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Dropout in keras
    from keras.layers import Dense, Dropout
    model = Sequential()
    model.add(Dense(500,
    input_dim=scaled_train_features.shape[1],
    activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(100, activation='relu'))
    model.add(Dense(1, activation='linear'))

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Test set comparison
    R2 values on AMD without dropout:

    train: 0.91

    test: -0.72

    With dropout:

    train: 0.46

    test: -0.22

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Ensembling

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Implementing ensembling
    # make predictions from 2 neural net models
    test_pred1 = model_1.predict(scaled_test_features)
    test_pred2 = model_2.predict(scaled_test_features)
    # horizontally stack predictions and take the average across rows
    test_preds = np.mean(np.hstack((test_pred1, test_pred2)), axis=1)

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Comparing the ensemble
    Model 1 R2 score on test set:

    -0.179

    model 2:

    -0.148

    ensemble (averaged predictions):

    -0.146

    MACHINE LEARNING FOR FINANCE IN PYTHON


    Dropout and
    ensemble!
    MACHINE LEARNING FOR FINANCE IN PYTHON

    You might also like