1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

Get started Open in app

Follow 616K Followers

You have 2 free member-only stories left this month. Sign up for Medium and get an extra one

Photo by Tadas Sar on Unsplash

Exploring Microsoft PowerPoint AI, using

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 1/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

Python
Get started Open in app
Here’s how to replicate the PowerPoint AI using Machine Learning and Python

Piero Paialunga 23 hours ago · 5 min read

A couple of days ago I was working on a PowerPoint presentation for my PhD research
and this happened:

Screenshot made by me. SEE THE ALT TEXT!

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 2/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

It was not the exact same image, it was actually way more explicit, with the x label,y
Get started Open in app
label, title and all of that, but it is not really important right now.

The very interesting thing is the Alt Text. The AI system of PowerPoint is not only able to
detect that we actually have a 2d plot (or Chart) but it recognizes that we are talking
about a boxplot!

Of course, I don’t exactly know how they do this, but as I work with Machine Learning
and Data Science all the days of my life I can try to take a guess. As the readers may
know, the technology that it is very widely used to classify images is known as
Convolutional Neural Networks (CNNs).

They may have used CNNs as a multi-class classifier. Here is an example of a Butterly
image classifier (more than 70 species/classes). A way more complicated thing that they
may have done is image captioning. Nonetheless, CNNs are surely used in their deep
learning algorithm, at the very minimum as basic bricks of something that is much
larger and complex.

In this very small example I will show how it is possible to build a Machine Learning
model that helps you distinguish boxplots and other kinds of plots, for example
lineplots.

Let’s do this.

0. The Libraries
These are the libraries that I used for this notebook:

import keras

import matplotlib.pyplot as plt

import numpy as np

import warnings

warnings.filterwarnings("ignore")

from matplotlib import image

import pandas as pd
import tensorflow as tf

from tensorflow.keras import datasets, layers, models

import numpy as np

import random

from os import listdir

f th i t i fil j i
https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 3/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science
from os.path import isfile, join

import math

Get started Open in app

import matplotlib.pyplot as plt

import seaborn as sns

from sklearn.preprocessing import LabelEncoder

from keras.models import Sequential

from keras.layers import Conv2D

from keras.layers import MaxPooling2D

from keras.layers import Flatten

from keras.layers import Dense

from keras.layers import Dropout

plt.style.use('ggplot')

plt.rcParams['font.family'] = 'sans-serif'

plt.rcParams['font.serif'] = 'Ubuntu'

plt.rcParams['font.monospace'] = 'Ubuntu Mono'

plt.rcParams['font.size'] = 14

plt.rcParams['axes.labelsize'] = 12

plt.rcParams['axes.labelweight'] = 'bold'

plt.rcParams['axes.titlesize'] = 12

plt.rcParams['xtick.labelsize'] = 12

plt.rcParams['ytick.labelsize'] = 12

plt.rcParams['legend.fontsize'] = 12

plt.rcParams['figure.titlesize'] = 12

plt.rcParams['image.cmap'] = 'jet'
plt.rcParams['image.interpolation'] = 'none'

plt rcParams['figure figsize'] = (12 10)

Hosted on Jovian View File

In a few words, I used keras, matplotlib, and a curious library known as

RandomWords that generate random english words. I used it to make up the x and y
axes.

1. Data Generation
The very fun part of this notebook is actually the data generation. I tried to build the
lineplots and boxplots in the most general way as possible, making up the x and y
labels, creating different lines and boxplots, again, in the most general way as
possible.

With this setup that you can virtually create an infinite numbers and kinds of plots. I
created two classes of data and performed a binary classification, but you can slightly
modify the code and create multiple classes.

Let’s dive in:

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 4/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

1.1 Line Plots

Get started Open in app
The code that I used to create the line plot is the following:

def plot_line():

plt.figure(figsize=(10,10))

n_max_line = 10

n_line = np.random.choice(np.arange(1,n_max_line)
x_max,x_min = 10,-10

x_lims = [np.random.choice(np.arange(x_min,0,0.1)
x = np.linspace(min(x_lims),max(x_lims),100)

k_min,k_max = -5,5

for n in range(n_line):

pick_degree = np.random.choice(np.arange(1,5,
y = 0

for degree in range(pick_degree):

k_random = np.random.choice(np.linspace(k
y=y+k_random*x**degree

plt.plot(x,y)

plt.xlabel(r.get_random_word(),fontsize=35)

plt.ylabel(r.get_random_word(),fontsize=35)

#plt.savefig(savename)
Hosted on Jovian View File

It has different degrees of randomness:

The x axis label and y axis label have random names

The x axis limits are random

The y axis show polynomials with random numbers of degrees and random values
of coefficients

The number of lines is random as well

Here is an example:

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 5/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

plot_line()
Get started Open in app

Hosted on Jovian View File

1.2 Box Plots

The code that I used to create the box plot is the following:

def plot_box():
n_max_box = 5

n_box = np.random.choice(np.arange(1,n_max_box))

x_max,x_min = 10,-10

column_names = []

sigma_s = np.arange(1,10,1)

column_values = []

plt.figure(figsize=(10,10))

for n in range(n_box):

x_lims = [np.random.choice(np.arange(x_min,0,
x = np.linspace(min(x_lims),max(x_lims),100)

column_name = r.get_random_word()

column_names.append(column_name)

pick_sigma = x.mean()/np.random.choice(sigma_
pick_sigma = np.abs(pick_sigma)

column_values.append(np.random.normal(x.mean(
column_values = np.array(column_values).T

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 6/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

data = pd.DataFrame(column_values,columns=column_
sns boxplot(data=data)
Get started Open in app
Hosted on Jovian View File

Different degrees of random here as well:

The x axis label and y axis label have random names

The x axis limits are random

The x axis quantities names are random

The y axis show samples from a gaussian distributions with random values of
standard deviation

The number of boxplots is random as well

def plot_box():
n_max_box = 5

n_box = np.random.choice(np.arange(1,n_max_box))

x_max,x_min = 10,-10

column_names = []

sigma_s = np.arange(1,10,1)

column_values = []

plt.figure(figsize=(10,10))

for n in range(n_box):

x_lims = [np.random.choice(np.arange(x_min,0,
x = np.linspace(min(x_lims),max(x_lims),100)

column_name = r.get_random_word()

column_names.append(column_name)

pick_sigma = x.mean()/np.random.choice(sigma_
pick_sigma = np.abs(pick_sigma)

column_values.append(np.random.normal(x.mean(
column_values = np.array(column_values).T

data = pd.DataFrame(column_values,columns=column_
sns boxplot(data=data)
Hosted on Jovian View File
https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 7/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

Get started Open in app

plot box()
Hosted on Jovian View File

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 8/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

1.2 Training Set and Test Set

Get started Open in app
Actually, the codes that I used to build the training set and test set are slightly
differences from the one above, that I was using to show you the results. Here is what
you will need:

Here you create the plots:

def plot_line(savename):

plt.figure(figsize=(10,10))

n_max_line = 10

n_line = np.random.choice(np.arange(1,n_max_line)
x_max,x_min = 10,-10

x_lims = [np.random.choice(np.arange(x_min,0,0.1)
x = np.linspace(min(x_lims),max(x_lims),100)

k_min,k_max = -5,5

for n in range(n_line):

pick_degree = np.random.choice(np.arange(1,5,
y = 0

for degree in range(pick_degree):

k_random = np.random.choice(np.linspace(k
y=y+k_random*x**degree

plt.plot(x,y)

plt.xlabel(r.get_random_word(),fontsize=35)

plt.ylabel(r.get_random_word(),fontsize=35)

plt.savefig(savename)

#plt.show()
plt.close()

def plot_box(savename):

n_max_box = 5

n_box = np.random.choice(np.arange(1,n_max_box))

x_max,x_min = 10,-10

column_names = []

sigma_s = np.arange(1,10,1)

l l []
Hosted on Jovian View File

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 9/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

Get started Open in app

Here you create k of them and store them. CREATE A TRAINING SET AND TEST
SET FOLDER FIRST OR IT WON’T WORK!

def build_training_set(num):

mypath = 'TrainingSet/'

for n in range(1,num+1):

print('%i instance has been started'%(n))

plot_box(mypath+'boxplot_'+str(n)+'.png')

print('Boxplot %i has been stored!'%(n))

plot line(mypath+'lineplot '+str(n)+' png')

Hosted on Jovian View File

Here you read them and label them

def extract_training_set():

mypath = 'TrainingSet/'

onlyfiles = [f for f in listdir(mypath) if isfile

training_set_arrays = []

training_set_labels = []

for file in onlyfiles:

split_file = file.split('.')

if split_file[-1]=='png':

training_set_labels.append(split_file[0].

Hosted on Jovian View File

After you define this function, you will have your dataset by doing this:

build_training_set(50)

1 instance has been started

Boxplot 1 has been stored!

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 10/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

Lineplot 1 has been stored!

Get started
2 instance Open
has in app
been started

Boxplot 2 has been stored!

Lineplot 2 has been stored!

3 instance has been started

Boxplot 3 has been stored!

Lineplot 3 has been stored!

4 instance has been started

Boxplot 4 has been stored!

Lineplot 4 has been stored!

5 instance has been started

Boxplot 5 has been stored!

Lineplot 5 has been stored!

6 instance has been started

Boxplot 6 has been stored!

Lineplot 6 has been stored!

7 instance has been started

Boxplot 7 has been stored!

Lineplot 7 has been stored!

Hosted on Jovian View File

Here are some examples of the training set:

plt.figure(figsize=(32,32))

for i in range(9):

plt.subplot(3,3,i+1)

plt.xticks([])

plt.yticks([])

plt.grid(False)

x=np.random.randint(len(X_train))

plt.imshow(X_train[x], cmap=plt.cm.binary)

plt.xlabel(labels_train[x], fontsize=60)

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 11/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science
p ( _ [ ], )
plt.show()
Get started Open in app

Hosted on Jovian View File

The exact same process has to be done for the test set and the strings has to be converted
to something more readable to a ML model (sklearn will do this for you with the so
called LabelEncoder feature):

def build_test_set(num):

mypath = 'TestSet/'

for n in range(1,num+1):

print('%i instance has been started'%(n))

plot_box(mypath+'boxplot_'+str(n)+'.png')

print('Boxplot %i has been stored!'%(n))

plot_line(mypath+'lineplot_'+str(n)+'.png')

print('Lineplot %i has been stored!'%(n))

build_test_set(10)
https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 12/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

1 instance
Get started has been
Open started

in app
Boxplot 1 has been stored!

Lineplot 1 has been stored!

2 instance has been started

Boxplot 2 has been stored!

Lineplot 2 has been stored!

3 instance has been started

Boxplot 3 has been stored!

Lineplot 3 has been stored!

4 instance has been started

Boxplot 4 has been stored!

Lineplot 4 has been stored!

5 i t h b t t d
Hosted on Jovian View File

2. Machine Learning Model

The Machine Learning model that we are going to use is basically the application of
different Convolutional layers and some Max Pooling operations, it will then end up
with a softmax that will tell you the probability of the image to belong to the first class.

The model that I used was the same of this article I published and you can find more
details about how the structure actually works.

size = X_train[0].shape[0]

classifier = Sequential()

# Step 1 - Convolution

classifier.add(Conv2D(3, (3, 3), input_shape = (size,

classifier.add(MaxPooling2D(pool_size = (2, 2)))

l ifi dd(C 2D(3 (3 3) i t h

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4
( i 13/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science
classifier.add(Conv2D(3, (3, 3), input_shape = (size,
Get started Open in app
classifier.add(MaxPooling2D(pool_size = (2, 2)))

classifier.add(Flatten())

#classifier.add(Dense(units = 32, activation = 'relu'

classifier.add(Dense(units = 1, activation = 'sigmoid

# Compiling the CNN

classifier.summary()

ERROR! Session/line number was not unique in database.

logging moved to new session 87

Model: "sequential_4"

______________________________________________________
Layer (type) Output Shape
======================================================
conv2d 7 (Conv2D) (None 718 718 3)
Hosted on Jovian View File

Here is how you train and test your model:

train_images, test_images = X_train,X_test

train_images=np.array(train_images)

test_images=np.array(test_images)

train_images, test_images = train_images / 255.0, tes

Train_images=[]

Test_images=[]

for i in range(len(train_images)):

a=train_images[i].reshape(size,size,3)

Train_images.append(a)

Train_images=np.array(Train_images)

for j in range(len(test_images)):

b=test_images[j].reshape(size,size,3)

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 14/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

Test_images.append(b)

Test_images=np.array(Test_images)

Get started Open in app

train_images,test_images=Train_images, Test_images

classifier.compile(optimizer = 'adam', loss = 'binary

history = classifier.fit(train_images, train_labels,
validation_data=(test_images, test_la

Train on 100 samples, validate on 20 samples

Epoch 1/10
100/100 [==============================] - 12s
122ms/step - loss: 15.4656 - accuracy: 0.5200 -
val_loss: 13.8420 - val_accuracy: 0.5000

Epoch 2/10
Hosted on Jovian View File

And as we can see, the final result is perfect. Even if it may sounds exciting, I have to
say that the experiment is pretty easy (we are all able to distinguish a plot with box
and a plot with lines) and the model is more than sufficiently powerful (a little bit of
overkill here).

3. Final Results
As a final prove that the model is correctly distinguish boxplots and lineplots, here are
some examples:

y_pred = classifier.predict(X_test).astype(int)

y_pred_string = le.inverse_transform(y_pred)

y_pred_string

array(['lineplot', 'boxplot', 'boxplot', 'lineplot',

'lineplot'
Hosted on Jovian View File

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 15/18
1/25/22, 2:16 PM Exploring Microsoft PowerPoint AI, using Python | by Piero Paialunga | Jan, 2022 | Towards Data Science

Get started Open in app

And here are the plots:

plt.figure(figsize=(32,32))

for i in range(9):

plt.subplot(3,3,i+1)

plt.xticks([])

plt.yticks([])

plt.grid(False)

x=np.random.randint(len(X_test))

plt.imshow(X_test[x], cmap=plt.cm.binary)

plt.title('Predicted label: "%s", Real label: "%s

plt.show()

Hosted on Jovian View File

https://towardsdatascience.com/exploring-powerpoint-ai-using-python-75f94d55f8f4 16/18