Numerical Machine Learning

PREFACE

In recent years, machine learning has become increasingly popular and pervasive, with applications ranging from self-driving cars and facial recognition to personalized website recommendations and stock market forecasting. The increased availability of data and advancements in computer power have made it possible to apply machine learning algorithms to a vast array of problems with impressive outcomes. Machine learning is currently utilized in a variety of areas, including banking, healthcare, marketing, and manufacturing, and it is anticipated that it will continue to play a significant role in the development of new technologies in the future. Consequently, machine learning has emerged as an essential subject of study for people interested in data science, artificial intelligence, and related fields. As machine learning continues to evolve and expand its reach, researchers and practitioners are constantly developing new techniques and algorithms to address specific challenges or improve upon existing methods. In this ever-changing landscape, it is crucial for those working in the field to stay up-to-date with the latest advancements and trends. This includes not only mastering the fundamental concepts and algorithms, but also understanding how to adapt and apply them in novel ways to solve real-world problems. By embracing the interdisciplinary nature of machine learning, and collaborating with experts from diverse fields, we can accelerate the development of innovative solutions that have the potential to transform industries, enhance the quality of life, and create a more sustainable future for all.

From our experiences of teaching machine learning using various textbooks, we have noticed that there tends to be a strong emphasis on abstract mathematics when discussing the theories of machine learning algorithms. On the other hand, in the application of machine learning, it usually straightaway goes to import offthe- shelf libraries such as scikit-learn, TensorFlow, Keras, and PyTorch. The disconnect between abstract mathematical theories and practical application creates a gap in understanding. This book bridges the gap using numerical examples with small datasets and simple Python codes to provide a complete walkthrough of the underlying mathematical steps of machine learning algorithms. By working through concrete examples step by step, readers/students can develop a well-rounded understanding of these algorithms, gain a more indepth knowledge of how mathematics relates to the implementation and performance of the algorithms, and be better equipped to apply them to practical problems.

Beginning with an introduction to machine learning in Chapter 1, the remaining chapters of the book cover seven commonly used machine learning algorithms and techniques, including both supervised and unsupervised learning, as well as both linear and nonlinear models. The book requires some prerequisite knowledge of basic probability and statistics, linear algebra, calculus, and Python programming. The book is intended for university students studying machine learning and is used as our primary teaching material for the Introduction to Machine Learning module at DigiPen Institute of Technology Singapore.

In conclusion, we would like to acknowledge Mr. Tan Chek Ming (Managing Director), Prof. Prasanna Ghali (Provost), Ms. Caroline Tan (Deputy Director), Ms. Angela Tay (Senior Manager), and all at DigiPen Institute of Technology Singapore, for their consistent support and help. We also wish to thank a number of our students (including Nelson Ng, Rhonda McGladdery, Farhan Fadzil, Lim Li Jia, Musa Ahmad Dahlan, Jeremy Yap, and Seah Jue Chen) for their diligence in spotting several typographical errors during their course of studies. Also, it has been a delight working with Bentham's professional editorial and production staff. We particularly thank Noor Ul Ain Khan, Humaira Hashmi, and Obaid Sadiq for their consistent, timely, and kind support throughout the development of this book. Furthermore, we extend our heartfelt appreciation to our families (including Xiaoyue Cui, Muyuan Wang, Safura Tazeen, Khasim BI, Shirleen Chow, Adler Teoh, Hriday Bhoyar, Swati Kolkhede, and all) for their unwavering encouragement throughout the creation of this book. We dedicate this book to them. The first author, Zhiyuan Wang, would also like to convey special thanks and appreciation to his Ph.D. advisors, Prof. Zhe Wu, Prof. Xiaonan Wang, and Prof. Gade Pandu Rangaiah from the National University of Singapore. Although they were not involved in this book, Zhiyuan deeply cherishes their sincere and invaluable guidance in his Ph.D. journey, which has helped him become a better researcher and educator.

Despite our best efforts to ensure the accuracy of the content within this book, errors may inadvertently persist. If you come across any inaccuracies or omissions, we kindly request that you bring them to our attention by emailing us at wangzhiyuan@u.nus.edu. We are committed to rectifying such oversights in future editions and will post corrections on our shared https://drive.google.com/drive/folders/1FqJvo4ZPazNbEH_GlHFoodqvegnQmHc n?usp=share_link

Zhiyuan Wang

DigiPen Institute of Technology Singapore Singapore

National University of Singapore Singapore

Sayed Ameenuddin Irfan

DigiPen Institute of Technology Singapore Singapore

National University of Singapore Singapore

Zhiyuan Wang

DigiPen Institute of Technology Singapore Singapore

Christopher Teoh

DigiPen Institute of Technology Singapore Singapore

&

Priyanka Hriday Bhoyar

DigiPen Institute of Technology Singapore Singapore

Introduction to Machine Learning

Zhiyuan Wang, Sayed Ameenuddin Irfan, Christopher Teoh, Priyanka Hriday Bhoyar

Abstract

Machine learning, a rapidly growing subfield of computer science, has had a significant impact on many industries and our lives. This chapter discusses the brief history of machine learning, its widespread adoption as a de facto feature, and fundamental concepts such as supervised and unsupervised learning, regression and classification, and underfitting and overfitting. We also emphasize the importance of understanding machine learning through numerical examples, which can bridge the gap between abstract mathematical theories and practical applications of machine learning algorithms. By developing a strong foundation in machine learning, readers/students can harness its potential to address challenges and opportunities across diverse sectors.

Keywords: Numerical Examples, Machine Learning History, Supervised Learning, Unsupervised Learning, Regression, Classification, Underfitting, Overfitting.

1.1. BRIEF HISTORY OF MACHINE LEARNING

Machine learning is a subfield of computer science that involves the creation of algorithms that can learn from data and make predictions. It has a long and rich history [1], with roots dating back to the 1950s when the field of artificial intelligence was founded. This field focused on developing machines that could perform tasks that typically require human-like intelligence, such as recognizing patterns, learning from experience, and making decisions. The first machine learning algorithms were developed in the 1960s, including decision tree and nearest neighbor algorithms. The 1980s saw the rapid growth of the field with the development of algorithms such as artificial neural network and support vector machine. These algorithms were applied to a wide range of applications in the 1990s, including natural language processing, computer vision, and speech recognition. In the 2000s, the field continued to evolve with the development of new algorithms, such as gradient boosting, and the increasing use of machine learning in industries such as finance and healthcare. The 2010s saw the widespread adoption of machine learning, aided by the advent of big data and the development of powerful graphics processing units (GPU) that could be used to train large and complex machine learning models. The subfield of deep learning [2], which typically involves the use of multi-layered neural networks, became particularly popular and found application across a diverse range of domains. Today, machine learning is a rapidly growing field that is currently being applied in various sectors. It has the potential to revolutionize many industries and has already had a significant societal impact.

1.2. MACHINE LEARNING AS A DE FACTO FEATURE

Machine learning is expected to be a transformative technology over the next two decades due to several factors. One key factor is the increasing availability of data, which is expected to continue to grow significantly in the coming years. As machine learning algorithms are particularly well suited for analyzing and making sense of large amounts of data, this will create new opportunities for their application in a variety of fields, including but not limited to healthcare, finance, transportation, education, manufacturing, and beyond. In these and other areas, machine learning has been adopted to automate some tasks that are currently performed by humans, freeing up humans to focus on more creative and high-level work [3].

In addition to automation, machine learning algorithms can be used to improve decision-making by analyzing large amounts of data and making predictions or recommendations based on that data. This can be particularly useful in fields such as finance, where machine learning can be used to identify