[PYTHON] [For beginners] Introduction to vectorization in machine learning


This article is the 5th day article of "Money Forward Advent Calendar 2015". Sorry for being late

Overview

Target

What not to do

Intro I feel that the excitement in this field is accelerating rapidly, with IT giants such as Googe, Microsoft, and IBM opening source machine learning systems. In my opinion (maybe hope), the third AI boom has taken root before winter, and we are now at the stage of being conscious of creating services that "use machine learning to make it natural + differentiate it." Isn't it?

So, I've been devoting my last few months of input activities exclusively to taking MOOCs and reading MLP series.

What I felt in that was also in The shortest route for working programmers who have avoided mathematics to start studying machine learning. At the very least, if you are not accustomed to handling matrices and vectors, even if you manipulate matrices in the course, it is easy to get into a state of what you are doing now (quote). " I specialized in control engineering and mechatronics at university, so I had some background, but I still struggled (excuse me because I had a four-year blank). I had a hard time myself, so it's tough for people in the same position or with no background! From a slightly above perspective, I would like to write about machine learning using vectors and matrices with an example.

Why use vectors / matrices

It is simply because the execution (learning) speed is fast. If you don't use a matrix, you will use for-loop to train each sample, but when n> 1000, the learning speed will drop dramatically. This is because Octave and Python, which are interpreted languages, incur overhead for each for minute. Therefore, it is recommended to learn by matrix rather than for-loop.

For reference, I will introduce the performance when I tried to implement MNIST data (28x28 pixels) learning of MLP (Multilayer Perceptron) model in both matrix version and for statement version.

How do you vectorize it?

Consider a simple logistic regression. This time, let's vectorize the calculus of z and the gradient.

State z vectorization

The vector z (each element) before plunging into the activation function can be calculated as follows. The above is first implemented with a for statement, but the goal is to implement it in a form using a matrix.

image

That is, we want to make z into the following form with just one command.

image

And each of the above columns can be transformed as follows. Please recognize it as ** such thing **.

image

As mentioned above, each element in z is the inner product of the vector x and the vector theta. To express this with x and theta without using a for statement, create an X with each vector x (transposed) superimposed on a row as shown below.

image

Then you can create a simple formula like the one below.

image

Implementation

If you implement this in Octave / python, you can get a neat shape as below.

octave.m


z = X * theta;

python.py


#With the method using numpy
np.dot(theta, X)

Example 2 Gradient vectorization

In order to find the optimum parameter, we need to implement the partial differential formula because we want to find a value that makes the partial differential for each parameter of the evaluation function 0 or below the threshold.

\frac{\partial J}{\partial \theta_j} = \frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)}) x_{j}^{(i)}

When this is vectorized

image

Now, to transform this, you can first use the following rules. Please recognize it as ** such thing **.

image

The matrix in which x is lined up is the transpose of X introduced in the vectorization of z, so

image

Therefore, you can write a vectorized partial derivative of the cost function as shown below.

image

Implementation

I will briefly call it as follows.

octave.m


h = activate_function(z)
grad = 1 / m * (X'*(h-y))

python.py


h = activate_function(z)
grad = 1/m * np.dot(X.T, h-y)

Summary

The above is the explanation about vectorization. I would like to make a splint for the hearts of beginners with this. However, I don't feel like I'm making a mistake, so I'd love to hear your opinions and comments.

Books I read / courses I took

Digression

Recommended Posts

[For beginners] Introduction to vectorization in machine learning
An introduction to OpenCV for machine learning
An introduction to Python for machine learning
Introduction to machine learning
An introduction to machine learning
Super introduction to machine learning
Everything for beginners to be able to do machine learning
Introduction to Deep Learning (1) --Chainer is explained in an easy-to-understand manner for beginners-
Introduction to machine learning Note writing
<For beginners> python library <For machine learning>
Introduction to Machine Learning Library SHOGUN
[Explanation for beginners] Introduction to convolution processing (explained in TensorFlow)
Introduction to Machine Learning: How Models Work
First Steps for Machine Learning (AI) Beginners
Introduction to ClearML-Easy to manage machine learning experiments-
Python learning memo for machine learning by Chainer Chapter 8 Introduction to Numpy
Before the introduction to machine learning. ~ Technology required for machine learning other than machine learning ~
Python learning memo for machine learning by Chainer Chapter 10 Introduction to Cupy
Python learning memo for machine learning by Chainer Chapter 9 Introduction to scikit-learn
[Python] Easy introduction to machine learning with python (SVM)
[Super Introduction to Machine Learning] Learn Pytorch tutorials
An introduction to object-oriented programming for beginners by beginners
Recommended study order for machine learning / deep learning beginners
Try to calculate RPN in Python (for beginners)
Take the free "Introduction to Python for Machine Learning" online until 4/27 application
[Super Introduction to Machine Learning] Learn Pytorch tutorials
Python beginners publish web applications using machine learning [Part 2] Introduction to explosive Python !!
[Introduction for beginners] Working with MySQL in Python
Introduction to Graph Database Neo4j in Python for Beginners (for Mac OS X)
I tried to predict the change in snowfall for 2 years by machine learning
How to Introduce IPython (Python2) to Mac OS X-Preparation for Introduction to Machine Learning Theory-
Python learning notes for machine learning with Chainer Chapters 11 and 12 Introduction to Pandas Matplotlib
A memorandum of method often used in machine learning using scikit-learn (for beginners)
[For beginners] How to use say command in python!
Build an interactive environment for machine learning in Python
Preparing to start "Python machine learning programming" (for macOS)
Machine learning beginners try to make a decision tree
Attempt to include machine learning model in python package
Cross-entropy to review in Coursera Machine Learning week 2 assignments
An introduction to machine learning from a simple perceptron
Data set for machine learning
[Learning memorandum] Introduction to vim
Japanese preprocessing for machine learning
Learning flow for Python beginners
Used in machine learning EDA
Introduction to Deep Learning ~ Learning Rules ~
Deep Reinforcement Learning 1 Introduction to Reinforcement Learning
~ Tips for beginners to Python ③ ~
Introduction to Python For, While
Introduction to Deep Learning ~ Backpropagation ~
How to adapt multiple machine learning libraries in one shot
Introduction to Machine Learning with scikit-learn-From data acquisition to parameter optimization
Python beginners publish web applications using machine learning [Part 1] Introduction
Made icrawler easier to use for machine learning data collection
For those who want to start machine learning with TensorFlow2
How to use machine learning for work? 03_Python coding procedure
Machine learning beginners try to reach out to Naive Bayes (2) --Implementation
For beginners, how to deal with common errors in keras
Machine learning to learn with Nogizaka46 and Keyakizaka46 Part 1 Introduction
[Python] Introduction to graph creation using coronavirus data [For beginners]
Try Q-learning in Dragon Quest-style battle [Introduction to Reinforcement Learning]