Data preprocessing

Hello! I am a web developer and have been interested in machine learning for several years now. Because in my day-to-day work park I have to solve problems that are less interesting to me, not related to machine learning, from time to time I forget what I once read or used. To create a memo for myself, strengthen my knowledge and share it with others, I decided to write this series of articles on machine learning. I'll start with data preprocessing.

In this article I will talk about what problems happen with data, how to solve them, as well as about the most commonly used methods of preparing data before feeding it to different models.

Skips

Consider the following dataset. I honestly invented it and will refer to it further in this article.

, . . — , . , .

, "" , . , , .

— , - , . , "" "" . , , - - . : , , , .

. , . .

, . , . : , . — , .

:

• "".

• .

, :

• .

, " " 1 .

• . , .

, . , , . , "" . — ( ).

, , , , . :

$$

$$IQR = Q_3 - Q_1,$$

$$$Q_1$ — — , 25% . $$$Q_3$ — — , 75% .

, , , : :

$$

$$[ Q_1 -1.5IQR, Q_3 + 1.5IQR ]$$

.

— . , [0, 1]. , . , . (, , ) .

. , , . , .

$$

$$x_{new} = \frac{ x_{old} - x_{min}} {x_{max} - x_{min}}$$

Z-. Z- :

$$

$$(-3σ[X], 3σ[X]),$$

$$$σ[X]$ — X.

Z- .

$$

$$x_{new} = \frac{x_{old} - M[X]} {σ[X]}$$

M[X] — X.

, Z- , .

One-hot encoding

. . , " " - . : . . . ( ).

, , ? . , " " 1, "" — 2. . , , . , . .

, , . , "" 4 :

, , .

, . . , , . . , .

Thanks for reading or browsing to here. I have described not all preprocessing methods, and this article is hardly useful for professional data scientists. However, if you are a beginner and do not know what to do with your data, you can safely return here. Good luck with your learning and interesting tasks!

List of sources

I am not a scientist and this article does not claim to be scientific. Therefore, I will not draw up sources according to GOSTs. Please excuse me for this.

1. Lecture course from Yandex and HSE "Introduction to Machine Learning" on the cursor.
2. Standardization, or mean removal and variance scaling - sklearn library documentation
3. Advanced machine learning data preparation tasks - Microsoft