Data Preprocessing

In this post, we introduce the basic process of data preprocessing (or data clearning). Data preprocessing is not a negligible and this topic is well discussed in the slides from Indiana and WashU.

Know Your Data

It is important to have a glance at your data before preprocessing or training, othewise you may find the impressive precision actually comes from the high rate of 1 in your data after you take several days to train the data.

Normalizing

Normalizing is the process of scaling the vector length to unit. Normalizing is an important step accodring to paper [1]. sklearn has provided the normalize to scale input vectors individually to unit norm (vector length).

Missing Data

Missing data is a very common problem for practical data. Pandas have a discussion on working with missing data.

Imbalanced Classes

The problem of imbalanced classes is very common in practical working, here we have a discussion on how to handle imblanced classes.

A python library can be found as imbalanced-learn.

Removing Outliers

Whether we should/need remove the outliers from the dataset is a tricky problem. Some comments here argue that outliers should not be removed from the data or we can conduct a sensitivity analysis given the existence of those outliers or not.

Some methods to remove the outliers from the data is summarized here.

A good practice is first make a boxplot of the data to have general image of the data.

Standard Deviation

A very simple method is to assume a normal distribution of all the samples and remove the outliers located beyond \(3\sigma\) away from the mean \(\mu\), where \(\sigma\) is the standard derivation of the sample data. That is

\[X =\{ x>\mu-3\sigma \text{ and }x < \mu+3\sigma | x\in X_{\text{raw}} \}\]

Boxplot

The boxplot is another very helpful method since it makes no distributional assumptions nor does it depend on a mean or standard deviation.

According to WashU, clustering is a good method to detect the outliers in the dataset. According to a paper, k-means can be a practical clustering method. sklearn has provide the KMeanas class to achieve K-Means clustering quickly.

Local Outlier Factor (LOF) is another classical method to perform outlier detection on moderately high dimensional datasets.

Ref.

[1] Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of machine learning research, 3(Mar), 1157-1182.