Htydjtk

I have data that I'm compressing with AutoEncoders (3-layer neural network) and I would like to normalize my data first. I would like to try to use the coded latent vector and feed it into an anomaly detection algorithm and see what happens.

I would like to normalize the data for the autoencoder so my values are either between 0,1 or -1,-1 because my output activation function will either be a sigmoid or tanh. This way my algorithm can train and the input will be in the same range as the output values of the NN.

However, when I normalized with

x(i)-xmean/(xmax-xmin) 

I ended up dividing by 0 in several features of the data which gave NaN. Is is possible to normalize my data so it is between -1,1 or 0,1 while avoiding dividing by 0 for my data?

While you could do this manually, Python also has a handy little function called MinMaxScaler, which will automatically apply max-min normalization to scale data between 0 and 1.

Assume we have an array of 200 values for variables s and t:

import numpy as np

import pandas as pd

from sklearn.preprocessing import MinMaxScaler



mu, sigma = 20, 10 # mean and standard deviation

s = np.random.normal(mu, sigma, 200)

t = np.random.normal(mu, sigma, 200)

Reshape your variables if necessary:

s=np.reshape(s,(-1,1))

t=np.reshape(t,(-1,1))

Now, you can see that we are forming two new variables, snew and tnew, which we are scaling using MinMaxScaler.

scaler = MinMaxScaler()

print(scaler.fit(s))

print(scaler.fit(s))

snew=scaler.transform(s)

tnew=scaler.transform(t)

Here is a sample of our new variables:

>>> snew

array([[0.24896606],

       [0.63121206],

       [0.60448469],

       .......

       [0.49044733],

       [0.28131596],

       [0.32909155]



>>> tnew

array([[0.91224005],

       [0.74540598],

       [0.3938718 ],

       .......

       [0.75749275],

       [0.80709325],

       [0.19440844]

As others pointed out, you can normalize or standardize your data using the following steps. I'm sure other libraries have similar functions but I think this is efficient.

Since you requested normalization, I'll cover that topic in this post. As others alluded, data normalization is the process in which researchers or data science practitioners make all the values in a given dataset be proportionally spread between 0 and 1.

To implement normalization, follow the steps below:

from sklearn.datasets import load_iris

from sklearn import preprocessing



iris = load_iris()

print(iris.data.shape)



X_data = iris.data

y_labels = iris.target



normalized_X_data = preprocessing.normalize(X_data)

You should subtract the xmin from x, not xmean.

Here is a normalization function generalized to rescale any new minimum and maximum as parameters (e.g., 0,1 or -1,-1):

def rescale(nums, new_min=0, new_max=1):

      "Rescale values to be between new min and max"

      return [(new_max - new_min) / (max(nums)-min(nums)) * (value-max(nums)) + new_max for value in nums]