Htydjtk

I've got 15 different datasets at about 10GB each. Each dataset comes with a binary 2D ground truth (10486147ish, 1) that I pull from it. I'm trying to figure out how to load each dataset, split them all with scikitlearn's train_test_split, then iterate over all 15 datasets per epoch. Under normal circumstances, the datasets would be shuffled as well, but I cannot figure out how to even do that since the data is too large to load all at once to shuffle (as such shuffling them is on the back burner for now).

Here's what my code looks like for one dataset.

import numpy as np

import pandas as pd

from keras.models import Sequential

from keras.layers import Dense

from keras.layers import LSTM

from sklearn.preprocessing import sequence

from sklearn.preprocessing import LabelEncoder

from sklearn.model_selection imporrt train_test_split







arr = np.load ('source/dir/dataset1.npy', allow_pickle = True, fix_imports = True)

arr[arr == -inf = -9999]

rehape = arr.reshape(((arr.shape[0])*(arr.shape[1])), (arr.shape[2]))

drop = reshape[~np.all(reshape == -9999, axis = 1)]

#additional work done with -9999 here

truth = drop[:,46]

data = drop[:,0:45]



#callbacks deleted in code sample



encoder = LabelEncoder()

encoder.fit(truth)

Y = encoder.transform(truth)

Y = Y.reshape(10486147, 1)

X = data.reshape(10486147, 45, 3)



seed = 7

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.33, random_state = seed)



model = sequential()

model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))

model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))

model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3)))

model.add(Dense(1, kernel_initializer = 'normal', activation = 'sigmoid'))



model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])



model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 500, batch_size = 1000, callbacks = [deleted callbacks])

So that makes sense for one dataset, but as I've said before I've got 15 datasets to iterate through, and I don't think retraining on new data is the right step.
Is there a way through dataset1.npy through dataset15.npy while properly splitting the ground truth as well.

Any suggestions?