Which Format is Faster, Matlab (.mat) or NumPy (.npy)?
$begingroup$
I am working on a deep learning problem to detect cancer in images of size 250 x 250. I have hardware limitations and I have been running out of memory.
I decided to convert my images to Matlab formatted files (".mat"), with some improvement; however, I still run out of memory. I have explored some resources that highly recommend using NumPy files (".npy").
It would be costly to convert my images to NumPy files, so I would like to make sure that converting will make a difference. I am not asking for memory enhancement algorithms (e.g. batching), just the memory difference between ".mat" and ".npy" files.
python deep-learning matlab numpy
edited Nov 16 '18 at 18:07
from keras import michael
29810
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
$endgroup$
bumped to the homepage by Community♦ 1 hour ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I am working on a deep learning problem to detect cancer in images of size 250 x 250. I have hardware limitations and I have been running out of memory.
I decided to convert my images to Matlab formatted files (".mat"), with some improvement; however, I still run out of memory. I have explored some resources that highly recommend using NumPy files (".npy").
It would be costly to convert my images to NumPy files, so I would like to make sure that converting will make a difference. I am not asking for memory enhancement algorithms (e.g. batching), just the memory difference between ".mat" and ".npy" files.
python deep-learning matlab numpy
edited Nov 16 '18 at 18:07
from keras import michael
29810
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
$endgroup$
bumped to the homepage by Community♦ 1 hour ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Can you post your loading code? Can you post benchmarking, including memory usage?
$endgroup$
– Brian Spiering
Nov 10 '18 at 18:07
add a comment |
$begingroup$
I am working on a deep learning problem to detect cancer in images of size 250 x 250. I have hardware limitations and I have been running out of memory.
I decided to convert my images to Matlab formatted files (".mat"), with some improvement; however, I still run out of memory. I have explored some resources that highly recommend using NumPy files (".npy").
It would be costly to convert my images to NumPy files, so I would like to make sure that converting will make a difference. I am not asking for memory enhancement algorithms (e.g. batching), just the memory difference between ".mat" and ".npy" files.
python deep-learning matlab numpy
edited Nov 16 '18 at 18:07
from keras import michael
29810
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
$endgroup$
I am working on a deep learning problem to detect cancer in images of size 250 x 250. I have hardware limitations and I have been running out of memory.
I decided to convert my images to Matlab formatted files (".mat"), with some improvement; however, I still run out of memory. I have explored some resources that highly recommend using NumPy files (".npy").
It would be costly to convert my images to NumPy files, so I would like to make sure that converting will make a difference. I am not asking for memory enhancement algorithms (e.g. batching), just the memory difference between ".mat" and ".npy" files.
python deep-learning matlab numpy
python deep-learning matlab numpy
edited Nov 16 '18 at 18:07
from keras import michael
29810
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
edited Nov 16 '18 at 18:07
from keras import michael
29810
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
edited Nov 16 '18 at 18:07
from keras import michael
29810
edited Nov 16 '18 at 18:07
from keras import michael
29810
edited Nov 16 '18 at 18:07
from keras import michael
29810
29810
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
asked Jun 28 '18 at 10:53
Andrew NaguibAndrew Naguib
676
676
bumped to the homepage by Community♦ 1 hour ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 1 hour ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Can you post your loading code? Can you post benchmarking, including memory usage?
$endgroup$
– Brian Spiering
Nov 10 '18 at 18:07
add a comment |
$begingroup$
Can you post your loading code? Can you post benchmarking, including memory usage?
$endgroup$
– Brian Spiering
Nov 10 '18 at 18:07
$begingroup$
Can you post your loading code? Can you post benchmarking, including memory usage?
$endgroup$
– Brian Spiering
Nov 10 '18 at 18:07
$begingroup$
Can you post your loading code? Can you post benchmarking, including memory usage?
$endgroup$
– Brian Spiering
Nov 10 '18 at 18:07
add a comment |
3 Answers
3
active
oldest
votes
$begingroup$
From my research:
- both can store data in binary format
- both store the data type of the data
- I am unsure about compression ratios and load time, which seems to be the subtext of your question.
One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.
That said, I found this post useful and thought it may be helpful to you if you have not seen it already. https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.
Have you considered the HDF5 format? If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.
Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
$endgroup$
add a comment |
$begingroup$
MATLAB has known for it's memory consumption. So even if you use same data for processing in Python overall system memory utilization will be less in Python.
Based on my experience so far using Python helped me dealing more data with better performance.
One other hand Python have many libraries/Frameworks out of box to further enhance the overall performance and Machine Learning/ Deep Learning (I am not much sure if similar Libraries & Frameworks are available in Matlab also).
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
answered Nov 13 '18 at 2:27
mannumannu
745
$endgroup$
add a comment |
$begingroup$
One way to reduce in-memory bottlenecks is to more efficiently handle data processing (regardless of the on-disk format).
There are software frameworks designed to improve the training process, especially for loading images. Dask is one such framework to scale existing Python workflows, thus mostly likely it will reduce the memory bottleneck for .npy files relative to .mat files (the only way to be sure is to benchmark).
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f33758%2fwhich-format-is-faster-matlab-mat-or-numpy-npy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
From my research:
- both can store data in binary format
- both store the data type of the data
- I am unsure about compression ratios and load time, which seems to be the subtext of your question.
One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.
That said, I found this post useful and thought it may be helpful to you if you have not seen it already. https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.
Have you considered the HDF5 format? If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.
Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
$endgroup$
add a comment |
$begingroup$
From my research:
- both can store data in binary format
- both store the data type of the data
- I am unsure about compression ratios and load time, which seems to be the subtext of your question.
One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.
That said, I found this post useful and thought it may be helpful to you if you have not seen it already. https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.
Have you considered the HDF5 format? If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.
Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
$endgroup$
add a comment |
$begingroup$
From my research:
- both can store data in binary format
- both store the data type of the data
- I am unsure about compression ratios and load time, which seems to be the subtext of your question.
One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.
That said, I found this post useful and thought it may be helpful to you if you have not seen it already. https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.
Have you considered the HDF5 format? If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.
Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
$endgroup$
From my research:
- both can store data in binary format
- both store the data type of the data
- I am unsure about compression ratios and load time, which seems to be the subtext of your question.
One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.
That said, I found this post useful and thought it may be helpful to you if you have not seen it already. https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.
Have you considered the HDF5 format? If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.
Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
answered Nov 9 '18 at 14:42
SkiddlesSkiddles
695210
695210
add a comment |
add a comment |
$begingroup$
MATLAB has known for it's memory consumption. So even if you use same data for processing in Python overall system memory utilization will be less in Python.
Based on my experience so far using Python helped me dealing more data with better performance.
One other hand Python have many libraries/Frameworks out of box to further enhance the overall performance and Machine Learning/ Deep Learning (I am not much sure if similar Libraries & Frameworks are available in Matlab also).
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
answered Nov 13 '18 at 2:27
mannumannu
745
$endgroup$
add a comment |
$begingroup$
MATLAB has known for it's memory consumption. So even if you use same data for processing in Python overall system memory utilization will be less in Python.
Based on my experience so far using Python helped me dealing more data with better performance.
One other hand Python have many libraries/Frameworks out of box to further enhance the overall performance and Machine Learning/ Deep Learning (I am not much sure if similar Libraries & Frameworks are available in Matlab also).
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
answered Nov 13 '18 at 2:27
mannumannu
745
$endgroup$
add a comment |
$begingroup$
MATLAB has known for it's memory consumption. So even if you use same data for processing in Python overall system memory utilization will be less in Python.
Based on my experience so far using Python helped me dealing more data with better performance.
One other hand Python have many libraries/Frameworks out of box to further enhance the overall performance and Machine Learning/ Deep Learning (I am not much sure if similar Libraries & Frameworks are available in Matlab also).
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
answered Nov 13 '18 at 2:27
mannumannu
745
$endgroup$
MATLAB has known for it's memory consumption. So even if you use same data for processing in Python overall system memory utilization will be less in Python.
Based on my experience so far using Python helped me dealing more data with better performance.
One other hand Python have many libraries/Frameworks out of box to further enhance the overall performance and Machine Learning/ Deep Learning (I am not much sure if similar Libraries & Frameworks are available in Matlab also).
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
answered Nov 13 '18 at 2:27
mannumannu
745
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
edited Nov 13 '18 at 3:11
Stephen Rauch♦
1,52551330
1,52551330
answered Nov 13 '18 at 2:27
mannumannu
745
answered Nov 13 '18 at 2:27
mannumannu
745
answered Nov 13 '18 at 2:27
mannumannu
745
745
add a comment |
add a comment |
$begingroup$
One way to reduce in-memory bottlenecks is to more efficiently handle data processing (regardless of the on-disk format).
There are software frameworks designed to improve the training process, especially for loading images. Dask is one such framework to scale existing Python workflows, thus mostly likely it will reduce the memory bottleneck for .npy files relative to .mat files (the only way to be sure is to benchmark).
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
$endgroup$
add a comment |
$begingroup$
One way to reduce in-memory bottlenecks is to more efficiently handle data processing (regardless of the on-disk format).
There are software frameworks designed to improve the training process, especially for loading images. Dask is one such framework to scale existing Python workflows, thus mostly likely it will reduce the memory bottleneck for .npy files relative to .mat files (the only way to be sure is to benchmark).
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
$endgroup$
add a comment |
$begingroup$
One way to reduce in-memory bottlenecks is to more efficiently handle data processing (regardless of the on-disk format).
There are software frameworks designed to improve the training process, especially for loading images. Dask is one such framework to scale existing Python workflows, thus mostly likely it will reduce the memory bottleneck for .npy files relative to .mat files (the only way to be sure is to benchmark).
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
$endgroup$
One way to reduce in-memory bottlenecks is to more efficiently handle data processing (regardless of the on-disk format).
There are software frameworks designed to improve the training process, especially for loading images. Dask is one such framework to scale existing Python workflows, thus mostly likely it will reduce the memory bottleneck for .npy files relative to .mat files (the only way to be sure is to benchmark).
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
edited Nov 10 '18 at 18:03
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
answered Nov 10 '18 at 17:51
Brian SpieringBrian Spiering
4,2781129
4,2781129
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f33758%2fwhich-format-is-faster-matlab-mat-or-numpy-npy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Can you post your loading code? Can you post benchmarking, including memory usage?
$endgroup$
– Brian Spiering
Nov 10 '18 at 18:07