Check data integrity after copying thousands of files
I copied thousands of files into an exFAT MicroSD card.
The number of files and bytes is identical, but how do I know whether the data is corrupt or not?
[Edit 1:]
← Before clicking this triangle that points down, please explain the downvote.
[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.
storage data integrity data-loss-prevention
add a comment |
I copied thousands of files into an exFAT MicroSD card.
The number of files and bytes is identical, but how do I know whether the data is corrupt or not?
[Edit 1:]
← Before clicking this triangle that points down, please explain the downvote.
[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.
storage data integrity data-loss-prevention
Please explain the -1 downvote. Thanks.
– neverMind9
3 hours ago
add a comment |
I copied thousands of files into an exFAT MicroSD card.
The number of files and bytes is identical, but how do I know whether the data is corrupt or not?
[Edit 1:]
← Before clicking this triangle that points down, please explain the downvote.
[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.
storage data integrity data-loss-prevention
I copied thousands of files into an exFAT MicroSD card.
The number of files and bytes is identical, but how do I know whether the data is corrupt or not?
[Edit 1:]
← Before clicking this triangle that points down, please explain the downvote.
[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.
storage data integrity data-loss-prevention
storage data integrity data-loss-prevention
edited 1 hour ago
neverMind9
asked 3 hours ago
neverMind9neverMind9
546314
546314
Please explain the -1 downvote. Thanks.
– neverMind9
3 hours ago
add a comment |
Please explain the -1 downvote. Thanks.
– neverMind9
3 hours ago
Please explain the -1 downvote. Thanks.
– neverMind9
3 hours ago
Please explain the -1 downvote. Thanks.
– neverMind9
3 hours ago
add a comment |
3 Answers
3
active
oldest
votes
Unmount, eject, and remount the device. Then use
diff -r source destination
In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.
I like that. Unfortunately, Android Terminal lacksdiff. Good nevertheless.
– neverMind9
1 hour ago
1
@neverMind9 I hope you will edit the question if this is an important consideration to you :-).
– sourcejedi
1 hour ago
@neverMind9 Install Termux for diff,rsync,etc on Android.
– user1133275
17 mins ago
add a comment |
Using MD5 sums is a good way, but the canonical way to use it is:
cdto the directory of the source files and issue:
md5sum * >/path/to/the/checksumfile.md5
If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.
Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).
cdto the directory of the copied files and issue:
md5sum -c /path/to/the/checksumfile.md5
With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).
Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.
add a comment |
It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.
Run these commands on both source and destination:
md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
md5sum hash.list.txt
…or with a single command:
md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum
The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).
Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.
– neverMind9
3 hours ago
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495506%2fcheck-data-integrity-after-copying-thousands-of-files%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
Unmount, eject, and remount the device. Then use
diff -r source destination
In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.
I like that. Unfortunately, Android Terminal lacksdiff. Good nevertheless.
– neverMind9
1 hour ago
1
@neverMind9 I hope you will edit the question if this is an important consideration to you :-).
– sourcejedi
1 hour ago
@neverMind9 Install Termux for diff,rsync,etc on Android.
– user1133275
17 mins ago
add a comment |
Unmount, eject, and remount the device. Then use
diff -r source destination
In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.
I like that. Unfortunately, Android Terminal lacksdiff. Good nevertheless.
– neverMind9
1 hour ago
1
@neverMind9 I hope you will edit the question if this is an important consideration to you :-).
– sourcejedi
1 hour ago
@neverMind9 Install Termux for diff,rsync,etc on Android.
– user1133275
17 mins ago
add a comment |
Unmount, eject, and remount the device. Then use
diff -r source destination
In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.
Unmount, eject, and remount the device. Then use
diff -r source destination
In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.
edited 2 hours ago
answered 2 hours ago
sourcejedisourcejedi
23.4k437103
23.4k437103
I like that. Unfortunately, Android Terminal lacksdiff. Good nevertheless.
– neverMind9
1 hour ago
1
@neverMind9 I hope you will edit the question if this is an important consideration to you :-).
– sourcejedi
1 hour ago
@neverMind9 Install Termux for diff,rsync,etc on Android.
– user1133275
17 mins ago
add a comment |
I like that. Unfortunately, Android Terminal lacksdiff. Good nevertheless.
– neverMind9
1 hour ago
1
@neverMind9 I hope you will edit the question if this is an important consideration to you :-).
– sourcejedi
1 hour ago
@neverMind9 Install Termux for diff,rsync,etc on Android.
– user1133275
17 mins ago
I like that. Unfortunately, Android Terminal lacks
diff. Good nevertheless.– neverMind9
1 hour ago
I like that. Unfortunately, Android Terminal lacks
diff. Good nevertheless.– neverMind9
1 hour ago
1
1
@neverMind9 I hope you will edit the question if this is an important consideration to you :-).
– sourcejedi
1 hour ago
@neverMind9 I hope you will edit the question if this is an important consideration to you :-).
– sourcejedi
1 hour ago
@neverMind9 Install Termux for diff,rsync,etc on Android.
– user1133275
17 mins ago
@neverMind9 Install Termux for diff,rsync,etc on Android.
– user1133275
17 mins ago
add a comment |
Using MD5 sums is a good way, but the canonical way to use it is:
cdto the directory of the source files and issue:
md5sum * >/path/to/the/checksumfile.md5
If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.
Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).
cdto the directory of the copied files and issue:
md5sum -c /path/to/the/checksumfile.md5
With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).
Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.
add a comment |
Using MD5 sums is a good way, but the canonical way to use it is:
cdto the directory of the source files and issue:
md5sum * >/path/to/the/checksumfile.md5
If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.
Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).
cdto the directory of the copied files and issue:
md5sum -c /path/to/the/checksumfile.md5
With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).
Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.
add a comment |
Using MD5 sums is a good way, but the canonical way to use it is:
cdto the directory of the source files and issue:
md5sum * >/path/to/the/checksumfile.md5
If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.
Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).
cdto the directory of the copied files and issue:
md5sum -c /path/to/the/checksumfile.md5
With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).
Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.
Using MD5 sums is a good way, but the canonical way to use it is:
cdto the directory of the source files and issue:
md5sum * >/path/to/the/checksumfile.md5
If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.
Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).
cdto the directory of the copied files and issue:
md5sum -c /path/to/the/checksumfile.md5
With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).
Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.
edited 1 hour ago
answered 2 hours ago
xenoidxenoid
2,7981724
2,7981724
add a comment |
add a comment |
It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.
Run these commands on both source and destination:
md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
md5sum hash.list.txt
…or with a single command:
md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum
The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).
Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.
– neverMind9
3 hours ago
add a comment |
It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.
Run these commands on both source and destination:
md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
md5sum hash.list.txt
…or with a single command:
md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum
The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).
Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.
– neverMind9
3 hours ago
add a comment |
It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.
Run these commands on both source and destination:
md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
md5sum hash.list.txt
…or with a single command:
md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum
The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).
It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.
Run these commands on both source and destination:
md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
md5sum hash.list.txt
…or with a single command:
md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum
The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).
edited 1 hour ago
answered 3 hours ago
neverMind9neverMind9
546314
546314
Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.
– neverMind9
3 hours ago
add a comment |
Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.
– neverMind9
3 hours ago
Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.
– neverMind9
3 hours ago
Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.
– neverMind9
3 hours ago
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495506%2fcheck-data-integrity-after-copying-thousands-of-files%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Please explain the -1 downvote. Thanks.
– neverMind9
3 hours ago