Check data integrity after copying thousands of files












1















I copied thousands of files into an exFAT MicroSD card.

The number of files and bytes is identical, but how do I know whether the data is corrupt or not?


[Edit 1:]
Before clicking this triangle that points down, please explain the downvote.



[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.










share|improve this question

























  • Please explain the -1 downvote. Thanks.

    – neverMind9
    3 hours ago


















1















I copied thousands of files into an exFAT MicroSD card.

The number of files and bytes is identical, but how do I know whether the data is corrupt or not?


[Edit 1:]
Before clicking this triangle that points down, please explain the downvote.



[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.










share|improve this question

























  • Please explain the -1 downvote. Thanks.

    – neverMind9
    3 hours ago
















1












1








1








I copied thousands of files into an exFAT MicroSD card.

The number of files and bytes is identical, but how do I know whether the data is corrupt or not?


[Edit 1:]
Before clicking this triangle that points down, please explain the downvote.



[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.










share|improve this question
















I copied thousands of files into an exFAT MicroSD card.

The number of files and bytes is identical, but how do I know whether the data is corrupt or not?


[Edit 1:]
Before clicking this triangle that points down, please explain the downvote.



[Edit 2:]
It would be good if the JackPal Android Terminal also supports the command, but if you have a solution, pleasa answer nevertheless.







storage data integrity data-loss-prevention






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 1 hour ago







neverMind9

















asked 3 hours ago









neverMind9neverMind9

546314




546314













  • Please explain the -1 downvote. Thanks.

    – neverMind9
    3 hours ago





















  • Please explain the -1 downvote. Thanks.

    – neverMind9
    3 hours ago



















Please explain the -1 downvote. Thanks.

– neverMind9
3 hours ago







Please explain the -1 downvote. Thanks.

– neverMind9
3 hours ago












3 Answers
3






active

oldest

votes


















4














Unmount, eject, and remount the device. Then use



diff -r source destination




In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.






share|improve this answer


























  • I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless.

    – neverMind9
    1 hour ago






  • 1





    @neverMind9 I hope you will edit the question if this is an important consideration to you :-).

    – sourcejedi
    1 hour ago













  • @neverMind9 Install Termux for diff,rsync,etc on Android.

    – user1133275
    17 mins ago





















4














Using MD5 sums is a good way, but the canonical way to use it is:





  1. cd to the directory of the source files and issue:



    md5sum * >/path/to/the/checksumfile.md5



If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.



Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).





  1. cd to the directory of the copied files and issue:



    md5sum -c /path/to/the/checksumfile.md5



With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).



Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.






share|improve this answer

































    2














    It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
    I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.



    Run these commands on both source and destination:



    md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
    md5sum hash.list.txt


    …or with a single command:



    md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum


    The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).






    share|improve this answer


























    • Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.

      – neverMind9
      3 hours ago











    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "106"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495506%2fcheck-data-integrity-after-copying-thousands-of-files%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    3 Answers
    3






    active

    oldest

    votes








    3 Answers
    3






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    4














    Unmount, eject, and remount the device. Then use



    diff -r source destination




    In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.






    share|improve this answer


























    • I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless.

      – neverMind9
      1 hour ago






    • 1





      @neverMind9 I hope you will edit the question if this is an important consideration to you :-).

      – sourcejedi
      1 hour ago













    • @neverMind9 Install Termux for diff,rsync,etc on Android.

      – user1133275
      17 mins ago


















    4














    Unmount, eject, and remount the device. Then use



    diff -r source destination




    In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.






    share|improve this answer


























    • I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless.

      – neverMind9
      1 hour ago






    • 1





      @neverMind9 I hope you will edit the question if this is an important consideration to you :-).

      – sourcejedi
      1 hour ago













    • @neverMind9 Install Termux for diff,rsync,etc on Android.

      – user1133275
      17 mins ago
















    4












    4








    4







    Unmount, eject, and remount the device. Then use



    diff -r source destination




    In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.






    share|improve this answer















    Unmount, eject, and remount the device. Then use



    diff -r source destination




    In case you used rsync to do the copy, rsync -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.







    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited 2 hours ago

























    answered 2 hours ago









    sourcejedisourcejedi

    23.4k437103




    23.4k437103













    • I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless.

      – neverMind9
      1 hour ago






    • 1





      @neverMind9 I hope you will edit the question if this is an important consideration to you :-).

      – sourcejedi
      1 hour ago













    • @neverMind9 Install Termux for diff,rsync,etc on Android.

      – user1133275
      17 mins ago





















    • I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless.

      – neverMind9
      1 hour ago






    • 1





      @neverMind9 I hope you will edit the question if this is an important consideration to you :-).

      – sourcejedi
      1 hour ago













    • @neverMind9 Install Termux for diff,rsync,etc on Android.

      – user1133275
      17 mins ago



















    I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless.

    – neverMind9
    1 hour ago





    I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless.

    – neverMind9
    1 hour ago




    1




    1





    @neverMind9 I hope you will edit the question if this is an important consideration to you :-).

    – sourcejedi
    1 hour ago







    @neverMind9 I hope you will edit the question if this is an important consideration to you :-).

    – sourcejedi
    1 hour ago















    @neverMind9 Install Termux for diff,rsync,etc on Android.

    – user1133275
    17 mins ago







    @neverMind9 Install Termux for diff,rsync,etc on Android.

    – user1133275
    17 mins ago















    4














    Using MD5 sums is a good way, but the canonical way to use it is:





    1. cd to the directory of the source files and issue:



      md5sum * >/path/to/the/checksumfile.md5



    If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.



    Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).





    1. cd to the directory of the copied files and issue:



      md5sum -c /path/to/the/checksumfile.md5



    With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).



    Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.






    share|improve this answer






























      4














      Using MD5 sums is a good way, but the canonical way to use it is:





      1. cd to the directory of the source files and issue:



        md5sum * >/path/to/the/checksumfile.md5



      If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.



      Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).





      1. cd to the directory of the copied files and issue:



        md5sum -c /path/to/the/checksumfile.md5



      With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).



      Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.






      share|improve this answer




























        4












        4








        4







        Using MD5 sums is a good way, but the canonical way to use it is:





        1. cd to the directory of the source files and issue:



          md5sum * >/path/to/the/checksumfile.md5



        If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.



        Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).





        1. cd to the directory of the copied files and issue:



          md5sum -c /path/to/the/checksumfile.md5



        With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).



        Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.






        share|improve this answer















        Using MD5 sums is a good way, but the canonical way to use it is:





        1. cd to the directory of the source files and issue:



          md5sum * >/path/to/the/checksumfile.md5



        If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.



        Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).





        1. cd to the directory of the copied files and issue:



          md5sum -c /path/to/the/checksumfile.md5



        With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).



        Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited 1 hour ago

























        answered 2 hours ago









        xenoidxenoid

        2,7981724




        2,7981724























            2














            It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
            I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.



            Run these commands on both source and destination:



            md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
            md5sum hash.list.txt


            …or with a single command:



            md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum


            The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).






            share|improve this answer


























            • Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.

              – neverMind9
              3 hours ago
















            2














            It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
            I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.



            Run these commands on both source and destination:



            md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
            md5sum hash.list.txt


            …or with a single command:



            md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum


            The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).






            share|improve this answer


























            • Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.

              – neverMind9
              3 hours ago














            2












            2








            2







            It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
            I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.



            Run these commands on both source and destination:



            md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
            md5sum hash.list.txt


            …or with a single command:



            md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum


            The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).






            share|improve this answer















            It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated.
            I use MD5 due to it's higher speeds. MD5 might not be the newest algorithm, but it is certainly good enough to verify offline data integrity.



            Run these commands on both source and destination:



            md5sum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts hashsum string only for the output, because I noticed that Android Terminal formats the output differently.
            md5sum hash.list.txt


            …or with a single command:



            md5sum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | md5sum


            The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).







            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited 1 hour ago

























            answered 3 hours ago









            neverMind9neverMind9

            546314




            546314













            • Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.

              – neverMind9
              3 hours ago



















            • Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.

              – neverMind9
              3 hours ago

















            Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.

            – neverMind9
            3 hours ago





            Yes, this is an answer to my own question. I would still appreciate your answer, if you know a better alternative.

            – neverMind9
            3 hours ago


















            draft saved

            draft discarded




















































            Thanks for contributing an answer to Unix & Linux Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495506%2fcheck-data-integrity-after-copying-thousands-of-files%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Callistus I

            Tabula Rosettana

            How to label and detect the document text images