How to big scale backup Gitlab?












8















When asking Gitlab support on how to do a 3TB backup on ones on-premise Gitlab they reply use our tool that produces a tarball.



This just seams wrong to me on all levels. This tarball contains the postgres dump, docker images, repo data, GIT LFS, etc config and so on. Backing up TB of static data together with KB very dynamic data doesn't seam right. And then comes the issue of, we want to do a backup every hour.



Question



I'd really like to know from others how they do it, to get a consistent backup.



ZFS on Linux would be fine with me, if that is part of the solution.






linux backup postgresql zfs gitlab







share|improve this question

























  • Why is this wrong? You back up your Gitlab completely to restore it completely. I don't think this is wrong. Of course it uses much more space than say, incremental backups, but...I wouldn't care about backup size.

    – Lenniey
    3 hours ago






  • 1





    Having a backup every hour is not unheard of, but it is impossible to make a 3TB in less than hour with their approach. And backups for just one day would be ~100TB, where there might only be 10MB of changes to the data.

    – Sandra
    3 hours ago











  • OK, this is a different question, not about the backup in general but about frequent backups.

    – Lenniey
    3 hours ago






  • 1





    In their official docs they even mention their method as being slow and suggest alternatives: If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. In this case you can consider using filesystem snapshots as part of your backup strategy. I can't speak from experience, though. But I may have to include something like this soon...

    – Lenniey
    3 hours ago


















8















When asking Gitlab support on how to do a 3TB backup on ones on-premise Gitlab they reply use our tool that produces a tarball.



This just seams wrong to me on all levels. This tarball contains the postgres dump, docker images, repo data, GIT LFS, etc config and so on. Backing up TB of static data together with KB very dynamic data doesn't seam right. And then comes the issue of, we want to do a backup every hour.



Question



I'd really like to know from others how they do it, to get a consistent backup.



ZFS on Linux would be fine with me, if that is part of the solution.






linux backup postgresql zfs gitlab







share|improve this question

























  • Why is this wrong? You back up your Gitlab completely to restore it completely. I don't think this is wrong. Of course it uses much more space than say, incremental backups, but...I wouldn't care about backup size.

    – Lenniey
    3 hours ago






  • 1





    Having a backup every hour is not unheard of, but it is impossible to make a 3TB in less than hour with their approach. And backups for just one day would be ~100TB, where there might only be 10MB of changes to the data.

    – Sandra
    3 hours ago











  • OK, this is a different question, not about the backup in general but about frequent backups.

    – Lenniey
    3 hours ago






  • 1





    In their official docs they even mention their method as being slow and suggest alternatives: If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. In this case you can consider using filesystem snapshots as part of your backup strategy. I can't speak from experience, though. But I may have to include something like this soon...

    – Lenniey
    3 hours ago
















8












8








8


1






When asking Gitlab support on how to do a 3TB backup on ones on-premise Gitlab they reply use our tool that produces a tarball.



This just seams wrong to me on all levels. This tarball contains the postgres dump, docker images, repo data, GIT LFS, etc config and so on. Backing up TB of static data together with KB very dynamic data doesn't seam right. And then comes the issue of, we want to do a backup every hour.



Question



I'd really like to know from others how they do it, to get a consistent backup.



ZFS on Linux would be fine with me, if that is part of the solution.






linux backup postgresql zfs gitlab







share|improve this question
















When asking Gitlab support on how to do a 3TB backup on ones on-premise Gitlab they reply use our tool that produces a tarball.



This just seams wrong to me on all levels. This tarball contains the postgres dump, docker images, repo data, GIT LFS, etc config and so on. Backing up TB of static data together with KB very dynamic data doesn't seam right. And then comes the issue of, we want to do a backup every hour.



Question



I'd really like to know from others how they do it, to get a consistent backup.



ZFS on Linux would be fine with me, if that is part of the solution.






linux backup postgresql zfs gitlab




linux backup postgresql zfs gitlab






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 2 hours ago







Sandra

















asked 3 hours ago









SandraSandra

4,4833279137




4,4833279137













  • Why is this wrong? You back up your Gitlab completely to restore it completely. I don't think this is wrong. Of course it uses much more space than say, incremental backups, but...I wouldn't care about backup size.

    – Lenniey
    3 hours ago






  • 1





    Having a backup every hour is not unheard of, but it is impossible to make a 3TB in less than hour with their approach. And backups for just one day would be ~100TB, where there might only be 10MB of changes to the data.

    – Sandra
    3 hours ago











  • OK, this is a different question, not about the backup in general but about frequent backups.

    – Lenniey
    3 hours ago






  • 1





    In their official docs they even mention their method as being slow and suggest alternatives: If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. In this case you can consider using filesystem snapshots as part of your backup strategy. I can't speak from experience, though. But I may have to include something like this soon...

    – Lenniey
    3 hours ago





















  • Why is this wrong? You back up your Gitlab completely to restore it completely. I don't think this is wrong. Of course it uses much more space than say, incremental backups, but...I wouldn't care about backup size.

    – Lenniey
    3 hours ago






  • 1





    Having a backup every hour is not unheard of, but it is impossible to make a 3TB in less than hour with their approach. And backups for just one day would be ~100TB, where there might only be 10MB of changes to the data.

    – Sandra
    3 hours ago











  • OK, this is a different question, not about the backup in general but about frequent backups.

    – Lenniey
    3 hours ago






  • 1





    In their official docs they even mention their method as being slow and suggest alternatives: If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. In this case you can consider using filesystem snapshots as part of your backup strategy. I can't speak from experience, though. But I may have to include something like this soon...

    – Lenniey
    3 hours ago



















Why is this wrong? You back up your Gitlab completely to restore it completely. I don't think this is wrong. Of course it uses much more space than say, incremental backups, but...I wouldn't care about backup size.

– Lenniey
3 hours ago





Why is this wrong? You back up your Gitlab completely to restore it completely. I don't think this is wrong. Of course it uses much more space than say, incremental backups, but...I wouldn't care about backup size.

– Lenniey
3 hours ago




1




1





Having a backup every hour is not unheard of, but it is impossible to make a 3TB in less than hour with their approach. And backups for just one day would be ~100TB, where there might only be 10MB of changes to the data.

– Sandra
3 hours ago





Having a backup every hour is not unheard of, but it is impossible to make a 3TB in less than hour with their approach. And backups for just one day would be ~100TB, where there might only be 10MB of changes to the data.

– Sandra
3 hours ago













OK, this is a different question, not about the backup in general but about frequent backups.

– Lenniey
3 hours ago





OK, this is a different question, not about the backup in general but about frequent backups.

– Lenniey
3 hours ago




1




1





In their official docs they even mention their method as being slow and suggest alternatives: If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. In this case you can consider using filesystem snapshots as part of your backup strategy. I can't speak from experience, though. But I may have to include something like this soon...

– Lenniey
3 hours ago







In their official docs they even mention their method as being slow and suggest alternatives: If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. In this case you can consider using filesystem snapshots as part of your backup strategy. I can't speak from experience, though. But I may have to include something like this soon...

– Lenniey
3 hours ago












2 Answers
2






active

oldest

votes


















5














I would review what you are backing up and possibly use a "multi-path" approach. For example, you could backup the Git repositories by constantly running through Git pulls on a backup servers. That would copy only the diff and leave you with a second copy of all Git repositories. Presumably you could detect new repos with the API.



And use the "built-in" backup procedures to backup the issues, etc. I doubt that the 3TB comes from this part so you would be able to do backups very often at very little cost. You could also set up the PostgreSQL database with a warm standby with replication.



Possibly your 3TB comes from container images in the Docker registry. Do you need to back those up? If so, then there may be a better approach just for that.



Basically, I would recommend really looking at what it is that makes up your backup and backup the data in various parts.



Even the backup tool from GitLab has options to include/exclude certain parts of the system such as the Docker Registry.






share|improve this answer































    3














    For such a short time between backups (1h), your best bet is to rely on filesystem-level snapshot and send/recv support.



    If using ZoL is not a problem in your environment, I would strongly advise to use it. ZFS is a very robust filesystem and you will really like all the extras (eg: compression) it offer. When coupled with sanoid/syncoid, it can provide a very strong backup strategy. The main disvantage is that it is not included into mainline kernel, so you need to install/update it separately.



    Alternatively, if you really need to restrict yourself to mainline-included stuff, you can use BTRFS. But be sure to understand its (many) drawbacks and pita.



    Finally, an alternative solution is to use lvmthin to take regular backups (eg: with snapper), relying on third party tools (eg: bdsync, blocksync, etc) to copy/ship deltas only.



    A different approach would be to have two replicated machines (via DRBD) where you take indipendent snapshots via lvmthin.






    share|improve this answer
























    • What about postgres? Would to stop gitlab and postgres for a minute, so a consistant shapshot could be made? Ideally it would be great if postgres could be put in a read-only mode while the snapshot is made.

      – Sandra
      2 hours ago






    • 1





      @Sandra restoring from a filesystem snapshots should appear to postgresql (and any other properly written databases) as a generic "host crash" scenario, triggering its own recovery procedure (ie: committing to main database any partially written page). In other words, you do not need to put postgres into read-only mode when taking snapshots.

      – shodanshok
      1 hour ago











    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "2"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f952392%2fhow-to-big-scale-backup-gitlab%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    5














    I would review what you are backing up and possibly use a "multi-path" approach. For example, you could backup the Git repositories by constantly running through Git pulls on a backup servers. That would copy only the diff and leave you with a second copy of all Git repositories. Presumably you could detect new repos with the API.



    And use the "built-in" backup procedures to backup the issues, etc. I doubt that the 3TB comes from this part so you would be able to do backups very often at very little cost. You could also set up the PostgreSQL database with a warm standby with replication.



    Possibly your 3TB comes from container images in the Docker registry. Do you need to back those up? If so, then there may be a better approach just for that.



    Basically, I would recommend really looking at what it is that makes up your backup and backup the data in various parts.



    Even the backup tool from GitLab has options to include/exclude certain parts of the system such as the Docker Registry.






    share|improve this answer




























      5














      I would review what you are backing up and possibly use a "multi-path" approach. For example, you could backup the Git repositories by constantly running through Git pulls on a backup servers. That would copy only the diff and leave you with a second copy of all Git repositories. Presumably you could detect new repos with the API.



      And use the "built-in" backup procedures to backup the issues, etc. I doubt that the 3TB comes from this part so you would be able to do backups very often at very little cost. You could also set up the PostgreSQL database with a warm standby with replication.



      Possibly your 3TB comes from container images in the Docker registry. Do you need to back those up? If so, then there may be a better approach just for that.



      Basically, I would recommend really looking at what it is that makes up your backup and backup the data in various parts.



      Even the backup tool from GitLab has options to include/exclude certain parts of the system such as the Docker Registry.






      share|improve this answer


























        5












        5








        5







        I would review what you are backing up and possibly use a "multi-path" approach. For example, you could backup the Git repositories by constantly running through Git pulls on a backup servers. That would copy only the diff and leave you with a second copy of all Git repositories. Presumably you could detect new repos with the API.



        And use the "built-in" backup procedures to backup the issues, etc. I doubt that the 3TB comes from this part so you would be able to do backups very often at very little cost. You could also set up the PostgreSQL database with a warm standby with replication.



        Possibly your 3TB comes from container images in the Docker registry. Do you need to back those up? If so, then there may be a better approach just for that.



        Basically, I would recommend really looking at what it is that makes up your backup and backup the data in various parts.



        Even the backup tool from GitLab has options to include/exclude certain parts of the system such as the Docker Registry.






        share|improve this answer













        I would review what you are backing up and possibly use a "multi-path" approach. For example, you could backup the Git repositories by constantly running through Git pulls on a backup servers. That would copy only the diff and leave you with a second copy of all Git repositories. Presumably you could detect new repos with the API.



        And use the "built-in" backup procedures to backup the issues, etc. I doubt that the 3TB comes from this part so you would be able to do backups very often at very little cost. You could also set up the PostgreSQL database with a warm standby with replication.



        Possibly your 3TB comes from container images in the Docker registry. Do you need to back those up? If so, then there may be a better approach just for that.



        Basically, I would recommend really looking at what it is that makes up your backup and backup the data in various parts.



        Even the backup tool from GitLab has options to include/exclude certain parts of the system such as the Docker Registry.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered 2 hours ago









        ETLETL

        5,02711842




        5,02711842

























            3














            For such a short time between backups (1h), your best bet is to rely on filesystem-level snapshot and send/recv support.



            If using ZoL is not a problem in your environment, I would strongly advise to use it. ZFS is a very robust filesystem and you will really like all the extras (eg: compression) it offer. When coupled with sanoid/syncoid, it can provide a very strong backup strategy. The main disvantage is that it is not included into mainline kernel, so you need to install/update it separately.



            Alternatively, if you really need to restrict yourself to mainline-included stuff, you can use BTRFS. But be sure to understand its (many) drawbacks and pita.



            Finally, an alternative solution is to use lvmthin to take regular backups (eg: with snapper), relying on third party tools (eg: bdsync, blocksync, etc) to copy/ship deltas only.



            A different approach would be to have two replicated machines (via DRBD) where you take indipendent snapshots via lvmthin.






            share|improve this answer
























            • What about postgres? Would to stop gitlab and postgres for a minute, so a consistant shapshot could be made? Ideally it would be great if postgres could be put in a read-only mode while the snapshot is made.

              – Sandra
              2 hours ago






            • 1





              @Sandra restoring from a filesystem snapshots should appear to postgresql (and any other properly written databases) as a generic "host crash" scenario, triggering its own recovery procedure (ie: committing to main database any partially written page). In other words, you do not need to put postgres into read-only mode when taking snapshots.

              – shodanshok
              1 hour ago
















            3














            For such a short time between backups (1h), your best bet is to rely on filesystem-level snapshot and send/recv support.



            If using ZoL is not a problem in your environment, I would strongly advise to use it. ZFS is a very robust filesystem and you will really like all the extras (eg: compression) it offer. When coupled with sanoid/syncoid, it can provide a very strong backup strategy. The main disvantage is that it is not included into mainline kernel, so you need to install/update it separately.



            Alternatively, if you really need to restrict yourself to mainline-included stuff, you can use BTRFS. But be sure to understand its (many) drawbacks and pita.



            Finally, an alternative solution is to use lvmthin to take regular backups (eg: with snapper), relying on third party tools (eg: bdsync, blocksync, etc) to copy/ship deltas only.



            A different approach would be to have two replicated machines (via DRBD) where you take indipendent snapshots via lvmthin.






            share|improve this answer
























            • What about postgres? Would to stop gitlab and postgres for a minute, so a consistant shapshot could be made? Ideally it would be great if postgres could be put in a read-only mode while the snapshot is made.

              – Sandra
              2 hours ago






            • 1





              @Sandra restoring from a filesystem snapshots should appear to postgresql (and any other properly written databases) as a generic "host crash" scenario, triggering its own recovery procedure (ie: committing to main database any partially written page). In other words, you do not need to put postgres into read-only mode when taking snapshots.

              – shodanshok
              1 hour ago














            3












            3








            3







            For such a short time between backups (1h), your best bet is to rely on filesystem-level snapshot and send/recv support.



            If using ZoL is not a problem in your environment, I would strongly advise to use it. ZFS is a very robust filesystem and you will really like all the extras (eg: compression) it offer. When coupled with sanoid/syncoid, it can provide a very strong backup strategy. The main disvantage is that it is not included into mainline kernel, so you need to install/update it separately.



            Alternatively, if you really need to restrict yourself to mainline-included stuff, you can use BTRFS. But be sure to understand its (many) drawbacks and pita.



            Finally, an alternative solution is to use lvmthin to take regular backups (eg: with snapper), relying on third party tools (eg: bdsync, blocksync, etc) to copy/ship deltas only.



            A different approach would be to have two replicated machines (via DRBD) where you take indipendent snapshots via lvmthin.






            share|improve this answer













            For such a short time between backups (1h), your best bet is to rely on filesystem-level snapshot and send/recv support.



            If using ZoL is not a problem in your environment, I would strongly advise to use it. ZFS is a very robust filesystem and you will really like all the extras (eg: compression) it offer. When coupled with sanoid/syncoid, it can provide a very strong backup strategy. The main disvantage is that it is not included into mainline kernel, so you need to install/update it separately.



            Alternatively, if you really need to restrict yourself to mainline-included stuff, you can use BTRFS. But be sure to understand its (many) drawbacks and pita.



            Finally, an alternative solution is to use lvmthin to take regular backups (eg: with snapper), relying on third party tools (eg: bdsync, blocksync, etc) to copy/ship deltas only.



            A different approach would be to have two replicated machines (via DRBD) where you take indipendent snapshots via lvmthin.







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered 2 hours ago









            shodanshokshodanshok

            25.7k34085




            25.7k34085













            • What about postgres? Would to stop gitlab and postgres for a minute, so a consistant shapshot could be made? Ideally it would be great if postgres could be put in a read-only mode while the snapshot is made.

              – Sandra
              2 hours ago






            • 1





              @Sandra restoring from a filesystem snapshots should appear to postgresql (and any other properly written databases) as a generic "host crash" scenario, triggering its own recovery procedure (ie: committing to main database any partially written page). In other words, you do not need to put postgres into read-only mode when taking snapshots.

              – shodanshok
              1 hour ago



















            • What about postgres? Would to stop gitlab and postgres for a minute, so a consistant shapshot could be made? Ideally it would be great if postgres could be put in a read-only mode while the snapshot is made.

              – Sandra
              2 hours ago






            • 1





              @Sandra restoring from a filesystem snapshots should appear to postgresql (and any other properly written databases) as a generic "host crash" scenario, triggering its own recovery procedure (ie: committing to main database any partially written page). In other words, you do not need to put postgres into read-only mode when taking snapshots.

              – shodanshok
              1 hour ago

















            What about postgres? Would to stop gitlab and postgres for a minute, so a consistant shapshot could be made? Ideally it would be great if postgres could be put in a read-only mode while the snapshot is made.

            – Sandra
            2 hours ago





            What about postgres? Would to stop gitlab and postgres for a minute, so a consistant shapshot could be made? Ideally it would be great if postgres could be put in a read-only mode while the snapshot is made.

            – Sandra
            2 hours ago




            1




            1





            @Sandra restoring from a filesystem snapshots should appear to postgresql (and any other properly written databases) as a generic "host crash" scenario, triggering its own recovery procedure (ie: committing to main database any partially written page). In other words, you do not need to put postgres into read-only mode when taking snapshots.

            – shodanshok
            1 hour ago





            @Sandra restoring from a filesystem snapshots should appear to postgresql (and any other properly written databases) as a generic "host crash" scenario, triggering its own recovery procedure (ie: committing to main database any partially written page). In other words, you do not need to put postgres into read-only mode when taking snapshots.

            – shodanshok
            1 hour ago


















            draft saved

            draft discarded




















































            Thanks for contributing an answer to Server Fault!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f952392%2fhow-to-big-scale-backup-gitlab%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            Callistus I

            Image
            Callistus I Res apud Vicidata repertae: Nativitas : unknown value ; Roma Obitus : 223 ; Roma Pontificatus 217–222 Praecessor : Zephyrinus Successor : Urbanus I Sanctus Callistus qui ieiunium instituit; e libro manu scripto Francogallico saeculo XIV Callistus (natus die ignoto; obiit anno 222), fuit episcopus Romae et Papa Ecclesiae Catholicae Romanae ab anno 217. Vita | Secundum adversarium suum, presbyterem, fuit Hippolytus Callistus, servus argentarii Christiani. Papae Zephyrino, Callistus Romae arcessivit. Callistus fuit primus diaconus ac dux catacumbae apud Viam Appiam, quae hodie Sancti Callisti appellatur. Pontificatus | Nexus externi | Vicimedia Communia plura habent quae ad Callistus I spectant. Fontes Latinae de papis usque ad annum 530 (Papa Felix IV) Liber pontificalis Opera Omnia .mw-parser-output .stipula{padding:3px;background:#F7F8FF;border:1px solid grey;margin:auto}.mw-parser-output .stipula td.cell1{bac...
            Read more

            Tabula Rosettana

            Image
            Tabula multilinguis Rosettana in Museo Britannico ostenditur. Tabula Rosettana, [1] etiam titulo OGIS 90 agnita, est stela decreto de rebus sacris in Aegypto anno 196 a.C.n. lato inscripta. Tabula iuxta Rosettam Aegypti, urbem in delta Nili et ad oram maris Mediterranei iacentem, anno 1799 a milite Francico reperta est. Inventio stelae, linguis duabus scripturisque tribus inscriptae, eruditis Instituti Aegypti statim nuntiata est; ibi enim iussu imperatoris Napoleonis eruditi omnium scientiarum (sub aegide Commissionis Scientiarum et Artium) properaverant cum expeditione Francica. Qua a Britannis mox debellata, tabula Rosettana Londinium missa hodie apud Museum Britannicum iacet. Textus Graecus cito lectus interpretationi textuum Aegyptiorum (in formis hieroglyphica et demotica expressorum) gradatim adiuvit. Denique textum plene interpretatus est Ioannes Franciscus Champollion. Ab opere eruditorum cumulativo coepit hodiernus scripturae hieroglyphicae linguaeque Aegyptiae a...
            Read more

            How to label and detect the document text images

            Image
            1 $begingroup$ This is what I mean as document text image: I want to label the texts in image as separate blocks and my model should detect these labels as classes. NOTE: This is how the end result should be like: The labels like Block 1, Block 2, Block 3,.. should be Logo, Title, Date,.. Others, etc. Work done: First approach : I tried to implement this method via Object Detection, it didn't work. It didn't even detect any text. Second approach : Then I tried it using PixelLink. As this model is build for scene text detection, it detected each and every text in the image. But this method can detect multiple lines of text if the threshold values are increased. But I have no idea how do I add labels to the text blocks. PIXEL_CLS_WEIGHT_all_ones = 'PIXEL_CLS_WEIGHT_all_ones' PIXEL_C...
            Read more