How does the MQTT server output a single raw file from multiple client publications?












1















I'm trying my first IoT project whereby I want to:




  • have an electret microphone capture audio

  • have an ESP8266 NodeMCU 12-E board submit captured audio to a remote server

  • have a remote server receive the audio data using an MQTT server

  • publish the audio data as a .WAV file on the server


I saw someone online demonstrate something similar in this YouTube video



https://www.youtube.com/watch?v=rU_Pw9Jb_PM



The author shared the project on github here



https://github.com/hjltu/esp8266-wifi-microphone



When I study the code, I think what I see is the author taking the value of analogRead(A) and appending it to some kind of string as a payload, which is then published to an MQTT server.



I see that the author expects the MQTT server or some other software to process the ESP8266 microphone audio data and output it as a .RAW file. This RAW file is eventually converted to a .WAV file with the help of ffmpeg.



My question is this: What command allows MQTT server to generate the .RAW file? Or is this done by an entirely different software? And it appears to me that for a single recording/audio file, the my_record() of esp8266-wifi-mic.ino file will send multiple payloads to the MQTT server. So how does the MQTT server know which published transmission belongs to which RAW file?










share|improve this question







New contributor




John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

























    1















    I'm trying my first IoT project whereby I want to:




    • have an electret microphone capture audio

    • have an ESP8266 NodeMCU 12-E board submit captured audio to a remote server

    • have a remote server receive the audio data using an MQTT server

    • publish the audio data as a .WAV file on the server


    I saw someone online demonstrate something similar in this YouTube video



    https://www.youtube.com/watch?v=rU_Pw9Jb_PM



    The author shared the project on github here



    https://github.com/hjltu/esp8266-wifi-microphone



    When I study the code, I think what I see is the author taking the value of analogRead(A) and appending it to some kind of string as a payload, which is then published to an MQTT server.



    I see that the author expects the MQTT server or some other software to process the ESP8266 microphone audio data and output it as a .RAW file. This RAW file is eventually converted to a .WAV file with the help of ffmpeg.



    My question is this: What command allows MQTT server to generate the .RAW file? Or is this done by an entirely different software? And it appears to me that for a single recording/audio file, the my_record() of esp8266-wifi-mic.ino file will send multiple payloads to the MQTT server. So how does the MQTT server know which published transmission belongs to which RAW file?










    share|improve this question







    New contributor




    John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.























      1












      1








      1








      I'm trying my first IoT project whereby I want to:




      • have an electret microphone capture audio

      • have an ESP8266 NodeMCU 12-E board submit captured audio to a remote server

      • have a remote server receive the audio data using an MQTT server

      • publish the audio data as a .WAV file on the server


      I saw someone online demonstrate something similar in this YouTube video



      https://www.youtube.com/watch?v=rU_Pw9Jb_PM



      The author shared the project on github here



      https://github.com/hjltu/esp8266-wifi-microphone



      When I study the code, I think what I see is the author taking the value of analogRead(A) and appending it to some kind of string as a payload, which is then published to an MQTT server.



      I see that the author expects the MQTT server or some other software to process the ESP8266 microphone audio data and output it as a .RAW file. This RAW file is eventually converted to a .WAV file with the help of ffmpeg.



      My question is this: What command allows MQTT server to generate the .RAW file? Or is this done by an entirely different software? And it appears to me that for a single recording/audio file, the my_record() of esp8266-wifi-mic.ino file will send multiple payloads to the MQTT server. So how does the MQTT server know which published transmission belongs to which RAW file?










      share|improve this question







      New contributor




      John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.












      I'm trying my first IoT project whereby I want to:




      • have an electret microphone capture audio

      • have an ESP8266 NodeMCU 12-E board submit captured audio to a remote server

      • have a remote server receive the audio data using an MQTT server

      • publish the audio data as a .WAV file on the server


      I saw someone online demonstrate something similar in this YouTube video



      https://www.youtube.com/watch?v=rU_Pw9Jb_PM



      The author shared the project on github here



      https://github.com/hjltu/esp8266-wifi-microphone



      When I study the code, I think what I see is the author taking the value of analogRead(A) and appending it to some kind of string as a payload, which is then published to an MQTT server.



      I see that the author expects the MQTT server or some other software to process the ESP8266 microphone audio data and output it as a .RAW file. This RAW file is eventually converted to a .WAV file with the help of ffmpeg.



      My question is this: What command allows MQTT server to generate the .RAW file? Or is this done by an entirely different software? And it appears to me that for a single recording/audio file, the my_record() of esp8266-wifi-mic.ino file will send multiple payloads to the MQTT server. So how does the MQTT server know which published transmission belongs to which RAW file?







      mqtt esp8266






      share|improve this question







      New contributor




      John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      share|improve this question







      New contributor




      John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this question




      share|improve this question






      New contributor




      John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      asked 2 hours ago









      JohnJohn

      1084




      1084




      New contributor




      John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      John is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






















          1 Answer
          1






          active

          oldest

          votes


















          2














          You are correct about the microphone input, void my_record() samples the microphone output level 1000 times, appending each reading to a string variable and publishes the resulting string to an MQTT broker.



          This process repeats 11 times every time my_record() is called.



          note: You are sort of misunderstanding about the .RAW file. It is a raw file, meaning that it is unprocessed and unformatted .... just a stream of bytes. Using the term .RAW implies a file name extension.



          What command allows MQTT server to generate the .RAW file?



          The MQTT broker (server) does not generate the raw file, the file is published to the MQTT broker by an outside source, the ESP8266 in this instance.



          So how does the MQTT server know which published transmission belongs to which RAW file?



          It does not know. All it does is to relay messages. It is up to the publisher to send to the correct topic and it is up to the subscriber to watch the data at the correct topic.



          The messages could arrive to the subscriber out of sequence, so a sequence number needs to be included with the message if a correct data sequence is desired.



          Have a look at these for a visual demo of MQTT messages.



          https://shiftr.io/shiftr-io/demo ... you can only subscribe to this one



          https://shiftr.io/try or https://shiftr.io/shiftr-io/try ... you can publish to this one (and subscribe)



          You can get your own account and watch your own messages being sent and received without the clutter of other messages.






          share|improve this answer










          New contributor




          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





















          • Oh....I see. So that means I need to create another client application to read subscription data from MQTT server, and save content to a .raw file?

            – John
            1 hour ago






          • 1





            exactly correct ..... the MQTT server is just a long pipe for your data

            – jsotola
            1 hour ago











          • data can arrive out of sequence though ..... updated answer

            – jsotola
            1 hour ago











          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "666"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });






          John is a new contributor. Be nice, and check out our Code of Conduct.










          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fiot.stackexchange.com%2fquestions%2f3838%2fhow-does-the-mqtt-server-output-a-single-raw-file-from-multiple-client-publicati%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          2














          You are correct about the microphone input, void my_record() samples the microphone output level 1000 times, appending each reading to a string variable and publishes the resulting string to an MQTT broker.



          This process repeats 11 times every time my_record() is called.



          note: You are sort of misunderstanding about the .RAW file. It is a raw file, meaning that it is unprocessed and unformatted .... just a stream of bytes. Using the term .RAW implies a file name extension.



          What command allows MQTT server to generate the .RAW file?



          The MQTT broker (server) does not generate the raw file, the file is published to the MQTT broker by an outside source, the ESP8266 in this instance.



          So how does the MQTT server know which published transmission belongs to which RAW file?



          It does not know. All it does is to relay messages. It is up to the publisher to send to the correct topic and it is up to the subscriber to watch the data at the correct topic.



          The messages could arrive to the subscriber out of sequence, so a sequence number needs to be included with the message if a correct data sequence is desired.



          Have a look at these for a visual demo of MQTT messages.



          https://shiftr.io/shiftr-io/demo ... you can only subscribe to this one



          https://shiftr.io/try or https://shiftr.io/shiftr-io/try ... you can publish to this one (and subscribe)



          You can get your own account and watch your own messages being sent and received without the clutter of other messages.






          share|improve this answer










          New contributor




          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





















          • Oh....I see. So that means I need to create another client application to read subscription data from MQTT server, and save content to a .raw file?

            – John
            1 hour ago






          • 1





            exactly correct ..... the MQTT server is just a long pipe for your data

            – jsotola
            1 hour ago











          • data can arrive out of sequence though ..... updated answer

            – jsotola
            1 hour ago
















          2














          You are correct about the microphone input, void my_record() samples the microphone output level 1000 times, appending each reading to a string variable and publishes the resulting string to an MQTT broker.



          This process repeats 11 times every time my_record() is called.



          note: You are sort of misunderstanding about the .RAW file. It is a raw file, meaning that it is unprocessed and unformatted .... just a stream of bytes. Using the term .RAW implies a file name extension.



          What command allows MQTT server to generate the .RAW file?



          The MQTT broker (server) does not generate the raw file, the file is published to the MQTT broker by an outside source, the ESP8266 in this instance.



          So how does the MQTT server know which published transmission belongs to which RAW file?



          It does not know. All it does is to relay messages. It is up to the publisher to send to the correct topic and it is up to the subscriber to watch the data at the correct topic.



          The messages could arrive to the subscriber out of sequence, so a sequence number needs to be included with the message if a correct data sequence is desired.



          Have a look at these for a visual demo of MQTT messages.



          https://shiftr.io/shiftr-io/demo ... you can only subscribe to this one



          https://shiftr.io/try or https://shiftr.io/shiftr-io/try ... you can publish to this one (and subscribe)



          You can get your own account and watch your own messages being sent and received without the clutter of other messages.






          share|improve this answer










          New contributor




          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





















          • Oh....I see. So that means I need to create another client application to read subscription data from MQTT server, and save content to a .raw file?

            – John
            1 hour ago






          • 1





            exactly correct ..... the MQTT server is just a long pipe for your data

            – jsotola
            1 hour ago











          • data can arrive out of sequence though ..... updated answer

            – jsotola
            1 hour ago














          2












          2








          2







          You are correct about the microphone input, void my_record() samples the microphone output level 1000 times, appending each reading to a string variable and publishes the resulting string to an MQTT broker.



          This process repeats 11 times every time my_record() is called.



          note: You are sort of misunderstanding about the .RAW file. It is a raw file, meaning that it is unprocessed and unformatted .... just a stream of bytes. Using the term .RAW implies a file name extension.



          What command allows MQTT server to generate the .RAW file?



          The MQTT broker (server) does not generate the raw file, the file is published to the MQTT broker by an outside source, the ESP8266 in this instance.



          So how does the MQTT server know which published transmission belongs to which RAW file?



          It does not know. All it does is to relay messages. It is up to the publisher to send to the correct topic and it is up to the subscriber to watch the data at the correct topic.



          The messages could arrive to the subscriber out of sequence, so a sequence number needs to be included with the message if a correct data sequence is desired.



          Have a look at these for a visual demo of MQTT messages.



          https://shiftr.io/shiftr-io/demo ... you can only subscribe to this one



          https://shiftr.io/try or https://shiftr.io/shiftr-io/try ... you can publish to this one (and subscribe)



          You can get your own account and watch your own messages being sent and received without the clutter of other messages.






          share|improve this answer










          New contributor




          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.










          You are correct about the microphone input, void my_record() samples the microphone output level 1000 times, appending each reading to a string variable and publishes the resulting string to an MQTT broker.



          This process repeats 11 times every time my_record() is called.



          note: You are sort of misunderstanding about the .RAW file. It is a raw file, meaning that it is unprocessed and unformatted .... just a stream of bytes. Using the term .RAW implies a file name extension.



          What command allows MQTT server to generate the .RAW file?



          The MQTT broker (server) does not generate the raw file, the file is published to the MQTT broker by an outside source, the ESP8266 in this instance.



          So how does the MQTT server know which published transmission belongs to which RAW file?



          It does not know. All it does is to relay messages. It is up to the publisher to send to the correct topic and it is up to the subscriber to watch the data at the correct topic.



          The messages could arrive to the subscriber out of sequence, so a sequence number needs to be included with the message if a correct data sequence is desired.



          Have a look at these for a visual demo of MQTT messages.



          https://shiftr.io/shiftr-io/demo ... you can only subscribe to this one



          https://shiftr.io/try or https://shiftr.io/shiftr-io/try ... you can publish to this one (and subscribe)



          You can get your own account and watch your own messages being sent and received without the clutter of other messages.







          share|improve this answer










          New contributor




          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.









          share|improve this answer



          share|improve this answer








          edited 1 hour ago





















          New contributor




          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.









          answered 1 hour ago









          jsotolajsotola

          1366




          1366




          New contributor




          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





          New contributor





          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.






          jsotola is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.













          • Oh....I see. So that means I need to create another client application to read subscription data from MQTT server, and save content to a .raw file?

            – John
            1 hour ago






          • 1





            exactly correct ..... the MQTT server is just a long pipe for your data

            – jsotola
            1 hour ago











          • data can arrive out of sequence though ..... updated answer

            – jsotola
            1 hour ago



















          • Oh....I see. So that means I need to create another client application to read subscription data from MQTT server, and save content to a .raw file?

            – John
            1 hour ago






          • 1





            exactly correct ..... the MQTT server is just a long pipe for your data

            – jsotola
            1 hour ago











          • data can arrive out of sequence though ..... updated answer

            – jsotola
            1 hour ago

















          Oh....I see. So that means I need to create another client application to read subscription data from MQTT server, and save content to a .raw file?

          – John
          1 hour ago





          Oh....I see. So that means I need to create another client application to read subscription data from MQTT server, and save content to a .raw file?

          – John
          1 hour ago




          1




          1





          exactly correct ..... the MQTT server is just a long pipe for your data

          – jsotola
          1 hour ago





          exactly correct ..... the MQTT server is just a long pipe for your data

          – jsotola
          1 hour ago













          data can arrive out of sequence though ..... updated answer

          – jsotola
          1 hour ago





          data can arrive out of sequence though ..... updated answer

          – jsotola
          1 hour ago










          John is a new contributor. Be nice, and check out our Code of Conduct.










          draft saved

          draft discarded


















          John is a new contributor. Be nice, and check out our Code of Conduct.













          John is a new contributor. Be nice, and check out our Code of Conduct.












          John is a new contributor. Be nice, and check out our Code of Conduct.
















          Thanks for contributing an answer to Internet of Things Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fiot.stackexchange.com%2fquestions%2f3838%2fhow-does-the-mqtt-server-output-a-single-raw-file-from-multiple-client-publicati%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          How to label and detect the document text images

          Vallis Paradisi

          Tabula Rosettana