MQTT topics and how exactly mosquitto broker deal with them

,
1

I don’t know if it is appropiate to post this here but here we go.

I wanna publish a lot of sensor data with mqtt to OpenHAB but I don’t know if my ESP8266 will support it. I have like a total of 16 sensors (temp,Hum,Lux,PIR) distributed in 3 floors, each sensor will provide a 2 byte data (PIR only 1 byte) and I am gonna publish it with 1 second samplig. The problem is that I don’t know how exactly works the PubSubClient.h, if it will send all the data at the same time or will send it by parts and how Mosquitto broker will send it to OpenHAB.

If by any chance I clear the payload char each time the ESP8266 published 4 data sensors per example, it will help or not?

Here is a part of the code, don’t mind some values please it’s not finished yet. I post only this part so the entire code is very large because it receives data from 4 Xbee’s and other things. If anyone could help me I appreciated.

char msg[50];  // Not sure about the exact size of the payload, I don't know if it will publish all the data at the same time or what 

// Dont mind please, it is not finihed yet
float tempC, Humidity, Lux = 0; 
int pir_int = 0;

float tempC, Humidity, Lux = 0; 
int pir_int = 0;

float tempC, Humidity, Lux = 0; 
int pir_int = 0;

float tempC, Humidity, Lux = 0; 
int pir_int = 0;

// Declaració dels topics mqtt
// Soterrani
#define st_temp "casa/st/soterrani/temperatura"
#define st_hum "casa/st/soterrani/humitat"
#define st_pir "casa/st/soterrani/pir"
#define st_llums "casa/st/soterrani/llums" // per codi actuador
#define st_lux "casa/st/soterrani/lux"

#define st_rent "casa/st/soterrani/rentadora" // per codi actuador
#define st_cald "casa/st/soterrani/caldera" // per codi actuador


// Cuina
#define st_temp "casa/pb/cuina/temperatura"
#define st_hum "casa/pb/cuina/humitat"
#define st_llums "casa/pb/cuina/llums" // per codi actuador
#define st_co "casa/pb/cuina/co"

// Saló
#define st_temp "casa/pb/salo/temperatura"
#define st_hum "casa/pb/salo/humitat"
#define st_pir "casa/pb/salo/pir"
#define st_llums "casa/pb/salo/llums" // per codi actuador
#define st_alarma "casa/pb/salo/alarma" // per codi actuador
#define st_lux "casa/pb/salo/lux"

// Dormitori
#define st_temp "casa/pp/dormitori/temperatura"
#define st_hum "casa/pp/dormitori/humitat"
#define st_pir "casa/pp/dormitori/pir"
#define st_llums "casa/pp/dormitori/llums" // per codi actuador
#define st_lux "casa/pp/dormitori/lux"

// Oficina
#define st_temp "casa/pp/oficina/temperatura"
#define st_hum "casa/pp/oficina/humitat"
#define st_llums "casa/pp/oficina/llums" // per codi actuador
#define st_lux "casa/pp/oficina/lux"

void loop()
{
  // Fem la primera connexió del client i les següents que siguin necessàries
  if (!client.connected()) {
    reconnect();
  }

    // Inicialitzem la funció de tractament de dades 
  parseData();


  // Publiquem les dades dels sensors cada un cert temps
  long now = millis();
  if (now - lastMsg > 29000) {
    lastMsg = now;
    // tractem les variables amb 2 enters i 2 decimals de precisió i les convertim a string
    // Soterrani
    String Pir_s = String(pir_int);
    client.publish(st_pir, Pir_s);
    client.publish(st_temp, dtostrf(tempC, 4, 2, msgbuf));
    client.publish(st_hum, dtostrf(Humidity, 4, 2, msgbuf));
    client.publish(st_lux, dtostrf(Lux, 4, 2, msgbuf); // revisar


// Not finished yet, here are the other sensor data I wanna publish

    String Pir_s = String(pir_int);
    client.publish(st_pir, Pir_s);
    client.publish(st_temp, dtostrf(tempC, 4, 2, msgbuf));
    client.publish(st_hum, dtostrf(Humidity, 4, 2, msgbuf));
    client.publish(st_lux, dtostrf(Lux, 4, 2, msgbuf); // revisar

    String Pir_s = String(pir_int);
    client.publish(st_pir, Pir_s);
    client.publish(st_temp, dtostrf(tempC, 4, 2, msgbuf));
    client.publish(st_hum, dtostrf(Humidity, 4, 2, msgbuf));
    client.publish(st_lux, dtostrf(Lux, 4, 2, msgbuf); // revisar

    String Pir_s = String(pir_int);
    client.publish(st_pir, Pir_s);
    client.publish(st_temp, dtostrf(tempC, 4, 2, msgbuf));
    client.publish(st_hum, dtostrf(Humidity, 4, 2, msgbuf));
    client.publish(st_lux, dtostrf(Lux, 4, 2, msgbuf); // revisar


    
  }

  // Fem una connexió en cada loop amb client per refrescar dades per publicar i mantenir la connexió 
  client.loop();

  
}
2

I can help with some of this. But the tl;dr is going to be who cares? Even an ESP8266 should be able to handle 16 messages per second. Mosquitto running on an RPi 0 can handle orders of magnitude more message traffic than that and OH certainly can as well. Don’t try to fix performance problems until you know it’s going to be a problem.

All the messages are adjudicated by the broker. Your ESPs will never be communicating directly with OH nor will OH be communicating directly with your ESPs. Each only ever talks to the broker.

How messages are sent and received depends on the Quality of Service set on the message.

QOS What it means What happens
0 At most once The client attempts to publish the message to the broker. If the broker gets it, great! If not the message is dropped.
1 At least once The client attempts to publish the message to the broker and waits a time for an acknowledgement that the message was received. If the ack doesn’t come in time, the message is sent again.
2 Exactly once There is a protocol of hand shakes and acks to ensure that the message is sent and received exactly once.

Obviously the higher the QOS, the more overhead there is involved to publish a message. Note that the subscriber also sets a QOS on the messages it receives. And that QOS does not need to match the QOS that was used when the message was published. So the ESP could publish at QOS 0 and OH could subscribe at QOS 2.

For sensor readings that report frequently, QOS 0 is usually more than adequate.

If the broker gets bogged down for some reason, it will start dropping QOS 0 messages. QOS 1 and 2 messages might be delayed.

MQTT messages are published as a single message, but whether or not that message gets split up into multiple packets is handled at a much lower level of the TCP/IP protocol stack. What I don’t know is whether the library being used on the ESP caches up multiple messages or sends them one at a time. I’m not entirely certain it matters either way. But you’ll have to look at the code or experiment to figure that out. In particular look at the publish and loop functions to see what it does. I suspect it caches all the “published” messages and then iterates over all the cached messages, sending each as a separate message to the broker individually.

Note that I don’t think openHAB can handle binary messages. It looks for Strings, even if the message is a number. But I could be wrong on that. Though it does look like you are converting the data to Strings anyway. So you’ll be sending several bytes per sensor reading, but that is going to be dwarfed by all the header and proforma required to construct an MQTT message in the first place.

You should review MQTT Essentials - All Core Concepts explained which is one of the best tutorials/explainers for how MQTT works on the web.

Having said all of that, this is a case of premature optimization. A second is forever, even for an ESP8266. Publishing 16 messages a second is nothing. You could probably publish 100 messages a second before you start to see any impacts at all and probably thousands before you need to do something about it. If you have a performance problem, it’s almost certainly not going to be MQTT that is the cause of the problem at these rates. Focus on making it work. Worry about performance later and only if performance becomes a problem.

But from a design perspective you should probably publish the PIR sensors immediately rather than waiting for a second. For slow changing values like temp and humidity you could drop it to once every 30 seconds and probably not suffer any negative consequences so if you do have a performance problem those are the sorts of places to look for improvements.

You could also save some load by constructing a single message that has all the values every second and then parse the values back out on the subscriber side, but that goes against the one of the MQTT guiding principles (Never impose extra work on the subscribers.)

2 Likes
3

Hi Rich, first of all, thanks for your quick and elaborated response.

I am still learning on fixing that bad habit of mine hahaha.

Great point, I didn’t know that.

Hopefully is like this, It must be like this or there will be no sense on using mqtt for home automation projects like mine where there are multiple sensor data.

Understood, first try and experiment, later fix only if necessary.

That’s the idea, I have PIR sensors in rules and I need them to be quick.

Thanks, I will check it later.

Exactly, I don’t wanna load Openhab with unncessary things. I have one ESP for the sensors and other one for the actuators. The sensor data is sent by some XBee to the ESP and then to the broker, later it’s showed on OpenHAB.

4

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.