MQTT Message lost/not found from ESP8266 to Embedded MQTT Broker [RESOLVED]

Running on OpenHab 2.x via Linux Mint

Hey Everyone!
So, I’ve run into an issue again, and I think it’s due to lack of experience or full understanding, but I really need some extra eyes and some advice to help determine A. if I’m diagnosing the issue properly, and B. how to solve the issue. Game Development’s what I’m used to, so working with Arduino and going through the debugging process has been entirely different from that of a video game. I’m struggling to determine what the issues are when the console says everything is fine & my code compiles.

Goal:
Using a NodeMCU ESP8266 with code I made on Arduino, I’m trying to send over the soil moisture percentage from my capacitive sensor to the embedded broker on OpenHab.
Eventually, I would like to program OpenHab to do stuff with the data once it receives the message, but that comes -after- I figure out how to send it to the broker in the first place.

The Problems:

  1. On occassion there’ll be intermittent network connection failures, so every minute or so the ESP will have to reconnect to the broker, and I have no clue why this is happening nor if I can prevent it. This isn’t the biggest priority so long as I can get the client to successfully send a message to the broker so that OpenHab can reasonably use it. and yes the clientid is unique to my knowledge.
  2. At least in every step of my Arduino code and the output console, it says it successfully sent the message, but neither MQTT Explorer nor MQTT.fx receives the message when they’re subscribed to the topic that my ESP is supposed to be publishing to. Essentially, my ESP is telling me it’s sending messages fine but it’s as if the broker is never actually receiving these messages.
    When I publish to the same topic via MQTT.fx however, those messages appear fine.
    I thought setting the client.publish to send retain messages that would resolve it but still nothing.
    and PubSubClient on Arduino appears to only be able to send QoS 0 messages so if it’s not appearing because of that then I have no clue how to solve this problem, but I don’t feel like that’s the cause of the issue either.
    Here’s my code:
#include <SPI.h>
#include <Wire.h>
#include <WiFiClient.h>
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

//WiFi Credentials (probably should include these in a secret .h file later)
char ssid[] = "XXXXXX"; // IoT devices can only connect to 2.4 GHz wifi, DO NOT connect to 5GHz it will fail
char pass[] = "XXXXXXXX";


int SensorPin = A0;
int SoilValue = 0;
int SoilPercent = 0; //this takes SoilValue, and converts it to a 0-100% based on AirValue & WaterValue
int AirValue = 866; //These two values are HAND MEASURED to determine air (no water) vs. full count of water.
int WaterValue = 540;
String Data;


WiFiClient wifiClient;
PubSubClient client(wifiClient);
const char broker[] = "XXXXXXXX";
int port = 1883;
const char topic[] = "tele/SoilSensor01";
const char clientID[] = "SoilSensor01";


const long interval = 1000;
unsigned long previousMillis = 0;

int count = 0;

void setup() {
  //Initialize serial and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  
  // attempt to connect to Wifi network:
  Serial.print("Attempting to connect to WPA SSID: ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    // failed, retry
    Serial.print(".");
    delay(5000);
  }

  Serial.println("You're connected to the network");
  Serial.println();

  Serial.print("Attempting to connect to the MQTT broker: ");
  Serial.println(broker);
  Serial.println("...");

  
  client.setServer(broker, port);
  client.setCallback(callback);
  client.connect(clientID);
  client.subscribe(topic);
  Serial.println("Connection Status: " + String(client.state()));
  if (!client.connected()) {
    reconnect();
  }
  Serial.println("You're connected to the MQTT broker!");
  
  Serial.println();

}

void loop() {
  client.loop();

  if (!client.connected()) {
    reconnect();
  }
  // avoid having delays in loop, we'll use the strategy from BlinkWithoutDelay
  // see: File -> Examples -> 02.Digital -> BlinkWithoutDelay for more info
  unsigned long currentMillis = millis();
  SoilValue = analogRead(SensorPin);
  SoilPercent = map(SoilValue, AirValue, WaterValue, 0, 100);
  if (currentMillis - previousMillis >= interval) {
    // save the last time a message was sent
    previousMillis = currentMillis;

    Data = String(SoilPercent);
    Serial.print("Sending message to topic: ");
    Serial.println(topic);
    Serial.println("Percent Value: " + Data + "%");
    
    //This allows the OpenHAB broker to read the message
    const char* buffer = Data.c_str();
    String JSONconverter = String("{\"moisture\": ") + String(buffer) + String("}");
    const char* ToJSON = JSONconverter.c_str();
    if(client.publish(topic, ToJSON, true) == true) {
        Serial.println("Message Succeeded!");
    }
    else if(client.publish(topic, ToJSON, true) == false) {
        Serial.println("Message Failed to Send!");
    }
    Serial.println();
    count++;
  }
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect(clientID)) {
      Serial.println("connected");
      client.subscribe(topic);
    } else {
      Serial.print("failed, state: ");
      Serial.println(client.state());
      Serial.println("attempting reconnection...");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

The thing is, if my code’s fine, then what exactly is going on? Why don’t the messages appear on MQTT.fx when I subscribe to the topic “tele/SoilSensor01”, or why don’t they appear anywhere for that matter? Am I not looking in the right place, or is this an issue with the embedded broker?
If you all think this is an issue due to the broker’s configuration, that’s already a huge step for me to figure this out! I’m not posting this on the server itself but my main computer that I use program with, so let me know if you need to see some of the openhab files and I can log in there and post it here.
Best,
-MTX

Do you see Message Succeeded! while monitoring the 8266 serial port?

Yep! it’s been saying “Message Succeeded!” Each time, and at least according to the API it only returns true if the publish succeeds. That’s at least why I’m assuming it’s actually getting published to the broker.

Can you test that it is publishing correcty by changing the topic and message in this line

if(client.publish(topic, ToJSON, true)

To somesthing like

if(client.publish(“test/one”, “hello world”, false)

Oh weird! So once I changed that, the message appeared in the MQTT monitors, but it only appears to be sent once. It’s not picking up any messages after that, even when it reconnects to the broker.

I suspect your currentMillis is a small number on restart and your previousMillis is a large number so it may wait for a while before it starts working.

Well, the problem with that is that the serial port is still publishing frequently around every second, meaning that the console is telling me “message succeeded!” every second, but the message only seems to get picked up once. I can reset the 8266 and it’ll publish a new message once, but after that nothing. If the moisture value changes, you can see it in real time on the console, but not on the broker.

The embeded broker in oh is not very good. Most people use mosquitto.

It will take some more debugging to find all the issues. (I on my phone)

You did get the topic and message correct into MQTT?

You can also achive this using tasmota

Yeah, I’m considering I might have to switch to mosquitto if problems like this persist.
I appreciate the help! And yes; I can verify I got the right topic and message into MQTT. It’s at least in the right spot.
And I’ve thought about using tasmota, which I might try out if there are no other options for this, but I’m also using this as a learning experience and I really want to see if I can build my own code using Arduino instead of having to upload someone else’s firmware. My thought process is that if I can set something like this up, then I can do all kinds of other cool stuff that something like tasmota wouldn’t do.

Ok np keep up whatever you want to learn.

So you are getting the correct message but only once in the broker.

Are you using mqtt explorer to look at this?

Yep!
I’m using MQTT explorer on my main computer, and MQTT.fx on the Openhab machine. Both receive the correct message once when the chip restarts.

Try adding a serial pl before message successful printing the topic and message.

Do you mean a serial.println()? the println is correctly showing up in the arduino console; I placed it right before message succeeded inside the if() statement and right before the if() statement itself. both show up.

So you can see in the serial it should be working.

I think its time to try mosquitto

Okey Doke!
So I was able to switch over to using mosquitto and set that up in openhab, at least I believe so. removed the moquette broker, so the embedded MQTT broker doesn’t appear anywhere in openhab. Got the mosquitto up and running on the same machine so I wouldn’t have to change much else (aka it still uses the same ip address).
The broker’s up and running fine, but unfortunately the chip is still only sending a message once in its lifetime.
I even tried to use another 8266 just in case to see if that would change anything (still a NodeMCU like the original), but the same issue occurs where I see it working in the serial but not in the broker.
When I get a chance myself, I’m going to try and use beginPublish() and client.print() that way to see if I get more success. Otherwise, I’m not sure if I should try out a different library or work through more diagnostics with openhab and the broker. I’m open to any other ideas/suggestions to figure out what’s going on and see what else I can test out.
Again thank you so much for the support and help!
I still want to get this to work myself, but since i have a few 8266s lying around I might install tasmota on 1 just to see if it interacts with the broker the same way as well.

Mosquitto does have its own log. If it is not seeing incoming messages from your ESP, there’s no point fiddling around in openHAB.

1 Like

I would suggest two changes to help debug this. First, remove the “if(client.publish(topic, ToJSON, true) == false)” after the publish. There is no need to test again – either the first publish was successful or it wasn’t. That second test is actually attempting to publish a second time.

Second, for debugging I would add a println of the topic and a println for the payload after the “message successful” println.

Change this:

if(client.publish(topic, ToJSON, true) == true) {
        Serial.println("Message Succeeded!");
    }
    else if(client.publish(topic, ToJSON, true) == false) {
        Serial.println("Message Failed to Send!");
    }

To this:

if(client.publish(topic, ToJSON, true) == true) {
        Serial.println("Message Succeeded!");
        Serial.println(topic);
        Serial.println(ToJason);
    }
    else {
        Serial.println("Message Failed to Send!");
    }

Okey doke, cleaned up that code and added the printlns. Thanks for that!
So far, it is still displaying message succeeded each time, and the printlns display the correct information.
So, interesting discovery: it DID successfully send a few messages overnight while I was asleep and AFK. From the time I left it alone, about 6 hours later it posted a message, and subsequently sent one message around every hour or so for about 4 hours until going dark again.
I have no clue why nor how it was able to do that, but lo and behold as I was reading the replies from you guys I discovered this in MQTT Explorer. It’s still really odd behavior considering it should be posting every second, and shouldn’t just be successful via extremely intermittent points in time, but we’re getting closer!
Thank you all so much for helping me work through this!

Just to check - do you have anything else which might be trying to connect to your MQTT broker with the same client ID?

I’ve had intermittent issue in the past and found them to be caused by something being overwhelmed. Could be the Arduino, the broker or the network. I would try slowing things down a bit. I would suggest moving the population of SoilValue and SoilPercent inside the once per second IF statement. Plus I would store the last value published and only publish again if the value has changed. (Actually I publish on change or every five minutes to accommodate OpenHab restarts (I don’t use Retain).

Comparing your code to some code that I have I found a few differences. I’m not saying any of these are the issue, just different.

You use

const char topic[] = "tele/SoilSensor01";
const char clientID[] = "SoilSensor01";

and I use

const char* topic = "tele/SoilSensor01";
const char* clientID = "SoilSensor01";

You have client.loop() before you check if client.connected, I have it after. And I have the Retain set to false on the publish.

HTH

1 Like