Hello Sara (and other interested developers),

In a rather simple way we can improve the quality of LoRa-transmissions in your nice LoRa-project (see your course “Learn ESP with Arduino IDE“, modules #6 and #8).

This can be done by transmitting every package twice (sender) and implement at the receiver side a little bit extra code to ignore/process this 2nd transmission if 1st one was received successful or not.

Here an explanation.

As you know, Sara, from our posts about this subject (see: “connection minus Solar Power Station to GND ESP32 missing?“,  “ESP32 pin 15 oneWire for DS18B20 conflict with LoRa” and “SD Library corrupt?“) I did extensive long lasting tests (for weeks) with one LoRa-sender and two LoRa-receivers.

I used the info as given in your project “ESP32 Publish Sensor Readings to Google Sheets (ESP8266 Compatible)” to retain the results of all LoRa-transmissions in a Google Sheet (GSheet). To bypass the limitation to maximal 3 data fields in a GSheet, I packed several less important values together in my first IFTTT-field (3th field in table below). This limitation is not imposed by the IFTTT-platform but by the Google Sheet system (see my post: IFTTT troubles).

Your original LoRa-project was apparently meant to “keep an eye” on the vegetable garden of Rui and you (see the pictures illustrating the environment for your sender), so missing some data packets or packages with corrupted content is not that important. Enough data for guarding this distant situation from your home to check if the soil is not getting too dry.

Things change if you want to use these LoRa-measurements to compare with other data or to show them on internet in the public area, as you demonstrate with these RNT-projects “ESP32/ESP8266 Plot Sensor Readings in Real Time Charts – Web Server” and “ESP32/ESP8266 Insert Data into MySQL Database using PHP and Arduino IDE“. Missing now and then some samples is here not convenient and gives a bit an unreliable impression to a visitor.

In my case I wanted to have all samples every 10 minutes also for comparing with other data (f.e. Dutch KNMI – weather forecast institute) exactly on every 10-minute-moment in an hour. So I added a RTC-module (Real Time Clock) to my LoRa-sender. I used the DS3231 as this is on, the longer terms, more precise as the DS1307 (see project “Guide for Real Time Clock (RTC) Module with Arduino (DS1307 and DS3231)“). To set the initial date+time I used your project “Getting Date and Time with ESP32 on Arduino IDE (NTP Client)“. The way I changed my sender code for this and adapted the calculations for the deep-sleep-time of the ESP32 is beyond my stuff for this post.

I tested for 3 weeks – during my holiday, so unattended – this LoRa-system, consisting of 1 sender (TTGO ESP32 with built-in LoRa-module ) and 2 receivers (the VDOIT ESP32 with external LoRa-RFM95W and also a TTGO ESP32 with built-in LoRa-module).

Here my sender, in my garden:

Returning from holidays I found that the transmission suffered from missing and corrupted packages. As an example I add here a composed picture with a few demonstrating lines out of the many lines – about 3000 – in my GSheet(s) .

(By the way: the values under ‘s’ in column C: number of seconds that the RTC in LoRa-sender is delayed related to real time as got from internet (NTP) – this is to check if the RTC-time is not lagging behind too much – at the last lines you see a lag of 7 seconds ‘running behind’ – the lag is calculated at the receiver side).

I added at receiver side some simple code to check if a received package was according the format as has been send by the LoRa-sender.

Here an example of a package as my sender code implementation will throw out:

    1524/2019-05-24T16:21:00Z\25.19&4095#95

    packID/sampleDateTime\temperature&soilHumidity#batteryCondition

My very simple code to perform some valid-transmission-check was only this:

#define maxPacketSize 45
bool validPackage = true;
 
 // Get readingID, temperature and soil moisture
 int pos1 = LoRaData.indexOf('/');
 int pos2 = LoRaData.indexOf('\\');
 int pos3 = LoRaData.indexOf('&');
 int pos4 = LoRaData.indexOf('#');

 if (pos1 == 0 || pos2 == 0 || pos3 == 0 || pos4 == 0 || LoRaData.length() > maxPacketSize)
 { validPackage = false; }
 else if (pos1 >= pos2 || pos2 >= pos3 || pos3 >= pos4)
 { validPackage = false; }

   //  if (!validPackage) then package will be rejected
   //  (not processed, so another missing sample)

On a 2000 transmitted packages some 20 packages were missing; both receivers had about same missing number of packages – mostly not the same packages were missing. So the LoRa-sender was working good; and I have to say that this solar powered station did very well in all tests. But as you can see from my table above also some bad packages came through my too simple validity check and resulted in corrupted values for packID, temperature, humidity and batteryLevel (%).

You have to know that the transmission conditions for my test were certainly NOT heavy. The distance between sender (in my garden) and the two receivers (in my house, but separated from outside only by double glass in windows) was respectively 2 meter and about 15 meter. Of course this is not where LoRa is intended for.
I could imagine that working with real long distances (1 to 2 km.) and experiments with the LoRa-transmission parameters (SF and such, see Andreas Spiess) could increase enormously the number of missing and corrupted packages.

Of course one can think about implementing a full-blown communication protocol in sender and receiver: the receiver has to acknowledge every good received package. For such the sender should switch to receiving mode after sending. And if receiving the acknowledge was missed or received corrupted…. etcetera, etcetera. And the sender should stay for longer in  not-sleeping-mode, so exhausting the solar-charged battery much more..  so too complex and too much hassles.

The best, and probably most simple way to improve quality of transmission is to send every package twice, so repeat every send action after a few seconds. With simple adjustment of code (and without checking package ID’s for doubles at receiver side) this can be implemented.

Here the simple adjustment in the sender code (for the complete sender code I refer to the course “Learn ESP with Arduino IDE“, module #8, unit 2):

In setup() insert after sendReadings() just this:

  #ifdef _sendRepeat    // enable/disable this by compiler directive
  delay(conWaitRepeat);
  sendReadings();
  #endif

You have to define the compiler directives _sendRepeat and conWaitRepeat; for this last one I choose 5000 (5 seconds), but it can be shorter to decrease the awake-time of the sender ESP32 (power consumption).
As you see in the original code of RNT the getReadings() still has to be done only once, saving on power consumption and – for my goals better – sending measurements on exactly that moment in time (10-minute-moment).

For adjustment at receiver side taking in account that every package can be received two times (or one time or not at all) one can think of complex code that compares the packageID of succeeding packages and throw away double received ones.

Too complex, it can be done very simple. If a package is received very short after successful receiving the ‘previous’ package you can assume this is the repeated-one, so ignore it. Makes adjustment as simple as this. And has more advantages, see at the end.

Adjustment receiver code: define some globals and at the end of getLoRaData() insert code to denote the moment of successful receiving a package:

// globals:

// t_minutes_LastUpdate: t in minutes since system start

//                       as moment of last good received package

unsigned long t_minutes_LastUpdate = 0;

// help flag denotes that 1st package has been received
bool flag1stReceived = false;

// insert this at end of getLoRaData()
t_minutes_LastUpdate = millis() / 60000;
 // Note(bug!): overflow within 49 days!
flag1stReceived = true;

In the main loop() you have to adapt the if (packetSize) { …. } statement to this code:

 if (packetSize) {
   bool processPackage = true; // local flag - default true
   if (flag1stReceived) { //this is to prevent small problem at startup
     long unsigned t_minutes_now = millis() / 60000; 
     if ((t_minutes_now - t_minutes_LastUpdate) <= 4) { 
       // Ignore! It's a repeat of same package
 
       // already received and processed
       processPackage = false; 
     }
   }

  if (processPackage) {
    getLoRaData();
    getTimeStamp();
    logSDCard();
  }
.... all other code in loop()
}

( mmm… about a possible bug within 49 days I did not bother now. )

The code I introduced above to perform some checking the validity of received data packages I extended a bit more to prevent corrupted packages as shown in the table above.

 #define maxPacketSize 45 
 #define minPacketSize 30

 bool validPackage = true;

 // Get positions readingID, temperature and soil moisture
 int pos1 = LoRaData.indexOf('/');
 int pos2 = LoRaData.indexOf('\\');
 int pos3 = LoRaData.indexOf('&');
 int pos4 = LoRaData.indexOf('#');
    
 if ( pos1 == 0 || pos2 == 0 || pos3 == 0 || pos4 == 0 ||
 
                                                   LoRaData.length() > maxPacketSize ) 
 { validPackage = false; } 
 else if ( pos1 >= pos2 || pos2 >= pos3 || pos3 >= pos4 ||
                                                   LoRaData.length() < minPacketSize )   
 { validPackage = false; }

 String readingID_C = LoRaData.substring(0, pos1);
 String temperature_C = LoRaData.substring(pos2+1, pos3);
 String soilMoisture_C = LoRaData.substring(pos3+1, pos4);
 String batteryLevel_C = 
LoRaData.substring(pos4+1, LoRaData.length());

 if (validPackage) {      
   validPackage = false;  // assume
   if (SisNumeric(readingID_C)) 
     if (SisFloat(temperature_C)) 
       if (SisNumeric(soilMoisture_C)) 
         if (SisNumeric(batteryLevel_C)) 
           validPackage = true;                                    
 }

   //  if !validPackage this package will be rejected 
   //  (not processed, so another missing sample)

if (validPackage) {
   // process content of this package
   ~....code...~
}

And below the routines called in code here above:

bool SisNumeric(String s) { // never negative values here
  // check if s is an valid integer
  for (int i = 0; i < s.length(); i++) {
    if (!isDigit(s[i])) {      
      return false;
    }
  }
  return true;
}

bool SisFloat(String s) {  // also negative floats possible
  // check if s is an valid float number
  bool flagDot = false;

  for (int i = 0; i < s.length(); i++) {
    if (!isDigit(s[i])) {            
      if (s[i] != '-' || i > 0) {
        if (s[i] == '.' && i > 0 && !flagDot) {
          flagDot = true;
        } else {
          return false;          
        }
      }        
    }
  }
  return true;
}

Checking validity is based here on the assumption that corrupted packages mostly have ‘weird’ or not allowed characters on inappropriate positions in the received string. Of course it would be better to implement a checksum-system at sender and receiver side. I leave this to an other clever developer – find with Google an simple and adequate algorithm for these small packages.

This improved code I’ve been testing for almost 2 weeks now with as result just a few missing packages (4 missing on 2000 transmitted) under the same rather soft conditions for LoRa.
But the real profit will get clear when I will enlarge the distance between sender and receiver to the much longer ranges where LoRa is meant for.

And here comes another plus or bonus for the simplicity of this approach and the small addition in the sender code. If we change this addition as given above a little bit into the following alternative version, we can extend In a simple way the quality of transmission, also for long ranges and bad transmission conditions.
At top of our code (global) we define:

// conCountRepeat: number of repeated sending of every package. 
//   undefine or set to 0 to disable repeated sending packages
#define conCountRepeat 3   // 0 => no repeat
#define conWaitRepeat 3000 // wait 3 seconds before next repeat

And in setup() add after sendReadings() just this:

  #if (conCountRepeat > 0)
  for (int i = 1; i <= conCountRepeat; i++) {
    delay(conWaitRepeat);
    sendReadings();   // or repeatSendReadings(); see Note
  }
  #endif

// Note: we could gain a bit on processor time (power consumption) in defining
// a routine repeatSendReadings() that will take global var ‘message’
// as composed in sendReadings() – and just send it.

We can play with the defined constants to test what are the lowest values for a given sender => receiver configuration (distance and transmission-parameters) to get acceptable results.

And what we have to change more at the receiver side in the code above to take in account more then 1 repeated packages? Nothing!
As long as we keep the combination of conCountRepeat and conWaitRepeat limited,
that is: (conCountRepeat * conWaitRepeat)  much lower as the time period of 4 minutes (minus 1 for calculation rounding) as we used in the adapted code for the receiver, all superfluous received packages will be ignored.
This it is and this is my contribution to LoRa-project.

For those interested in how my webserver page looks, here a screenshot.

Yeah, you have to learn a little bit Dutch, to understand this all.

Systeemstart is moment LoRa-receiver was started/reset – LoRa-sender runs already 1,5 week)
(between brackets some cryptic info for me to monitor).
The software at receiver side is also maintaining min- and max-values for every day (24 hours, reset at 00:00h) and over-all since the last start-up/reset of receiver.
Actual values for moment last update, temperature, soil humidity (bodemvochtigheid) and battery level are updated by scripts without reloading the whole page.
The Dutch word ‘Klomp–temperatuur‘ denotes the temperature at 10 cm height (klomp) above the soil (ground); this temperature can be rather different from the temperature according the standardized way of measurement (1.5 meter above ground). In winter and spring, with cold nights, this difference can be -2 to -5 degrees Celsius and can tell us if we experience frost at the ground (warning!); also bigger differences have been seen in the past (to even -8 degrees!).
Frost at the ground, for some a fright (gardeners), is giving others a hopeful perspective (ice skaters, many of them in Holland).
And ‘klomp’ is the Dutch word for the famous Dutch wooden shoe.
They are about 10 cm. high. 

At the end of this long post, I will thank you, Sara, and also Rui, for all the nice material on your educational site of very high quality and the very good and patient support you both gave me at my sometimes stupid questions. I saw you both also to be very precise and conscious in correcting small errors in your nice projects; this is important for novices like me to understand all things.
I started this spring with developing for Arduino and for the most on ESP32 and learned in little time a lot.

And it was fun!

With a lot of respect and regards,

Jop