Remove duplicates from influxdb

Hello everybody,

on my first openhab setup i made a mistake on configuring the persist strategy for influx.
It has not written anything in the first try, so i changed to have my temperature data saved every minute to db (which is in fact also not that what i really wanted, but it worked)

Unfortunately, I made another terrible mistake here. I was not aware of the cron notation and i thought that this here is for having data persisted every minute

everyMinute: "* * * * * ?"

But its every second (!).
However, one or two write-crashed sdcards later i noticed, that i have collected 25521585 data entries for one measurement in a half year where the most are all basically the same. :confused:

Here is a sample

 timestamp | temp_value
    1581122976210000000 3
    1581122977211000000 3
    1581122978213000000 3
    1581122979216000000 3
    1581122980217000000 3
    1581122981220000000 3
    1581122982222000000 3
    1581122983223000000 3
    1581122984226000000 3
    1581122985229000000 3
    1581122986232000000 3
    1581122987233000000 3
    1581122988235000000 3
    1581122989238000000 3
    1581122990241000000 3
    1581122991244000000 3
    1581122992246000000 3
    1581122993247000000 3
    1581122994250000000 3
    1581122995253000000 3
    1581122996254000000 3
    1581122997256000000 3
    1581122998258000000 3
    1581122999260000000 3
    1581123000262000000 3
    1581123001263000000 3
    1581123002266000000 3
    1581123003269000000 2.9
    1581123004272000000 2.9
    1581123005274000000 2.9
    1581123006275000000 2.9
    1581123007277000000 2.9
    1581123008280000000 2.9
    1581123009282000000 2.9
    1581123010284000000 2.9
    1581123011287000000 2.9
    1581123012290000000 2.9
    1581123013292000000 2.9
    1581123014295000000 2.9
    1581123015298000000 2.9
    1581123016301000000 2.9
    1581123017304000000 2.9
    1581123018306000000 2.9
    1581123019309000000 2.9
    1581123020312000000 2.9
    1581123021315000000 2.9
    1581123022318000000 2.9
    1581123023320000000 2.9
    1581123024323000000 2.9
    1581123025326000000 2.9
    1581123026329000000 2.9
    1581123027331000000 2.9
    1581123028334000000 2.9
    1581123029336000000 2.9
    1581123030339000000 2.9
    1581123031342000000 2.9
    1581123032344000000 2.9
    1581123033347000000 2.9
    1581123034348000000 2.9
    1581123035351000000 2.9
    1581123036354000000 2.9
    1581123037356000000 2.9
    1581123038357000000 2.9
    1581123039359000000 2.9
    1581123040361000000 2.9
    1581123041363000000 2.9
    1581123042365000000 2.9
    1581123043368000000 2.9
    1581123044371000000 2.9
    1581123045374000000 2.9
    1581123046375000000 2.9
    1581123047377000000 2.9
    1581123048380000000 2.9
    1581123049383000000 2.9
    1581123050384000000 2.9
    1581123051387000000 2.9
    1581123052389000000 2.9
    1581123053390000000 2.9
    1581123054393000000 2.9
    1581123055396000000 2.9
    1581123056399000000 2.9
    1581123057400000000 2.9
    1581123058402000000 2.9
    1581123059405000000 2.9
    1581123060407000000 2.9
    1581123061408000000 2.9
    1581123062410000000 2.9
    1581123063413000000 2.9
    1581123064416000000 2.9
    1581123065418000000 2.9
    1581123066421000000 2.9
    1581123067423000000 2.9
    1581123068425000000 2.9
    1581123069427000000 2.9
    1581123070430000000 2.9
    1581123071433000000 2.9
    1581123072435000000 2.9
    1581123073438000000 2.9
    1581123074441000000 2.9
    1581123075444000000 2.9
    1581123076447000000 2.9
    1581123077449000000 2.9
    1581123078450000000 2.9
    1581123079453000000 2.9
    1581123080456000000 2.9
    1581123081459000000 2.9
    1581123082461000000 2.9
    1581123083462000000 2.9
    1581123084465000000 2.9
    1581123085468000000 2.9
    1581123086471000000 2.9
    1581123087472000000 2.9
    1581123088474000000 2.9
    1581123089477000000 2.9
    1581123090480000000 2.9
    1581123091483000000 2.9
    1581123092485000000 2.9
    1581123093486000000 2.9
    1581123094489000000 2.9
    1581123095492000000 2.9
    1581123096493000000 2.9
    1581123097495000000 2.9
    1581123098497000000 2.9
    1581123099500000000 2.9
    1581123100502000000 2.9
    1581123101504000000 2.9
    1581123102506000000 2.9
    1581123103508000000 2.9
    1581123104511000000 2.9
    1581123105514000000 2.9
    1581123106517000000 2.9
    1581123107518000000 2.9
    1581123108520000000 2.9
    1581123109523000000 2.9
    1581123110524000000 2.9
    1581123111526000000 2.9
    1581123112528000000 2.9
    1581123113530000000 2.9
    1581123114532000000 2.9
    1581123115535000000 2.9
    1581123116538000000 2.9
    1581123117539000000 2.9
    1581123118541000000 2.9
    1581123119544000000 2.9
    1581123120546000000 2.9
    1581123121549000000 2.9
    1581123122550000000 2.9
    1581123123551000000 2.9
    1581123124554000000 2.9
    1581123125556000000 2.9
    1581123126558000000 2.9
    1581123127559000000 2.9
    1581123128562000000 2.9
    1581123129564000000 2.9
    1581123130567000000 2.9
    1581123131570000000 2.9
    1581123132572000000 2.9
    1581123133575000000 2.9
    1581123134578000000 2.9
    1581123135581000000 2.9
    1581123136584000000 2.9
    1581123137586000000 2.9
    1581123138589000000 2.9
    1581123139592000000 2.9
    1581123140595000000 2.9
    1581123141598000000 2.9
    1581123142599000000 2.9
    1581123143602000000 2.9
    1581123144605000000 2.9
    1581123145609000000 2.9
    1581123146612000000 2.9
    1581123147614000000 2.9
    1581123148618000000 2.9
    1581123149621000000 2.9
    1581123150623000000 2.9
    1581123151625000000 2.9
    1581123152626000000 2.9
    1581123153628000000 2.9
    1581123154631000000 2.9
    1581123155633000000 2.9
    1581123156635000000 2.9
    1581123157636000000 2.9
    1581123158638000000 2.9
    1581123159642000000 2.9
    1581123160645000000 2.9
    1581123161648000000 2.9
    1581123162650000000 2.9
    1581123163652000000 2.9
    1581123164655000000 2.9
    1581123165657000000 2.9
    1581123166660000000 2.9
    1581123167662000000 2.9
    1581123168665000000 2.9
    1581123169667000000 2.9
    1581123170670000000 2.9
    1581123171672000000 2.9
    1581123172673000000 2.9
    1581123173675000000 2.9
    1581123174677000000 2.9
    1581123175680000000 2.9
    1581123176684000000 2.9
    1581123177685000000 2.9
    1581123178686000000 2.9
    1581123179688000000 2.9
    1581123180690000000 2.9
    1581123181692000000 2.9
    1581123182694000000 2.9
    1581123183696000000 2.9
    1581123184699000000 2.9

Since the data would be interesting anyway … does anybody know if there is a function to eliminate duplicated entries?

So basically i would like to simulate the “only persist when changed” behaviour, but therefore, the duplicate datapoints must be deleted in between … any idea?

Further complications:

select *

for a single measurement without any limits is not a good idea …

Best regards and a happy new year :slight_smile:

Seems not that there is a elegant solution to this. I wrote a python script in the meanwhile which worked pretty well for my case … i just gathered data chunk by chunk and skipped entries that does not represent a change. The essence was just 65000 datapoints out of originally 25 millions. The database shrinked from 500mb to 2mb :slight_smile:

Mission complete for me :slight_smile: but anyway … if somebody has a elegant query based solution, please provide!

Best