I’m doing experiments regarding the usage of an AI library which is supporting LSTM models. Goal is to predict an energy forecast based on weather API (temperature, cloudiness, precipitation, …) and predict
is my HVAC cooling tomorrow?
how long will HVAC run?
which energy is needed (higher temperature - higher energy needed)
Same use case for heat pump: with lower outside temperatures it’ll run more often to heat up during the day.
I didn’t found any recommendations in the forum. Deeplearning4j looks for me outdated. Deep Java library looks fresher on github. But maybe there’s a better choice in another language which can be connected?
So my question:
Is there any library recommendation / experience of the forum or @maintainers which maybe a candidate of the core or addons in the future?
Unfortunately I can’t share any experience with Java AI libraries, apart from LangChain4j being really capable when working with LLMs, RAG etc.
Speaking about better choices in other languages to connect:
GraalPython allows running Python (including native modules IIRC) on the JVM. Given most of the AI tooling is based on Python, this might be a great help. If you are interested in trying out Python on the JVM, you might take a look at MQTT Home Assistant and Python Scripting. @holger_hees and @ccutrer have contributed those, best to ask them for help.
If you’re just wanting to use this in your own rules, building on top of Python Scripting to use some AI libraries is probably your best bet. If you’re wanting to create an automation addon (an addon that provides extensions or actions to be used from rules written in any language) and you land on a Python-based library as the best implementation, then the Home Assistant binding is probably your best example - it has pip packages wrapped into the JAR file, straight Python code in the openHAB repository that’s put into the JAR file, and a small amount of Python-in-Java files that glues the Java to the Python. But beware that while in theory GraalPy can support packages with native (CPython) extensions, I’ve avoided them because you’d have to figure out how to pre-compile them for all necessary platforms/architectures and get them all packaged up in the openHAB build process.
One other possibility is that I know Ruby has several gems for accessing LLMs that run remotely (mostly cloud, but maybe others?), so you could also write openHAB rules in Ruby and access AI that way.
Most of LLMs will be accessible through openai interface which is openapi compatible. Been involved in a voice assistant project which did interact with few different tools and all I can say - a lot depends on the model and how conversation is shaped. I can confirm that my colleagues got decent results with gpt-oss-20b for tool selection. Much better than with llama3-70b.
Integration with LLM is fairly easy, and once you get a relatively clear tool definitions, model will be able to extract arguments and let you execute user intent. There is a lot of hype on it.
I know deeplearning4j been in the market for very long time, but it always seemed to be more low level approach.
I would not discourage it in favor of python yet, as a lot of python stuff will rely on tensor architecture and tensorflow library (I suppose). As tensorflow have C++ part its possible to interact with it through tensorflow java project. Deeplearning4j seem also to have some integration with it.
Since we know a technology aspect we can also look into theorem behind:
In mathematics, a tensor is an algebraic object that describes a multilinear relationship between sets of algebraic objects associated with a vector space
Given above I think the hardest part is finding relationship (and describing transformation) between various measurements as they will vary a lot, possibly missing some of observations impacting the relationship itself. Yet, if you can model a valid LTSM with sensors you have, it will be definitely an interesting experiment to repeat across other people installations to see how accurate it is.
If you start with tensorflow, then you may be able to build a quick prototype and only then put an effort into porting it into Java if needed and if possible.
I think it’s not (yet) AI, but “akkudoktor” (the german “Dr Backe” guy from YouTube) started an OpenSource project for optimisation with inputs from current PV, forecast PV, tibber, battery usage, etc. plus you can add some devices and their need for energy.
I’m just starting to get into this and as it is early alpha stage the documentation is not soooo helpful. Here’s the github: https://github.com/Akkudoktor-EOS/EOS
I recently opened an issue for having SoC with UoM in openHAB not working as source, and it was fixed within hours…
Yes, there’s no AI involved, but I’m not sooo sure, if machine learning is really needed? You’ve got some less precise predictions (solar) and some precise predictions (tibber). On the other side you’ve got some good to calculate energy demands from heating pumps, some precise calculated ones from let’s say EV. The ones that could really need some machine learning are dish washer, washing machine or cooking sessions… But they’re usually not the really demanding devices?
I’m assuming that I’m not the only one that don’t keep track of all the different terms used within ML (so-called “AI”), but to me, the difference between LLM and LSTM is just that they are using different theoretical/mathematical models for how they “work”. I think very few people have enough insight into these things to actually know how the different models work.
As far as I know, everything that is called “AI” today is some kind of machine learning model, including so-called “conversational AI”. So, to me, it’s hard to make some clear distinction. Add in the extreme amount of hype and direct lies on this topic these days, and it’s really hard to remain oriented.
To me, it boils down to: there are several “theoretical models” of how to do the machine learning, many of them probably quite closely related. These have different names, some become more popular than others and make it to more headlines. Regardless of which model, some fundamental principles seem to apply: Small (training) data sets give inaccurate results, larger ones give better results, depending on the quality of the training data. Large ones have very quite serious hardware demands, really large ones require so much hardware and power that it’s completely out of the realm of a private person to run.
The commercial ones won’t stay free or cheap to use for long either, we’re currently in the phase where people are throwing money into the game to try to outlast the others, and be one of the few remaining that can cash in. That’s why it’s relatively cheap at the moment. This won’t last.
I can’t quite imagine how all this can translate into ML doing something “useful” on weak hardware like what we typically run OH on. Maybe I just lac imagination, but I’m skeptical, unless you want to base it on some commercial “cloud” solution that is guaranteed to become quite expensive to use in the future.
Maybe something as simple as a Proportional-Integral-Derivative (PID) could handle this?
Machine Learning: an subset of “AI”, meaning that systems can learn from historical data
Deep Learning: an subset of Machine Learning, which build an artificial neural network capable of interpreting and extract features from said historical data
Large Language Model: also an subset (or correctly a model) of Deep Learning capable of understanding natural languages with billions of parameters
All rely of training data, ML has simpler models, DL complex multilayered models and LLMs use extremely complex models with billions of data sets and parameters. Trainig each of them is faster oder takes longer of course with more complex data/models.
So, with that said, you could deploy all of them on a raspi (if the model is already trained). of course it’s restricted if it’s offline and running on an ARM SoC system.
What I think Bernd is aiming at is the “simple” Machine Learning aspect. You’d have to train the model with data from openHAB (e.g. items and their changing values). If done properly you can do “predictive” work with it. ML could detect some patterns in your smarthome and predict energy demands. If you’re not as chaotic as our family perhaps there’s alwasy some time intervals where you need more power other time intervals you need less power. So the thinking is: detecting patterns and react accordingly.
I can give you an example of non-smarthome use, which is straight forward and used widely. If you install some sensors on let’s say an elevator and record vibrations, temperature, useage, etc. you can react predictive if the elevator shows aberrations in one or more data points. Let’s say, the vibrations differ from the usual pattern and you have other (similar) historic data, that an elevator pulley needs replacing. So you can replace it before the elevator shows an error and you have to put the elevator out of order and potentially lose clients or at least complaints rise…
Back to our smarthome: You could try to use the energy consumption pattern, add some ML to it and optimize it. Let’s say automatically preheat the heating pump, plan EV charging, etc. pp.
But - and that was my 2 cents - is ML for that really superior to a set of “static” rules? which could of course be PID oriented.
This matches my understanding, except that according to your own definitions, ML encompasses both DL and LLM, so you can’t really say that “ML has simpler models”, since they are all ML. That’s why I like to use the term ML, it lets me describe it on a general level without going into a lot of details about the different models that I don’t know anyway.
When it comes to ML being “a subset of AI”, I guess that is true on a hypothetical level, but as far as I know, no “AI” that isn’t ML exists today, which is why I don’t like the term “AI” at all - because I think it implies something that ML is not, actual “intelligence”. By that, I mean abstraction, application of knowledge across different domains/concepts etc. I know that a lot of people want us to believe there is actual “intelligence” there, because they want to make money, but that doesn’t make it so.
I know that training requires more resources than using the model, but that doesn’t mean that using it is “cheap”. With my 12 GB GPU, I’ve found that LLM models above 7-8 GB starts to be “painfully slow” to run. Those aren’t the high quality ones, and even the ones I can reasonably run there, would be completely impossible to run from an RPi as far as I can understand.
In a way, ML is “the opposite” of what we have been doing up until now. We’ve tried to make things smart/effective by designing code to do specific things in an effective way. ML goes “the other way” by not trying to be smart/effective at all, but instead just throw a lot of hardware resources at the problem. The “benefit” of this is that no human has to sit down and design a code tailored to do the specific task, the downside is wasted processing power, memory and electricity. This is my main reason for being skeptical of the “optimized hardware” that we’re supposed to get in the near future that will allow all this to be done on “weak hardware”. To me, it sounds like a contradiction, but maybe I’ve gotten it all wrong.
I understand this in theory, what I don’t understand is how to translate this into a practical solution. You would have to find some existing model that would fit the purpose, that in itself might require significant insight into the whole “world of ML”. Then you’d have to figure out how to train the model to do something useful. You can’t just throw historic data at it, you must figure out a way to make it understand what your goals are, what is desirable behavior and not, when the historical data describe “good” and “bad” situations. I just don’t see how you do that without diving really deep into ML. But, again, this could just be me lacking imagination.
How is this “straight-forward”? The whole challenge is how to teach the model how to know when something is wrong. It doesn’t understand anything about elevators or pulleys, so much somehow train it so that it knows how to correlate a certain “combination of sensor inputs” with a certain error condition. To, me, making this training meaningful and effective seems difficult.
Alternatively, you can make it really stupid and just teach it “what is within normal parameters”, and then produce a “deviation message” if the readings fall outside this normal. If so, a human must come and evaluate what is wrong, all it can say is “this isn’t normal”. I understand that doing this isn’t too difficult, but I’m not sure how useful it is either. You could probably easily program “traditional software” to monitor a similar set of sensors and react if they went “out of envelope”, using much less hardware resources to perform the same task.
It depends on whether you are using the modern sense of the term AI or the traditional sense of the term. Bayesian nets, declarative programming (e.g. Prolog), etc. used to be called AI once-upon-a-time. But the “machine” doesn’t learn in these approaches. It’s told all the facts then applies relatively rigid algorithms to reason about the facts.
I expect these general purpose LLMs to become paired down and specialized for certain tasks over time. The model can be much smaller if its only job is, for example, helping to write openHAB rules in JS. The more specialized the model, the less is required to hold it.
I expect over time we’ll start to see the efficiency of the models improve also as we continue to hit up against hardware and power consumption limits and being more efficient gives a significant competitive edge.
But even now, Gemma2 (2B) runs reasonably well on an RPi 5 with 8GB RAM without an external AI co-processor. Other similarly sized models work well enough on an RPi too I’m told.
I think, that’s a bit out-of-scop for this thread, but there’s a whole toolset to achieve just that. You have to think “outside” of a given ruleset via hard If-Then-Else. There’s a whole industry around just that.
