Any toughts about how these kind of frameworks will change the future of smart home systems for ever.
I think it’s too soon to tell. I’m not certain the whole agent approach is really needed in an openHAB/HA type home automation environment. We can already hook our system up to an LLM and have it make decisions and run analysis and such. We don’t need an agent to achieve this.
I’ve run some experiments myself with ChatGPT (there’s an add-on) and it’s not super hard to craft usable prompts.
I imagine for those stuck in closed ecosystems like Google Home, Homekit, etc. and do not have a central home automation hub like openHAB/HA, it will be more useful. The agent(s) could theoretically be able to interact with those systems without a hub in the middle.
Perhaps an agent could be used to integrate with an API that OH doesn’t already support. Though it seems to me it would be more effective to have an AI code up an add-on instead.
I’m not sure if I would trust Openclaw in particular right now with direct access to my home automation, but maybe with a local LLM and some bigger guardrails, it could be fun to experiment with. But I don’t see what it brings over what is already possible.
@rlkoshak we should have a rolling thread with cool ChatGPT use cases…. (Like the one where we share what we built)…
Regarding how smart homes will change… honestly I have no clue. I think that one of the first use cases might be to make smart vacuums act like drones, but one I would really like to see is not having to create rules. Just speak normally and the llm creates the rule for you. That would be cool 
Anyone can start such a thread. My use of LLMs right now is a bit underwhelming. It doesn’t make sense for me to start it. I use ChatGPT to make an announcement in the morning when it’s time to leave for school early because of traffic (so it’s not the same words every time).
I also messed around based on ChatGPT / AI - Event Log Analysis to show how it could be done with just a couple lines of rules code and the ChatGPT add-on. That’s about it.
1 Like
I just told OpenClaw “I want to integrate,” it did its magic, generated its own prompt, and that was it - everything works.
As of now, I’ve slightly optimized it: once per hour it retrieves the list of items and uses them to process my requests. It works perfectly - from simple control commands to queries like, “Analyze the water pressure pattern over the past 24 hours - does it look like there might be a leak?”
I also did quite a few experiments in the last few months with agents and MCP servers with tools for openHAB interaction. I didn’t do it with openCLAW but with my own tools. In my experience there is quite a lot you can do with agents but there is not much or nothing that you coulnd’t do without them.
I tried both: an agent that helps you with openHAB administration and one more user centric approach (only getting item states, sending commands and the like).
Administration:
In my experience the aforementioned guardrails are especially important for complex administrative tasks. This includes strong validation of all inputs, very extensive error handling with very helpful error messages that help the agents to improve but also detailed logging of any changes made to the system. During my experiments the agent was sometimes extremely helpful when supporting complex tasks like analyzing shortcomings in the semantic model and fixing them but more than once it also broke stuff pretty badly causing more work to clean up after them.
When writing rules the main issues are usually that the models are very proficient in writing JS Code but they are not very familiar with the specific openHAB interfaces. Especially when it comes to date/time handling models often use methods that may exist in other frameworks but not in the JSScripting libraries. Also for newer APIs like the JS persistence APIs the LLMs are often outdated and write code that does not work in OH5. So this always needs to be supported with additional tools and resources to assist the LLM.
User Assistance:
This is a far less complex scenario which can be supported very well however the problem here is that the gaps between the capabilities of the openHAB platform and LLMs are surprisingly small. Answering questions about item states is something that works well but you do that e.g. with Alexa or Google as well though the advantage is that you can directly profit from a well defined semantic model without additional administration. LLMs shine when you want query complete equipments with different item types or aggregation of information that is not modeled as groups in your system.
Sending commands is something that works just as good but again only with well defined semantics and also there is no big difference to other existing interfaces. Especially when you chat with the agents: A well designed Main UI dashboard is much quicker for the most commonly required information or commanded items and voice assistants still beat all other interfaces in terms of speed and ease of use. Again an agent with voice command interface could help a lot but also requires very explicit semantic modeling.
However I also experimented with Rule-like mechanics: I created tools that allow the agent to monitor items and react to item changes according to user requests. This also works quite well but here as well in my experience the main use cases are short-term temporary jobs that don’t justify a separate rule:
- I’m out of the house for three hours, please let me know if the kids turn on the TV
- I’m on vacation for the next week, please monitor all doors and windows
Where two is already on the edge: If you are on vacation at least once a year you might just want to create a rule for that and toggle a vacation-mode switch when you leave…
All in all, I haven’t really found the killer feature yet. Therefore I also thought about a more autonomous approach to make agents inform you about relevant events or send commands autonomously. But when I thought more about it I couldn’t come up with a lot of ideas. Unless you really don’t care about annoying messages there are very few categories that would justify autonomous information by your agent:security related (e.g. doors/windows open), saving money (cold outside and window open, left the coffeemaker on when leaving the house), info about finished devices (washer/dryer/dishwasher) and… well thanks but no more messages for me at least… The question is: For which of these do you need an agent or what could an agent do what a rule couldn’t. You can make the test and ask you favorite LLM: My LLMs didn’t come up with any better ideas which is rather disappointing. After all these models should be the ones in charge here…
Also there are two factors that do not convince me yet:
Cost: The costs for using agents are low if you ask for something every once in a while. I do have agent at hand on my phone that I use a couple of times a day. To be honest, I do not do it because it has a lot of advantages but just to test this and because the agent has a funny personality. More autonomous agents like the one you created can however produce potentially relatively high costs given the comparably low benefit. Also I think this applies to local models (hardware and energy) just the way as does for cloud based models (usage). In my opinion the approach you are describing is however very cost inefficient. I would really recommend to use RAG- and Tool-based approaches which can reduce token uses significantly.
Speed & Reliability: My agents are in general slower and less reliable than Main UI or Alexa… Speed is not always super important, but everything less then 100% reliability can easily be disappointing: “You had one job…”
What I like most about agent integration is that I do not need to configure everything manually for Alexa and Google and other platforms. It’s so painful to create all the explicit metadata for other platforms that I only have very few devices integrated there. Maybe that is another use case for agents: Create my Alexa metadata config based on my semantic model A sound semantic model however is a good start to integrate an agent and it can instantly do a lot for you. I don’t have a voice channel yet connected to my agent but this could really be another boost I guess.
3 Likes
The point here Is that openclaw Is not a LLM Is an operating systems for AI Agents. Can deliver messages, orchestrate tasks, it is a control unit between inputs, outputs and AI Agents Tasks. So basically can be a smart home hub itself.
Yes, you’re right this is the interesting part about this system. If you skip the steps of setting up and configuring a classical smart home hub altogether in the first place this could get really magical.
However given the complexity of such a system I have my doubts that it will be able to really manage that efficiently as a whole. The coding agents that I use are already struggling with overviewing smaller codebases. But of course this is just the beginning and not the end of the development so it will be very interesting to see what comes out of it.