What to do with my AI-generated bindings?

Development Add-ons
1

I have ported/updated two bindings with the help of AI and would appreciate some guidance:

  • How should I make them available? Should I post them in the forum first and potentially move them to the Marketplace later? (It seems a bit of a waste to only keep them for myself).

  • How should the use of AI be disclosed? The code was 100% generated by AI. I didn’t do a review of the code, but mainly tested the resulting bindings for functionality. Do we have standard disclaimers for this.

  • Are there any best practices to improve quality? My current approach was:

    1. Generate the binding with Sonnet 5
    2. Have Opus 5 review the result
    3. Feed most of the findings back to Sonnet and fix them
    4. Run a similar review cycle specifically looking for resource-related risks

The bindings are: (I’ll post them in seperate threads):

  • Balboa binding: Updated from OH3 to OH5.2, including autodetection, performance improvements and fixes for network-related bugs.

  • Jablotron binding: A new, local USB binding generated from an existing Python implementation. This one particularly surprised me — with relatively few iterations, AI produced a correctly working binding. Interestingly, the AI itself recommended not creating a full openHAB binding, but instead connecting the Python script via MQTT, arguing that a proper openHAB binding would be quite complex.

I understand, and actually share, some of the concerns about AI-generated bindings. On the other hand, smart home has a very long tail of devices and integrations, and AI-generated bindings could potentially be a way to achieve much broader coverage.

So: What would be the recommended way to proceed with these bindings, and are there any guidelines or best practices for AI-generated openHAB add-ons?

2

Why not review the code yourself, and see whether you understand/agree with AI’s proposals?

3

No, just go straight to the Marketplace.

State that near the top of your description of what the binding does. As long as you are up front and honest, no one can fault you. Those who refuse to use AI generated code will know to avoid it, and everyone else will know what they are getting may or may not have problems.

Make sure your agents read and review the code against AGENTS.md. For better results, have it review the code against small sections of AGENTS.md at a time.

There is also a GEMINI.md and CLAUDE.md files. I think they have the same contents or just point at AGENTS.md. I’ve only really looked at AGENTS.md.

But the key with having an agent review the code is they will only review for exactly what you ask it. And if you are too generic, it will stop once it finds one or two things. So you have to be persistent, and you will get better results if you narrow the review focus. At least that’s my experience. YMMV.

I will also say, based on my experience with EyeOnWater even after having the agents review everything against the “rules” in AGENTS.md, it will still need a lot of work before it can become suitable to be merged. But so far, every finding by copilot and every comment from the maintainers have been valid (except for maybe one where I think the maintainer is going against AGENTS.md and I need to determine whether AGENTS.md is wrong or the maintainer is mistaken.

Submitting the add-on will definitely make it better, but it will be a lot of work and a lot of tokens. I’m still at it though because I think OH has a real problem with add-ons. It shouldn’t be this hard.

4

I usually perform an AI review on all my PRs (regardless whether it’s 100% hand-written or AI-generated) using GitHub - wborn/github-review-policy · GitHub (Wouter/wborn is an openHAB maintainer, and uses this for AI assisted reviews in core and add-ons).
Before I used that, I was using a more generic review skill and always had the AI check the review checklist that the add-on maintainers also use during their review: Review Checklist · openhab/openhab-addons Wiki · GitHub.
Not sure whether the review checklist is already referenced by AGENTS.md …

5

Well I thought I had the answer - I used the following approach for a binding I was developing:

@author firstname lastname - Initial contribution with input from Gemini

But then I tried the review policies as mentioned above

But then I got a bit fat slap across the wrists from that:

Author Attribution Tags

Problem: @author firstname lastname- Initial contribution with input from Gemini mentions AI tools. openHAB repository guidelines require using the contributor’s real human name without AI references.

So looks like I will be removing the Gemini reference from there…

So I guess it’s disclosure is just limited to posts in this forum, and marketplace, when it gets put on there… I think I have mentioned vibe-coding in my original post more than a few times AAP (Arrowhead) ECi Alarm Binding , but when I eventually get to finalising it, I think I will need to be aiming for a PR to openhab-addons, not marketplace, due to some dependencies which are a pain via marketplace. Being my first binding, I just trying to convince myself that its truly ready for the real world, with well over 100 hours of vibe coding, integration testing with an emulator, and running against my real alarm… Having run another review using the above MD files, tells me I have a few new things to address…Fortunately none look too ominous…

But I don’t think it matters if its a binding or whatever - I have built/hacked a few integrations in the past using python, for integration to OpenHAB via MQTT, and generally just throw them up on the forum, so people could use them, warts and all, if they wanted. If it was partly based on someone elses work and I leveraged that, I would credit that, and I think it is no different for AI these days.

6

When it comes to commit messages, the Linux kernel’s approach sounds like a good idea to me:

7

@Larsen I’m curious, how much did it cost in (token? subscription? credit?) to build a binding from zero to “usable” (not final polished version)

8

For me using Gemini Flash mostly it cost about $8.80 to get an add-on that works. I’ve since spent $66.75 dealing with Copilot findings and maintainer comments. And of course that doesn’t count the stuff I’ve done manually. But I’m still not done. I’m on round three, but I can’t sustain this spend rate (I use Gemini for other stuff too) so I have to do more manually (sometimes I become the Chief Copy and Paste Officer just to save tokens).

This particular binding will enable me to get a warning when the water bill is going to be too high before the end of the month so it is likely to save me more money than I’ve spent next summer. So I can justify the cost.

But other integrations I can’t and for those I’ll build a sidecar in Python and integrate with OH using MQTT. I was going to go that path for this one too but decided to make it an add-on to see how that goes. A sidecar type implementation has been running me between $5-$10 per sidecar to get something working and polished. I spent just over $10 to get a residio2mqtt sidecar working for my thermostat and around $5 to build a BTLE sniffer that captures and records the SpO2 readings from a Berrymed O2 sensor.

So far, with Gemini at least, I find it’s cheapest to build in Python compared to Node.js. And creating an OH add-on (that could be merged) is looking like it is going to cost an order of magnitude more.

9

How do you use the paid “Gemini”? Through antigravity?

10

I do pay-as-you-go. https://ai.google.dev/gemini-api/docs/billing

I started on the free tier, set up billing and added $25 to move to tier 2 as I found tier 1 too limiting. When that runs out I throw another $25 at it (I do not do automatic top-up). Thanks to the add-on I’m now on tier 3.

For the harness I’m using Cline, mainly because when I first started I was hoping to use Gemini for the planning stage and a local LLM for the acting stage. But I tried a few LLMs in Ollama and was never able to get one to work. So I have it set up using Gemini on both the planning and action stage. But Cline is the only harness I’ve encountered so far that lets you split the planning from the acting so you can use a more powerful LLM to plan but the acting agent doesn’t have to be that smart. YMMV. I’m planning to try some new models locally again (I’ve read the older gemma 2.5 code might work better than the newer model. :person_shrugging:

I do plan on trying out Antigravity at some point. I just haven’t got around to it yet.

Note: Cline tracks the spend which is how I was able to provide the costs above. It’s not 100% accurate but close enough for budgeting.

11

Oh that’s some crazy amount of money IMO when thinking about the work you did with it.
This makes it totally understandable that more and more companies pivot away from AI coding and hire back their old employees … if you get these spendings on a small binding, I don’t want to see the spendings on big enterprise software projects.

You should definitely try out Antigravity with a paid subscription, I’ve been on the Google AI Pro tier (regularly 20$/month, students 5$/month, first year free for students) for a year now and the 5 hour windows and weekly limits are insanely high, running out of credit really takes a lot of PR work and AI reviews of large PRs. Seems like the better value compared to pay-per-token.

12

Already heading down that path. It’s been a long time since I was a student so that won’t help, but if my wife cancels her ChatGPT subscription, with what we’ll save by no longer needing the google home subscription we will actually save some money in the long run.

I did try out Antigravity and I like the way it works. But I ran out of credits pretty fast and have to wait a week before I can try again, or just decide to get the Google AI Pro subscription.

13

If the Gemini models are good enough for your use case and your use case involves a lot of input and output tokens, the Google AI Pro subscription is definitely worth it. I think Google at the moment provides the most usage per money when comparing with the other big US players.

Unfortunately their models aren’t the best ones and you notice that. Luckily though, Google hosts Anthropic models on GCP and provides Claude models in Antigravity with way lower limits than their own models, but enough to let Claude do the plan and then execute it with Gemini models.

14

Most of what I’m doing is pretty simple and straight forward. Gemini is capable enough, though it does keep guessing at environment variables I need to pass to Docker containers which is annoying to no end. I mostly use it for ansible playbooks, trouble shooting problems, and little python scripts that solve annoying problems.

15

I’ve been using Claude for a couple of months, and it’s been spot on about openHAB stuff on practically every statement.

16

First of all: thank you, thank you for all the replies and helpful tips!

Maybe one thing should be clarified up front: I have no intention of getting these bindings into an official openHAB release. In my opinion, using AI to create something like this for an official release should probably only be done by people who are sufficiently experienced to ensure the code quality with a high degree of confidence.

My goal was, at most, to get them into the Marketplace. The bindings in question are niche bindings anyway, with perhaps only a handful of users.

So, regarding:

The short and honest answer is: because I can’t.

My Java experience dates back to around the turn of the millennium, and my knowledge of OSGi is, at best, at a very abstract level.

Of course, out of curiosity, I did look at the code and tried to understand what the AI had produced. But I would never dare to call that a proper code review.

I’ll do that. This also gave me another idea: could AI-generated bindings be formally tagged in some way, so that openHAB could make use of that information?

For example, perhaps there could be a setting in openHAB that lists all AI-generated bindings and allows the user to disable them with a checkbox.

The thinking behind this is that, as the number of AI-generated bindings increases, there might also be an increasing risk of performance issues or other unforeseen problems. Having an explicit way to identify and disable such bindings could give users some additional control.

Related to this, another idea would be to indicate the level of human review the binding has received. For example:

  • generated with AI, no human review
  • reviewed against Agents.md / coding guidelines
  • personally reviewed by the author
  • reviewed by an experienced openHAB developer

That’s not so easy to answer.

I initially subscribed to Claude Pro for one month just to try it out. So far, the included quota has been sufficient for my private projects.

Updating an existing binding to OH5 took roughly 30 minutes and used perhaps 20% of the daily quota.

Porting the Jablotron binding from Home Assistant to openHAB took maybe 2 hours. I actually found this the most interesting part — I was surprised by how little effort was required to port a component from Home Assistant to openHAB. So the cost for this initial working binding were less than 1€.

However, I’m not experienced with AI-assisted programming. I’m sure there is a lot of room for optimization once you know how to work effectively with these tools.

Initially, I mainly used Sonnet 5, and later I used Opus 5 for most of the work. With Opus 5, the quota decreases significantly faster, but since I have a very limited amount of time to spend on this, it has still been more than sufficient.

17

I’ve used Opus 5 to do similar python to MQTT Sidecar type integration for LitterRobot and SmartRent apartments, both ported from existing HA integrations.

It’s been incredibly helpful and took maybe 1.5 hours for both…

18

yes, MQTT with HA-standard is a good solution too. Probably much leaner than a full binding. But the dependencies are a MQTT-server, -binding and the HA-binding. So that potentially distracts people from using it.

19

If you don’t mind running Home Assistant as the “sidecar”, this opens up a universe of available integrations to openHAB without MQTT.

20

You’re right, of course, that’s the perfect way to benefit from the HA universe, but I still have a blockage somehow to…