Chatgpt to generate core code to openHab (Policy / Restriction)

Development
1

Hello All,

I’ve come asking myself a question recently.
I’ve used chatgpt to help me construct a new class to modify header of mp4 files on the fly.
This is possibly for inclusion in the new feature-media branch, in the base openhab.core.media or openhab.core.audio package.

But when doing it, I ask myself is there is any policy about using Chatgpt code generation for Openhab. Is it ok for everyone (of couse after review) to have a full class generated with help of chatgpt into base code. Or do we have any restriction on this.

@Lolodomo, @lsiepel, @Nadahar : any advice on it.

Thanks,

Laurent

2

I can’t speak to policy, but it sounds like a terrible idea to me. These "AI"s mostly come up with complete crap as “code”, you’re very lucky if it even compiles, but what’s worse is all the logical flaws it creates. Remember that they don’t “understand” anything of the code, it’s basically an “advanced search” that glues together different pieces of code it has found that it “think sounds relevant”.

If you want to use those for anything, I think the only thing that makes any sense is to make them generate something to give you an idea of what must be done, and then implement it yourself based on that information.

Copyright-wise, there might also be challenges, but I’m not really paying attention to these lawsuits and “where the needle is pointing” at any point in time.

1 Like
3

I have to agree with Nadahar.

I think AI is a good way for programming beginners to get inspired and learn programming using simple examples. It can also serve as a reference for things like how to implement a loop as a stream, etc.

But it’s completely unsuitable for professional, efficient programming.

AI code is often cumbersome and unnecessarily complex.

AI code usually contains errors.

AI code can create security vulnerabilities.

AI code will never implement anything new for you because AI is never creative.

If you think about it for a while, you could go on like this forever…

2 Likes
4

I’m partially agreed with you.

I’m aware about copyright issue as I’ve used to work on IP legacy before in corporate business, and that was the initial sense of my question. It’s why I ask about eventual policies for Openhab. On my side, I’m agree that LLM are based mainly on sucking code on internet. But I’m also on my own experience think that LLM make some progress to not propose copyright code in their response, that was not the case in the first time of LLM.

About if the result code is crappy or note, or even compiled. I’m surprised about how the response is more and more consistent. I would not say it it’s perfect, many time I’ve just go around in circles with LLM, don’t find a correct working answer to what I want.

But for exemple, on my Mpg4, I was able to in a very short time to understand structure of Mp4, having a basic parser in place, and adding my metadata tags to the stream, all working ok with all my players at home.

Of course, I’m not saying it would handle all Mpg 4 standard fully, and that it won’t need review to have something that would handle it perfectly. But it would certainly have require me to read mpg4 specification for several hours before having a result.

In comparison, I’ve also look to include external library like org.jcodec, or sannies/mp4parser. But this libraries are not maintained correctly for several years, and even worst, are not working properly.

So finally, don’t know what are best or worst :slight_smile:

5

this it what I mean with getting inspired by an KI. You understand how it could work. But at the end, you have to finish the code, you have to clean and optimize it.

just an example. Some weeks ago, someone posted a fix for a bug here. It was a code snipped from a KI. 15 lines long. It was successful fixing the problem. But at the end, if you think about the problem, the same problem could be solved by a one liner.

6

“Working ok with my players” just isn’t a very good test. That might be coincidental and depend on many different things. Also, the metadata structure of MP4 isn’t very complicated as far as I remember, so I don’t think you’d need much time to just figure out what you need to do manually.

But, there are so many things that can happen that will corrupt the stream under certain circumstances. That is certainly possible when you write the code yourself too, but it’s “almost guaranteed” when you use LLMs to “generate” the code.

If you want to look at Java code that handles the basic metadata structure of MP4, I know that metadata-extractor has at least partial support for the MP4/ISO Base container. It doesn’t allow writing them though, because that’s not relevant for what the library does - but it should be more than enough to get an idea how to access these data.

I generally use FFmpeg for all such operations because they have people with knowledge of most of these formats, which means that they mostly “get it right” when they manipulate media streams (but far from always!).

7

When it comes to legal aspects, I’m really not sure who’s the right person to ask, maybe @hmerk has something to add?

8

I have to agree with Nadahar.

I’m not want to seems to be the advocate of LLMs.
I know to some extents how they work, and agree with you about their creativity.
But a few answer to your feedback.

I think AI is a good way for programming beginners to get inspired and learn programming using simple examples. It can also serve as a reference for things like how to implement a loop as a stream, etc.

But it’s completely unsuitable for professional, efficient programming.

I’m not totally agree with this. More and more programmers can’t know about any domains in computer sciences. LLMs can be also a source of reference, even if not perfect.

I was used for several years to browse internet or ask a colleague when I don’t know how to make something. But it can take several hours browsing so many site to find right answer. I’m saying that LLMs can points you fast to some direction that would take several couple of minutes of ours with other methods.

AI code is often cumbersome and unnecessarily complex.

AI code usually contains errors.

This is certainly the most accurate assumptions for me. Sometimes it take me long time to adjust / understand what the proposed code does not work on specific case, and or to adjust to make it work as except / with acceptable performance. The worst is when LLMs proposed you severals variants, looping on them, because none are working correctly.

AI code can create security vulnerabilities.

Not fully sure about this one. This can be the case of course. But I’ve also work on Appsec for several years, and more and more, IA are used to complete / or even replace existing SCA/SAST tools to make code more secure.

AI code will never implement anything new for you because AI is never creative.

I’m agree with this, but If you think, I’m better take time to do the creative part on new high level functionality, and not on low level writing technical stuff.

I’ve remembered of an old colleague telling me in earlier 20’s that is not “normal” that on that time we are still have to handle some stupid thing like code misspelling error on compilation, and that it will be better if we can concentrate only high level business code.

1 Like
9

Let’s not even get started on how much I disagree with this :laughing: We’ve had high-level languages for a very long time, if that’s what you want, then use one of those. You get crappy, slow, wasteful programs as a result, but you can “save” a lot of developer time if you don’t care about doing things with precision, speed or efficiency.

Any developer that can’t be bothered to handle misspellings, can’t be trusted to handle all the other crucial details in code that determines if things work as they should or shouldn’t. Bugs that leads to compile errors are “no problem”, because the compiler points them out for you. It’s all the errors that don’t cause compile errors that should worry you.

10

I’m not sure to understand where you want to get me.
Have to said that at my starting time as developper around 14 years old I’m done some programmation on commodore 64 with 6502 assembly.

Of course we can certainly wrote Openhab using assembly level. I was thinking myself that Java was already such a high level languages, but was not thinking it was a piece of crappy, slow and wasteful results ! I don’t ever talks of others that use crappy languages like python.

11

I´ve used AI to build some rules and none of them worked out of the box.
There was still some work to do and give the correct prompts to get a working rule.
As already mentioned, some lines were overly complex and i could reduce them with some logical thinking.
It´s good for users that doesn´t work in development but nothing to work on something like the core.

1 Like
12

It was a comment on what your colleague told you, not on you :wink: I’m just saying that there have always been (business interests) that have wanted to cut down development costs. We’ve had plenty of high level solutions in the past, but they have mostly failed, despite being pushed quite a lot at times. I see that current “AI push” to be more of the same, it’s just another twist of trying to do more for less money. I don’t think it will achieve much at all, but I guess we’ll all just have to see.

There are much more high level languages than Java (I’m thinking of things like 4GLs). But, I must admit that for a long time, I wouldn’t touch Java, because it’s so slow and inefficient compared to native code. But, as time went by and the alternatives kept getting worse and worse, Java started looking better and better. I still don’t really see Java as a “proper language”, because it generates bytecode, not native code. But, when the alternatives are things like JavaScript or Python, Java is like a Ferrari :wink:

I’ve never taken on the pain of getting familiar enough with all the quirks of C++ to make myself “useful” with it. So, when I want to write efficient code, I still turn to Pascal (FreePascal/Lazarus). Yeah, I know it’s niche, but it’s what I know. Unfortunately, Lazarus has gone down the path of not supporting UTF-16 properly, which makes it “challenging” to use to make good quality code for Windows.

13

The other question though is who owns the copyright on the code the LLM produces (assuming the LLM isn’t itself plagiarizing)? There have been a few cases in the US that have ruled that an LLM cannot hold copyright, but I don’t know about the rest of the world.

How much do you have to change the code an LLM produces before it becomes your own? :person_shrugging: I think the legal situation is very murky, but it could be the case that LLM generated code can’t be licensed with a FOSS license at all, because it’s by definition public domain or owned by the company that created the LLM. Do you even have the right to submit the code to the project in the first place? :person_shrugging:

It’s a very complicated legal situation right now in terms of copyright.

LLMs are particularly bad at creating OH rules. It’s too niche of a programming environment that LLMs just are not trained for. I suspect a custom trained LLM might do better but haven’t had the time to experiment with training one up. I do find better results by prompting with something like

You are an openHAB rules programmer. Using https://www.openhab.org/addons/automation/jsscripting/ as a reference, create a rule that ...

And even then it will often generate nonsense, mixing Rules DSL, JS Scripting, and Nashorn JS, accessing non-existant APIs, etc.

1 Like
14

Thanks @rlkoshak,

This was my initial questioning in fact, more that the code quality issue.
That why I talk into initial post of Policy / restriction.
I’m don’t do thorough search, but little surprise that this question was not discuss before.
I’m pretty sure that some other développer (perhaps not core developer :slight_smile: ) have already submit code coming from LLM.

15

You are probably right, and I guess it wouldn’t hurt if it was explicitly stated that LLM code shouldn’t be submitted somewhere, e.g. in the CLA.

PS! I still claim that, no matter how significant the legal challenges might be, code quality is a much bigger problem :wink:

16

Unfortunately, I cannot add much to this topic. Being a full board meber of the foundation, I was into finance and tax, but not copyright stuff.

First reference I would check is the EPL, which is our licence. It then should be addressed as a question on github in the core repo, to get feedback from the maintainers.

2 Likes
17

Interesting topic…

On a side note: I figured that writing good prompts is the absolute foundation to elicit good responses from the AI/KI. I have created quite a few projects (Python, C), where the AI spat out code that functioned on first try; did what I asked it to do, and enhanced functionality based on my subsequent inputs.

Even DSL rules for openHAB worked without corrections.

I also use it work in psychology, legal, medical and for product research.

… but then I pay for the use of AI.

However, I can acknowledge, and this happens in particular with longer chats, it loses the plot entirely; it hallucinates, switches roles, in claim I did something it did or buggered up. The minute I sense that, I start a new chat.

In any case it has severely reduced my calls for help on this forum :slight_smile:

18

There’s a big difference between code “working” and the working correctly and optimally. If code just is “an enigma” that you can’t really penetrate, I can see how it can be difficult to know the difference, but for anybody that can see what the code actually does, it doesn’t take much complexity before it all comes crashing down.

Some factors will also decide how much “success” you will experience, if you typically just asks it to “do” simple things that have lots and lots of examples online, your chances are somewhat better. But, as soon as you ask it to do anything the slightest out of the ordinary, I’m sure you’ll have a very different “success rate”.

Another thing is that writing prompts have some pretty significant limitations itself. If I were to try to describe a piece of code in English, instead of just writing the code, I would often spend much more time and effort than I would by just writing the code. It doesn’t take much complexity before your “prompt” becomes a small book.

The whole thing is actually putting everything on its head. We have developed “programming languages” as an interface between man and machine, the goal has been to find a way where we can accurately describe what we want the machine to do as efficiently and effortlessly as possible. This is what we call “computer code”. The idea that “plain English” (or any other “natural” language) is somehow an easier way to tell the machine what to do is… bizarre. What you do is you take a tool that is very unsuitable for the task, and use it “as best you can”, hoping that this “AI” will magically “make the right guesses” on all the details that can’t easily be defined using “natural” language.

It is in many ways as if you wanted to have mathematicians throw all their math symbols and rules for how to express math out the window, and instead “describe” the math using plain English. It would a very effective way of setting them back a few thousand years.

There are two sides to that coin. If people stop asking questions that humans can answer, the “AI” will run out of “knowledge” to draw on, and will instead increasingly have to start trying to serve a derivate of its own babble. It’s easy to see that the quality will drop rapidly if humans stop doing questions and answers “manually”.

I see the LLMs as nothing more than “advanced search tools”. It can save you a lot of time finding the information you’re after, no doubt, but it can also mislead you wildly; so you can’t really use it on an area where you don’t have a solid foundation of knowledge, because how can you then tell facts from fiction?

1 Like
19

You make good points about the difference between “working now” and “correct/optimal/robust.” I fully agree that AI output still needs human review, and I only use it in areas where I can validate the results.

Where we differ a bit is the idea that natural language interfaces replace programming. I don’t see them as a substitute for code, but as another abstraction layer, similar to how we moved from assembly to C, or from C to Python. I still end up editing, refining, or improving the AI-generated code manually.

In practice, I find that for well-defined tasks – even beyond trivial examples – the AI gives me a solid starting point. For me that reduces boilerplate and helps me explore ideas faster, not replace understanding or skill.

And as for forums, I don’t see AI reducing the need for human discussion. If anything, the questions I bring to human communities are now more specific and informed because I’ve already explored the easy angles with AI.

So yes, AI isn’t magic, and it still hallucinates. But as a productivity tool and accelerator, it’s been very effective for me when used with the right expectations.

I was simply offering my own experience from using these tools day-to-day, rather than stepping into the bigger ideological debates or older narratives. There’s room for different views here.

1 Like
20

“We” didn’t move from C to Python. C is very low level, relatively close to assembly, and I doubt that anybody experiences in C would move to Python - which is “at the opposite end of the spectrum”. I view Python more as either a very advanced scripting language or as a “rapid prototyping tool”, than an actual programming language. I know that others view it differently, but that’s how I see it.

My point about prompt writing is that it’s just inferior to code in every way, the only “benefit” it has, is that people who don’t know code can do it - but at the same time, those are the very same people that should not use it, because they cannot validate the result.

Using it for “templating” to save time and avoid boilerplate is a very different thing. I don’t have a problem with that, except that I think it has the risk of sneaking in bugs or flaws that you might not notice when “filling out the template”. In many ways, you have the exact same challenge when you find a snippet on SO or elsewhere on the net and use that as a template. Which is why I prefer to rather have the “template” open in another window (to peek at) while I write the code myself.

If it can fill that role, it’s a good thing. Nobody benefits from the same questions being answered over and over again. I’m just afraid that it can be much more trouble to help somebody out of a jam they have gotten themselves into by following “AI” instructions.

This is the crucial, yet difficult point. The alure is that you can do thing that you otherwise couldn’t do because of lack of knowledge, and if you do, you’re quite “defenseless” to resist being mislead.

I didn’t mean to imply that there isn’t room. I’m one of those people that often dig in and try to sort things out when then have already gone wrong. It can be hard enough to “backtrace” what humans have done and thought to “find the flaw”, having to do this with “AI” generated nonsense scares me.

I guess my primary motivation for “resisting” is to try to warn people of letting this “be the new norm”, causing a lot more or less unsolvable headaches. With all the financial backing behind the “AI push”, reason is pretty much guaranteed to lose, but I can’t help it: I have to try.