ChatGPT binding with consecutive chat / conversation?

[Hundertvolt1 (Nico Steinhardt)]

Hi, I just had a crazy idea with ChatGPT as a natural conversation engine capable of triggering OH actions. Well, doesn’t seem too far away, see below, but one thing which I obviously did not get working is a real conversation as you get on the OpenAI website.

I have a working ChatGPT binding and created a new chat channel, nothing weird about that. The issue I have is that every time I command a string to this channel, the chat starts “dumb” without a conversation history, and I would like to know if I missed something, did anything wrong or if it’s not possible.

Here you can see what could be possible with a first prompt (this is from the openAI website):

User
You are an interpreter between a human and a computer. You know the functions of the computer and the corresponding commands, I'll explain that later.
The human will approach you, and your task is to find out which function the human desires. When you found this out, you give the command to the computer. 
The computer can turn on the light in the bathroom, the command is BATH_LIGHT_ON.
The computer can turn off the light in the bathroom, the command is BATH_LIGHT_OFF.
The computer can open the blinds in the bathroom, the command is BATH_BLINDS_OPEN.
The computer can close the blinds in the bathroom, the command is BATH_BLINDS_SHUT.
You give a command to the computer by writing the tag <CMD>, followed by the command exactly as described. You answer the human by writing the tag <ANS>, followed by the message to the human. No other tags are allowed.
No computer commands are allowed to be contained in the message to the human. When you gave a command to the computer, inform the human that you did.
From now on you'll wait until a human approaches you.

ChatGPT
<ANS>Hello there! How can I assist you today?

User
Hey!
ChatGPT
<ANS>Hello! How can I help you today? If there's something specific you'd like the computer to do, feel free to let me know.

User
What can you do for me?
ChatGPT
<ANS>I can assist with various functions related to the bathroom. For example, I can turn on the light, turn off the light, open the blinds, or close the blinds. Please let me know which function you'd like to perform.

User
I'd like to turn off the light
ChatGPT
<CMD>BATH_LIGHT_OFF

<ANS>The light in the bathroom has been turned off. Is there anything else you need assistance with?

User
Yes, can you open the blinds?
ChatGPT
<CMD>BATH_BLINDS_OPEN

<ANS>The blinds in the bathroom have been opened. Is there anything else you would like to do?

just imagine what you could do with the ANS and CMD tag by simple string processing in a rule…
Of course, it may fail in rare cases and generate crap, but this as well is nothing too bad for string processing. Of course, don’t use this for (better: don’t even enable it to do) critical or dangerous actions. But for all the rest, this is an incredible and way easy opportunity.

What I observed when I tried this with the binding, I got the same answer for the initial prompt and for the “Hey”, but when I asked it “What can you do for me?”, I get this:

I'm here to assist you with a variety of tasks! Here are some things I can help you with:

Answering Questions: I can provide information on a wide range of topics, explain concepts, and offer insights.

Language Translation: I can translate text between different languages.

Coding Help: If you're working on a programming or coding task, I can help you understand concepts, debug code, or provide examples.

Writing Assistance: Whether you need help with creative writing, essay composition, or general content creation, I can assist you in generating text.

Learning New Topics: I can help you learn new subjects by providing explanations and answering questions.

Problem Solving: If you have a problem or a decision to make, I can offer suggestions and help you think through the issue.

Conversation and Entertainment: If you just want to chat or have some fun, I can engage in casual conversation, tell jokes, or play text-based games.

Remember that while I can provide information and assistance, I don't have real-time capabilities, and my responses are generated based on patterns and information available up to my last training cut-off in January 2022. If you have a specific question or task in mind, feel free to let me know!

…which is the standard reply for this question in a new chat, without any history. With this, all the cool stuff you can think of does not work, so I hope that someone knows how to establish a chat with history there…

[dalgwen (Gwendal Roulleau)]

Basically, you want to use ChatGPT as an intent classifier ?

I didn’t test, but maybe you can use the “systemMessage” channel for this.
The text you put in this channel will be inserted before each command you utter.

This is not a complete history, but it clearly is a first step.

[Hundertvolt1 (Nico Steinhardt)]

@dalgwen

Basically, you want to use ChatGPT as an intent classifier ?

Well, in a second step kind of, but not just.
My first step actually was using the “systemMessage” before - I experimented and ended up with giving a prompt primer with this channel, added stringified measurement data from my environment sensors and told ChatGPT to inform me when my dishwasher has done its job and justify this with the weather
Besides causing bursts of laughter in the whole family, I saw something interesting: ChatGPT almost naturally interpreted the sensor data to how this actually feels (as the reason not to go out but care for the dishes ). All that already with a combined constant (systemMessage) and variable (channel) prompt.

This gave me an idea of what could be possible. When I got the Signal Binding running (we already wrote there ), I thought how it would be to chat with my smart home in natural language, but with the ability to also act. So I tested some prompts, as written before, in the OpenAI chat website and found that all of this looks very possible.

Some further tests I did also went quite well when commanding multiple things at a time, and besides interpreting, it will friendly support you if you forget a function or the like. I have, seriously, several hundred sensors and actuators. You can’t remember all of them

At some later point, I plan to test that the “AI interpreter” can also query the computer for measurement data. I would strongly suppose that ChatGPT would give you a reasonable answer based on real local measurement data if you asked it “Which clothing should I wear today” in that context.

The further these things get, the more it becomes impossible to do this without history and the same prompt allover again. Commanding actuators may still be possible if you exactly know with the first message what you want, but the number of identical tokens being sent each time will explode.

When querying for sensor data, history is definitely required. When ChatGPT recognizes that it needs data (minimum one prompt from the user), it must answer to the “computer” (first answer), awaits its response (minimum second prompt), interpret the response and answer (second answer) to the user.

So, long story short: without history, ChatGPT is not able to play one of its greatest strengths - a real dialog, as on the OpenAI chat site.
So technically speaking, to get full advantage over ChatGPT from the binding, it’s required that it can control the same properties as on the OpenAI site:

• One command to open a new, virgin chat without any history with a one-time init prompt
• All further communication with history

So at least, these are my ideas What do you think about it?

[dalgwen (Gwendal Roulleau)]

OK I see !
When LLMs explode, I, like you and many others, thought about their capabilities as an assistant for our home. It seems promising, indeed.

What you really need is :

• More than “just” a binding, a Human Langage Interpreter addon with a LLM
• That use function, like the ChatGPT function (it is basically what you used when you ask ChatGPT to answer with your command like “BATH_LIGHT_OFF”, but in the “proper” way to do it)

Good news : it’s totally doable, and I’m nearly sure that it will be, some day, as it seems the best way to use an assistant.
Bad news : it could take time before someone get on it

In the meantime, it should be possible to indeed “hack” your way IF a chat history was to be activated, like you suggest with predefined “command”.
But looking at the ChatGPT binding code, it actually isn’t.
The proper way to do it is to record questions and answers and provides it to the ChatGPT api with every call with the right parameter in the api call (i.e. a list of previous messages { “role”: “user”, “content”: “my first question” } { “role”: “assistant”, “content”: “the first answer” } etc.).

The binding doesn’t use this functionnality as for now, so if you want it, you will have to be proficient in one of the scripting langage openHAB offers. You should then be able to record every message/answers each time the channel receive updated/command and append them before your next prompt.

[Hundertvolt1 (Nico Steinhardt)]

@dalgwen

That use function, like the ChatGPT function (it is basically what you used when you ask ChatGPT to answer with your command like “BATH_LIGHT_OFF”, but in the “proper” way to do it)

Amazing! Thanks for the link, I did not have an idea that this is actually possible.
With a Java package and JSON output. Sam Altman must have thought of OpenHAB when he returned as CEO
Yeah, well, seriously, this seems like the way to go. Of course with the grain of salt that it might act unexpectedly from time to time, but hey, the really safety critical stuff (door lock or the like) or potentially dangerous devices (e.g. high-power heaters) should not depend on a cloud service anyway. For all the rest, it’s a great opportunity.
So desperately hoping that the time until it works won’t be soooo long

@dalgwen

In the meantime, it should be possible to indeed “hack” your way IF a chat history was to be activated, like you suggest with predefined “command”.
But looking at the ChatGPT binding code, it actually isn’t.

Also here, I’m not sufficiently deep in the details. My expectation somehow was that a “chat” is an instance of an object in the ChatGPT API, and as long as you don’t reset or delete it, it will automatically keep its history - same thing as with the ChatGPT Website - you have your sideboard even with very old chat sessions you can revive anytime. Or delete them, if they are obsolete - and that the binding in its current state would create a new chat object with every input and lose track of any object before.

What you say now sounds way different - more like that there is no inherent history at all (at least not in the API). Or is there, and it’s just not implemented (yet) in the binding?

How are they actually doing this on the website?

I also was assuming that giving the complete history would make ChatGPT act as expected, but wouldn’t this also include repeating the behaviour setting prompt allover again? And especially for longer chat sessions, wouldn’t this consume massive amounts of tokens? Of course, the costs per token in the API is super low, but even then, this sounds very wasteful. Or is the way you describe (give the recording via parameter in the API call) avoiding exactly this scenario and only raising token count for what is really new?

@dalgwen

The binding doesn’t use this functionnality as for now, so if you want it, you will have to be proficient in one of the scripting langage openHAB offers. You should then be able to record every message/answers each time the channel receive updated/command and append them before your next prompt.

From what’s required in the script, this does not worry me for one single processing cycle
But I’m not sure if I got this correctly. In my understanding, you can use the channel to chat with the AI. The channel uses string format only. What you say now sounds like you can actually include JSON directly within the chat string which the API will recognize even before the actual AI processing takes place?

Thank you very much for all you already described so far!

[dalgwen (Gwendal Roulleau)]

@Hundertvolt1

I also was assuming that giving the complete history would make ChatGPT act as expected, but wouldn’t this also include repeating the behaviour setting prompt allover again? And especially for longer chat sessions, wouldn’t this consume massive amounts of tokens? Of course, the costs per token in the API is super low, but even then, this sounds very wasteful.

Exactly. This is the way of all LLM. They are “completion” model. They predict the next word from a previous set of words. They have no memory or whatever. And yes, the previous chat also counts as token : the longer your chat is, the more costly (in resource and so in money) each next request is.

So, yes, the ChatGPT GUI also does that : it stores your old conversation and send it back every new prompt you give. But there is some optimization : from what I’ve read, ChatGPT GUI does (server-side) a summary of the previous prompt, keeping important informations only instead of sending it raw to the LLM, to save some token for subsequent requests.

When you use the API (as with this binding), you have to do it by yourself. The API is “raw” to let consumer use it as they want, and does not store anything.

@Hundertvolt1

What you say now sounds like you can actually include JSON directly within the chat string which the API will recognize even before the actual AI processing takes place?

No, I was not clear, sorry : I was giving a technical response on how the API can be used when you have access to all the request parameters.
In your case, I’m saying that you could simulate such a functionnality by appending all questions and responses yourself to the prompt, recreating what the API probably does when the server receives the list of old request/answer in json format. Probably by appending before each previous request / answer who is the author for the LLM to recognize what is a request and what is an answer (as you did in your first post).

Something I forgot : you can also open a feature request with your use case on github for the ChatGPT binding to use the history properly. Someone may find it interesting and take it.

[Hundertvolt1 (Nico Steinhardt)]

Hi again, thanks for your hints and explanations!

Meanwhile, I experimented a bit. Seems we’re on a way there!
I set up a ChatGPT history-enabled conversation with a Signal account:

// Rules
rule "ChatGPT Test Dialog - User"
when
  Item Signal_Message_Received received update
then
  val receivedMsg = Signal_Message_Received.state.toString().trim()
  val receivedJson = "{\"from\": \"user\", \"to\": \"James\", \"message\": \"" + receivedMsg + "\"}\n"
  if (dialog == "") {
    OpenAI_Account_Assistant_Chat.sendCommand(startMsg + receivedJson)
    dialog += receivedJson
  } else {
    OpenAI_Account_Assistant_Chat.sendCommand(historyMsg + dialog + newMsg + receivedJson + reminder)
    dialog += receivedJson
  }
  logInfo("Assistant.User", "dialog = {}", dialog)
end

rule "ChatGPT Test Dialog - AI"
when
  Item OpenAI_Account_Assistant_Chat received update
then
  val receivedMsg = OpenAI_Account_Assistant_Chat.state.toString()
  val userMsg = transform("JSONPATH", "$.message", receivedMsg)
  Signal_Send_Message.sendCommand(userMsg)
  dialog += receivedMsg + "\n"
  logInfo("Assistant.AI", "dialog = \n{}", dialog)
end

This actually works like a charm - see test conversation below.

By the way, I set the “temperature” parameter in the channel I used for this to 0.2 to make it act rather reasonable.

What I found most interesting:

• I first made a prompt to explain ChatGPT what to do and to use JSON. I forgot to add this variable to the string and just confronted ChatGPT with a JSON input. This, even including correct use of the names in the “from” and “to” fields, worked immediately
• I added the prompt later on when I found my mistake. This lead to very erratic behavior, like just repeating my inputs or the like. So I just threw away the prompt
• There is a slight issue: ChatGPT tends to forget to use JSON in a reply it considers somewhat final in the conversation or which is very long. It was for example the case for the “brand recommendation” and “recipe” answer in the test I did. This seems to be fixed with the “reminder” message which is added at the end of each prompt (I saw no more JSON flaws after adding this reminder!)

The Signal chat of course is parsed and looks like a natural conversation (no measures taken to enforce waiting for a reply yet, though). Way more interesting is the actual history - this is the final history content after a “nice chat” via Signal:

dialog = 
{"from": "user", "to": "James", "message": "Hi there"}
{"from": "James", "to": "user", "message": "Hello! How can I assist you today?"}
{"from": "user", "to": "James", "message": "I am looking for a small present"}
{"from": "James", "to": "user", "message": "Sure! What is the occasion and budget for the present?"}
{"from": "user", "to": "James", "message": "A Christmas party"}
{"from": "James", "to": "user", "message": "Great! What is your budget for the Christmas party present?"}
{"from": "user", "to": "James", "message": "Just a small gift, nothing expensive"}
{"from": "James", "to": "user", "message": "Understood. I can suggest a few small gift options within your budget. Can you please provide me with an approximate budget range?"}
{"from": "user", "to": "James", "message": "Just some cookies"}
{"from": "James", "to": "user", "message": "Great choice! Cookies make for a delicious and thoughtful gift. Is there a specific type of cookies you would like, or any dietary restrictions I should be aware of?"}
{"from": "user", "to": "James", "message": "I love chocolate"}
{"from": "James", "to": "user", "message": "That's great to hear! Chocolate cookies are always a popular choice. We have a variety of chocolate cookies available. Would you like me to recommend a specific brand or flavor?"}
{"from": "user", "to": "James", "message": "Yes"}
{"from": "James", "to": "user", "message": "Sure! I can recommend a few popular brands for chocolate cookies. Some options include: 
1. Nestle Toll House Chocolate Chip Cookies
2. Pepperidge Farm Milano Dark Chocolate Cookies
3. Oreo Chocolate Sandwich Cookies
4. Keebler Fudge Stripes Dark Chocolate Cookies
Please let me know if any of these options interest you, or if you have any other preferences."}
{"from": "user", "to": "James", "message": "Do you have a recipe?"}
{"from": "James", "to": "user", "message": "Yes, I can provide you with a recipe for chocolate cookies. Here's a simple and delicious recipe you can try:
Ingredients:
- 1 cup (2 sticks) unsalted butter, softened
- 1 cup granulated sugar
- 1 cup packed brown sugar
- 2 large eggs
- 1 teaspoon vanilla extract
- 2 1/4 cups all-purpose flour
- 1/2 cup unsweetened cocoa powder
- 1 teaspoon baking soda
- 1/2 teaspoon salt
- 2 cups chocolate chips
Instructions:
1. Preheat your oven to 350°F (175°C) and line a baking sheet with parchment paper.
2. In a large mixing bowl, cream together the softened butter, granulated sugar, and brown sugar until light and fluffy.
3. Add the eggs one at a time, beating well after each addition. Stir in the vanilla extract.
4. In a separate bowl, whisk together the flour, cocoa powder, baking soda, and salt. Gradually add the dry ingredients to the butter mixture, mixing until just combined.
5. Fold in the chocolate chips.
6. Drop rounded tablespoons of dough onto the prepared baking sheet, spacing them about 2 inches apart.
7. Bake for 10-12 minutes, or until the edges are set and the centers are slightly soft.
8. Allow the cookies to cool on the baking sheet for 5 minutes, then transfer them to a wire rack to cool completely.
Enjoy your homemade chocolate cookies! Let me know if you have any other questions or if there's anything else I can assist you with."}

I guess this is something to be extended and to work with

[dalgwen (Gwendal Roulleau)]

Very nice hack !
Now you make me curious and I want to try it, maybe with a LocalAI.
Thanks for the share.