Git based non-public versioning and deploy workflow

Being neither overly familiar with Linux systems nor with git, I always wanted a versioning workflow for my (text based) openHAB config, but couldn’t find an easy to follow guide. Now I finally found the time to get to know git a bit better (see source) and decided to share my results with you.

What do I try to archive:

  • file versioning
  • keeping more than one up to date copy of my config files
  • no coding on my openHAB server
  • no cloud server involved
  • keeping the workflow as hassle free and as easy to use as possible

What I’m not aspiring to:
This is not a fully fledged backup solution, for that you should look at Amanda

Main Source
For git related steps I will reference Pro Git throughout this post. If you are unfamiliar with git or want to deepen your understanding, it would not hurt to read the whole book, but you can follow this post without doing so.


Setup

  • openHAB server (openHABianPi on Raspi 3, paths used in this post might vary on different setups)
  • text / file based openHAB 2 configuration
  • main computer (Mac-/Windows-Client)

Installation

Both platforms: To check if git is already installed, run the following command in your shell / Terminal / PowerShell:

git --version

OpenHABianPi does already come with a git installation, if your platform is missing git, have a look at Pro Git, Installation.

Initialization

Both plattforms: You might want to configure git and at least to set your credentials.

$ git config --global user.name "John Doe"
$ git config --global user.email johndoe@example.com

Additional explanations / options see Pro Git, First Time Git Setup

openHAB: Initialize a git repository in your config folder and stage & commit your existing files.
Optional: Additionally I create a .gitignore file, telling git to ignore temporary files, see also Pro Git, Recording Changes to the Repository, Section Ignoring Files:

$ cd /etc/openhab2/
$ git init
$ echo '*~' > .gitignore
$ git commit -a -m "<Initial Commit>"

Cloning

main computer: Now we create a new repository on our main computer as a copy of our openHAB repo.
So fire up your Terminal / PowerShell, navigate to a folder of your choice and use the following command:

$ git clone ssh://openhabian@openHABianPi/etc/openhab2/ --origin openHABianPi config

Explanation:

  • openhabian: A user with write / read permissions in your config folder on your openHAB server
  • openHABianPi: The DNS name of your openHAB server, you could use the IP address instead
  • /etc/openhab2/: Path to your config folder on your openHAB server
  • –origin openHABianPi: Name for the remote repo, used for push/pull commands in the future. If you omit this, the remote repo will get the default name ‘origin’
  • config: Folder name for the repo on your main machine
    After you send the command, you might be asked for the password of specified user and a new folder called ‘config’ will be created in the location of your choice, containing a copy of your openHAB repo.

Optional: SSH Keys

Avoiding password request for every ssh connection / push / pull / …
main computer: Follow Pro Git, Generating Your SSH Public Key
openHAB:

A second method is to create a single git user account on the machine, ask every user who is to have write access to send you an SSH public key, and add that key to the ~/.ssh/authorized_keys file of that new git account. At that point, everyone will be able to access that machine via the git account. This doesn’t affect the commit data in any way — the SSH user you connect as doesn’t affect the commits you’ve recorded.
(From Pro Git, Getting Git on a Server)

The single git account would be a user with write / read permissions in your config folder on your openHAB server (for example openhabian on openHABianPi setups), the public key is the key you just generated on you main computer.

Push to Deploy

main computer: If we now change stuff in our working repo, commit this stuff and want to push it to our openHAB repo, we will get an error message:

$ cd /path/to/working/repo
$ cd config/rules
$ vim example.rules
$ git commit -a -m "Did some changes to example.rules"
$ git push

The reason for the error message is, that git isn’t happy with pushes to non-bare repositories - reasons / details see here.

openHAB: To change that we have to set a flag in our openHAB repo:

$ cd /etc/openhab2/
$ git config --local receive.denyCurrentBranch updateInstead

Now your openHAB repo will accept pushed changes and will update the content of your config folder accordingly.
This can lead to trouble and lost changes if you have non-committed changes in your openHAB repo. If you set this flag change files only anymore in your working repo!

Remark: If you don’t want to push to deploy, you have to make the working repo available as remote repo in your openHAB repo and to pull, merge and checkout changes manually.

Finished

Now you don’t have to touch your openHAB server anymore to change your config, you just make your changes in your working repo, commit them and than push the change to your openHAB repo:
main computer

$ cd /path/to/working/repo
$ cd config/rules
$ vim example.rules
$ git commit -a -m "Did some changes to example.rules"
$ git push

Additional Options:

  • I’m using now my own git server as describted here: Setting up my own git server on a pi
  • VS Code, the supported editor for openHAB config files, does support git, meaning that you can push / pull changes directly in the editor without using a shell / Terminal / PowerShell.
  • put your working repo inside your Dropbox folder to have it available on more than one computer without using a cloud git server
  • Use a cloud git server if you don‘t care about an local workflow / if you want to publish your config (gitignore/encrypt sensitive files (e.g. files containing passwords)!)
  • A way to organize your repo with branches if you have more than one openHAB server
  • Use git for more than only your config folder (for example your userdata folder) you can either create more than on repo, or use symlinks to use only one repo (inspired by @rlkoshak). If you prefere the second option, you have to move the folders into your git folder and create the symlinks in the original location (git doesn’t follow symlinks):
sudo systemctl stop openhab2.service
sudo mkdir /home/openhabian/git
sudo mkdir /home/openhabian/git/config
sudo mkdir /home/openhabian/git/userdata
sudo mv /etc/openhab2/* /etc/openhab2/.* /home/openhabian/git/config/
sudo mv /var/lib/openhab2/* /var/lib/openhab2/.* /home/openhabian/git/userdata/
sudo rm -r /etc/openhab2/
sudo rm -r /var/lib/openhab2/
sudo ln -s /home/openhabian/git/config /etc/openhab2
sudo ln -s /home/openhabian/git/userdata /var/lib/openhab2
sudo chown -R openhabian /home/openhabian/git/
sudo systemctl start openhab2.service

EDIT: Be wary - at least in my case moving /var/lib/openhab2/ into my git repo caused so many permission based problems that I have to do a fresh openHAB installation.
EDIT: Added link ‘A way to organize your repo with branches if you have more than one openHAB server’


Questions

  • Which folders, other than config & userdata, might also be of interest for versioning?

Please let me know when you find mistakes / see ways to improve this post.

6 Likes

Thanks for the tutorial. I’ve been meaning to write something like this for quite awhile.

I’m not sure there would be any advantage in doing this. You are still relying on a cloud service. If you use a cloud git server like GitHub or Bitbucket at least the interface is specially designed to work with git.

I can endorse this approach. I used it until I moved to Docker. Once I moved to Docker the symlinks were no longer required.

An additional additional option is to run your own git server. Both Gogs and GitLab provide fully featured server and are relatively easy to install. This will give you the nice web UIs and tooling without needing to move anything off premises. I use Gogs primarily because I’m too lazy to make sure I don’t accidentally check in secrets.

Also, I do believe that Github at least, maybe Bitbucket too, now offers free private repos so even if you do accidentally check in something that you shouldn’t, the repo can only be accessed by you and users you have authorized.

In regard to the use of Dropbox:

  • You are right, a cloud service is a cloud service and I don’t know if Dropbox is more secure than for example GitHub (does anyone know more about the security of these two offers?). That’s one of the reasons why I put the Dropbox option under additional options - the basic setup should be usable without any cloud connection.
  • I might be wrong, but I would guess that I would have to open firewall ports allowing my openHAB server to be reachable from outside if I would configure GitHub in a way to keep the push to deploy mechanism (working repo -> GitHub -> openHAB repo)? That would be a no-go for me, my openHAB server is strictly restricted to my LAN :thinking:
  • having the working repo in a Dropbox folder it should not matter on which computer I work - the repo should always be in the same state regardless of commits / pushes :thinking:

I acknowledge the discussion about security and cloud providers. I am using gitlab with private repos and I trust gitlab.

Interesting tidbit. Gitlab provides gitlab-ci which is Continuous Integration feature of gitlab. With this it is fairly simple to deploy the conf files to the production system. Using this, after approx 20sec the code is pushed automatically to the server. You only need to start a small service which is lightweight. I have the gitlab-runner integrated in my openhab docker. Works like a charm.

You commit/push to origin and booom, deployed. So cool.

Anyone interested let me know.

2 Likes

Based on what I know of GitHub (I know very little of Dropbox) I’d say that GitHub is the more secure platform, at least in terms of login and access.

No, you would probably lose the push to deploy and have to do a pull to deploy on the production server. But that would be a bit more production like. You may have some long running changes that you are making that you want to incrementally check in and push to the upstream repo. You can’t do that if pushing to the upstream means deploying the changes.

I’m the last person on this forum who would suggest you open any port on your LAN to the internet. And in this case, that wouldn’t even work.

That’s a completely different problem. The way github works is you clone a repo and work in it. You can make changes, check stuff in, and all the rest in that local repo. As such, the proper git way would be to have a clone on each machine and use push and pull to keep them in sync.

To a large degree, you are using git in a way that runs counter to how it is designed to work. Using a source control like Subversion might be closer to your desired workflow.

openhab@openhab:/etc/openhab2$ git commit -a -m "Did some changes to example.rules"
On branch master

Initial commit

Untracked files:
	.gitignore
	OPENHAB.code-workspace
	automation/
	html/
	icons/
	items/
	persistence/
	rules/
	scripts/
	services/
	sitemaps/
	sounds/
	things/
	transform/

nothing added to commit but untracked files present
openhab@openhab:/etc/openhab2$  git push
fatal: No configured push destination.
Either specify the URL from the command-line or configure a remote repository using

    git remote add <name> <url>

and then push using the remote name

    git push <name>


Not sure but I think I messed up ?

openhab@openhab:/etc/openhab2$ git push -u origin master
error: src refspec master does not match any.
error: failed to push some refs to 'https://github.com/AllenSlaghuis/Openhab-Configs.git'
openhab@openhab:/etc/openhab2$ git push openhab-config
error: src refspec refs/heads/master does not match any.
error: failed to push some refs to 'openhab-config'
openhab@openhab:/etc/openhab2$ git push 
fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use

    git push --set-upstream origin master

openhab@openhab:/etc/openhab2$ 

One of the nice things about git is it often tells you how to fix the problems it encounters.

Run the two commands indicated.

git remote add <name> <url>
git push <name>

I think the two commands from before might fix this, but if not run git push --set-upstream origin master.

2 Likes

Thank you.
it’s working now. made the normal noob mistake . I jumped in to something new with out reading the documentation. did a lot of intense reading last night after receiving your reply.
got it working.
Love it. No more getting lost on what changed and what still needs to be changed.

:clap:

Warming up this rather old thread :smile:

I followed the tutorial and it works fine as long as I trigger push commands from my main computer via git push -u origin main from terminal / git bash. When doing so, I am then (always :roll_eyes:) prompted to type in the password of my openhabian user on the openHAB server. Once typed in everything works as expected.

…except I’m trying to run the whole thing out of Visual Studio Code, trying to use the integrated source control section: I can commit - no problem - but once I click on the “Sync changes” button I get an error message. In the output window I can read:

openhabian@192.168.***.***: Permission denied (publickey,password).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Is there a way to “save” my password somewhere or use similar methods to avoid the prompt in the terminal that might also be the reason for the error message when using the “Sync changes” button?

No trouble - you are looking for SSH keys. Best to google for a guide, but the gist is:

  • you generate a SSH key on your computer,
  • copy the public key to your git server into the file ~/.ssh/authorized_keys

From now on your computer will use said SSH key to authorize SSH connections to your git server and you don’t have to type a password anymore.