I ran into a problem with software that is not openHAB. But I’m posting what I’ve done here as it could be useful for openHAB users as well who wish to run the risk of exposing their home network to the Internet. I’m a computer security engineer by profession so I know what risks I’m assuming and how to mitigate them. If you do not know how to set this up, monitor it for attack and compromise, keep it up to date, and secure the rest of your network then this tutorial is not for you.
The Problem
I’m experimenting with Nextcloud. Unfortunately the folks over at Nextcloud have take the whole “security is built in” a little too far for a user like me. I access all of my home servers through OpenVPN and rely on that to provide the network security. But the sticking point is the Nextcloud Android app will only work through HTTPS with a valid certificate.
I do not want to get in the Certificate Authority business. It’s already a pain to keep up with for my SSH logins and Mosquitto. Generating and distributing self signed certificates to all my client devices is a step too far.
The Approach
Recently (within the last year) LetsEncrypt has added supports for wild card subdomains. This means you can get one certificated for *.foo.bar. Then you can run openhab.foo.bar, nextcloud.foo.bar, nightscout.foo.bar, and so on. This is great because lots of services one might run at home, such as openHAB, do not do well with reverse proxies trying to put them at foo.bar/openhab.
So we need to acquire a domain name that allows us to create subdomains. Then we can get the LetsEncrypt certs. Then we can set up pfSense and HAProxy as our reverse proxy. Finally I’ll discuss a little bit about monitoring.
Acquire a domain name
In order for this to work you need to acquire a domain name that supports:
- Dynamic DNS updates
- Gives you access to set TXT records
- Supports subdomains
- (optional but highly recommended) provides an API or supports NSupdate/RFC2136
I chose Namecheap because I was able to get a domain for $1 for the first year and < $10 per year each following year. However, I discovered
It supports all four of the listed requirements above and it even provides an API. But to use the API
you must have: Account balance of $50+, 20+ domains in your account, or purchases totaling $50+ within the last 2 years.
I don’t see that happening for me. So I will have some manual steps I’ll need to perform every 90 days or so to renew my LetsEncrypt certificate.
So you probably want to go with some other provider.
NOTE: I made a huge mistake here as I used my personal phone number when I registered, not realizing that that number was going to end up in my Whois record. If you have the option of not providing a phone number then don’t. If you must, use a secondary number like something offered from Google Voice. The spam, oh my lord the spam!
Set up Dynamic DNS
Once acquired, you will need to do some setting up of the domain name. namecheap provides a CPanel to configure everything. Your provider will probably have something similar. I’ll describe what I did as your screens will probably look different, and I don’t want to have to go through the trouble of redacting a bunch of information from screenshots.
What I had to do first was set up Dynamic DNS. For Namecheap one has to have at least one host to use with Dynamic DNS. I created two records, one using * as the host and the other using “ssh” as the host. I don’t know if both are necessary as once I got it working with “ssh” I added one for “*” in pfSense and both work but it might be because I did “ssh” first. Namecheap provides a Dynamic DNS Password which you will want to copy for the next step.
In pfSense go to Services > Dynamic DNS
Make sure you have a Check IP Services added and enabled. Then click the Add button in the Dynamic DNS Clients section to add a new client. Use the best information for your DNS provider. I used:
| Field | Value | Comments |
|---|---|---|
| Service Type | Namecheap | |
| Interface to monitor | WAN | |
| Hostname | * foo.bar | There are two fields and Namecheap requires both be filled out, there is information about what is needed for other providers under the fields |
| Wildcards | checked | |
| Password | copied from Namecheap above |
Everything else I left as the default.
When you save and apply it will immediately go out and update Namecheap. You should see your external IP address listed next to the Dynamic DNS record you created above now. If your pfSense is configured to respond to pings on the WAN interface than you should be able to ping it. But even if it isn’t, you can still see if the name resolves to your external IP address.
It might take a few minutes for the DNS records to resolve so be patient if it doesn’t work right away. If it shows the wrong IP address, see the next step.
Set up Host Records
Namecheap sets up a few records for you, namely a record to drop you on a landing page they host. This can cause your ping from resolving to the wrong IP address from the previous step.
I removed all the prepopulated host records and created three new ones. Again, I don’t know if the “ssh” record is required.
| Type | Host | Value |
|---|---|---|
| A + Dynamic DNS Record | * | Enter any valid IP address and it will be replaced with the dyndns IP address from the previous step |
| A + Dynamic DNS Record | ssh | This one may not be needed. |
| TXT Record | _acme-challenge | Enter any random stuff for the value for now. We will paste in a random string later. This field is what proves to LetsEncrypt that you own the domain. |
Again, these are what Namecheap calls it. Other DNS providers may call these something or work differently.
It has probably been long enough, your pings should start working if they were not working before.
Let’s Encrypt!
If you haven’t already, on pfSense go to System > Package Manager and install the ACME plugin. ACME is the protocol and software that LetsEncrypt uses to verify you own the domain and distribute the certificate.
Once it’s installed you will find a new entry under Services called Acme Certificates. Click on that.
Account Keys
First you need to create some account keys. LetsEncrypt is rate limited so you want to make sure that you have everything configured correctly before requesting a real cert. To help people test, LetsEncrypt provides a test service that you can use as you figure out your settings without bumping into the rate limit on the production servers. Certs obtained from these test services cannot be used.
So we will create two Account Keys.
Click the Add button and create a key for the testing server. I used:
| Field | Value |
|---|---|
| Name | test key |
| Description | Test Account Key |
| ACME Server | Let’s Encrypt Staging ACME v2 (for TESTING purposes) |
| E-Mail Address | your email |
| Account key | leave blank |
It is really important that you choose the Staging ACME v2 server. Only the v2 will support wildcard domains.
Now click the “+ Create new account key” button and wait for the box to fill in with a new RSA private key.
Finally click the “Register ACME Account key”. The little cog will spin and if it worked the cog will turn into a check.
Now repeat the process to create an account key on the Let’s Encrypt Production ACME v2 server. I named mine “prod key”.
Get you some certs!
Click on the Certificates tab and click the Add button.
| Field | Value | Comment |
|---|---|---|
| Name | I used my new domain name | |
| Description | An appropriate description | |
| Status | Active | |
| Acme Account | test key | |
| Private Key | 2048-bit RSA | |
| OCSP Must Staple | unchecked | |
| DNS-Sleep | blank | |
| Actions list | We will come back to this later | |
| Certificate renewal after | 90 | Leave blank if you are using a service that requires DNS-Manual (see below) |
Notice we skipped the Domain SAN List. You need to add two entries.
| Mode | Domainname | Method |
|---|---|---|
| Enabled | foo.bar | Appropriate method for your DNS provider, I had to chose DNS-Manual |
| Enabled | *.foo.bar | Same method chosen above |
Every level of the domain needs to have it’s own certificate. So we will get a certificate that covers both foo.bar and *.foo.bar.
Click on “Issue” and review the output. There will be a line that tells you to copy a randomly generated string to a TXT record under _acme-challenge.foo.bar. Copy this string (excluding the single quotes) and paste them into the Value filed for the _acme-challenge host record created above. Wait a couple of minutes.
Now click on “Renew” and review the output. It should tell you whether the request was successful or not. If there are any errors, review your Domain SAN List for errors. If it shows success then you have a brand new LetsEncrypt cert. You will find it listed under System > Cert Manager > Certificates. It will show the issuer as something like “Acmecert: 0=Let’s Encrypt,CN=Let’s Encrypt Authority X3,C=US”.
Once you’s shown you have everything configured correctly, edit the certificate and change the Acme Account from “test key” to “prod key” and repeat the issue and renew steps to generate a usable certificate.
Firewall Rules
Now we need to allow ports 80 and 443 through the firewall so that the Internet can reach our HAProxy. In pfSense browse to Firewall > Rules. Click Add and enter:
| Field | Value |
|---|---|
| Action | Pass |
| Disabled | unchecked |
| Interface | WAN |
| Address Family | IPv4 |
| Protocol | TCP |
| Source | any |
| Destination | This firewall (self) |
| Destination Port Range | From HTTP(80) To HTTP(80) |
| Log | Checked |
| Description | Allow HTTP Inbound |
Save and appy.
Click Add and enter:
| Field | Value |
|---|---|
| Action | Pass |
| Disabled | unchecked |
| Interface | WAN |
| Address Family | IPv4 |
| Protocol | TCP |
| Source | any |
| Destination | This firewall (self) |
| Destination Port Range | From HTTPS(443) To HTTPS(443) |
| Log | Checked |
| Description | Allow HTTPS Inbound |
HAProxy
In pfSense, return to System > Package Manager and install HAProxy. HAProxy is a special purpose reverse proxy and it will do the same job for us that nginx or Apache does as described here.
Once successfully installed, go to Services > HAProxy. HAProxy consists of Frontends and Backends. The Backends represent your services running in your LAN. This is where you define your machine host or IP address and port number hosting your service. You will create a separate Backend for each service you want to expose.
Frontends are where you map a subdomain to a given Backend. This is where http-https redirects and adding authentication will be configured.
HAProxy Settings
| Field | Value | Notes |
|---|---|---|
| Enable HAProxy | checked | |
| Maximum connections | 100 | I chose something low because I will only have two to three simultaneous users. I will increase if this causes problems. |
| Number of processes to start | left blank | |
| Reload behavior | checked | |
| Reload stop behavior | left blank | |
| Carp monitor | Disabled | |
| Internal stats port | 2200 | |
| Internal stats refresh rate | left blank | |
| Stickable page refresh rate | left blank | |
| Logging | left defaults | Setting up an rsyslog server or some other centralized logging server is a good idea |
| Global DNS resolvers for haproxy | left defaults | |
| Global email notifications | left defaults | |
| Max SSL Diffie-Hellman size | 2048 | If left blank a warning will appear every time you apply changed settings because it defaults to 1024 which is too small. |
Users and Passwords
In pfSense, navigate to Services > HAProxy and click on Settings. Scroll down to “Global Advanced pass thru”. We will be adding a list of usernames and passwords to this box. These are the users that will be used to authenticate. The first think you may worry about is having passwords in plain text in your pfSense configs. Fret not, you can use a salted hash for the password.
You will enter the following into the box:
userlist UserGroup
group is-user
user User1 password SECRET groups is-user
user User2 password SECRET groups is-user
This will create a UserGroup with two users, User1 and User2. This also creates an is-user group. You can create more than one. This can be handy if you need separate users for separate services.
So how do you create an encrypted password? HAProxy supports DES, MD5, SHA-256, and SHA-512. I’ll show SHA-512.
There are a number of ways to do this. The easiest is if you have python3 installed. In this case you can copy the following and paste it on your command line, replacing the password of course:
python3 -c 'import crypt; print(crypt.crypt("TheSuperSecretPasswordHere", crypt.mksalt(crypt.METHOD_SHA512)))'
The salted and hashed password will be printed out. Paste that over “SECRET” for the given user.
If you are uncomfortable having your password in your command line history and visible on your screen, you can use the following and then type in your password.
python3 -c 'import crypt,getpass; print(crypt.crypt(getpass.getpass(), crypt.mksalt(crypt.METHOD_SHA512)))'
All credit for the above goes to User Authentication with HAProxy on pfSense.
HAProxy Backends
First let’s create a Backend. In pfSense go to Services > HAProxy and click on Backend. Then click the Add button. The following settings are for exposing openHAB with a username and password authentication.
Name: openhab.koshak.us
Server list:
| Mode | Name | Forwardto | Address | Port | Encrypt(SSL) | SSL checks | Weight |
|---|---|---|---|---|---|---|---|
| active | openhab | Address+Port | IP of openHAB server | 8080 | unchecked | unchecked | leave blank |
| Leave the rest of the options blank and unchecked. |
Now for the ACLs and actions. This will be where we add the authentication.
Access Control lists:
| Name | Expression | Value |
|---|---|---|
| BackendAccess | Custom acl: | http_auth(UserGroup) |
Actions:
| Action | Parameters | Condition acl names | Notes |
|---|---|---|---|
| http-request auth | realm FooBar unless BackendAccess | blank | enter the Parameters into the “realm:” box that pops up after selecting http-request auth |
I ran into some trouble here largely because of some extra security built into Nextcloud and the monitoring that HAProxy does on the services. You will see in the Backend configuration a Health Checking section. This control the settings for how HAProxy periodically checks that the backend server is alive. You can leave everything with the defaults for openHAB. Unfortunately, Nextcloud operates using a whitelist approach rather than a blacklist approach to security. Consequently I had to go in and whitelist nextcloud.foo.bar, and the local machine’s IP address in order for it to accept connections from HAProxy. In addition, I had to check method to GET and set the header to “HTTP/1.1\r\nHost:\ name.of.host:8080” where name.of.host is the fdqn of the host serving Nextcloud. Without the whitelisting and these other configuration tweaks HAProxy kept seeing the service as offline.
HAProxy Frontends
Shared Frontend
All of the connection requests will be coming in to the same IP address and port but we need a way to distinguish between requests so that those for openhab.foo.bar go to the openhab backend and those for nextcloud.foo.bar go to the nextcloud backend. So we will create a shared front end and then sub front ends for each subdomain.
Click on Frontend and click the Add button.
| Field | Value | Notes |
|---|---|---|
| Name | shared-fontend | |
| Description | Shared frontend | |
| Status | Action | |
| Shared Frontend | unchecked | We will check this for the subdomains |
| External address | Select WAN address, Port 433, and check SSL Offloading | |
| Type | http / https(offloading) | |
| Use “forwarder” option | checked | |
| Certificate | Select your Let’s Encrypt certificate | Under SSL Offloading. Check the two checkboxes under this field |
Leave the rest of the fields blank or at their default values.
Save and apply.
openHAB Fronend
Click Add again.
| Field | Value | Notes |
|---|---|---|
| Name | openhab.foo.bar | |
| Description | openHAB authenticated frontend | |
| Status | Action | |
| Shared Frontend | checked | |
| Primary frontend | shared-frontend |
Now for the ACLs. These will match the hostname (i.e. openhab.foo.bar) to the openhab Backend. This will also be where we add the authentication.
First we will set up the authentication, then the forwarder.
Access Control lists:
| Name | Expression | Value |
|---|---|---|
| openhab-acl | Host matches: | openhab.foo.bar |
Actions:
| Action | Parameters | Condition acl names | Notes |
|---|---|---|---|
| Use Backend | openhab.foo.bar | openhab-acl | select the backend in the "backend box that pops up after selecting Use Backend |
HTTP to HTTPS Redirect
Now we want to set up a redirect so any attempts to connect using HTTP gets forwarded to HTTPS. In pfSense click on Frontend and click the Add button.
Field|Value
-|-|-
Name|http-to-https
Description|http to https redirect
Status|Action
Shared Frontend|unchecked|
External address|Select WAN address, Port 80, and uncheck SSL Offloading
Type| http / https(offloading)
In the Default backend, access control lists and actions section create a new Action.
| Action | Parameter | Notes |
|---|---|---|
| http-request redirect | scheme https | enter in the rule: box that appears when selecting http-request redirect. |
Troubleshooting
I’m no expert and will not be of much comprehensive help. But in pfSense if you go to Status > HAProxy Stats you will see the status of all your Frontends and Backends. Look here first if you get 503 errors. If everything isn’t Green then research the error to figure out what’s wrong. This is how I diagnoed and solved the Nextcloud problem. Essentially I searched Google for the error then did some more searching for “Nextcloud HAProxy pfSense”.
Security and monitoring
As I mentioned above, this section is not comprehensive and it is not a tutorial. At best it is a list of tutorials you should research and implement on your home system.
Before I go into it, a little bit about the overall goal of security. If you will be exposing your home network to the Internet in this way, you need to be aware that it is a matter of when, not if, your network will come under attack. Remember, you are not a target because of who you are, you are a target because of what ports you have exposed to the internet and what services you have running at those ports. With services like Shodan, it’s a matter of a simple search for someone to find a bunch of potential targets. And there are plenty of less legitimate services that offer the same service.
If you expose a port, you are a target and you will be constantly under attack. The good news it will be by scripts and automated systems. The bad news is the reasons those scripts and automated attacks work is because they work.
So what is an administrator to do? Your goal will be to make it so should an attack succeed, the amount of damage that can be done is minimized. This is sometimes called defense in depth.
The following are some ways you can provide defense in depth in your system.
| Name | Description | Notes |
|---|---|---|
| Strict firewall rules | if you know only certain IPs will talk to other IPs, set up rules, either in your main firewall or on the hosts to enforce that. | Should a host be compromised, the attacker’s ability to move laterally on your network is reduced. |
| Demilitarized Zone (DMZ) | Ideally, HAProxy above would be running on a hardened machine in an isolated network called a DMZ. There would be two firewalls, one between the DMZ and the Internet and another between the DMZ and your LAN. | Should your reverse proxy become compromised, they are stuck inside a network and host that doesn’t have anything of value. By using myopenhab.org, you are in a sense letting the openHAB Foundation run a DMZ for you. |
| Network segmentation | Create separate networks for your data processing and your home automation. Configure your home automation network so that devices can’t see each other. | This limits the ability of an attacker from moving laterally on your network. |
| Centralized logging | Centralize all of your audit logging into one place so it’s easier to monitor. | You must review the audit logs periodically for signs of a successful attack. |
| Change all the passwords | Leave no default password in place | |
| Use encryption for sensitive data | Use a program like VeraCrypt and put any sensitive data in an encrypted volume. Only mount this volume when you need to. Encrypt sensitive files on disk (e.g. Nextcloud offers an encryption option). | Should an attack be successful, even if they acquire your files, they cannot read them. |
| Limit what individual services can do | Run all services under a limited user account, never as root. Provide further isolation using a chroot or container (though the container could bring in new vulnerabilities so be careful). | This is why OH runs under a restricted openhab user and why you should never give that user unrestricted sudo rights. |
| Run attack detection/prevention | There are programs that look for behaviors that indicate an attack and they can be configured to block them. Snort and Bro are two commonly used ones. | These are very challenging to set up and require close monitoring. |
| Encrypt network traffic | If it leaves a host (e.g. it’s not just traffic between two containers on the same host) then the traffic should be encrypted to prevent an attacker from sniffing useful information. | |
| Use two factor authentication where possible | This can be two factor like you are used to such the SMS message with a key to type in, or it could be by using password protected client certificates for authentication. | The something you have is the certificate, the something you know if the password for the certificate. HAProxy has options for setting up either type of authentication. |
| Use fail2ban | This script looks at audit logs and bans IP addresses (i.e. sets a firewall rule) that fails multiple times. | Sadly pfSense doesn’t support fail2ban through the GUI. I plan on looking to see if I can make it work from the command line, though it’s FreeBSD so that may not work. |
| Configure TLS certs on all services | For internal networks, you will need to establish your own CA, add a CA cert to all your client devices, and then create self signed certs for all your services. | This prevents man-in-the-middle attacks. |
| System hardening | Especially for machines that are directly exposed to the Internet there should be some significant hardening. This can include blocking all but absolutely necessary ports with a host based firewall, stopping and removing any unneeded software, and using kernel modules/systems to further prevent actions that software or users can do beyond what is allowed by standard Discretionary Access (DAC) such as SELinux, Apparmor, etc. For monitoring, a strict Tripwire policy (or equivalent) is recommended. | |
| Monitor | This was mentioned above but I’ll give it it’s own category. You need to be made aware of any failed action and any change. You can’t just set all the above up and let it run. You must get alerts from them all and actually read the alerts. You must review the audit logs and the syslogs for suspicious behavior. |
The above table is really just off the top of my head. Creating a secure service that is exposed to the Internet takes a lot of planning, configuration, and ongoing work. I do not recommend it unless you have the knowledge and skills to do all of the above and the corresponding understanding of the risks you are accepting should you fail to do all of the above.
Conclusion
Hopefully the above will be found useful to someone!
Additions, comments, and corrections are welcome!
I currently have the above set up and working with several services and it works great. Being able to use subdomains (openhab.foo.bar) instead of suppaths (foo.bar/openhab) has really made setup and use super flexible and easy to set up.
