Google Family Location Sharing in OpenHAB via python and MQTT

Rationale:

For advanced presence detection (and another project I’m working on) I wanted to get the GPS locations from the smartphones of my family members. At first, I considered using a specialized app, however since my family already uses Google Maps location sharing extensively and it works extremely well, I decided it would be easiest and probably the most reliable of all the options available to me.

Approach:

The major complication is that Google does not provide a real API to get locations from the location sharing. That said, there are workarounds, namely locationsharinglib, a python library used to do just this. Then, to get these actual values into OpenHAB, I decided to use MQTT as it’s what I’m most familiar with and already use extensively. This could probably also be done using the REST API or something similar.

The script:

This Python script is what fetches the location data and publishes it to MQTT.

import random
import time

from paho.mqtt import client as mqtt_client
from locationsharinglib import Service

#Location Sharing Library config
cookies_file = 'COOKIE_FILE_PATH.txt'
google_email = 'GOOGLE_USER@gmail.com'

#MQTT Configuration
broker = '192.168.1.114'
port = 1883
topic = "googlelocation/"
client_id = 'python-mqtt-{random.randint(0, 1000)}'
username = ''
password = ''

#Update Interval (in seconds)
update_interval = 60

service = Service(cookies_file=cookies_file, authenticating_account=google_email)

def connect_mqtt():
    def on_connect(client, userdata, flags, rc):
        if rc == 0:
            print("Connected to MQTT Broker!")
        else:
            print("Failed to connect, return code %d\n", rc)

    client = mqtt_client.Client(client_id)
    client.username_pw_set(username, password)
    client.on_connect = on_connect
    client.connect(broker, port)
    return client

def publish(client):
    while True:
        time.sleep(update_interval)
        for person in service.get_all_people():
            for data in dir(person):
                if not data.startswith('_'):
                    print(topic + person.nickname + "/" + data, str(getattr(person, data)))
                    client.publish(topic + person.nickname + "/" + data, str(getattr(person, data)))


def run():
    client = connect_mqtt()
    client.loop_start()
    publish(client)


if __name__ == '__main__':
    run()

Things, items, and transforms

.things file

//NAME Location
Thing mqtt:topic:googlelocation_NAME "Google Location: NAME" (mqtt:broker:e0c4d01e) @ "Server Room"{
Channels:
    Type number   : Accuracy         [ stateTopic="googlelocation/NAME/accuracy"]
    Type string   : Address          [ stateTopic="googlelocation/NAME/address"]
    Type number   : Battery_Level    [ stateTopic="googlelocation/NAME/battery_level"]
    Type contact  : Charging         [ stateTopic="googlelocation/NAME/charging", on="True", off="False"]
    Type string   : Country_Code     [ stateTopic="googlelocation/NAME/country_code"]
    Type datetime : Timestamp        [ stateTopic="googlelocation/NAME/datetime", transformationPattern="JS:googletimestamp.js"]
    Type string   : Full_Name        [ stateTopic="googlelocation/NAME/full_name"]
    Type string   : Id_number        [ stateTopic="googlelocation/NAME/id"]
    Type number   : Latitude         [ stateTopic="googlelocation/NAME/latitude"]
    Type number   : Longitude        [ stateTopic="googlelocation/NAME/longitude"]
    Type string   : Photo_URL        [ stateTopic="googlelocation/NAME/picture_url"]
}

.items file

//NAME Location
Number   GPSLocation_NAME_Accuracy       "Accuracy"       (Logged) {channel="mqtt:topic:googlelocation_NAME:Accuracy"           }
String   GPSLocation_NAME_Address        "Address"        (Logged) {channel="mqtt:topic:googlelocation_NAME:Address"            }
Number   GPSLocation_NAME_Battery_Level  "Battery Level"  (Logged) {channel="mqtt:topic:googlelocation_NAME:Battery_Level"      }
Contact  GPSLocation_NAME_Charging       "Chargin"        (Logged) {channel="mqtt:topic:googlelocation_NAME:Charging"           }
String   GPSLocation_NAME_Country_Code   "Country Code"   (Logged) {channel="mqtt:topic:googlelocation_NAME:Country_Code"       }
DateTime GPSLocation_NAME_Timestamp      "Timestamp"      (Logged) {channel="mqtt:topic:googlelocation_NAME:Timestamp"          }
String   GPSLocation_NAME_Full_Name      "Full Name"      (Logged) {channel="mqtt:topic:googlelocation_NAME:Full_Name"          }
String   GPSLocation_NAME_Id_number      "ID Number"      (Logged) {channel="mqtt:topic:googlelocation_NAME:Id_number"          }
Number   GPSLocation_NAME_Latitude       "Latitude"       (Logged) {channel="mqtt:topic:googlelocation_NAME:Latitude"           }
Number   GPSLocation_NAME_Longitude      "Longitude"      (Logged) {channel="mqtt:topic:googlelocation_NAME:Longitude"          }
String   GPSLocation_NAME_Photo_URL      "Photo URL"      (Logged) {channel="mqtt:topic:googlelocation_NAME:Photo_URL"          }

googletimestamp.js

(function(timestamp){
    return timestamp.substring(0,10) + 'T' + timestamp.substring(11,23) + timestamp.substring(26,32) 
})(input)

Step by step:

  1. Install locationsharing lib and paho-mqtt on my OH server (pip3 install locationsharinglib and pip3 install paho-mqtt. Python3 and pip3 required)

  2. Put the script somewhere useful. I put mine in OpenHAB’s scripts folder, because it seems appropriate and gets included in the automatic backups, but it really doesn’t matter where it is exactly.

  3. Set up the cookies for locationsharinglib. Basically, log out of the google account you intend to use, log in again (make sure you’re on .com) and then go to maps.google.com. Once you’ve done this, export the cookies as text files (there are browser extensions that can do this) and place the cookie file in the same directory as the script. More details can be found here on how to do this.

  4. Customize the python script. Put the cookie.txt in the same directory as the script, and put the name along with your email in the appropriate slots. Then, adjust the MQTT settings to whatever you want and put in an interval (in seconds) on how often the script should send updates.

  5. Test it out! I use mosquitto, so listening to MQTT is as simple as mosquitto_sub -v -t 'googlelocation/#'

  6. Configure the script to start automatically when the computer starts. I did this via Systemd, and there are plenty of good tutorials on how to do this. Make sure to set WorkingDirectory properly. Using Systemd has lots of advantages, including automatically restarting the script if something doesnt work like the MQTT connection or similar.

  7. Configure those OpenHAB things, items, and scripts. See my examples above. The script simply converts from the format used by google to the OpenHAB acceptable DateTime format.

  8. Enjoy!

Results

So far, it’s been running without any hiccups since I started it, and the simplicity of the python script means that once configured, it shouldn’t need to be updated. New users can simply share their location with the main google account, and it should seamlessly also read their locations to MQTT.

Potential issues might crop up if the cookie used to authenticate to google expires, or if google changes the way this unofficial ‘API’ works.

7 Likes

Hello
I’m also implementing liblocationsharing, by the way in a slighty different way (using Exec binding, without MQTT).
Another benefit of using Google Maps tracking is that I’m using it through Family Link and I’m sure it cannot be disabled by my kids.
OpenHab pushes latitude and longitudes to influxDB and I can render a tracking map with Grafana and TrackMap plugin.
But I’m facing a issue : the cookies always expire after a few days.
How long can you run with your cookies ?

Well, I haven’t had a cookie expire yet but it’s only been running for four days. I did have a bit of difficulty getting a cookie that worked though: I had to try multiple browsers multiple times and repeated logging in and out until it finally worked.

I’ll update if one does expire

are you aware of the gpstracker binding ?

Well, I considered using OwnTracks or GPSLogger, but I’ve had bad experiences with these types of apps simply not working reliably in the background and only updating very rarely. Also, installing an untested extra app on everyone’s phones is a hassle when I can simply use the family Google account which already has all the shares setup and configured.

A side benefit is that this is also integrated with Google’s maps API, so the ‘address’ channel reports a human-readable street location

Here is how I get cookies, it works every time :

  • Install Export cookies extension for firefox
  • Open a private tab
  • Login to maps.google.com (and not a country specific URL)
  • Click “Export cookies” button, select “All Domains”, do not check “Prefix httpOnly cookies”
  • Import the generated file content into the cookies files

Hello
Tested several times, the cookies always expire after 3 or 4 days.

Any feedback at expiring cookies?I want to use this nice location solution but not if i must extract cokies after 3 or 4 days…

1 Like

Thanks for the script, its working well.

I wanted to display the location using a mapview item within my sitemap. Therfore, I am using a ECMAScript-2021 to generate the required pointType item. It gets triggered, when the longitude item changes (or longitude) :


var now = time.ZonedDateTime.now();

var DecimalType = org.openhab.core.library.types.DecimalType;

var PointType = org.openhab.core.library.types.PointType;

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

/// Combining Coordinates

lat = new DecimalType(items.getItem("GoogleLocationLatitude").state);

long = new DecimalType(items.getItem("GoogleLocationLongitude").state);

location = new PointType(lat, long);

/// update Location item

items.getItem("GoogleLocationLocation").postUpdate(location);

/// update timestamp

items.getItem("GoogleLocationLocation_lastupdate").postUpdate(now.toLocalDateTime().toString());

Hey there,
Thanks for the script, I modified it to work with Owntracks binding:

import json
import os
import time
from dotenv import load_dotenv
from locationsharinglib import Service, Person
from unidecode import unidecode
import requests

# Update Interval (in seconds)
update_interval = 60

load_dotenv(override=True)

service = Service(cookies_file=os.environ.get("cookies_file"), authenticating_account=os.environ.get("google_email"))


class OwntracksPerson:
    pass

    def toJSON(self):
        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=True, indent=4)


def make_tid(full_name: str):
    return unidecode("".join([initial[0] for initial in full_name.split()]))


def create_owntracks_person(person: Person):
    print(f"Person: {person}")
    owntracks_person = OwntracksPerson()
    owntracks_person._type = "location"
    owntracks_person.tid = make_tid(person.full_name)
    owntracks_person.tst = int(round(person.datetime.timestamp()))
    owntracks_person.lon = person.longitude
    owntracks_person.lat = person.latitude
    owntracks_person.batt = person.battery_level
    owntracks_person.acc = person.accuracy
    return owntracks_person


def make_request(otp: OwntracksPerson):
    r = (requests
         .post(url=os.environ.get("openhab_host"),
               auth=(os.environ.get('openhab_username'), os.environ.get('openhab_password')),
               data=otp.toJSON(),
               headers={"Content-type": "application/json"}
               ))
    print(f"Status code: {r.status_code}: {r.text}")


def publish():
    print("Starting script")
    while True:
        for person in service.get_all_people():
            otp = create_owntracks_person(person)
            make_request(otp)
        time.sleep(update_interval)
        print("Done sleeping")


def run():
    publish()


if __name__ == '__main__':
    run()

The config values are taken from .env file residing in the same folder:

google_email=mail
openhab_username=username
openhab_password=password
openhab_host=https://openhab-host/gpstracker/owntracks

And the requirements.txt:

locationsharinglib~=4.1.8
python-dotenv~=0.21.0
Unidecode~=1.3.6
requests~=2.28.1

So far, it’s running smoothly. I went from Owntracks (unreliable, especially for new Android devices with long time screen off), to Life360 (the most reliable when it worked, but with privacy concerns and some battery degradation; also with no Polish translation, so my parents couldn’t use it) and now to Google based solution.

The Things were automatically created in Openhab UI, I just connected them to existing Items.

Unfortunately, as @ cyrilpawelko said - the cookie spontaneously expires after 3-4 days making this method quite unreliable.

thanks for the script.
I use another script to call the website “maps.google.com” every 30 minutes. So far, this prevents the cookies from expiring, it is working for 6 days now, which is 2 days better than when I tried without this additional script.
I will continue to monitor this, if it’s expiring again my next idea is to rewrite the cookies file, but so far it’s working ok like this.

import requests
import pickle
import http.client 
import re
import requests
import pprint
import logging
http.client._MAXHEADERS = 1000

logging.basicConfig(format='%(asctime)s - %(message)s', filename='/logs/cookie-renew.log', level=logging.INFO)

def parseCookieFile(cookiefile):
    """Parse a cookies.txt file and return a dictionary of key value pairs
    compatible with requests."""

    cookies = {}
    with open (cookiefile, 'r') as fp:
        for line in fp:
            if not re.match(r'^\#', line):
                lineFields = line.strip().split('\t')
                if len (lineFields) >= 6:
                    cookies[lineFields[5]] = lineFields[6]
			    
    return cookies

cookies = parseCookieFile('google.com_cookies.txt')


pprint.pprint(cookies)

response = requests.get('https://maps.google.com', cookies=cookies)

# Get the cookies from the response
cookies = response.cookies
logging.info('Cookies reloaded')
1 Like

Any updates on if this has been continually working? I’ve implemented this in my setup using systemd and a RestartSec=1800s to do the visit every half hour and it seems to be working so far–but it hasn’t been four days yet.

UPDATE: As of 10. Feb 2023, has been working continuously

UPDATE 2: On the 15. Feb 2023, the system broke. I had to load in a fresh cookie. Problem is, I was messing around with my Google account settings around that time, so I may have inadvertently somehow killed the old cookie.

UPDATE 3: The system runs rather reliably, however I need to add a new cookie roughly every three months or so

So far it is working continuously for me, more than 2 weeks now.

I know that many have problems with the reliability of this approach - I use a docker container that I restart at midnight and this works for me for many months now. But there was an an update in the mqtt library used in the described solution. If you would like to setup this you’ll need to do a small update (add mqtt.CallbackAPIVersion.VERSION1):

import time
import paho.mqtt.client as mqtt

from locationsharinglib import Service
# ...
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION1, client_id)
docker-compose.yaml

image

version: '3.9'
services:
# ...
  location:
    image: my/location:1.0.0
    dns:
      - 8.8.8.8
      - 8.8.4.4
    build:
      context: ../images/location
    container_name: ${COMPOSE_PROJECT_NAME}-location
    depends_on:
      - mosquitto
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro

      - ${MOUNT}/location/secrets:/app/secrets
Dockerfile
FROM python:3

LABEL maintainer="..." \
      description="Google Location to MQTT"

RUN mkdir app
COPY /src/* /app

WORKDIR /app
RUN apt update
RUN apt install iputils-ping -y
RUN pip install --no-cache-dir -r dependencies.txt

CMD [ "python", "google-location.py" ]
dependencies.txt
locationsharinglib
paho-mqtt