Android Phone Finder from OpenHAB (Google Assistant and Alexa)

Goal:

  • Use voice command (Google Assistant and/or Alexa) to initiate Android phone finder feature
    • “Hey Google, turn on Jim’s Phone Finder”
    • “Alexa, turn on Bob’s Phone Finder”
  • Programmatically initiate phone finder feature

When you have multiple android phones, asking Google to find my phone results in it prompting about each phone one by one. It can get quite tedious. Sometimes it would just ring the first phone.

This solution lets you ring the specific phone using an openhab item.

Updates:

  • 2023-03-22 fixed find_elements to find multiple phones
  • 2023-03-21 Adapted to Google’s obfuscated class names
  • 2022-12-30 Refactor and move cookie storage into metadata
  • 2022-07-23 Google stopped showing the find my phone feature in the search result. Changed the find my phone url to google.com/android/find - the FindMyPhone#ring method is also updated accordingly

Requirements:

  • selenium standalone docker
  • JRuby scripting (it can probably be done in other language too)

Selenium Docker Setup

Using docker-compose.yml:

version: "3"
services:
  selenium:
    container_name: selenium
    image: selenium/standalone-chrome
    restart: always
    shm_size: 2gb
    ports:
      - "4444:4444"

Pull the image and start the docker:

$ docker-compose pull selenium
$ docker-compose up -d selenium

Items setup

Create openhab items to receive command and be linked to Google Assistant and Alexa

Group   gPhoneFinders

// To store google cookies
String FindMyPhone_Cookies 

Switch Find_Jims_Phone "Jim's Phone Finder" (gPhoneFinders) {phone="Jim's Phone", expire="3s", ga="Switch", alexa="Switch"}
Switch Find_Bobs_Phone "Bob's Phone Finder" (gPhoneFinders) {phone="Bob's Phone", expire="3s", ga="Switch", alexa="Switch"}

The phone metadata should contain the phone’s name as displayed on your Find my phone page

Don’t forget to tell Google to Sync my devices and Alexa to Discover devices

Obtain your google credential cookies

  • Open your browser, and go to www.google.com
  • Make sure you’re logged in to the account that can perform the “find my phone” function.
  • Enable “Developer tool” of your browser
  • Refresh the google page
  • Go to `Network" tab and find the first item. This should be the request to www.google.com
  • Find the request header and copy the cookies. It should look like “SIDCC=xxxxx; __Secure-3PSIDCC=xxxxx; blahblah=xxxxx”
  • Send the cookie as a string command to FindMyPhone_Cookies item. This can be done in Karaf console, i.e. send FindMyPhone_Cookies "xxxxxxxx"

Rules

Using JRuby:

# frozen_string_literal: true

require "json"
require "cgi/cookie"

gemfile do
  source "https://rubygems.org"
  gem "selenium-webdriver"
end

provider!(FindMyPhone_Cookies => :persistent) if defined?(:provider!)

class FindMyPhone
  GOOGLE_HOME_URL = "https://www.google.com"
  GOOGLE_FIND_URL = "https://www.google.com/android/find"

  def initialize(server = "127.0.0.1", port = 4444)
    options = Selenium::WebDriver::Options.chrome
    options.add_argument("--headless")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-gpu")

    @driver = Selenium::WebDriver.for :remote, url: "http://#{server}:#{port}/wd/hub", options: options
  end

  def close
    @driver.quit
  end

  def logged_in?
    return false unless @driver.title.include? "Google"

    login_btn = @driver.find_elements(xpath: "//a[contains(@href,'ServiceLogin')]")
    login_btn.nil? || login_btn.empty?
  end

  def cookies=(cookies)
    cookies&.each do |cookie|
      cookie = cookie.compact.except(:expires)
      @driver.manage.add_cookie(**cookie)
    end
  end

  def cookies
    @driver.manage.all_cookies
  end

  def refresh
    @driver.navigate.refresh
  end

  def home
    logger.trace "Navigating to HOME"
    @driver.navigate.to(GOOGLE_HOME_URL)
  end

  def refresh_cookies
    home
    load_cookies
    refresh
    return save_cookies if logged_in?.tap { |o| logger.trace "Logged in 2: #{o}" }

    logger.warn "Failed to login to Google Find My Phone. Update your cookies!"
  end

  def ring(phone)
    @driver.navigate.to(GOOGLE_FIND_URL)

    # create a hash of phone_name => id
    # This is the list of phone as icons on the top left corner
    xpath = %{//div[contains(@class, "NAEdUe")]//div[@role="button"]}
    devices = @driver.find_elements(xpath: xpath)
                     .to_h { |elem| [elem.find_element(tag_name: "img")["aria-label"], elem] }

    device = devices[phone]
    device.click
    # This is the "PLAY SOUND" button element on the left
    ring_button = @driver.find_element(xpath: %(.//button[contains(@class, "IMd82d")]))
    ring_button.click
    sleep 5
  end
end

def save_cookies(cookies)
  FindMyPhone_Cookies.metadata["cookies"] = cookies.to_json
end

def saved_cookies
  raise "Cookies not set" unless FindMyPhone_Cookies.metadata["cookies"]

  JSON.parse(FindMyPhone_Cookies.metadata["cookies"].value).map { |cookie| cookie.transform_keys(&:to_sym) }
end

def refresh_cookies(current_cookies)
  fmp = FindMyPhone.new
  fmp.home
  fmp.cookies = current_cookies
  fmp.refresh
  save_cookies(fmp.cookies)
  logger.info "Find my phone cookies refreshed"
ensure
  fmp.close
end

rule "Find my phone: Set new cookies" do
  received_command FindMyPhone_Cookies
  run do |event|
    cookies = CGI::Cookie.parse(event.command.to_s)
                         .values
                         .map do |cookie|
      { name: cookie.name, value: cookie.value.map { |v| CGI.escape(v) }.join("&"),
        path: cookie.path, domain: cookie.domain, expires: cookie.expires,
        secure: cookie.secure, http_only: cookie.httponly }.compact
    end
    refresh_cookies(cookies)
  end
end

rule "Find my phone: Refresh cookies" do
  every :day
  run { refresh_cookies(saved_cookies) }
end

rule "Find my phone: Ring" do
  received_command gPhoneFinders.members, command: ON

  triggered do |item|
    phone = item.metadata["phone"].value
    logger.info "#{item}: #{phone}..."
    fmp = FindMyPhone.new
    begin
      fmp.home
      fmp.cookies = saved_cookies
      fmp.ring phone
      save_cookies(fmp.cookies)
    rescue
      say "There's something wrong with #{item.label}"
      raise
    ensure
      fmp.close
    end
  end
end

Note:

4 Likes

Google stopped showing the find my phone feature in the search result. Changed the find my phone url to google.com/android/find

Nice work. Unfortunately I was not able to install selenium standalone docker on openHABian and I dont like JRuby. So I re-implemented the code using Node and used ChromeDriver directly.
How to use?

  • Install Node.js,
  • clone https://github.com/Plustig/Android-Phone-Finder/
  • install app using ‘npm install’ in project’s root directory
  • Install ChromeDriver using ‘sudo apt-get install chromium-chromedriver’
  • copy path of chromedriver binary in constructor method of app.js

Support for multiple phones is missing. I invoked the node app like this

/*
This rule scrapes the google's 'find my phone' page to let the phone ring
*/

var Exec = Java.type("org.openhab.core.model.script.actions.Exec");


rules.JSRule({
    name: "Ring My Phone",
    description: "This rule scrapes the google 'find my phone' page to let the phone ring",
    triggers: [triggers.ChannelEventTrigger("deconz:switch:homeserver:tdfr_shortcut_findphone:buttonevent", "1002")],
    execute: data => {
        console.log("Let your phone ring...");
        Exec.executeCommandLine( 'node', '/home/openhabian/selenium-test-app/app.js', 'ring');
    }
});
  
rules.JSRule({
  name: "Refresh google cookies",
  description: "This rule reloads the google cookies needed for authentification.",
  triggers: [triggers.GenericCronTrigger("0 0 */3 ? * *")],
  execute: data => {
      console.log("Refresh google cookies...");
      Exec.executeCommandLine( 'node', '/home/openhabian/selenium-test-app/app.js', 'refresh');
  }
});

Hi guys,

I’m trying to use your solution as well (the JS one in a dockerized environment). I’ve successfully deployed the selenium container and are able to debug the JS code. All seems fine, except the fact, that I’m not able to sign in to my google account. I’ve a feeling that the cookie string is not good I’m sending. My default cookie value looks like this:

‘1P_JAR=VALUE_COMES_HERE;OTZ=;SIDCC=VALUE_COMES_HERE;__Secure-1PSIDCC=VALUE_COMES_HERE;__Secure-3PSIDCC=VALUE_COMES_HERE;’

Is this correct? Or am I doing something wrong? Is there any way to debug, what selenium is doing?

Thanks!

UPDATE: I managed to set the correct cookies using the method from the original repo, so the issue was caused by the incorrect or insufficient cookies.

Blockquote
Open a private tab in your browser, login to https://google.com and export the cookies with EditThisCookie

Update:

  • Previously the cookies didn’t get refreshed/saved correctly so the original cookies were used and they’ll eventually expire
  • Cookies are now set using an openhab command instead of saving it in the script.
  • Store updated cookies in metadata.

Ughhhh!!! Google has now obfuscated their phone finder page.
I wish there’s an official API

I finally spent some time looking into this and adjusted the script to match Google’s new obfuscated class ids. It’s working again now!