Guide to Network UPS Tools (NUT) on multiple Raspberry PI’s (Openhab 2.5 and other systems)

This tutorial will cover the installation of NUT on a Pi running Openhab (OH) 2.5 with other systems powered by the same UPS.

First off I would like to thank everyone in the OH community who has written tutorials, commented, asked and answered questions. You have all been a great help. This is my first tutorial/lesson learned; your comments, suggestions and critique are welcome.

This all started after I had bought a UPS for my systems, most of which are running on Raspberry Pi’s and then did not set up anything to monitor the UPS.

We live in a remote area that has power outages. We have a generator that starts automatically and provides power whenever required. The generator and power to the house are monitored. We know if it doesn’t start and if the house is without power. The UPS is only necessary to carry its connected systems through the short interval between the utility power outage and when the generator picks up the load. Why monitor the UPS? Wrong assumption on my part! I have learned.

A while ago we had a utility power outage. The generator started, the house had power, but the UPS kept supplying its loads from battery until the batteries were depleted and the connected systems lost power very unceremoniously. Since OH uses zram (nothing wrong with that by the way) all logged and persistence data was lost up to the last time I had restarted the system many months ago. I would like to acknowledge the reliability of OH, again thanks developers and community!

Solution: Monitor UPS and have the system react appropriately
I found Network UPS Tools (NUT), rpwongs great guide (see below) and lots of other useful information.
https://community.openhab.org/t/beginners-guide-to-network-ups-tools-nut-on-a-raspberry-pi/78443

As I delved deeper I found that I had to customize the solution for my systems which led me down a number of rabbit holes, lots of work, testing and many failures, hence this (hopefully) useful guide.
The guide deals mainly with the installation of NUT. It explains why specific modification were required to successfully integrate it with our OH systems. We have a number of OH systems powered by the UPS, one that has control, i.e. the one that runs the house and we connect to with our phones, one for code development and testing of ideas, one running OH3 to which we will transition.

Executive summary of this solution (so you don’t have to read through the whole document)

  1. One NUT Master and a number of NUT Slaves powered by a single UPS
  2. The OH Control system on this Master
  3. The Master monitors UPS, communication to UPS via USB and displays results in OH
  4. The Network UPS Monitor (NUT) Binding is used for UPS parameters
  5. The Master to UPS communications is also displayed in OH (NUT binding does not have this feature, yet?)
  6. Systems reboot after a set time on battery to save logs; either because they use zram, log2ram or some other software to protect the Pi sd card
  7. Slaves (other UPS powered systems) may shutdown early to extend UPS battery life

I read Mel Grubb’s guide to setting up NUT. It provides a lot of useful information, well worth reading. As such I will try to not repeat what Mel has already explained. https://melgrubb.com/2016/12/11/rphs-v2-ups/

Installing NUT

sudo apt update
sudo apt install nut-client nut-server

After reading many tutorials on NUT installations I found it easier to install both the nut client and server on all systems. I do not differentiate between Master and Slave for the software installation. The Master is the system that is connected directly to the UPS and communicates with it (in my case via USB) and will be the last to shut down. Slaves are all other systems, they may or may not shutdown early. If you only have a single system, a Master, then ignore the Slave settings.

Configuring NUT

Even though some of the files below are only required on the NUT Master system. I chose to have all files on all systems, regardless if Master or Slave. My reasons were that:

  1. It would allow me to easily switch Masters if required
  2. I only need to maintain a single copy of all the files
  3. Settings within the files are used to customize them for the Master or Slave

Change to nut configuration file directory and editing NUT files

cd /etc/nut

NUT Configuration

sudo nano nut.conf
On Master
MODE=netserver
On Slave
MODE=netclient

The above is pretty well self-explanatory. If this system is the Master it’s also the server for UPS information, all other systems are Slaves and therefore clients.

NUT UPS Configuration

sudo nano ups.conf
maxstartdelay = 45
maxretry = 5
retrydelay = 5
[ec850lcd]
    driver = usbhid-ups
    port = auto
    desc = "CyberPower EC850LCD"
    pollinterval = 15
    offdelay = 600
    ondelay = 660
    ignorelb
    override.battery.charge.low = 30
    override.battery.charge.warning = 35
    override.battery.runtime.low = 1000

The important parameters:
[ec850lcd] - UPS name, change this to whatever you would like for your UPS. Remember to make the same change in all the files below.
pollinterval - Because I have a Cyberpower UPS. See https://nmaggioni.xyz/2017/03/14/NUT-CyberPower-UPS/ . This worked for me. This is also the reason why it was important for me to see Master to UPS communications in OH.
offdelay - OH 2.5 on my Pi 3B+ takes 6:15 minutes to shut down. This sets the UPS to shut down in 10 minutes after receiving a shutdown command from the Master; sufficient time for the Master to shutdown and then the UPS (default was 20 seconds)
ondelay - Per NUT documentation this should be longer than the offdelay (default is 30 seconds)
ignorelb and the overrides - This sets the charge.low, charge.warning and runtime.low values to be sufficient for the Master and other systems to shutdown. The default values in the UPS are too low and would shut down the UPS before the would Pi saved logs and shut down.

On the Master system you can now check if the driver and nut-server are configured correctly.

sudo upsdrvctl start
sudo service nut-server restart

If you get any error messages you can stop and start each of the above or reboot the system.

To check if your Master can communicate with the UPS.

upsc ec850lcd@[Master IP address]

NUT Server TCP Connections

sudo nano upsd.conf
# upsd.conf - Configuration for Network UPS Tools

# Which interfaces listen to TCP connections from clients
# 0.0.0.0. means all IP addresses on our network can connnect
# port 3493 is default port
# This will only be read at startup of upsd.  If you make changes here,
# you'll need to restart upsd, reload has no effect.

LISTEN 0.0.0.0 3493

These settings lets all NUT clients (Slaves) communicate with the server (Master).

NUT Users

sudo nano upsd.users
# upsd.users - User definitions for NUT upsd

# [user] = user
#    password = password for that user
#    action = set allow changing values of certain UPS variable
#    action = fsd allow forced shutdown setting
#    instcmd = instant commands, use upscmd -l for a full list of what UPS supports
#    upsmon either master or slave

[admin]
    password = admin
    actions = set
    actions = fsd
    instcmds = all

[monmaster]
    password = monmaster
    upsmon master

[monslave]
    password = monslave
    upsmon slave

Mel Grubb’s tutorial provides a good explanation on the upsd.users file. I chose to simplify these to an admin, master and slave user.

NUT Client Configuration File

sudo nano upsmon.conf
#upsmon.conf - Configuration for network UPS Tools upsmon

# Run user nut
RUN_AS_USER nut
# List of UPS devices to monitor
# Enable either master of slave
# Master
#MONITOR ec850lcd@[ip address of master] 1 monmaster monmaster master
# Slave
#MONITOR ec850lcd@[ip address of master] 1 monslave monslave slave

# Number of power supplies receiving power to keep this system running
MINSUPPLIES 1
# Command to run when shutting down this system, ensure path is correct to shutdown
SHUTDOWNCMD "/sbin/shutdown -h now"
# Command to run on +EXEC events below, if using upssched then ensure path is correct
NOTIFYCMD /sbin/upssched
# Poll the UPS every 5 seconds.
POLLFREQ 5
# If the UPS is on battery poll this often
POLLFREQALERT 5
# Wait this many seconds for "Notify Shutdown" messages from slaves
HOSTSYNC 15
# time UPS may go missing before being declared dead
# see https://nmaggioni.xyz/2017/03/14/NUT-CyberPower-UPS/
# for DEADTIME setting
# default = 15
DEADTIME 25
# Location of the flag-file to make UPS turn itself off.
POWERDOWNFLAG /etc/killpower
# Messages for the events %s will be replaced with the UPS system identifier in question
NOTIFYMSG ONLINE "UPS %s on line power"
NOTIFYMSG ONBATT "UPS %s on battery"
NOTIFYMSG LOWBATT "UPS %s battery is low"
NOTIFYMSG FSD "UPS %s: forced shutdown in progress"
NOTIFYMSG COMMOK "Communications with UPS %s established"
NOTIFYMSG COMMBAD "Communications with UPS %s lost"
NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
NOTIFYMSG REPLBATT "UPS %s battery needs to be replaced"
NOTIFYMSG NOCOMM "UPS %s is unavailable"
NOTIFYMSG NOPARENT "upsmon parent process died - shutdown impossible"
# Change behavior of upsmon on certain events.
# Possible values for the flags:
# SYSLOG - Write the message in the syslog
# WALL - Write the message to all users on the system
# EXEC - Execute NOTIFYCMD (see above, in our case see upssched.conf) with the message
# IGNORE - Don't do anything
# If you use IGNORE, don't use any other flags on the same line.
# NOTIFYFLAG <notify type> <flag>[+<flag>][+<flag>] ...
NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC
NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
NOTIFYFLAG SHUTDOWN SYSLOG+WALL+EXEC
NOTIFYFLAG REPLBATT SYSLOG+WALL
NOTIFYFLAG NOCOMM   SYSLOG+WALL+EXEC
NOTIFYFLAG NOPARENT SYSLOG+WALL
# Warn every 12 hours if battery needs to be replaced
RBWARNTIME 43200
# Warn every 5 minutes, if UPS is unreachable
NOCOMMWARNTIME 300
# Wait 5 seconds before starting to shut down
FINALDELAY 5

There’s lots pf settings above. The important learnings for my systems:

RUN_AS_USER nut

The user and group nut are created when NUT is installed. Some tutorials suggest setting this to “RUN_AS_USER root”. It may be easier but also enable some pitfalls. I chose to utilize the group and user created by NUT. In the section “Creating upssched Directory and Adjusting Nut Rights and Privileges” below we adjust the ownership and rights for the nut user.

# Enable either master of slave
# Master
#MONITOR ec850lcd@[Master IP address] 1 monmaster monmaster master
# Slave
#MONITOR ec850lcd@[Master IP address] 1 monslave monslave slave

Enter the ip address of the Master where shown and remove the # in front of the MONITOR keyword if this file is on a Master or Slave system; and change the ups name if yours if different.

NOTIFYCMD /sbin/upssched

I use NUT’s upssched to process the NOTIFYFLAG’s EXEC which are set in the configuration file above. I noticed while testing NUT that in some cases upssched installed in usr/sbin/ ; ensure that you have the correct path to upssched in the statement above.

NUT upssched Configuration File

sudo nano upssched.conf
# Network UPS Tools - upssched.conf file

#Command script to run
CMDSCRIPT /etc/nut/upssched-cmd

# command for pipe and lock files
PIPEFN /etc/nut/upssched/upssched.pipe
LOCKFN /etc/nut/upssched/upssched.lock

# Execution of flags on same order as upsmon.conf file
# UPS is online = supplied from utility power
# Cancel earlyshutdown timer, beeperenable timer and re-disable beeper
AT ONLINE * EXECUTE online
AT ONLINE * EXECUTE beeperdisable
AT ONLINE * CANCEL-TIMER earlyshutdown
AT ONLINE * CANCEL-TIMER beeperenable
AT ONLINE * CANCEL-TIMER reboot

# UPS on battery
# Execute onbatt code
# Start earlyshutdown timer (see code, to shut down devices that do not
# need to be running on UPS) earlyshutdown timer to be longer that generator
# start delay plus multiple crank cycles
AT ONBATT * EXECUTE onbatt
AT ONBATT * START-TIMER earlyshutdown 600
# enable beeper if ups has been on for 1 hour
AT ONBATT * START-TIMER beeperenable 3600
# enable reboot of pi (eg. to save zram data) if ups has been on for 1 hour
AT ONBATT * START-TIMER reboot 3600

# UPS low battery
AT LOWBATT * EXECUTE lowbatt
# UPS fsd = forced shutdown
AT FSD * EXECUTE fsd
# UPS commok
AT COMMOK * EXECUTE commok
AT COMMOK * CANCEL-TIMER commbad
# UPS commbad
AT COMMBAD * START-TIMER commbad 30
# UPS shutdown
AT SHUTDOWN * EXECUTE shutdown
# UPS replbatt
AT REPLBATT * EXECUTE replbatt
# UPS nocomm
AT NOCOMM * START-TIMER commbad 30
# UPS noparent
AT NOPARENT * EXECUTE noparent

CMDSCRIPT /etc/nut/upssched-cmd
Script that we execute, see next section, ensure directory path and file name are correct.

PIPEFN /etc/nut/upssched/upssched.pipe
LOCKFN /etc/nut/upssched/upssched.lock
Pipe and Lock files required by NUT. The directory for these files and its rights are created below.

AT [commands]
The AT commands execute a section of the script in the upssched-cmd file whenever NUT has detected a new event, e.g. the AT ONLINE lines execute whenever NUT detects that the UPS has switch
to Online mode.
In those cases where there’s a START-TIMER or CANCEL-TIMER we either start a timer that delays the execute of a script for the set number of seconds or we cancel a timer that is running.

The AT EXECUTE ONBATT section is worthy of further explanation
AT ONBATT * START-TIMER earlyshutdown 600
Starts a timer that is used to shutdown systems that are non-critical to conserve UPS battery capacity if the UPS has been on battery for more than 10 minutes
AT ONBATT * START-TIMER beeperenable 3600
Starts a timer that enables the UPS beeper if the UPS has been on battery for more than 1 hour
AT ONBATT * START-TIMER reboot 3600
Starts a timer that reboots the system if the UPS has been on battery for more than 1 hour. This is primarily to ensure that logs are saved to the sd card.

NUT upssched Script File

sudo nano upssched-cmd
#!/bin/sh

# Is system master = directly communicating with a ups, then true 
# or slave = not directly communicating with a ups, then false
MASTER="true"

# SYSTEMTYPE defines how this system reacts to power outages
# primary = 1; systems that stay on until lowbattery event, master is always type 1, slaves if important
# secondary = 2; systems that shutdown at earlyshutdown (after ups has been on battery for some time)
SYSTEMTYPE=1

# Values only required for masters
UPS_USERNAME="admin"
UPS_PASSWORD="admin"
UPS_LINK="ec850lcd@[Master IP address] "

# If system reboots during onbattery create this file
REBOOTED="/etc/nut/upssched/rebooted"
# If communication to ups is down create this file
COMMBAD="/etc/nut/upssched/commbad"


case $1 in
   online)
      # if previously rebooted, then delete file
      logger -t upssched-cmd "NUT - Online"
      if [ -f $REBOOTED ]; then
         rm $REBOOTED
      fi
   ;;
   onbatt)
      logger -t upssched-cmd "UPS running on battery"
   ;;
   earlyshutdown)
      logger -t upssched-cmd "NUT - Early shutdown time-out reached"
      # if secondary system then shutdown now
      if [ $SYSTEMTYPE -eq 2 ]; then
         logger -t upssched-cmd "NUT - UPS on battery too long, early shutdown on this system"
         # on secondary systems shutdown now sudo /sbin/shutdown -h now
         # ensure path is correct to upsmon
         /sbin/upsmon -c fsd
      fi
   ;;
   lowbatt)
      # shutdown all systems now
      logger -t upssched-cmd "NUT - UPS on battery critical, forced shutdown"
      # enable fsd on all system - if we missed the earlyshutdown, take it down
      # ensure path is correct to upsmon
      /sbin/upsmon -c fsd
   ;;
   fsd)
      logger -t upssched-cmd "NUT - UPS forced shutdown started"
   ;;
   commok)
      logger -t upssched-cmd "NUT - UPS communications OK"
      # if file was created then delete file
      if [ -f $COMMBAD ]; then
         rm $COMMBAD
      fi
   ;;
   commbad)
      logger -t upssched-cmd "NUT - UPS communications has been offline too long"
     # check if file has already been created
     if [ -f $COMMBAD ]; then
        logger -t upssched-cmd "NUT - System has already created COMMBAD file"
     # if not create file for openhab
     else
        logger -t upssched-cmd "NUT - Creating COMMBAD file"
        touch $COMMBAD
     fi
   ;;
   shutdown)
      logger -t upssched-cmd "NUT - System shutdown started"
   ;;
   beeperenable)
      # if master and primary system then enable beeper on ups
      if [ $MASTER -eq "true" ]; then
         if [ $SYSTEMTYPE -eq 1 ]; then
            logger -t upssched-cmd "NUT - Enabling UPS beeper"
            upscmd -u ${UPS_USERNAME} -p ${UPS_PASSWORD} ${UPS_LINK} beeper.enable
         fi
      fi
   ;;
   beeperdisable)
      # if master and primary system then enable beeper on ups
      if [ $MASTER -eq "true" ]; then
         if [ $SYSTEMTYPE -eq 1 ]; then
            logger -t upssched-cmd "NUT - Disabling UPS beeper"
            upscmd -u ${UPS_USERNAME} -p ${UPS_PASSWORD} ${UPS_LINK} beeper.disable
         fi
      fi
   ;;
   reboot)
     # do not reboot if system has already reboot
     if [ -f $REBOOTED ]; then
        logger -t upssched-cmd "NUT - System has already been rebooted while on battery"
     # else reboot system
     else
        logger -t upssched-cmd "NUT - Rebooting system"
        touch $REBOOTED
        # ensure path is correct to shutdown
        sudo /sbin/shutdown -r now
     fi
   ;;
   *)
      logger -t upssched-cmd "No code defined for command: $1"
   ;;
esac

I am at the earlier stages of learning shell script so I’m sure there are better ways to accomplish some of these tasks, but here we go with the explanations.

MASTER=“true”
Is set “true” or “false” is this or is not the Master. This is used to determine if we can control the UPS’s beeper. TODO read upsmon.conf file and check which MONITOR line is enabled.

SYSTEMTYPE=1
Per the comments in the file above this variable determines if this system is shutdown early or not.

REBOOTED="/etc/nut/upssched/rebooted"
COMMBAD="/etc/nut/upssched/commbad"
REBOOTED and COMMBAD are files that are created if the system was rebooted while the UPS was on battery or if communications to the UPS has been lost.
If REBOOTED is present this system has already been rebooted while on battery.
If COMMBAD is present is indicates that the Master is no longer communicating with the UPS.
Both files are deleted when the relevant conditions have been resolved.

UPS_LINK="ec850lcd@[Master IP address] "
Change this to your ups name and IP address

Creating upssched Directory and Adjusting Nut Rights and Privileges

sudo mkdir upssched
sudo chown nut:nut *
sudo chmod a+x upssched-cmd 
sudo chmod a+rw upssched

The above creates the upssched subdirectory in /etc/nut, changes ownership of all files and subdirectories to the nut group and nut user, allows execution of the upssched-cmd file and read/write access to the upssched subdirectory.

Allow nut User to Shut down and Reboot System without Sudo Password

cd /etc/sudoers.d
sudo visudo 020_nut
# 020_nut
# allows user nut to shutdown and reboot system without entering password
nut ALL=(ALL) NOPASSWD: /sbin/shutdown, /sbin/reboot

Very important to use visudo, it will check the syntax before saving the file (ignore the fact that it says it’s saving it with a .tmp extension) and will not let you create a file with syntax errors. Syntax errors here are very, very bad, will seriously mess up your system and are very difficult to resolve! I know, I have done it!
The code above allows the user nut to run shutdown and reboot, in our case in the upssched-cmd script, without having to enter the sudo password. The file name was chose conveniently so this file is sorted after the existing files.

Modify Nutshutdown Script

cd /lib/system/system-shutdown
#!/bin/sh
# initial single line in file, sometimes did not shutdown rpi
# changed to script below with sleep 2 added 
# /sbin/upsmon -K >/dev/null 2>&1 && /sbin/upsdrvctl shutdown

# added from https://github.com/networkupstools/nut/blob/master/scripts/systemd/nutshutdown.in and modified
# This script requires both nut-server (drivers)
# and nut-client (upsmon) to be present locally
# and on mounted filesystems
[ -x "/sbin/upsmon" ] && [ -x "/sbin/upsdrvctl" ] || exit

if /sbin/upsmon -K >/dev/null 2>&1; then
  /sbin/upsdrvctl shutdown
  /bin/sleep 2
fi

exit 0

I found that my UPS would not reliably shutdown every time NUT issued the force shutdown command (fsd). Perhaps a timing issue between upsdrvctl issuing a shutdown command to the UPS and the Pi itself shutting down as well. Only after modified the file above and adding the /bin/sleep 2 did the UPS shutdown work reliably. Suggestions for a better approach are welcome.

Let’s Check if Everything Works

At this point I restart NUT on the Master with the following commands

sudo service nut-server restart
sudo service nut-client restart
sudo upsdrvctl stopt
sudo upsdrvctl start

On the slave

sudo service nut-client restart

Check communications at both Master and Slaves with the following:

upsc ec850lcd@[Master IP address]

Init SSL without certificate database
battery.charge: 100
battery.charge.low: 30
battery.charge.warning: 35
battery.mfr.date: CPS
battery.runtime: 14650
battery.runtime.low: 1000
battery.type: PbAcid
battery.voltage: 14.3
battery.voltage.nominal: 12
device.mfr: CPS
device.model: EC850LCD
device.type: ups
driver.flag.ignorelb: enabled
driver.name: usbhid-ups
driver.parameter.offdelay: 600
driver.parameter.ondelay: 660
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 15
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: CyberPower HID 0.4
driver.version.internal: 0.41
input.transfer.high: 142
input.transfer.low: 96
input.voltage: 117.0
input.voltage.nominal: 120
output.voltage: 118.0
ups.beeper.status: disabled
ups.delay.shutdown: 600
ups.delay.start: 660
ups.load: 0
ups.mfr: CPS
ups.model: EC850LCD
ups.productid: 0501
ups.realpower.nominal: 450
ups.status: OL
ups.test.result: No test initiated
ups.timer.shutdown: -60
ups.timer.start: 0
ups.vendorid: 0764

OH Setup for UPS Monitoring

I then followed rpwongs guide from Section 3. Connect OH to NUT to create things and items in OH.

OH Master to UPS Communications Alarm

One additional feature I wanted in OH was to see the state of the Master to UPS Communication. The COMMBAD file created by the upssched-cmd script above provides that information.

commbad.sh in the OH scripts folder

#!/bin/bash

# commbad.sh - checks for presence of commbad file generated by Nut ups monitor
# presence of file indicates that communications to ups is bad
COMMBAD="/etc/nut/upssched/commbad"
if [ -f $COMMBAD ]; then
    # if file exists return 0
    retval="0"
else
    # if file does not exist return 1
    retval="1"
fi
echo "$retval"

In the OH misc folder add to exec.whitelist file

# added for ups commbad check
/bin/bash /etc/openhab2/scripts/commbad.sh

To ups.things (you may need to install Exec Binding if you have not installed it)

Thing exec:command:commbad "UPS Commbad" [command="/bin/bash /etc/openhab2/scripts/commbad.sh", interval=30,timeout=5]

To ups.items

//cpu commbad
String  vUPS_CommbadRaw "UPS Communications" {channel="exec:command:commbad:output"}
String  vUPS_Commbad "UPS Communications [%s]"

To ups.rules

// ups.rule
// checks for presence of commbad file
// file indicates that communications to ups are bad
rule "rCheckCommbad"
when
    Item vUPS_CommbadRaw changed
then
    if (vUPS_CommbadRaw.state == "1") {
        vUPS_Commbad.postUpdate("Commok")
    }
    if (vUPS_CommbadRaw.state == "0") {
        vUPS_Commbad.postUpdate("Commbad")
    }
end

The item vUPS_Commbad is now used to show the state of the Master to UPS communications. This helped me to troubleshoot the Master to UPS communications.

I knowledge this may be a lengthy tutorial / lessons learned. It did take quite some time to resolve the various nuances of NUT, my UPS and the Pi’s that were running OH (production and development systems) as well as other systems connected to the UPS. Hopefully this provides others with a template to install NUT and monitor a UPS for their OH and other systems.

6 Likes

Thanks for writing up.
Would you evetually also be willing to contribute to openHABian with a NUT installer code piece?

Hi Markus, it’s difficult to say yes since I’m unsure what a installer code piece entails and, as I noted in my write up, I am at the early stages of learning shell script. My coding skills were learned in the previous millennium, with programming languages that are no longer in popular demand :grimacing:. With those caveats, I would certainly be willing to help and contribute where possible.

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.