Amanda howto for openhabian and NAS

backup
openhabian
amanda
Tags: #<Tag:0x00007f2135579678> #<Tag:0x00007f2135579538> #<Tag:0x00007f21355793a8>

(Me) #61

Reporting back as promised.

I was able to restore a file from one pi to another one with amrestore without using the database :slight_smile:

Setup:
2 identical x86Raspberian with openhabian in a VirtualBox setup.

  1. Set up amanda on one instance and backup some directories
  2. Install amanda on the second instance, pointing to the exact same directory for creating the virtual tapes -> I expected it to overwrite the existing backups there, but surprisingly it didn’t
  3. Locate the Slot you want to restore from (This has to be done manually. Best Option to find out which one to use is to check the timestamps of the backups which can also be done with amrestore once you have the vtape in the device)
  4. Put the vtape (slot) into the device with amtape
  5. Restore the files with amrestore

After this procedure I can happily open and use the file on the second instance of the pi. So it’s possible to restore the files without having the database at hand in the very worst case. I’ll write a more comprehensive how-to later on, also including an example with backuped amanda directories … so maybe we can add this to documentation or put it somewhere else if someone is in desperate need :slight_smile:.

What is most important for me is the conclusion, that there is always a way to restore the backups, no matter what is broken :grin:


(Niklas) #62

Will I be able to restore a dump to a somewhat smaller disk?

I had this problem with an earlier backup strategy where my two 8 Gb SD-cards weren’t exactly the same size. So when trying to restore the backup it wouldn’t fit. I solved this by shrinking the partition somewhat before doing the backup.

How would I solve a similar “problem” with an Amanda dump?

Anyway I tried to do an amfetchdump to see what I would end up with, this is my output:

[08:37:16] backup@openHABianPi:~$ amfetchdump -p openhab-dir openhabianpi /dev/mmcblk0 > /home/openhabian/backupsDS/openhabianpi-image-180112
WARNING: Fetch first dump only because of -p argument
1 volume(s) needed for restoration
The following volumes are needed: openHABian-openhab-dir-008
Press enter when ready

ERROR: Volume '' not found        

I also had to change the working directory from “openhab” to “openhab-dir”. The docs text uses both directory’s making it a bit confusing.
What am I doing wrong? Also, I want to restore the latest dump and not the “first”. Is there an option for this?


(Branden Smale) #63

how does one go about mounting a flash drive, for backup with a usb ssd running openhabian?


(Markus Storm) #64

I don’t understand your question.


(Branden Smale) #65

i have 2 usb attached storage. one is a ssd that im running openhabian on, the other is a flash drive for backups only.

how do i prepare for backup the flash drive? instructions say to remove all usb attached storage solutions, then plug in the one used for backup, but i cannot remove the ssd, as the system wont run without it.


(Markus Storm) #66

I still don’t understand your question.
“The one used for backup” ("= the one to backup TO") would be your flash drive not your SSD, and “the one TO backup” would be your SSD.
Instructions DON’T say to remove all USB attached storage, dunno where you got that from.
Or did you mean to ask how to use (a set of) removable media (such as several USB sticks or SD cards) as your backup destination ?
Normally, you need an accessible destination directory by the time Amanda is being installed, so you permanently need to have your backup storage attached and have it mounted as a filesystem, that’s where your backups go to. This could be your flash drive, so you should not remove it. And it has to provide enough space for the cumulated backed-up data. Size varies and depends on some parameters, but usually this means at least 2-3 times the size of your medium you want to backup (which is your SSD).
BTW does not sound as if you read the Amanda README ? It’s all explained in there.


(Markus Storm) #67

If you backup/restore “raw” data, the backup system cannot know what’s inside so it cannot know what to omit when restoring. Yes, one strategy to work around that is to shrink the active partition before backing up. The other one is to restore to a file or larger device and copy the file contents. Another possible solution is to have an additional empty partition at the end (one that does not contain any data you need so it does not do harm when it’s incompletely restored).
But all of this is just tricks to work around a design fault: to use SD cards of different size is a particularly bad idea.

On your error message, can’t tell from the outside what’s wrong without looking at the config and logs, eventually.
Guess Amanda does not find the dump file in the openHABian-openhab-dir-008/ directory where it expects to find it (because that’s where it recorded it was stored to when you ran the backup). Maybe your Amanda database is broken or you changed mountpoints or directory names or access rights or whatever since you ran the backup. Eventually the amadmin command helps in finding your dump file.
Short of that, you need to dig the logs to find out. Find them in /var/log/amanda/*.
For further support, I suggest to check the resources mentioned in the README such as the Amanda FAQ.

Omit -p if you want to select a different dump. See the manpage (or type man amfetchdump in your shell) for this and more restore related options.


(Niklas) #68

Thank you Markus for your answers and advice!
I have three 8 Gb SD-cards, luckily the one in operation is the smallest one with the fewest number of bytes and sectors. Anyway, shrinking a partition is not a difficult task.

As I was troubleshooting my unsuccessful “fetchdump” I discovered that it is really troublesome to export log files from the Amanda log directory. I actually didn’t manage to do this and was forced to copy-paste the parts of interest. Amanda obviously adds quite strict permission attributes to all files created by the backup user.

I anyhow solved my underlying problem. With the given command Amanda was trying to fetch the first dump made. But as the “tapes” are continuously overwritten the first dump was now incomplete. This situation must be quite common right? I would need tape sizes larger than my raw dump to avoid this, right?

[19:41:06] backup@openHABianPi:~$ amadmin openhab-dir find openhabianpi /dev/mmcblk0

date                host         disk         lv tape or file               file part status
2018-01-08 01:00:03 openHABianPi /dev/mmcblk0  0 openHABian-openhab-dir-014    1  2/2 OK FAIL Missing part
2018-01-09 01:00:02 openHABianPi /dev/mmcblk0  0 openHABian-openhab-dir-015    3  1/2 OK
/---/ (continuing for all tapes)
2018-01-15 01:00:03 openHABianPi /dev/mmcblk0  0 openHABian-openhab-dir-012    3  1/2 OK
2018-01-15 01:00:03 openHABianPi /dev/mmcblk0  0 openHABian-openhab-dir-013    1  2/2 OK

By fetching a later dump (the most recent from 20180115) the operation was successful.

[19:19:10] backup@openHABianPi:~$ amfetchdump -p openhab-dir openhabianpi /dev/mmcblk0 20180115 > /home/openhabian/backupsDS/openhabianpi-image-20180115
2 volume(s) needed for restoration
The following volumes are needed: openHABian-openhab-dir-012 openHABian-openhab-dir-013
Press enter when ready

amfetchdump: 3: restoring split dumpfile: date 20180115010003 host openHABianPi disk /dev/mmcblk0 part 1/UNKNOWN lev 0 comp N program APPLICATION

1602176 kb
amfetchdump: 1: restoring split dumpfile: date 20180115010003 host openHABianPi disk /dev/mmcblk0 part 2/UNKNOWN lev 0 comp .gz program APPLICATION
2980172 kb
[19:41:24] backup@openHABianPi:~$

Regarding the -p option. The manual says the following:
“Unless -p is used, backup images are extracted to files in the current directory named:”

"-p: Pipe exactly one complete dump file to stdout, instead of writing the file to disk. This will restore only the first matching dumpfile (where “first” is determined by the dump log search facility).

The option -p seems to be more important to be able to generate a restoration file rather than a restoration. The warning message makes it quite confusing: “WARNING: Fetch first dump only because of -p argument”

So I’m quite pleased with the setup now. I will add the persistence folder to the backup list and I will also try to adjust the frequency of the raw backup (complete dump) to perhaps once a week.

Thank you for your hard work!


(Markus Storm) #69

Yeah, that’s because of its origins in datacenter sphere, explained in the README.
But you can simply do sudo bash and work as root to access the logs.

Well, if dumpsize>tapesize then Amanda splits a dump across several tapes (even virtual tapes) so on restoring that one you need to insert one tape after another. That’s why amfetchdump prompts you to insert the tape. Same still if you use vtapes. So your dump on tape 1 is incomplete in that case but it is still restoreable unless you waited so long that one of the parts 2 to X (all on a different (v)tapes) got overwritten.
EDIT: Possibly what is meant by the warning message you get when you use -p is that it’ll only restore one part (i.e. the first one). If amfetchdump did print ‘insert 2nd tape now’, this text output would be piped to the following unix command as well, right inbetween restore data part 1 and 2. That would mess up the stream of backup data.

If you use vtapes (which I assume you do as that’s the default setup), tapesize is artificially restricted, but unlike with physical tapes of defined length/capacity, it doesn’t do harm to simply increase this setting (well, unless #vtapes * their size is larger than your storage capacity, then you’ll run out of storage). See tapetype definition in amanda.conf.

Check the dumpcycle parameter in amanda.conf and feel free to change it.
Be aware that Amanda automatically schedules level 0/1 so you’ll have one complete dump within that period. It sometimes ‘promotes’ L0 dumps (i.e. runs them sooner than it would need to) in trying to achieve a more or less equal amount of data to backup each day, but that probably does not work well if you have just one big raw device and some small directories for Amanda to juggle with.
But you should not reduce the frequency at which you run amdump from cron as that would result in less data safety.


(Jean) #70

Hi,
I am digging into openhab now since some days and while not beeing experienced with Linux it was ok so far.
This backup topic now is making me crazy and I have spend more time reading through this thread and related documentation than for all other topics together.

I followed the readme and tried the additional steps discussed here but need help now or have to give up.

I am running openhabian 2.2. on Raspi 3, backup should go on USB stick as I thought it would be more easy than the NAS solution.

The installation of Amanda from openhabian config is not sucessful and I can not fix it.

Here the lines from the console

[17:37:13] openhabian@openHABianPi:~$ sudo openhabian-config
2018-02-18_17:37:27_CET [openHABian] Checking for root privileges... OK
2018-02-18_17:37:27_CET [openHABian] Loading configuration file '/etc/openhabian.conf'... OK
2018-02-18_17:37:27_CET [openHABian] openHABian configuration tool version: [master]v1.4-372(8bc6d7d)
2018-02-18_17:37:27_CET [openHABian] Checking for changes in origin... OK
2018-02-18_17:38:03_CET [openHABian] Setting up the Amanda backup system ...
$ apt -y install amanda-common amanda-server amanda-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
amanda-client is already the newest version (1:3.3.9-5).
amanda-common is already the newest version (1:3.3.9-5).
amanda-server is already the newest version (1:3.3.9-5).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ln: failed to create symbolic link '/storage/usbstick-msdos/slots/drive0': Operation not permitted
ln: failed to create symbolic link '/storage/usbstick-msdos/slots/drive1': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot1': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot2': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot3': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot4': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot5': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot6': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot7': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot8': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot9': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot10': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot11': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot12': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot13': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot14': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots/slot15': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos/slots': Operation not permitted
/bin/chown: changing ownership of '/storage/usbstick-msdos': Operation not permitted
2018-02-18_17:39:12_CET [openHABian] Checking for default openHABian username:password combination... OK
2018-02-18_17:39:12_CET [openHABian] We hope you got what you came for! See you again soon ;)

[17:39:12] openhabian@openHABianPi:~$ sudo su - backup
[17:40:18] backup@openHABianPi:~$ amcheck openhab-dir
Amanda Tape Server Host Check
-----------------------------
slot ?: Can't make directory '/storage/usbstick-msdos/slots/drive0': Permission denied
ERROR: Can't make directory '/storage/usbstick-msdos/slots/drive0': Permission denied
NOTE: host info dir /var/lib/amanda/openhab-dir/curinfo/openHABianPi does not exist
NOTE: it will be created on the next run.
NOTE: index dir /var/lib/amanda/openhab-dir/index/openHABianPi does not exist
NOTE: it will be created on the next run.
Server check took 0.712 seconds

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 2.770 seconds.  0 problems found.

(brought to you by Amanda 3.3.9)
[17:40:43] backup@openHABianPi:~$

It seems there is an issue with rights to create the drives and slots and while I had the impression this got mentioned here in the discussion and tried to fix it is always the same.

The USB is mounted and backup user got rights to access them according to the hints in this thread.

Any hint to solve this is appreciated.
Thank you


(Thomas Binder) #71

Are you 100% sure, this is the case? The posted message hints otherwise…
and are you also 100% sure, you read and understood the document, especially the part on the USB mounting?

please post the result of ls -laR /storage/ on your Pi


(Jean) #72

I read it several times and did my best to understand it.

[18:18:17] backup@openHABianPi:~$ ls -laR /storage/
/storage/:
total 24
drwxr-xr-x  3 root root  4096 Feb 18 10:51 .
drwxr-xr-x 22 root root  4096 Feb 18 10:51 ..
drwxr-xr-x  3 root root 16384 Feb 18 11:32 usbstick-msdos

/storage/usbstick-msdos:
total 36
drwxr-xr-x  3 root root 16384 Feb 18 11:32 .
drwxr-xr-x  3 root root  4096 Feb 18 10:51 ..
drwxr-xr-x 17 root root 16384 Feb 18 11:32 slots

/storage/usbstick-msdos/slots:
total 272
drwxr-xr-x 17 root root 16384 Feb 18 11:32 .
drwxr-xr-x  3 root root 16384 Feb 18 11:32 ..
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot1
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot10
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot11
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot12
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot13
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot14
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot15
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot2
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot3
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot4
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot5
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot6
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot7
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot8
drwxr-xr-x  2 root root 16384 Feb 18 11:32 slot9

/storage/usbstick-msdos/slots/slot1:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot10:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot11:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot12:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot13:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot14:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot15:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot2:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot3:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot4:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot5:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot6:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot7:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot8:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..

/storage/usbstick-msdos/slots/slot9:
total 32
drwxr-xr-x  2 root root 16384 Feb 18 11:32 .
drwxr-xr-x 17 root root 16384 Feb 18 11:32 ..
[18:38:53] backup@openHABianPi:~$

So there are quite some dirs…?!?


(Thomas Binder) #73

yes, there are a lot of “Slots” - representing tapes (the origin of Amanda)
The privileges on the subdirectories are for “root” only, that’s why you get those errors.
and this is, what startles me, because those directories will be generated by the openHABian install-script…

We could now work backwards through all your current installation - but best thing would be, if you have an image of your SD Card before your tries with Amanda? Then you can start over again - I don’t know Amanda yet and just started today and I had no problems (I used the NAS version).

If you can start over, I think the doc misses something which could lead to that mess…? If you have connected the USB stick and are ready to go, did you sudo bash before? I guess you did, because otherwise you wouldn’t have the privilege to echo into/etc/fstab … But it’s worth a shot.
and as we’re at it - you did only the MSDOS (meaning FAT16) version in the stick?

If that’s all set and done - I’m sorry, but then I can’t help you. But I’m sure someone with Amanda and/or knowledge of the openHABian scripts can help surely.


(Jean) #74

Thomas,
thank you for your approach to help me anyway.

This is also confusing me but thats why I write here.

I have no image of the SD, so far I used the backup method desribed here: https://docs.openhab.org/installation/linux.html#backup-and-restore

But I did some other changes since the last backup which I would like to keep if possible.

Actually I did not manage to do the changes into fstap with echo but have edited the file with vim:

[10:53:43] openhabian@openHABianPi:~$ echo "/dev/sda1     /storage/usbstick-msdos    vfat                                        defaults,noatime  0       1" >> /etc/fstab
-bash: /etc/fstab: Permission denied
[10:54:39] openhabian@openHABianPi:~$ sudo echo "/dev/sda1     /storage/usbstick-msdos    vfa                                   t     defaults,noatime  0       1" >> /etc/fstab
-bash: /etc/fstab: Permission denied

[19:12:19] openhabian@openHABianPi:~$ cat /etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=16d75cfa-01  /boot           vfat    defaults          0       2
PARTUUID=16d75cfa-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

/usr/share/openhab2          /srv/openhab2-sys           none bind 0 0
/etc/openhab2                /srv/openhab2-conf          none bind 0 0
/var/lib/openhab2            /srv/openhab2-userdata      none bind 0 0
/var/log/openhab2            /srv/openhab2-logs          none bind 0 0
/usr/share/openhab2/addons   /srv/openhab2-addons        none bind 0 0
/dev/sda1                    /storage/usbstick-msdos    vfat    defaults,noatime 0 1

I only used the FAT part because I did not see the need to have two different partitions.
But I thought it would be FAT32.

Did I get something wrong on that part?

Again thank you for your time!


(Thomas Binder) #75

at first: not really… as it is described in the docs, you only need one of these. FAT16/vfat is readable by MS-DOS and above (vfat is just FAT16 with longer filenames than 8.3). Thing is, not every OS can read ext4. So, if you just want to have a look on the USB-Stick from another OS, you must have an ext4-reader installed under Windows e.g. If you only use the USB stick for your Amanda backup you can also use ext4. If you want to read without additional Software under Windows, stick with FAT16.

[10:54:39] openhabian@openHABianPi:~$ sudo echo "/dev/sda1     /storage/usbstick-msdos    vfa                                   t     defaults,noatime  0       1" >> /etc/fstab
-bash: /etc/fstab: Permission denied

this one explains it I guess… you didn’t sudo bash before everything else! sudo echo ... alone won’t let you write fstab, you need full root here. I don’t know how the openHABian script reacts, but please try again:

  1. Format your USB-stick (you could just as easy assign ext4 this time, if you like)
  2. start over with the whole procedure, but before the fdisk-operation in the doc, do sudo bash, that way you gain full root-privileges and you won’t need sudo
  3. do as the doc says and echo "/dev/sda1 /storage/usbstick-msdos vfat defaults,noatime 0 1" >> /etc/fstab will work now - as the openhabian-config script can access and change the directory privileges then.

This should do it, hopefully…


(Markus Storm) #76

Just guess but the problem might be that the install script does a change ownership to the storage dir and that this does not work for vfat filesystem.
There’s a mount option you can add in the /etc/fstab line to make the whole filesystem be owned by the ‘backup’ user, see here for a description. Edit fstab, remount and reinstall.
But the better idea is to avoid using Windows concepts such as vfat on a UNIX system (I know I shouldn’t have put it into the docs, but you noticed the warning lines on top of and below the code lines, didn’t you ?) . Windows people tend to choose these as they know 'em, but in the end that’s always a bad idea because they need to be compatible with the UNIX software (openHABian and Amanda, these are) and not with the user. Better go with the ext4 version of the stick as shown in the README. Or at least create two partitions, one for backup storage using ext4.


(Jean) #77

Hi Markus,

thank you for creating the readme and your help here.
Got it working now with the additional support and better understanding.

I deleted the previously created dirs and the entry in fstab and formated the usb stick with ext4.
Then went through the steps again and it went all well.

The traps I got in were following, may be others can avoid this:

I thought there is no root usage forseen for openhabian so I tried to do by using sudo and got in some trouble and missunderstandings. Now I found out that root can be used (read here)

The command mkexfatfs seems not to be available in openhabian installation which also confused me a bit.
I have now skipped the FAT part.

Again thank you. Great to be able to benefit from your work!

Now that the installation is done I wonder if there is anything else to be done for scheduling the backups.
I somehow understood Amanda is taking care of it, but how can I see when it is planned or happening?


(Markus Storm) #78

Glad to hear.

[21:31:49] root@openHABianPi:/# cat /etc/cron.d/amanda
0 1 * * * backup /usr/sbin/amdump openhab-dir &>/dev/null
0 18 * * * backup /usr/sbin/amcheck -m openhab-dir &>/dev/null

(Jean) #79

Perfekt, danke!


(Frantisek) #80

Hi, I have installed and configured Amanda on my system but I am recieving an error and would kindly like to ask community for any hint what could be wrong. Thanks in advance!

My setup:
RPI 3, openhabian installation, running and booting from 16GB micro SD card (shrinked to 10GB to avoid potential issues when reimaging to another 16GB card which is few sectors smaller maybe as well as to decrease the time needed for full backup).

Directly to RPI is attached external 2,5" 60GB HDD, mounted to /mnt/HDD formated to NTFS as I wanted to easily copy backup files also to my WIN PC. Maybe this (NTFS) is the issue, I am not sure, but from what I read, this should work. I have enabled on RPI the higher power for USB and the HDD worked normally until I installed Amanda and run the first backup.

After Amanda installation (the improved readme is now really helpful and I read it about 3 times as well as this discussion before I started the installation so big thanks to @mstormi and others) there were correctly created folders and subfolders on the HDD (I am using only the option with local storage, not the Amazon nor NAS) although, I have inserted as the mount point during the Amanda setup the path in this format: “/mnt/HDD/” including the last slash, which was maybe not correct as Amanda in amreport reffered to “/mnt/HDD//slots/drive…” with double slahes after HDD so the slash was there from my input in the installation as well as Amanda probably added it itself. But as the folder structure on the mounted HDD semeed to be successfully created by amanda, I kept it for the first time as it was (those double slashes). Maybe the installation script could be adjusted to avoid this in the future or the readme could mention that.

Whatever, I have manually run the first backup and after several hours checked with amreport which showed some errors (unfortunately do not have it there was some issue with the raw backup of the card). I couldn’t access the HDD from the RPI (I tried list directories on HDD using ls but didn’t work). However, there WERE created some files on the HDD and I was able to access the HDD after RPI reboot. So I let that be and there was the first overnight backup scheduled by cron.

I had again an error though and the HDD was somehow locked again. So I edited the amanda.conf to correct the double slashes in the path to HDD mount so now is:

tpchanger "chg-disk:/mnt/HDD/slots"    # The tape-changer glue script

and installed the mailing on RPI so amanda could send me reports via email (btw very easy guideline to use gmail account for sending emails from RPI is here, maybe it’s worth add the link to the readme as well). So finally I received the log into my email:

Hostname: openHABianPi
Org     : openHABian openhab-dir
Config  : openhab-dir
Date    : březen 22, 2018

These dumps were to tape openHABian-openhab-dir-002.
Not using all tapes because taper found no tape.
No dumps are left in the holding disk.

The next 10 tapes Amanda expects to use are: 10 new tapes.


FAILURE DUMP SUMMARY:
  openHABianPi /dev/mmcblk0 lev 0  FAILED [data write: Connection reset by peer]
  openHABianPi /dev/mmcblk0 lev 0  partial taper: Error writing device fd 6: Input/output error
  openHABianPi /dev/mmcblk0 lev 0  FAILED [data write: Broken pipe]
  openHABianPi /dev/mmcblk0 lev 0  partial taper: taper found no tape



STATISTICS:
                          Total       Full      Incr.   Level:#
                        --------   --------   --------  --------
Estimate Time (hrs:min)     0:15
Run Time (hrs:min)          0:27
Dump Time (hrs:min)         0:00       0:00       0:00
Output Size (meg)            1.3        0.0        1.3
Original Size (meg)          1.3        0.0        1.3
Avg Compressed Size (%)    100.0        --       100.0
DLEs Dumped                    2          0          2  1:2
Avg Dump Rate (k/s)        538.2        --       538.2

Tape Time (hrs:min)         0:11       0:11       0:00
Tape Size (meg)              1.3        0.0        1.3
Tape Used (%)                0.0        0.0        0.0
DLEs Taped                     4          2          2  1:2
Parts Taped                    4          2          2  1:2
Avg Tp Write Rate (k/s)      2.0        0.0     6700.0

USAGE BY TAPE:
  Label                        Time         Size      %  DLEs Parts
  openHABian-openhab-dir-002   0:11     2380828k   69.8     4     4



FAILED DUMP DETAILS:
  /-- openHABianPi /dev/mmcblk0 lev 0 FAILED [data write: Connection reset by peer]
  sendbackup: info BACKUP=APPLICATION
  sendbackup: info APPLICATION=amraw
  sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/lib/amanda/application/amraw restore [./file-to-restore]+
  sendbackup: info COMPRESS_SUFFIX=.gz
  sendbackup: info end
  \--------
  /-- openHABianPi /dev/mmcblk0 lev 0 FAILED [data write: Broken pipe]
  sendbackup: info BACKUP=APPLICATION
  sendbackup: info APPLICATION=amraw
  sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/lib/amanda/application/amraw restore [./file-to-restore]+
  sendbackup: info COMPRESS_SUFFIX=.gz
  sendbackup: info end
  \--------



NOTES:
  planner: Adding new disk openHABianPi:/dev/mmcblk0.
  planner: Last full dump of openHABianPi:/etc/openhab2 on tape openHABian-openhab-dir-001 overwritten in 1 run.
  planner: Last full dump of openHABianPi:/var/lib/openhab2 on tape openHABian-openhab-dir-001 overwritten in 1 run.
  planner: WARNING: no history available for openHABianPi:/var/lib/openhab2; guessing that size will be 51730 KB
  planner: WARNING: no history available for openHABianPi:/etc/openhab2; guessing that size will be 1465 KB
  taper: Slot 2 without label can be labeled
  taper: Slot 3 without label can be labeled
  taper: tape openHABian-openhab-dir-002 kb 1340 fm 3 [OK]
  taper: while labeling new volume: Error checking directory /mnt/HDD/slots/drive2/data/: No such file or directory


DUMP SUMMARY:
                                                         DUMPER STATS   TAPER STATS
HOSTNAME     DISK              L ORIG-kB  OUT-kB  COMP%  MMM:SS   KB/s MMM:SS    KB/s
-------------------------------- ---------------------- -------------- --------------
openHABianPi /dev/mmcblk0      0                     --    PARTIAL      10:58     0.0 PARTIAL
openHABianPi /etc/openhab2     1      20      20     --    0:01   18.3   0:00   200.0
openHABianPi /var/lib/openhab2 1    1320    1320     --    0:01  945.7   0:00 13200.0

(brought to you by Amanda version 3.3.9)

The storagestate file contains:

$STATE = {
           'meta' => undef,
           'drives' => {
                         '/mnt/HDD/slots/drive1' => {},
                         '/mnt/HDD/slots/drive2' => {},
                         '/mnt/HDD//slots/drive1' => {},
                         '/mnt/HDD//slots/drive0' => {
                                                       'pid' => 10185
                                                     }
                       }
         };

Of course I googled the input/output error and most of the links says that the HDD is corrupted. However, this seems strange as after RPI restart it works normally. I am not experienced so can only think of some access rights issues, which is strange if amanda could create folders as well as some files…

Here is the content of my openhab-dir if that helps (bře = March):

openhabian@openHABianPi:/etc/amanda/openhab-dir $ ls -lh
celkem 24K
-rw-r--r-- 1 root   root     13 bře 20 13:11 amanda-client.conf
-rw-r--r-- 1 root   root   2,8K bře 21 22:21 amanda.conf
-rw-r--r-- 1 root   root   2,8K bře 21 21:01 amanda.conf.bkp
-rw-r--r-- 1 root   root    144 bře 20 13:11 disklist
-rw------- 1 backup backup  450 bře 22 10:36 storagestate
-rw------- 1 backup backup  122 bře 22 01:26 tapelist
-rw------- 1 backup backup    0 bře 20 14:01 tapelist.lock

This is the content of the slots folder on HDD (suprisingly I can ls it now without the reboot):

openhabian@openHABianPi:/mnt/HDD/slots $ ls -lh --color=auto
celkem 8,5K
lrwxrwxrwx 1 root root   18 bře 22 10:36 data -> slot3
drwxrwxrwx 1 root root    0 bře 20 13:47 drive0
drwxrwxrwx 1 root root    0 bře 22 01:00 drive1
drwxrwxrwx 1 root root 4,0K bře 20 14:01 slot1
drwxrwxrwx 1 root root    0 bře 20 13:10 slot10
drwxrwxrwx 1 root root    0 bře 20 13:10 slot11
drwxrwxrwx 1 root root    0 bře 20 13:10 slot12
drwxrwxrwx 1 root root    0 bře 20 13:10 slot13
drwxrwxrwx 1 root root    0 bře 20 13:10 slot14
drwxrwxrwx 1 root root    0 bře 20 13:10 slot15
drwxrwxrwx 1 root root 4,0K bře 22 01:15 slot2
drwxrwxrwx 1 root root    0 bře 20 13:10 slot3
drwxrwxrwx 1 root root    0 bře 20 13:10 slot4
drwxrwxrwx 1 root root    0 bře 20 13:10 slot5
drwxrwxrwx 1 root root    0 bře 20 13:10 slot6
drwxrwxrwx 1 root root    0 bře 20 13:10 slot7
drwxrwxrwx 1 root root    0 bře 20 13:10 slot8
drwxrwxrwx 1 root root    0 bře 20 13:10 slot9

and the content of the slot1 folder:

openhabian@openHABianPi:/mnt/HDD/slots/slot1 $ ls -lh --color=auto
celkem 3,0G
-rwxrwxrwx 1 root root  32K bře 20 14:01 00000.openHABian-openhab-dir-001
-rwxrwxrwx 1 root root 102M bře 20 14:01 00001.openHABianPi._var_lib_openhab2.0
-rwxrwxrwx 1 root root 2,9M bře 20 14:01 00002.openHABianPi._etc_op짉nhab2.0
-rwxrwxrwx 1 root root 2,9G bře 20 14:15 00003.openHABianPi._dev_mmcblk0.0

slot2 content (other slots are empty):

openhabian@openHABianPi:/mnt/HDD/slots/slot2 $ ls -lh --color=auto
celkem 2,2G
-rwxrwxrwx 1 root root  32K bře 22 01:15 00000.openHABian-openhab-dir-002
-rwxrwxrwx 1 root root  52K bře 22 01:15 00001.openHABianPi._etc_openhab2.1
-rwxrwxrwx 1 root root 1,4M bře 22 01:15 00002.openHABianPi._var_lib_openhab2.1
-rwxrwxrwx 1 root root 2,2G bře 22 01:25 00003.openHABianPi._dev_mmcblk0.0

So it seems to me that only the first manually run backup worked (with error that I do not have) but not the automated ones. May this really be the ownership issue as the owner of the slot folders seems to be root so user backup can’t write there? If yes, I thought that he amanda installation should take care of that… Is the chown the solution then? It would be strange anyway as the first manual backup I started logged as backup user.

Thanks a lot for any hint!

UPDATE 22.03.2018 14:10
I have created new profile for testing that is backing up only small folder with influxdb data, using same settings for slots and tapes copied from openhab-dir and it works! See the log from email:

Hostname: openHABianPi
Org     : openHABian influx-dir
Config  : influx-dir
Date    : březen 22, 2018

These dumps were to tape openHABian-influx-dir-001.
The next 10 tapes Amanda expects to use are: 10 new tapes.


STATISTICS:
                          Total       Full      Incr.   Level:#
                        --------   --------   --------  --------
Estimate Time (hrs:min)     0:00
Run Time (hrs:min)          0:00
Dump Time (hrs:min)         0:00       0:00       0:00
Output Size (meg)           15.6       15.6        0.0
Original Size (meg)         15.6       15.6        0.0
Avg Compressed Size (%)    100.0      100.0        --
DLEs Dumped                    1          1          0
Avg Dump Rate (k/s)       4536.2     4536.2        --

Tape Time (hrs:min)         0:00       0:00       0:00
Tape Size (meg)             15.6       15.6        0.0
Tape Used (%)                0.5        0.5        0.0
DLEs Taped                     1          1          0
Parts Taped                    1          1          0
Avg Tp Write Rate (k/s)  15990.0    15990.0        --

USAGE BY TAPE:
  Label                       Time         Size      %  DLEs Parts
  openHABian-influx-dir-001   0:00       15990k    0.5     1     1



NOTES:
  planner: Adding new disk openHABianPi:/var/lib/influxdb/data.
  planner: WARNING: no history available for openHABianPi:/var/lib/influxdb/data; guessing that size will be 1000000 KB
  taper: Slot 3 without label can be labeled
  taper: Slot 4 without label can be labeled
  taper: tape openHABian-influx-dir-001 kb 15990 fm 1 [OK]
  big estimate: openHABianPi /var/lib/influxdb/data 0
                  est: 1000032k    out 15990k


DUMP SUMMARY:
                                                              DUMPER STATS   TAPER STATS
HOSTNAME     DISK                   L ORIG-kB  OUT-kB  COMP%  MMM:SS   KB/s MMM:SS    KB/s
------------------------------------- ---------------------- -------------- --------------
openHABianPi /var/lib/influxdb/data 0   15990   15990     --    0:04 4535.4   0:01 15990.0

(brought to you by Amanda version 3.3.9)

and the slots content now (slot3 used for the influx-dir backup):

[14:12:42] backup@openHABianPi:/mnt/HDD/slots$ ls
celkem 8,5K
lrwxrwxrwx 1 root root   18 bře 22 14:08 data -> slot4
drwxrwxrwx 1 root root    0 bře 20 13:47 drive0
drwxrwxrwx 1 root root    0 bře 22 01:00 drive1
drwxrwxrwx 1 root root 4,0K bře 20 14:01 slot1
drwxrwxrwx 1 root root    0 bře 20 13:10 slot10
drwxrwxrwx 1 root root    0 bře 20 13:10 slot11
drwxrwxrwx 1 root root    0 bře 20 13:10 slot12
drwxrwxrwx 1 root root    0 bře 20 13:10 slot13
drwxrwxrwx 1 root root    0 bře 20 13:10 slot14
drwxrwxrwx 1 root root    0 bře 20 13:10 slot15
drwxrwxrwx 1 root root 4,0K bře 22 01:15 slot2
drwxrwxrwx 1 root root  360 bře 22 14:08 slot3
drwxrwxrwx 1 root root    0 bře 20 13:10 slot4
drwxrwxrwx 1 root root    0 bře 20 13:10 slot5
drwxrwxrwx 1 root root    0 bře 20 13:10 slot6
drwxrwxrwx 1 root root    0 bře 20 13:10 slot7
drwxrwxrwx 1 root root    0 bře 20 13:10 slot8
drwxrwxrwx 1 root root    0 bře 20 13:10 slot9

and the content of the slot3 folder:

[14:14:16] backup@openHABianPi:/mnt/HDD/slots/slot3$ ls
celkem 16M
-rwxrwxrwx 1 root root 32K bře 22 14:08 00000.openHABian-influx-dir-001
-rwxrwxrwx 1 root root 16M bře 22 14:08 00001.openHABianPi._var_lib_influxdb_data.0