Backupninja running out of disk space
Backupninja starting giving me errors along the lines of:
[Errno 28] No space left on device Fatal Error: Lost connection \ to the remote system
Yet, df on that server showed plenty of room:
0 munson:/var# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda1 1829159 663080 1068486 39% / tmpfs 95492 0 95492 0% /lib/init/rw udev 10240 48 10192 1% /dev tmpfs 95492 0 95492 0% /dev/shm /dev/mapper/main-var 982728 216664 713636 24% /var /dev/mapper/main-srv 206424760 69507240 126431760 36% /srv 0 munson:/var
All backups were going to the /srv partition, which has a lot of room, both before and after the backup started and ended.
Next I ran:
To see how it changed in realtime while the backup was running, and I found that it’s the var partition that was growing. Fortunately, backupninja was cleaning up after itself when it quit, leaving the filesystem exactly the same as when it started.
I wanted to know what directory was getting written to, so I tried running:
find . /var > /root/before.txt
Before running the backup, followed by:
find . /var > /root/after.txt
During the backup.
diff -u /root/after.txt /root/before.txt
Revealed no changes.
So, I tried:
lsof | grep rdiff
And found, among other things:
rdiff-bac 7624 rbi-backup 7u REG 254,2 231211008 \ 14172176 /srv/tmp/tmphgYI7a (deleted)
It looks like rdiff backup was creating a temp file using the /var/tmp directory.
Since this server only serves one purpose: backing up, I decided to make things easier and simply move the /var/tmp directory to /srv/tmp and then leave a symlink in /var/tmp.