Backupninja running out of disk space
Backupninja starting giving me errors along the lines of:
[Errno 28] No space left on device Fatal Error: Lost connection \
to the remote system
Yet, df on that server showed plenty of room:
0 munson:/var# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 1829159 663080 1068486 39% /
tmpfs 95492 0 95492 0% /lib/init/rw
udev 10240 48 10192 1% /dev
tmpfs 95492 0 95492 0% /dev/shm
/dev/mapper/main-var 982728 216664 713636 24% /var
/dev/mapper/main-srv 206424760 69507240 126431760 36% /srv
0 munson:/var
All backups were going to the /srv partition, which has a lot of room, both before and after the backup started and ended.
Next I ran:
watch df
To see how it changed in realtime while the backup was running, and I found that it’s the var partition that was growing. Fortunately, backupninja was cleaning up after itself when it quit, leaving the filesystem exactly the same as when it started.
I wanted to know what directory was getting written to, so I tried running:
find . /var > /root/before.txt
Before running the backup, followed by:
find . /var > /root/after.txt
During the backup.
However:
diff -u /root/after.txt /root/before.txt
Revealed no changes.
So, I tried:
lsof | grep rdiff
And found, among other things:
rdiff-bac 7624 rbi-backup 7u REG 254,2 231211008 \
14172176 /srv/tmp/tmphgYI7a (deleted)
It looks like rdiff backup was creating a temp file using the /var/tmp directory.
Since this server only serves one purpose: backing up, I decided to make things easier and simply move the /var/tmp directory to /srv/tmp and then leave a symlink in /var/tmp.
Problem solved.