Backupninja running out of disk space

2007-01-08 2-minute read

Backupninja starting giving me errors along the lines of:

[Errno 28] No space left on device Fatal Error: Lost connection \
to the remote system

Yet, df on that server showed plenty of room:

0 munson:/var# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda1              1829159    663080   1068486  39% /
tmpfs                    95492         0     95492   0% /lib/init/rw
udev                     10240        48     10192   1% /dev
tmpfs                    95492         0     95492   0% /dev/shm
/dev/mapper/main-var    982728    216664    713636  24% /var
/dev/mapper/main-srv 206424760  69507240 126431760  36% /srv
0 munson:/var

All backups were going to the /srv partition, which has a lot of room, both before and after the backup started and ended.

Next I ran: watch df

To see how it changed in realtime while the backup was running, and I found that it’s the var partition that was growing. Fortunately, backupninja was cleaning up after itself when it quit, leaving the filesystem exactly the same as when it started.

I wanted to know what directory was getting written to, so I tried running:

find . /var > /root/before.txt

Before running the backup, followed by:

find . /var > /root/after.txt

During the backup.

However:

diff -u /root/after.txt /root/before.txt

Revealed no changes.

So, I tried:

lsof | grep rdiff

And found, among other things:

rdiff-bac  7624 rbi-backup    7u      REG      254,2  231211008   \
14172176 /srv/tmp/tmphgYI7a (deleted)

It looks like rdiff backup was creating a temp file using the /var/tmp directory.

Since this server only serves one purpose: backing up, I decided to make things easier and simply move the /var/tmp directory to /srv/tmp and then leave a symlink in /var/tmp.

Problem solved.