You read that right. Today I had a partition with over 150 gigabytes of space left on it and a drive error reporting that said partition was full. Not only that, but this was an important partition for me, the one that I keep in a mirrored RAID because it stores the main copy of all my school work, programming and more, so I was very concerned about disk errors.

The problem turned out to be related to the differential backups that I do to the partition - backups that created many small increments files on each backup, files which hadn’t been deleted for years. It turned out that the inodes, which are the data structures that store each file on disk, were all in use.

I was able to get a display of the used inodes with the command

df -i

which yielded output like this

Filesystem Inodes IUsed IFree IUse% Mounted on

/dev/md1 790144 790144 0 100% /mnt/maindrive

Unfortunately there were only two solutions that I could figure out to this problem. One was to create a new filesystem on the partition that has more inodes, and migrate over to it. Unfortunately, to do this required that I have enough space to store the data while the partition’s filesystem was changed - which would mean I would have to get a new drive. The other option was to delete enough files to free up inodes to last me a while. This was simple enough to do, as I could have rdiff-backup delete all backups older than a certain date, as well as delete some other old files that I was unlikely to use again. I quickly decided to implement the second option.

After having deleted all my desktop backups more than a month old, and all the backups of my Minecraft server over two weeks old, df -i reported back 8% of the inodes free. Hopefully this will be enough to keep me going for a few more months, until I have time to rework the partitioning scheme.

In the future, I plan on creating a separate partition for backups, one which has more inodes to support the huge number of increment files that rdiff-backup creates over the long term. I may even leave behind ext filesystems for this purpose, assuming I can find an filesystem more optimized for backups (which I suspect I will). It also reminds me that the time for getting a new set of hard drives is drawing near, as I slowly edge towards completely filling the last 100 gigabytes of those old 1 terabyte drives.