When a ZFS file system has no space left then the deletion of files can fail with “disk quota exceeded“. The post provides different ways to create free space to overcome the situation.
1. Truncating files
If the files can not be removed directly, we can first truncate them and then delete them.
# cat /dev/null > /file/to/delete # rm /file/to/delete
Alternatives: there are other options to free up space in the zpool, e.g.
1. increase the quota if there is space in the zpool left
2. Shrink the size of a zvol
3. temporarily destroy a dump device (if the rpool is affected)
4. delete unused snapshots
5. increase the space in the zpool by enlarging a vdev or adding a vdev
6. Temporarily decrease refreservation of a ZVol
7. Rolling back the log
2. Increasing the quota
You can check the current quota with
# zfs get quota {filesystem}
# zfs get refquota {filesystem}
If the quota is set then you can increase it with
# zfs set quota={value} {dataset}
# zfs set refquota={value} {dataset}
3. Shrink the size of a zvol
If the zvol of the affected zpool has some space left then it can be shrinked.
# zfs set volsize=[newsize] ZPOOL/ZVOL
For the rpool and its dump zvol it looks like this:
# zfs set volsize=[newsize] rpool/dump
4. Destroying a dump device on an rpool (or an unused zvol)
The rpool’s dump device can temporarily be destroyed, as it is used for storing a livedump (“savecore -L”) or a crash dump in case of a panic.
# dumpadm Dump content : kernel with ZFS metadata Dump device : /dev/zvol/dsk/rpool/dump (dedicated) Savecore directory: /var/crash Savecore enabled : yes Save compressed : on
You can inspect all zvols with
# zfs list -t volume NAME USED AVAIL REFER MOUNTPOINT rpool/dump 5.16G 11.4G 4.00G - rpool/swap 2.06G 10.3G 2.00G - rpool/testvol 103M 10.3G 8.13M -
# zfs list rpool/dump NAME USED AVAIL REFER MOUNTPOINT rpool/dump 5.16G 11.4G 4.00G -
Please note the volsize for the re-creation of the zvol later on; in this case 4 GB of space can be created by destroying the zvol.
# zfs destroy rpool/dump
Later you can re-create the zvol with (note that the size of 4 GB is used here according to the example; specify the volsize you saw in your output of “zfs get volsize rpool/dump”):
# zfs create -V 4g rpool/dump
5. Delete unused snapshots
List the snapshots with
# zfs list -t snapshot
You can sort the output; here the “used” property is used:
# zfs list -t snapshot -S used
You can see the contents of a snapshot; there is a special directory “.zfs/snapshot” directory in the root of the file system. By default the directoy “.zfs” is hidden. You need to change the property “snapdir” to see the directory:
# zfs set snapdir=visible {filesystem}
Now you can change to the directoy /{filesystem}/.zfs/snapshot with cd(1) and inspect the contents. Then decide which snapshot you do not need and delete it with
# zfs destroy {snapshot}
6. Increase the space in the zpool by enlarging a vdev or adding a vdev
You can add another vdev to the zpool to make it bigger. If your zpool consists of raidzN vdevs please make sure that you add another raidzN vdevs. If you would erroneously add a single vdev then your zpool is no longer fully redundant; you could attach another vdev to have a mirror added to the raidzN vdevs and have another type of redundancy. But if you make a mistake then this can just be corrected by re-creating the whole zpool.
You can grow a vdev by growing the underlying LUN or make the partition larger. Please make sure that the following property is set for the zpool:
# zpool get autoexpand {zpool}
# zpool set autoexpand=on {zpool}
This way ZFS will detect changes in the size of the used storage devices and will propagate the change to the vdev.
7. Temporarily decrease refreservation of a ZVol
Temporarily decrease the refreservation of any zvols that are not using all of their reserved space. This frees space within the zpool to allow a more permanent solution to be implemented, before the refreservation is restored to its original value.
List the zvols:
# zfs list -t vol NAME PROPERTY VALUE SOURCE rpool/dump referenced 4.00G - rpool/dump refreservation 5.16G local
Select a zvol and display its current refreserv and usedbyrefreserv:
# zfs get refreservation,usedbyrefreservation rpool/dump NAME PROPERTY VALUE SOURCE rpool/dump refreservation 5.16G local rpool/dump usedbyrefreservation 1.16G -
Decrease the refreserv:
# zfs set refreservation=5.0G rpool/dump
Note: ‘refreservation’ may be greater than necessary. You can use ‘auto’ to reserve enough space for a non-sparse volume. After you successfully deleted the data restore the refreservation back to the original value, here:
# zfs set refreservation=5.16G rpool/dump
Note: ‘refreservation’ may be greater than necessary. You can use ‘auto’ to reserve enough space for a non-sparse volume.
8. Rolling back the log
If the import of a zpool fails due to “out of space” then you can import with rolling the log back to the previous commit to the log; however all updates to the filesystem after that commit was made are lost. You can achieve this by
# zpool import -F ZPOOLNAME
It will tell you the point in time to which the log was rolled back, e.g.:
# zpool import -F rpool
Pool rpool returned to its state as of Thu Jun 09 07:12:01 2016. If you just want to check whether this is possible then you can do
# zpool import -Fn ZPOOLNAME