ZFS no space left on device error giving you trouble? Are you finding it difficult to delete files on your ZFS file system? Our Support Team has come up with eight different solutions for this specific issue.

Many of our customers run into the disk quota exceeded message when they attempt to delete files on their ZFS file system. Fortunately, our Support Team has put together a bouquet of solutions to help resolve this error without too much trouble.

Find out which one works for you and let us know in the comments.

What is ZFS no space left on device error?

The ZFS file system offers the features of a file system as well as a volume manager. In other words, it is different from conventional file systems and RAID arrays. It comes with several built-in storage features like deduplication, replication, compression, clones, snapshots, data protection, and so on.

However, you may find yourself facing a “disk quota exceeded “ error when the ZFS has no more space left. This error often arises when you attempt to delete files when free space is not available.

Fortunately, our Support Team has come up with eight different ways you can resolve this specific issue. Let’s take a look at each of these solutions in detail.

How to fix ZFS: No space left on device?

1. Truncating files
2. Shrinking the size of zvol
3. Increasing quota
4. Rolling back the log
5. Destroying dump device on rpool
6. Increasing space in zpool via vdev
7. Deleting unused snapshots
8. Temporarily decreasing reservation of zvol

 Truncating Files to solve ZFS no space left error

In case the files are not removable directly, we fill first truncate then and then proceed with the deletion.

# cat /dev/null > /file/to/delete
# rm /file/to/delete

Shrinking the size of zvol

Another way to resolve the “disk quota exceeded “ is to shrink the size of a zvol of the affected zpool. This is done using the command:

# zfs set volsize=[newsize] ZPOOL/ZVOL

You can also shrink the size of the dump zvol of the rpool with the command:

# zfs set volsize=[newsize] rpool/dump

Increasing quota to solve ZFS no space left error

An alternative way to solve the no space issue would be to first check the current quota, and then increase if it is not enough.

1. First, check the current quote with the following commands:

   # zfs get quota {filesystem}
   # zfs get refquota {filesystem}

2. Then, increase it with the following commands:

   # zfs set quota={value} {dataset}
   # zfs set refquota={value} {dataset}

Rolling back the log

If importing a zpool fails because of no disk space, you can attempt importing by rolling the log to the previous commit. However, any updates made to the file system after that specific commit will be lost.

# zpool import -F ZPOOLNAME

This command tells us when the log was rolled back. For instance:

# zpool import -F rpool

The above example returns the rpool to its state as of Thu Oct 09 07:12:01 2021. However,if you just want to verify whether you can roll back with the following statement as well:

# zpool import -Fn ZPOOLNAME

Destroying dump device on rpool

A different way to deal with the disk quota exceeded error would be to temporarily destroy the rpool’s dump device. The dump device is used to store a crash dump or a livedump in case of a panic.

# dumpadm
Dump content      : kernel with ZFS metadata
Dump device       : /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash
Savecore enabled  : yes
Save compressed   : on

Then, you can inspect the zvols with the following command:

# zfs list -t volume
NAME            USED  AVAIL  REFER  MOUNTPOINT
rpool/dump     5.15G  11.3G  4.00G  -
rpool/swap     2.05G  10.2G  2.00G  -
rpool/testvol   102M  10.2G  8.12M  -

# zfs list rpool/dump
NAME         USED  AVAIL  REFER  MOUNTPOINT
rpool/dump  5.15G  11.3G  4.00G  -

Remember to note the volsize. We will use this later on during the re-creation of the zvol. In the above example, we can create 4 GB space by destroying the zvol.

Increasing space in zpool via vdev

This additional way to solve the no disk space error involves adding another vdev to make the zpool bigger. In other words, adding a raidzN vdev to the zpool to make it bigger.

In case you make the mistake of adding a single vdev when the zpool is not fully redundant, you can attach another vdev. This will be similar to having a mirror added to the raidzN and also create an additional type of redundancy.

However, if you wind up making a mistake, it can be easily fixed by re-creating the whole zpool.

A vdev can be enlarged by making the partition bigger or growing the underlying LUN. Remember to verify that the following property is set before proceeding:

# zpool get autoexpand {zpool}
# zpool set autoexpand=on {zpool}

This ensures that the ZFS detects the size changes of the used storage devices and propagates the change to the vdev.

Deleting unused snapshots

1. First, we have to list the snapshots with the following command:

   # zfs list -t snapshot

   Then we will sort the output with the help of the “used” property as seen below:

   # zfs list -t snapshot -S used

   This will display the contents of the snapshot, including the .zfs/snapshot directory in the file system’s root.
2. If the directory is not visible, you will have to run the following command to change the property snapdir in order to see the directory:

   # zfs set snapdir=visible {filesystem}

3. Then, change to the /{filesystem}/.zfs/snapshot with the cd command. Then inspect its content to decide which snapshots can be deleted. After that, run the following command to delete the selected snapshot.

   # zfs destroy {snapshot}

Temporarily decreasing refreservation of zvol

The last solution offered by our Support Team would be to temporarily decrease the refreservation of the zvols that are not utilizing their reserved space. This will free up plenty of space within the zpool.

Moreover, it also makes way for a more permanent solution before the refreservation is returned to its initial value.

1. First, get a list of the zvols by running the following command:

   # zfs list -t vol
   NAME        PROPERTY        VALUE  SOURCE
   rpool/dump  referenced      4.00G  -
   rpool/dump  refreservation  5.15G  local

2. Then choose a zvol and display its usedbyrereserv and current refreserv:

   # zfs get refreservation,usedbyrefreservation rpool/dump
   NAME        PROPERTY              VALUE  SOURCE
   rpool/dump  refreservation        5.15G  local
   rpool/dump  usedbyrefreservation  1.15G  -

3. After that, decrease the refreserv with the command:

   # zfs set refreservation=5.0G rpool/dump

   According to our Support Team, you can also use auto to reserve sufficient space for a non-sparse volume. Once the data is deleted successfully, restore the refreservation to the initial value as seen below:

   # zfs set refreservation=5.15G rpool/dump

[Is Server Management giving you trouble? Give us a call.]

Conclusion

To conclude, the skilled Support Engineers at Bobcares introduced eight different ways to deal with the ZFS no space left on device error. This will enable us to delete files as and when required without running into the disk quota exceeded message.