Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

ZFS no space left on device: How to resolve

by | Oct 17, 2021

ZFS no space left on device error giving you trouble? Are you finding it difficult to delete files on your ZFS file system? Our Support Team has come up with eight different solutions for this specific issue.

Many of our customers run into the disk quota exceeded message when they attempt to delete files on their ZFS file system. Fortunately, our Support Team has put together a bouquet of solutions to help resolve this error without too much trouble.

Find out which one works for you and let us know in the comments.

What is ZFS no space left on device error?

The ZFS file system offers the features of a file system as well as a volume manager. In other words, it is different from conventional file systems and RAID arrays. It comes with several built-in storage features like deduplication, replication, compression, clones, snapshots, data protection, and so on.

However, you may find yourself facing a “disk quota exceeded “ error when the ZFS has no more space left. This error often arises when you attempt to delete files when free space is not available.

Fortunately, our Support Team has come up with eight different ways you can resolve this specific issue. Let’s take a look at each of these solutions in detail.

How to fix ZFS: No space left on device?

  1. Truncating files
  2. Shrinking the size of zvol
  3. Increasing quota
  4. Rolling back the log
  5. Destroying dump device on rpool
  6. Increasing space in zpool via vdev
  7. Deleting unused snapshots
  8. Temporarily decreasing reservation of zvol

 Truncating Files to solve ZFS no space left error

In case the files are not removable directly, we fill first truncate then and then proceed with the deletion.

# cat /dev/null > /file/to/delete
# rm /file/to/delete

Shrinking the size of zvol

Another way to resolve the “disk quota exceeded “ is to shrink the size of a zvol of the affected zpool. This is done using the command:

# zfs set volsize=[newsize] ZPOOL/ZVOL

You can also shrink the size of the dump zvol of the rpool with the command:

# zfs set volsize=[newsize] rpool/dump

Increasing quota to solve ZFS no space left error

An alternative way to solve the no space issue would be to first check the current quota, and then increase if it is not enough.

  1. First, check the current quote with the following commands:
    # zfs get quota {filesystem}
    # zfs get refquota {filesystem}
  2. Then, increase it with the following commands:
    # zfs set quota={value} {dataset}
    # zfs set refquota={value} {dataset}

Rolling back the log

If importing a zpool fails because of no disk space, you can attempt importing by rolling the log to the previous commit. However, any updates made to the file system after that specific commit will be lost.

# zpool import -F ZPOOLNAME

This command tells us when the log was rolled back. For instance:

# zpool import -F rpool

The above example returns the rpool to its state as of Thu Oct 09 07:12:01 2021. However,if you just want to verify whether you can roll back with the following statement as well:

# zpool import -Fn ZPOOLNAME

Destroying dump device on rpool

A different way to deal with the disk quota exceeded error would be to temporarily destroy the rpool’s dump device. The dump device is used to store a crash dump or a livedump in case of a panic.

# dumpadm
Dump content      : kernel with ZFS metadata
Dump device       : /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash
Savecore enabled  : yes
Save compressed   : on
Then, you can inspect the zvols with the following command:
# zfs list -t volume
NAME            USED  AVAIL  REFER  MOUNTPOINT
rpool/dump     5.15G  11.3G  4.00G  -
rpool/swap     2.05G  10.2G  2.00G  -
rpool/testvol   102M  10.2G  8.12M  -
# zfs list rpool/dump
NAME         USED  AVAIL  REFER  MOUNTPOINT
rpool/dump  5.15G  11.3G  4.00G  -

Remember to note the volsize. We will use this later on during the re-creation of the zvol. In the above example, we can create 4 GB space by destroying the zvol.

Increasing space in zpool via vdev

This additional way to solve the no disk space error involves adding another vdev to make the zpool bigger. In other words, adding a raidzN vdev to the zpool to make it bigger.

In case you make the mistake of adding a single vdev when the zpool is not fully redundant, you can attach another vdev. This will be similar to having a mirror added to the raidzN and also create an additional type of redundancy.

However, if you wind up making a mistake, it can be easily fixed by re-creating the whole zpool.

A vdev can be enlarged by making the partition bigger or growing the underlying LUN. Remember to verify that the following property is set before proceeding:

# zpool get autoexpand {zpool}
# zpool set autoexpand=on {zpool}

This ensures that the ZFS detects the size changes of the used storage devices and propagates the change to the vdev.

Deleting unused snapshots

  1. First, we have to list the snapshots with the following command:
    # zfs list -t snapshot

    Then we will sort the output with the help of the “used” property as seen below:

    # zfs list -t snapshot -S used

    This will display the contents of the snapshot, including the .zfs/snapshot directory in the file system’s root.

  2. If the directory is not visible, you will have to run the following command to change the property snapdir in order to see the directory:
    # zfs set snapdir=visible {filesystem}
  3. Then, change to the /{filesystem}/.zfs/snapshot with the cd command. Then inspect its content to decide which snapshots can be deleted. After that, run the following command to delete the selected snapshot.
    # zfs destroy {snapshot}

Temporarily decreasing refreservation of zvol

The last solution offered by our Support Team would be to temporarily decrease the refreservation of the zvols that are not utilizing their reserved space. This will free up plenty of space within the zpool.

Moreover, it also makes way for a more permanent solution before the refreservation is returned to its initial value.

  1. First, get a list of the zvols by running the following command:
    # zfs list -t vol
    NAME        PROPERTY        VALUE  SOURCE
    rpool/dump  referenced      4.00G  -
    rpool/dump  refreservation  5.15G  local
  2. Then choose a zvol and display its usedbyrereserv and current refreserv:
    # zfs get refreservation,usedbyrefreservation rpool/dump
    NAME        PROPERTY              VALUE  SOURCE
    rpool/dump  refreservation        5.15G  local
    rpool/dump  usedbyrefreservation  1.15G  -
  3. After that, decrease the refreserv with the command:
    # zfs set refreservation=5.0G rpool/dump

    According to our Support Team, you can also use auto to reserve sufficient space for a non-sparse volume. Once the data is deleted successfully, restore the refreservation to the initial value as seen below:

    # zfs set refreservation=5.15G rpool/dump

[Is Server Management giving you trouble? Give us a call.]

Conclusion

To conclude, the skilled Support Engineers at Bobcares introduced eight different ways to deal with the ZFS no space left on device error. This will enable us to delete files as and when required without running into the disk quota exceeded message.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF