Tired of getting OnApp resize disk error? The Support Engineers at Bobcares have an easy fix for this issue.

One of our customers recently faced errors when they tries to resize the virtual machine disk on LVM storage, Find out how our OnApp Hosting Support Services can help you with all your queries, big and small.

How to deal with OnApp resize disk error

The following error is an indication of OnApp resize disk error:

Error running lvchange: LV onapp-e4jsf92mjh6wk7/e7salolw28scol has open snapshot backup-hsak2e90z2m2jt: not deactivating\n

The error is seen in most OnApp cloud versions that use LVM datastore.
According to our Support Team, first, log on to the server listed in the failed log. Then run lvdisplay command as seen below to find the specific VM disk on the datastore.

lvdisplay /dev/onapp-e4jsf92mjh6wk7/e7salolw28scol  -

The server IP is present in the call if you are looking at storageAPI error. Additionally, you will find the disk identifier and the datastore in the URL: Datastore/<datastore identifier>Vdisk/<diskidentifier>

For instance, here is an LV snapshot status:

/dev/onapp-e4jsf92mjh6wk7/backup-hsak2e90z2m2jt [active]

Next, we will run lvremove in order to remove the snapshot. For instance:

[root@hv1-onappCorp ~]# lvremove /dev/onapp-e4jsf92mjh6wk7/backup-hsak2e90z2m2jt
Can't remove open logical volume "backup-hsak2e90z2m2jt"

If you wind up with this error, track down the process that is responsible for holding the snapshot open. Or, in case the snapshot is mounted or mapped, release them as seen below:

[root@hv1-onappCorp ~]# lsof /dev/onapp-e4jsf92mjh6wk7/backup-hsak2e90z2m2jt
[root@hv1-onappCorp ~]# mount

After this, try lvremove again, and you will see something similar to this:

Do you really want to remove active logical volume backup-hsak2e90z2m2jt? [y/n]: y. Logical volume "backup-hsak2e90z2m2jt" successfully removed

If you see an error about -real device, but cannot locate the snapshot. We recommend examining the dmsetup tree.

OnApp resize disk error: Track the process preventing dmentry from being removed

1. 1. First, SSH into the hypervisor shown in the error log.
   2. Next, run the following command to examine the device-mapper heirarchy for the backup responsible for the -real snapshot in the error log:

       – Run: dmsetup ls –tree | grep -B5 onapp–xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real

      We would like to point out that you may have to increase -B swtich value to a value higher than 5 in case the root backup device is not visible. For instance:

      [root@10.0.0.97 ~]# dmsetup ls --tree | grep -B5 onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real
      onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq (253:12)
       |-onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq-cow (253:21)
       |  `-mpath2 (253:6)
       |     |- (8:64)
       |     `- (8:32)
       `-onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real (253:20)
   3. After that, we have to find out which process is utilizing the top dm entry in the heirarchy with lsof.

      – onapp–xxxxxxxxxxxxxx-backup–yyyyyyyyyyyyyy
      – Run:  lsof /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy

      For instance:

      [root@10.0.0.97]# lsof /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq 
      COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
      tgtd    5991 root   15u   BLK 253,12      0t0 331375079 /dev/mapper/onapp—y0jmntchnsldn8-backup--c00fxwrzcy1jhq

      However, if there isn’t a backup server, lsof will not show anything. We recommend verifying the backup is mounted with “/proc/mounts/”. In case it is mounted, unmount it and then clear the dmsetup entries.
   4. Verify if any iSCSI targets are active with the following command since the backup is used by tgtd.

      – Run: tgtadm –lld iscsi –mode target –op show

      For instance:

      [root@10.0.0.97]# tgtadm --lld iscsi --mode target --op show
      Target 1: iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq
          System information:
              Driver: iscsi
              State: ready
          I_T nexus information:
          LUN information:
      
              LUN: 0
                  Type: controller
                  SCSI ID: IET     00010000
                  SCSI SN: beaf10
                  Size: 0 MB, Block size: 1
                  Online: Yes
                  Removable media: No
                  Readonly: No
                  Backing store type: null
                  Backing store path: None
                  Backing store flags: 
      
              LUN: 1
                  Type: disk
                  SCSI ID: IET     00010001
                  SCSI SN: beaf11
                  Size: 644245 MB, Block size: 512
                  Online: Yes
                  Removable media: No
                  Readonly: No
                  Backing store type: rdwr
                  Backing store path: /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq
                  Backing store flags: 
      
          Account information:
          ACL information:
              10.0.50.99
      
   5. Next, scrutinize the backup server for any related ISCSI sessions. Then log out of them.
      • First log in to the backup server in the “ACL information” and run iscsiadm -m session.
      • Then, compare the target name from the earlier step. After that, log out of the ISCSI sessions with iscsiadm -m node -u -T <target name>.
      • In the absence of any more corresponding sessions, skip to the next step.

      For instance:

      [root@10.0.0.99]# iscsiadm -m session
      tcp: [171] X.X.X.X:XXXX,1 iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq
      [root@10.0.0.99]# iscsiadm -m node -u -T iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq

      Our Engineers would like to point out that in case the iscsiadm command does not include -T after -u, you will wind up logging out of all iscsi targets.
   6. After that, delete the iSCSI target. Return to hypervisor and run the following command:

      tgtadm –lld iscsi –op delete –mode target –tid *

      Here, tid represents Target ID. It is displayed when the command in step 4 is run. For instance:

      [root@10.0.0.97]# tgtadm --lld iscsi --op delete --mode target --tid 1
   7. Finally, run the dm entries as well as -real snapshot. Run this for entries that have X1 at the end:

      kpartx -d -p X /dev/mapper/backup-XXXXXXXXXX

      Followed by:

      – Run: dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy
      – Run: dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy-cow
      – Run: dmsetup remove /dev/mapper/onapp–xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real

      For instance:

      [root@10.0.0.97]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq
      [root@10.0.0.97]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq-cow
      [root@10.0.0.97]# dmsetup remove /dev/mapper/onapp—y0jmntchnsldn8-yz2zvp2xygmg72-real

      However, never run dmsetup remove on a non-real or non-backup device. This will lead to data loss for the virtual machine.

[Looking for assistance? Give Bobcares a try.]

Conclusion

In short, we learned how to resolve OnApp resize disk errors. With Bobcares by your side, you can leave this to the experts.