Tired of getting OnApp resize disk error? The Support Engineers at Bobcares have an easy fix for this issue.
One of our customers recently faced errors when they tries to resize the virtual machine disk on LVM storage, Find out how our OnApp Hosting Support Services can help you with all your queries, big and small.
How to deal with OnApp resize disk error
The following error is an indication of OnApp resize disk error:
Error running lvchange: LV onapp-e4jsf92mjh6wk7/e7salolw28scol has open snapshot backup-hsak2e90z2m2jt: not deactivating\n
The error is seen in most OnApp cloud versions that use LVM datastore.
According to our Support Team, first, log on to the server listed in the failed log. Then run lvdisplay command as seen below to find the specific VM disk on the datastore.
lvdisplay /dev/onapp-e4jsf92mjh6wk7/e7salolw28scol -
The server IP is present in the call if you are looking at storageAPI error. Additionally, you will find the disk identifier and the datastore in the URL: Datastore/<datastore identifier>Vdisk/<diskidentifier>
For instance, here is an LV snapshot status:
Next, we will run lvremove in order to remove the snapshot. For instance:
[[email protected] ~]# lvremove /dev/onapp-e4jsf92mjh6wk7/backup-hsak2e90z2m2jt Can't remove open logical volume "backup-hsak2e90z2m2jt"
If you wind up with this error, track down the process that is responsible for holding the snapshot open. Or, in case the snapshot is mounted or mapped, release them as seen below:
[[email protected] ~]# lsof /dev/onapp-e4jsf92mjh6wk7/backup-hsak2e90z2m2jt [[email protected] ~]# mount
After this, try lvremove again, and you will see something similar to this:
Do you really want to remove active logical volume backup-hsak2e90z2m2jt? [y/n]: y. Logical volume "backup-hsak2e90z2m2jt" successfully removed
If you see an error about -real device, but cannot locate the snapshot. We recommend examining the dmsetup tree.
OnApp resize disk error: Track the process preventing dmentry from being removed
- First, SSH into the hypervisor shown in the error log.
- Next, run the following command to examine the device-mapper heirarchy for the backup responsible for the -real snapshot in the error log:
– Run: dmsetup ls –tree | grep -B5 onapp–xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real
We would like to point out that you may have to increase -B swtich value to a value higher than 5 in case the root backup device is not visible. For instance:
[[email protected] ~]# dmsetup ls --tree | grep -B5 onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq (253:12) |-onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq-cow (253:21) | `-mpath2 (253:6) | |- (8:64) | `- (8:32) `-onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real (253:20)
- After that, we have to find out which process is utilizing the top dm entry in the heirarchy with lsof.
– onapp–xxxxxxxxxxxxxx-backup–yyyyyyyyyyyyyy – Run: lsof /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy
[[email protected]]# lsof /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME tgtd 5991 root 15u BLK 253,12 0t0 331375079 /dev/mapper/onapp—y0jmntchnsldn8-backup--c00fxwrzcy1jhq
However, if there isn’t a backup server, lsof will not show anything. We recommend verifying the backup is mounted with “/proc/mounts/”. In case it is mounted, unmount it and then clear the dmsetup entries.
- Verify if any iSCSI targets are active with the following command since the backup is used by tgtd.
– Run: tgtadm –lld iscsi –mode target –op show
[[email protected]]# tgtadm --lld iscsi --mode target --op show Target 1: iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 644245 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq Backing store flags: Account information: ACL information: 10.0.50.99
- Next, scrutinize the backup server for any related ISCSI sessions. Then log out of them.
- First log in to the backup server in the “ACL information” and run iscsiadm -m session.
- Then, compare the target name from the earlier step. After that, log out of the ISCSI sessions with iscsiadm -m node -u -T <target name>.
- In the absence of any more corresponding sessions, skip to the next step.
[[email protected]]# iscsiadm -m session tcp:  X.X.X.X:XXXX,1 iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq [[email protected]]# iscsiadm -m node -u -T iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq
Our Engineers would like to point out that in case the iscsiadm command does not include -T after -u, you will wind up logging out of all iscsi targets.
- After that, delete the iSCSI target. Return to hypervisor and run the following command:
tgtadm –lld iscsi –op delete –mode target –tid *
Here, tid represents Target ID. It is displayed when the command in step 4 is run. For instance:
[[email protected]]# tgtadm --lld iscsi --op delete --mode target --tid 1
- Finally, run the dm entries as well as -real snapshot. Run this for entries that have X1 at the end:
kpartx -d -p X /dev/mapper/backup-XXXXXXXXXX
– Run: dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy – Run: dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy-cow – Run: dmsetup remove /dev/mapper/onapp–xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real
[[email protected]]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq [[email protected]]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq-cow [[email protected]]# dmsetup remove /dev/mapper/onapp—y0jmntchnsldn8-yz2zvp2xygmg72-real
However, never run dmsetup remove on a non-real or non-backup device. This will lead to data loss for the virtual machine.
[Looking for assistance? Give Bobcares a try.]
In short, we learned how to resolve OnApp resize disk errors. With Bobcares by your side, you can leave this to the experts.