Problem:
Unable to delete an inaccessible datastore
I was switching my storage array, so I migrated the VMs from that old datastore/storage to a new datastore/storage.
The old datastore was shared by 3 ESXi 6.7 hosts, with no cluster. After migrating the VMs and unmount/delete the datastore, it was still presented in two of the ESXi hosts and was marked as Inaccessible. I am pretty sure that I did something wrong, but don’t know what.
Attempting a few times to delete the datastore keep popping up the error below:
I tried and check:
- There was no VM or template allocated in the old datastore.
- There was no files, snapshots or templates. In fact, the datastores show 0 B of capacity.
- Storage I/O Control (SIOC) wasn’t enabled on the datastore.
- Access via SSH to delete the datastore from there, but these two datastores were not listed.
- This wasn’t an Orphaned Datastore, the Delete Datastore option wasn’t grayed out, it just didn’t work.
- I tried more options. See resources.
Solution:
Yet another "simple" solution. Restart management agents in the ESXi host, or restart the ESXi host itself.
To restart management agents on the ESXi 6.7 login into SSH as root and run the command below:
services.sh restart
You can also restart the management agents from the ESXi host console.
My major problem was that one of these two ESXi hosts was hosting the vCenter Server and all other servers of the infrastructure. This is not suitable nor friendly for productions environments but was the only solution for me.
Besides users being more careful when removing datastores, I hope VMware had another solution to solve this kind of issue.
Resources:
https://kb.vmware.com/s/article/2004605
https://communities.vmware.com/thread/571912
https://communities.vmware.com/message/2662944#2662944
https://community.spiceworks.com/topic/1972169-remove-inactive-datastore-from-esxi
worked – thanks!
Nice! I was going to just reboot my VCSA, that wouldn't have done anything for me though.. luckily I googled, otherwise I woulda had a few minutes of confusion..
What's awkward here is that: Previously I was told to always unmount then delete datastores, then remove the mapping from my SAN, then delete the volume on the SAN- which always worked in 5.5 and in 6.5..
But when following this in 6.7, the datastores reappear after about 20 seconds as inaccessible as if I only deleted them from the SAN.
You don't need to run services.sh restart, I was able to clear out mine by running:
/etc/init.d/hostd restart
followed by
/etc/init.d/vpxa restart
-I'm pretty sure this leaves a couple of the management services alone, your hosts will still drop from vCenter momentarily, but it will not impact running VMs.
For me it was simpler. Select the datastore and go to “VMs”, do VM’s have an ISO mounted on this datastore? If so remove it from the VM settings (Change CD/DVD drive to “client device”). Still have the problem? VM still listed? Check if there are any snapshots and delete them (Old vm settings will still have the old datastore mounted). That should remove all the VMs. However now check the “Templates” tab, any templates listed? If so you will need to do the same on the template. For me, I couldn’t convert the template to a VM as the template was in a contradicted state as it would only power on if it could get to the datastore. I had to clone the template to a new VM, and delete the old template.
Had this too. Hooked ISO's (even though the DS is inaccessable) can cause much runaround. It's become my first check when having challenges with VM's, Hosts and Storage. Nice one!
If anyone else comes across this, my solution was to put the hosts in Maintenance Mode, then once the running VMs are migrated off, navigate to the host > Configure tab > System > Services, then restart the VMware vCenter Agent service. Took longer to migrate the VMs off than it did to cycle the service. Once the host was back up, the datastore was no longer listed on the Datastore tab.
If anyone else come accross this issue, before you try it. its worh trying to do a rescan from datacenter level, so right click on Datacenter> Storage > Rescan Storage.
I had deleted the LUN from SAN before unmounting or deleting from vcenter and doing the above step removed the inaccessibable datastore from my vcenter and rest of the ESXi hosts.
@Sam you are the real MVP 😀 Did the rescan on DC level and it worked like a charm... Thank you!