Cannot connect to the Docker daemon. Is the docker daemon running on this host?

A user raised a ticket to increase /var filesystem because he needed more space for docker images. We normally don’t add space to /var as it is considered a system filesystem. If application needs to write to a directory under /var, we create a new filesystem mounted at for example /var/application-dir.

As docker store images etc. under /var/lib/docker, we created a new filesystem mounted at /var/lib/docker. The user had no images to migrate, therefore we just cleared /var/lib/docker before mounting the new filesystem. After this, docker did not work. systemctl status docker shows docker is up and loaded but running any “docker command” ends up in:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

We reinstalled docker a few times without success. In the midst of troubleshooting, I killed all processes except my login shell by running killall5. At which point I have no choice apart from reboot. The issue went away after reboot.


killall vs killall5

I learned this the hard way. In Linux, killall kills process by name whereas in Unix System V, killall does what is it is called- kill all processes. killall5 is the SystemV killall command in Linux. If you run killall5 in Linux, it will kill all processes except kernel threads(those surrounded by square bracket in ps aux output) and your login shell.

man killall5: Continue reading

ESXi hosts cannot see LUNs from NetApp storage

Newly built ESXi 6 hosts will not detect LUNs zoned to them. The storage is NetAPP FAS8000 series FC. After multiple scanning and refreshing, nothing changed. The storage guys said they got everything right at their end.

I found this VMware KB which says RSCN event be enabled on the switch in order for the host to detect LUNs after a zone change without reboot. I shared it with the Storage guys and they agreed to check it later.

Meanwhile, I rebooted the hosts after which all hosts detected the LUNs. Additional LUNs were added and detected after a rescan. The storage guys said they didn’t change anything at their end though.

Add second VMKernel interface to multiple ESXi hosts

I have to add a VMKernel interface for NAS traffic to a series of hosts that I just built. I am going to call the portgroup NAS as that is the purpose. The portgroup will be in vSwitch0 along with the MGMT network.

These are the commands I need

esxcli network vswitch standard portgroup add -p NAS -v vSwitch0
esxcfg-vmknic -a -i a.b.c.d -n -p NAS
esxcli network ip interface  set -i vmk1 -m 9000    

But I need to run this through a series of hosts. I place the hostnames and IPs in a file in this format. Continue reading

Configure hostname for multiple ESXi hosts

I build a bunch of new ESXi hosts and this is how I configure hostname for each host.

First, I have the list of the hosts’ FQDN in a file

$ cat list

I use cat and awk to generate an ssh command for each host. This assumes you have ssh keys configured. Continue reading