The lack of jumbo frames burns our world down

NAS Datastores are greyed out in ESXi hosts. VMs becomes inaccessible, lots of them. One host in the cluster was rebooted and everything looks fine except we can see what’s in the datastores. NFS MaxQueueDepth is set to 64 pretty much in all hosts. This cluster has it set to the Max value

~# esxcfg-advcfg -g /NFS/MaxQueueDepth
Value of MaxQueueDepth is 4294967295

We change it to the recommended value of 64 and rebooted the host. We are still stuck. Then came our savior. After hours of troubleshooting, it takes NetApp senior support engineer to ask us “Is Jumbo frame working”?

No. It isn’t we find. We have mtu 9000 set on the vmkernel interface for NAS but we cannot send jumbo frames traffic to he NAS filer

vmkping -I vmkX -d -s 8000 NAS_FILER_IP

The reason being the vSwitch has mtu set to 1500. So we change vSwitch to 9000 mtu and the fire is gone.

The hosts in the process of migrating to vCenter 6 from vCenter 5.x, were migrated to Standard Switch from Nexus 1KV vDS and in this process jumbo frame setting was missed on the vSwitch.

For any NAS issue, the questions have always been:

1) Is this volume exported?
2) Can you ping the NAS filer IP?

Henceforth
3) Is jumbo frame working? Can you ping the NAS filer with mtu > 1500? That is if you think you have jumbo frame enabled. If the vmk interface and vSwitch/vDS Port Profile mtu settings match, it should still work, be it 1500 or 9000.

The savior from NetApp explained-
There will be no problem mounting the datastore because small frames are used. But to access data, it will use big frames. Because of the vmkernel interface and vSwitch mtu setting mismatch, the frames could not go through.

Advertisements

2 thoughts on “The lack of jumbo frames burns our world down

  1. Pingback: vMotion failed at 20% ” The operation is not allowed in the current state .” – rtfmp

  2. Pingback: vMotion fails at 20% ” The operation is not allowed in the current state .” – rtfmp

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s