What Right Looks Like: Part 2 – Unadulterated Nerdery

Sometimes the gear we have is just a bit intimidating. Every Marine with an 065X MOS was taught at some point one form of a Cisco and/or Microsoft class, so it is not as prevalent with those technologies, but the NetApp (storage) seems to be that piece of gear that everyone either fears, or just plain does not understand.

That’s not what right looks like! Whether you are using the FAS-2040 in the DDS-M DSM or the FAS-2050 in the TCWS, you’ve got two controllers. This is a huge capability when you employ them in HA (High Availability) mode. You are most likely using your NetApp to centralize where you store your VMs and possibly to host your ShareDrive. Doing this significantly increases your capabilities and the flexibility of your datacenter, but anytime you hear “centralization of resources”, it should immediately trigger the single-point of failure center of your brain. Your FAS has RAID to protect against disc failure, dual-redundant power supplies to protect against the failure of a power supply, AND dual-controllers to protect against the failure of a controller IF you have HA enabled. The Marine Corps has paid for all issued hardware to contain dual-controllers, and the requisite licenses but it is up to you to ensure the gear has been configured properly to support this configuration. You can see from the picture above that HA is not enabled. We know this because there is a picture of a single storage system in the OnCommand System Manager console screen. Aside from looking at the management software, you could tell if the system was configured for HA by looking at the cabling at the back of the FAS. If only one of the controllers has a network cable connected to it, you can’t be running in HA mode. This is normally one of the first things I look for when I am told “the gear is configured and ready to go”.

That’s what right looks like! In the top picture, not only do both controllers have network cables connected, they have redundant cabling (etherchannel). Keep in mind, it takes more than just connecting two cables to each controller. You need to configure the etherchannel (multiple physical ports configured to act as one) on both your storage system and the switch you plug your storage system into. On the second picture, we see the icon for a storage system configured for HA (actually two storage systems configured for HA, the first a FAS 2050 and the second a 2040).

Special thanks to Lt Skehan and the outstanding Marines of A Co for the pictures and screenshots!

Robert Lauterbach says:

January 23, 2015 at 8:51 pm

Another great point. Many times I have students coming to me not understanding why their filers come equipped with 2 controllers. I have received many phone calls asking what the below prompt means.

>waiting for giveback

This always means that you experienced a controller failure or one of the many other events which can kick off a controller fail over. This commonly occurs when someone powers down a filer incorrectly. The easy fix for this is to console or SSH in to the surviving controller issue the command:

>cf giveback

This will allow the ‘crashed’ controller to come back online and resume what is referred to as normal operations.

Although simple to fix this can be difficult to detect. If clustered fail-over has worked as designed you will not incur service interruptions. It will take routine checks (log monitoring) or properly setup SNMP monitoring to identify when a fail-over has occurred.

John says:

January 23, 2015 at 10:11 pm

Thanks for the comment. Whenever I have the Marines turn on HA, I like to have them failover to the other controller and recover. It is a really easy process, and as you mentioned will save you when it counts.

Leave a Reply Cancel reply