A guide in updating Brocade SAN switch firmwares

I would like to share some of my personal best-practices in upgrading firmwares on Brocade SAN switches and/or directors. This blog post is divided in several sections:

  • Pre-upgrade tasks;
  • Upgrading the SAN switch;
  • Post-upgrade tasks;
  • Alternative FTP server;
  • Putty log of a SAN switch upgrade.

First of all, it’s important to mention every Brocade SAN switch has two firmware partitions. The Fabric OS (also referred to as “FOS“) is booted from the active partition, whereas the secondary partition is used to provide the ability to perform a non-disruptive firmware upgrade or as a fallback mechanism in case the firmware on the primary partition is damaged.

SANSWITCH:admin> firmwareshow
Appl Primary/Secondary Versions
FOS v7.0.0c
FOS v7.0.0c

The firmware on the SAN switch can be upgraded disruptively or, non-disruptively which will take some more time. When you are upgrading SAN switch components in a live production environment, it’s highly advisable to use the non-disruptive approach.

The firmware upgrade path can be collected from the “Brocade Fabric OS vA.B.CD Release Notes“. In general, we can say non-disruptive upgrade is supported from the previous version (identified as “B” in the version information). For example: v7.1.2b > v7.2.1c > v7.3.1d > v7.4.1.

A disruptive upgrade is supported within the same version (identified as “A” in the version information, for example: v7.2.x to v7.4.1). Refer to the release notes to confirm the firmware migration path as this can differ from version to version.

Overall, we can say the firmware update takes about 20 minutes. When upgrading two fabrics consisting out of 2 switches per fabric running on version 7.1.2b, we need to perform 3 updates to reach version 7.4.1. Bringing the total number of updates to 12 for both fabrics. It’s unnecessary to say this will take a significant time (estimated 240 minutes or 4 hours).

There are some possibilities to speed some things up!

  • Update SAN switches from the inside out. Start with the SAN switches in the “core” of the fabric, and finish with the switches on the “edge” of the fabric.
  • As a general best-practice, don’t upgrade SAN switches over both fabrics at the same time. I usually work on a version-by-version fabric-by-fabric basis to limit the fabric convergence to one specific fabric, allowing data communication to occur on the “standby” fabric.
  • Initiate the second firmware update once the initial switch is performing an “HA reboot“, this will avoid multiple SAN switches to reboot within the same fabric as the same time!
  • Prepare everything upfront! Download all the firmware files and store them on the FTP you will use to perform the upgrades. Make sure the SAN switch is able to connect to the FTP server over the network. Create an overall planning, allowing you to easily follow-up on the various tasks during the update(s).
  • The first firmware in the migration path can be installed on the secondary partition without rebooting the switch. This shortens the intervention with 15 minutes per switch. This is achieved by executing the command: “firmwaredownload” with the operand “-r” followed by an “hareboot” and a “firmwarecommit” during the maintenance time window.

As a general recommendation, I would advise to schedule this intervention in a low-IO time-window (outside production hours). A downtime for the systems is not required as the multi-path software should fail-over storage paths without interrupting the applications.

PREPARATION TASKS:

  1. Get a view on the SAN topology and it’s connected devices. The easiest way is to use Brocade SAN Health or HP SAN Visibility.
  2. Verify the firmware compatibility between SAN switch firmware, host bus adapters, storage firmware, etc. For HP material, please refer to HP Single Point Of Connectivity Knowledge (SPOCK). Refer to the vendor documentation for any remaining information.
  3. Download the firmware files in advance. The firmware files can be rather big (up to 1GB a piece) and take some time to be downloaded.
  4. Update server components (HBA cards, MPIO, etc) upfront.
  5. Verify if all systems are redundantly connected/configured on the fiber channel fabrics. Do not conclude if the system is redundantly connected, all paths to the various storage components are available. My own experiences taught me storage paths can be unavailable even though it should not be the case.
  6. Schedule a blackout of the backup environment during the firmware updates. Tape drives do not support MPIO and power-cycling the switches tend to give hardware errors in the backup application.
  7. Validate the health state of the SAN switches by executing the command: “switchstatusshow“. In case something is faulty, the issue can be resolved before the maintenance.

SANSWITCH:admin> switchstatusshow
Switch Health Report Report time: 12/17/2015 08:50:16 AM
Switch Name: SANSWITCH
IP address: 192.168.101.220
SwitchState: HEALTHY
Duration: 14016:36

Power supplies monitor HEALTHY
Temperatures monitor HEALTHY
Fans monitor HEALTHY
Flash monitor HEALTHY
Marginal ports monitor HEALTHY
Faulty ports monitor HEALTHY
Missing SFPs monitor HEALTHY
Error ports monitor HEALTHY

PRE-UPGRADE TASKS:

In order to prepare the intervention, the following steps should be executed:

  1. Before updating the firmware, it’s advised to reboot the SAN switch to release any pending commands and/or locks. The switch can be easily rebooted by executing the command “hareboot” and takes a few minutes. A switch reboot is transparent for redundantly configured servers.
  2. Collect a supportsave as a baseline (command: “supportsave” and follow the guidance outlined in the interactive prompt). This will allow reverse analysis in case of any issues without wasting time.
  3. Save the configuration of the switch to an FTP site by executing the command: “configdownload” and follow the steps in the interactive prompt.
  4. Verify the health state of the SAN switch before updating the SAN switch firmware. This can be performed in the GUI or by executing the CLI-command: “switchstatusshow“.
  5. Take a closer look the switch event logs by executing the command: “errshow” or “errdump“.
  6. Clear the counters on the fiber channel ports by executing the command “statsclear” or the command “portstatsclear -i <start-to-end-index>“.

UPGRADING THE SAN SWITCH FIRMWARE:

First of all, I would advise to use the non-disruptive approach to update the SAN switch firmware. Start with a SAN switch in the core of the fabric, once the system is performing an HA reboot, continue with another switch in the core or continue with an edge switch. The best way to address this is to work from the inside out. Once fabric “A” has been updated with version “X“, update the switches in fabric “B” with version “X“, before installing version “Y” on the switches in fabric “A“.

  1. Verify the health state of the SAN switch by using the command “switchstatusshow“.
  2. The firmware installation can be performed by using the GUI which is pretty straightforward, but does not provide any decent follow-up.  An alternative is to use the commandline. The command to be executed is “firmwaredownload” and provides a set of operands, such as “-s” or “-r” which allows the firmware to be downloaded to the secondary partition without a system reboot. In case you want to use the interactive prompt, execute the command: “firmwaredownload” and follow the steps in the wizard.
  3. Once the “firmwaredownload” has been started without any additional operands, the system performs an “hareboot” automatically, followed with a “firmwarecommit“. The status of the firmware installation process, can be followed by executing the command “firmwaredownloadstatus” once the switch has been rebooted.
  4. After switch reboot, execute the three steps listed above on the remaining switches in the fabrics (one-by-one).

POST-UPGRADE ACTIONS:

Once the firmware has been installed on the system, execute the following steps to collect a baseline after firmware installation on the Fiber-channel Switches:

  1. Clear the counters on the fiber channel ports by executing the command “statsclear” or the command “portstatsclear -i <start-to-end-index>“.
  2. Take a closer look the switch event logs for any errors by executing the command: “errshow” or “errdump“.
  3. Verify the health state of the SAN switch by using the command “switchstatusshow“.
  4. Collect a supportsave as a baseline (command: “supportsave” and follow the guidance outlined in the interactive prompt). This will allow reverse analysis in case of any issues without wasting time.

Alternative for FTP SERVER:

In case no FTP server is available within the network, or you work as an IT consultant and you have to do this often, it can be a lot easier to install FileZilla FTP server on your laptop.

Additionally, some switches allow to update the firmwares by using an USB device plugged into the SAN switch. This can be useful when no FTP server is available, but requires physical access.

PUTTY LOG OF A SAN SWITCH UPGRADE:

This section provides some additional information on how to upgrade a Brocade SAN switch from the commandline. The switch being upgraded is an HP StorageWorks 4/8 SAN switch (type: 34.0) from version v6.1.0d to v6.2.2e. It’s a rather old switch with a rather old firmware which was still lying around at my place.

STEP 1: rebooting the SAN switch.

SANSWITCH:admin> hareboot
HA Rebooting …
SANSWITCH:admin>

STEP 2: collecting a supportsave.

SANSWITCH:admin> supportsave
This command will collect RASLOG, TRACE, supportShow, core file, FFDC data and other support information and then transfer them to a FTP/SCP server or a USB device. This operation can take several minutes.
NOTE: supportSave will transfer existing trace dump file first, then automatically generate and transfer latest one. There will be two trace dump files transfered after this command.
OK to proceed? (yes, y, no, n): [no] y

Host IP or Host Name: 192.168.0.121
User Name: cfg
Password:
Protocol (ftp or scp): ftp
Remote Directory: .

Saving support information for switch:SANSWITCH, module:CONSOLE0……tSave_files/SANSWITCH-S0-201601081536-CONSOLE0.gz: 535.00 B 9.45 kB/s

Saving support information for switch:SANSWITCH, module:SSAVELOG…
…tSave_files/SANSWITCH-S0-201601081545-sslog.ss.gz: 1.87 kB 93.85 kB/s
SupportSave completed
SANSWITCH:admin>

STEP 3: saving the switch configuration to the FTP server as a fall-back mechanism.

SANSWITCH:admin> configupload
Protocol (scp, ftp, local) [ftp]: ftp
Server Name or IP Address [host]: 192.168.0.121
User Name [user]: cfg
File Name [config.txt]: SANSWITCH-CFG.txt
Password:

configUpload complete: All config parameters are uploaded
SANSWITCH:admin>

STEP 4: verifying the switch health state prior to new firmware installation.

SANSWITCH:admin> switchstatusshow
Switch Health Report Report time: 01/08/2016 03:57:32 PM
Switch Name: SANSWITCH
IP address: 192.168.0.241
SwitchState: HEALTHY
Duration: 00:23

Power supplies monitor HEALTHY
Temperatures monitor HEALTHY
Fans monitor HEALTHY
Flash monitor HEALTHY
Marginal ports monitor HEALTHY
Faulty ports monitor HEALTHY
Missing SFPs monitor HEALTHY
Fabric Watch is not licensed
Detailed port information is not included
SANSWITCH:admin>

STEP 5: consulting the switch logs.

SANSWITCH:admin> errdump
2016/01/08-15:34:11, [FSSM-1002], 232,, INFO, SilkWorm200E, HA State is in sync.
2016/01/08-15:34:11, [FSSM-1003], 233,, WARNING, SilkWorm200E, HA State out of sync.
2016/01/08-15:35:10, [SEC-1203], 234,, INFO, SANSWITCH, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 192.168.0.121
2016/01/08-15:46:18, [SS-1000], 235,, INFO, SilkWorm200E, supportSave has uploaded support information to the host with IP address 192.168.0.121.
2016/01/08-15:54:47, [CONF-1001], 236,, INFO, SANSWITCH, configUpload completed successfully. All config parameters are uploaded.
SANSWITCH:admin>

STEP 6: clearing the port statistics. All error counters are resetted to “zero“.

SANSWITCH:admin> statsclear
SANSWITCH:admin> porterrshow

STEP 7: installing the newer firmware revision on the SAN switch.

SANSWITCH:admin> firmwaredownload
Server Name or IP Address: 192.168.0.121
User Name: flash
File Name: .
Network Protocol(1-auto-select, 2-FTP, 3-SCP) [1]: 2
Password: *****
Checking system settings for firmwaredownload…
Server IP: 192.168.0.121, Protocol IPv4
System settings check passed.

You can run firmwaredownloadstatus to get the status of this command.

This command will cause a warm/non-disruptive boot on the switch, but will require that existing telnet, secure telnet or SSH sessions be restarted.

Do you want to continue [Y]: Y
Firmware is being downloaded to the switch. This step may take up to 30 minutes.
Preparing for firmwaredownload…
Removing fabos-drivers
Start to install packages…
dir ##################################################
ldconfig ##################################################

STEP 8: consulting the firmware revision on both partitions. During the upgrade process, you will notice the secondary partition will transition from the old version (v6.1.0d) into an “Unknown” state, before it will display the newer software revision (v6.2.2e).

SANSWITCH:admin> firmwareshow
Appl Primary/Secondary Versions
FOS v6.2.2e
FOS unknown

STEP 9: verify the firmware installation status by executing the command “firmwaredownloadstatus“. In the log, you will notice the upgrade on the primary partition first, the hareboot, the commit of the firmware on the secondary partition.

SANSWITCH:admin> firmwaredownloadstatus
[1]: Fri Jan 8 16:16:50 2016
Firmware is being downloaded to the switch. This step may take up to 30 minutes.

[2]: Fri Jan 8 16:22:49 2016
Firmware has been downloaded to the secondary partition of the switch.

[3]: Fri Jan 8 16:24:54 2016
The firmware commit operation has started. This may take up to 10 minutes.

[4]: Fri Jan 8 16:30:05 2016
The commit operation has completed successfully.

[5]: Fri Jan 8 16:30:05 2016
Firmwaredownload command has completed successfully. Use firmwareshow to verify the firmware versions.

STEP 10: once the switch has been updated to the latest version, execute the following commands again to close the intervention:

  • SANSWITCH:admin> statsclear
  • SANSWITCH:admin> switchstatusshow
  • SANSWITCH:admin> supportsave

Hope this helps!

Ruben

Leave a Reply