Direct Updating of Modern Dell Server and Peripheral Firmware on XenServer 7.0

by Tobias Kreidl, CTP, Steering Committee

There are various ways to update the BIOS and firmware on Dell servers, such as with the Dell Repository Manager (https://www.vmadmin.co.uk/other/332-dellbiosfirmwareupdaterepomanager) and even without needing Windows as a means of booting up and performing the updates by creating a bootable ISO, e.g. http://www.dell.com/Support/Article/us/en/04/SLN296511 or other means, such as listed in https://wiki.ubuntu.com/DellBIOS. For XenServer, these procedures require extra work and in some cases, multiple reboots. You really don’t want to incur any more downtime than necessary nor do you want to create problems by making changes on dom0 that could cause issues later with support or functionality.

One procedure to streamline this was mentioned a number of years ago in the very nice article by Neil Spellings, CTP, at http://neil.spellings.net/2012/03/03/updating-dell-firmware-from-within-xenserver-dom0/ which entails creating a supplementary yum repository that can be used to leverage the Dell Linux Repository (DLR) utilities.

Unfortunately, this has issues under XenServer 7.0, which jumped from being based on CentOS 5 to CentOS 7. The way yum repositories are now set up makes it harder (deliberately, of course, as Citrix doesn’t want people creating problems with foreign installs) to download or update “foreign” RPMs via yum. One such workaround was posted at http://discussions.citrix.com/topic/378880-xenserver-70-repositories/ but this doesn’t work as it stands any more, as for one, the release version is now different (and even making it a generic “7” which is then pointed to by the latest release isn’t all that’s necessary). It also calls for an additional xfsprogs utility to be installed.

The DSU Utility

One option under newer installations of Red Hat or SUSE is in lieu of DLR to instead use the DSU (Dell System Update) utility. This is a more flexible, newer option which is unfortunately currently limited to only the following operating systems:

  • Red Hat Enterprise Linux 6.8 (x86_64)
  • Red Hat Enterprise Linux 7.2(x86_64)
  • SUSE Linux Enterprise Server 11SP4 (x86_64)
  • SUSE Linux Enterprise Server 12SP1 (x86_64)

That means of course that at present, only XenServer 7.0 falls into that category and also from my tests, is only compatible (at least for BIOS updates) with PERC 7 and higher controllers (7, 8. 9 and including the H700 and H800 series, as well as potentially some of the more recent H300 series), so likely just Dell 11th generation and higher servers. Even though the package can be made to install correctly on older servers, you get an error message indicting the DSU software is not compatible with the controller. I’m not sure about the H310, but it and the H330 are new enough to possibly also be compatible. Details on Dell server controllers can be found here: http://www.dell.com/support/article/us/en/19/SLN292279/.

The DSU utility (aptly named dsu) has certain other advantages over the Dell Linux Repository (DLR) utility. First off, you can select individual updates to apply. It will also show you the current and latest versions of what’s installed and what is available. Under certain circumstances, packages can even be reverted (though I didn’t encounter any in my investigation that were). Instead of being built on the Open Build System, the packages are based on Dell Update Packages (DUP). It is even possible to create selected packages bundled into a bootable ISO or a directory.

Setting up the repository on standard Linux boxes with the operating system listed above should be as simple as running the commands:

#wget -q -O – https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi | bash

#yum install dell-system-update

Unfortunately, this is not so simple given the XenServer 7.0 environment and how even enabling base and mirror collections isn’t always successful. Doing so can be quite messy and the thought here is to be as nonintrusive as possible.

I have worked out a way to accomplish this quickly and easily with minimal changes to dom0. The goal is to keep the server from getting “polluted” with unnecessary changes and prevent any interference with the intended functionality of XenServer. Here are the steps:

1) Create the file /etc/yum.repos.d/dell-system-update.repo and insert the contents:

[dell-system-update_independent]

name=dell-system-update_independent
baseurl=http://linux.dell.com/repo/hardware/dsu/os_independent/
gpgcheck=1
gpgkey=http://linux.dell.com/repo/hardware/dsu/public.key
enabled=1
exclude=dell-system-update*.i386

[dell-system-update_dependent]

name=dell-system-update_dependent
mirrorlist=http://linux.dell.com/repo/hardware/dsu/mirrors.cgi?osname=el$releasever&basearch=$basearch&native=1
gpgcheck=1
gpgkey=http://linux.dell.com/repo/hardware/dsu/public.key
enabled=1

(Note that if need be, the wget and yum commands can be run on a different server and the contents of the resulting repository copied and pasted into the file on the target XenServer, which would also be a good way to check for possible updates in the future. Also, should there be an issue resolving linux.dell.com, try substituting the IP address 143.166.82.43 for it in the four occurrences above.)

2) Run the command “yum install dell-system-update” and respond “y” to the various questions.

3) If necessary, in particular if you have private network, firewalls or border router limitations, you may need to make use of a yum proxy server. Assuming there is such machine available and is accessible, on your XenServer, edit the file /etc/yum.conf and enter at the end of the file the one line:

proxy=http://IP-address-of-proxy-host:3128

for example, proxy=http://10.11.12.13:3128 ,where “3128” is the standard squid Web proxy port used by yum (change the port number to match your site’s requirements, if needed).

That’s it. You should now have access to the “dsu” utility. The various options can be seen by running “dsu –help”. More about dsu and its options can be found here: https://linux.dell.com/repo/hardware/dsu/ . While the configuration of a yum proxy server is beyond the scope of this article, there are a number of good guides describing how to create reverse proxies, for example, this one for a squid proxy server: https://www.cyberciti.biz/tips/howto-rhel-centos-fedora-squid-installation-configuration.html. Depending on the environment, this last step may not be required at all.

An Example

Here is an example of the output when running the “inventory” option. Each of the individual components is identified and numbered and shown along with the current firmware level:


# dsu –inventory

Verifying catalog installation …

Installing catalog from repository …

Fetching dsucatalog …

Reading the catalog …

Installing inventory collector …

Fetching invcol_WF06C_LN64_16.12.200.896_A00 …

Verifying inventory collector installation …

Getting System Inventory …


1. BIOS  ( Version : 2.5.2 )

2. Lifecycle Controller, 1.4.2.12, A00  ( Version : 1.4.2.12 )

3. Dell 64 Bit uEFI Diagnostics, version 4247A0 Installer Revision 14.05.00, 4247.1  ( Version : 4247A0 )

4. Power Supply  ( Version : 09.13.59 )

5. Power Supply  ( Version : 09.13.59 )

6. CPLD  ( Version : 1.0.3 )

7. PERC H710 Mini Controller 0 Firmware  ( Version : 21.3.0-0009 )

8. Firmware for  – Disk 0 in Backplane 1 of PERC H710 Mini Controller 0    ( Version : DB08 )

9. 12G SEP Firmware   ( Version : 1.00 )

10.  iDRAC  ( Version : 1.66.65.00 )

11. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth0)  ( Version : 7.10.18 )

12. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth1)  ( Version : 7.10.18 )

13. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth2)  ( Version : 7.10.18 )

14. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth3)  ( Version : 7.10.18 )

15. Intel(R) Ethernet Converged Network Adapter X540-T2  ( Version : 0.0.0 )

16. Intel(R) Ethernet Converged Network Adapter X540-T2  ( Version : 0.0.0 )

Here is a list of dsu options available:
# dsu –help
Usage: dsu [OPTION]…
Options:
-h, –help                                     Help
-v, –version                                  Version
-i, –inventory                                Perform Inventory
-n, –non-interactive                          Non-interactive mode
-g, –get-categories                           Get category values
-c, –category=<CATEGORY1,CATEGORY2,…>       List only updates of <CATEGORY1,…> categories
-u, –apply-upgrades-only                      List only upgradable updates
-d, –apply-downgrades-only                    List only downgradable updates
-e, –apply-equivalent-updates                 Enable equivalent updates
-l, –update-list=<FILENAME1,FILENAME2,…>    Apply <FILENAME1,…> updates
    –config=                            Configuration file
    –source-type=                       TYPE=PDK
    –source=                            Source location
    –destination-type=                  TYPE=ISO|CBD
    –destination-location=               Output location when destination-type is provided
    –bootable-log-location=             File to write logs during server update
    –ic-location=                       Inventory Collector binary
    –input-inventory-file=              Inventory XML file
    –output-inventory-xml=              Path to save the Inventory XML file
    –preview                                  Preview updates

Typically, you would run dsu with option “a” (select all) and then “c” (commit).

Below is the appearance of a different machine after an update was performed. Afterwards, the output below from the plain command “dsu” shows most are now at the highest rev level (the exception in this example is the iDRAC):

# dsu
Verifying catalog installation …
Installing catalog from repository …
Fetching dsucatalog …
Reading the catalog …
Verifying inventory collector installation …
Getting System Inventory …
Determining Applicable Updates …

|———–Dell System Updates———–|
[ ] represents ‘not selected’
[*] represents ‘selected’
[-] represents ‘Component already at repository version (can be selected only if -e option is used)’
Choose:  q – Quit without update, c to Commit, – To Select/Deselect, a – Select All, n – Select None

[-]1 BIOS

Current Version : 6.4.0 same as : 6.4.0

[-]2 Dell 32 Bit Diagnostics, version 5162 Installer Revision 14.05.00, 5162A0, 5162.1

Current Version : 5162A0 same as : 5162A0

[*]3 iDRAC6

Current Version : 1.97 Upgrade to : 2.85

[-]4 Firmware for – Disk 0 in Backplane 0 of PERC 6/i Integrated Controller 0

Current Version : HS11 same as : HS11

[-]5 PERC 6/i Integrated Controller 0 Firmware

Current Version : 6.3.3.0002 same as : 6.3.3-0002

[-]6 Dell LifeCycle Controller v1.7.5, 1.7.5.4, A00

Current Version : 1.7.5.4 same as : 1.7.5.4

[-]7 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth3)

Current Version : 08.07.26 same as : 08.07.26

[-]8 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth2)

Current Version : 08.07.26 same as : 08.07.26

[-]9 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth9)

Current Version : 08.07.26 same as : 08.07.26

[-]10 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth1)

Current Version : 08.07.26 same as : 08.07.26

[-]11 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth8)

Current Version : 08.07.26 same as : 08.07.26

[-]12 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth0)

Current Version : 08.07.26 same as : 08.07.26

[-]13 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth7)

Current Version : 08.07.26 same as : 08.07.26

[-]14 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth6)

Current Version : 08.07.26 same as : 08.07.26

[-]15 Firmware for – Disk 2 in Backplane 0 of PERC 6/i Integrated Controller 0

Current Version : D598 same as : D598

Enter your choice : 3

[*]3 iDRAC6

Current Version : 1.97 Upgrade to : 2.85

Enter your choice: c

(It turns out it’s not compatible, since this is a PERC 6 controller, and hence it stays as the level it already is. On servers with newer controllers, it would have allowed you to proceed.)

Finally, here is an example of performing a successful update of an individual component. The inventory report showed the following for item 15:


[ ]15 Dell Lifecycle Controller, 1.6.5.12, A00
 Current Version : 1.6.5.12 Upgrade to : 1.7.5.4

To perform the update you would enter 15 to select that item,which then appears with an asterisk, indicating it has been selected:

[*]15 Dell Lifecycle Controller, 1.6.5.12, A00
 Current Version : 1.6.5.12 Upgrade to : 1.7.5.4

To then apply the update to this item, enter option “c”. The results are as follows:

Fetching Lifecycle-Controller_Legacy_Application_0WFGM_LN_1.7.5.4_A00 …
Installing Lifecycle-Controller_Legacy_Application_0WFGM_LN_1.7.5.4_A00 …
Collecting inventory…
…………………………………..
Running validation…

Dell Lifecycle Controller, 1.6.5.12, A00

The version of this Update Package is newer than the currently installed version.
Software application name: Dell Lifecycle Controller, 1.6.5.12, A00
Package version: 1.7.5.4
Installed version: 1.6.5.12

Executing update…
WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE UPDATE IS IN PROGRESS.
THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!
………………………………………………………………………………………………………………………..
………………………………………………………………………………………………………………………..
………………………………………………………………………………………………………………………..
………………………………………………………………………………………………………………………..
…………………………………

The update completed successfully.


Done! Please run ‘dsu –inventory’ to check the inventory

(Note that a reboot should be performed first or in one case I encountered, the inventory command just hung. I found several instances in which a cold boot was necessary.)

Conclusions

As with BIOS/firmware updates, a reboot at the end is necessary. A nice aspect of DSU is that the updates can be applied while the host is running and the reboot can be scheduled to take place at any convenient time. The DSU process is simple and flexible. It is, unfortunately, not functional currently with any XenServer version prior to 7.0, partly because of a library incompatibility with libcurl.so, and as mentioned above, Dell only officially supports DSU on RHEL 6.8 and 7.2, and given that XenServer 6.5 SP1 is based on CentOS 5, the incompatibility is not surprising.

Looking forward to XenServer 7.1 and beyond, DSU can provide the desired functionality and versatility to perform firmware updates and above all, with a more pronounced degree of simplicity compared to DLR. The installation is also simplified by not having to manipulate any of the existing repositories, which can definitely be seen as a bonus as it is much less intrusive than what otherwise would be necessary.

Disclaimer: Citrix frowns upon modifications to, as well as the installation of, additional packages onto XenServer. Do so at your own risk, knowing that such actions might void any service contracts and support.

Leave a Reply