Rubrik Basics – Cluster Upgrade Process

I’ve been doing some work with Rubrik in our lab and thought it worth covering some of the basic features that I think are pretty neat. In this edition of Rubrik Basics, I thought I’d quickly cover off software upgrades. There are two ways to upgrade the Rubrik software on your Brik – via USB and SFTP. Either way, you’ll need access to the Downloads section of the support site. If you’re a customer, you’ll have this already. If this all sounds too hard, you can raise a ticket with the support team and they’ll tunnel in and do the upgrade for you (assuming you’ve allowed remote tunnel capability).

 

USB

The good thing about using a USB drive is that you can still keep appliances in “dark” sites up to date. Before you begin you’ll need to do two things:

  • Download the compressed upgrade archive and the matching signature file from the customer portal.
  • Format a removable drive with the FAT32 file system.

You’ll need to copy the upgrade file and matching signature file to the removable drive. Plug that into any node in the cluster. Log in to that node as the admin user. Mount the USB drive by typing the following command:

mount --usb_device

Type the following command to begin the upgrade:

upgrade start

The upgrade system scans the file system for upgrade archives. If multiple archives are available, it display a list of choices. Once you’ve finished, you can unmount the device.

umount --usb_device

 

SFTP

You can also run the upgrade via SFTP. I found the instructions on how to do that here. It’s not too dissimilar to the USB method. You’ll want to use your favourite SFTP client to upload the files to the /upgrade directory. Once you’ve done that, ssh on to the node and you can run a pre-flight check. If everything comes up Milhouse you’ll be good to go for the next step.

Using username "admin".

admin@10.xxx.yyy.131's password:

=======================

Welcome to Rubrik CLI

=======================

Type 'help' or '?' to list commands

RVM165Sxxxx55 >> upgrade start --mode prechecks_only
Do you want to use --share rubrik-4.1.2-2366.tar.gz [y/N] [N]: y
Upgrade status: Started pre-checks successfully
RVM165Sxxxx55 >> upgrade status
Current upgrade mode: prechecks_only
Current upgrade pre-checks node: RVM165Sxxxx55
Current upgrade pre-checks tarball name: --share rubrik-4.1.2-2366.tar.gz
Current upgrade pre-checks status: In progress
Current run started at: 2018-07-19 00:48:04.437000 UTC+0000

Current state (3/6): VERIFYING
Current task: Verify authenticity of new software
Current state progress: 0.0%

Finished states (2/6): ACQUIRING, COPYING
Pending states (3/6): UNTARING, DEPLOYING, PRECHECKING

Time taken so far: 18.38 seconds
Overall upgrade progress: 6.0%

To check on progress, run “upgrade status” to, erm, check on the status of the upgrade.

RVM165Sxxxx55 >> upgrade status
Last upgrade mode: prechecks_only
Last upgrade pre-checks node: RVM165Sxxxx55
Last upgrade pre-checks tarball name: --share rubrik-4.1.2-2366.tar.gz
Last upgrade pre-checks status: Completed successfully
Last run ended at: 2018-07-19 00:51:03.129000 UTC+0000
Current state: IDLE

Now you’re ready to do it for real. Run “upgrade start” to start.

RVM165Sxxxx55 >> upgrade start
Do you want to use --share rubrik-4.1.2-2366.tar.gz [y/N] [N]: y
Upgrade status: Started upgrade successfully
RVM165Sxxxx55 >> upgrade status
Current upgrade mode: normal
Current upgrade node: RVM165Sxxxx55
Current upgrade tarball name: --share rubrik-4.1.2-2366.tar.gz
Current upgrade status: In progress
Current run started at: 2018-07-19 00:52:56.882000 UTC+0000

Current state (4/9): UNTARING
Current task: Extract new software
Current state progress: 0.0%

Finished states (3/9): ACQUIRING, COPYING, VERIFYING
Pending states (5/9): DEPLOYING, PRECHECKING, PREPARING, UPGRADING, RESTARTING

Time taken so far: 22.52 seconds
Overall upgrade progress: 3.5%

It’s a pretty quick process, and eventually you’ll see this message.

RVM165Sxxxx55 >> upgrade status
Last upgrade mode: normal
Last upgrade node: RVM165Sxxxx55
Last upgrade tarball name: --share rubrik-4.1.2-2366.tar.gz
Last upgrade status: Completed successfully
Last run ended at: 2018-07-19 01:19:09.719000 UTC+0000

Current state: IDLE
RVM165Sxxxx55 >>

And you’re all done. Note that you only have to upload the data and run the process on one node in the cluster.

OpenMediaVault – Updating from 2.2.x to 3.0.x

I recently upgraded my home-brew NAS from OpenMediaVault 2.2.14 (Stone burner) to openmediavault 3.0.86 (Erasmus). It’s recommended that you do a fresh install but I thought I’d give the upgrade a shot as it was only a 10TB recovery if it went pear-shaped (!). They also recommend you disable all your plugins before you upgrade.

 

Apt-get all of the things

It’s an OS upgrade as well as an application upgrade. In an ssh session I ran

apt-get update && apt-get dist-upgrade && omv-update

This gets you up to date, then upgrades your distro (Debian), and then gets the necessary packages for omv. I then ran the omv upgrade.

omv-release-upgrade

This seemed to go well. I rebooted the box and could still access the shared data. Happy days. When I tried to access the web UI, however, I could enter my credentials but I couldn’t get in. I then ran

omv-firstaid

And tried to reconfigure the web interface. It kept complaining about a file not being found. So I ran

dpkg -l | grep openmediavault

This told me that there was still a legacy plugin (openmediavault-dnsmasq) running. I’d read on the forums that this might cause some problems. So I used apt-get to remove it.

apt-get remove openmediavault-dnsmasq

The next time I ran apt-get it told me there were some legacy packages present that I could remove. So I did.

apt-get autoremove dnsmasq dnsmasq-base libnetfilter-conntrack3

After that, I was able to login in to the web UI with no problems and everything now seems to be in order. When my new NAS arrives I’ll evacuate this one and rebuild it from scratch. There are a fair few changes in version 3 and it’s worth checking out. You can download the ISO image from here.

 

DNS Matters

The reason I had the dnsmasq plugin installed in the first place was that I’d been using the NAS as a DHCP / DNS server. This had been going reasonably well, but I’d heard about Pi-hole and wanted to give that a shot. That’s a story for another time, but I did notice that my OMV box hadn’t updated its /etc/resolv.conf feel correctly, despite the fact that I’d reconfigured DNS via the web GUI. If you run into this issue, just run

dpkg-reconfigure resolvconf

And you’ll find that resolv.conf is correctly updated. Incidentally, if you’re a bit old-fashioned and don’t like to run everything through DHCP reservations, you can add a second set of static host entries to dnsmasq on your pi-hole machine by following these instructions.

EMC – Next-Generation VNX – Data In Place Upgrades

Approximately 4 or 500 years ago, I spent a number of nights in various data centres around the country upgrading customers’ CLARiiON arrays from CX200s to Cx500s, CX300s to CX3-20s, and so on. The neat thing about the CLARiiON was that EMC had a pretty reasonable way of doing data in place (DIP) upgrades, including across generations if required. With the introduction of the VNX, that changed. Primarily because of the switch from FC to SAS on the back-end. And with the “Next-Generation” VNX (VNX2), you also can’t go from VNX to VNX2. Which some people have been understandably unhappy about. The procedure hasn’t changed much over the years, and you can read Rob’s post here for a pretty thorough look at what’s involved.

So why would you want to do this anyway? Especially given that, if you’re upgrading a VNX5200 for example, you’ve probably only had the array in operation for a few years. Well, requirements change, companies grow, people need more horsepower. Sometimes EMC makes it a commercially viable option to do a DIP upgrade rather than replace the array with another one. There’re are a bunch of reasons.

I don’t want to go into exactly what the steps are, as your friendly EMC service folk or partner will be able to go through that with you, but I thought it might be an idea to share a few things to know prior to launching into one of these procedures (or even making the decision to upgrade in this fashion).

The supported source systems include:

  • VNX5200;
  • VNX5400;
  • VNX5600; and
  • VNX5800.

Note that you cannot convert a VNX7600, nor can you go from VNX to VNX2 (as I mentioned before). Also, the VNX8000 can’t be a source system, because that’s already as big as the VNX goes.

Supported targets for upgrade include:

  • VNX5400;
  • VNX5600;
  • VNX5800; and
  • VNX7600.

You can’t go to a VNX8000. You can also upgrade the type of array as follows:

  • Block to block;
  • File to file; and
  • Unified to unified.

You can’t convert from a block system to a higher performing unified system. You can, however, do a block conversion, and do a block-to-unified upgrade. It generally takes about six hours to complete a DIP conversion. As always, if you’re considering this approach, talk to EMC about it.

 

Updated Articles page

It’s been too long since I wrote up a how-to article. But this one came from a really interesting problem. My colleagues were recently faced with an issue at a site where the customer wanted to upgrade from vSphere 5.1 to 5.5. Which was fine, but they’d forgotten / misplaced / couldn’t remember the SSO master password. So I’ve added a brief article covering the steps involved in getting it sorted out. Full credit to Michael, Vincent and our Partner SE Charles for piecing together the steps. I’m really just the messenger.

While you’re there, have a look through my other articles. While dated, some are still useful.

EMC – VNX: USM reports an error “Assistance needed for upgrade”

If you’re trying to do an OE upgrade on a VNX you might get the following error after you’ve run through the “Prepare for Installation” phase.

USM_error

Turns out you just need to upgrade USM to the latest version. You can do this manually or via USM. Further information on this error can be found on support.emc.com by searching for the following Primus ID: emc321171.

Incidentally, I’d just like to congratulate EMC on how much simpler it is upgrade FLARE / VNX OE nowadays than it was when I first started on FC and CX arrays. Sooo much nicer …

 

 

EMC – New VNX OE available

I’m a bit behind on my news, but just a quick note to say that FLARE 33 for the next-generation VNX (Block) has received an update to version 05.33.000.5.038. The release notes are here. Notably, a fix related to ETA 175619 is included. This is a good thing. As always, go to EMC’s Support site for more info, and talk to your local EMC people about whether it’s appropriate to upgrade.

Cisco 9124(e) firmware downgrade

Sometimes, for any number of reasons, you’ll find yourself wanting to downgrade the firmware on your Cisco edge devices to match what you have running in the core. Fortunately, at least for the 9100-series switches, this is basically the same as upgrading the firmware. I’ve included the commands to run here, and also the full output of the process. For the director-class switches, there are a few more things to do, such as clearing out the space on the standby supervisor as well as the active sup card. I’ll try and post something 9500-series specific in the next few weeks.

In short, do this (assuming you’re loading version 3.3(4a) of the code):

copy running-config startup-config

copy startup-config tftp://192.168.101.9/startup-config_FOSLAB5A08_28072010

show module

copy tftp://192.168.101.9/m9100-s2ek9-mz.3.3.4a.bin bootflash:m9100-s2ek9-mz.3.3.4a.bin

copy tftp://192.168.101.9/m9100-s2ek9-kickstart-mz.3.3.4a.bin bootflash:m9100-s2ek9-kickstart-mz.3.3.4a.bin

dir bootflash:

show version image bootflash:m9100-s2ek9-mz.3.3.4a.bin

show incompatibility system m9100-s2ek9-mz.3.3.4a.bin

install all system bootflash:m9100-s2ek9-mz.3.3.4a.bin kickstart bootflash:m9100-s2ek9-kickstart-mz.3.3.4a.bin

y

show module

show version

You can also see the full output here. Note that this process works equally well for HP’s 9124e switches (the type you find in the back of c7000 blade chassis for instance), although you should be downloading the firmware from HP’s site, not Cisco’s.

vSphere 4.1 GA

VMware vSphere 4.1 is now available for download. Release notes can be found here. Enjoy!

2009 and penguinpunk.net

It was a busy year, and I don’t normally do these type of posts, but I thought I’d try to do a year in review type thing so I can look back at the end of 2010 and see what kind of promises I’ve broken. Also, the Exchange Guy will no doubt enjoy the size comparison. You can see what I mean by that here.

In any case, here’re some broad stats on the site. In 2008 the site had 14966 unique visitors according to Advanced Web Statistics 6.5 (build 1.857). But in 2009, it had 15856 unique visitors – according to Advanced Web Statistics 6.5 (build 1.857). That’s an increase of some 890 unique visitors, also known as year-on-year growth of approximately 16.82%. I think. My maths are pretty bad at the best of times, but I normally work with storage arrays, not web statistics. In any case, most of the traffic is no doubt down to me spending time editing posts and uploading articles, but it’s nice to think that it’s been relatively consistent, if not a little lower than I’d hoped. This year (2010 for those of you playing at home), will be the site’s first full year using Google analytics, so assuming I don’t stuff things up too badly, I’ll have some prettier graphs to present this time next year. That said, MYOB / smartyhost are updating the web backend shortly so I can’t make any promises that I’ll have solid stats for this year, or even a website :)

What were the top posts? Couldn’t tell you. I do, however, have some blogging-type goals for the year:

1. Blog with more focus and frequency – although this doesn’t mean I won’t throw in random youtube clips at times.

2. Work more on the promotion of the site. Not that there’s a lot of point promoting something if it lacks content.

3. Revisit the articles section and revise where necessary. Add more articles to the articles page.

On the work front, I’m architecting the move of my current employer from a single data centre to a 2+1 active / active architecture (from a storage and virtualisation perspective). There’s more blades, more CLARiiON, more MV/S, some vSphere and SRM stuff, and that blasted Cisco MDS fabric stuff is involved too. Plus a bunch of stuff I’ve probably forgotten. So I think it will be a lot of fun, and a great achievement if we actually get anything done by June this year. I expect there’ll be some moments of sheer boredom as I work my way through 100s of incremental SAN Copies and sVMotions. But I also expect there will be moments of great excitement when we flick the switch on various things and watch a bunch of visio illustrations turn into something meaningful.

Or I might just pursue my dream of blogging about the various media streaming devices on the market. Not sure yet. In any case, thanks for reading, keep on reading, tell your friends, and click on the damn Google ads.

VMware VirtualCenter 2.5 Update 4

I’ve been nuts deep in a SAN migration project recently and promptly missed the announcement that VMware VirtualCenter 2.5 Update 4 is now available for download. I haven’t had time to put it through its paces yet, but noticed in the release notes that some plugins have been updated, some more useful things have been added to Virtual Machine monitoring, and this little nugget with esxcfg-mpath (a command dear to my heart) still isn’t fixed. But, hey, it’s still better than Sun’s CAM.