OT – Upgrading From macOS Mojave To Catalina (The Hard Way)

This post is really about the boring stuff I do when I have a day off and isn’t terribly exciting. TL;DR I had some problems upgrading to Catalina, and had to start from scratch.

 

Background

I’ve had an Apple Mac since around 2008. I upgraded from a 24″ iMac to a 27″ iMac and was super impressed with the process of migrating between machines, primarily because of Time Machine’s ability to recover settings, applications, and data in a fairly seamless fashion. I can’t remember what version of macOS I started with (maybe Leopard?), but I’ve moved steadily through the last few versions with a minimal amount of fuss. I was running Mojave on my iMac late last year when I purchased a refurbished 2018 Mac mini. At the time, I decided not to upgrade to Catalina, as I’d had a few issues with my work laptop and didn’t need the aggravation. So I migrated from the iMac to the Mac mini and kept on keeping on with Mojave.

Fast forward to April this year, and the Mac mini gave up the ghost. With Apple shutting down its stores here in response to COVID-19, it was a 2 week turnaround at the local repair place to get the machine fixed. In the meantime, I was able to use Time Machine to load everything on a 2012 MacBook Pro that was being used sparingly. It was a bit clunky, but had an internal SSD and 16GB of RAM, so it could handle the basics pretty comfortably. When the Mac mini was repaired, I used Time Machine once again to move everything back. It’s important to note that this is everything (settings, applications, and data) that had been accumulated since 2008. So there’s a bit of cruft associated with this build. A bunch of 32-bit applications that I’d lost track of, widgets that were no longer really in use, and so on.

 

The Big Update

I took the day off on Friday last week. I’d been working a lot of hours since COVID-19 restrictions kicked in here, and I’d been filling my commuting time with day job work (sorry blog!). I thought it would be fun to upgrade the Mac mini to Catalina. I felt that things were in a reasonable enough state that I could work with what it had to offer, and I get twitchy when there’s an upgrade notification on the Settings icon. Just sitting there, taunting me.

I downloaded the installer and pressed on. No dice, my system volume wasn’t formatted with APFS. How could this be? Well, even though APFS has been around for a little while now, I’d been moving my installation across various machines. At the time when the APFS conversion was part of the macOS upgrade, I was running an iMac with a spinning disk as the system volume, and so it never prompted to do that upgrade. When I moved to the Mac mini, I didn’t do any macOS upgrade, so I guess it just kept working with the HFS+ volume. It seems a bit weird that Catalina doesn’t offer a workaround for this, but I may just have been looking in the wrong place. Now, there was a lot of chatter in the forums about rebooting into Recovery Mode and converting the drive to an APFS volume. No matter what I tried, I was unable to do this effectively (either using the Recovery Mode console with Mojave or with Catalina booting from USB). I followed articles like this one but just didn’t have the same experience. And when I erased the system drive and attempted to recover from Time Machine backups, it would re-erase the volume as HFS+. So, I don’t know, I guess I’m an idiot. The solution that finally worked for me was to erase the drive, format it as APFS, install Mojave from scratch, and recover from a Time Machine backup. Unfortunately, though, this seemed to only want to transfer around 800KB of settings data. The normal “wait a few hours while we copy your stuff” just didn’t happen. Sod knows why, but what I did know was that I was really wasting my day off with this stuff.

I also ran in to an issue trying to do the installation from USB. You can read about booting from external devices and the T2 security chip here, here, and here. I lost patience with the process and took a different approach.

 

Is That So Bad?

Not really. I have my Photos library and iTunes media on a separate volume. I have one email account that we have used POP with over the years, but I installed Thunderbird, recovered the profile from my Time Machine data, and modified profiles.ini to point to that profile (causing some flashbacks to my early days on a help desk supporting a Netscape user base). The other thing I had to do was recover my Plex database. You can read more on that here. It actually went reasonably well. I’d been storing my iPhone backups on a separate volume too, and had to follow this process to relocate those backup files. Otherwise, Microsoft, to their credit, has made the reinstallation process super simple with Microsoft 365. Once I had most everything setup again, I was able to perform the upgrade to Catalina.

 

Conclusion

If this process sounds like it was a bit of a pain, it was. I don’t know that Apple has necessarily dropped the ball in terms of usability in the last few years, but sometimes it feels like it. I think I just had really high expectations based on some good fortune I’d enjoyed over the past 12 years. I’m not sure what the term is exactly, but it’s possible that because I’ve invested this much money in a product, I’m more forgiving of the issues associated with the product. Apple has done a great job historically of masking the complexity of technology from the end user. Sometimes, though, you’re going to come across odd situations that potentially push you down an odd path. That’s what I tell myself anyway as I rue the time I lost on this upgrade. Was anyone else’s upgrade to Catalina this annoying?

Rubrik Basics – Cluster Upgrade Process

I’ve been doing some work with Rubrik in our lab and thought it worth covering some of the basic features that I think are pretty neat. In this edition of Rubrik Basics, I thought I’d quickly cover off software upgrades. There are two ways to upgrade the Rubrik software on your Brik – via USB and SFTP. Either way, you’ll need access to the Downloads section of the support site. If you’re a customer, you’ll have this already. If this all sounds too hard, you can raise a ticket with the support team and they’ll tunnel in and do the upgrade for you (assuming you’ve allowed remote tunnel capability).

 

USB

The good thing about using a USB drive is that you can still keep appliances in “dark” sites up to date. Before you begin you’ll need to do two things:

  • Download the compressed upgrade archive and the matching signature file from the customer portal.
  • Format a removable drive with the FAT32 file system.

You’ll need to copy the upgrade file and matching signature file to the removable drive. Plug that into any node in the cluster. Log in to that node as the admin user. Mount the USB drive by typing the following command:

mount --usb_device

Type the following command to begin the upgrade:

upgrade start

The upgrade system scans the file system for upgrade archives. If multiple archives are available, it display a list of choices. Once you’ve finished, you can unmount the device.

umount --usb_device

 

SFTP

You can also run the upgrade via SFTP. I found the instructions on how to do that here. It’s not too dissimilar to the USB method. You’ll want to use your favourite SFTP client to upload the files to the /upgrade directory. Once you’ve done that, ssh on to the node and you can run a pre-flight check. If everything comes up Milhouse you’ll be good to go for the next step.

Using username "admin".

admin@10.xxx.yyy.131's password:

=======================

Welcome to Rubrik CLI

=======================

Type 'help' or '?' to list commands

RVM165Sxxxx55 >> upgrade start --mode prechecks_only
Do you want to use --share rubrik-4.1.2-2366.tar.gz [y/N] [N]: y
Upgrade status: Started pre-checks successfully
RVM165Sxxxx55 >> upgrade status
Current upgrade mode: prechecks_only
Current upgrade pre-checks node: RVM165Sxxxx55
Current upgrade pre-checks tarball name: --share rubrik-4.1.2-2366.tar.gz
Current upgrade pre-checks status: In progress
Current run started at: 2018-07-19 00:48:04.437000 UTC+0000

Current state (3/6): VERIFYING
Current task: Verify authenticity of new software
Current state progress: 0.0%

Finished states (2/6): ACQUIRING, COPYING
Pending states (3/6): UNTARING, DEPLOYING, PRECHECKING

Time taken so far: 18.38 seconds
Overall upgrade progress: 6.0%

To check on progress, run “upgrade status” to, erm, check on the status of the upgrade.

RVM165Sxxxx55 >> upgrade status
Last upgrade mode: prechecks_only
Last upgrade pre-checks node: RVM165Sxxxx55
Last upgrade pre-checks tarball name: --share rubrik-4.1.2-2366.tar.gz
Last upgrade pre-checks status: Completed successfully
Last run ended at: 2018-07-19 00:51:03.129000 UTC+0000
Current state: IDLE

Now you’re ready to do it for real. Run “upgrade start” to start.

RVM165Sxxxx55 >> upgrade start
Do you want to use --share rubrik-4.1.2-2366.tar.gz [y/N] [N]: y
Upgrade status: Started upgrade successfully
RVM165Sxxxx55 >> upgrade status
Current upgrade mode: normal
Current upgrade node: RVM165Sxxxx55
Current upgrade tarball name: --share rubrik-4.1.2-2366.tar.gz
Current upgrade status: In progress
Current run started at: 2018-07-19 00:52:56.882000 UTC+0000

Current state (4/9): UNTARING
Current task: Extract new software
Current state progress: 0.0%

Finished states (3/9): ACQUIRING, COPYING, VERIFYING
Pending states (5/9): DEPLOYING, PRECHECKING, PREPARING, UPGRADING, RESTARTING

Time taken so far: 22.52 seconds
Overall upgrade progress: 3.5%

It’s a pretty quick process, and eventually you’ll see this message.

RVM165Sxxxx55 >> upgrade status
Last upgrade mode: normal
Last upgrade node: RVM165Sxxxx55
Last upgrade tarball name: --share rubrik-4.1.2-2366.tar.gz
Last upgrade status: Completed successfully
Last run ended at: 2018-07-19 01:19:09.719000 UTC+0000

Current state: IDLE
RVM165Sxxxx55 >>

And you’re all done. Note that you only have to upload the data and run the process on one node in the cluster.

OpenMediaVault – Updating from 2.2.x to 3.0.x

I recently upgraded my home-brew NAS from OpenMediaVault 2.2.14 (Stone burner) to openmediavault 3.0.86 (Erasmus). It’s recommended that you do a fresh install but I thought I’d give the upgrade a shot as it was only a 10TB recovery if it went pear-shaped (!). They also recommend you disable all your plugins before you upgrade.

 

Apt-get all of the things

It’s an OS upgrade as well as an application upgrade. In an ssh session I ran

apt-get update && apt-get dist-upgrade && omv-update

This gets you up to date, then upgrades your distro (Debian), and then gets the necessary packages for omv. I then ran the omv upgrade.

omv-release-upgrade

This seemed to go well. I rebooted the box and could still access the shared data. Happy days. When I tried to access the web UI, however, I could enter my credentials but I couldn’t get in. I then ran

omv-firstaid

And tried to reconfigure the web interface. It kept complaining about a file not being found. So I ran

dpkg -l | grep openmediavault

This told me that there was still a legacy plugin (openmediavault-dnsmasq) running. I’d read on the forums that this might cause some problems. So I used apt-get to remove it.

apt-get remove openmediavault-dnsmasq

The next time I ran apt-get it told me there were some legacy packages present that I could remove. So I did.

apt-get autoremove dnsmasq dnsmasq-base libnetfilter-conntrack3

After that, I was able to login in to the web UI with no problems and everything now seems to be in order. When my new NAS arrives I’ll evacuate this one and rebuild it from scratch. There are a fair few changes in version 3 and it’s worth checking out. You can download the ISO image from here.

 

DNS Matters

The reason I had the dnsmasq plugin installed in the first place was that I’d been using the NAS as a DHCP / DNS server. This had been going reasonably well, but I’d heard about Pi-hole and wanted to give that a shot. That’s a story for another time, but I did notice that my OMV box hadn’t updated its /etc/resolv.conf feel correctly, despite the fact that I’d reconfigured DNS via the web GUI. If you run into this issue, just run

dpkg-reconfigure resolvconf

And you’ll find that resolv.conf is correctly updated. Incidentally, if you’re a bit old-fashioned and don’t like to run everything through DHCP reservations, you can add a second set of static host entries to dnsmasq on your pi-hole machine by following these instructions.

EMC – Next-Generation VNX – Data In Place Upgrades

Approximately 4 or 500 years ago, I spent a number of nights in various data centres around the country upgrading customers’ CLARiiON arrays from CX200s to Cx500s, CX300s to CX3-20s, and so on. The neat thing about the CLARiiON was that EMC had a pretty reasonable way of doing data in place (DIP) upgrades, including across generations if required. With the introduction of the VNX, that changed. Primarily because of the switch from FC to SAS on the back-end. And with the “Next-Generation” VNX (VNX2), you also can’t go from VNX to VNX2. Which some people have been understandably unhappy about. The procedure hasn’t changed much over the years, and you can read Rob’s post here for a pretty thorough look at what’s involved.

So why would you want to do this anyway? Especially given that, if you’re upgrading a VNX5200 for example, you’ve probably only had the array in operation for a few years. Well, requirements change, companies grow, people need more horsepower. Sometimes EMC makes it a commercially viable option to do a DIP upgrade rather than replace the array with another one. There’re are a bunch of reasons.

I don’t want to go into exactly what the steps are, as your friendly EMC service folk or partner will be able to go through that with you, but I thought it might be an idea to share a few things to know prior to launching into one of these procedures (or even making the decision to upgrade in this fashion).

The supported source systems include:

  • VNX5200;
  • VNX5400;
  • VNX5600; and
  • VNX5800.

Note that you cannot convert a VNX7600, nor can you go from VNX to VNX2 (as I mentioned before). Also, the VNX8000 can’t be a source system, because that’s already as big as the VNX goes.

Supported targets for upgrade include:

  • VNX5400;
  • VNX5600;
  • VNX5800; and
  • VNX7600.

You can’t go to a VNX8000. You can also upgrade the type of array as follows:

  • Block to block;
  • File to file; and
  • Unified to unified.

You can’t convert from a block system to a higher performing unified system. You can, however, do a block conversion, and do a block-to-unified upgrade. It generally takes about six hours to complete a DIP conversion. As always, if you’re considering this approach, talk to EMC about it.

 

Updated Articles page

It’s been too long since I wrote up a how-to article. But this one came from a really interesting problem. My colleagues were recently faced with an issue at a site where the customer wanted to upgrade from vSphere 5.1 to 5.5. Which was fine, but they’d forgotten / misplaced / couldn’t remember the SSO master password. So I’ve added a brief article covering the steps involved in getting it sorted out. Full credit to Michael, Vincent and our Partner SE Charles for piecing together the steps. I’m really just the messenger.

While you’re there, have a look through my other articles. While dated, some are still useful.

EMC – VNX: USM reports an error “Assistance needed for upgrade”

If you’re trying to do an OE upgrade on a VNX you might get the following error after you’ve run through the “Prepare for Installation” phase.

USM_error

Turns out you just need to upgrade USM to the latest version. You can do this manually or via USM. Further information on this error can be found on support.emc.com by searching for the following Primus ID: emc321171.

Incidentally, I’d just like to congratulate EMC on how much simpler it is upgrade FLARE / VNX OE nowadays than it was when I first started on FC and CX arrays. Sooo much nicer …

 

 

EMC – New VNX OE available

I’m a bit behind on my news, but just a quick note to say that FLARE 33 for the next-generation VNX (Block) has received an update to version 05.33.000.5.038. The release notes are here. Notably, a fix related to ETA 175619 is included. This is a good thing. As always, go to EMC’s Support site for more info, and talk to your local EMC people about whether it’s appropriate to upgrade.

Cisco 9124(e) firmware downgrade

Sometimes, for any number of reasons, you’ll find yourself wanting to downgrade the firmware on your Cisco edge devices to match what you have running in the core. Fortunately, at least for the 9100-series switches, this is basically the same as upgrading the firmware. I’ve included the commands to run here, and also the full output of the process. For the director-class switches, there are a few more things to do, such as clearing out the space on the standby supervisor as well as the active sup card. I’ll try and post something 9500-series specific in the next few weeks.

In short, do this (assuming you’re loading version 3.3(4a) of the code):

copy running-config startup-config

copy startup-config tftp://192.168.101.9/startup-config_FOSLAB5A08_28072010

show module

copy tftp://192.168.101.9/m9100-s2ek9-mz.3.3.4a.bin bootflash:m9100-s2ek9-mz.3.3.4a.bin

copy tftp://192.168.101.9/m9100-s2ek9-kickstart-mz.3.3.4a.bin bootflash:m9100-s2ek9-kickstart-mz.3.3.4a.bin

dir bootflash:

show version image bootflash:m9100-s2ek9-mz.3.3.4a.bin

show incompatibility system m9100-s2ek9-mz.3.3.4a.bin

install all system bootflash:m9100-s2ek9-mz.3.3.4a.bin kickstart bootflash:m9100-s2ek9-kickstart-mz.3.3.4a.bin

y

show module

show version

You can also see the full output here. Note that this process works equally well for HP’s 9124e switches (the type you find in the back of c7000 blade chassis for instance), although you should be downloading the firmware from HP’s site, not Cisco’s.

vSphere 4.1 GA

VMware vSphere 4.1 is now available for download. Release notes can be found here. Enjoy!

2009 and penguinpunk.net

It was a busy year, and I don’t normally do these type of posts, but I thought I’d try to do a year in review type thing so I can look back at the end of 2010 and see what kind of promises I’ve broken. Also, the Exchange Guy will no doubt enjoy the size comparison. You can see what I mean by that here.

In any case, here’re some broad stats on the site. In 2008 the site had 14966 unique visitors according to Advanced Web Statistics 6.5 (build 1.857). But in 2009, it had 15856 unique visitors – according to Advanced Web Statistics 6.5 (build 1.857). That’s an increase of some 890 unique visitors, also known as year-on-year growth of approximately 16.82%. I think. My maths are pretty bad at the best of times, but I normally work with storage arrays, not web statistics. In any case, most of the traffic is no doubt down to me spending time editing posts and uploading articles, but it’s nice to think that it’s been relatively consistent, if not a little lower than I’d hoped. This year (2010 for those of you playing at home), will be the site’s first full year using Google analytics, so assuming I don’t stuff things up too badly, I’ll have some prettier graphs to present this time next year. That said, MYOB / smartyhost are updating the web backend shortly so I can’t make any promises that I’ll have solid stats for this year, or even a website :)

What were the top posts? Couldn’t tell you. I do, however, have some blogging-type goals for the year:

1. Blog with more focus and frequency – although this doesn’t mean I won’t throw in random youtube clips at times.

2. Work more on the promotion of the site. Not that there’s a lot of point promoting something if it lacks content.

3. Revisit the articles section and revise where necessary. Add more articles to the articles page.

On the work front, I’m architecting the move of my current employer from a single data centre to a 2+1 active / active architecture (from a storage and virtualisation perspective). There’s more blades, more CLARiiON, more MV/S, some vSphere and SRM stuff, and that blasted Cisco MDS fabric stuff is involved too. Plus a bunch of stuff I’ve probably forgotten. So I think it will be a lot of fun, and a great achievement if we actually get anything done by June this year. I expect there’ll be some moments of sheer boredom as I work my way through 100s of incremental SAN Copies and sVMotions. But I also expect there will be moments of great excitement when we flick the switch on various things and watch a bunch of visio illustrations turn into something meaningful.

Or I might just pursue my dream of blogging about the various media streaming devices on the market. Not sure yet. In any case, thanks for reading, keep on reading, tell your friends, and click on the damn Google ads.