Cisco MDS, NVMe, and Flexibility

Disclaimer: I recently attended Storage Field Day 20.  My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event.  Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

Cisco recently presented at Storage Field Day 20. You can see videos of the presentation here, and download my rough notes from here.


NVMe, Yeah You Know Me

Non-Volatile Memory Express, known more commonly as NVMe, is a protocol designed for high performance SSD storage access.  In the olden days, we used to associate fibre channel and iSCSI networking options with high performance block storage. Okay, maybe not the 1Gbps iSCSI stuff, but you know what I mean. Time has passed, and the storage networking landscape has changed significantly with the introduction of All-Flash and NVMe. But NVMe’s adoption hasn’t been all smooth sailing. There have been plenty of vendors willing to put drives in storage arrays that support NVMe while doing some translation on the backend that negated the real benefits of NVMe. And, like many new technologies, it’s been a gradual process to get end-to-end NVMe in place, because enterprises, and the vendors that sell to them, only move so fast. Some vendors support NVMe, but only over FC. Others have adopted the protocol to run over RoCEv2. There’s also NVMe-TCP, in case you weren’t confused enough about what you could use. I’m doing a poor job of explaining this, so you should really just head over to Dr J Metz’s article on NVMe for beginners at SNIA.


Cisco Are Ready For Anything

As you’ve hopefully started to realise, you’ll see a whole bunch of NVMe implementations available in storage fabrics, along with a large number of enterprises continuing to have conversations about and deploy new storage equipment that uses traditional block fabrics, such as iSCSI or FC or, perish the thought, FCoE. The cool thing about Cisco MDS is that it supports all this crazy and more. If you’re running the latest and greatest NVMe end to end implementation and have some old block-only 8Gbps FC box sitting in the corner they can likely help you with connectivity. The diagram below hopefully demonstrates that point.

[image courtesy of Cisco]


Thoughts and Further Reading

Very early in my storage career, I attended a session on MDS at Cisco Networkers Live (when they still ran those types of events in Brisbane). Being fairly new to storage, and running a smallish network of one FC4700 and 8 Unix hosts, I’d tended to focus more on the storage part of the equation rather than the network part of the SAN. Cisco was still relatively new to the storage world at that stage, and it felt a lot like it had adopted a very network-centric view of the storage world. I was a little confused why all the talk was about backplanes and port density, as I was more interested about the optimal RAID configuration for mail server volumes and how I should protect the data being stored on this somewhat sensitive piece of storage. As time went on, I was invariably exposed to larger and larger environments where decisions around core and edge storage networking devices started to become more and more critical to getting optimal performance out of the environment. A lot of the information I was exposed to in that early MDS session started to make more sense (particularly as I was tasked with deploying larger and larger MDS-based fabrics).

Things have obviously changed quite a bit since those heady days of a network upstart making waves in the storage world. We’ve seen increases in network speeds become more and more common in the data centre, and we’re no longer struggling to get as many IOPS as we can out of 5400 RPM PATA drives with an interposer and some slightly weird firmware. What has become apparent, I think, is the importance of the fabric when it comes to getting access to storage resources in a timely fashion, and with the required performance. As enterprises scale up and out, and more and more hosts and applications connect to centralised storage resources, it doesn’t matter how fast those storage resources are if there’s latency in the fabric.

The SAN still has a place in the enterprise, despite was the DAS huggers will tell you, and you can get some great performance out of your SAN if you architect it appropriately. Cisco certainly seems to have an option for pretty much everything when it comes to storage (and network) fabrics. It also has a great story when it comes to fabric visibility, and the scale and performance at the top end of its MDS range is pretty impressive. In my mind, though, the key really is the variety of options available when build a storage network. It’s something that shouldn’t be underestimated given the plethora of options available in the market.

Cisco – Restoring MDS configurations from somewhere else

We recently had to replace a Cisco MDS 9124e in our lab. I used to use this method to copy and restore configuration files to MDS switches.

switch# copy tftp:// startup-config
Trying to connect to tftp server......
Connection to server Established. Copying Started.....
TFTP get operation was successful
This command is deprecated. To obtain the same results, please use
the sequence 'write erase' + 'reload' + 'copy <file> running-config' + 'copy running-config startup-config'.

It was rough, but it used to work. So now I do this.

switch# copy tftp:// bootflash:
Trying to connect to tftp server......
Connection to server Established. Copying Started.....
TFTP get operation was successful
switch# dir
      15155    Feb 05 21:37:37 2013  switch.cfg

write erase
copy switch.cfg running-config
copy run start

It makes sense, as the write erase and reload commands make you think about what you’re doing, and you need to be sure that you want to overwrite the running or startup config.

Updated articles page

I’ve added another document to my articles page. This one covers the creation of port-channels between Cisco MDS 9513 switches. I was clueless about a lot of this until a friend from EMC took me through the steps. So I’ve created this document as a way to capture those steps for future reference. Hopefully you’ll find it of use.

Cisco MDS 9XXX Basics – Part 2

Setting up ntp on your shiny, new Cisco MDS 9XXX switch is fairly simple. And important. Time man, it’s really important.

From a terminal session, issue the following commands:

conf t
ntp server prefer
ntp server

copy running-config startup-config

And that’s it …

Cisco MDS 9XXX Basics – Part 1

So we’ve finally started delivering on the project that I’ve been working on for the last 12 – 18 months. It’s fun to see my detailed designs turn into running infrastructure.

As part of this, I’ve been doing some configuration of some new Cisco 9513 and 9124e switches for our fabric. I have every intention of writing a downloadable article with some of the basic stuff, but I thought I’d do a few, smaller articles for my own reference more than anything else.

Now, most Cisco nerds will already know this stuff, but for someone like me who cut their teeth on Brocade Fabric OS, it’s a little different.

To connect to a 9124e (Cisco’s blade switch), I recommend using the HP OA’s serial connection.

Connect to the active OA via serial, login using your normal credentials and run

connect interconnect 3

This will connect you to the serial console of the first 9124e switch in the chassis. This assumes that you have other devices in bays 1 and 2, such as Cisco 3120s, or whatever.

If this is the first time you’ve connected to the switch, or if you’ve not configured it yet, you’ll get to a very useful first setup screen.

Press [Enter] to display the switch console:
  Enter the password for “admin”:
  Confirm the password for “admin”:

         —- Basic System Configuration Dialog —-

This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.

Please register Cisco MDS 9000 Family devices promptly with your
supplier. Failure to register may affect response times for initial
service calls. MDS devices must be registered to receive entitled
support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime
to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): yes


  Create another login account (yes/no) [n]:

  Configure read-only SNMP community string (yes/no) [n]:

  Configure read-write SNMP community string (yes/no) [n]:

  Enter the switch name : FCswitch1

  Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]:

    Mgmt0 IPv4 address :

    Mgmt0 IPv4 netmask :

  Configure the default gateway? (yes/no) [y]:

    IPv4 address of the default gateway :

  Configure advanced IP options? (yes/no) [n]:

  Enable the ssh service? (yes/no) [y]:

    Type of ssh key you would like to generate (dsa/rsa) [rsa]:

    Number of rsa key bits <768-2048> [1024]:

  Enable the telnet service? (yes/no) [n]:

  Enable the http-server? (yes/no) [y]:

 Configure clock? (yes/no) [n]:

 Configure timezone? (yes/no) [n]:

 Configure summertime? (yes/no) [n]:

  Configure the ntp server? (yes/no) [n]:

  Configure default switchport interface state (shut/noshut) [shut]:

  Configure default switchport trunk mode (on/off/auto) [on]:

  Configure default switchport port mode F (yes/no) [n]:

  Configure default zone policy (permit/deny) [deny]:

  Enable full zoneset distribution? (yes/no) [n]:

  Configure default zone mode (basic/enhanced) [basic]:

The following configuration will be applied:
  password strength-check
  switchname FCswitch1
  interface mgmt0
    ip address
    no shutdown
  ip default-gateway
  ssh key rsa 1024 force
  feature ssh
  no feature telnet
  feature http-server
  system default switchport shutdown
  system default switchport trunk mode on
  no system default zone default-zone permit
  no system default zone distribute full
  no system default zone mode enhanced

Would you like to edit the configuration? (yes/no) [n]:

Use this configuration and save it? (yes/no) [y]:

At this point, the switch does a copy run start and reboots. For some reason we’ve been getting this error.

 Error: There was an error executing at least one of the commands
Please verify the following log for the command execution errors.
Disabling ssh: as its enabled right now:
 ssh: Cannot disable both telnet and SSH

I’ve been ignoring this error. So, too, has NX-OS. You’ll then see the following:

Would you like to save the running-config to startup-config? (yes/no) [n]: y

[########################################] 100%

The switch then reboots and you can monitor it for any errors. Once you’re satisfied with the config, use CTRL-SHIFT-_ and press d to disconnect from the 9124e terminal. The process is identical for the Cisco MDS 9513, except for the bit about it being a blade switch :)