EMC – naviseccli – checking your iSCSI ports are running at the correct speed

It’s been a while since I wrote about naviseccli and I admit I’ve missed it. I once wrote about using naviseccli to identify MirrorView ports on a CLARiiON array. Normally the MirrorView port is consistently located, but in that example we’d upgraded from a CX3-80 to a Cx4-960 and it was in a different spot. Oh how we laughed when we realised what the problem was. Anyway, we’ve been doing some work on an ever so slightly more modern VNX5300 and needed to confirm that some newly installed iSCSI SLICs were operating at the correct speed. (Note that these commands were run from the Control Station).

The first step is to list the ports

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2016.09.07 08:59:37 =~=~=~=~=~=~=~=~=~=~=~=
[nasadmin@NAS001 ~]$ navicli -h A_VNXSP connection -getport

SP:  A
Port ID:  8
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.a8
iSCSI Alias:  0017.a8
IP Address:  192.168.0.13
Subnet Mask:  255.255.255.0
Gateway Address:  192.168.0.254
Initiator Authentication:  false

SP:  A
Port ID:  9
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.a9
iSCSI Alias:  0017.a9

SP:  A
Port ID:  10
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.a10
iSCSI Alias:  017.a10

SP:  A
Port ID:  11
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.a11
iSCSI Alias:  017.a11

SP:  B
Port ID:  8
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.b8
iSCSI Alias:  0017.b8
IP Address:  192.168.0.14
Subnet Mask:  255.255.255.0
Gateway Address:  192.168.0.254
Initiator Authentication:  false

SP:  B
Port ID:  9
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.b9
iSCSI Alias:  0017.b9

SP:  B
Port ID:  10
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.b10
iSCSI Alias:  017.b10

SP:  B
Port ID:  11
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.b11
iSCSI Alias:  017.b11

Once you’ve done that, you can list the port speed for a particular port

[nasadmin@NAS001 ~]$ navicli -h A_VNXSP connection -getport -sp a -portid 8 -speed
SP:  A
Port ID:  8
Port WWN:  iqn.1992-04.com.emc:cx.cetv2223700017.a8
iSCSI Alias:  0017.a8
IP Address:  192.168.0.13
Subnet Mask:  255.255.255.0
Gateway Address:  192.168.0.254
Initiator Authentication:  false
Port Speed:  1000 Mb
Auto-Negotiate:  Yes
Available Speeds:  10 Mb
-               :  100 Mb
-               :  1000 Mb
-               :  Auto

If you have a lot of ports to check this may not be the most efficient way to do it (ioportconfig may be more sensible), but if your network team are reporting on one particular port being an issue – this is a great way to narrow it down.

EMC – Next-Generation VNX – Data In Place Upgrades

Approximately 4 or 500 years ago, I spent a number of nights in various data centres around the country upgrading customers’ CLARiiON arrays from CX200s to Cx500s, CX300s to CX3-20s, and so on. The neat thing about the CLARiiON was that EMC had a pretty reasonable way of doing data in place (DIP) upgrades, including across generations if required. With the introduction of the VNX, that changed. Primarily because of the switch from FC to SAS on the back-end. And with the “Next-Generation” VNX (VNX2), you also can’t go from VNX to VNX2. Which some people have been understandably unhappy about. The procedure hasn’t changed much over the years, and you can read Rob’s post here for a pretty thorough look at what’s involved.

So why would you want to do this anyway? Especially given that, if you’re upgrading a VNX5200 for example, you’ve probably only had the array in operation for a few years. Well, requirements change, companies grow, people need more horsepower. Sometimes EMC makes it a commercially viable option to do a DIP upgrade rather than replace the array with another one. There’re are a bunch of reasons.

I don’t want to go into exactly what the steps are, as your friendly EMC service folk or partner will be able to go through that with you, but I thought it might be an idea to share a few things to know prior to launching into one of these procedures (or even making the decision to upgrade in this fashion).

The supported source systems include:

  • VNX5200;
  • VNX5400;
  • VNX5600; and
  • VNX5800.

Note that you cannot convert a VNX7600, nor can you go from VNX to VNX2 (as I mentioned before). Also, the VNX8000 can’t be a source system, because that’s already as big as the VNX goes.

Supported targets for upgrade include:

  • VNX5400;
  • VNX5600;
  • VNX5800; and
  • VNX7600.

You can’t go to a VNX8000. You can also upgrade the type of array as follows:

  • Block to block;
  • File to file; and
  • Unified to unified.

You can’t convert from a block system to a higher performing unified system. You can, however, do a block conversion, and do a block-to-unified upgrade. It generally takes about six hours to complete a DIP conversion. As always, if you’re considering this approach, talk to EMC about it.

 

EMC – Using naviseccli to configure a VNX domain

The concept of domains have been with CLARiiON and (later) VNX arrays since the early part of the 21st Century. The configuration is fairly simple, and, in keeping with the idea that you can do anything with naviseccli, I thought I’d do a quick post on using naviseccli to join SPs to a domain. This assumes you have security setup with your naviseccli environment, and you know the IPs of the SPs you’re trying to add to the domain.

You can the set the master node for a domain with this command. Note that the nominated node can’t be a member of another domain at the time.

naviseccli -h SPA-IP-Address domain -setmaster SPB-IP-Address
 WARNING: You are about to set the following node as the master of the domain: SPB-IP-Address
 Proceed? (y/n) y

If a node is a problem, or you’re about to remove an array from your environment, it’s a good idea to remove it from the domain before you rip it out of the rack.

naviseccli -h SPA-IP-Address domain -remove SPA-IP-Address
 WARNING: You are about to remove the following node from the domain: SPA-IP-Address
 Proceed? (y/n) y

You may also wish to add another couple of nodes, particularly if you have a number of arrays in the environment.

naviseccli -h SPB-IP-Address domain -add SPA-IP-Address
 WARNING: You are about to remove the following node from the domain: SPA-IP-Address
 Proceed? (y/n) y

And that’s it. I recommend you check out EMC’s white paper – Domain Management with EMC Unisphere for VNX (p/n h8853.4) – for more information on VNX domain management.

EMC – VNX – Slow Disk Rebuild Times

I’ve been a bit behind on my VNX OE updates, and have only recently read docu59127_VNX-Operating-Environment-for-Block-05.33.000.5.102-and-for-File-8.1.6.101,-EMC-Unisphere-1.3.3.1.0096-Release-Notes covering VNX OE 5.33…102. Checking out the fixed problems, I noticed the following item.

VNX_OE_RN

The problem, you see, came to light some time ago when a few of our (and no doubt other) VNX2 customers started having disk failures on reasonably busy arrays. EMC have a KB on the topic on the support site – VNX2 slow disk rebuild speeds with high host I/O (000187088). To quote EMC “The code has been written so that the rebuild process is considered a lower priority than the Host IO. The rebuild of the new drive will take much longer if the workload from the hosts are high”. Which sort of makes sense, because host I/O is a pretty important thing. But, as a number of customers pointed out to EMC, there’s no point prioritising host I/O if you’re in jeopardy of having a data unavailable or data loss event because your private RAID groups have taken so long to complete.

Previously, the solution was to “[r]educe the amount of host I/O if possible to increase the speed of the drive rebuild”. Now, however, updated code comes to the rescue. So, if you’re running a VNX2, upgrade to the latest OE if you haven’t already.

 

 

EMC – VNX – Configuring LDAP Authentication

I’m surprised that I haven’t done an article on configuring Active Directory (AD) authentication on the VNX. It’s pretty easy to do, and a good idea. Big thanks to Sean Thulin for documenting this in a clear and concise fashion, and to EMC Support‘s website for filling in some of the blanks I had (via Primus emc308583).

DNS

Firstly, you should have DNS configured on your array. This is just a basic thing that you should do. Stop making excuses.

Dsquery

For AD authentication, you need the following information:

  • Domain Controller (DC) hostname;
  • A basic account on AD with read permission on AD on Users and Group containers – this account is called the Bind DN; and
  • Full path information for the Bind DN, the User container, and the Group container.

To obtain this, log in to a Windows computer with dsquery installed. You don’t need Domain Admin rights to get this information.

To determine the DC hostname, run set | findstr “LOGONSERVER” to return the hostname.

If there isn’t a Bind DN account created, you’ll need one. This can be a normal user account with the password preferably set to “Not Expired” to avoid issues down the track. Once the user is created anywhere in AD, use Dsquery thusly:

C:\Users\dan>dsquery user -name ldap_account

You’ll get this:

"CN=ldap_account,OU=Service Accounts,DC=domain,DC=com"

The above is fully qualified path name for the account “ldap_account,” which will be used as the Bind DN. You’ll need access to the password of this service account.

The User container is where the VNX will look for the user login be used for authentication. In this example the user name is “Storage User”.

C:\Users\dan>dsquery user -name "Storage User"
"CN=Storage User,OU=Storage Admins,OU=Administrators,DC=domain,DC=com"

The User Container path here that you need to note is: OU=Storage Admins,OU=Administrators,DC=domain,DC=com

For the group, you can do the same thing. In a number of environments, this will be the same location as the Users.

C:\Users\dan>dsquery group -name "Storage Admins"
"CN=Storage Admins,OU=Storage Admin Groups,OU=Administrators,DC=domain,DC=com"

The path name for group container is : OU=Storage Admin Groups,OU=Administrators,DC=domain,DC=com

Manage LDAP

Now you’re ready to set things up. Go to Domain -> Manage LDAP  and configure using the above collected information.

LDAP1

You can configure two service connections. These would usually be DCs that are at discrete data centres.

LDAP2

Click on Add or Modify.

LDAP3

Here’s what you need to fill in:

  • Host Name or IP Address – Use the FQDN, it’s 2015 and DNS should work in your environment;
  • Port 389 for LDAP, 636 for LDAPS – This will change depending on whether you select LDAP or LDAPS as the protocol;
  • Server Type – Choose “Active Directory”;
  • Domain Name – Specify the domain name;
  • BindDN – This is where you put the distinguished name of the LDAP service account;
  • Bind Password – The password for the LDAP service account;
  • Confirm Bind Password – Confirmed;
  • User Search Path – This is the info we got earlier;
  • Group Search Path – Ditto; and
  • Add certificate – If you’re using LDAPS, you’ll need this.

Role Mapping

LDAP4

Note that it is recommended to use group names with no special characters and with fewer than 32 characters. The main roles include:

  • Operator – Read-only privilege for storage and domain operations; no privilege for security operations.
  • Network Administrator – All operator privileges and privileges to configure DNS, IP settings, and SNMP.
  • NAS Administrator – Full privileges for file operations. Operator privileges for block and security operations.
  • SAN Administrator – Full privileges for block operations. Operator privileges for file and security operations.
  • Storage Administrator – Full privileges for file and block operations. Operator privileges for security operations.
  • Security Administrator – Full privileges for security operations including domains. Operator privileges for file and block operations.
  • Administrator – Full privileges for file, block, and security operations. This role is the most privileged role.
  • VM Administrator – Enables you to view and monitor basic storage components of your VNX system through vCenter by using VMware’s vSphere Storage APIs for Storage Awareness (VASA).

Note that some of these roles apply to “Unified” configs (NAS), rather than block-only.

Conclusion

Don’t forget to synchronise the information once you’ve created the connections. And that’s it. you should now be able to log in to your VNX with your AD credentials. Just make sure “Use LDAP” is ticked.

EMC – VSI for VMware vSphere 6.5 Linked Mode Issue – Redux

I wrote in a previous post about having some problems with EMC’s VSI for VMware vSphere 6.5 when running in vCenter 5.5 in Linked Mode. I spoke about deploying the appliance in just one site as a workaround. Turns out that wasn’t much of a workaround. Because workaround implies that I was able to get some functionality out of the situation. While the appliance deployed okay, I couldn’t get it to recognise the deployed volumes as EMC volumes.

 

A colleague of mine had the same problem as me and a little more patience and logged a call with EMC support. Their response was “[c]urrent VSI version does not support for Linked mode, good news is recently we have several customers requesting that enhancement and Dev team are in the process of evaluating impact to their future delivery schedule. So, the linked mode may will be supported in the future. Thanks.”

 

iStock-Unfinished-Business-3

While this strikes me as non-optimal, I am hopeful, but not optimistic, that it will be fixed in a later version. My concern is that Linked Mode isn’t the problem at all, and it’s something else stupid that I’m doing. But I’m short of places I can test this at the moment. If I come across a site where we’re not using Linked Mode, I’ll be sure to fire up the appliance and run it through its paces, but for now it’s back in the box.

EMC – Using naviseccli to create a VNX Snapshot

If you’re a VNX customer you’ve probably heard someone bang on about how easy to use VNX Snapshots are, particularly if they’ve used SnapView in the past. If you’re after the good word on VNX Snapshots, check out this whitepaper from EMC here. Tomek has a reasonable write-up here as well.

In any case I’ve been working with a customer on some migration scripts and they wanted to take VNX Snapshots as well as VM snapshots while they update their OS and apps. I wrote about creating SnapView Clones with naviseccli some time ago, but I find VNX Snapshots a shedload easier to work with. This is will, as always, be dictated by your own set of requirements, circumstances and religious beliefs.

So here’s what you need to do to get from start to finish. Note that I haven’t covered creating Snapshot Mount Points (SMPs) in this, nor do I talk about using host-based tools such as SnapCLI. I’ll follow up in the future with some words around this.

[Update] I forgot to mention @Dynamoxxx / Storage Monkey‘s excellent posts on this subject too – have a look here for Linux and here for Windows.

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Program Files (x86)\EMC\Navisphere CLI>NaviSECCli.exe
Not enough arguments
  Usage:
    [-User <username>] [-Password <password>]
    [-Scope <0 - global; 1 - local; 2 - LDAP>]
    [-Address <IPAddress | NetworkName> | -h <IPAddress | NetworkName>]
    [-Port <portnumber>] [-Timeout <timeout> | -t <timeout>]
    [-AddUserSecurity | -RemoveUserSecurity | -DeleteSecurityEntry]
    [-Parse | -p] [-NoPoll | -np] [-cmdtime]
    [-Xml] [-f <filename>] [-Help] CMD <Optional Arguments>
    [security -certificate]

You’ll need to set yourself up if you’re using a fresh installation.

C:\Program Files (x86)\EMC\Navisphere CLI>NaviSECCli.exe -addusersecurity -scope 0 -user sysadmin

You can then create a snapshot of LUN 7 called “testsnap1” which is read/write and will be kept for 4 hours.

C:\Program Files (x86)\EMC\Navisphere CLI>NaviSECCli.exe -address 192.168.0.100 snap -create -res 7 -resType LUN -name "testsnap1" -descr "snap via CLI" -keepFor 4h -allowreadwrite yes
Unable to validate the identity of the server.  There are issues with the certificate presented.
Only import this certificate if you have reason to believe it was sent by a trusted source.
Certificate details:
Subject:        CN=192.168.0.100,CN=SPA,OU=CLARiiON
Issuer: CN=192.168.0.100,CN=SPA,OU=CLARiiON
Serial#:        fcd99068
Valid From:     2015:01:15:02:55:01
Valid To:       2020:01:14:02:55:01
Would you like to [1]Accept the certificate for this session, [2] Accept and store, [3] Reject the certificate?
Please input your selection(The default selection is [1]):
2

Note that there’s no output from this command. If you want to check out the snapshots you have, you can list them.

C:\Program Files (x86)\EMC\Navisphere CLI>naviseccli -address 192.168.0.100 snap -list

Name:  testsnap1
Description:  snap via CLI
Creation time:  05/19/15 10:22:37
Source LUN(s):  7
Source CG:  N/A
State:  Ready
Allow Read/Write:  Yes
Modified:  No
Allow auto delete:  No
Expiration date:  05/19/15 14:22:37

Want to change the ID of the snapshot or change the autodelete setting?

C:\Program Files (x86)\EMC\Navisphere CLI>naviseccli -address 192.168.0.100 snap -modify -id "testsnap1" -name "testsnap2" -allowautodelete yes
Setting auto-delete on this Snapshot will clear expiration date on it. Are you sure you want to perform this operation?(y/n): n
C:\Program Files (x86)\EMC\Navisphere CLI>naviseccli -address 192.168.0.100 snap -modify -id "testsnap1" -name "testsnap2"

Great, now let’s get rid of it.

C:\Program Files (x86)\EMC\Navisphere CLI>naviseccli -address 192.168.0.100 snap -destroy -id "testsnap2"
Are you sure you want to perform this operation?(y/n): y

And that’s about it.

EMC – VSI for VMware vSphere 6.5 Linked Mode Issue

As part of a recent deployment I’ve been implementing EMC VSI for VMware vSphere Web Client v6.5 in a vSphere 5.5 environment. If you’re not familiar with this product, it “enables administrators to view, manage, and optimize storage for VMware ESX/ESXi servers and hosts and then map that storage to the hosts.” It covers a bunch of EMC products, and can be really useful in understanding where your VMs sit in relation to your EMC storage environment. It also really helps non-storage admins get going quickly in an EMC environment.

To get up and running, you:

  • Download the appliance from EMC;
  • Deploy the appliance into your environment;
  • Register the plug-in with vCenter by going to https://ApplianceIP:8443/vsi_usm/admin;
  • Register the Solutions Integration Service in the vCenter Web Client; and
  • Start adding arrays as required.

So this is all pretty straightforward. BTW the default username is admin, and the default password is ChangeMe. You’ll be prompted to change the password the first time you log in to the appliance.

 

So the problem for me arose when I went to register a second SIS appliance.

VSI1

By way of background, there are two vCenter 5.5 U2 instances running at two different data centres. I do, however, have them running in Linked Mode. And I think this is the problem. I know that you can only register one instance at a time with one vCenter. While it’s not an issue to deploy a second appliance at the second DC, every time I go to register the service in vCenter, regardless of where I’m logged in, it always points to the first vCenter instance. Which is a bit of a PITA, and not something I’d expected to be a problem. As a workaround, I’ve deployed one instance of the appliance at the primary DC and added both arrays to it to get the client up and running. And yes, I agree, if I have a site down I’m probably not going to be super focused on storage provisioning activities at my secondary DC. But I do enjoy whinging about things when they don’t work the way I expected them in the first instance.

 

I’d read in previous versions that Linked Mode wasn’t supported, but figured this was no longer an issue as it’s not mentioned in the 6.5 Product Guide. This thread on ECN seems to back up what I suspect. I’d be keen to hear if other people have run into this issue.

 

EMC – Basics – Accessing RemotelyAnywhere on VNX with MCx

Many moons ago I wrote a brief article about accessing RemotelyAnywhere on the CX4. This was prompted by changes in Release 29 of FLARE that changed the access mechanism for remote console access on the SPs. I’ve been working on some VNX2s recently (or VNX with MCx – as EMC really would like them to be known), and I was curious as to whether the process was the same.

Pretty much, yep.

 

Overview

Nowadays there are a few ways to access RemotelyAnywhere on the VNX SP. There are a few different ports on the array that can be used, depending on your circumstances. In some environments, where you’re not allowed to touch the customer’s network with your own gear, the service port may be more appropriate. Here’s an image of the ports from EMC. The model of VNX you’re using will dictate the layout of the ports.

SP_mgmt_ports

You can go via:

  • the SP’s management port: http://<SP IP address>:9519;
  • the SP’s service port: https://128.221.1.250:9519 (SP A) or https://128.221.1.251:9519 (SP B); and
  • the SP’s serial port: https://192.168.1.1:9519 (this assumes you’re connected via serial already – more on that below).

 

Management Port

This is fairly straightforward, and you’ll need to be on a network that has access to the management ports.

 

Service Port

So, you’re probably already aware that the best way to connect to the service port is to set your laptop TCP/IP settings as follows:

  • IP Address – 128.221.1.249 or 128.221.1.254
  • Subnet Mask – 255.255.255.248
  • Default Gateway – leave blank
  • DNS server entries – leave blank

 

Serial Cable

If you want to connect via the serial cable, you’ll need to setup a PPP connection on your laptop. The following steps assume that you’ve got a USB to serial adapter and you’re using a Windows 7 machine.

Device Manager

  • Right Click on your Computer icon and Select Manage
  • Click on Device Manager
  • Expand Ports (COM & LPT)
  • Look for the USB-to-Serial Comm Port (COM##)
  • The COMM number will be the one you will select during the configuration of your PPP connection.

 

Create the COM Port

  • Click Start -> Control Panel -> Phone and Modem.
  • Click the “modem tab” and click Add.
  • On the Install new Modem Pane, select the Don’t detect my modem box, then click Next.
  • Select Communications cable between two computers, then click Next.
  • Select the COM port from the previous step, then click Next.
  • Click Finish.
  • Highlight the new modem and click Properties.
  • Select the “modem tab”  and adjust the max speed to 115200 then click OK.
  • Click OK again to exit the Phone and Modem screen.
  • In the Computer Management window, disable and then re-enable the USB Serial connection in Device Manager. Do this by right-clicking on it.

 

Setting Up the PPP Connection

  • Click Start -> Control Panel -> Network and Sharing Center, click Set up a new connection or network (at the bottom).
  • Click Next, select Set up a dial-up connection and click Next.
  • This screen should list modems and select the Communications cable between two computers created above.
  • On the next screen put in a random phone number.  This is required in order to complete this step. You need at least one digit, but you’ll remove it later.  Next put in the username and password and give it a name. Then click on Connect. This connection will fail displaying: Connection Failed with error 777. Click on Set up the connection anyway.
  • You will get: “The connection to the Internet is ready to use”. Select Close.
  • The above connection should now appear in Network Connections. Open Control Panel and select Change Adapter Settings.
  • Right-click on your new Modem connection and select Cancel as Default Connection.
  • Right Click on your new Modem connection again and select Properties.

 

Modify the Settings

  • In the General tab, remove the phone number entry and leave it blank.
  • In the General tab, click configure and set Max speed to 115200 and select enable hardware flow control.
  • In the Options tab, click PPP settings and check that the top two boxes are selected (LCP extensions and SW compression).
  • In the Security tab, check Data Encryption is Optional Encryption (connect even if no encryption) is set.
    In the Networking tab, check Internet Protocol Version 4 is selected and click on Properties.
  • In the Networking tab, choose Internet Protocol Properties, then the Advanced button. Uncheck Use default gateway on remote network.
  • Click OK.
  • Click OK.
  • Click OK.

 

Conclusion

Here’s what it looks like when you log in – enjoy.

screenshot2

EMC – VNX2, Unisphere and Java Support

In my current role, I don’t do a lot of array configuration from scratch anymore. I generally do the detailed design and hand it over to a colleague to go and make it so. Recently, however, I’ve had to step in and do some actual work myself because reasons. Anyway, I was deploying a new VNX5400 and having a heck of a time getting Unisphere to work. And by Unisphere I mean Java. I initially wanted to blame my work-issued Windows 8.1 laptop, but it was ultimately a Java issue. It turns out my Java version was high. Not Cypress Hill high, but still too high for Unisphere.

EMC’s snappily titled “EMC VNX Operating Environment for Block 05.33.006.5.096, EMC VNX Operating Environment for File 8.1.6.96, EMC Unisphere 1.3.6.1.0096 Release Notes” talks fairly explicitly about Java support on page 6, and I thought it was worth repeating here for schmucks like me who do this stuff part-time. You can find this document on the EMC support site.

“Java support

The following 32 bit Java Platforms are verified by EMC and compatible for use with Unisphere, the Unified Service Manager (USM), and the VNX Installation Assistant (VIA):

  • Oracle Standard Edition 1.7 up to Update 75
  • Oracle Standard Edition 1.8 up to Update 25

The 32-bit JRE is required – even on 64 bit systems. JRE Standard Edition 1.6 is not recommended because Oracle has stopped support for this edition”.

I think I was running 1.8 Update 31, and saw that, regardless of the browser, Unisphere just wouldn’t load. If you need to track down an older version of Java to work on stuff like this – Oracle has a site you can go to here. Incidentally, I can confirm that it is not necessary to install the Ask Toolbar in order for Unisphere to function correctly.

*Update (2016.05.20): Current link to 1.8 U25 is here.

*Update (2016.06.04): Jirah Cox (@vJirah) pointed out that http://filehippo.com keeps an extensive archive of versions too.