CLARiiON CX700 FLARE Recovery – Part 3

In the previous series of posts, I covered the despair and ultimate joy of getting to a point where I could recover a munted CLARiiON CX700. In this post, I’ll cover the process to recover the array to a working state, and the steps required to get the array functioning at a useful level.

Having successfully performed a Utility Partition Boot, it’s ecessary to get the LAN service ports on the array configured in order to be able to ftp the recovery image to the array. Obviously, you’ll need the array and your service laptop plugged into a network-type thing that will enable frank communication between the arrays and you.

===============================================================================
CLARiiON Utility Toolkit Main Menu
===============================================================================
1) About the Utility Toolkit
2) About this Array
3) Reset Storage Processor
4) Image Repository Sub-Menu
5) Plugin Sub-Menu
6) NVRAM Sub-Menu
7) Enable LAN Service Port
8) Enable Engineering Mode
9) Install Images

Enter Option: 7
===============================================================================
Please enter the network settings you wish to use for this SP
===============================================================================
IP Address:  192.168.0.2
Subnet Mask:  255.255.255.0
Default Gateway:  192.168.0.255
Host Name:  spa
Domain Name:  

===============================================================================
Confirm Network Settings
===============================================================================
IP Address:      192.168.0.2  
Subnet Mask:     255.255.255.0
Default Gateway: 192.168.0.255
Host Name:       spa          
Domain Name:                  

Enable LAN Service Port with these settings? y/n [y] 
The LAN Service Port has been enabled

Automatically enable the LAN Port with these settings in the future? y/n [y] n

Press the Enter key to continue… 

Once you’ve enabled the LAN port on the SP you’re connected to, you need to ftp the image to the SP’s repository. The username to use is Clariion, and the password is clariion!. Once you’ve logged in, run a put command to put the file up there. It doesn’t really matter what you call it, but it should be a file of type mif. Here’s a pointless text capture of the ftp login process:

C:\>ftp 192.168.0.2
Connected to 192.168.0.2.
220-FileZilla Server version 0.8.3 beta test release 1
220-written by Tim Kosse (
[email protected])
220 Please visit
http://sourceforge.net/projects/filezilla/
User (192.168.0.2:(none)): Clariion
331 Password required for clariion
Password:
230 Logged on
ftp> ls
200 Port command successful
150 Opening data channel for directory list.
FLARE.mif
226 Transfer OK
ftp: 11 bytes received in 0.00Seconds 11000.00Kbytes/sec.
ftp>

Once you’ve successfully uploaded the recovery image, you’ll be good to go. It’s also important to note that the FLARE recovery image should be for the release that you intend to run. I didn’t consider uploading a Release 19 image, as I knew that these arrays had run Release 26 previously. In any case, jumping back into the Image menu on the terminal, it’s now time to copy the image from the RAM disk and then load it.

===============================================================================
CLARiiON Utility Toolkit Main Menu
===============================================================================
1) About the Utility Toolkit
2) About this Array
3) Reset Storage Processor
4) Image Repository Sub-Menu
5) Plugin Sub-Menu
6) NVRAM Sub-Menu
7) View LAN Service Port Settings
8) Enable Engineering Mode
9) Install Images

Enter Option: 4

===============================================================================
CLARiiON Utility Toolkit Image Repository Menu
===============================================================================
1) Back to the Main Menu
2) List Image Repository Contents
3) Delete Files from the Image Repository
4) Copy Files from the RAM Disk to the Image Repository
5) Copy Files from the Image Repository to the RAM Disk

Enter Option: 4

===============================================================================
Select files to copy to the Image Repository
===============================================================================
1) FLARE.mif

Enter comma separated list of options: 1

Copying FLARE.mif to the Image Repository… Success

Press the Enter key to continue…
 
===============================================================================
CLARiiON Utility Toolkit Image Repository Menu
===============================================================================
1) Back to the Main Menu
2) List Image Repository Contents
3) Delete Files from the Image Repository
4) Copy Files from the RAM Disk to the Image Repository
5) Copy Files from the Image Repository to the RAM Disk

Enter Option: 1

===============================================================================
CLARiiON Utility Toolkit Main Menu
===============================================================================
1) About the Utility Toolkit
2) About this Array
3) Reset Storage Processor
4) Image Repository Sub-Menu
5) Plugin Sub-Menu
6) NVRAM Sub-Menu
7) View LAN Service Port Settings
8) Enable Engineering Mode
9) Install Images

Enter Option: 9

===============================================================================
Select Images to Install
===============================================================================
1) FLARE.mif

Enter comma separated list of options: 1

===============================================================================
Confirm Image Installation
===============================================================================

  FLARE.mif

You need to install this only on the SP that you have visibility of, as troubleshooting the installation to both SPs is tricky.

Are you sure you want to install these images? y/n [n] y

===============================================================================
Select Storage Processors to install images for
===============================================================================
1) This SP (SP A)
2) Peer SP (SP B)
3) Both SP’s

Enter Option: 1
 
Installing Data Directory Boot Service 02.12
0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..100%
|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|  
***************************************************
The COPY operation has completed successfully.

Installing BIOS 03.70
0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..100%
|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|  
***************************************************
The COPY operation has completed successfully.

Installing Extended POST 02.38
0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..100%
|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|  
***************************************************
The COPY operation has completed successfully.

Installing FLARE Image 02.26.700.5.005
0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..100%
|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|  
***************************************************
The COPY operation has completed successfully.

Press the Enter key to continue… 

Once the copy has completed successfully, the system needs to be reset, and you’ll see the SP reboot up to three times before it’s useable.

===============================================================================
CLARiiON Utility Toolkit Main Menu
===============================================================================
1) About the Utility Toolkit
2) About this Array
3) Reset Storage Processor
4) Image Repository Sub-Menu
5) Plugin Sub-Menu
6) NVRAM Sub-Menu
7) View LAN Service Port Settings
8) Enable Engineering Mode
9) Install Images

Enter Option: 3

Requesting System Reset         

Once this is complete, you can either load the recovery image to the other SP via Navisphere in Engineering Mode, or you can use the same method as described above. Note that, once the image is copied to the repository, it is not necessary to re-upload it, as both SPs have access to the files.

The normal process needs to be followed as would normally be followed to initialise an array. In my case I initialised security, setup IP addresses for the SPs, logged in, committed the FLARE (R26.005), enabled Access Logix, configured cache settings, upgraded FLARE to the latest (R26.028), reloaded the latest Utility Partition and Recovery Images, and went about loading the appropriate enablers for the array. And now we have a working lab :)

32 Comments

  1. Great information! Exactly what I was looking for. Question is now, what if you didn’t have drives that contained a utility partition? Then what? Exactly my case.

  2. Hi Dave,

    This is what I haven’t been able to test yet. Although we’ll be coming into possession of a CX300 and CX3-20 arrays shortly, and I’d like to find out for myself. My hunch is that you’re going to need Vault disks from somewhere – possibly by hotsparing from a working array. I don’t know whether the arrays have enough code running on the SPs to bootstrap the utility partition otherwise, as my understanding was it’s spread across the first 4 disks. I’ll let you know when I find out.

  3. Dan,

    I have a CX300 here with no drives. It won’t boot the utility partition. Yes, you are correct. The boot ROM doesn’t have the capacity for it. I’m assuming you could take one drive from any of the 4 slots (Slot 0-3) in a working array and that would be enough to boot the utility partition. I’ll try that tomorrow. It is spread or is it mirrored? Drives slots 0-2 and 1-3.

    At first I thought they put the utility partition on the DCO/HPA areas of the drive. However, that is not the case. They do however hide code in these areas to keep you from using a standard off the shelf drive.

  4. Hi Dave,

    I was under the impression it was spread, although this seems like a silly thing for them to have done. I’m also assuming you could take any of the first 4 and boot something from there. But I won’t be able to confirm that for a few weeks. As I’m sure you’ll beat me to the punch on this one, let me know how it goes. Cheers

  5. Hmm.. well here’s one to throw into the mix. I have a CX3-20f with the first 5 Vault/OS disks.. however I’m using a different DAE to the origanal and cannot boot the utility partition.

    Looking at the DAE.. it’s sat on Bus ID 7 not 0.

    How on earth do you change that!

    Jim

  6. Just to add.. I was being a little thick on that last post, DAE is now at 0/0. However it seems my SP’s just cannot see the disks ( all 5 disks present and in the correct order and slots ).

    DAE connects fine, 4gb link lights on both LCC’s but from trying a variety of tests, SP’s simply cannot see the DAE or disks.

    Hope sombody had a bright idea! Seems a bit overkill for a paper weight.. : )

    Jim

    Running Fibre Loop Validation Algorithm (this might take a while)…
    Single drive loop configuration validation complete…
    Beginning iterative drive validation…

    ErrorCode: 0x0000018F
    ErrorDesc:
    FRU: All System FRUs
    Device: BOOT PATH
    Description: Data Directory Boot Service Not Loaded Error!
    Rev: 01.50
    EndError:
    ErrorTime: 02/01/2010 12:52:10

  7. Dan,

    You are correct. One drive would not boot the utility partition. Two drives did though. Drives 0 and 1. Now it looks like a have a faulty SP which doesn’t even seem to POST anymore.

  8. Dave,

    If you back out the faulty SP do you get the other one booting? Because you can use the ftp recovery image method to reload the code on the munted SP, assuming that it’s not hardware-related …

    Jim, what password do you use to access the menu? ie Do you get the full diagnostics? When I did this I had mismatched FLARE and DAE-OS, but it kind of just dealt with it. It’s something I’m going to test when I get hold of the lab CX3-20s, but that won’t be for at least a month or two.

  9. Hello, well it seems that the SP can see the DAE (reading serial’s seems to work from the Menu and i can power down drives) but booting the OS or Utility fails with the same error.

    I have another DAE coming today which was a 1st DAE in another setup so i’m hoping that magicly sorts things.. will keep you posted.

    Will post the menu later today, slightly different to your one ( option 33 was 13 in my case ).

    Jim

  10. Dan,

    It looks like I may have removed the SP when it was doing a BIOS update. I’m not sure. If I serial into the SP when power is applied it does nothing. The green light comes on but the amber light does nothing. I’m assuming since I can’t get to a menu or past BIOS stage 1 there would be no way to repair this SP. Is there a way to flash the BIOS from the other SP? I noticed there are no removable chips on the SP’s so swapping the chips would do me no good.

  11. Well here’s an update to my previous post. I loaded the .mif file to SPB but it only gets loaded to the disks. Since SPB won’t even POST there is no way to even get to a menu to flash the firmware or load images. I guess I am SOL on this SP since EMC didn’t put in a chip you can remove nor did they provide a way for us “commoners” to flash/load BIOS information. The SP has mulex connectors and such but unless it’s documented I’m afraid this SP is toast.

  12. Hi Dave,

    if you can POST SPA and boot the utility partition, it’s possible to at least get the image loaded onto SPB. The image repository is located on disk 4, and should be accessible to both SPs. I think the reason the guide recommends you do it to the SP you’re connected to is that
    a) you should be able to see the progress and look for errors on the copy
    b) you need to reset the SP once the image is loaded.

    SPA’s FLARE is on disks 0 and 2, Utility partition is on 1 and 3 and image repository is on disk 4. Reverse for SPB. Hope that helps …

  13. I loaded the images on SPB. However, it’s misleading because it actually isn’t loading it on SPB it’s loading it on the disks that SPB uses.

    The problem I’m having is that SPB won’t POST and has no output out of the serial connection. I think the PhoenixBIOS or something has been corrupt. The SP worked yesterday when I first put the drives in and then I believe I may have removed the power when it was updating stuff. Sigh.

  14. I noticed there is a Enable Engineering Mode area from the Utility Toolkit. Any idea what functions you have from this area/menu? Wondering if you can flash firmware. I know you can flash firmware from the SHIP_it menu’s but since I cannot get to a menu on this dead SP I can’t do anything.

  15. I tried looking at that, but the standard engineering password wouldn’t work. When you copy the recovery image to the SP from the repository, I know that it copies BIOS files as well as FLARE. So you I guess you could try copying the image, waiting some amount of time, and then resetting the SP. Does SPA boot Navisphere at all?

  16. True, but then you go into the image menu and select “Install Images”. Then there’s an option to select the SP you want to recover. Although I assume you’ve been down this road already.

  17. SPA is completely fine. SPB I am unable to do anything. No menus and no output to serial connection. So, that means no recovery. I loaded from SPA the images for SPB but as mentioned before it just loads that information to the SPB drives. Since this SP doesn’t boot it doesn’t even know about it. I need something that flashes directly to the SPB chips I’m assuming which you can do from the SHIP_it menus. However,since I can’t get to the menus on SPB, I cannot do it.

  18. Hello chaps, just an update my 2nd DAE turned up and upon connecting that as first everything booted fine.

    ( which is a little odd as the other DAE was set to 0/0 only thing i can think is that it was a Dell DAE as opposed to an EMC Dae? Have now added it as a 2nd DAE. So soon as I can get into Navisphere I’ll see what’s what ).

    Not quite out of the woods yet, so far only managed to connect to the IP off one of the SP’s. But it seems Navisphere does not get on with Win 7 x64 reagardless of which version of Java you have.

    Will now boot into the utility partion, pretty sure it’s booting fully (SP’s stop blinking amber and stay on green ) will grab an XP laptop and see how that fairs.

    Dave, Hope you recover your SP!

    Now if i could just find some cheap 300gb 15k disks…

  19. Well I got a new SP up and running. Ran into a problem though. It seems that my PSM is corrupt or not functioning correctly. I am unable to go to /setup and when I hit the IP of either SP navisphere eventually pops up with a message that says “Unable to access security information”. Using EMCremote I can see it has a problems talking to the PSM it looks like. I can’t uninitialize the array nor can I get in to kill the security information. Is there a back door to kill the PSM? I basically need to get back to an uninitialized array.

  20. Can someone email me this article? emc66373 I only have customer viewing and I can’t see the resolution.

  21. Dan,

    Thanks for the information. Unfortunately that document didn’t help. Using the procedures in the article I didn’t seem to have a LUN number of 1024. I have two LUNs using the ls command. I’m not sure if I have a PSM or not. I used your procedures for installing the .mif file to both SP’s. The funny part is when you recover both SP’s you would think that the storage array would be “uninitialized” but this is not the case. It comes up with the same SP names and IP addresses after boot. There has to be a way to uninitialize the array without using the /setup

    I’ll start reviewing the logs in more detail.

  22. Hello Dave,

    Not sure if it helps… but i had major problems with Java not showing the correct web pages.

    Once I grabbed an old version of JRE i could get into the /setup and kill the domain security ( allthough you also have to change the IP’s despite wiping everything ).

    Jim

  23. Hello Im have some problem with restart to default ex200. I run array and put “esc” write SHIP_it
    but this runing diagnostec menu – all diagnostic is sucessful.When array started -> http://paste.debian.net/60022/ and restart…. wher is or what is a step to a restart to default optnions array cx200

    Diagnostic Menu

    1) Reset Controller 17) BE0 FCC Sub-Menu
    2) Enter Debugger 18) PP0 FCC Sub-Menu
    3) Display Warnings/Errors 19) PP1 FCC Sub-Menu
    4) Boot OS 20) FE0 FCC Sub-Menu
    5) POST Sub-Menu 21) POST ROM Sub-Menu
    6) Display/Change Privilege 22) BIOS ROM Sub-Menu
    7) Boot UProc Sub-Menu 23) System Test Sub-Menu
    8) LRU Sub-Menu 24) Image Sub-Menu
    9) RAM Sub-Menu 25) Disk Sub-Menu
    10) NOVRAM Sub-Menu 26) Resume PROM Sub-Menu
    11) Real Time Clock Sub-Menu 27) Voltage Margin Sub-Menu
    12) Console UART Sub-Menu 28) Information Display
    13) SPS UART Sub-Menu 29) ICA Sub-Menu
    14) LCC 0 UART Sub-Menu 30) DDBS Service Sub-Menu
    15) Debug Port UART Sub-Menu 31) JBOD Mode
    16) LAN Service Port Sub-Menu

    Enter Option :

  24. Hi there,

    You need to:
    – boot the array from the utility partition;
    – enter engineering mode;
    – select “Invalidate Data Directory” in the engineering mode menu;
    – install an appropriate recovery image on the SPs (in your case I imagine this would be R19);
    – reboot the arrays.

    Hope that helps

  25. I have a Cx3-80, and I need to reimage SPA. The system does not contain a valid image, so I have to throw one up on the ftp. However, the standard clariion user name and password doesnt work. I’ve tried Clariion/ clariion!, clariion/ clariion!, clariion/ clariion1992, and clariion1992/ clariion1992. Any ideas? Is there another way to get the mif file on the system besides using the ftp? SPB is working fine.

  26. Hi,
    we have a problem with CX200 array – we want to bring it to the defaults, so we could connect to it via NaviSphere. Until now we only managed to connect to it via rs232 and gain access to the diagnostic menu (SHIP_it, DB_key). Probably the spa spb system image are damaged (image checksum error). We have the CX200-Bundle-02.19.200.5.045.pbu system image, but we don’t have the FLARE.mif file (we don’t even know if it’s needed). We are able to set a LAN FTP connection to the storage, so we can upload any image (via default login/password Clariion/clariion!) but to install it we’ll need a “Enable engineering mode” password (which we do not have – we don’t know the default password and don’t know if it was changed).

    The main problem is that the array manager isn’t starting. Theoretically via the rs232 connection (com1/ppp) we should have access to storage address (192.168.1.1) but unfortunately the default user/password (for ftp) in this case doesn’t word. In every information we managed to find there’s this “flare.mif” and it’s recovery.

    Do you have any idea’s that could help us? Maybe you could send us this “flare.mif”?

    Regards
    Justyn

  27. We were able to run but with a flare CX200 version R16. R19 flare after loading matrix restarts all the time .. little idea? Can flare is damaged? : (

  28. I was able to access the ftp through the command prompt using the standard passwords (clariion/clariion!)and successfully transferred my mif file and the 3-80’s been working fine for a few weeks now. The password still doesnt work through explorer or third party ftp client’s. At least there is a work around….

  29. this is all goopd stuff. i have 3 x cx400’s and 6 spare sp’s. i am on flare 13 :( my SPA is dead. bad flare in 1-3 . my 0-2 boots into spa fine and navi loads. booting up to utility on sp b 1-3 is okay but it says no image. i can not find any images. i also have disks that say wwn seed do not match. where can i obtain flare 13 or 14.mif. i have full ndu for 14, 16, &19 but no .mif. [email protected]. also how can someone change the wwn and seed info?

  30. John, you should be able to install the R13 recovery image via NDU in Navisphere in Engineering Mode – assuming you have a bootable SP. If you don’t, you’ll need to use the ftp method. It’s important that the recovery image being loaded is the same release as the surviving SP. Let me know if you have trouble getting hold of the R13 mif file. To change seed info, you’ll need to be in Engineering Mode again, right-click the array, Engineering Mode, Change WWN Seed. The “Change WWN Seed” window will display.

  31. Does anyone have a procedure for installing to a fresh set of vault drives? (No utility partition to boot to)

    Pls. email jg (at) 50micron.com or post here.

    For reference, got a CX3-20i Head I’d like to make into a real CX3-20i so I can play with it.

    Thanks. :)

  32. Jesse,

    I have to replace one of our CX700s with a CX3-20 in the next few weeks – I’ll see if I can get it working then. FWIW I thought the redneck IT post was gold :)

    Dan

Comments are closed.