EMC – DIY Heatmaps

My friend Mat has developed a pretty cool script that can make pretty pictures out of Analyzer files from CLARiiON and VNX arrays. He’s decided to release it to the world, so you can download it here. I’ve also added the following instructions to a pdf document available here. Here’s a sample output file from the script. He’s after feedback as well, so send it through and I’ll make sure it gets to him.

Purpose:

The heatmap script allows you to generate heatmaps of various metrics over time from NAR files generated from EMC Clariion/VNX arrays.

Requirements/Notes:

  • This script was developed and tested using  Strawberry Perl (v5.12.3), but there is no reason it won’t work with other flavours or versions of Perl, but it is possible that other perl modulse other than the one listed below may need to be installed
  • It requires the  Text::CSV module for Perl.  Available here http://search.cpan.org/CPAN/authors/id/M/MA/MAKAMAKA/Text-CSV-1.21.tar.gz , if you can get cpan working  (run “cpan Text::CSV”) it will be easy to install, otherwise you’ll have to do it manually …

Unzip the files into eg C:\Text-CSV-1.21 and run the following commands

C:\Text-CSV-1.21>perl Makefile.PL

C:\Text-CSV-1.21>dmake

C:\Text-CSV-1.21>dmake test

C:\Text-CSV-1.21>dmake install

  • The script uses the Microsoft tool logparser.exe to manipulate various CSV files available here http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=24659
  • The script uses naviseccli to convert the NAR files into CSV files.
  • At the moment the script will only run on a windows server/PC due to its reliance on the logparser tool, if people are interested I will investigate other tools available in the Linux/Unix worlds.
  • The default min/max figures that I am using may be unrealistic, I am by no means an expert in analysing the performance of an array, any feedback is welcome.

Usage:

The following are the command line options available for the script

–logparser <path\filename>

–naviseccli <path\filename>

These options allow you to set the location of the various 3rd party binaries that the script relies on

–avg_interval <seconds>

The interval that the analyser stats are averaged down to, ie 1800 seconds will set the average interval to 30 minutes

–nar_file <filename>

The input NAR file(s), more than one NAR file can be specified

–out <filename>

The filename of the generated HTLM output file, the default is HEATMAP.HTML

–array_name <name>

The name of the array, this is for reporting purposes only

–summary

Generate a summary for each attribute type, displaying minimum, average and maximum figures for each metric

Metric

Min

Max

Avg

Disk Utilization (%)

0.27

97.38

8.16

Disk Total Throughput (IO/s)

0.72

309.06

23.03

SP Utilization (%)

38.03

61.11

49.78

SP Total Throughput (IO/s)

5588.70

25693.24

10088.56

–mash

–mashonly

This generated a mash table for each metric type (currently only Storage Processor and Disk), averaging each of the metrics for each type down into a single table, allowing you to combine multiple metrics into a single table.

The –mash option will display the mash-up table alongside the selected metrics, and the –mashonly options will only display the mash-up tables.

–get_drive_type

–array_ip <ip addresS>

This option allows you to query the array (as it is currently configured) to determine drive types (currently only SATA II, FIBRE CHANNEL, and SATA II SSD), currently this options only affects the IOPS calculations for drives.

–user_id <user id>

–pwd <password>

–scope <0|1>

These options are only used in conjunction with the –get_drive_type option, if you have cached credentials for the array configured then these options should not be necessary.

–display_drive_type

This option is only used in conjunction with the –get_drive_type option, it will display another drive table, and allow you to view the different drive type, drive size, pool and RAID Group layouts.

–disk_highlight

This option is only used in conjunction with the –get_drive_type and –display_drive_type options, it will highlight the corresponding disks in the other heatmaps, it will also allow you to select all of the drives that have the same attributes as displayed by the –display_drive_type option.

–config_file <filename>

–generate_config <filename>

These options allow you to generate a configuration file, using the –generate_config option, this will generate a config file with all of the default attributes, and to use a configuration file using the –config_file option.

–help

This option will display the following help information.

 

Heatmap Generator

 

Usage: heatmap.3.010.pl <options>

Where options can be the following:

–logparser <path\filename> – The path to the logpaser executable (c:/Program Files/Log Parser 2.2/LogParser.exe)

–naviseccli <path\filename> – The path to the naviseccli executable (c:/Program Files/EMC/Navisphere CLI/NaviSECCli.exe)

–avg_interval <seconds>     – The interval in seconds that the stats are averaged at (1800 seconds)

–nar_file <filename>        – NAR input file(s) this option can be specified multiple times

–out <filename>             – The output filename (heatmap.html)

–array_name <name>          – Set the name of the array in the report

–summary                    – Displays a summary of each metric

–mash                       – Creates a mash-up of each metric per object type, and displays alongside other metrics

–mashonly                   – Creates a mash-up of each metric per object type, and only displays the mash-ups

–get_drive_type             – Query Array to get drive type information

–array_ip <ip addresS>      – Set the IP address of the array

–user_id <user id>          – Set the user ID to log into the array

–pwd <password>             – Set the user password of the array

–scope <0|1>                – Set the Scope of the array account

–display_drive_type         – Display the drive types in the charts

–disk_highlight             – Highlight drives on mouse over

–config_file <filename>     – Use a configuration file to set attribute min/maxes

–generate_config <filename> – Generate a configuration file using the defined defaults

–help                       – This help

–attrib <attribute>         – Set the attribute to graph, where attribute can be the following

d_utilization – display stats based on disk utilization (%)

d_iops        – display stats based on disk Total Throughput (IOPS)

d_r_iops      – display stats based on disk read IOPS

d_w_iops      – display stats based on disk write IOPS

d_queue       – display stats based on disk Queue Length

d_b_queue     – display stats based on disk Average Busy Queue Length

d_response    – display stats based on disk Response Time

d_service     – display stats based on disk Service Time

d_bandwidth   – display stats based on disk Total Bandwidth

d_r_bandwidth – display stats based on disk Read Bandwidth

d_w_bandwidth – display stats based on disk Write Bandwidth

d_r_size      – display stats based on disk Read Size

d_w_size      – display stats based on disk Write Size

d_seek        – display stats based on disk Average Seek Distance

 

s_utilization – display stats based on SP utilization

s_response    – display stats based on SP Response Time

s_bandwidth   – display stats based on SP Total Bandwidth

s_iops        – display stats based on SP Total Throughput (IOPs)

s_b_queue     – display stats based on SP Average Busy Queue Length

s_service     – display stats based on SP Service Time

s_c_dirty     – display stats based on SP Cache Dirty Pages (%)

s_c_flush     – display stats based on SP Cache Flush Ratio

s_c_flush_mb  – display stats based on SP Cache MBs Flushed (MB/s)

s_c_hw_flush  – display stats based on SP Cache High Water Flush On

s_c_i_flush   – display stats based on SP Cache Idle Flush On

s_c_lw_flush  – display stats based on SP Cache Low Water Flush Off

s_wc_flush    – display stats based on SP Write Cache Flushes/s

s_fc_dirty    – display stats based on FAST Cache Dirty Pages (%)

s_fc_flush_mb – display stats based on FAST Cache MBs Flushed (MB/s)

 

Running the script without any –nar_file options will result in the script prompting the user to supply NAR file(s)

–attrib <attribute>

Allows you to select which attributes you want to display on the heatmap.

7 Comments

  1. Hi.

    I’ve got a heatmap.pl file from this website to make pretty pictures out of Analyzer files from CLARiiON and VNX array.

    I’m not a good at per script so could you send me the example how can I use the heatmap.pl with NAR file that is gathered from Clariion.

    Thanks in advance.

  2. I am assuming that you have perl installed and the Text::CSV module installed, as long as you have internet connectivity from your PC/server using cpan shouldnt be a problem

    1. its probably best to put the heatmap perl script and the nar file(s) that you want to process in the same directory
    2. “cd” to the directory that the heatmap perl script is located in.
    3. run the following command (I am assuming that when perl was installed it modified the path variable, and you should be able run the perl binary from anywhere, otherwise you’ll have to specify the path to the perl binary

    perl heatmap.3.010.pl –nar_file analyzer_nar_file.nar –attrib d_utilization –attrib d_iops –attrib s_utilization –attrib s_iops –summary

    where the command line parameters
    –nar_file analyzer_nar_file.nar – where analyzer_nar_file.nar is the NAR file that you want to analyze
    –attrib d_utilization
    –attrib d_iops
    –attrib s_utilization
    –attrib s_iops – the 4 –attrib options will display disk and SP utilization and IOPS heatmaps
    –summary – will produce a summary table

    this is a fairly basic run, as you can see from the readme there are alot more options, but just start off with something simple …. hopefully this helps, let me know if you have any other questions

  3. This is a very neat script. Would be neat to get this style as real time view in Unisphere Analyzer too.

    Kudos to you and friend for posting this!

  4. Hi Preston – thanks for the kind words. We hope you find it useful – and feel free to spread the word :)

  5. Hey,

    I am trying your script for the first time, I am using strawberry perl, I had no issues with the CPAN command, however the first time I run your script I get and error @ line 1182 “Type of arg 1 to keys must be hash (not array dereference)

    I did change your path setttings to the following:

    # path and filename pointing to microsoft logparser
    my $log_parser=”c:/Program\ Files\ \(x86\)/Log\ Parser\ 2.2/LogParser.exe”;

    # path and filename pointing to emc naviseccli
    my $naviseccli=”c:/Program\ Files\ \(x86\)/EMC/Navisphere\ CLI/NaviSECCli.exe”;

  6. hey Tommy,

    Unfortunately I’m not in front of a computer that I can do much debugging on, at first look the path settings seem ok – although the quotes around the path/filename are different from a standard double quote, but I’m guessing they are getting converted when being posted to the web.

    The area of code that the script is having a problem in is building a list of pool names, so I am guessing that you are using the “–get_drive_type” option. If this is the case could you try running the script without this option, preferably with just a basic set of options, something like

    perl heatmap.3.010.pl –nar_file analyzer_nar_file.nar –attrib d_utilization –attrib d_iops –attrib s_utilization –attrib s_iops –summary

    I’m guessing that you have a pool name that is causing some problems for script, are you able to e-mail me a copy of your modified script and if possible one of your NAR files, or a list of pool names to matharvest@gmail.com

    thanks

    Mat.

Comments are closed.