QNAP – TR-004 Firmware Issue Workaround

I’ve been a user of QNAP products for over 10 years now. I have a couple of home systems running at the moment, including a TS-831X with a TR-004 enclosure attached to it. Last week I was prompted to update the external enclosure firmware to 1.1.0. After I did that, I had an issue where, once the unit spun down its disks, the volume would be marked as “Not active” by the system and I’d lose access to the data. Recovery was simple enough – I could either reboot the box or manually recover the enclosure via the QTS interface. I raised a job with QNAP web support, and we went back and forth with troubleshooting over the course of a week. The ticket was eventually escalated, and it was acknowledged that the current fix was to rollback to version 1.0.4 of the enclosure firmware.

The box is only used for media storage for Plex, but I figured it was worth backing up the contents of the external enclosure to another location in case something went wrong with the rollback. In any case, I’ve not done a downgrade on a QNAP device before, so I thought it was worth documenting the procedure here.

For some reason I needed to use Chrome over Safari in this example. I don’t know why that is. But whatever. In QTS, click on Storage & Snapshots, then Storage. Click on External RAID Management and then click on Check for Update.

You’ll see in this example, the installed TR-004 version is 1.1.0. Click on Browse to get the firmware file you want to roll back to.

You’ll get a stern warning that this kind of thing might cause problems.

Take a backup. Then tick the box.

The update will progress. It doesn’t take too long.

You then need to power off the enclosure and power it back on.

And, hopefully, your data will still be there. One side effect I noted was that the shared folder on that particular volume no longer had the correct permissions associated with the share. Fortunately, this is a home environment, and I’m using one user account to provide access to the share. I don’t know what you’d do if you had a complicated permissions situation in place.

And there you go. Like most things with QNAP, it’s a fairly simple process. This is the first time I’ve had to use QNAP support, and I found them responsive and helpful. I’ll report back if I get any other issues with the enclosure.

FalconStor Announces StorSafe

Remember FalconStor? You might have used its VTL product years ago? Or perhaps the Network Storage Server product? Anyway, it’s still around, and recently announced a new product. I had the opportunity to speak to Todd Brooks (CEO) and David Morris (VP Products) to discuss StorSafe, something FalconStor is positioning as “the industry’s first enterprise-class persistent data storage container”.

 

What Is It?

StorSafe is essentially a way to store data via containers. It has the following features:

  • Long-term archive storage capacity reduction drives low cost;
  • Multi-cloud archive storage;
  • Automatic archive integrity validation & journaling in the cloud;
  • Data egress fee optimisation; and
  • Unified Management and Analytics Console.

Persistent Virtual Storage Container

StorSafe is a bit different to the type of container you might expect from a company with VTL heritage.

  • Does not rely on traditional tape formats, e.g. LTO constraints
  • Variable Payload Capacity of Archival Optimisation by Type
  • Execution capabilities for Advanced Features
  • Encryption, Compression, and Best-in-Class Deduplication
  • Erasure coding for Redundancy across On-premise/Clouds
  • Portable – Transfer Container to Storage System or any S3 Cloud
  • Archival Retention for 10, 25, 50, & 100 years

[image courtesy of FalconStor]

Multi-Cloud Erasure Coding

  • The VSC is sharded into multiple Mini-Containers that are protected with Erasure Coding
  • These Mini-Containers can then be moved to multiple local, private data centres, or cloud destinations for archive
  • Tier Containers depending on Access Criticality or Limited Access needs

[image courtesy of FalconStor]

 

Thoughts And Further Reading

If you’re familiar with my announcement posts, you’ll know that I try to touch on the highlights provided to me by the vendor about its product, and add my own interpretation. I feel like I haven’t really done StorSafe justice however. It’s a cool idea, in a lot of ways. This idea that you can take a bunch of storage and dump it all over the place in a distributed fashion and have it be highly accessible and resilient. This isn’t designed for high performance storage requirements. This is very much focused on the kinds of data you’d be keen to store long-term, maybe on tape. I can’t tell you what this looks like from an implementation or performance perspective, so I can’t tell you whether the execution matches up with the idea that Falconstor has had. I find the promise of portability, particularly for data that you want to keep for a long time, extremely compelling. So let’s agree that this idea seems interesting, and watch this space for more on this as I learn more about it. You can read the press release here, and check out Mellor’s take on it here.

Random Short Take #31

Welcome to Random Short Take #31. Lot of good players have worn 31 in the NBA. You’d think I’d call this the Reggie edition (and I appreciate him more after watching Winning Time), but this one belongs to Brent Barry. This may be related to some recency bias I have, based on the fact that Brent is a commentator in NBA 2K19, but I digress …

  • Late last year I wrote about Scale Computing’s big bet on a small form factor. Scale Computing recently announced that Jerry’s Foods is using the HE150 solution for in-store computing.
  • I find Plex to be a pretty rock solid application experience, and most of the problems I’ve had with it have been client-related. I recently had a problem with a server update that borked my installation though, and had to roll back. Here’s the quick and dirty way to do that on macOS.
  • Here’s are 7 contentious thoughts on data protection from Preston. I think there are some great ideas here and I recommend taking the time to read this article.
  • I recently had the chance to speak with Michael Jack from Datadobi about the company’s announcement about its new DIY Starter Pack for NAS migrations. Whilst it seems that the professional services market for NAS migrations has diminished over the last few years, there’s still plenty of data out there that needs to be moved from on box to another. Robocopy and rsync aren’t always the best option when you need to move this much data around.
  • There are a bunch of things that people need to learn to do operations well. A lot of them are learnt the hard way. This is a great list from Jan Schaumann.
  • Analyst firms are sometimes misunderstood. My friend Enrico Signoretti has been working at GigaOm for a little while now, and I really enjoyed this article on the thinking behind the GigaOm Radar.
  • Nexsan recently announced some enhancements to its “BEAST” storage platforms. You can read more on that here.
  • Alastair isn’t just a great writer and moustache aficionado, he’s also a trainer across a number of IT disciplines, including AWS. He recently posted this useful article on what AWS newcomers can expect when it comes to managing EC2 instances.

WekaIO And A Fresh Approach

Disclaimer: I recently attended Storage Field Day 19.  My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event.  Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

WekaIO recently presented at Storage Field Day 19. You can see videos of their presentation here, and download my rough notes from here.

 

More Data And New Architectures

Liran Zvibel (Co-founder and CEO) spent some time talking about the explosion in data storage requirements in the next 4 – 5 years. It was suggested that most of this growth will come in the form of unstructured data. The problem with today’s storage systems, he suggested, was that storage is broken into “Islands of Compromise” categories – each category carries a leader. What does that mean exactly? DAS and SAN cannot share data easily, and the performance of a number of NAS and Object architectures isn’t great.

A New Storage Category

WekaIO is positioning itself in a new storage category. One that delivers:

  • The highest performance for any workload
  • Complete data shareability
  • Cloud native, hybrid cloud support
  • Full enterprise features
  • Simple management

Unique Product Differentiation

So what is that sets WekaIO apart from the rest of the storage industry? Zvibel listed a number of differentiators, including:

  • Only POSIX namespace that scales to exabytes of capacity and trillions of files
  • Only networked file system that is faster than local storage
    • Massively parallel
    • Lowest latency
  • Snap to object
    • Unique blend of All-Flash and Object storage for instant backup to cloud storage (no backup software required)
  • Cloud burst from on-premises to public cloud
    • Fully hybrid cloud enabled with highest performance
  • End-to-end data encryption with no performance degradation
    • Critical for modern workloads and compliance

[image courtesy of Barbara Murphy]

 

Customer Examples

This all sounds great, but where is WekaIO really being used effectively? Barbara Murphy spent some time talking with the delegates about a number of customer examples across the following market verticals.

Life sciences

  • Genomics sequencing and analytics
  • Drug discovery
  • Microscopy

Deep Learning

  • Machine Learning / Artificial Intelligence
  • Real-time analytics
  • IoT

 

Thoughts and Further Reading

I’ve written enthusiastically about WekaIO before. It’s easy to get caught up in some of the hype that seems to go hand in hand with WekaIO presentations. But WekaIO has a lot of data to back up its claims, and it’s taken an interesting approach to solving traditional storage problems in a non-traditional fashion. I like that there’s a strong cloud story there, as well as the potential to leverage the latest hardware advancements to deliver the performance companies need.

The analysts and storage vendors drone on and on about the explosion in data growth over the coming years, but it’s a real problem. Our workload challenges are changing as well, and it seems like a new approach is needed for how we approach some of these challenges. The scale of the data that needs to be crunched doesn’t always mean that DAS is a good option. You’re more likely to see these kinds of challenges show up in the science and technology industries. And WekaIO seems to be well-positioned to meet these challenges, whether it’s in public cloud or on-premises. It strikes me that WekaIO’s focus on performance and resilience, along with a robust software-defined architecture, has it in a good position to tackle the types of workload problems we’re seeing at the edge and in AI / ML focused environments. I’m really looking forward to seeing what comes next for WekaIO.

Random Short Take #30

Welcome to Random Short Take #30. You’d think 30 would be an easy choice, given how much I like Wardell Curry II, but for this one I’m giving a shout out to Rasheed Wallace instead. I’m a big fan of ‘Sheed. I hope you all enjoy these little trips down NBA memory lane. Here we go.

  • Veeam 10’s release is imminent. Anthony has been doing a bang up job covering some of the enhancements in the product. This article was particularly interesting because I work in a company selling Veeam and using vCloud Director.
  • Sticking with data protection, Curtis wrote an insightful article on backups and frequency.
  • If you’re in Europe or parts of the US (or can get there easily), like writing about technology, and you’re into cars and stuff, this offer from Cohesity could be right up your alley.
  • I was lucky enough to have a chat with Sheng Liang from Rancher Labs a few weeks ago about how it’s going in the market. I’m relatively Kubernetes illiterate, but it sounds like there’s a bit going on.
  • For something completely different, this article from Christian on Raspberry Pi, volumio and HiFiBerry was great. Thanks for the tip!
  • Spinning disk may be as dead as tape, if these numbers are anything to go by.
  • This was a great article from Matt Crape on home lab planning.
  • Speaking of home labs, Shanks posted an interesting article on what he has running. The custom-built rack is inspired.

Dell EMC, DevOps, And The World Of Infrastructure Automation

Disclaimer: I recently attended Storage Field Day 19.  My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event.  Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

Dell EMC recently presented at Storage Field Day 19. You can see videos of the presentation here, and download my rough notes from here.

 

Silos? We Don’t Need No Silos

The data centre is changing, as is the way we manage it. There’s been an observable evolution of the applications we run in the DC and a need for better tools. The traditional approach to managing infrastructure, with siloed teams of storage, network, and compute administrators, is also becoming less common. One of the key parts of this story is the growing need for automation. As operational organisations in charge of infrastructure and applications, we want to:

  • Manage large scale operations across the hybrid cloud;
  • Enable DevOps and CI/CD models with infrastructure as code (operational discipline); and
  • Deliver self service experience.

Automation has certainly gotten easier, and as an industry we’re moving from brute force scripting to assembling pre-built modules.

 

Enablers for Dell EMC Storage (for Programmers)

REST

All of our automation Power Tools use REST

  • Arrays have a REST API
  • REST APIs are versioned APIs
  • Organised by resource for simple navigation

Secure

  • HTTPS, TLS 1.2 or higher
  • Username / password or token based
  • Granular RBAC

With REST, development is accelerated

 

Ansible for Storage?

Ansible is a pretty cool automation engine that’s already in use in a lot of organisations.

Minimal Setup

  • Install from yum or apt-get on a Linux server / VM
  • No agents anywhere

Low bar of entry to automation

  • Near zero programming
  • Simple syntax

 

Dell EMC and vRO for storage

VMware’s vRealize Orchestrator has been around for some time. It has a terrible name, but does deliver on its promise of simple automation for VMware environments.

  • Plugins allow full automation, from storage to VM
  • Easily integrated with other automation tools

The cool thing about the plugin is that you can replace homegrown scripts with a pre-written set of plugins fully supported by Dell EMC.

You can also use vRO to implement automated policy based workflows:

  • Automatic extension of datastores;
  • Configure storage the same way every time; and
  • Tracking of operations in a single place.

vRO plugs in to vRealize Automation as well, giving you self service catalogue capabilities along with support for quotas and roles.

What does the vRO plugin support?

Supported Arrays

  • PowerMax / VMAX All-Flash (Enterprise)
  • Unity (Midrange)
  • XtremIO

Storage Provisioning Operations

  • Adds
  • Moves
  • Changes

Array Level Data Protection Services

  • Snapshots
  • Remote replication

 

Thoughts and Further Reading

DevOps means a lot of things to a lot of people. Which is a bit weird, because some smart folks have written a handbook that lays it all out for us to understand. But the point is that automation is a big part of what makes DevOps work at a functional level. The key to a successful automation plan, though, is that you need to understand what you want to automate, and why you want to automate it. There’s no point automating every process in your organisation if you don’t understand why you do that process in the first place.

Does the presence of a vRO plugin mean that Dell EMC will make it super easy for you to automate daily operations in your storage environment? Potentially. As long as you understand the need for those operations and they’re serving a function in your organisation. I’m waffling, I know, but the point I’m attempting to make is that having a tool bag / shed / whatever is great, and automating daily processes is great, but the most successful operations environments are mature enough to understand not just the how but the why. Taking what you do every day and automating it can be a terrifically time-consuming activity. The important thing to understand is why you do that activity in the first place.

I’m really pleased that Dell EMC has made this level of functionality available to end users of its storage platforms. Storage administration and operations can still be a complicated endeavour, regardless of whether you’re a storage administrator comfortably ensconced in an operational silo, or one of those cool site reliability engineers wearing jeans to work every day and looking after thousands of cloud-native apps. I don’t think it’s the final version of what these tools look like, or what Dell EMC want to deliver in terms of functionality, but it’s definitely a step in the right direction.

MinIO – Not Your Father’s Object Storage Platform

Disclaimer: I recently attended Storage Field Day 19.  My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event.  Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

MinIO recently presented at Storage Field Day 19. You can see videos of the presentation here, and download my rough notes from here.

 

MinIO – A Sharp Looking 3-Piece Suite

AB Periasamy spoke to the delegates first, describing MinIO as “a high performance, software-defined, distributed object storage server, designed for peta-scale data infrastructure”. It was built from scratch with the private cloud as its target and is comprised of three components:

  • MinIO Server
  • MinIO Client
  • MinIO SDK

He noted that “the private cloud is a very different beast to the public cloud”.

Why Object?

The MinIO founders felt strongly that data would continue to grow, S3 would overtake POSIX, and that the bulk of data would exist outside of AWS. It’s his opinion that private cloud finally started emerging as a real platform last year.

 

Architecture

A number of guiding principles were adopted by MinIO when designing the platform. MinIO is:

  • Focused on performance. They believe it is the fastest object store in existence;
  • Cloud native. It is the most K8s-friendly solution available for the private cloud;
  • 100% open-source enables an increasingly dominant position in the enterprise;
  • Built for scale using the same philosophy as web scalers; and
  • Designed for simplicity. Simplicity scales – across clients, clouds, and machines.

 

[image courtesy of MinIO]

 

Other Features

Some of the other key features MinIO is known for include:

  • Scalability;
  • Support for erasure coding;
  • Identity and Access Management capability;
  • Encryption; and
  • Lifecycle Management.

MinIO is written in Go and is 100% open source. “The idea of holding customers hostage with a license key – those days are over”.

 

Deployment Use Cases

MinIO delivers usable object storage capability in all of the places you would expect it to.

  • Big Data / Machine Learning environments
  • HDFS replacements
  • High performance data lake / warehouse infrastructure
  • Cloud native applications (replacing file and block)
  • Multi-cloud environments (portability)
  • Endpoint for streaming workloads

 

Thoughts and Further Reading

If you watch the MinIO presentation, or check my notes, you’ll see a lot of slides with some impressive numbers in terms of both performance and market penetration. MinIO is not your standard object storage stack. A number of really quite big customers use it internally to service their object storage requirements. And, because it’s open source, a whole lot of people are really curious about how it all works, and have taken it for a spin at some stage or another. The story here isn’t that MinIO is architecturally a bit different from some other vendors’ storage offerings. Rather, it’s the fact that it’s open source and accessible to every punter who wants to grab it. This is exactly the reason why neckbeards get excited about open source products. Because you can take a core infrastructure function, and build a product that does something extremely useful from it. And you can contribute back to the community.

The big question, though, is how to make money of this kind of operating model. A well known software company made a pretty decent stab at leveraging open source products as a business model, delivering enhanced support services as a way to keep the cash coming in. This is very much what MinIO is doing as well. It has a number of very big customers willing to pay for an enhanced support experience via a subscription. It’s an interesting idea. Come up with a product that does what it says it will quite well. Make it easy to get hold of. Help big companies adopt it at scale. Then keep them up and running when said open source code becomes a mission critical piece of their business workflow. I want this model to work, I really do. And I have no evidence to say that it won’t. The folks at MinIO were pretty confident about what they could deliver with SUBNET in terms of the return on investment. I’m optimistic that MinIO will be around for a while longer, as the product looks the goods, and the people behind the product have spent some time thinking through what this will look like in the future. I also recommend checking out Chin-Fah’s recent article for another perspective.

Tiger Technology Is Bridging The Gap

Disclaimer: I recently attended Storage Field Day 19.  My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event.  Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

Tiger Technology recently presented at Storage Field Day 19. You can see videos of the presentation here, and download my rough notes from here.

 

What’s A Tiger Then?

Tiger Technology has been around since 2004. It’s been primarily focused on the world of media and entertainment and the myriad of storage problems (opportunities?) associated with that market segment. Tiger Technology is:

  • Privately held;
  • Based in Bulgaria and the USA;
  • Has around 50 employees; and
  • Has more than 3000 customers worldwide.

As I mentioned earlier, they’ve been focused on the media and entertainment segment. This is on of the most demanding markets, with applications that are:

  • Mission critical;
  • High-bandwidth;
  • Low-latency;
  • Leveraging application specific functions + non-standard implementation;
  • Time-sensitive; and
  • Intolerant of data loss.

 

Tiger Bridge

So what exactly does Tiger Bridge do? It might be better to think of it in terms of what it doesn’t do. There’s capacity management, cloud integration, hierarchical storage management, and some cool collaboration options. It’s a filter driver that lets you talk between all kinds of storage platforms via a global namespace. Key features include:

  • Software only
  • Transparent
  • Seamless
  • Native
  • Unified storage space
  • Multi-cloud, multi-tier
  • Non-proprietary, no vendor lock
  • No external cache
  • Full Active Directory security features

 

Tiger Bridge Architecture

Alexander Lefterov took us through some of the initial technical and architectural choices that were made as part of the product development, including:

  • File vs Block
  • Internal to the data flow vs external
  • Native vs cross platform
  • File system vs database
  • Open vs proprietary format on target
  • System administrator vs end user

[image courtesy of Tiger Technology]

 

Thoughts and Further Reading

I first came across Tiger Technology when speaking to Backblaze about an integration that had been developed between B2 and Veeam. It strikes me that Tiger Technology is pretty good at gluing storage together to make it do what people need it to do. The appeal of cloud storage is that you can theoretically expand and shrink the capacity you consume based on your workload requirements on any given day. Unfortunately, it’s not always a simple exercise to put data where you need it in a timely fashion. Tiger Bridge enables you to easily access cloud storage, and move data between on-premises storage and the cloud.

There are a bunch of cool storage solutions available on the market, servicing all kinds of workload requirements. Some are fast, some are slow, some are expensive, and some are cheap. But sometimes you need to do stuff with storage that might reside both on-premises and in the cloud. In that case, you want the experience to be non-intrusive from a usability perspective. My father-in-law always talks about the importance of the user experience versus wasting time trying to get the technology working as expected. It strikes me that Tiger Technology has done a good job of ensuring that the bridge between on-premises storage and multiple cloud storage options is a fairly robust one.

People sometimes find themselves trying to do things that aren’t natively supported by the technologies they’re using. Oftentimes the response from the technology vendor is to change how the user is doing a particular activity. If you’ve worked in an enterprise service management environment, you may have heard this being trotted out by the service management consultant. But I think it’s a bit rich to expect that users should build their business activities based on the capabilities of the technology, and not the other way around. Solutions like Tiger Bridge are a nice compromise between how users want to work, and how the storage vendors deliver their technology to those users. It’s not a solution that’s going to work for everyone, but if you have the need, I think it’s worth checking out.

InfiniteIO And Your Data – Making Meta Better

InfiniteIO recently announced its new Application Accelerator. I had the opportunity to speak about the news with Liem Nguyen (VP of Marketing) and Kris Meier (VP of Product Management) from InfiniteIO and thought I’d share some thoughts here.

 

Metadata Is Good, And Bad

When you think about file metadata you might think about photos and the information they store that tells you about where the photo was taken, when it was taken, and the kind of camera used. Or you might think of an audio file and the metadata that it contains, such as the artist name, year of release, track number, and so on. Metadata is a really useful thing that tells us an awful lot about data we’re storing. But things like simple file read operations make use of a lot of metadata just to open the file:

  • During the typical file read, 7 out of 8 operations are metadata requests which significantly increases latency; and
  • Up to 90% of all requests going to NAS systems are for metadata.

[image courtesy of InfiniteIO]

 

Fire Up The Metadata Engine

Imagine how much faster storage would be if it only has to service 10% of the requests it does today? The Application Accelerator helps with this by:

  • Separating metadata request processing from file I/O
  • Responding directly to metadata requests at the speed of DRAM – much faster than a file system

[image courtesy of InfiniteIO]

The cool thing is it’s a simple deployment – installed like a network switch requiring no changes to workflows.

 

Thoughts and Further Reading

Metadata is a key part of information management. It provides data with a lot of extra information that makes that data more useful to applications that consume it and to the end users of those applications. But this metadata has a cost associated with it. You don’t think about the amount of activity that happens with simple file operations, but there is a lot going on. It gets worse when you look at activities like AI training and software build operations. The point of a solution like the Application Accelerator is that, according to InfiniteIO, your primary storage devices could be performing at another level if another device was doing the heavy lifting when it came to metadata operations.

Sure, it’s another box in the data centre, but the key to the Application Accelerator’s success is the software that sits on the platform. When I saw the name my initial reaction was that filesystem activities aren’t applications. But they really are, and more and more applications are leveraging data on those filesystems. If you could reduce the load on those filesystems to the extent that InfiniteIO suggest then the Application Accelerator becomes a critical piece of the puzzle.

You might not care about increasing the performance of your applications when accessing filesystem data. And that’s perfectly fine. But if you’re using a lot of applications that need high performance access to data, or your primary devices are struggling under the weight of your workload, then something like the Application Accelerator might be just what you need. For another view, Chris Mellor provided some typically comprehensive coverage here.

Random Short Take #26

Welcome to my semi-regular, random news post in a short format. This is #26. I was going to start naming them after my favourite basketball players. This one could be the Korver edition, for example. I don’t think that’ll last though. We’ll see. I’ll stop rambling now.