Arcitecta Announces Mediaflux Universal Data System

I had the opportunity to speak to Jason Lohrey and Robert Murphy from Arcitecta a little while ago about the company’s Mediaflux announcement. It was a great conversation, and I’m sad that I hadn’t heard about the company beforehand. In any case I figured I’d share some thoughts on the announcement.

 

What Is It?

The folks at Arcitecta describe the Mediaflux Universal Data System as “a convergence of data management, data orchestration, multi-protocol access, and storage in one platform”. The idea is that the system manages your data across all of your storage platforms. It’s not just clustered or distributed storage. It’s not just a control plane that gives you multi-protocol access to your storage platforms. It’s not just an orchestration engine that can move your data around as required. It’s all of these things and a bit more too. Features include:

  • Converges data management, orchestration and storage within a single platform – that’s right, it’s all in the one box.
  • Manages every aspect of the data lifecycle: On-premises and cloud, with globally distributed access.
  • Offers multi-protocol access and support. The system supports NFS, SMB, S3, SFTP and DICOM, among many others.
  • Empowers immense scalability. Mediaflux licensing is decoupled from the volume of data stored so organisations can affordably scale storage needs to hundreds of petabytes, accommodating hundreds of billions of files without the financial strain typically associated with such vast capacities. Note that Mediaflux’s pricing is based on the number of concurrent users.
  • Provides the option to forego third-party software and clustered file systems.
  • Supports multi-vendor storage environments, allowing customers to choose best-of-breed hardware.

Seem ambitious? Maybe, but it also seems like something that would be super useful.

 

Global Storage At A Worldwide Scale

One of the cool features of Mediaflux is how it handles distributed file systems, not just across data centres, but across continents. A key feature of the platform is the ability to deliver the same file system to every site.

[image courtesy of Arcitecta]

It has support for centralised file locking, as well as replication between sites. You can also configure variable retention policies for different site copies, giving you flexibility when it comes to how long you store your data in various locales. According to the folks at Arcitecta, it’s also happy to make the most of your bandwidth, and able to use up to 95% of the available bandwidth.

 

Thoughts And Further Reading

There have been a few data management / orchestration / unified control plane companies that have had a tilt at doing universal storage access well, across distances, and with support for multiple protocols. Sometimes the end result looks like an engineering project at best, and you have to hold your mouth right to have any hope of seeing your data again once you send it on its way. Putting these kinds of platforms together is no easy task, and that’s why this has been something of a journey for the team at Arcitecta. The company previously supported Mediaflux on top of third-party file and object systems, but customers needed a solution that was more flexible and affordable.

So why not just use the cloud? Because some people don’t like storing stuff in hyperscaler environments. And sometimes there’s a requirement for better performance than you can reasonably pay for in a cloud environment. And not every hyperscaler might have a presence where you want your data to be. All that said, if you do have data in the cloud, you can manage it with Mediaflux too.

I’m the first to admit that I haven’t had any recent experience with the type of storage systems that would benefit from something like Mediaflux, but on paper it solves a lot of the problems that enterprises come across when trying to make large datasets available across the globe, while managing the lifecycle of those datasets and keeping them readily available. Given some of the reference customers that are making use of the platform, it seems reasonable to assume that the company has been doing something right. As with all things storage, your mileage might vary, but if you’re running into roadblocks with the storage platforms you know and love, it might be time to talk to the nice people in Melbourne about what they can do for you. If you’d like to read more, you can download a Solution Brief as well.

Random Short Take #91

Squeezing this one in before the end of the year. It’s shorter than normal but we all have other things to do. Let’s get random.

  • Like the capacity and power consumption of tape but still want it on disk? Check out this coverage of the Disk Archive Corporation over at Blocks and Files.
  • This was a great series of posts on the RFC process. It doesn’t just happen by magic.
  • Jeff Geerling ran into some issues accessing media recently. It’s a stupid problem to have, and one of the reasons I’m still such a sucker for physical copies of things. I did giggle a bit when I first read the post though. These kind of issues come up frequently for folks outside the US thanks to content licensing challenges and studios just wanting us to keep paying for the same thing over and over again and not have any control over how we consume content.
  • My house was broken into recently. It’s a jarring experience at best. I never wanted to put cameras around my house, but now I have. If you do this in Queensland you can let the coppers know and they can ask for your help if there’s a crime in the area. I know it’s not very punk rock to surveil people but fuck those kids.
  • You didn’t think I’d get to 91 and not mention Dennis Rodman, did you? One of my top 5 favourite players of all time. Did everything on the court that I didn’t: played defence, grabbed rebounds, and gave many a high energy performance. So here’s some highlights on YouTube.

That’s it for this year. Stay safe, and see you in the future.

Random Short Take #90

Welcome to Random Short Take #90. I remain somewhat preoccupied with the day job and acquisitions. It’s definitely Summer here now. Let’s get random.

  • You do something for long enough, and invariably you assume that everyone else knows how to do that thing too. That’s why this article from Danny on data protection basics is so useful.
  • Speaking of data protection, Preston has a book on recovery for busy people coming soon. Read more about it here.
  • Still using a PDP-11 at home? Here’s a simple stack buffer overflow attack you can try.
  • I hate it when the machines shout at me, and so do a lot of other people it seems. JB has a nice write-up on the failure of self-service in the modern retail environment. The sooner we throw those things in the sea, the better.
  • In press release news, Hammerspace picked up an award at SC2023. One to keep an eye on.
  • In news from the day job, VMware Cloud on AWS SDDC Version 1.24 was just made generally available. You can read more about some of the new features (like Express Storage Architecture support – yay!) here. I hope to cover off some of that in more detail soon.
  • You like newsletters? Sign up for Justin’s weekly newsletter here. He does thinky stuff, and funny stuff too. It’s Justin, why would you not?
  • Speaking of newsletters, Anthony’s looking to get more subscribers to his daily newsletter, The Sizzle. To that end, he’s running a “Sizzlethon”. I know, it’s a pretty cool name. If you sign up using this link you also get a 90-day free trial. And the price of an annual subscription is very reasonable. There’s only a few days left, so get amongst it and let’s help content creators to keep creating content.

Random Short Take #89

Welcome to Random Short Take #89. I’ve been somewhat preoccupied with the day job and acquisitions. And the start of the NBA season. But Summer is almost here in the Antipodes. Let’s get random.

  • Jon Waite put out this article on how to deploy an automated Cassandra metrics cluster for VCD.
  • Chris Wahl wrote a great article on his thoughts on platform engineering as product design at scale. I’ve always found Chris to be a switched on chap, and his recent articles diving deeper into this topic have done nothing to change my mind.
  • Curtis and I have spoken about this previously, and he talks some more about the truth behind SaaS data recovery over at Gestalt IT. The only criticism I have for Curtis is that he’s just as much Mr Recovery as he is Mr Backup and he should have trademarked that too.
  • Would it be a Random Short Take without something from Chin-Fah? Probably not one worth reading. In this article he’s renovated his lab and documented the process of attaching TrueNAS iSCSI volumes to his Proxmox environment. I’m fortunate enough to not have had to do Linux iSCSI in some time, but it looks mildly easier than it used to be.
  • Press releases? Here’s one for you: Zerto research report finds companies lack a comprehensive ransomware strategy. Unlike the threat of World War 3 via nuclear strike in the eighties, ransomware is not a case of if, but when.
  • Hungry for more press releases? Datadobi is accelerating its channel momentum with StorageMAP.
  • In other PR news, Nyriad has unveiled its storage-as-a-service offering. I had a chance to speak to them recently, and they are doing some very cool stuff – worth checking out.
  • I hate all kinds of gambling, and I really hate sports gambling, and ads about it. And it drives me nuts when I see sports gambling ads in apps like NBA League Pass. So this news over at El Reg about the SBS offering consumers the chance to opt out of those kinds of ads is fantastic news. It doesn’t fix the problem, but it’s a step in the right direction.

StorPool Announces Version 21

StorPool recently announced version 21 of its storage platform, offering improvements across data protection, efficiency, availability, and compatibility. I had the opportunity to speak to Boyan Krosnov and Alex Ivanov and wanted to share some thoughts.

 

“Magic” Scale-out Erasure Coding

One of the main features announced with Version 21 was “magic” scale-out erasure coding. Previously, StorPool offered triple replication protection of data across nodes. Now, with at least five all-NVMe storage servers, you can take advantage of this new erasure coding. Key capabilities include:

  • Near-zero performance impact even for Tier 0/Tier 1 workloads;
  • Data redundancy across nodes, as information is protected across servers with two parity objects so that any two servers can fail and data remains safe and accessible;
  • Great flexibility and operational efficiency. With per-volume policy management, volumes can be protected with triple replication or Erasure Coding, with per-volume live conversion between data protection schemes;
  • Always-on, non-disruptive operations – up to two storage nodes can be rebooted or brought down for maintenance while the entire storage system remains running with all data remaining available; and
  • Incremental mesh encoding and recovery.

 

Other New Features

But that’s not all. There’s also been work done in the following areas:

  • Improved iSCSI scalability – with support for exporting up to 1000 iSCSI targets per server
  • CloudStack plug-in improvements – introduces support for CloudStack’s volume encryption and partial zone-wide storage that enables easy live migration between compute hosts.
  • OpenNebula add-on improvements – now supports multi-cluster deployments where multiple StorPool sub-clusters behave as a single large-scale primary storage system with a unified global namespace
  • OpenStack Cinder driver improvements – Easy deployment with Canonical Charmed OpenStack and OpenStack instances managed with kolla-ansible
  • Deep integration with Proxmox Virtual Environment – introduces end-to-end automation of all storage operations in Proxmox VE deployments
  • Additional hardware and software compatibility – increased the number of validated hardware and operating systems resulting in easier deployment of StorPool Storage in customers’ preferred environments

 

Thoughts and Further Reading

It’s been a little while since I’ve written about StorPool, and the team continues to add features to the platform and grow in terms of customer adoption and maturity in the market. Every time I speak to Alex and Boyan, I get a strong sense that they’re relentlessly focussed on making the platform more stable, more performance-oriented, and easier to operate. I’m a fan of many of the design principles the company has adopted for its platform, including the use of standard server hardware, fitting in with customer workflows, and addressing the needs of demanding applications. It’s great that it scales linearly, but it’s as equally exciting, at least to me, that it “fades into the background”. Good infrastructure doesn’t want to be mentioned every day, it just needs to work (and work well). The folks at StorPool understand this, and seem to working hard to ensure that the platform, and the service that supports it, meets this requirement to fade into the background. It’s not necessarily “magic”, but it can be done with good code. StorPool has been around since 2012, is self-funded, profitable, and growing. I’ve enjoyed watching the evolution of the product since I was first introduced to it, and am looking forward to seeing what’s next in future releases. For another perspective on the announcement, check out this article over at Gestalt IT.

VMware Cloud on AWS – Check TRIM/UNMAP

This a really quick follow up to one of my TMCHAM articles on TRIM/UNMAP on VMware Cloud on AWS. In short, a customer wanted to know whether TRIM/UNMAP had been enabled on one of their clusters, as they’d requested. The good news is it’s easy enough to find out. On your cluster, go to Configure. Under vSAN, you’ll see Services. Expand the Advanced Options section and you’ll see whether TRIM/UNMAP has been enabled for the cluster or not.

VMware Cloud Disaster Recovery – Ransomware Recovery Activation

One of the cool features of VMware Cloud Disaster Recovery (VCDR) is the Enhanced Ransomware Recovery capability. This is a quick post to talk through how to turn it on in your VCDR environment, and things you need to consider.

 

Organization Settings

The first step is to enable the ransomware services integration in your VCDR dashboard. You’ll need to be an Organisation owner to do this. Go to Settings, and click on Ransomware Recovery Services.

You’ll then have the option to select where the data analysis is performed.

You’ll also need to tick some boxes to ensure that you understand that an appliance will be deployed in each of your Recovery SDDCs, Windows VMs will get a sensor installed, and some preinstalled sensors may clash with Carbon Black.

Click on Activate and it will take a few moments. If it takes much longer than that, you’ll need to talk to someone in support.

Once the analysis integration is activated, you can then activate NSX Advanced Firewall. Page 245 of the PDF documentation covers this better than I can, but note that NSX Advanced Firewall is a chargeable service (if you don’t already have a subscription attached to your Recovery SDDC). There’s some great documentation here on what you do and don’t have access to if you allow the activation of NSX Advanced Firewall.

Like your favourite TV chef would say, here’s one I’ve prepared earlier.

Recovery Plan Configuration

Once the services integration is done, you can configure Ransomware Recovery on a per Recovery Plan basis.

Start by selecting Activate ransomware recovery. You’ll then need to acknowledge that this is a chargeable feature.

You can also choose whether you want to use integrated analysis (i.e. Carbon Black Cloud), and if you want to manually remove other security sensors when you recover. You can, also, choose to use your own tools if you need to.

And that’s it from a configuration perspective. The actual recovery bit? A story for another time.

VMware Cloud Disaster Recovery – Firewall Ports

I published an article a while ago on getting started with VMware Cloud Disaster Recovery (VCDR). One thing I didn’t cover in any real depth was the connectivity requirements between on-premises and the VCDR service. VMware has worked pretty hard to ensure this is streamlined for users, but it’s still something you need to pay attention to. I was helping a client work through this process for a proof of concept recently and thought I’d cover it off more clearly here. The diagram below highlights the main components you need to look at, being:

  • The Cloud File System (frequently referred to as the SCFS)
  • The VMware Cloud DR SaaS Orchestrator (the Orchestrator); and
  • VMware Cloud DR Auto-support.

It’s important to note that the first two services are assigned IP addresses when you enable the service in the Cloud Service Console, and the Auto-support service has three public IP addresses that you need to be able to communicate with. All of this happens outbound over TCP 443. The Auto-support service is not required, but it is strongly recommended, as it makes troubleshooting issues with the service much easier, and provides VMware with an opportunity to proactively resolve cases. Network connectivity requirements are documented here.

[image courtesy of VMware]

So how do I know my firewall rules are working? The first sign that there might be a problem is that the DRaaS Connector deployment will fail to communicate with the Orchestrator at some point (usually towards the end), and you’ll see a message similar to the following. “ERROR! VMware Cloud DR authentication is not configured. Contact support.”

How can you troubleshoot the issue? Fortunately, we have a tool called the DRaaS Connector Connectivity Check CLI that you can run to check what’s not working. In this instance, we suspected an issue with outbound communication, and ran the following command on the console of the DRaaS Connector to check:

drc network test --scope cloud

This returned a status of “reachable” for the Orchestrator and Auto-support services, but the SCFS was unreachable. Some negotiations with the firewall team, and we were up and running.

Note, also, that VMware supports the use of proxy servers for communicating with Auto-support services, but I don’t believe we support the use of a proxy for Orchestrator and SCFS communications. If you’re worried about VCDR using up all your bandwidth, you can throttle it. Details on how to do that can be found here. We recommend a minimum of 100Mbps, but you can go as low as 20Mbps if required.

Brisbane VMUG – October 2023

 

Event Overview

This month I’ll be presenting a recap of VMware Explore for all those that could not make it overseas, exploring what’s new and innovations. The agenda covers:

  • Cloud & Edge infrastructure
  • Modernize infrastructure, operating models and applications
  • Networking & Security
  • Automating app experiences with a comprehensive and secure network
  • Modern Applications & Cloud Management
  • Develop, operate and optimize apps at scale on any cloud
  • Hybrid Workforce
  • Enable work anywhere with secure and frictionless experiences

An introduction to VyOS by Shah Anupam covering how VyOS networking can be leveraged within the VMware ecosystem.

 

Primary Venue

Brisbane VMware Office

Queen St 8/324, 4000 Brisbane, QLD, AU

Brisbane VMware Office – Goondiwindi Room

Register here. Hope to see you there. [Edit] I should mention it’s happening on Wednesday October 18th, 2023 from 12:00 – 1:30pm.

Random Short Take #88

Welcome to Random Short Take #88. This one’s been sitting in my drafts folder for a while. Let’s get random.