Using A Pure Storage FlashBlade As A Veeam Repository

I’ve been doing some testing in the lab recently. The focus of this testing has been primarily on Pure Storage’s ObjectEngine and its associated infrastructure. As part of that, I’ve been doing various things with Veeam Backup & Replication 9.5 Update 4, including setting up a FlashBlade NFS repository. I’ve documented the process in a document here. One thing that I thought worthy of noting separately was the firewall requirements. For my Linux Mount Server, I used a CentOS 7 VM, configured with 8 vCPUs and 16GB of RAM. I know, I normally use Debian, but for some reason (that I didn’t have time to investigate) it kept dying every time I kicked off a backup job.

In any case, I set everything up as per Pure’s instructions, but kept getting timeout errors on the job. The error I got was “5/17/2019 10:03:47 AM :: Processing HOST-01 Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond NFSMOUNTHOST:2500“. It felt like it was probably a firewall issue of some sort. I tried to make an exception on the Windows VM hosting the Veeam Backup server, but that didn’t help. The problem was with the Linux VM’s firewall. I used the instructions I found here to add in some custom rules. According to the Veeam documentation, Backup Repository access uses TCP ports 2500 – 5000. Your SecOps people will no doubt have a conniption, but here’s how to open those ports on CentOS.

Firstly, is the firewall running?

[danf@nfsmounthost ~]$ sudo firewall-cmd --state
[sudo] password for danf:
running

Yes it is. So let’s stop it to see if this line of troubleshooting is worth pursuing.

[danf@nfsmounthost ~]$ sudo systemctl stop firewalld

The backup job worked after that. Okay, so let’s start it up again and open up some ports to test.

[danf@nfsmounthost ~]$ sudo systemctl start firewalld
[danf@nfsmounthost ~]$ sudo firewall-cmd --add-port=2500-5000/tcp
success

That worked, so I wanted to make it a more permanent arrangement.

[danf@nfsmounthost ~]$ sudo firewall-cmd --permanent --add-port=2500-5000/tcp
success
[danf@nfsmounthost ~]$ sudo firewall-cmd --permanent --list-ports
2500-5000/tcp

Remember, it’s never the storage. It’s always the firewall. Also, keep in my mind this article is about the how. I’m not offering my opinion about whether it’s really a good idea to configure your host-based firewalls with more holes than Swiss cheese. Or whatever things have lots of holes in them.

VMware – Unmounting NFS Datastores From The CLI

This is a short article, but hopefully useful. I did a brief article a while ago linking to some useful articles about using NFS with VMware vSphere. I recently had to do some maintenance on one of the arrays in our lab and I was having trouble unmounting the datastores using the vSphere client. I used some of the commands in this KB article (although I don’t have SIOC enabled) to get the job done instead.

The first step was to identify if any of the volumes were still mounted on the individual host.

[root@esxihost:~] esxcli storage nfs list
Volume Name  Host            Share                 Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
-----------  --------------  --------------------  ----------  -------  ---------  -----  ---------------------
Pav05        10.300.300.105  /nfs/GB000xxxxxbbf97        true     true      false  false  Not Supported
Pav06        10.300.300.106  /nfs/GB000xxxxxbbf93        true     true      false  false  Not Supported
Pav01        10.300.300.101  /nfs/GB000xxxxxbbf95        true     true      false  false  Not Supported

In this case there are three datastores that I haven’t been able to unmount.

[root@esxihost:~] esxcli storage nfs remove -v Pav05
[root@esxihost:~] esxcli storage nfs remove -v Pav06
[root@esxihost:~] esxcli storage nfs remove -v Pav01

Now there should be no volumes mounted on the host.

[root@esxihost:~] esxcli storage nfs list
[root@esxihost:~]

See, I told you it would be quick.

VMware vSphere and NFS – Some Links

Most of my experience with vSphere storage has revolved around various block storage technologies, such as DAS, FC and iSCSI. I recently began an evaluation of one of those fresh new storage startups running an NVMe-based system. We didn’t have the infrastructure to support NVMe-oF in our lab, so we’ve used NFS to connect the datastores to our vSphere environment. Obviously, at this point, it is less about maximum performance and more about basic functionality. In any case, I thought it might be useful to include a series of links regarding NFS and vSphere that I’ve been using to both get up and running, and troubleshoot some minor issues we had getting everything running. Note that most of these links cover vSphere 6.5, as our lab is currently running that version.

Basics

Create an NFS Datastore

How to add NFS export to VMware ESXi 6.5

NFS Protocols and ESXi

Best Practice

Best Practices for running VMware vSphere on Network Attached Storage

Troubleshooting

Maximum supported volumes reached (1020652)

Increasing the default value that defines the maximum number of NFS mounts on an ESXi/ESX host (2239)

Troubleshooting connectivity issues to an NFS datastore on ESX and ESXi hosts (1003967)

Caringo Announces SwarmNFS

Caringo recently announced SwarmNFS, and I recently had the opportunity to be briefed by Caringo’s Adrian J Herrera (VP Marketing). If you’re not familiar with Caringo, their main platform is Swarm, which “provides a platform for data protection, management, organization and search at massive scale”. You can read an overview of Swarm here, and there’s also a technical overview here.

 

So what is it?

SwarmNFS is a “stateless Linux process that integrates directly with Caringo Swarm. It delivers a global namespace across NFSv4, HTTP, SCSP (Caring’s protocol), S3, and HDFS, delivering data distribution and data management at scale”.

SwarmNFS is basically an NFS server modified with proprietary code. It is:

  • Stateless and lightweight;
  • Has no caching or spooling;
  • Supports parallel data streaming; and
  • Has no single point of failure, with built-in high availability.

Caringo tell me this makes it a whole lot easier to centralise, distribute and manage data, while using a bunch less resources than a traditional file gateway. You can run it as either a Linux process, an appliance or via a VM. Caringo also tell me that, since they connect directly into Swarm, there are less bottlenecks than the traditional approach using gateways, FUSE and proxies.

Caringo_001

Everything in the UI can be done via the API as well, and it has support for multi-tenancy. As I mentioned before, there’s a global namespace with “Universal Access”, meaning that files can be written, read and edited through any interface (NFSv4, SCSP/HTTP, S3, HDFS). Having been a protocol prisoner in previous roles it’s nice to think the there’s a different way to do things.

 

What do I use it for?

You can use this for all kinds of stuff Adrian ran me through some use cases, including:

  • Media and entertainment (think media streaming / content delivery); and
  • Street view type image storage.

One of the key things here is that, because the platform uses NFS, a lot of application re-work doesn’t necessarily need to occur to take advantage of the object storage platform. In my opinion this is a pretty cool feature of the platform, and one that should definitely see people look at SwarmNFS fairly seriously when evaluating their object storage options.

 

Conclusion

Caringo are doing some really cool stuff. If you haven’t checked out FileFly before, it’s also worth a look. The capabilities of the Swarm platform are growing at a rapid place. And the storage world is becoming more object and less block and file as each day passes. Enrico‘s been telling me that for ages now, and everything I’m seeing supports that. Caringo’s approach to metadata – storing metadata with the object itself – also means you can do a bunch of cool stuff with it fairly easily, like replicating it, applying erasure coding to it, and so forth. The upshot is that now the data’s truly portable. So, if you’re object-curious but still hang out with file types, maybe SwarmNFS might be a nice compromise for everyone.

Caringo_002