We’re not done with backup and recovery yet though. This article from Ben Liebowitz on setting up immutable backups for Vembu using S3 was comprehensive.
Tom Hollingsworth wrote a great article on clearing out the cruft in your home – specifically the electronic kind. I have a cupboard full of cables I tell myself I’ll need at some point, but I really need to just get rid of them. In my defence, I did unplug my LaserDisc player last weekend. Baby steps and all that.
Speaking of streaming, this article covered some of the best mechanisms to purchase digital content with. I still prefer buying discs, but I’m a bit weird too.
Finally, I’ve been a fan of John Birmingham’s writing since I was a misspending my youth at university in the 90s, so it makes sense that I’d enjoy his food reviews too (mainly because it’s not just about food). It should come as no surprise that I, too, love pork rillette.
This is one of those articles I’ve been meaning to post for a while, simply because I forget every time I do it how to do it properly. One way to expand the capacity of your QNAP NAS (non-disruptively) is to replace the drives one at a time with larger capacity drives. It’s recommended that you follow this process, rather than just ripping the drives out one by one and waiting for the RAID Group to expand. It’s a simple enough process to follow, although the QNAP UI has always struck me as a little on the confusing side to navigate, so I took some pictures. Note that this was done on QNAP firmware 5.0.0.1986.
Firstly, go to Storage/Snapshots under Storage in the ControlPanel. Click on Manage.
Select the Storage Pool you want to expand, and click on Manage again.
This will give you a drop-down menu. Select Replace Disks One by One.
Now select the disk you want to replace and click on Change.
Once you’ve done this for all of the disks (and it will take some time to rebuild depending on a variety of factors), click on Expand Capacity. It will ask you if you’re sure and hopefully you’ll click OK.
It’ll take a while for the RAID Group to synchronise.
You’ll notice then that, while the Storage Pool has expanded, the Volume is still the original size. Select the Volume and click on Manage.
Now you can click on Resize Volume.
The Wizard will give you information on the Storage Pool capacity and give you the option to set the new capacity of the volume. I usually click on Set to Max.
It will warn you about that. Click on OK because you like to live on the edge.
It will take a little while, but eventually your Volume will have expanded to fill the space.
Disclaimer: I recently attended Storage Field Day 22. Some expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event. Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.
CTERA recently presented at Storage Field Day 22. You can see their videos from Storage Field Day 22 here, and download a PDF copy of my rough notes from here.
CTERA?
In a nutshell, CTERA is:
Enterprise NAS over Object
100% Private
Multi-cloud, hybrid consistent
Delivers data placement policy and mobility
Caching, not tiering
Zero-trust
The Problem
So what’s the problem we’re trying to solve with unstructured data?
Every IT environment is hybrid
More data is being generated at the edge
Workload placement strategies are driving storage placement
Storage must be instrumented and accessible anywhere
Seems simple enough, but edge storage is hard to get right.
[image courtesy of CTERA]
What Else Do You Want?
We want a lot from our edge storage solutions, including the ability to:
Migrate data to cloud, while keeping a fast local cache
Connect branches and users over a single namespace
Enjoy a HQ-grade experience regardless of location
Achieve 80% cost saving with global dedupe and cloud economics.
The Solution?
CTERA Multi-cloud Global File System – a “software-defined file over object with distributed SMB/NFS edge caching and endpoint collaboration”.
[image courtesy of CTERA]
CTERA Architecture
Single namespace connecting HQ, branches and users with ACL support
Object-native backend with cache accelerated access for remote sites
Multi-cloud scale-out to customer’s private or public infrastructure
Source-based encryption and global deduplication
Multi-tenant administration scalable to thousands of sites
Data management ecosystem for content security, analytics and DevOps automation
[image courtesy of CTERA]
Use Cases?
NAS Modernisation – Hybrid Edge Filer, Object-based Filesystem, Elastic scaling, Built-in Backup & DR
Media – Large Dataset Handling, Ultra-Fast Cloud Sync, MacOS Experience, Cloud Streaming
Multi-site Collaboration – Global File System Distributed Sync Scalable Central Mgt.
Edge Data Processing – Integrated HCI Filers Distributed Data Analysis Machine-Generated Data
Container-native – Global File System Across Distributed Kubernetes Clusters and Tethered Cloud Services
Thoughts and Further Reading
It should come as no surprise that people expect data to be available to them everywhere nowadays. And that’s not just sync and share solutions or sneaker net products on USB drives. No, folks want to be able to access corporate data in a non-intrusive fashion. It gets worse for the IT department though, because your end users aren’t just “heavy spreadsheet users”. They’re also editing large video files, working on complicated technical design diagrams, and generating gigabytes of log files for later analysis. And it’s not enough to say “hey, can you download a copy and upload it later and hope that no-one else has messed with the file?”. Users are expecting more from their systems. There are a variety of ways to deal with this problem, and CTERA seems to have provided a fairly robust solution, with many ways of accessing data, collaborating, and storing data in the cloud and at the edge. The focus isn’t limited to office automation data, with key verticals such as media and entertainment, healthcare, and financial services all having solutions suited to their particular requirements.
CTERA’s Formula One slide is impressive, as is the variety of ways it works to help organisations address the explosion unstructured data in the enterprise. With large swathes of knowledge workers now working more frequently outside the confines of the head office, these kinds of solutions are only going to become more in demand, particularly those that can leverage cloud in an effective (and transparent) fashion. I’m excited to see what’s to come with CTERA. Check out Ray’s article for a more comprehensive view of what CTERA does.
Welcome to Random Short Take #31. Lot of good players have worn 31 in the NBA. You’d think I’d call this the Reggie edition (and I appreciate him more after watching Winning Time), but this one belongs to Brent Barry. This may be related to some recency bias I have, based on the fact that Brent is a commentator in NBA 2K19, but I digress …
I find Plex to be a pretty rock solid application experience, and most of the problems I’ve had with it have been client-related. I recently had a problem with a server update that borked my installation though, and had to roll back. Here’s the quick and dirty way to do that on macOS.
I recently had the chance to speak with Michael Jack from Datadobi about the company’s announcement about its new DIY Starter Pack for NAS migrations. Whilst it seems that the professional services market for NAS migrations has diminished over the last few years, there’s still plenty of data out there that needs to be moved from on box to another. Robocopy and rsync aren’t always the best option when you need to move this much data around.
There are a bunch of things that people need to learn to do operations well. A lot of them are learnt the hard way. This is a great list from Jan Schaumann.
Analyst firms are sometimes misunderstood. My friend Enrico Signoretti has been working at GigaOm for a little while now, and I really enjoyed this article on the thinking behind the GigaOm Radar.
Alastair isn’t just a great writer and moustache aficionado, he’s also a trainer across a number of IT disciplines, including AWS. He recently posted this useful article on what AWS newcomers can expect when it comes to managing EC2 instances.
New year, same old format for news bites. This is #28 – the McKinnie Edition. I always thought Alfonzo looked a bit like that cop in The Deuce. Okay – it’s clear that some of these numbers are going to be hard to work with, but I’ll keep it going for a little while longer (the 30s are where you find a lot of the great players).
In what seems like pretty big news, Veeam has been acquired by Insight Partners. You can read the press release here, and Anton Gostev shares his views on it here.
This one looks like a bit of a science project, but I find myself oddly intrigued by it. You can read the official announcement here. Pre-orders are open now, and I’ll report back some time in March or April when / if the box turns up.
I loved this article from Chin-Fah on ransomware and NAS environments. I’m looking forward to catching up with Chin-Fah next week (along with all of the other delegates) at Storage Field Day 19. Tune in here if you want to see us on camera.
Speaking of ransomware, this article from Joey D’Antoni provided some great insights into the problem and what we can do about it.
A lot of my friends overseas are asking about the bush fires in Australia. There’s a lot in the media about it, and this article about the impact on infrastructure from Preston made for some thought-provoking reading.
I still use Plex heavily, and spend a lot of time moving things from optical discs to my NAS. This article covers a lot of the process I use too. I’ve started using tinyMediaManager as well – it’s pretty neat.
All the kids (and vendor executives) are talking about Kubernetes. It’s almost like we’re talking about public cloud or big data. Inspired in part by what he saw at Cloud Field Day 6, Keith weighs in on the subject here and I recommend you take the time to read (and understand) what he’s saying.
I enjoy reading Justin’s disclosure posts, even when he throws shade on my state (“Queensland is Australia’s Florida”). Not that he’s wrong, mind you.
Cohesity NAS Data Migration, part of SmartFiles, was recently announced as a generally available feature within the Cohesity DataPlatform 6.4 release (after being mentioned in the 6.3 release blog post). The idea behind it is that you can use the feature to perform the migration of NAS data from a primary source to the Cohesity DataPlatform. It is supported for NAS storage registered as SMB or NFS (so it doesn’t necessarily need to be a NAS appliance as such, it can also be a file share hosted somewhere).
What To Think About
There are a few things to think about when you configure your migration policy, including:
The last time the file was accessed;
Last time the file was modified; and
The size of the file.
You also need to think about how frequently you want to run the job. Finally, it’s worth considering which View you want the archived data to reside on.
What Happens?
When the data is migrated an SMB2 symbolic link is left in place of the file with the same name as the file and the original data is moved to the Cohesity View. Note that on Windows boxes, remote to remote symbolic links are disabled, so you need to run these commands:
C:\Windows\system32>fsutil behavior set SymlinkEvaluation R2R:1
C:\Windows\system32>fsutil behavior query SymlinkEvaluation
Once the data is migrated to the Cohesity cluster, subsequent read and write operations are performed on the Cohesity host. You can move data back to the environment by mounting the Cohesity target View on a Windows client, and copying it back to the NAS.
Configuration Steps
To get started, select File Services, and click on Data Migration.
Click on the Migrate Data to configure a migration job.
You’ll need to give it a name.
The next step is to select the Source. If you already have a NAS source configured, you’ll see it here. Otherwise you can register a Source.
Click on the arrow to expand the registered NAS mount points.
Select the mount point you’d like to use.
Once you’ve selected the mount point, click on Add.
You then need to select the Storage Domain (formerly known as a ViewBox) to store the archived data on.
You’ll need to provide a name, and configure schedule options.
You can also configure advanced settings, including QoS and exclusions. Once you’re happy, click on Migrate and the job will be created.
You can then run the job immediately, or wait for the schedule to kick in.
Other Things To Consider
You’ll need to think about your anti-virus options as well. You can register external anti-virus software or install the anti-virus app from the Cohesity Marketplace
Thoughts And Further Reading
Cohesity have long positioned their secondary storage solution as something more than just a backup and recovery solution. There’s some debate about the difference between storage management and data management, but Cohesity seem to have done a good job of introducing yet another feature that can help users easily move data from their primary storage to their secondary storage environment. Plenty of backup solutions have positioned themselves as archive solutions, but many have been focused on moving protection data, rather than primary data from the source. You’ll need to do some careful planning around sizing your environment, as there’s always a chance that an end user will turn up and start accessing files that you thought were stale. And I can’t say with 100% certainty that this solution will transparently work with every line of business application in your environment. But considering it’s aimed at SMB and NFS shares, it looks like it does what it says on the tin, and moves data from one spot to another.
You can read more about the new features in Cohesity DataPlatform 6.4 (Pegasus) on the Cohesity site, and Blocks & Files covered the feature here. Alastair also shared some thoughts on the feature here.
Happy 2019. I’ve been on holidays for three full weeks and it was amazing. I’ll get back to writing about boring stuff soon, but I thought I’d post a quick summary of some issues I’ve had with my home-built NAS recently and what I did to fix it.
Where Are The Disks Gone?
I got an email one evening with the following message.
I do enjoy the “Faithfully yours, etc” and the post script is the most enlightening bit. See where it says [UU____UU]? Yeah, that’s not good. There are 8 disks that make up that device (/dev/md0), so it should look more like [UUUUUUUU]. But why would 4 out of 8 disks just up and disappear? I thought it was a little odd myself. I had a look at the ITX board everything was attached to and realised that those 4 drives were plugged in to a PCI SATA-II card. It seems that either the slot on the board or the card are now failing intermittently. I say “seems” because that’s all I can think of, as the S.M.A.R.T. status of the drives is fine.
Resolution, Baby
The short-term fix to get the filesystem back on line and useable was the classic “assemble” switch with mdadm. Long time readers of this blog may have witnessed me doing something similar with my QNAP devices from time to time. After panic rebooting the box a number of times (a silly thing to do, really), it finally responded to pings. Checking out /proc/mdstat wasn’t good though.
Notice the lack of, erm, devices there? That’s non-optimal. The fix requires a forced assembly of the devices comprising /dev/md0.
dan@openmediavault:~$ sudo mdadm --assemble --force --verbose /dev/md0 /dev/sd[abcdefhi][sudo] password for dan:mdadm: looking for devices for /dev/md0mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.mdadm: /dev/sdc is identified as a member of /dev/md0, slot 3.mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.mdadm: /dev/sde is identified as a member of /dev/md0, slot 5.mdadm: /dev/sdf is identified as a member of /dev/md0, slot 4.mdadm: /dev/sdh is identified as a member of /dev/md0, slot 7.mdadm: /dev/sdi is identified as a member of /dev/md0, slot 6.mdadm: forcing event count in /dev/sdd(2) from 40639 upto 40647mdadm: forcing event count in /dev/sdc(3) from 40639 upto 40647mdadm: forcing event count in /dev/sdf(4) from 40639 upto 40647mdadm: forcing event count in /dev/sde(5) from 40639 upto 40647mdadm: clearing FAULTY flag for device 3 in /dev/md0 for /dev/sddmdadm: clearing FAULTY flag for device 2 in /dev/md0 for /dev/sdcmdadm: clearing FAULTY flag for device 5 in /dev/md0 for /dev/sdfmdadm: clearing FAULTY flag for device 4 in /dev/md0 for /dev/sdemdadm: Marking array /dev/md0 as 'clean'mdadm: added /dev/sdb to /dev/md0 as 1mdadm: added /dev/sdd to /dev/md0 as 2mdadm: added /dev/sdc to /dev/md0 as 3mdadm: added /dev/sdf to /dev/md0 as 4mdadm: added /dev/sde to /dev/md0 as 5mdadm: added /dev/sdi to /dev/md0 as 6mdadm: added /dev/sdh to /dev/md0 as 7mdadm: added /dev/sda to /dev/md0 as 0mdadm: /dev/md0 has been started with 8 drives.
In this example you’ll see that /dev/sdg isn’t included in my command. That device is the SSD I use to boot the system. Sometimes Linux device conventions confuse me too. If you’re in this situation and you think this is just a one-off thing, then you should be okay to unmount the filesystem, run fsck over it, and re-mount it. In my case, this has happened twice already, so I’m in the process of moving data off the NAS onto some scratch space and have procured a cheap little QNAP box to fill its role.
Conclusion
My rush to replace the homebrew device with a QNAP isn’t a knock on the OpenMediaVault project by any stretch. OMV itself has been very reliable and has done everything I needed it to do. Rather, my ability to build semi-resilient devices on a budget has simply proven quite poor. I’ve seen some nasty stuff happen with QNAP devices too, but at least any issues will be covered by some kind of manufacturer’s support team and warranty. My NAS is only covered by me, and I’m just not that interested in working out what could be going wrong here. If I’d built something decent I’d get some alerting back from the box telling me what’s happened to the card that keeps failing. But then I would have spent a lot more on this box than I would have wanted to.
I’ve been lucky thus far in that I haven’t lost any data of real import (the NAS devices are used to store media that I have on DVD or Blu-Ray – the important documents are backed up using Time Machine and Backblaze). It is nice, however, that a tool like mdadm can bring you back from the brink of disaster in a pretty efficient fashion.
Incidentally, if you’re a macOS user, you might have a bunch of .ds_store files on your filesystem. Or stuff like .@Thumb or some such. These things are fine, but macOS doesn’t seem to like them when you’re trying to move folders around. This post provides some handy guidance on how to get rid of a those files in a jiffy.
As always, if the data you’re storing on your NAS device (be it home-built or off the shelf) is important, please make sure you back it up. Preferably in a number of places. Don’t get yourself in a position where this blog post is your only hope of getting your one copy of your firstborn’s pictures from the first day of school back.
A good friend of mine is friends with someone who works at Panasas and suggested I might like to hear from them. I had the opportunity to speak to some of the team, and I thought I’d write a brief overview of what they do. Hopefully I’ll have the opportunity to cover them in the future as I think they’re doing some pretty neat stuff.
It’s HPC, But Not As You Know It
I don’t often like to include that slide where the vendor compares themselves to other players in the market. In this case, though, I thought Panasas’s positioning of themselves as “commercial” HPC versus the traditional HPC storage (and versus enterprise scale-out NAS) is an interesting one. We talked through this a little, and my impression is that they’re starting to deal more and more with the non-traditional HPC-like use cases, such as media and entertainment, oil and gas, genomics folks, and so forth. A number of these workloads fall outside HPC, in the sense that traditional HPC has lived almost exclusively in government and the academic sphere. The roots are clearly in HPC, but there are “enterprise” elements creeping in, such as ease of use (at scale) and improved management functionality.
[image courtesy of Panasas]
Technology
It’s Really Parallel
The really value in Panasas’s offering is the parallel access to the storage. The more nodes you add, the more performance improves. In a serial system, a client can access data via one node in the cluster, regardless of the number of nodes available. In a parallel system, such as this one, a client accesses data that is spread across multiple nodes.
What About The Hardware?
The current offering from Panasas is called ActiveStor. The platform is comprised of PanFS running on Director Blades and Storage Blades. Here’s a picture of the Director Blades (ASD-100) and the Storage Blades (ASH-100). The Director has been transitioned to a 2U4N form factor (it used to be sit in the blade chassis).
[image courtesy of Panasas]
Director Nodes are the Control Plane of PanFS, and handle:
Metadata processing: directories, file names, access control checks, timestamps, etc.
Uses a transaction log to ensure atomicity and durability of structural changes
Coordination of client system actions to ensure single-system view and data-cache-coherence
“Realm” membership (Panasas’s name for the storage cluster), realm self-repair, etc.
Realm maintenance: file reconstruction, automatic capacity balancing, scrubbing, etc.
Storage Nodes are the Data Plane of PanFS, and deal with:
Storage of bulk user data for the realm, accessed in parallel by client systems
Also stores, but does not operate on, all the metadata of the system for the Director Nodes
API based upon the T10 SCSI “Object-Based Storage Device” that Panasas helped define
Storage nodes offer a variety of HDD (4TB, 6TB, 8TB, 10TB, or 12TB) and SSD capacities (480GB, 960GB, 1.9TB) depending on the type of workload you’re dealing with. The SSD is used for metadata and files smaller than 60KB. Everything else is stored on the larger drives.
DirectFlow Protocol
DirectFlow is a big part fo what differentiates Panasas from your average scale-out NAS offering. It does some stuff that’s pretty cool, including:
Support for parallel delivery of data to / from Storage Nodes
Support for fully POSIX-compliant semantics, unlike NFS and SMB
Support for strong data cache-coherency across client systems
It’s a proprietary protocol between clients and ActiveStor components, and there’s an installable kernel module for each client system (Linux and macOS). They tell me that pNFS is based upon DirectFlow, and they had a hand in defining pNFS.
Resilience
Scale out NAS is exciting but us enterprise types want to know about resilience. It’s all fun and games until someone fat fingers a file, or a disk dies. Well, Panasas, as it happens, have a little heritage when it comes to disk resilience. They use a N + 2 RAID 6 (10 wide + P & Q). You could have more disks working for you, but this number seems to work best for Panasas customers. In terms of realms, there are 3, 5 or 7 “rep sets” per realm. There’s also a “realm president”, and every Director has a backup director. There’s also:
Per-file erasure coding of striped files allows the whole cluster to help rebuild a file after a failure;
Only need to rebuild data protection on specific files instead of entire drives(s); and
The percentage of files in the cluster affected by any given failure approaches zero at scale.
Thoughts and Further Reading
I’m the first to admit that my storage experience to date has been firmly rooted in the enterprise space. But, much like my fascination with infrastructure associated wth media and entertainment, I fancy myself as an HPC-friendly storage guy. This is for no other reason than I think HPC workloads are pretty cool, and they tend to scale beyond what I normally see in the enterprise space (keeping in mind that I work in a smallish market). You say genomics to someone, or AI, and they’re enthusiastic about the outcomes. You say SQL 2012 to someone and they’re not as interested.
Panasas are positioning themselves as being suitable, primarily, for commercial HPC storage requirements. They have a strong heritage with traditional HPC workloads, and they seem to have a few customers using their systems for more traditional, enterprise-like NAS deployments as well. This convergence of commercial HPC, traditional and enterprise NAS requirements has presented some interesting challenges, but it seems like Panasas have addressed those in the latest iteration of its hardware. Dealing with stonking great big amounts of data at scale is a challenge for plenty of storage vendors, but Panasas have demonstrated an ability adapt to the evolving demands of their core market. I’m looking forward to seeing the next incarnation of their platform, and how they incorporate technologies such as InfiniBand into their offering.
There’s a good white paper available on the Panasas architecture that you can access here (registration required). El Reg also has some decent coverage of the current hardware offering here.
I recently had the opportunity to replace a bunch of ageing 2TB drives in my OpenMediaVault NAS with some 3TB drives. I run it in a 6+2 RAID-6 configuration (yes, I know, RAID is dead). I was a bit cheeky and replaced 2 drives at a time and let it rebuild. This isn’t something I recommend you do in the real world. Everything came up clean after the drives were replaced. I even got to modify the mdadm.conf file again to tell it I had 0 spares. The problem was that the size of the filesystem in OpenMediaVault was the same as it was before. When you click on Grow it expects you to be adding drives. So, you can grow the filesystem, but you need to expand the device to fill the drives. I recommend taking a backup before you do this. And I unmounted my shares before I did this too.
If you’re using a bitmap, you’ll need to remove it first.
In this example, /dev/md0 is the device you want to grow. It’s likely that your device is called /dev/md0. Note, also, that this will take some time to complete. The next step is to expand the filesystem to fit the RAID device. It’s a good idea to run a filesystem check before you do this.
fsck /dev/md0
Then it’s time to resize (assuming you had no problems in the last step).
resize2fs /dev/md0
You should then be able to remount the device and see the additional capacity. Big thanks to kernel.org for having some useful instructions here.
I recently upgraded my home-brew NAS from OpenMediaVault 2.2.14 (Stone burner) to openmediavault 3.0.86 (Erasmus). It’s recommended that you do a fresh install but I thought I’d give the upgrade a shot as it was only a 10TB recovery if it went pear-shaped (!). They also recommend you disable all your plugins before you upgrade.
Apt-get all of the things
It’s an OS upgrade as well as an application upgrade. In an ssh session I ran
This gets you up to date, then upgrades your distro (Debian), and then gets the necessary packages for omv. I then ran the omv upgrade.
omv-release-upgrade
This seemed to go well. I rebooted the box and could still access the shared data. Happy days. When I tried to access the web UI, however, I could enter my credentials but I couldn’t get in. I then ran
omv-firstaid
And tried to reconfigure the web interface. It kept complaining about a file not being found. So I ran
dpkg -l | grep openmediavault
This told me that there was still a legacy plugin (openmediavault-dnsmasq) running. I’d read on the forums that this might cause some problems. So I used apt-get to remove it.
apt-get remove openmediavault-dnsmasq
The next time I ran apt-get it told me there were some legacy packages present that I could remove. So I did.
After that, I was able to login in to the web UI with no problems and everything now seems to be in order. When my new NAS arrives I’ll evacuate this one and rebuild it from scratch. There are a fair few changes in version 3 and it’s worth checking out. You can download the ISO image from here.
DNS Matters
The reason I had the dnsmasq plugin installed in the first place was that I’d been using the NAS as a DHCP / DNS server. This had been going reasonably well, but I’d heard about Pi-hole and wanted to give that a shot. That’s a story for another time, but I did notice that my OMV box hadn’t updated its /etc/resolv.conf feel correctly, despite the fact that I’d reconfigured DNS via the web GUI. If you run into this issue, just run
dpkg-reconfigure resolvconf
And you’ll find that resolv.conf is correctly updated. Incidentally, if you’re a bit old-fashioned and don’t like to run everything through DHCP reservations, you can add a second set of static host entries to dnsmasq on your pi-hole machine by following these instructions.
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
