Insane Data Requirements
We spoke briefly about just how insane modern data requirements are becoming, in terms of both volume and performance requirements. The example offered up was that of an Advanced Driver-Assistance System (ADAS). These things need a lot of capacity to work, with training data starting at 15PB of data with performance requirements approaching 100GB/s.
- Autonomy – Level 2+
- 10 Deep neural networks needed
- Survey car – 2MP cameras
- 2PB per year per car
- 100 NVIDIA DGX-1 servers per car
When your hot data is 15 – 30PB and growing – it’s a problem.
What’s New In 7?
SwiftStack has been working to address those kinds of challenges with version 7.
Ultra-scale Performance Architecture
SwiftStack has managed to get some pretty decent numbers under its belt, delivering over 100GB/s at scale with a platform that’s designed to scale linearly to higher levels. The numbers stack up well against some of their competitors, and have been validated through:
ProxyFS Edge takes advantage of SwiftStack’s file services to deliver distributed file services between edge, core, and cloud. The idea is that you can use it for “high-throughput, data-intensive use cases”.
[image courtesy of SwiftStack]
- Containerised deployment of ProxyFS agent for orchestrated elasticity
- Clustered filesystem enables scale-out capabilities
- Caching at the edge, minimising latency for improved application performance
- Load-balanced, high-throughput API-based communication to the core
1space File Connector
But what if you have a bunch of unstructured data sitting in file environments that you want to use with your more modern apps? 1space File Connector brings enterprise file data into the cloud namespace, and “[g]ives modern, cloud-native applications access to existing data without migration”. The thinking is that you can modernise your workflows at an incremental rate, rather than having to deal with the app and the storage all in one go. incrementally
- Containerised deployment 1space File Connector for orchestrated elasticity
- File data is accessible using S3 or Swift object APIs
- Scales out and is load balanced for high-throughput
- 1space policies can be applied to file data when migration is desired
The SwiftStack AI Architecture
SwiftStack has also developed a comprehensive AI Architecture model, describing it as “the customer-proven stack that enables deep learning at ultra-scale”. You can read more on that here.
- Shared-nothing distributed architecture
- Keep GPU compute complexes busy
Elasticity from Edge-to-Core-to-Cloud
- With 1space, ingest and access data anywhere
- Eliminate data silos and move beyond one cloud
- Data can be retained and referenced indefinitely as it was originally written
- Enabling traceability, accountability, confidence, and safety throughout the life of a DNN
- Compelling savings compared to public cloud or all-flash arrays Real-World Confidence
- Notable AI deployments for autonomous vehicle development
The final piece is the SwiftStack PRO offering, a support service delivering:
- 24×7 remote management and monitoring of your SwiftStack production cluster(s);
- Incorporating operational best-practices learned from 100s of large-scale production clusters;
- Including advanced monitoring software suite for log aggregation, indexing, and analysis; and
- Operations integration with your internal team to ensure end-to-end management of your environment.
Thoughts And Further Reading
The sheer scale of data enterprises are working with every day is pretty amazing. And data is coming from previously unexpected places as well. The traditional enterprise workloads hosted on NAS or in structured applications are insignificant in size when compared to the PB-scale stuff going on in some environments. So how on earth do we start to derive value from these enormous data sets? I think the key is to understand that data is sometimes going to be in places that we don’t expect, and that we sometimes have to work around that constraint. In this case, SwiftStack has recognised that not all data is going to be sitting in the core, or the cloud, and it’s using some interesting technology to get that data where you need it to get the most value from it.
Getting the data from the edge to somewhere useable (or making it useable at the edge) is one thing, but the ability to use unstructured data sitting in file with modern applications is also pretty cool. There’s often reticence associated with making wholesale changes to data sources, and this solution helps to make that transition a little easier. And it gives the punters an opportunity to address data challenges in places that may have been inaccessible in the past.
SwiftStack has good pedigree in delivering modern scale-out storage solutions, and it’s done a lot of work ensure that its platform adds value. Worth checking out.