I’ve been a Dropbox user for some time, and as disk-slinger in a former life, I really enjoy reading the company’s tech blog. This post on cloud storage abstraction with Object Store was great.
Data breaches – everyone’s having them. Are you missing out? It’s probably just a matter of time. While you’re waiting, check out this article from Preston – John Proctor and Data Breaches.
It’s always sad when a startup stops. Mellor writes that Pavilion Data – last NVMe over Fabrics flash array startup – has died. I had the opportunity to take the kit for a spin in a previous role, and spent some time with some of their people. Smart folks doing interesting things. I hope those impacted land on their feet.
I always enjoy talking to Chris about his Plex setup, because he’s taken it to the next level. Here’s a great article he put together on how to automate lighting with Plex playback.
Finally, our old friend VMware vCenter Converter is back. You can download it from here.
Here are a few links to some random news items and other content that I recently found interesting. You might find them interesting too. Episode 16 – please enjoy these semi-irregular updates.
Scale Computing has been doing a bit in the healthcare sector lately – you can read news about that here.
This was a nice roundup of the news from Apple’s recent WWDC from Six Colors. Hat tip to Stephen Foskett for the link. Speaking of WWDC news, you may have been wondering what happened to all of your purchased content with the imminent demise of iTunes on macOS. It’s still a little fuzzy, but this article attempts to shed some light on things. Spoiler: you should be okay (for the moment).
There’s a great post on the Dropbox Tech Blog from James Cowling discussing the mission versus the system.
The more things change, the more they remain the same. For years I had a Windows PC running Media Center and recording TV. I used IceTV as the XMLTV-based program guide provider. I then started to mess about with some HDHomeRun devices and the PC died and I went back to a traditional DVR arrangement. Plex now has DVR capabilities and it has been doing a reasonable job with guide data (and recording in general), but they’ve decided it’s all a bit too hard to curate guides and want users (at least in Australia) to use XMLTV-based guides instead. So I’m back to using IceTV with Plex. They’re offering a free trial at the moment for Plex users, and setup instructions are here. No, I don’t get paid if you click on the links.
Speaking of axe-throwing, the Cohesity team in Queensland is organising a social event for Friday 21st June from 2 – 4 pm at Maniax Axe Throwing in Newstead. You can get in contact with Casey if you’d like to register.
VeeamON Forum Australia is coming up soon. It will be held at the Hyatt Regency Hotel in Sydney on July 24th and should be a great event. You can find out more information and register for it here. The Vanguards are also planning something cool, so hopefully we’ll see you there.
There’s a lot of talk and slideware devoted to digital transformation, and a lot of it is rubbish. But I found this article from Chin-Fah to be particularly insightful.
Disclaimer: I recently attended Storage Field Day 15. My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event. Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.
Dropbox recently presented at Storage Field Day 15. You can see videos of their presentation here, and download my rough notes from here.
What’s That In Your Pocket?
James Cowling spent some time talking to us about Dropbox’s “Magic Pocket” system architecture. Despite the naff name, it’s a pretty cool bit of tech. Here’s a shot of James answering a question.
Magic Pocket
Dropbox uses Magic Pocket to store users’ file content:
1+ EB of user file data currently stored
Growing at over 10PB per month
Customising the stack end-to-end allowed them to:
Improve performance and reliability for our unique use case
Improve economics
Inside the Magic Pocket
Brief history of development
Prototype and development
Production validation
Ran in dark phase to find any unknown bugs
Deleted first byte of data from third party cloud provider in February 2015
Scale out and cut over
600,000+ disks
3 regions in USA, expanding to EU
Migrated more than 500PB of user data from third party cloud provider into MP in 6 months
It’s worth watching the video to get a feel for the scale of the operation. You can also read more on the Magic Pocket here and here. Chan also did a nice write-up that you can access here.
Beyond Public Cloud
A bit’s been made of Dropbox’s move from public cloud back to its own infrastructure, but Dropbox were careful to point out that they used third parties where it made sense for them, and still leveraged various public cloud and SaaS offerings as part of their daily operations. The key for them was understanding whether building their own solution made sense or not. To that end, they asked themselves three questions:
At what scale is investment in infrastructure cost effective?
Will this scale enable innovation by building custom services and integrating hardware / software more tightly?
Can that innovation add value for users?
From a scale perspective, it was fairly simple, with Dropbox being one of the oldest, largest and most used collaboration platforms around. From an integration perspective, they needed a lot of network and storage horsepower, which set them apart from some of the other web-scale services out there. They were able to add value to users through an optimised stack, increased reliability and better security.
It Makes Sense, But It’s Not For Everyone
That all sounds pretty good, but one of the key things to remember is that they haven’t just cobbled together a bunch of tin and a software stack and become web-scale overnight. While the time to production was short, all things considered, there was still investment (in terms of people, infrastructure and so forth) in making the platform work. When you commit to going your own way, you need to be mindful that there are a lot of ramifications involved, including the requirement to invest in people who know what they’re doing, the capacity to do what you need to do from a hardware perspective, and the right minds to come up with the platform to make it all work together. The last point is probably hardest for people to understand. I’ve ranted before about companies not being anywhere near the scale of Facebook, Google or the other hyperscalers and expecting that they can deliver similar services, for a similar price, with minimal investment.
Scale at this level is a hard thing to do well, and it takes investment in terms of time and resources to get it right. And to make that investment it has to make sense for your business. If your company’s main focus is putting nuts and bolts together on an assembly line, then maybe this kind of approach to IT infrastructure isn’t really warranted. I’m not suggesting that we can’t all learn something from the likes of Dropbox in terms of how to do cool infrastructure at scale. But I think they key takeaways should be that Dropbox have:
Been around for a while;
Put a lot of resources into solving the problems they faced; and
Spent a lot of time deciding what did and did not make sense to do themselves.
I must confess I was ignorant of the scale at which Dropbox is operating, possibly because I saw them as a collaboration piece and didn’t really think of them as an infrastructure platform company. The great thing, however, is they’re not just a platform company. In the same way that Netflix does a lot of really cool stuff with their tech, Dropbox understands that users value performance, reliability and security, and have focused their efforts on ensuring that the end user experience meets those requirements. The Dropbox backend infrastructure makes for a fascinating story, because the scale of their operations is simply not something we come across every day. But I think the real success for Dropbox is their relentless focus on making the end user experience a positive one.
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.