DIY cloud music libraries are not an option in 2021
June 10, 2021 in digital music by Dan Gravell
Following my research into cloud music libraries and services I was contacted to ask about hosting bliss in the cloud. The idea was: you could upload your music and have bliss manage it, then allow synchronisation of bliss’s changes back to your home music network.
You’d think in today’s age of cloud-enlightenment this would be straightforward. It turns out this isn’t just a pain to set up, it’s downright scary in terms of potential cost. So I didn’t bother writing the write-up, but I did decide to write my findings!
Hand waving in the general direction of the Cloud
It started with a comment…
To which I flippantly replied… spin up a new EC2 instance […], install bliss… then work out how to sync the music backwards and forwards. I made it sound easy, huh?
Well, technically it’s not too difficult. There are plenty of cloud storage services where your music library can be stored. We can upload the library into one of those, then create a new computer “in the cloud”. Just like any other computer, you can install bliss on this. Connect the computer to the storage and we’re pretty much done! However, I got stuck on the very first step_ of uploading my music because I got scared about the potential pricing.
Wait, what? As I’ve written in the past, services like iDrive and Backblaze are reasonably affordable and are good solutions for “dumb” storage of your music library. The current price points are about $5/month for 1TB.
This makes these services obvious candidates to be backup hosts, or in more elaborate setups you could use them as the “source of truth” of your music library, synchronizing them with your devices (home music server, mobile apps etc) as required.
However, we’re going further here. We don’t just want to have data in the cloud, but services too. Data isn’t worth much if you never do anything with it. Music isn’t much good if you never listen to it. Services use data to deliver something of use. For music libraries, that’s things like music servers, transcoders, music library organizers (I have one of those) etc.
Location, location, location
There are two types of locations for the services most of us use for music on the Internet. The first and by far the most popular is that we outsource this to the likes of Spotify or YouTube Music. Some of these services support storing your own music and all of them support streaming it. That’s the “service” part of the coin. There’s a library, and it can be streamed to you.
It should be noted: very few consumer grade services offer services other than streaming (like the ones I described above: transcoding or organisation of libraries) but that’s something I’m keeping tabs on.
But what about self hosting your music library? Well what most people have been doing in the past few years in combining cloud storage with self hosting is to run their music software locally, i.e. within their home network. They can take advantage of cheap storage online and have a second copy in their home network. You can even stream directly from these services, if playback is your only concern.
The disadvantage of this is now you have two problems - i.e. your music library, duplicated. It must be said that in most cases the storage is used for cloud music backups and is a one-way thing. But even so, it’s something else to think about.
So, in this case I wanted to run the services in the cloud. Ideally - I wanted to set up a cloud based “gold” music library from which all my musical life is derived. The library could be secured by backups, the library could be organised by bliss, the library could be synchronised to my home network. Idyllic!
The trouble is that the prominently-cheap storage providers I mentioned above do not allow you to run services. Instead, you need to go elsewhere for that, e.g. to Amazon Web Services (AWS), Azure (Microsoft’s equivalent) or any of the other platforms. These tend to be wider in scope than simple data storage.
The cost
So far I’d identified that I would try to store the library somewhere cheap, and then create a computer inside AWS (I’m most familiar with AWS) to connect to the library and install bliss so they can work together. And because I’m quite mindful of the cost, I wanted to keep an eye on on the finances.
So let’s start breaking the cost down. We’ve already said we can pay about $5/month for 1TB on one of the cheaper storage providers.
Now, for ‘compute’ (that’s what techies call the capability to process data in the Cloud - it also gets applied to being able to ‘create’ a virtual computer that you’d use like any physical one), you can get a very modest server running for $10/month. I’m being generous here calling this “modest”. Think about mid-level NAS in terms of performance. $10/month is not a great amount, but I’m trying to keep costs down and as a result I’m having to use a pretty mediocre setup.
Now, in theory we could hook that server up with the cheap storage and that means we can use the storage services price. This seems like a best-of-both worlds.
But there are two problems. And the two problems multiply to create one ginormous headache.
Data transfer: the elephant in the room
The first is that transferring data out of the virtual computer in AWS is metered. There are two ways in which data could be transferred. If I was streaming the music to play it - that would be one way. Another way is that if bliss was running, and it ever updated a music file (which is kind-of the point) that is also transferring data out. Similarly, if I was batch converting some FLACs to MP3s, for example.
Data transfer into compute is often free, but not out. For example, streaming data would be free for first GB then 9c per GB. That means you get about four albums streamed for free in FLAC, then it’s about 2.5c per album streamed. Not that much, but wait for the kicker…
The second problem is that connecting the computer to the music library is technically challenging. To reduce the amount of data transfer we need a way of lazily connecting the two rather than eagerly mirroring the entire library. There are great tools for this, such as rclone’s mount command. However, there’s no real assurance about the amount of data that will be transferred.
Now, combine these two factors: metered data transfer with unlimited potential data transfer and we have our scary monster problem.
This is not a hypothetical problem. With metered storage you need to know exactly what your software is likely to do, and the amount of data that might use. Simply scanning a media folder with Plex for example, might involve a large amount of data transfer (if it’s just reading files, it should be free, but who knows if Plex creates some sort of metadata cache in each folder, for example?). Reports abound of student developers racking up AWS bills.
I don’t want to scare you - these projects are fun and can be useful. If your use case is more modest; merely streaming from the cloud, the cost might not be bad. Let’s say you listen to, on average, one album per night. In this case the total cost per month ends up being:
| Storage | $5/month |
| Compute | $10/month |
| Data transfer | $0.7/month |
The trouble, and risk, comes when you don’t feel in total control - in case the software you use performs some activity (which might be totally inconsequential in a non-Cloud environment) which burdens you with a huge cost.
We could avoid this potential by using AWS’s own file storage service (EBS). But if you wanted to hit the magic 1TB size, that will cost $80/month - 16x as much for storage alone.
This all leads me to conclude that DIY hosting music services, in addition to the library itself, holds too much risk if you’re trying to optimise cost as many consumers are. This makes it not viable for music collectors at the moment.
Did I miss something? Are there any clever hacks to optimise this cost? Let me know in the comments below.
Photo by Timon Studler on Unsplash
Photo by Yaopey Yong on Unsplash
