I've often wondered why there aren't any (that I'm aware of anyway) Linux hierarchical storage managers (HSM). I realize they are typically used in big iron with things like tape as the base level of storage, but especially when it looked like Optane was going to be a thing it seemed like spinning<->SSD<->Optane<->RAM could be a huge performance opportunity. Though, I guess that's a "layering violation"... Honestly, at this point we've gone almost entirely SSD except for our ZFS backup servers, and that layering violation already has an SSD and RAM cache in it...
It’s just easier done at the application layer. At a previous engagement, in the Before AI era, we originally had a block-level tiering solution that went from RAM to local nvme to NAS on striped flash to S3. It worked okay, but the application-level tiering worked better.
Would have loved a read-through union filesystem but you pretty much had to do your own.
When you're running ZFS (or probably some other clever layering violation like bcachefs) on heterogeneous workloads, you will quickly run into the limits of built-in assumptions. At least that's my experience:
- ARC may require manual tuning for size and metadata size to accommodate page cache, in-app caches, file count etc
- L2ARC and SLOGs are very obviously useful for some clear workloads, but have some cliffs you don't expect, such as ram required for L2ARC mapping, unexpected cold/hot/boot behavior
- Reasoning about reliability becomes harder the more tiers you have
Just imagine, to pick a simple example, you're running postgres, thousands of sqlite3 dbs and one huge duckdb (larger-than-ram) on a single such system - it would perhaps manage to adapt to one such workload at a time, but would it balance all three at the same time?
So much as I'd love to see this direction, I can also imagine the pains coming with it.
The best solution is metadata-only caching, as that's mostly random I/O and would leave the mechanical bandwidth fully for the data itself. The downside is that L2ARC won't have everything, as it isn't proactive; there is the special device, but then the metadata is SSD-only, with that one failure point unless you run mirrors.
If they'd get around to merging Special Failsafe [1] and hybrid drives could address the flash space separately like under SAS, it would be a no-brainer.
Right now at the ZFS-using desktop PC I write this with:
640GB of total L2ARC, 66% utilized (34% available). 664MB of RAM is used to keep track of this.
The machine only has an uptime of a little over a week right now, but over that period L2ARC had a 93% hit rate.
L2ARC would fill more-completely but the machine is low on free RAM right now; Firefox is being a pig, and that means that some things get ejected from ARC before they can be sent to L2ARC.
It's generally quite responsive despite being a box that exclusively uses spinning rust as primary storage.
In terms of absolute reliability: RAIDZ2 is in-use, which allows two drives to fail without data loss. Adding the L2ARC did not change this: Either or both of L2ARC devices could die in an instant and the machine would keep moving along (at reduced performance).
(I'd planned to upgrade the hard drives to something bigger, which is a simple process with ZFS, and also add more RAM, but the market moved faster than I did and I'll be stuck with this for awhile.)
There is LVM Cache which uses dm-cache. That allows you to use SSDs to cache HDDs. Either can be RAIDed too. You can choose your file system because it happens at the block layer.
Besides the trivial caching on SSD there's also ways to do write buffering quite easily. I did that once a long time back when SSDs were too expensive for getting a 2TB disk. So probably ~10 years back.
RAM cache is already layered in front of persistent storage. When it comes to easy hybrid SSD/HDD setups bcachefs was close to deliver, but then things jumped the shark.
That's exactly why I keep Windows on my NAS. You can configure storage tiering with arbitrary redundancy. You can have different redundancy on SSD and HDD tiers. As far as I am concerned ReFS gives me all the features of ZFS and doesn't eat my RAM. Also BitLocker + Secure Boot aren't pain in the ass there.
I used to have one of their 2.5in hybrid drive in a laptop. Seemed to work well for games. But once SSD's became cheap the tech was not longer worth it. Also, 2.5in drives topped out at 2TB of storage, while NVME and 2.5 Sata SSD's continued to grow in size.
Yeah, wow. I hadn’t realised that the prices had gone up that much on hard disks as well. A pair of Synology 4TBs I bought for 2500SEK 18 months ago are now 3400SEK each from the same vendor.
I have 6x unused 6TB SAS drives I bought for $50 each in 2024. I assume they are $150 each now. I can't afford cold storage right now so I rely on raidz2 and can replace drives as they die.. But they have 2 years and no SMART errors.
Worse than that, sellers on Amazon all seem to be scams now[1], and my trusted local supplier (who has genuine parts) is only selling one hard disk per customer. Good luck populating a NAS.
"According to Seagate’s press release, the Alkemi intelligent storage platform takes a different approach. Where Apple’s Fusion Drives used tiering managed by macOS, Alkemi manages the storage pool directly from the enclosure."
Soooooo... it's gonna be like the old days of Drobo, where a little VXworks box did all the heavy lifting of BeyondRAID and transparently exposed it as a big ol' block device to your OS? Good times.
I’m not sure it’s realistic to expect an ordinary user to manage and tune tiered storage, so making it opaque to the OS, or even somewhat aware of the OS running on the computer via a device or filesystem driver, might be a sensible decision.
The 8TB version seems like very poor value for money. For comparison, Amazon is selling 24 TB Seagate Exos drives for $799.
Would have loved a read-through union filesystem but you pretty much had to do your own.
- ARC may require manual tuning for size and metadata size to accommodate page cache, in-app caches, file count etc
- L2ARC and SLOGs are very obviously useful for some clear workloads, but have some cliffs you don't expect, such as ram required for L2ARC mapping, unexpected cold/hot/boot behavior
- Reasoning about reliability becomes harder the more tiers you have
Just imagine, to pick a simple example, you're running postgres, thousands of sqlite3 dbs and one huge duckdb (larger-than-ram) on a single such system - it would perhaps manage to adapt to one such workload at a time, but would it balance all three at the same time?
So much as I'd love to see this direction, I can also imagine the pains coming with it.
If they'd get around to merging Special Failsafe [1] and hybrid drives could address the flash space separately like under SAS, it would be a no-brainer.
[1]https://github.com/openzfs/zfs/pull/16185
640GB of total L2ARC, 66% utilized (34% available). 664MB of RAM is used to keep track of this.
The machine only has an uptime of a little over a week right now, but over that period L2ARC had a 93% hit rate.
L2ARC would fill more-completely but the machine is low on free RAM right now; Firefox is being a pig, and that means that some things get ejected from ARC before they can be sent to L2ARC.
It's generally quite responsive despite being a box that exclusively uses spinning rust as primary storage.
In terms of absolute reliability: RAIDZ2 is in-use, which allows two drives to fail without data loss. Adding the L2ARC did not change this: Either or both of L2ARC devices could die in an instant and the machine would keep moving along (at reduced performance).
(I'd planned to upgrade the hard drives to something bigger, which is a simple process with ZFS, and also add more RAM, but the market moved faster than I did and I'll be stuck with this for awhile.)
Spoiled Americans. That drive is $1393 (14160 SEK) here in Sweden now[1]. A year ago it was 4160 SEK...
Three times as expensive in 12 months seems true for most drives.
I'm just praying no drives in my NAS are going to crap out anytime soon...
[1] https://www.prisjakt.nu/produkt.php?p=12519318
[1] example from when I was trying to buy NAS drives a few weeks back: https://www.amazon.co.uk/sp?marketplaceID=A1F83G8C2ARO7P&sel...
Soooooo... it's gonna be like the old days of Drobo, where a little VXworks box did all the heavy lifting of BeyondRAID and transparently exposed it as a big ol' block device to your OS? Good times.