[NEW] Data Tiering - Valkey as general data store not just in-memory data store

[selemuse]

Now that Valkey is beginning to set its own course, wonder if Valkey can be enhanced to work as general data store, rather than just in-memory data store? It will widen the use-case for Valkey and remove those memory constraint.

[sanel]

I'm not sure if my vote counts, but why, may I ask? Valkey/Redis is good at what it does, and we have plenty of battle-tested KV stores out there (e.g., HBase) that have stood the test of time. However, if the goal of this is to improve snapshotting to be "less visible" and more streamlined like in real databases, I'm for it :)

[mattsta]

If you truly want One System to Rule them All, just learn clickhouse and be at peace forever. Clickhouse has turned into basically what I wish redis would have been if it had proper management.

if anything, this project should try to have a more narrow focus before trying to grow again. feature deprecation would help more than trying to add more complexity. every new feature will have fewer users and take more time to implement and maintain.

"but what about backwards compatibility!" some people yell. What about it? Versions don't stop existing. Just run old versions if you need old versions for legacy systems. You'd be surprised how much industrial equipment still runs on Windows 3.1 out there.

I think what people actually want is a core "lightweight + high performance data management platform" they can extend for multiple use cases, but the current architecture mixes all concerns together. The unlimited mixing of concerns has become unmanageable. Every part of the system from networking, server, data structures, protocols, storage, replication, clustering are all mixed together where you can't do a major refactor to one component without touching almost all the other components too (which then breaks compatibility, so nobody does it).

Once nice thing about a non-profit foundation model is there's no push to constantly "grow" or "expand" or "capture market share." The project can just grind in the dark to be best.

[edit: also random idea after writing this: would it make sense to organize a new responsibility structure where there's a per-feature "leader" single person responsible for directing each logical component of networking, server, data structures, protocols, storage, replication, clustering, etc? The project has always had mainly an "everybody knows everything" organization which is great for micromanaging authority, but not the best for long term detailed feature growth in different areas or running concurrent development cycles over the long term.]

[kam193]

I might misunderstand the idea of the issue's author, but what I understand under 'not just in-memory' is an ability to avoid data loss on eviction in critical situations when the real memory usage is higher than expected. I don't think it would be above Valkey scope ;)

But this is just an example important to me. I'd refer to three more general-purpose on-disk features from the world around Valkey:

https://docs.keydb.dev/docs/flash - KeyDB on Flash Storage
https://docs.redis.com/6.4/rs/databases/redis-on-flash/ - Redis on Flash, a feature of Redis Enterprise in 6.x
https://docs.redis.com/latest/rs/databases/auto-tiering/ - Auto Tiering, a feature of Redis Enterprise in 7.x

I'd be happy if I could see a similar feature (or even just the eviction alternative) it in the successor of Redis OSS. But this probably require more talks and some support for development :)

[mattsta]

Yeah, it's one of those "would be fun/neat/interesting" ideas to have and technically fun to build over time, but it's worth considering if other projects already do this better and the effort to build new things vs. who would use it.

These systems have been tried different ways in the past (keep every key in memory, but note which are live vs. paged out to disk; or if a memory check fails check the disk index before returning failures, etc) and it also depends how much you want to optimize for storage systems like the specific flash notes you mentioned above.

Another amusing part is hardware risk. This also seemed like a fun project to explore when Optane was really taking off 8-10 years ago, but now Optane is EOL so all that work around programming for a specific single-vendor "persistent-RAM-reboot-safe" storage modality is just wasted. yay tech industry.

[zuiderkwast]

@soloestoy you mentioned there are some services in China that do this as a budget alternative, because disk is cheaper than ram. How do these work? Just curious.

I generally agree with Matt that what we need is not more features. If we do less, we can do it better.

[PingXie]

I see a balance to achieve here between sound engineering and user requirements.

To use the OS analogy, I think we need something similar to the microkernel architecture, i.e., we need new features but we don't need to build all of them to the "kernel". I would imagine that we need a core engine, that owns the infrastructure such as networking, replication, (multi)threading, and core data structures (strings/lists/set/hash/...). This bare minimum system would cater to all standalone caching use cases. Features like cluster support, scripting (via Lua or other languages), etc should be built as part of this project but as modules only. This also includes the data tiering feature that @soloestoy explained before. By moving these features into their own modules, we will significantly reduce the coupling between the core engine and the modules and among modules themselves, hence speeding up the innovations in both the core engine and the new features (such as support "data tiering").

[soloestoy]

Disks can offer larger capacity and lower cost, but disk-based storage and memory-based storage represent two very different development directions and research areas. Indeed, in China, there are many disk storage products compatible with the Redis protocol. They all use RocksDB as the disk storage engine, and then they have a coding layer that maps complex data structures to pure key-value pairs in RocksDB, for example:

And to enhance the efficiency of disk access, multi-threading is used to access the disk, which also introduces designs for concurrency control. Overall, this is a complex engineering task. Currently, using disk storage is not our top priority I think.

I also want to share some of my views on disk-based storage. Many people think that Redis/Valkey uses memory to store data, and since memory is volatile, it can lead to data loss and cannot guarantee data reliability. Only disk storage can ensure data reliability. I do not fully agree with this view.

First of all, although Valkey uses memory for storage, it also supports persistence. For example, when appendonly (AOF) is enabled, write commands are appended to the log, and even if the process crashes abnormally, data can be recovered from the AOF file. If there is a high demand for persistence, setting appendfsync to always can ensure that every write command is "immediately" flushed to disk. This is not too different from the Write-Ahead Logging (WAL) mechanism of traditional disk-based databases.

Furthermore, through the above methods, whether it is memory-stored Valkey or traditional databases stored on disks, they can only ensure the reliability of data on a single machine. If the machine crashes or the disk is damaged, data recovery is impossible. I believe data reliability relies more on replicas; storing data across multiple replicas to avoid data loss due to single points of failure.

However, data replication between primary and secondary replicas in a multi-replica setup is a serious topic. Currently, because we use an asynchronous replication mechanism, we cannot fully guarantee data consistency between primary and secondary replicas. There are data discrepancies between the primary and secondary replicas, which may lead to the loss of data not yet replicated to the secondary replica when the primary database crashes. Addressing the consistency issues between primary and secondary replicas is a challenge, but I believe it is a problem and direction we should focus on solving in the future in Valkey.

[PingXie]

@soloestoy I think you touched upon a few points that resonate with me really well. I see two high level requirements in this topic:

1. cost efficiency
   a. I too see value in "data tiering" and RocksDB is a great/common engine option but I am sure there are other options too.
   b. My understanding of the "data tiering" value comes mostly from the Redis ecosystem (such as the client, tooling, and experience/expertise) and the cost benefits. It is a cost play at the end of the day.
   c. AOF also doesn't help with the "cost" ask. You still need to hold all of your data in RAM.

2. data durability
   a. AOF does not provide true durability in an HA environment, even if the always policy is used.
   b. In the non-HA case, the AOF user sacrifices "availability" for "durability", also not an ideal situation.
   c. The lack of synchronous replication is indeed the first hurdle IMO that needs to be overcome in order to achieve RPO_0-like true durability. This is quite a departure from where Redis started, philosophically speaking, but IMO can be introduced as an opt-in mode.

BTW, the multi-threading support to improve disk access efficiency should be a separate concern from the core engine IMO. I can also see solutions like RocksDB could help in both cases (though not completely) though I feel helpful to look at the problems on their own first.

[zuiderkwast]

Looking at docs, I find that there was an early feature "virtual memory" which was deprecated in Redis 2.6 and later removed. There's still a document about it in our doc repo: https://github.com/valkey-io/valkey-doc/blob/main/topics/internals-vm.md

[madolson]

Reposting a previous comment from: #553

Yeah, naively mapping disk to memory doesn't work very well, since you see huge latency spikes when you have a virtual memory miss and need to fetch the memory page from disk. You can theoretically hide that if we were multi-threaded, since other threads would continue to get scheduled, but our single threaded architecture get's hurt too much by it.

A virtual memory like approach could work though if we built it ourself in userland. We could make a pretty minor change to the main hash-table to indicate that a key is "in-memory" vs "on-disk". Before executing a command, we can check if the command is in-memory, and if it is we execute the command normally. If it's on-disk, we can do one of two things:

Implement logic to go execute the command for an on-disk operation. I think this is similar to what Zhao mentioned with rocksdb in other threads.
We fetch the data into memory, and once it's there we execute the command as normal.
We would need a way to spill items to disk as well.

[zuiderkwast]

Reposting another comment from #553.

With the "on-disk" flag per key, the key's name still consumes memory. I have another idea: We use a probabilistic filter for on-disk keys. If the key is not found in memory (main hash table) and the feature is enabled, then we check the probabilistic filter. If we have a match, we go and fetch the key from disk. This can allow a larger number of small keys on disk that what we even want to store metadata for in memory.

We can use new maxmemory policies for this. Instead of evicting, we move a key to disk.

If we implement some module API for these actions (evict hook, load missing key hook), then the glue to rocksdb or another storage backend can be made pluggable.