Redis and Memcached are Dead: The Case for DB-Backed Hierarchical Caching

Despite my years of contributions to Drupal's support for Redis and Memcached, I think it's time to phase them out. We have several services we like run on these daemons, including caching, locking, and queues.

The database has gotten lots better:

• Internal lock optimization in MySQL and MariaDB
• Better HA options: MySQL MHA and MariaDB with Galera
• The near-universality of InnoDB
• Cheap, prevalent solid-state storage (SSD)
• Support for certificate authentication and TLS

Redis and memcached have not improved much:

• No coherency with Memcached failover
• The new Redis replication (Sentinel) isn't even as good as MySQL's in 2012
• Security based on iptables or a password (not great for cloud)
• No durability for queues or locks

There are better need-specific options:

• Hierarchical caching with apcu and coherency through the database
• Modern distributed lock/queue daemons for non-caching purposes

In short, we should stop using Redis and Memcached:

• Small sites should continue using the database backends for everything
• Big sites should deploy lock/queue daemons and using hierarchical caching

Why this matters for core:

• Certain cache items do not need cluster-wide consistency, but a consistent API is the only one we offer right now.
• Some cache items are content-addressed and can use lighter-weight consistency mechanisms.
• For some cache items, it may be better to hit a stale item (in the local cache) than miss or escalate to "L2" or "L3."