Redis and Memcached are Dead: The Case for DB-Backed Hierarchical Caching

David Strauss

Despite my years of contributions to Drupal's support for Redis and Memcached, I think it's time to phase them out. We have several services we like run on these daemons, including caching, locking, and queues.

The database has gotten lots better:

  • Internal lock optimization in MySQL and MariaDB
  • Better HA options: MySQL MHA and MariaDB with Galera
  • The near-universality of InnoDB
  • Cheap, prevalent solid-state storage (SSD)
  • Support for certificate authentication and TLS

Redis and memcached have not improved much:

  • No coherency with Memcached failover
  • The new Redis replication (Sentinel) isn't even as good as MySQL's in 2012
  • Security based on iptables or a password (not great for cloud)
  • No durability for queues or locks

There are better need-specific options:

  • Hierarchical caching with apcu and coherency¬†through the database
  • Modern distributed lock/queue daemons for non-caching purposes

In short, we should stop using Redis and Memcached:

  • Small sites should continue using the database backends for everything
  • Big sites should deploy lock/queue daemons and using hierarchical caching

Why this matters for core:

  • Certain cache items do not need cluster-wide consistency, but a consistent API is the only one we offer right now.
  • Some cache items are content-addressed and can use lighter-weight consistency mechanisms.
  • For some cache items, it may be better to hit a stale item (in the local cache) than miss or escalate to "L2" or "L3."

Session Track

DevOps

Experience Level

Advanced

Drupal Version