Delayed operations with Queues

Have you ever noticed a site, especially an extranet, or any Drupal site as a logged-in user becoming suddenly unresponsive ? If so, it way well have been because many naive site implementations rely on performing blocking or heavy operations during the page generation cycle, possibly introducing SPOFs (Single Point Of Failure) on the upstream resource.

Typical cases include:

• data acquisition in client mode, in which the site performs outgoing HTTP requests, even with HTTPRL, to download data from a news feed, or an ERP; or perform a web services request of any kind ; even Drupal core includes such a behaviour.
• data acquisition in server mode, in which an external service is injecting data into Drupal over a REST API at too high a rate, causing excessive resource usage, and slowing down the site for its actual users
• page content generation, be it blocks or main content, performing too many requests and causing the site to be sluggish

Such blocking or slow operations should be performed "offline" on most projects, and especially high load ones, to improve time response and user experience, and avoid overload.

In this session we will talk about different approaches to offloading work from your website using queues and workers that can handle massive jobs without impacting the web heads response times.

We will talk about:

• The mechanisms provided by Drupal through the Queue API, currently available on Drupal 6 to 8.
• Architectural patterns to combine such offline work with the Drupal main web UI, like:
  • deferred submits
  • asynchronous push and pull data feeds
  • lazy content or block generation
  • use cases for stale cache data
  • cache pre-warming
  • anticipated content generation
  • static content pre-generation
• Available implementations in contrib, including:
  • custom solutions for queues (Redis) and phpdaemon for workers
  • the available Queue API implementations in contrib for Redis and MongoDB
  • Gearman (http://gearman.org/)
  • PHP Resque (https://github.com/chrisboulton/php-resque) that is port of github queuing system to php
  • RabbitMQ
  • Beanstalkd

We will also talk about sysadmin aspecs of deploying the system, how to start workers (supervisor), monitor queues etc.