HAC, a DevOps framework for Drupal leveraging Drupal and Jenkins

HighWire Press hosts more than 2,000 sites and is moving all of these sites to Drupal over the next 12 months.  Transitioning to a DevOps culture and developing a robust DevOps framework has been a key component in this process.  With a distributed team of 40+ developers and site builders, security, transparency, consistency, and efficiency are critical.  To support this, we have developed the HighWire Administration Console (HAC). HAC is currently controlling ~350 Drupal sites with greater accuracy and efficiency than was present prior to its adoption.

In HAC we model each site hosted on our Infrastructure as a node along with fields for site attributes, configuration settings (including those for Varnish and Apache), required repositories and other values. HAC supports multiple instances of each site for development, staging, QA and production. 

Through an architecture focused on Services, a ctools plugin system leveraging a custom API module for executing jobs in Jenkins, Rake, and Capistrano, we are able to quickly model and execute Drupal site actions, perform maintenance, and expand reporting.  Additionally, LDAP integration and Drupal user management provide authentication at a platform level and roles-based permissions to run various maintenance and reporting tools.

HAC has allowed us to reduce time to update sites from 4 hours to 3 minutes per site. We have nearly eliminated human errors from deployments and updates resulting in greater uptime. Site auditing through HAC has resulted in faster site performance and improved transparency.

HAC also models the clusters of webheads, memcache, varnish, and mySQL machines that support thousands of sites for hundreds of customers. HAC manages the configuration files for these applications to allow each site to be individually configured but centrally managed for greater accuracy and transparency for our whole organization.

With HAC, we have the ability to add/remove machines from a cluster, clone a site, deploy a site, update code repositories, run drush commands, run site audits, perform content migration, install features, and change settings. HAC also provides an audit trail of when sites are updated and by whom, along with the ability to roll back updates.  

HAC includes a ticket management utility for organizing weekly and monthly updates and has allowed us to execute a cultural shift to a DevOps approach for site management. This shift has resulted in a reduction in time to update sites, reduced error rates, increased uptime, and increased transparency across our organization.

In HAC we demonstrate that Drupal has a role to play in the development of a DevOps/CI system for maintaining Drupal sites.