Building High Available ELK for Drupal

Setting up your own ELK solution is easy, there are many resources available, for example this DrupalCon NOLA presentation Drupal and Logstash: centralised logging. But how do you make it cope with a massive spike in number of logs, when you need to be able to processs hundreds of events per second? How do you make it scale? This session will help you to answer these questions.

This will be a technical talk targeting sysadmims and systems savvy developers, presenting a possible High Available ELK solution, capable of receiving logs and metrics from different servers and Drupal environments.

We will cover some advanced topics and comon problems, for example

  • designing scalable ELK stack
  • logstash indexer autoscaling
  • preventing elasticsearch to run out of diskspace (S3 backup, curator)
  • securing log transmission with TLS/SSL, ssl offloading tricks, ELB
  • upgrading your ELK stack without downtime
  • different ways of getting logs from Drupal to logstash

Technologies: logstashelasticsearchkibana, beats, AWS, High Availability, AMI, auto-scaling, message queues, syslog, elastic filebeat and topbeat, server metrics, S3 backup, curator.

About the presenter

Marji is a co-founder and the chief systems administrator for Morpht, interested in DevOps, Ansible, ELK / Elastic stack, Puppet, Jenkins, server configuration and developer workflow – with Drupal being the centre of his attention.

Marji has been running ELK stacks for about 2 years, receiving and analysing logs from a couple of hundreds of  Drupal servers.

Follow up blog post: ha-elk-for-drupal

Session Track

DevOps

Experience level

Intermediate

Drupal Version

N/A

Session Time Slot(s)

Time: 
09/27/2016 - 17:00 to 09/27/2016 - 18:00
Room: 
Liffey Meeting 2 | Amazee Labs