Better backups and faster syncs: Stream-wrappers for generated content.

manarth

I recently joined an organisation with a Drupal site that have been running for several years, and started getting my local dev environment set up.

  • New VM created? Check.
  • Checkout the repo from Git? Check.
  • Import the DB? Check.
  • Sync the files from prod? Copied 1MB of 9GB…2MB of 9GB…
  • Guess I'd better go fetch a coffee, this could take a while.

Sound familiar?

Whilst that rsync command was running, I thought I'd have a look at the files on prod, and see why they were so big…Oh…half of those files are image-styles: thumbnails and so on. <ctrl>-c, rsync --exclude…Copied 1B of 4.5GB, 2MB of 4.5GB…

I had still had plenty of time for a coffee, but I'd managed to cut the time by half.

It got me wondering, why do we use sites/default/files as a dumping ground for everything? Is there a better way?

What if files that could be automatically generated - image thumbnails, aggregated css and js - were stored elsewhere? What if they went to sites/default/generated-files? Our backups would be smaller, faster, easier to manage. As would synchronising stage or testing environments.

After positive feedback from conversations at Drupalcon Asia, I've been encouraged to create a ticket on d.o. - https://www.drupal.org/node/2670404 - and submit a Core Conversation proposal, to talk about how to get this feature into core, what are the challenges and issues we might face, how to handle backward-compatibility, and if we then backport it to D7, do we treat BC the same way?

Register NOW

Session Track

Core Conversations

Experience Level

Intermediate

Drupal Version