Drupal 8 Development / Deployment Workflow

I've been putting off learning how to build sites in Drupal 8, and migrating my existing Drupal 7 sites over to Drupal 8. Why? Drupal 8 uses a lot of new tools. I want to learn how to set up a Drupal 8 site in the "right" (optimal) way so that I don't incur technical debt for myself later on. That means I have a lot of tools to learn. That takes time, which I don't have a lot of. So I've procrastinated.

Now, two deadlines are creeping up on me. The first is the release of Drupal 9, planned for June 3, 2020. At that point, there's a bit longer before Drupal 7 goes unsupported (November 2021 is the current plan), but it's time to move.

The second is the requirement for Strong Customer Authentication (SCA) on European payments the middle of this coming September. ~~One of my D7 sites runs Ubercart and uses Ubercart Stripe, which won't support SCA.The only alternative to upgrading is to disable Stripe as a payment method.~~ Actually, AndraeRay has taken maintainer status of Ubercart Stripe and nearly has an SCA-compliant 7.x-3.x branch ready. Many thanks!

So it's time to start. But how do I set up my development workflow? Having done a lot of reading, here's what I propose. I'd really value input from the Drupal community here; I'm bound to have missed something important, and I've one big question I can't answer (see below). Please pile in at the Comments.

My Proposed Approach

Set up a development server, separate from the server that live sites will be deployed on. This will have plenty of RAM, the same version of PHP as the deployment environment, and Composer installed.
I plan to use a private git repository to store each site's code and configuration.
On the development server, use Composer to install drupal-composer/drupal-project.
Use .gitignore to exclude all core and contributed modules and themes from the git repository.
I'll also move environment-specific settings (such as database credentials) to a settings.local.php file, which will also be excluded using .gitignore.
Add everything else to git.
As I work on the site, I'll commit, add and push changes to the git repository.
I'll also set up a sync directory using $config_directories[‘sync’], so that I can use Drupal console to export the site's configuration and include that in git as well.
On the production server, I can periodically pull everything from the git repository. When I do so, I'd use composer install (never composer update). This removes the need for the production server to have a large amount of RAM (since generating composer.lock is done on the development server). I'd then use Drupal console to import the synchronised configuration, and clear all caches.
For theming, on Drupal 7 I built my own responsive themes by subtheming Zen. Zen for Drupal 8 is in Alpha only; the last (alpha) release was 3 years ago and the last commit was 20 months ago. So it looks like the best approach is to subtheme Classy directly.
Previously, I've found Sass and Compass to be a great help with building a custom theme, so I'd plan to use them again. I'd include one simple .scss file that is excluded from git to override the base hue of the theme; this allows my development site to have a different colour scheme to the production one, to help me always remember which site I'm on.

All of the above can be modified to allow a staging copy of the site as well, if that becomes helpful.

Questions

I have one big question.

This workflow allows me to make configuration changes on the development site, and then push them up to the deployment site when ready.

It seems to me that, whilst configuration needs pushing from development to production, content needs pushing the other way. That's to say, I'll want regularly to make sure my development site contains the latest content on the production site. New and altered pages need pushing down, as do nodes that use the new content types I've developed in development.

Configuration: Development => Staging => Production Content: Production => Staging => Development

How do you do this?

Specifically, I don't think I want to dump the entire database from production, and replace the development database contents with the sqldump. Doing that would also override configuration changes on the development site, and that's not the way this whole workflow is designed.

So is there a generally adopted method to use, to dump the site content from production and import it to development, but without making any changes to the configuration that is stored in the development database?

Over to you

So over to you, reader. I've learnt a lot, but I have a lot to learn still.

Can you help answer that question?

Have I missed anything really important? Would anything in the approach above be unwise, and if so what and why? Am I right that Classy is the best base theme, or is another preferable? (It would need to be actively developed, likely to stick around as Drupal moves through 8.8, 8.9, 9.x and so on, and not slow the site down significantly or make it harder to maintain by adding lots of preprocessing or complex div structures / css libraries). Am I using git and composer in the right way? Any other advice?

Thanks in advance!

Blog Category:

Drupal Planet

Comments

Since you're exporting the

Since you're exporting the development config to the sync folder, you can simply import it again after syncing the production database to your envrionment.

Thanks

Makes sense and easy enough. (a) Drop all tables. (b) Import output from production mysqldump. (c) Import config from development.

Anything that could go wrong with that?

Some Random Ideas

I haven't yet had any opportunity of looking at this problem in full, but here are some ideas that may be of help:

Use --dev flag for composer commands (require, install, update?) on local/dev environments. Make sure to only add modules used for development purposes that way. Just don't use the flag on prod to get a non-dev build.
Use config_split in order to have small chunks of configuration that you use selectively on a given env. Here's an excellent presentation from DrupalCon Seattle 2019 with a systematic approach.
For the actual deploy script:
- Make sure to run drush updb before drush config:import.
- Be careful if planning to use symlinks, I remember reading somewhere that Drupal core would not allow using those for a fast directory switch when deploying. Not sure if that's still true, though.
Learn the different ways to store small bits of information. This is mostly related to custom code, but also helps when having to set some values in settings.*.php files to move across envs. Also, integrate the concept of "secrets" when handling settings.*.php files, so that you never commit things like API keys, DB credentials, etc.
Test your script/workflow to ensure it'll do fine for Drupal core updates. I think 8.4 was somewhat hard for some people; related to this: drush may be a challenge if core changes in a way that requires you to also get a newer drush at the same time.
Make sure to setup a proper backup strategy. If you don't want to be super-fancy for this, there is a super-simple script (leveraging drush) over at my personal blog that you can use for reference.

Additionally, I highly recommend using docker + docker-compose for setting up your local env. It gives lots of flexibility given the kind of isolation it provides (easy to rebuild if an env breaks, easy to test out new env options). You may want to have a look at the custom setup I use for any ideas that may be useful for you. BTW, I/O is better on Linux (it's virtualized in macOS).

Hope it helps!

Thank you - very helpful

Thanks for taking the time to type that - that's a huge amount of very helpful advice. I'll work through it.

Quick question: Is there any particular reason you say to use drush rather than Drupal console to import the configuration back in (having run updb first).

Staging

Your comment on testing is helpful. It means that creating a staging installation is a crucial step for me. I need to have a subdomain (that is not accessible to the world at large) on the same domain as the production site, to which I can run every deployment before running it to the actual production area. That way, if something critical has changed that means my deployment script will wipe every taxonomy entry (for example), I get to find out before doing it on the live site.

Drush and Staging

I mentioned drush just because of muscle memory, I haven't yet checked to see if there's any difference in what they do for commands like config:import or updb.

Great point about having a staging server; even if "simulated" in the same host as prod, it's very useful. Of course, extra care must be taken to ensure DB, files directory, etc are properly isolated between the two envs living on the same physical machine.

I forgot to mention in my first comment that something that helped a lot in a project I was maintaining for a couple years was implementing a small set of WebTestCase tests, exercising the critical functionality; having that allowed me to save some time when publishing updates (especially critical ones) since I just let the automated tests inform me about any negative impact of a given update. I didn't get to the point of automating the build + test with something like Jenkins, but it should have been doable should the client have continued using the site.

config workflow

I use config-split module. This way after I copy the prod db to dev and test, i just have to run "drush cs-im" to get all the proper config for both lower environments. I literally have everything refreshed and ready to go in less than 10 mins-- most of which is spent on the db restore.

This is one of the few bright spots of D8 development that is light years better than D7 + Features.