Introducing Drush CMI tools

Now we've got the experience of a number of production D8 sites under our belt we took the time to consolidate our CMI workflow into some useful drush commands.

And naturally we've open sourced them.

Read on to find out more about our drush CMI tools.

by lee.rowlands /

Use case

Say you're working on a local development environment for a project where the client is adding and editing configuration. For example, the project might be using contact-forms or Yaml forms for user interaction. Each of these and their associated fields, form and view displays are a config object. As the client is editing these, you don't want that configuration tracked in your source control.

So you start working on a new feature, the first thing you do is sync down a QA or production database and then you run your config import so that your local environment is in a clean state.

You work on some features and the time has come to export those changes to your config export folder, in order to check in the new work into git ready for deployment.

Enter drush cexy

drush cexy

Its like drush cex but with some powersauce.

So the normal drush cex command comes with a --skip-modules option that prevents configuration from say devel module from being exported. But let's go back to our original use case.

We want to export all configuration, but we want to exclude certain patterns.

This is where the --ignore-list option of drush cexy comes in.

In our project we have a ./drush folder, so we stick a file in their called config-ignore.yml with contents as follows.

ignore:
  - field.field.contact_message.*
  - field.storage.contact_message.*
  - contact.form.*
  - core.entity_form_display.contact_message*
  - core.entity_form_display.contact_form*
  - core.entity_view_display.contact_message*
  - core.entity_view_display.contact_form*
  - system.site
  - workbench_email.workbench_email_template.*

You'll note there are some wildcards there. We're ignoring all contact message fields and forms as well as any form or view display configuration. Additionally we're ignoring Workbench Email templates and the system site settings.

So now we run drush cexy like so

drush cexy --destination=/path/to/config-export --ignore-list=/path/to/drush/config-ignore.yml

So what this does is export the active configuration, and then apply the ignore list to remove unwanted configuration.

So now when you run git status you should only see changes you want to commit.

Single install configuration

So lets assume you're working on a feature branch that requires installation of the Google Analytics module.

You download and enable the module

drush dl google_analytics
drush en -y google_analytics

And then you export your configuration with drush cexy using your build tool of choice (make in this case - cause remembering all the flags is bound to go wrong)

make export

After running that you find you have a new file in your exported configuration folder:

google_analytics.settings

Now you know that you want this configuration to be editable by the client, as they'll have different GA urchin codes on different environments.

So you don't want to check this into git. But, you do need it to be deployed at least once. And that's where drush cexy's sibling comes in drush cimy.

drush cimy

drush cimy is the import equivalent of drush cexy. We've found it significantly increases our CMI workflow productivity.

So returning to our single install of the google analytics settings. You'd just exported your config using drush cexy and found yourself with a new google_analytics.settings.yml file that you needed to deploy, but only once.

drush cimy combines the following features

  • The power of drush cim --partial
  • The ability to perform config deletes
  • The ability to perform one-time installs

The format is as follows

drush cimy --source=/path/to/config-export --install=/path/to/config-install --delete-list=/path/to/config-delete.yml

So we move the google_analytics.settings.yml out of our config-export folder and into our config-install folder. And then we add it to our drush/config-ignore.yml file, so it doesn't get exported in the future.

Partial imports

So as alluded above, drush cimy is similar to drush cim --partial in that it does partial imports.

The way drush cim --partial works is equivalent to the following

  • firstly it creates a temporary folder
  • then it exports all active configuration
  • then it copies your nominated config-export folder (the one under source control) over the top (in the temporary folder)
  • then it imports from the temporary folder

So what you get imported is all active config plus and new config from the config export, with changes in the exported config taking precedence over the active config.

The main pain point with using --partial is you don't get config deletes.

e.g. if you delete a config file from git (it is no longer in your config-export folder) because it is still present in the active configuration, it still remains after import.

So why is this a problem. So let's consider a scenario where someone enabled dblog module on QA, and saved the settings so that dblog.settings.yml is in the active configuration.

Your core.extensions.yml that is tracked in git does not contain dblog module. But dblog.settings.yml depends on dblog module.

So you work away on your feature and go to deploy to QA. But the import step of your deployment automation fails, because --partial places a copy of dblog.settings.yml in the temporary folder and tries to import it, but because dblog module is going to be disabled by the import, you have an unmet config dependency.

This is where the --delete-list flag kicks in. Let's look at a sample delete list file

delete:
  - dblog.settings

So this is where drush cimy varies from drush cim, its (equivalent) logic is as follows

  • As with drush cim --partial, first it creates a temporary folder
  • Move one-time install configuration into the folder first - so that active configuration takes precendence over initial state
  • export all active configuration
  • delete any configuration found in active configuration that is listed in the delete list
  • copy the nominated config-export (tracked in source control) over the top, taking final precendence
  • import the result

So this means you get the good bits of partial imports, without the dependency dramas that can result. It also allows you to perform valid deletes, something that isn't possible with drush cim --partial - for example you might want to delete a field from active configuration. Previously you'd have to write an update hook to do that before you performed your config import. Now you just list it in the config-delete.yml file

Installation

cd ~/.drush
wget https://raw.githubusercontent.com/previousnext/drush_cmi_tools/8.x-1.x/drush_cmi_tools.drush.inc
drush cc drush

 

Tagged

drush, CMI

lee.rowlands
Senior Developer

Comments

James Williams

HOORAY! I'd looked into drush cim --partial way back last year before it was reliable enough to work well, but I'd lost track of it since. Thanks for bringing it back to my attention, and for the extensions to it that you've added. Brilliant :-)

I have to say, I'm actually okay with scripting deletes in update hooks (they feel more controlled that way). And I'm not sure I want to be tracking core.extensions.yml in source control, as I think it's useful to easily have development modules enabled in staging environments, as well as the dependencies issue you mentioned.

The one-time installs idea is clever though. I've been balancing using /config/install directories (using config_devel to package config into them) for installation, and then managing the files sync directory separately, but that does introduce redundancy, so I like this. Might have to put it into practise :-)

Moshe Weitzman

Great article! I'm open to changing core Drush commands to make this easier, or to move some/all into core Drush. Haven't looked at the code but I do see the usefulness of these commands.

Sam Marley-Jarrett

Hi! It'd be absolutely ace if you could include a composer.json file and set the type correctly to be a drush plugin, so those of us using strict VCS and Composer could take advantage of this :-)

heddn

+1 to Sam's comment. A composer.json example here would be super helpful.

scott_euser

Looks like an excellent tool, thanks for posting! Just curious if you have had any progress moving this into Drush? I would would like to combine this with Config Readonly if we can get traction here https://www.drupal.org/node/2826274

scott_euser

I've submitted a patch to config read only that makes it work automatically with your drush/config-ignore.yml file so the two stay in sync without effort.

Johan

What is the idea behind the --install switch
drush cimy --source=/path/to/config-export --install=/path/to/config-install

lee.rowlands

See above under 'Single install configuration' section

Sander Van Camp

Hey,

Great work, we use your tool intensively! However, since today (Thu 25 January 2018), we get an error while Exception your command.

[Exception]
The configuration directory type 'sync' does not exist

My best guess is that this has something to do with a drush update. Is this something that you can fix? Or should we try to stay on a lower version of drush?

Thanks for the great work and help.

lee.rowlands

This should be set in your settings.php

Sander

yes I know how to change the config directories, but the problem persists while using Drush 9.

lee.rowlands

There is a separate branch for Drush 9 support, I'm not sure if it is ready yet

joske

Hi,

Thanks for this great tool.

It seems the import command ignores files in the language folder. E.g config/dev/language/nl/views.view.slideshow_images.yml

With the regular drush import command changes in translations are detected.

Any fix for this?

Thanks!

Joris

lee.rowlands

I think there is a pull request for collection support

joske

thanks! I looked at it but this issue is about having the option to ignore collections, which is the opposite of what we want. We want the language folders to be taken into account. They are being exported properly but during import we get a: nothing to import message. Any ideas on how to tackle this?

lee.rowlands

The code was based of what drush was doing, so perhaps drush is doing something else at the moment and we need to update accordingly.

If you work it out, please send a Pull Request.

Thanks

Mario Madera

Hi, I'm trying to install dev-9.x branch for cmi-tools using composer global require (the same manner I had used to install drush)
It's seems to run ok but, after clearing drush cache and running drush to see the whole command list, cimy and cexy are not listed.
could you advice me?

Diosbel Mezquia

Hi, I am trying to do this, but I need to do it only for when it is a delete operation, to avoid that the confgurations that I have in production are deleted when doing drush cim.

Check this article.
How to prevent config import from deleting webforms on a target site if they don't exist in code?
https://www.drupal.org/project/webform/issues/2931104

How could I do this only for the delete operation?
Thank.