Skip to main content

Introducing Drush CMI tools

Now we've got the experience of a number of production D8 sites under our belt we took the time to consolidate our CMI workflow into some useful drush commands.

And naturally we've open sourced them.

Read on to find out more about our drush CMI tools.

by Lee Rowlands /

Use case

Say you're working on a local development environment for a project where the client is adding and editing configuration. For example, the project might be using contact-forms or Yaml forms for user interaction. Each of these and their associated fields, form and view displays are a config object. As the client is editing these, you don't want that configuration tracked in your source control.

So you start working on a new feature, the first thing you do is sync down a QA or production database and then you run your config import so that your local environment is in a clean state.

You work on some features and the time has come to export those changes to your config export folder, in order to check in the new work into git ready for deployment.

Enter drush cexy

drush cexy

Its like drush cex but with some powersauce.

So the normal drush cex command comes with a --skip-modules option that prevents configuration from say devel module from being exported. But let's go back to our original use case.

We want to export all configuration, but we want to exclude certain patterns.

This is where the --ignore-list option of drush cexy comes in.

In our project we have a ./drush folder, so we stick a file in their called config-ignore.yml with contents as follows.

ignore:
  - field.field.contact_message.*
  - field.storage.contact_message.*
  - contact.form.*
  - core.entity_form_display.contact_message*
  - core.entity_form_display.contact_form*
  - core.entity_view_display.contact_message*
  - core.entity_view_display.contact_form*
  - system.site
  - workbench_email.workbench_email_template.*

You'll note there are some wildcards there. We're ignoring all contact message fields and forms as well as any form or view display configuration. Additionally we're ignoring Workbench Email templates and the system site settings.

So now we run drush cexy like so

drush cexy --destination=/path/to/config-export --ignore-list=/path/to/drush/config-ignore.yml

So what this does is export the active configuration, and then apply the ignore list to remove unwanted configuration.

So now when you run git status you should only see changes you want to commit.

Single install configuration

So lets assume you're working on a feature branch that requires installation of the Google Analytics module.

You download and enable the module

drush dl google_analytics
drush en -y google_analytics

And then you export your configuration with drush cexy using your build tool of choice (make in this case - cause remembering all the flags is bound to go wrong)

make export

After running that you find you have a new file in your exported configuration folder:

google_analytics.settings

Now you know that you want this configuration to be editable by the client, as they'll have different GA urchin codes on different environments.

So you don't want to check this into git. But, you do need it to be deployed at least once. And that's where drush cexy's sibling comes in drush cimy.

drush cimy

drush cimy is the import equivalent of drush cexy. We've found it significantly increases our CMI workflow productivity.

So returning to our single install of the google analytics settings. You'd just exported your config using drush cexy and found yourself with a new google_analytics.settings.yml file that you needed to deploy, but only once.

drush cimy combines the following features

  • The power of drush cim --partial
  • The ability to perform config deletes
  • The ability to perform one-time installs

The format is as follows

drush cimy --source=/path/to/config-export --install=/path/to/config-install --delete-list=/path/to/config-delete.yml

So we move the google_analytics.settings.yml out of our config-export folder and into our config-install folder. And then we add it to our drush/config-ignore.yml file, so it doesn't get exported in the future.

Partial imports

So as alluded above, drush cimy is similar to drush cim --partial in that it does partial imports.

The way drush cim --partial works is equivalent to the following

  • firstly it creates a temporary folder
  • then it exports all active configuration
  • then it copies your nominated config-export folder (the one under source control) over the top (in the temporary folder)
  • then it imports from the temporary folder

So what you get imported is all active config plus and new config from the config export, with changes in the exported config taking precedence over the active config.

The main pain point with using --partial is you don't get config deletes.

e.g. if you delete a config file from git (it is no longer in your config-export folder) because it is still present in the active configuration, it still remains after import.

So why is this a problem. So let's consider a scenario where someone enabled dblog module on QA, and saved the settings so that dblog.settings.yml is in the active configuration.

Your core.extensions.yml that is tracked in git does not contain dblog module. But dblog.settings.yml depends on dblog module.

So you work away on your feature and go to deploy to QA. But the import step of your deployment automation fails, because --partial places a copy of dblog.settings.yml in the temporary folder and tries to import it, but because dblog module is going to be disabled by the import, you have an unmet config dependency.

This is where the --delete-list flag kicks in. Let's look at a sample delete list file

delete:
  - dblog.settings

So this is where drush cimy varies from drush cim, its (equivalent) logic is as follows

  • As with drush cim --partial, first it creates a temporary folder
  • Move one-time install configuration into the folder first - so that active configuration takes precendence over initial state
  • export all active configuration
  • delete any configuration found in active configuration that is listed in the delete list
  • copy the nominated config-export (tracked in source control) over the top, taking final precendence
  • import the result

So this means you get the good bits of partial imports, without the dependency dramas that can result. It also allows you to perform valid deletes, something that isn't possible with drush cim --partial - for example you might want to delete a field from active configuration. Previously you'd have to write an update hook to do that before you performed your config import. Now you just list it in the config-delete.yml file

Installation

cd ~/.drush
wget https://raw.githubusercontent.com/previousnext/drush_cmi_tools/8.x-1.x/drush_cmi_tools.drush.inc
drush cc drush

 

Tagged

drush, CMI

Posted by Lee Rowlands
Senior Drupal Developer

Dated

Comments

Comment by James Williams

Dated

HOORAY! I'd looked into drush cim --partial way back last year before it was reliable enough to work well, but I'd lost track of it since. Thanks for bringing it back to my attention, and for the extensions to it that you've added. Brilliant :-)

I have to say, I'm actually okay with scripting deletes in update hooks (they feel more controlled that way). And I'm not sure I want to be tracking core.extensions.yml in source control, as I think it's useful to easily have development modules enabled in staging environments, as well as the dependencies issue you mentioned.

The one-time installs idea is clever though. I've been balancing using /config/install directories (using config_devel to package config into them) for installation, and then managing the files sync directory separately, but that does introduce redundancy, so I like this. Might have to put it into practise :-)

Comment by Moshe Weitzman

Dated

Great article! I'm open to changing core Drush commands to make this easier, or to move some/all into core Drush. Haven't looked at the code but I do see the usefulness of these commands.

Comment by Sam Marley-Jarrett

Dated

Hi! It'd be absolutely ace if you could include a composer.json file and set the type correctly to be a drush plugin, so those of us using strict VCS and Composer could take advantage of this :-)

Comment by heddn

Dated

+1 to Sam's comment. A composer.json example here would be super helpful.

Comment by scott_euser

Dated

Looks like an excellent tool, thanks for posting! Just curious if you have had any progress moving this into Drush? I would would like to combine this with Config Readonly if we can get traction here https://www.drupal.org/node/2826274

Comment by scott_euser

Dated

I've submitted a patch to config read only that makes it work automatically with your drush/config-ignore.yml file so the two stay in sync without effort.

Comment by Johan

Dated

What is the idea behind the --install switch
drush cimy --source=/path/to/config-export --install=/path/to/config-install

Comment by Lee Rowlands

Dated

See above under 'Single install configuration' section

Comment by Sander Van Camp

Dated

Hey,

Great work, we use your tool intensively! However, since today (Thu 25 January 2018), we get an error while Exception your command.

[Exception]
The configuration directory type 'sync' does not exist

My best guess is that this has something to do with a drush update. Is this something that you can fix? Or should we try to stay on a lower version of drush?

Thanks for the great work and help.

Comment by Lee Rowlands

Dated

This should be set in your settings.php

Comment by Sander

Dated

yes I know how to change the config directories, but the problem persists while using Drush 9.

Comment by Lee Rowlands

Dated

There is a separate branch for Drush 9 support, I'm not sure if it is ready yet

Comment by joske

Dated

Hi,

Thanks for this great tool.

It seems the import command ignores files in the language folder. E.g config/dev/language/nl/views.view.slideshow_images.yml

With the regular drush import command changes in translations are detected.

Any fix for this?

Thanks!

Joris

Comment by Lee Rowlands

Dated

I think there is a pull request for collection support

Comment by joske

Dated

thanks! I looked at it but this issue is about having the option to ignore collections, which is the opposite of what we want. We want the language folders to be taken into account. They are being exported properly but during import we get a: nothing to import message. Any ideas on how to tackle this?

Comment by Lee Rowlands

Dated

The code was based of what drush was doing, so perhaps drush is doing something else at the moment and we need to update accordingly.

If you work it out, please send a Pull Request.

Thanks

Comment by Mario Madera

Dated

Hi, I'm trying to install dev-9.x branch for cmi-tools using composer global require (the same manner I had used to install drush)
It's seems to run ok but, after clearing drush cache and running drush to see the whole command list, cimy and cexy are not listed.
could you advice me?

Pagination

Add new comment

Restricted HTML

  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd> <h2> <h3> <h4> <h5> <h6>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.
Not sure where to start? Try typing "hello" or "help" if you get stuck.