Now we've got the experience of a number of production D8 sites under our belt we took the time to consolidate our CMI workflow into some useful drush commands.
And naturally we've open sourced them.
Read on to find out more about our drush CMI tools.
Use case
Say you're working on a local development environment for a project where the client is adding and editing configuration. For example, the project might be using contact-forms or Yaml forms for user interaction. Each of these and their associated fields, form and view displays are a config object. As the client is editing these, you don't want that configuration tracked in your source control.
So you start working on a new feature, the first thing you do is sync down a QA or production database and then you run your config import so that your local environment is in a clean state.
You work on some features and the time has come to export those changes to your config export folder, in order to check in the new work into git ready for deployment.
Enter drush cexy
drush cexy
Its like drush cex but with some powersauce.
So the normal drush cex command comes with a --skip-modules option that prevents configuration from say devel module from being exported. But let's go back to our original use case.
We want to export all configuration, but we want to exclude certain patterns.
This is where the --ignore-list option of drush cexy comes in.
In our project we have a ./drush folder, so we stick a file in their called config-ignore.yml with contents as follows.
ignore: - field.field.contact_message.* - field.storage.contact_message.* - contact.form.* - core.entity_form_display.contact_message* - core.entity_form_display.contact_form* - core.entity_view_display.contact_message* - core.entity_view_display.contact_form* - system.site - workbench_email.workbench_email_template.*
You'll note there are some wildcards there. We're ignoring all contact message fields and forms as well as any form or view display configuration. Additionally we're ignoring Workbench Email templates and the system site settings.
So now we run drush cexy like so
drush cexy --destination=/path/to/config-export --ignore-list=/path/to/drush/config-ignore.yml
So what this does is export the active configuration, and then apply the ignore list to remove unwanted configuration.
So now when you run git status you should only see changes you want to commit.
Single install configuration
So lets assume you're working on a feature branch that requires installation of the Google Analytics module.
You download and enable the module
drush dl google_analytics drush en -y google_analytics
And then you export your configuration with drush cexy using your build tool of choice (make in this case - cause remembering all the flags is bound to go wrong)
make export
After running that you find you have a new file in your exported configuration folder:
google_analytics.settings
Now you know that you want this configuration to be editable by the client, as they'll have different GA urchin codes on different environments.
So you don't want to check this into git. But, you do need it to be deployed at least once. And that's where drush cexy's sibling comes in drush cimy.
drush cimy
drush cimy is the import equivalent of drush cexy. We've found it significantly increases our CMI workflow productivity.
So returning to our single install of the google analytics settings. You'd just exported your config using drush cexy and found yourself with a new google_analytics.settings.yml file that you needed to deploy, but only once.
drush cimy combines the following features
- The power of
drush cim --partial - The ability to perform config deletes
- The ability to perform one-time installs
The format is as follows
drush cimy --source=/path/to/config-export --install=/path/to/config-install --delete-list=/path/to/config-delete.yml
So we move the google_analytics.settings.yml out of our config-export folder and into our config-install folder. And then we add it to our drush/config-ignore.yml file, so it doesn't get exported in the future.
Partial imports
So as alluded above, drush cimy is similar to drush cim --partial in that it does partial imports.
The way drush cim --partial works is equivalent to the following
- firstly it creates a temporary folder
- then it exports all active configuration
- then it copies your nominated config-export folder (the one under source control) over the top (in the temporary folder)
- then it imports from the temporary folder
So what you get imported is all active config plus and new config from the config export, with changes in the exported config taking precedence over the active config.
The main pain point with using --partial is you don't get config deletes.
e.g. if you delete a config file from git (it is no longer in your config-export folder) because it is still present in the active configuration, it still remains after import.
So why is this a problem. So let's consider a scenario where someone enabled dblog module on QA, and saved the settings so that dblog.settings.yml is in the active configuration.
Your core.extensions.yml that is tracked in git does not contain dblog module. But dblog.settings.yml depends on dblog module.
So you work away on your feature and go to deploy to QA. But the import step of your deployment automation fails, because --partial places a copy of dblog.settings.yml in the temporary folder and tries to import it, but because dblog module is going to be disabled by the import, you have an unmet config dependency.
This is where the --delete-list flag kicks in. Let's look at a sample delete list file
delete: - dblog.settings
So this is where drush cimy varies from drush cim, its (equivalent) logic is as follows
- As with
drush cim --partial, first it creates a temporary folder - Move one-time install configuration into the folder first - so that active configuration takes precendence over initial state
- export all active configuration
- delete any configuration found in active configuration that is listed in the delete list
- copy the nominated config-export (tracked in source control) over the top, taking final precendence
- import the result
So this means you get the good bits of partial imports, without the dependency dramas that can result. It also allows you to perform valid deletes, something that isn't possible with drush cim --partial - for example you might want to delete a field from active configuration. Previously you'd have to write an update hook to do that before you performed your config import. Now you just list it in the config-delete.yml file
Installation
cd ~/.drush wget https://raw.githubusercontent.com/previousnext/drush_cmi_tools/8.x-1.x/drush_cmi_tools.drush.inc drush cc drush
Comments
FYI - your "YAML Forms" link is returning a 404 - https://drupal.org/project/yaml_form should be https://www.drupal.org/project/yamlform
HOORAY! I'd looked into drush cim --partial way back last year before it was reliable enough to work well, but I'd lost track of it since. Thanks for bringing it back to my attention, and for the extensions to it that you've added. Brilliant :-)
I have to say, I'm actually okay with scripting deletes in update hooks (they feel more controlled that way). And I'm not sure I want to be tracking core.extensions.yml in source control, as I think it's useful to easily have development modules enabled in staging environments, as well as the dependencies issue you mentioned.
The one-time installs idea is clever though. I've been balancing using /config/install directories (using config_devel to package config into them) for installation, and then managing the files sync directory separately, but that does introduce redundancy, so I like this. Might have to put it into practise :-)
Great article! I'm open to changing core Drush commands to make this easier, or to move some/all into core Drush. Haven't looked at the code but I do see the usefulness of these commands.
Yeah, I tried to ping you on irc to discuss before I posted this, but timezones being what they are etc. Greg suggested something similar at https://github.com/previousnext/drush_cmi_tools/issues/1
Hi! It'd be absolutely ace if you could include a composer.json file and set the type correctly to be a drush plugin, so those of us using strict VCS and Composer could take advantage of this :-)
Sure, can do
+1 to Sam's comment. A composer.json example here would be super helpful.
Almost a year, but I hope It helps, it comes from the documentation https://github.com/previousnext/drush_cmi_tools
"repositories": [
{
"url": "https://github.com/previousnext/drush_cmi_tools.git",
"type": "git"
}
],
"require": {
"drupal/drush-cmi-tools": "dev-8.x-1.x"
},
Looks like an excellent tool, thanks for posting! Just curious if you have had any progress moving this into Drush? I would would like to combine this with Config Readonly if we can get traction here https://www.drupal.org/node/2826274
I've submitted a patch to config read only that makes it work automatically with your drush/config-ignore.yml file so the two stay in sync without effort.
thanks!
What is the idea behind the --install switch
drush cimy --source=/path/to/config-export --install=/path/to/config-install
See above under 'Single install configuration' section
Hey,
Great work, we use your tool intensively! However, since today (Thu 25 January 2018), we get an error while Exception your command.
[Exception]
The configuration directory type 'sync' does not exist
My best guess is that this has something to do with a drush update. Is this something that you can fix? Or should we try to stay on a lower version of drush?
Thanks for the great work and help.
This should be set in your settings.php
Hi,
Thanks for this great tool.
It seems the import command ignores files in the language folder. E.g config/dev/language/nl/views.view.slideshow_images.yml
With the regular drush import command changes in translations are detected.
Any fix for this?
Thanks!
Joris
I think there is a pull request for collection support
thanks! I looked at it but this issue is about having the option to ignore collections, which is the opposite of what we want. We want the language folders to be taken into account. They are being exported properly but during import we get a: nothing to import message. Any ideas on how to tackle this?
The code was based of what drush was doing, so perhaps drush is doing something else at the moment and we need to update accordingly.
If you work it out, please send a Pull Request.
Thanks
Hi, I'm trying to install dev-9.x branch for cmi-tools using composer global require (the same manner I had used to install drush)
It's seems to run ok but, after clearing drush cache and running drush to see the whole command list, cimy and cexy are not listed.
could you advice me?
Hi, I am trying to do this, but I need to do it only for when it is a delete operation, to avoid that the confgurations that I have in production are deleted when doing drush cim.
Check this article.
How to prevent config import from deleting webforms on a target site if they don't exist in code?
https://www.drupal.org/project/webform/issues/2931104
How could I do this only for the delete operation?
Thank.
Comment by Justin Winter
Dated