Using Behat and DrupalDriver? Beware pathauto.

Background

In case you hadn't heard of it before, Behat is a PHP testing framework for Behaviour Driven Development. It lets you write human-readable stories that describe the behaviour of your application, that can be automatically executed to test if it's working as expected.

For some of our projects, we've embraced Behat so much that we've taken to writing the acceptance criteria for our stories in the Given, When, Then format. This has the advantage that our tests are already written when we start the story. So because we want to adapt our tests to the natural language of the acceptance criteria, rather than the other way around, we are regularly adding to our FeatureContext class to define the custom step definitions.

In order to make these step definitions as flexible as possible, we're using the DrupalDriver on some of our projects instead of the BlackBox or DrushDriver.

This means we have access to the full Drupal API's in our step definitions and any content changes that are made as part of the tests can be cleaned up in the after scenario method.

Using the Drupal Driver

In order to use the Drupal Driver you need to configure your behat.yml to nominate the api_driver. Our behat template looks like below. The ${app.url} and ${app.dir} placeholders are populated from our build.properties file using a phing build target so each developer can specify their local environment settings, such as the URL to their dev site, and the directory it lives in.

annotations:
  paths:
    features: features/annotations

closures:
  paths:
    features: features/closures

default:
  paths:
    features:  features
    bootstrap: features/bootstrap
  extensions:
    Behat\MinkExtension\Extension:
      base_url: ${app.uri}
      goutte: ~
      selenium2: ~
    Drupal\DrupalExtension\Extension:
      api_driver: "drupal"
      blackbox: ~
      drush:
        root: ${app.dir}
      drupal:
        drupal_root: ${app.dir}

drupal7:
  filters:
    tags: "@api"
  extensions:
    Drupal\DrupalExtension\Extension:
      api_driver: "drupal"
      drupal:
        drupal_root: ${app.dir}
      drush:
        root: ${app.dir}

So with that in place you can access Drupal API's in your custom step definitions.

The problem we hit

So what was our problem? Well testing manually, functionality worked just fine, but testing with Behat would net weird results. Our tests would pass fine for the first few steps, then after a particular point in the test suite, every path that used a page-manager page would fail.

The problem was easily diagnosable by adding:

And then show last response

to our steps and inspecting the page. Using this we could see that the standard node page view was being used instead of our page manager variant.

Our project scaffold includes a basic Behat feature to check that the home-page returns a 200 Ok response, for this project it too was a page manager page and even it was failing. We knew for certain that the front page wasn't a 404, but that's what Behat was telling us, and if we visited the home-page immediately after running the tests it in fact was a 404.

As soon as we cleared the cache - we got the home page back.

Diagnosing the issue

Now this is one of those times and another one of those reasons where contributing to core makes you more knowledgeable and your day-job just that little bit easier :-).

Because of our familiarity with core, we were able to diagnose the issue straight away. It was clear that something was triggering a menu rebuild and because we run Behat using phing from outside the Drupal root, we weren't getting all of the necessary include files loaded, particularly those include files that defined the default page manager variants.

Now the DrupalExtension knows this, and if you use the 'and the cache has been cleared' the step definition it provides in DrupalContext is aware of this. It does a chdir() to the Drupal root first, then resets the working directory afterwards. But it was clear that something else was triggering a menu rebuild during our test suite.

So knowing that menu_rebuild() is the function that does the rebuild and that it lives in includes/menu.inc, we opened up the file and hacked in some code to print the backtrace when the function was called. We re-ran our tests using phing and saw this trace right where things started going astray.

Array
(
    [0] => menu_rebuild
    [1] => menu_get_item
    [2] => _pathauto_path_is_callback
    [3] => _pathauto_set_alias
    [4] => pathauto_create_alias
    [5] => pathauto_user_update_alias
    [6] => pathauto_user_insert
    [7] => user_module_invoke
    [8] => user_save
    [9] => Drupal\Driver\Cores\Drupal7::userCreate
    [10] => Drupal\Driver\DrupalDriver::userCreate
    [11] => Drupal\DrupalExtension\Context\DrupalContext::assertAuthenticatedByRole
    [12] => call_user_func_array
    [13] => Behat\Behat\Definition\Annotation\Definition::run
    [14] => Behat\Behat\Tester\StepTester::executeStepDefinition
    [15] => Behat\Behat\Tester\StepTester::executeStep
    [16] => Behat\Behat\Tester\StepTester::visit
    [17] => Behat\Gherkin\Node\AbstractNode::accept
    [18] => Behat\Behat\Tester\ScenarioTester::visitStep
    [19] => Behat\Behat\Tester\OutlineTester::visitOutlineExample
    [20] => Behat\Behat\Tester\OutlineTester::visit
    [21] => Behat\Gherkin\Node\AbstractNode::accept
    [22] => Behat\Behat\Tester\FeatureTester::visit
    [23] => Behat\Gherkin\Node\AbstractNode::accept
    [24] => Behat\Behat\Console\Command\BehatCommand::runFeatures
    [25] => Behat\Behat\Console\Command\BehatCommand::execute
    [26] => Symfony\Component\Console\Command\Command::run
    [27] => Symfony\Component\Console\Application::doRunCommand
    [28] => Symfony\Component\Console\Application::doRun
    [29] => Behat\Behat\Console\BehatApplication::doRun
    [30] => Symfony\Component\Console\Application::run
)

So most of our steps start with Given I am logged in as a user with the role "foo" because we're testing functionality on the behalf of a particular user, and this in turn uses DrupalContext::createUser(), which uses user_save(), which in turn invokes hook_user_insert()

Now because our site uses pathauto, pathauto_user_insert() fires which in turn calls menu_get_item() which can't find an item for the given user object (as it's being created) so a menu rebuild occurs, and because this is outside the Drupal root, we get a partial rebuild, as a fair swag of include files aren't loaded.

Fixing the problem

So because Behat and DrupalExtension live in an Object oriented world, fixing it wasn't too difficult, our FeatureContext already extends DrupalContext, so we just reimplemented the assertAuthenticatedByRole() method, being sure to add:

'path' => array('pathauto' => 0)

to the user object passed to DrupalDriver::userCreate(). This prevents pathauto from calling menu_get_item() and thereby rebuilding the menu.

Then we fired up our tests again, and got a little bit further but hit the same issue, but with a different backtrace.

This time the trace went something like this -> DrupalDriver::createNode() -> node_save() -> invoke hook_node_insert() -> pathauto_node_insert(). I.e. the same thing again but for nodes, so we added the same code to anywhere our FeatureContext called DrupalDriver::createNode(), cleared the cache using drush and then ran it again.

Success!

Summing up

• Behat and DrupalExtension are awesome, coupled with phing and a CI build, you can automate your testing process and run a test suite against your acceptance criteria to ensure you don't get regressions.
• Working with the Drupal driver means you can get pretty creative with your step definitions and access the full Drupal API. But because you're running Drupal from outside the Drupal root, sometimes things can go astray.
• If they do, having a solid knowledge of core built up from your experiences as a contributor can help you pinpoint the issues without much loss of time
• Backtraces are your friend
• Be sure to check out Boris Gordon'ssession on phing and CI at Drupalcon Prague.