Last year at DrupalCon Portland 2022, Dries announced that "Drupal is for ambitious site builders." It refined the "Drupal is for ambitious digital experiences" vision. It chooses to focus on a specific persona and improve their experience with Drupal. It didn't sit perfectly well with me, so I wrote about how improving the Drupal developer experience empowers the Ambitious Site Builder. And with Drupal 10, there has been a huge shift in the development experience for frontend developers. Drupal 10.1.0 will bring even more improvements to the frontend developer experience. One of those will be the new Development Settings form to manage Twig development mode and markup caching.

The pain of bypassing markup caching when building Drupal

One reason I love building applications in Drupal is its Cache API. There isn't anything like it in other frameworks. We have fragment caching for our markup. That then builds into dynamic page caching for partially caching responses of pages with uncacheable content. Then we have the page cache for full response caching like a reverse proxy. Amazing for production performance but a roadblock for local development.

There is a way to bypass these caching layers, and you can do so if you know to read the sites/example.settings.local.php file in your Drupal code base. Here are the steps required to setup bypassing these caches:

1. Copy sites/example.settings.local.php to sites/default/settings.local.php
2. Remove comment to allow render cache bin to be bypassed (source)
3. Remove comment to allow page cache bin to be bypassed (source)
4. Remove comment to allow dynamic_page_cache cache bin to be bypassed (source)
5. Run chmod 775 sites/default to ensure it's writeable after Drupal hardened the directory's permissions
6. Edit your sites/default/settings.php to allow including your new settings.local.php file
7. Rebuild caches using Drush or the user interface

All right! We did it! Wait, no? Yes, that's right. This was only part of the battle! This will bypass markup caching for your content within render arrays in your blocks and controllers. It does not bypass Twig's template caching. That means changes to Twig templates won't show up automatically yet! This next part is convoluted and the most painful part of the process.

1. Open sites/development.services.yml in your editor
2. Copy the twig.config parameter defaults from sites/default.services.yml
3. Paste the twig.conifg parameter defaults into the development.services.yml file under parameters
4. Modify debug to be true to enable Twig's debug mode
5. Optionally set cache to be false to bypass template cache and only reload templates if they have been modified
6. Rebuild caches using Drush or the user interface

(Note you need to copy and paste the entire twig.config contents because array-based parameters are not merged with other defaults when defining service parameters.)

And now we are there! If you modify a Twig template in your theme the changes will automatically appear without being caught behind various layers of cache. That is, until you run composer install or composer update and Drupal's scaffolding plugin destroys your modifications to sites/development.services.yml. The sites/development.services.yml is provided by Drupal core and automatically placed as part of Drupal's scaffolding.

1. Read how to exclude scaffolding files from the documentation
2. Edit your project's composer.json file
3. In extra.drupal-scaffold.file-mapping set [web-root]/sites/development.services.yml to false
4. Run composer update --lock to ignore checksum warnings

Many Drupal agencies have set up project templates or tooling that scaffold this setup. I am one of them. You can browse some boilerplate that I copy around: composer.json and development.services.yml.

As you can see, this requires understanding these details as an established Drupal agency or consultant. What about all of the new developers we want to attract to Drupal? What about those new frontend developers being brought in to work on a Drupal site instead of WordPress or Next.js? They need to have a delightful experience that makes them want to work with Drupal again.

The new Development Settings form in Drupal 10.1.0

One fix would have been to improve the default development.services.yml to have defaults for Twig and Render debugging. But that still wouldn't solve the root problem. It's hard to get template modifications to "show up" when building your Drupal site. It should be easier than having to copy and edit files. And now, it will be as simple as pressing a button on the Development Settings form in Drupal 10.1.0, releasing in June!

I opened an issue after DrupalCon Portland that we needed to make it easier for theme builders to enable Twig debugging and disable the render cache. We couldn't have just one or the other. If you have Twig debug mode, your markup is still being cached. If you bypass markup caching, your Twig templates are still cached. They are very much intertwined. I worked on the concept for a few days on personal time but had to pivot as I had to begin focusing on Drupal 10 upgrade tooling.

Many contributors worked on this issue to provide user experience feedback, testing, and code contribution. I recommend reading Mike Herchel's blog post about this new feature as it provides more history about managing to land this feature for Drupal 10.1.0. We just made the 10.1.x code freeze. Otherwise, this feature would have to wait for 10.2.0 in December 2023!

There is currently a follow up ticket. We never invalidate the Twig template cache or anny markup cache bins. That means when you re-enable Twig caching and markup caching the Drupal site shows stale content, like a time machine. I have an initial merge request opened to resolve this problem. It needs tests and review of the approach.

How it works

Now to the fun part! I am excited about this feature because it brings a new approach to manipulating Drupal's application state. These development settings rely on customizing the service container by editing various settings files. All previous steps have been consolidated into one form and two checkboxes.

State API versus Configuration API

When we think of administrative forms in Drupal, they are associated with modifying configuration objects to control how Drupal works. The new Development Settings form utilizes the State API for storing these settings. The State API provides a "simple system for the storage of information about the system's state." Values stored in the state are not meant to be exported and are intended to be specific to an individual environment. That is exactly where development settings should be stored.

If we had used configuration objects to store these settings, someone may perform a configuration export and accidentally commit these settings. Then they could be deployed and pushed to multiple environments.

If you have any sort of auditing tools for your Drupal site, the settings are stored with the following keys. If they exist and their value is true, then they are active.

• twig_debug
• twig_cache_disable
• disable_rendered_output_cache_bins

Modifying the service container with state values

For these development settings to take effect, we need to modify the service container when it is being built. To do so, we implemented a compiler pass to modify the service container when it is being compiled. The new DevelopmentSettingsPass compiler pass reads values from the state information and applies changes to the service container. It is kind of funky because it causes initialization of the state service during container compilation. But, this compiler pass runs after all services have been registered and any service modifiers have been executed. Here's an exerpt of the code in CoreServiceProvider where the new compiler pass is registered, amongst the other existing ones:

// Add the compiler pass that lets service providers modify existing
// service definitions. This pass must come first so that later
// list-building passes are operating on the post-alter services list.
$container->addCompilerPass(new ModifyServiceDefinitionsPass());

$container->addCompilerPass(new DevelopmentSettingsPass());

Why put the logic in a compiler pass rather than in the CoreServiceProvider class? Honestly it's for encapsulation of that code. Most modules that modify the service container implement a MyModuleServiceProvider and register or alter services in that service provider. Compiler passes are the method provided by Symfony's DependencyInjection component. The ServiceModifierInterface and ServiceProviderInterface concepts are additions Drupal has provided (so modules could add their own compiler passes, or other manipulations of the service container.) If this is a new concept, I recommend reading the documentation page Altering existing services, providing dynamic services.

The compiler pass checks if Twig debug mode is enabled or if Twig cache has been disabled. It then updates the twig.config parameter with the proper values. It is a lot simpler than having to copy a block of YAML content from one file to another!

/** @var \Drupal\Core\State\StateInterface $state */
$state = $container->get('state');
$twig_debug = $state->get('twig_debug', FALSE);
$twig_cache_disable = $state->get('twig_cache_disable', FALSE);

if ($twig_debug || $twig_cache_disable) {
  $twig_config = $container->getParameter('twig.config');
  $twig_config['debug'] = $twig_debug;
  $twig_config['cache'] = !$twig_cache_disable;
  $container->setParameter('twig.config', $twig_config);
}

Next, when disabling markup caching, we have to do two things. The cache.backend.null service is only defined if sites/development.services.yml has been included. The compiler pass registers the service if it does not exist. We then iterate through the relevant cache bins and set their default backend to cache.backend.null.

if ($state->get('disable_rendered_output_cache_bins', FALSE)) {
  $cache_bins = ['page', 'dynamic_page_cache', 'render'];
  if (!$container->hasDefinition('cache.backend.null')) {
    $container->register('cache.backend.null', NullBackendFactory::class);
  }
  foreach ($cache_bins as $cache_bin) {
    if ($container->has("cache.$cache_bin")) {
      $container->getDefinition("cache.$cache_bin")
        ->clearTag('cache.bin')
        ->addTag('cache.bin', ['default_backend' => 'cache.backend.null']);
      }
    }
  }
}

Fun fact! Before this code wasn't in a compiler pass. I refactored it into the compiler pass later to help keep the code clean and tidy for review. When I did this, I registered the DevelopmentSettingsPass compiler pass last. When I did that I hit a regression. Disabling markup caching no longer work! It runs out it was because the DevelopmentSettingsPass compiler pass ran after BackendCompilerPass. That compiler pass reads default_backend and properly injects the storage backend service. For the changes in DevelopmentSettingsPass to take effect it needed to run before BackendCompilerPass. That is why it currently runs after ModifyServiceDefinitionsPass.

Applying changes by invalidating the service container

Whenever you make changes to sites/development.services.yml, the cache bins in sites/default/settings.local.php, or sites/default/services.yml (if present) you need to rebuild Drupal's caches to have them take effect. That is because Drupal caches the service container definition and it needs to be rebuilt. However, rebuilding Drupal's caches causes it to have to a lot of other work besides rebuild the service container. Luckily it is possible to just invalidate the container so the container cache is purged. This provides a more precise action and keeps other caches intact – like plugin definitions.

When you submit the Development Settings form, we invalidate the service container so that settings are applied immediately. The logic is very verbose as we try to minimize invalidating the service container unless we have to when values change. I recommend checking it out in full: DevelopmentSettingsForm.php#L120-147.

But here is a quick highlight of how it works. Our from has DrupalKernel injected. The kernel is the root entrypoint of Drupal and handles requests and managing the service container.

  public function __construct(
    protected StateInterface $state,
    protected DrupalKernelInterface $kernel
  ) {
  }

  public static function create(ContainerInterface $container) {
    $instance = new static(
      $container->get('state'),
      $container->get('kernel')
    );
    $instance->setMessenger($container->get('messenger'));
    return $instance;
  }

Then in our submitForm method we call the kernel's invalidateContainer method. This deletes the cached container definition, which causes Drupal to recompile the container the next time it is requested.

$this->kernel->invalidateContainer();

And then, magic! The page load after form submit will have Twig debug comments in its markup and bypass markup caching.