matt glaman

Next stages for the Drupal Starshot trial experience

Published on

Drupal CMS is the official name for Drupal Starshot. We officially have the Drupal CMS project on Drupal.org, where the previous prototype has been converted into the official codebase. This monolithic repository will contain the Composer project and packages (recipes) that makeup Drupal CMS. 

The announcement came at a perfect time. Shortly before the announcement, I had a call with some folks at the Drupal Association about plans for hosting the trial experience I have been working on. During the call, we devised a solution to leverage GitLab Pages. This way, there is no bespoke hosting setup, and it leverages existing tools. After checking in with the rest of the Drupal Starshot team, it was decided the trial codebase would be added to the monolithic repository. 

Alongside these developments, the root causes of some bugs have been identified and are being squashed (and explained later). I am also working to make the JavaScript code that connects various Web APIs and WebAssembly shareable so others can build with the same tools.

Try out Drupal CMS here: https://project.pages.drupalcode.org/drupal_cms/

Using GitLab pages to host the trial experience

Before adding the trial code to the Drupal CMS repository, Adam G-H (phenaproxima) built a GitLab CI job that creates the trial artifact. He also discovered that artifacts on GitLab can be made public and accessible over a specific URL! GitHub Actions does not support this and requires uploading to S3 so that it is available. The job is called build trial artifact, and the artifact is named trial.zip. It's accessible via the following URL:

https://git.drupalcode.org/api/v4/projects/157093/jobs/artifacts/0.x/raw/trial.zip?job=build+trial+artifact

The web application can download the artifact directly without including it in the deployed assets. Any time Drupal CMS is updated, the trial artifact is updated and immediately usable without redeploying the web application.

Next, I opened an issue and provided a few merge requests to get the trial web application added to the repository and deployed via GitLab Pages. This part was easy as I copied and pasted files. I just needed to set up GitLab CI. The trial experience codebase is in the trial directory.

There are a few tests, so there is a run trial tests job:

run trial tests:
  stage: test
  script:
    - cd trial
    - npm ci
    - npm run test

This way, we can provide some stability regarding what gets deployed.

Setting up GitLab Pages was easy. You need a job named pages in the .gitlab-ci.yml file. All static files for the trial experience are in trial/public. We tell GitLab CI that trial/public is an artifact path and the publish path. By default, GitLab assumes the publish path is a root-level public directory. This part took a little trial and error as we weren't aware of the publish⁠⁠⁠⁠⁠⁠⁠ setting, and the deployment failed initially.

pages:
  stage: deploy
  needs:
    - 'run trial tests'
  artifacts:
    paths:
      - trial/public
  publish: trial/public
  rules:
    # Only deploy to pages if we're on the default branch, on a push, and not a fork.
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_ROOT_NAMESPACE == "project"
      changes:
        - trial/**/*
        - .gitlab-ci.yml
  script:
    - cd trial
    - npm ci
    - npm run build
    - echo "Trial experience was deployed at ${CI_PAGES_URL}"

And now we have the Drupal CMS trial experience running at https://project.pages.drupalcode.org/drupal_cms/. In the future, a URL such as try.drupal.org/drupal-cms or try.drupalcode.org/drupal-cms will be provided.

In the future, I aim to have a @mglaman/wasm-drupal package on NPM to contain the core functionality that makes using php-wasm possible in the trial experience codebase. This will allow other organizations to run their own WebAssembly experiences with Drupal and provide a centralized area to handle bugs that aren't specific to Drupal CMS.

Safari support

Safari support has been an issue since I first started the trial prototype. After spending time creating debuggable builds of php-wasm, it turns out the error has nothing to do with the WebAssembly code itself. Eventually, I was able to get a few stack traces with named functions returned, including the root error:

RangeError: Maximum call stack size exceeded.
php-worker.mjs.wasm.wasm-function[zendparse]
php-worker.mjs.wasm.wasm-function[zend_compile]
php-worker.mjs.wasm.wasm-function[compile_file]
php-worker.mjs.wasm.wasm-function[compile_filename]
php-worker.mjs.wasm.wasm-function[zend_include_or_eval]
php-worker.mjs.wasm.wasm-function[ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER]

I had read that sometimes if the code is too optimized with Emscripten or not optimized enough, there could be errors in some runtimes. However, it didn't matter how much I optimized PHP with Emscripten; the error always happened. I then started to read more about the specific call stack size error. I found an Emscripten issue that provided some insight. It has to do with the stack size of the JavaScript engine. Which means it is something set when the browser is compiled. That's near impossible to fix.

I don't have an iPhone or iPad. Maybe iOS Safari has a higher stack size and it's only an issue on macOS Safari.

I still want to work on this bug, but I've been focusing on supporting Drupal CMS with some other long-standing bugs (which are almost all fixed!)

Removing the "too many redirects" error on the Starshot prototype

One module added to the Starshot prototype early on was the Webform module; a demo form is also provided when you install the Starshot prototype. There was one problem. Any form submitted (including user login) when using the Starshot prototype within the WebAssembly runtime errored on "too many redirects." This did not happen when only running Drupal core, so it had to be a contributed module. Since we merged the trial experience into the Drupal CMS repository and had proper builds deployed to GitLab Pages, I knew this needed to be resolved quickly. 

I wish I knew why, but it's due to the Redirect module, even though no redirects are defined. For now, we're disabling Redirect from the trial artifact. Since I can't use Xdebug in WebAssembly, there will be some good old-fashioned die and throw new Exception debugging from within the Redirect module. I've read the code and honestly cannot fathom what would be causing the module to cause a redirect. Or at least during the bug, the service worker is reporting an HTTP 307 response (which isn't a regular redirect status code from Drupal.)

You can follow the issue to learn more about the details once they're figured out: https://www.drupal.org/project/drupal_cms/issues/3468968

Resolving random session loss when logged into Drupal

An issue that has bothered me even more than the "too many redirects" issue was the apparent session loss for users. After logging into Drupal, after 30 seconds to a minute, the user would get logged out. I had no idea why! I tackled the problem from the wrong angle. I assumed it had to be a quirk with session handling and something gone awry with session garbage collection or who knows what. Then I realized: what if it's not the PHP session going away? What if the session cookie is lost?

It turns out that was the case. Inside php-cgi-wasm, cookies set by PHP code are tracked as a property.

this.cookies    = cookies    || new Map;

When a response from the PHP code sends the Set-Cookie header, the service worker adds it to the map.

if(parsedResponse.headers['Set-Cookie'])
{
	const raw = parsedResponse.headers['Set-Cookie'];
	const semi  = raw.indexOf(';');
	const equal = raw.indexOf('=');
	const key   = raw.substr(0, equal);
	const value = raw.substr(1 + equal, -1 + semi - equal);

	this.cookies.set(key, value,);
}

Then, when a request is made, it sets those cookies to PHP's HTTP_COOKIE server variable. 

putEnv(php, 'HTTP_COOKIE', [...this.cookies.entries()].map(e => `${e[0]}=${e[1]}`).join(';') );

However, a service worker's lifecycle isn't guaranteed. The cookies are stored in memory and can be lost whenever the service worker is reinitialized. There needed to be persistent storage to place the cookies. Web Workers cannot access session or local storage since they do not run on the main thread. They do have access to IndexDB. So, I wrote a class that extends Map and uses IndexDB as persistent storage and the map as in-memory storage. You can see it here: https://github.com/mglaman/wasm-drupal/blob/main/public/cookie-map.mjs.

This has solved the case of missing cookies and lost sessions.

 

#

Want more? Sign up for my weekly newsletter