This is the second article in a series of articles looking at migrating from Jadu into a LocalGov Drupal (LGD) site for Central Bedfordshire. In the first article we looked at the Jadu API and setting things up so that we could make calls to the API and parse the XML data using the migration systems available.
In the last article I mentioned something about the Jadu API that caused me a lot of headaches. The API contains most of the information for a page, but critically, the Jadu API contains no information about the path of a page. There is basically no way to get the URL of a page in Jadu from the XML API.
I'm quire sure that this makes creating anything useful in the API a real pain since referring back to the site needs to be done with manually placed links, but it's clearly like this by design. I couldn't find any documentation on why it is like this, but it almost feels like vendor lock-in. Please correct me if I'm wrong here.
If you migrate a page from one system to another then it is highly important that you maintain the URL structure of the site. If you change the URL of a page then you need to add in a step that adds a redirect from the old system to the new so that all of your search engine results, the existing links from other sites, and any user bookmarks that have been created work correctly. This is critical to get right for a public facing council site like this.
Since I was migrating into a LGD site, it made sense to use the Drupal path auto system and LGD path management plugins to manage the paths on the Drupal site. We therefore needed to know the existing Jadu URLs so that we could create these redirects.
To get the URLs during the migration caused quite a bit of experimentation, but I did solve the issue with a solution that had a high success rate.
In this article we will look at methods used to find the URLs of the Jadu site before moving onto the solution that solved the problem. We will then look at how those URLs are used in the migration of the site.
What I Tried
Before I get into how I solved this issue let's look at some of the methods I looked at to show what didn't work.
Constructing the Jadu URL from information in the API XML
I actually went down this road as there is documentation from Jadu about how URLs are generated and I had success doing this for a different (non-Jadu) project.
According to the documentation, Jadu URLs can be generated from the title of the page, using the following rules:
- Replacing spaces with dashes
- Removing stop words from the title
- Changed to lowercase
- Special characters removed
Unfortunately, from testing the migrated content with known Jadu URLs I found that this method had about a 60% success rate, which isn't good enough. The main issue is that these instrcutions are clearly incorrect, many URLs on the site do not remove uppercase characters, or strip out dashes or stop words. There were even a few cases where special characters were still present in the path.
Also, there was an issue where it was impossible to create paths for hierarchical pages as there wasn't enough information at hand to generate the path in those situations.
Pulling data from the Jadu site
I toyed with the idea of looking at the front end of the site (i.e. not the API) to find the URLs for each page, but not only was it difficult to generate the Jadu paths, it was also impossible to confirm that I had the correct URL.
For example, let's say we had a page with the path /news/article/1473/housing_matters_-_january_2026 for a news article. I could attempt to generate the path using the title of the page and Jadu's rules on creating URL slugs. Unfortunately, if I attempt to access the same page with the completely incorrect URL /news/article/1473/monkey-monkey-monkey it results in a 200 status code with the correct content. The page itself doesn't have a canonical URL metatag so there is no way to verify that the URL I attempted to fetch is "correct" using the site.
Also, because the Jadu API is rate limited you need to slow down how often you access the site of you will be banned. To bypass this rate limit I added an artificial delay of 3 seconds to all of the API requests in the migration, which was an acceptable delay in the migration. Adding more API calls would have slowed down the migration quite a bit. In fact, to migrate the 48K URLs into the site with a standard delay of 3 seconds would take about 40+ hours. That clearly wasn't going to work.
Finally, there is also the issue of hierarchical pages. It's not possible to find out the position of a page in the hierarchical structure of the site based on the metadata and URL of the page.
Solving The URL Problem
It became clear that what I needed was a way to find the URL of a page from a known list of URLs, rather than trying to reconstruct or attempt to pull them from the site.
The solution was to pull together a list of URLs and compile them into a data structure that I could then use during the migration.
The site had a sitemap.xml file that I could use, although it was clearly corrupt as the XML wasn't valid I could at least get lots of URLs from the file. In the run up to the migration the client had been collecting URLs from the site via SEO sites scanning tools so this was able to fill in some of the gaps that existed in the sitemap.xml file. Outside of that I also had a number of URLs that didn't appear to exist anywhere else, but were valid pages on the site (found whilst running the migration and comparing the new paths with the imported redirects).
In the end this was around 48K URLs, which included links to images and other files in use on the site.
This mess of URLs was processed and collected together into a single array so that I could filter out any duplicates and make sure they were all of the same format. The array was then used to generate a PHP class that contained a single, static, property that contained all of the URLs available. I used a simple PHP script to generate this class, but the result was that I could add any number of URLs to the list and ensure that everything was accounted for and that I could easily add more URLs to the list.
Here is the class I generated, sans the 40K+ URLs of course.
<?php
declare(strict_types=1);
namespace Drupal\jadu_migrate\Service;
/**
* Lookup class for the Jadu URLs.
*/
class JaduUrls extends BaseJaduUrls {
/**
* The URLs.
*/
public const array JADU_URLS = [
// A whole load of URLs.
];
}
Notice that the JaduUrls class extends a base class called BaseJaduUrls. I decided to extend a class so that I could add some URL finding functionality to the JaduUrls class, without needing to open a file with a colossal array defined in it. This base class took care of all of the logic behind pulling information out of the list and returning the found URL and the parent class contained the list of URLs.
Here is a section of that class, containing the logic for finding the original URL of a service landing page or a news page.
<?php
declare(strict_types=1);
namespace Drupal\jadu_migrate\Service;
/**
* Lookup class for the JaduUrls.
*/
class BaseJaduUrls {
/**
* The URLs.
*/
public const array JADU_URLS = [];
/**
* Find a URL by an ID and the type of page.
*
* @param string $id
* The ID.
* @param string $type
* The type of page.
* @param string|null $parentId
* An optional parent ID. This is used for guides pages especially.
*
* @return string|null
* The path, or null if nothing was found.
*/
public static function findUrlById(string $id, string $type, ?string $parentId = NULL): ?string {
// Set the parent ID place to be 0, which will never match.
$parentIdPlace = 0;
// Create the URL information from the paths.
// Note that the path contains a starting "/" so the counts here are all
// increased by 1 to allow for that.
switch ($type) {
case 'localgov_services_landing':
// Generated from category pages.
// URL format is:
// "/info/<id>/page_title".
$urlStart = 'info';
$urlParts = 4;
$pathIdPlace = 2;
break;
case 'localgov_news_article':
// Generated from news pages.
// URL format is:
// "/news/article/<id>/page_title".
$urlStart = 'news';
$urlParts = 5;
$pathIdPlace = 3;
break;
////////////////////////////////
// More types of page added here.
////////////////////////////////
}
if (!isset($urlParts) || !isset($urlStart) || !isset($pathIdPlace)) {
// Something we don't know about.
return NULL;
}
foreach (static::JADU_URLS as $url) {
$parts = explode('/', $url);
if ($parts[1] === $urlStart && count($parts) == $urlParts) {
if ($parentId !== NULL) {
if ($parts[$parentIdPlace] == $parentId && $parts[$pathIdPlace] === $id) {
return $url;
}
}
elseif ($parts[$pathIdPlace] === $id) {
if (isset($parts[3]) && $parts[3] === 'a_to_z') {
// This produces the wrong URL for the page.
continue;
}
return $url;
}
}
}
return NULL;
}
}
With this in place we can now interrogate the URL list using the ID of the original page and the type of page we are migrating.
$url = JaduUrls::findUrlById("123", 'localgov_news_article');
The base class also allowed me to create a subclass in order to test the URLs. Unit testing this URL finding code is critical to ensure that everything worked correctly and the right ID/type combination found the right URL from the list.
Here is the outline of the unit test URL mocking class.
<?php
declare(strict_types=1);
namespace Drupal\Tests\jadu_migrate;
use Drupal\jadu_migrate\Service\BaseJaduUrls;
/**
* Creates a test class with a known set of URLs to test with.
*/
class MockJaduUrls extends BaseJaduUrls {
/**
* The URLs.
*/
public const array JADU_URLS = [
'/news/article/123/test-test-test',
// More URls here to test.
];
}
Using this class I could then create a simple test that used a data provider to run a number of tests on URLs that I could control. The idea behind creating a mock URL class was so that I could add in a number of test URLs, and also a number of URLs designed to test the logic so that we could be sure that the correct URL would be picked up. As those test URLs shouldn't have redirects in the site they were therefore not part of the actual URL list. This allowed the tests to run correctly, even if the main URL list was updated and the URL list updated. I was testing the functionality of the URL finder so it didn't really matter what the URL was, as long as it was correct.
The test function is pretty simple and consists of the following.
public function testJaduUrlFind(string $id, string $type, string $expectedResult, ?string $parentId = NULL): void {
$jaduUrl = MockJaduUrls::findUrlById($id, $type, $parentId);
$this->assertEquals($expectedResult, $jaduUrl);
}
With all of that in place I could now concentrate on looking at the migration.
Redirect Migration
As I mentioned above, when generating new content on the site we used the Drupal and LGD path auto systems to generate the paths for the content. This meant that we didn't have to change how the paths are generated in LGD, which would create a maintenance issue as the site would have deviated from the core path structure.
Every time we created a page we needed to also create a redirect between the old Jadu URL and the new Drupal one. This allows the new site to function in the same way as the old site so that all search engine listings, external links, and any bookmarks still worked with the new content. A critical consideration, especially for a public facing government site as this is.
Migrating URLs into a site can be done in two ways. If you know the old and the new URLs you can just migrate directly into the redirect system since that's a known quantity. Due to us not knowing this structure exactly before hand I therefore had to use the second method, which is to hook into the migration and run the needed logic to generate the redirects as the content is created. In order to do this we need to use the post row save event so that we can add redirects to the site after the new item of content has been added.
The following class is the even subscriber for the Jadu migrate module that will fire the onMigratePostRowSave method when the MigrateEvents::POST_ROW_SAVE event is triggered. Using this event we know that the new entity is in place and so we can add extra items to the database to fill in any gaps in the data (like the redirects for example).
<?php
declare(strict_types=1);
namespace Drupal\jadu_migrate\EventSubscriber;
use Drupal\Core\Database\Connection;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\jadu_migrate\Service\AliasRedirectInterface;
use Drupal\migrate\Event\MigrateEvents;
use Drupal\migrate\Event\MigratePostRowSaveEvent;
use Drupal\migrate\MigrateLookupInterface;
use Drupal\migrate\Plugin\MigrationPluginManagerInterface;
use Drupal\node\NodeInterface;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
/**
* Migration event subscriber for Jadu migrate.
*/
class MigrationEventSubscriber implements EventSubscriberInterface {
/**
* Creates a MigrationEventSubscriber object.
*
* @param \Drupal\Core\Entity\EntityTypeManagerInterface $entityTypeManager
* Drupal entity type manager.
* @param \Drupal\migrate\MigrateLookupInterface $migrateLookup
* Drupal migration lookup service.
* @param \Drupal\migrate\Plugin\MigrationPluginManagerInterface $migrationPluginManager
* Drupal migration plugin manager.
* @param \Drupal\jadu_migrate\Service\AliasRedirectInterface $aliasRedirect
* Drupal alias redirect service.
* @param \Drupal\Core\Database\Connection $database
* Drupal database connection.
* @param \Drupal\jadu_migrate\Service\FixMarkup $fixMarkupService
* Fix markup service.
* @param \Drupal\jadu_migrate\Service\WysiwygFileInterface $wysiwygFile
* Wysiwyg file service.
*/
public function __construct(
protected EntityTypeManagerInterface $entityTypeManager,
protected MigrateLookupInterface $migrateLookup,
protected MigrationPluginManagerInterface $migrationPluginManager,
protected AliasRedirectInterface $aliasRedirect,
protected Connection $database
) {}
/**
* Get subscribed events.
*
* @inheritdoc
*/
public static function getSubscribedEvents(): array {
$events[MigrateEvents::POST_ROW_SAVE][] = ['onMigratePostRowSave'];
return $events;
}
/**
* Triggered after a row is saved in the migration.
*
* @param \Drupal\migrate\Event\MigratePostRowSaveEvent $event
* The event object.
*/
public function onMigratePostRowSave(MigratePostRowSaveEvent $event): void {
$migrationId = $event->getMigration()->getBaseId();
if ($migrationId === 'news') {
$row = $event->getRow();
$destination = $event->getDestinationIdValues();
$this->aliasRedirect->createRedirect(
'localgov_news_article',
(int) $row->getSourceProperty('id'),
(int) $destination[0]
);
}
}
}
There's a lot more code in this file, but I've reduced it down to just the new migration here to save on space for the news articles. What we do in this event is to collect data from the current migration and pass this to a service class created for the project called AliasRedirect. This takes care of the creation of the redirect.
The AliasRedirect service class takes the Drupal\redirect\RedirectRepository class as a single dependency. It can then use that redirect service to find out if a redirect exists and to generate a new redirect for a given path. The path is found by using the JaduURls class created above. Much of the code in this class is used to ensure that the redirection path is good and that we aren't creating duplicate redirects.
<?php
declare(strict_types=1);
namespace Drupal\jadu_migrate\Service;
use Drupal\redirect\Entity\Redirect;
use Drupal\redirect\RedirectRepository;
/**
* Create a path alias based add a redirect from the old URL where necessary.
*/
class AliasRedirect implements AliasRedirectInterface {
public function __construct(protected RedirectRepository $redirectRepository) {}
public function redirectExists($redirectSource):bool {
$redirects = $this->redirectRepository->findBySourcePath($redirectSource);
if (count($redirects) > 0) {
return TRUE;
}
return FALSE;
}
public function createRedirect(string $type, int $oldId, int $newId, ?int $parentId = NULL):void {
switch ($type) {
case 'localgov_news_article':
$jaduPath = JaduUrls::findUrlById((string) $oldId, $type);
break;
default:
// No redirect logic for this type.
return;
}
if ($jaduPath === NULL) {
// Don't create a redirect for a path we don't have.
return;
}
if (str_starts_with($jaduPath, '/')) {
// Remove the first slash if present.
$jaduPath = substr($jaduPath, 1);
}
if ($this->redirectExists($jaduPath) === TRUE) {
// Redirect already exists.
return;
}
$redirect = Redirect::create([
'redirect_source' => $jaduPath,
'redirect_redirect' => 'internal:/node/' . $newId,
'language' => 'und',
'status_code' => '301',
]);
$redirect->save();
}
}
With this event and service class in place we now have a mechanism to generate redirects for the pages of content going into the site. The above example shows this for the news pages only, but the same mechanism was also used for documents, events, pages, events, and other Jadu content types.
Conclusion
With everything working together here we had a good mechanism for both "generating" Jadu URLs and for creating redirects between one site and the other.
Getting to this point took quite a bit of trial and error. It quickly became apparent that generating the Jadu paths in the Drupal site was not going to work and that simply collecting together all of the information we had at hand was the best solution.
I could have changed things slightly so that the URLs could have been read from a CSV file or a database table, but both of those mechanisms require extra work, extra code, and pre-migration setup to get working. In my experience, once the migration is done there is normally a bit of a gap where you concentrate on working on other bits of the site. I had lots of good documentation on how the migration was created, setting things up, and running the migration, but having extra steps in the migration would have complicated things further than they needed to be.
Creating a massive array in a PHP class meant that it was as simple as it could be. I didn't need to rely on making sure a database table was up to date or make sure that a CSV file was in the right place. I could also use the Drupal unit testing functions to test the code without mocking any Drupal services.
In addition to what you see here I also created a Drush command that generated a report on the pages in Drupal and all of the redirects created for those pages. This allowed us to see if any gaps existed in the redirects before we launched as we knew that every page created in Drupal after the migration must have at least one redirect one created, which is from the original Jadu URL.
As this system wasn't perfect we did spot a few missing URLs. Creating a report to show gaps was important as it allowed us to track down a couple of edge cases where the logic wasn't working and where we didn't have records of the URLs in question. Even then there were some redirects that needed adding in manually and we are keeping an eye on the 404 responses on the site to catch any other missing pages that might have crept in.
Migrating redirects is never simple, but it is also never the end of the process. You should always look at any 404 messages coming from the site to ensure that you add redirects for any missing content to relevant pages.
I will release the source code for this migration in the coming weeks. As I write these articles I am tidying things up so that the module can be released as a self contained Jadu to LGD migration starter module. There is some custom logic for Central Bedfordshire that might not be useful for everyone so I'm working on either removing that or creating configuration to allow those customisations to be configured.
If you are looking to migrate from Jadu, or are trying to get to grips with the migration system then please get in touch!
In the next article I will look at migrating document and document content from Jadu into Drupal.
Add new comment