This is the second article in a series looking at migrating from Jadu into a LocalGov Drupal (LGD) site for the Central Bedfordshire site. In the first article we looked at the Jadu API and setting things up so that we could make calls to the API and parse the XML data using the migration systems available.
In the second article we looked at reproducing Jadu URLs to create redirects for migrated content, even though the Jadu API doesn't contain any URL information.
Now that we have a the Jadu connection created and redirects working we can start tackling other aspects of the migration, so let's look at migrating the main content of the site from Jadu. We can pull structured pages of content out of the Jadu API and apply them to the structure of a LGD site, maintaining the same hierarchical structure and order of pages.
Much of the content of a LGD site is built using just a handful of content types, and Central Bedfordshire was built in the same way. LGD comes with a number of content types that allow content editors to add content in different ways, which allows for some decent customisation in the structure and layout of a site. Not only that, but as we are using Drupal it is possible to customise this structure as much as we need to suit the needs of the site.
For Central Bedfordshire, we had the standard structure of the site built with Service Landing pages, which gave editors the ability to link out to other pages that contained all of the content. These inner pages were set up as Guide pages, and the migration of that content is what we will be focusing on in this article. Please note that I will need to gloss over a little bit of the detail in this article, but rest assured that the source code will be available soon.
First, let's have a quick look at Guides in LGD.
Guide Pages
LGD Guide pages are useful if you have a collection of pages that you want to group together, but those pages are necessarily linear in their content. They are useful to give users a range of information on a given subject and come with a page list block, along with "Previous" and "Next" navigation buttons.
Guide pages consist of two different content types:
- Guide page - An individual page, containing content for the subject at hand.
- Guide Overview page - Used to collect together a set of guide pages into a collection and give an overview of what the content the guide contains.
The Guide Overview pages are used to create the navigation structure for the pages that they link to. When a content editor creates a Guide page they are required to inject the page into a Guide Overview page. After saving the page an entry is added to the Guide page and the Guide Overview page, linking the two pages together.
In Jadu, the bulk of the content we needed for the Guide pages is stored in the document and pages API (with pages being stored under documents). Let's look at getting the content we want out of Jadu, starting with documents.
Documents API
A document in Jadu is an item that has a URL, but doesn't have any content of its own. Instead, it's a stub that stores the position of the inner pages within the structure of the site. When a user visits the document the site will load in the first page of content.
The document endpoint can be referenced by the following endpoint.
https://[domain]/api/documents/all.xml
This returns an XML file that might look something like this.
<?xml version="1.0" encoding="utf-8"?>
<documents page="1" pages="30" per_page="50" total="1474">
<document id="1" pages="3" visible="true">
<title>The Document Title</title>
<date>1786880366</date>
<categories>
<category id="1" name="Some category"/>
</categories>
</document>
... More documents here ...
</documents>
As you can see, there isn't much here to migrate, just the title of the document a (undocumented) timestamp that we used as the creation date of the document.
This is pretty straightforward and the migration scripts just pulled the items out of this documents API and created them in the LGD site as Guide Overview pages. The migration script for documents was called, naturally, documents to show the data source.
Pages API
The pages of each document are pulled from a different API endpoint, and requires a document ID in order to access them.
This is the structure of the URL we need to access the pages of a document.
https://[domain]/api/documents/[DOCUMENT_ID]/pages.xml
We might get a response that looks like this.
<?xml version="1.0" encoding="utf-8"?>
<pages document_id="1" page="1" pages="1" per_page="10" total="1">
<page id="1" page_number="1">
<title>Page 1</title>
<image/>
<date>1786880366</date>
<content><![CDATA[<p>The page content.</p>]]></content>
</page>
<page id="2" page_number="2">
<title>Page 2</title>
<image/>
<date>1786880366</date>
<content><![CDATA[<p>The page content.</p>]]></content>
</page>
</pages>
We have the title of the page, an image (that never had any value so we didn't use it) and a (again undocumented) timestamp that we used for the creation date of the document. This can all be mapped from the XML into fields that we inject into Drupal and created as Guide pages. It's not quite as straightforward as a one-to-one migration as the first page in the list needs to become the Guide Overview page.
What we need to do first is inject the document ID into the page migration.
To facilitate the processing of XML data form the Jadu API we created a JaduSimpleXmlPager class, which I detailed in the first article in this series. The document ID needs to be injected into the URL of the page so we needed to update this class to allow this to happen.
The migration script in the project for pages was called documents_content, just to specify that it is connected to the documents migration and what the function of the migration is.
To add the needed components to the URL we create an entry in our migration script to hold the document ID, and a placeholder in the URL to inject the document ID into. We set the value of the document ID to 0 by default as we don't know what it will be until after we run the document migration.
source:
...
document_id: 0
urls:
- 'https://forms.centralbedfordshire.gov.uk/api/documents/[DOCUMENT_ID]/pages.xml'
Because this is a migration that pulls from two data sources (i.e. the document and the page) we need to create that mapping in the migration script.
source:
...
ids:
document_id:
type: integer
id:
type: integer
To create the hierarchical structure between Guide Overview and Guide pages we need to populate a field in the Guide page. This is done by using the migration_lookup plugin to populate the localgov_guides_parent field with the mapped value from the Jadu document ID. The migration lookup will find the corresponding Drupal ID of the document ID so that this link can be created.
process:
type:
plugin: default_value
default_value: localgov_guides_page
title: title
localgov_guides_parent:
- plugin: migration_lookup
migration: documents
no_stub: true
source: document_id
LGD will see this ID and create the entry in the Guide Overview page to link that page to the Guide page automatically.
Inside the JaduSimpleXmlPager class we now need to update a couple of methods.
In order to associate the page with the parent document we then need to inject the value of the document ID into the current item so that it can be saved correctly. The fetchNextRow() method can be updated in order to populate each row found with the document ID value from the configuration.
protected function fetchNextRow(): void {
parent::fetchNextRow();
if ($this->valid()) {
if (isset($this->configuration['document_id'])) {
// For the documents_content migration we need to inject the parent
// document ID for use in the migrate lookup of parent pages.
$this->currentItem['document_id'] = $this->configuration['document_id'];
}
}
}
The addJaduConfigToUrl() method is used to add credentials and pagination to the URL of the API callback. In addition to this, we also look for the presence of the document ID value in the migration configuration and swap this in the current URL.
public function addJaduConfigToUrl(string $url): string {
$path = UrlHelper::parse($url);
if (isset($this->configuration['jadu_key'])) {
$path['query']['api_key'] = $this->configuration['jadu_key'];
}
if (isset($this->configuration['document_id'])) {
$path['path'] = str_replace('[DOCUMENT_ID]', (string) $this->configuration['document_id'], $path['path']);
}
if (isset($this->configuration['pager'])) {
// Grab the pager configuration setup.
// We need to ensure that the per_page configuration is always added to
// the URL or we will get odd results with duplicate data.
$pager = $this->configuration['pager'];
if (isset($pager['per_page']) && str_contains($url, 'per_page=') === FALSE) {
$path['query']['per_page'] = $pager['per_page'];
}
}
return Url::fromUri($path['path'], [
'query' => $path['query'],
])->toString();
}
With that in place we now have a mechanism to pull documents and pages from the API and map them correctly in the Drupal migration data for the guide pages. One thing that's missing so far is how we add the document ID to the page migration, so let's look into that next.
Running The Documents Content Migration
Whilst we now have a migration script to pull pages from the Jadu site into Guide pages in LGD it doesn't do a lot on its own. In fact you can't run it directly as the document ID is missing and so it will not migrate anything.
In order to run a page migration we needed to have a document ID, but we first need a mechanism to inject that into the migration itself. We can inject the plugin.manager.migration Drupal service into a class to create an instance of our documents_content migration script. A second parameter is added to the createInstance() method to override anything that is in the configuration of the migration script itself. This means that we can easily inject the document ID into the migration script and run the migration for that set of pages using the following code.
/**
* Trigger the documents_content migration.
*
* @param int $documentId
* The document ID.
*
* @throws \Drupal\Component\Plugin\Exception\PluginException
*/
public function migratePagesContent(int $documentId): void {
/** @var \Drupal\migrate\Plugin\MigrationInterface $migration */
$migration = $this->migrationPluginManager->createInstance('documents_content', [
'source' => [
'document_id' => $documentId,
],
]);
// Set the migration to update any existing pages.
$migration->getIdMap()->prepareUpdate();
$executable = new MigrateExecutable($migration, new MigrateMessage());
$executable->import();
}
This assumes that we have injected the plugin.manager.migration service into the migrationPluginManager class property, which provides an instance of the class \Drupal\migrate\Plugin\MigrationPluginManagerInterface.
So where do we trigger this migration? Rather than create a Drush command or a custom migration source built around the mapping table, the documents_content migration is triggered automatically at the end of the documents migration. To do this we add the migratePagesContent() method to an event subscriber that is listening to the MigrateEvents::POST_IMPORT event being fired from the migration system, which is done when a migration is complete.
When this happens we look for the migration ID of documents and in the migration system. As we know that the documents migration has finished then we know that all of the document IDs from Jadu are stored in the database in the form of a migration mapping table. We then run a database query to pull out the data from this mapping table (using the migrate API to find out the table name) and run the document_pages migration against each document ID that we find.
class MigrationEventSubscriber implements EventSubscriberInterface {
public static function getSubscribedEvents(): array {
$events[MigrateEvents::POST_IMPORT][] = ['onMigratePostImport'];
return $events;
}
/**
* Triggered after the migration is complete.
*
* @param \Drupal\migrate\Event\MigrateImportEvent $event
* The event object.
*/
public function onMigratePostImport(MigrateImportEvent $event): void {
$migrationId = $event->getMigration()->getBaseId();
if ($migrationId === 'documents') {
// The documents migration has completed, so we now need to migrate
// the page content from those pages. We do this by loading all of the
// source IDs that have been migrated and asking Jadu for the page
// content via the API, which we do via a migration.
$migration = $event->getMigration();
// @phpstan-ignore method.notFound (mapTableName is not in the interface.)
$tableName = $migration->getIdMap()->mapTableName();
if ($this->database->schema()->tableExists($tableName) === FALSE) {
// No mapping table exists, so do not run the secondary migration.
return;
}
$query = $this->database->select($tableName, 'map')
->fields('map', ['sourceid1', 'destid1'])
->isNotNull('destid1')
->orderBy('sourceid1');
$results = $query->execute()->fetchAll();
foreach ($results as $result) {
$this->migratePagesContent((int) $result->sourceid1);
}
}
}
As the number of documents is quite small on the site (i.e. less than 3K) we run it was possible to run this simply in the trigger without any problems.
The small downside here is that even if you trigger the documents migration with a single item it will then run the full documents_content migration for all imported documents. That could be changed by altering the way in which the onMigratePostImport() event works, but for the project there was no need to migrate bits of the site like that.
Another issue is that we are running the entire set of documents_content migrations in one go, bit by bit. It would be better to split them up using the Drupal Batch API so that they ran sequentially and also printed out messages to the command line. For the purposes of the Central Bedfordshire site this mechanism was fine due to the (relatively) small number of pages and the fact that we ran everything through the command line. If you are wanting to migrate 10 or 100 times that number I would highly suggest using Drupal queue or batch operations to segment the workload a little.
With all of this in place we now have documents being pulled from the Jadu API to create Guide Overview pages, and then pages being pulled from the Jadu API to create Guide content pages that are linked to those outer pages. What we need to solve now is making sure that the first page of content is added to the correct place.
Moving Content Into Guide Overview
Due to the difference in architecture between Jadu and Drupal there is a step we need to perform in order to populate the Guide Overview page with content. As the documents in Jadu contain no content of their own we need to add some extra functionality so that we can populate the Guide Overview page with actual content from the pages API endpoint.
To do this we need to use another migration event to intercept the migration of a page before it is added to the site. This is done using the MigrationEvents::PRE_ROW_SAVE event, which is triggered after everything is ready to be created, but before any Drupal objects have been generated. The purpose of the trigger is to look at the page_number property in the source data and if the page number is 1 then redirect that content into the parent guide overview document.
public static function getSubscribedEvents(): array {
$events[MigrateEvents::PRE_ROW_SAVE][] = ['onMigratePreRowSave'];
return $events;
}
/**
* Triggered before a row is saved in the migration.
*
* @param \Drupal\migrate\Event\MigratePreRowSaveEvent $event
* The event object.
*/
public function onMigratePreRowSave(MigratePreRowSaveEvent $event): void {
$migrationId = $event->getMigration()->getBaseId();
if ($migrationId === 'documents_content') {
$row = $event->getRow();
if ($row->getSourceProperty('page_number') === "1") {
$parentID = $row->getSource()['document_id'];
$parent = $this->loadNodeByMigrationLookup('documents', $parentID);
if ($parent instanceof NodeInterface) {
// Attach the content of this page to the parent page.
$parent->set('body', [
'format' => 'wysiwyg',
'value' => $row->getSource()['content'],
]);
$parent->save();
}
// Skip the row.
throw new MigrateException('Skipping');
}
}
}
When we detect the first page in the list and have moved the content we then throw a MigrateException exception. This tells the migration to skip the current row, which means that the page itself doesn't get created in Drupal, only the content is migrated.
Now, when running the document pages migration, the first page encountered will form the content of the Guide Overview pages, with the subsequent pages in the series forming the Guide pages connected to that main overview page.
Conclusion
As you can see, there were a number of steps to go through when running this migration. To summarise though, we are performing the following steps:
- Migrate the document pages from Jadu and create Guide Overview pages as stubs for the content.
- When the document pages migration finishes, trigger a separate migration to migrate each page for that document.
- The first page in the page migration is skipped and instead the content of that page is injected into the Guide Overview page. Other pages are automatically injected into the Guide Overview page structure by LGD.
To run the entire migration for documents we only need to run the documents migration, the documents_content migration is then run automatically.
When set set up the site we originally started migrating Documents and Pages from Jadu into Services Sub-Landing Pages (with the Service Pages being a level above, migrated from Jadu categories). This changed and we opted to migrate documents into the guide pages structure, which was actually a little easier to setup.
This was one of the longest sections of the migration to run, but it still only took 90 minutes or so on a site that had no caches. In terms of numbers, there were around 3K document pages and a separate 20K pages to migrate.
I will release the source code for this migration in the coming weeks. As I write these articles I am tidying things up so that the module can be released as a self contained Jadu to LGD migration starter module. There is some custom logic for Central Bedfordshire that might not be useful for everyone so I'm working on either removing that or creating configuration to allow those customisations to be configured.
If you are looking to migrate from Jadu, or are trying to get to grips with the migration system then please get in touch!
In the next article I will look at migrating directories from Jadu into Drupal.
Add new comment