This is the first post in a series of posts looking at migrating content from a Jadu site into a Drupal site running the LocalGov Drupal (LGD) distribution. The site I was migrating content into was the new site for Central Bedfordshire, and whilst other implementations of Jadu may differ from what I describe here, it should be enough information for you to understand the process.
Jadu is a proprietary CMS application that is mostly written in PHP. Back in 2011 I went to a talk by one of the developers from Jadu at the PHPNW11 conference, so I was aware of the system. That talk, by the way, was interesting as they built a system called Phalanger that allowed PHP (and Jadu) to be run on a .NET environment. That project is now called PeachPie and it looks like it is still under active development.
LocalGov Drupal is a Drupal distribution that combines Drupal, some configuration, and a collection of modules with the aim of making it easier for councils to create websites. The functionality provided includes content pages, news pages, bus timetables, and waste collection systems. What's more, it's maintained by a vibrant community of people.
In this article we look at how the Jadu API is used to get hold of site content, and what we can do in Drupal to facilitate the migration of this content. We then look at a simple part of the migration as an introduction to the whole process. This won't be an introduction to the migration API as I will skip over some of the fine details of the API to keep things brief.
First, let's look at the Jadu API.
The Jadu API
Jadu provides an XML API that can be accessed via a key assigned to your user. This means that you'll need to have a valid user and a valid API key in order to access the XML feeds in a Jadu site. Getting the key is the responsibility of your Jadu administrator and is beyond the scope of this article. Once you have access you can grab the XML using a normal GET request.
For example, to get the news items available on the site you might make a request to the following URL:
http://[domain]/api/news/all.xml?api_key=[api_key]
Sending the request to the news feed returns a structure that looks a bit like this:
<?xml version="1.0" encoding="utf-8"?>
<newsItems page="1" pages="1" per_page="10" total="4">
<news id="2">
<title>3D printed moon building designs revealed</title>
<summary>The architects behind ...</summary>
<content><![CDATA[<p>An inflatable structure...]]></content>
<imageURL></imageURL>
<newsDate>1359729360</newsDate>
<categories>
<category id="20000" name="Category One"/>
</categories>
</news>
<news id="3">
<title>British army stages record-breaking virtual battle</title>
<summary>The British army has...</summary>
<content><![CDATA[<p>The experiment was carried out...]]></content>
<imageURL></imageURL>
<newsDate>1359729420</newsDate>
<categories>
<category id="20001" name="Category Two"/>
</categories>
</news>
</newsItems>
Note that both of these examples are taken from the Jadu documentation pages, they are not from Central Bedfordshire. I won't be posting any of their actual data or keys here.
We can also pass in additional parameters to alter the results of the request, including a couple of global parameters that effect pagination. The news feed has options to set the number of items per page (using the per_page parameter) and the page to fetch (using the page parameter). This means that if we have a number of news items to fetch we just need to grab the first page, look at the number of pages available, and proceed to fetch the rest of the pages in the set.
Here's the updated URL, containing the different parameters.
http://[domain]/api/news/all.xml?api_key=[api_key]&per_page=50&page=1
What I did in the migration was set a default value for the per_page parameter so that I was always fetching 50 items at a time, which means less requests made to the Jadu API.
Whilst the API is useful for getting the basic content of the site, there are a number of limitations. The XML data we get back from Jadu doesn't contain the following information.
- Created/Updated dates - One or two API endpoints (e.g. news and events) do contain the date of the item, but we don't have access to the page creation dates or last updated dates.
- Author information - All endpoints contain no information about who created the page.
- File metadata - Endpoints do contain references to files in the form of URLs, but there is no metadata available for those files. This isn't too bad as we can work that out from the file itself.
- Blocks or other ancillary page elements - There were a few instances where we would import a page of content and find that it was missing something from the page. These blocks needed to be added in by hand after the migration had run.
- Page URL - None of the API endpoints contain any URL information so you must manually construct all the site URLs yourself. I'll dig into this in more detail in a later article, but this really was a pain.
The API also has a number of gotchas that you need to be aware of.
Pagination
The Jadu API pagination system (at least on the Central Bedfordshire site) acts a little strangely. For example, let's say that we requested the news items from the API and the API told us there were 30 pages of items, with 10 items per page. If your script doesn't properly keep track of the last page in the list and happens to ask for page 31 then you will get a seemingly correct response.
In fact, what you will receive if you ask for an out of bounds page is the first page of the results repeated, with a 200 response code!
My first attempt at writing the integration attempted to look for the change in status code and produced an infinite loop with the first page being updated many thousands of times (the Drupal migration mapping meant the pages weren't duplicated). I quickly saw what was going on and corrected it, but it's certainly an odd feature.
I think I'm just spoiled by API systems that return sensible status codes when you ask for data that doesn't exist.
Rate limits
It's not clearly documented, but the Jadu API has a built in rate limit. This means that you can't just run through all of the pages in a large collection of documents or your script will be blocked from the site.
You therefore need to engineer artificial delays into the API callbacks to prevent the API being overloaded. Adding a 1 or 2 second delay before you make any call to the API is a good way of preventing this from happening, but it does slow everything down.
Unpublished Pages
During the migration work I had a couple of instances where the migration was referencing pages that had been unpublished since the original migration was run. This meant that the script would attempt to download a page that was no longer available in the API due to it being removed.
This particularly effected the directory migration, where unpublished directories would return the following data:
<directory id="-1" entries="0" public_submission="0">
<name/>
<content>
<![CDATA[ ]]>
</content>
<categories/>
<fields/>
</directory>
The migration system does not like trying to fit the value "-1" into an unsigned integer value as the ID value in the migration mapping table. What's worse is that page responded with a 200 response code! I needed to add a skip clause to a number of migration scripts to stop processing anything if this value is found. This should ideally be a 404 or 403 so that the value could have been rejected upstream.
The Jadu documentation states that it returns different status codes, but I found that everything outside of a missing API key was just returned a 200 and it was then up to the migration scripts to handle broken data.
With the API out of the way, let's look at an example migration, which will expand into how I built the migration systems.
Migrating News Items From Jadu
The simplest migration that I performed during the Central Bedfordshire migration work was for the News items. This was actually the starting point for the whole process since it is fairly simple (i.e. each page is self contained) and means that I could understand the requirements of the API and build the interface to use it. I will cover some of the more complex Jadu migrations later in this series.
It is possible to migrate XML data from an external resource using the Migrate Plus module using a url migration source plugin. The Migrate Plus module comes with the ability to change how the url source plugin works through a number sub-plugins that can be referenced in the migrate script as configuration items.
These configuration items are as follows:
data_fetcher_plugin - Controls how the data is fetched from the remote resource. This can be either http to make a HTTP request, or file to access a file on the local system.data_parser_plugin - Controls how the data retrieved from the endpoint is extracted into usable migration data. This plugin type also handles things like the URLs that are to be used in the migration (you can enter more than one). There are a number of different plugins available to parse XML and Json data, including an interface around SOAP if you really need it.
These Migrate Plus suite of plugins are really good and do the job well, but I needed to make some changes to how the HTTP requests are made, how the URLs are constructed, and how the XML data is extracted from the request. To this end I created a custom data_fetcher_plugin plugin to handle the HTTP requests and a custom data_parser_plugin plugin to handle the XML data extraction.
Let's look into the custom plugins I created in detail.
data_fetcher_plugin: http_delay
Extends the Migrate Plus module's http plugin, but adds in a configurable delay between requests. This allows us to run a full migration without hitting Jadu's rate limits. The class we extend is called \Drupal\migrate_plus\Plugin\migrate_plus\data_fetcher\Http and all we need to do is override the getResponse() method to add the delay, make the request, and return the content of the result.
As this is the main route for HTTP requests to Jadu I also included a couple of cache mechanisms so that if we request the same page again then we can get the result from the cache. The class uses the normal Drupal database cache for quick access to the XML contents, and another file based cache as a backup cache.
I added the file cache as an aid when developing the migration as it's quite easy (and necessary sometimes) to clear the Drupal database cache, but that means needing to download everything all over again. It's also very good for debugging what happened during the migration as you can just search through the files for the correct XML data. This does mean there's an extra step in fully clearing the migration cache, but that's easily done.
The class is a quite short, but I'll just add the central elements here. You can probably construct the rest using the information I've provided.
<?php
declare(strict_types=1);
namespace Drupal\jadu_migrate\Plugin\migrate_plus\data_fetcher;
use Drupal\Core\Cache\CacheBackendInterface;
use Drupal\Core\File\FileExists;
use Drupal\Core\File\FileSystemInterface;
use Drupal\Core\StringTranslation\TranslatableMarkup;
use Drupal\file\FileRepositoryInterface;
use Drupal\migrate_plus\Attribute\DataFetcher;
use Drupal\migrate_plus\Plugin\migrate_plus\data_fetcher\Http;
use Psr\Http\Message\ResponseInterface;
use GuzzleHttp\Psr7\Response;
use Symfony\Component\DependencyInjection\ContainerInterface;
/**
* Retrieve data over an HTTP connection for migration (with delay).
*
* Example:
*
* @code
* source:
* plugin: url
* data_fetcher_plugin: http
* delay: 3
* @endcode
*/
#[DataFetcher(
id: 'http_delay',
title: new TranslatableMarkup('HTTP')
)]
class HttpDelay extends Http {
/**
* The default request delay.
*
* @var int
*/
protected int $requestDelay = 3;
/**
* {@inheritdoc}
*/
public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition): self {
$instance = new static(
$configuration,
$plugin_id,
$plugin_definition,
$container->get('http_client'),
$container->get('plugin.manager.migrate_plus.authentication'),
);
$instance->cacheBackend = $container->get('cache.default');
$instance->fileSystem = $container->get('file_system');
$instance->fileRepository = $container->get('file.repository');
return $instance;
}
/**
* {@inheritdoc}
*
* Make a request to the given URL, but with a delay. This allows us to make
* lost of requests to Jadu without being rate limited.
*
* Note that this uses two different cache mechanisms:
* - The database cache is used for immediate caches.
* - The file cache is used for a longer term cache.
*
* Since the database cache can be cleared easily (and accidentally) during
* development, the file cache is used for a more stable cache when building
* migration systems.
*/
public function getResponse($url): ResponseInterface {
$cacheId = 'jadu_migrate:http_delay:' . hash('sha1', $url);
if ($cache = $this->cacheBackend->get($cacheId)) {
$response = new Response();
$response->getBody()->write($cache->data);
return $response;
}
$this->fileSystem->prepareDirectory($this->cacheDir, FileSystemInterface::CREATE_DIRECTORY | FileSystemInterface::MODIFY_PERMISSIONS);
$filename = $this->cacheDir . '/' . $cacheId . '.xml';
if (file_exists($filename)) {
$content = file_get_contents($filename);
$response = new Response();
$response->getBody()->write($content);
return $response;
}
// Sleep for a configurable time, and then make the request.
sleep($this->configuration['delay'] ?? $this->requestDelay);
$response = parent::getResponse($url);
// Extract the data of the request.
$data = $response->getBody()->getContents();
// Set our caches.
$this->cacheBackend->set($cacheId, $data);
$this->fileSystem->saveData($data, $filename, FileExists::Replace);
return $response;
}
}
The usage instructions for this plugin are included in the above example, but I'll show them in the migration script later in the article.
data_parser_plugin: jadu_simple_xml_pager
Extends the Migrate Plus module's simple_xml parser to handle the pagination of Jadu requests and to give us the ability to inject the Jadu API key into the migration process. The class extends the \Drupal\migrate_plus\Plugin\migrate_plus\data_parser\SimpleXml class and overrides a few methods to control things the structure and number of URLs.
Rather than print out everything I'll highlight parts of the class that are important to functionality. First, the footprint of the class, showing the needed namespaces and plugin definitions, is as follows.
<?php
declare(strict_types=1);
namespace Drupal\jadu_migrate\Plugin\migrate_plus\data_parser;
use Drupal\Component\Utility\UrlHelper;
use Drupal\Core\StringTranslation\TranslatableMarkup;
use Drupal\Core\Url;
use Drupal\migrate\MigrateException;
use Drupal\migrate_plus\Attribute\DataParser;
use Drupal\migrate_plus\Plugin\migrate_plus\data_parser\SimpleXml;
/**
* Represents an XML pager system that extends the SimpleXml parser.
*
* This makes use of the Jadu page information in the payload to create the
* pagination system.
*/
#[DataParser(
id: 'jadu_simple_xml_pager',
title: new TranslatableMarkup('Simple XML')
)]
class JaduSimpleXmlPager extends SimpleXml {
}
The class here extends the SimpleXml class from the Migrate Plus module, which in turn extends the \Drupal\migrate_plus\DataParserPluginBase class that is responsible for managing the URLs needed in the migration.
All of the Jadu APIs are paginated, so rather than adding a bunch of URLs for every page of content I needed I thought a better way would be to "seed" each migration with the starting point and the number of items per page required. In order to populate the other URLs in the list we needed to override the getNextUrls() method from this base class and populate all of the Jadu URLs for a given endpoint.
The getNextUrls() method makes a request to the first URL in the list (with a page number of 1) and then parses the results of that to find out how many pages are available in the list. It then generates the URLs needed to pull data from every page. This method gets called at different stages of the migration process so it needs to understand that we might not always be starting at page 1.
The addJaduConfigToUrl() is used to inject the API key into the URL, and we'll come to that in a second.
protected function getNextUrls(string $url): array {
$url = $this->addJaduConfigToUrl($url);
// Clear XML error buffer. Other Drupal code that executed during the
// migration may have polluted the error buffer and could create false
// positives in our error check below. We are only concerned with errors
// that occur from attempting to load the XML string into an object here.
libxml_clear_errors();
$xml_data = $this->getDataFetcherPlugin()->getResponseContent($url);
$xml = simplexml_load_string(trim($xml_data));
foreach (libxml_get_errors() as $error) {
$error_string = self::parseLibXmlError($error);
throw new MigrateException($error_string);
}
$this->registerNamespaces($xml);
// Extract the page number and total number of pages from the XML.
$pager = $this->configuration['pager'];
$pageSelection = $xml->xpath($pager['page_selector']);
$pageSelector = $xml->xpath($pager['pages_selector']);
$perPage = $pager['per_page'] ?? 25;
$page = (int) reset($pageSelection);
$pages = (int) reset($pageSelector);
if ($page + 1 > $pages) {
// We have reached the last page, so return an empty array.
return [];
}
$nextUrls = [];
// Build the next URLs for the next page based on the current page and the
// page count.
$path = UrlHelper::parse($url);
$path['query']['page'] = $page + 1;
$path['query']['per_page'] = $perPage;
$nextUrls[] = Url::fromUri($path['path'], [
'query' => $path['query'],
])->toString();
return $nextUrls;
}
The addJaduConfigToUrl() method is used to add the API key to the URL and ensure that the URL contains the correct page information. This deconstructs the URL and then recreates it using the Drupal \Drupal\Component\Utility\UrlHelper and \Drupal\Core\Url classes. This method is added to a couple of places in the class to ensure that these elements are always present in the URL.
public function addJaduConfigToUrl(string $url): string {
$path = UrlHelper::parse($url);
if (isset($this->configuration['jadu_key'])) {
$path['query']['api_key'] = $this->configuration['jadu_key'];
}
if (isset($this->configuration['pager'])) {
// Grab the pager configuration setup.
// We need to ensure that the per_page configuration is always added to
// the URL or we will get odd results with duplicate data.
$pager = $this->configuration['pager'];
if (isset($pager['per_page']) && str_contains($url, 'per_page=') === FALSE) {
$path['query']['per_page'] = $pager['per_page'];
}
}
return Url::fromUri($path['path'], [
'query' => $path['query'],
])->toString();
}
Instead of hard coding the API into the migration scripts (don't do this!) I opted to inject the key into the migration script using Drupal's config update processes.
To do this, we first need to add the following string to the migration script, so that there is a blank value we can target. Well, technically we don't need to do this, but it means that the migration script doesn't break because the variable is missing and instead will complain that the API key is incorrect.
jadu_key: ''
Then, we can add the API key to the Drupal settings.php file.
$config['migrate_plus.migration.news']['source']['jadu_key'] = $jaduApiKey;
Drupal will find this config override in the settings file and inject it into the migration script. This means that the API key can be stored and deployed in a secure fashion, without needing to be hard coded into the migration scripts. There are a number of other ways that API keys can be securely added to a Drupal site, but this plays nicely with the deployment mechanism we were using, which injects variables securely into the site.
The Migration Source
With these plugins created and out API key in place we can create the header of the source migration configuration, which looks a bit like this for the news feed.
source:
plugin: url
data_fetcher_plugin: http_delay
data_parser_plugin: jadu_simple_xml_pager
delay: 3
headers:
User-Agent: 'Jadu'
pager:
page_selector: '@page'
pages_selector: '@pages'
per_page: 50
jadu_key: ''
urls:
- 'https://www.centralbedfordshire.gov.uk/api/news/all.xml'
item_selector: //newsItems/news
I've left the API endpoint here as it doesn't exist anymore, we can since launched the site and Jadu is no longer available on that address.
All we need to then is map the elements in the XML document into the data in the row. This is done using xpath queries based on the root item_selector query, which means that these selectors are relative to the xpath query //newsItems/news and so target individual news items in the XML feed.
fields:
-
name: id
label: 'News ID'
selector: '@id'
-
name: title
label: 'News title'
selector: title
-
name: news_date
label: 'News Date'
selector: newsDate
-
name: image_url
label: 'Image URL'
selector: imageURL
-
name: summary
label: 'Summary'
selector: summary
-
name: content
label: 'Content'
selector: content
-
name: categories
label: 'Categories'
selector: categories
defaults:
# The newsroom can either be created before hand, or included in the migration.
newsroom_id: 1
ids:
id:
type: integer
In LGD, each News page is connected to a parent Newsroom page. This allows news to be segmented easily into different types of news without having to create complex taxonomy views. In the migration above I reference an existing newsroom with an ID that I created in separate migration that used the embedded_data plugin to contain the newsroom information in a single script. I've not included it here as it's a little beyond the scope of this article.
After this, everything about the migration is the same as any other migration. We take the values from the fields and inject them into the data where required.
Whilst the path auto system in Drupal will handle the URLs of the newly created content, we still need to create the redirect from the old Jadu URL to the new Drupal URL. That subject, is quite involved so I will leave that for another article in the series.
Conclusion
Once I had the code functionality working adding the other parts of the migration was just a case of pulling data and working with it. With the classes above I didn't need to worry about API keys or pagination, I just needed to tell the migration handler what the base URL was and it would handle the rest.
When running the migration there's a small delay as it churns through the URLs to fetch all the data, and because of the delay introduced this takes a little bit of time. Once that steps is complete the migration from XML to the Drupal database is quite quick.
I'm currently going through the source code and ensuring that it is fit for release. I want to create a "generic" Jadu to Drupal migration module that will get you 80% of the way there so I'm adding documentation and tests to the code I created and removing some of the hard coded things for Central Bedfordshire. As I only have a single Jadu instance as reference I'm hesitant to remove anything, but there are some customisations to the workflow added for that project in particular that don't need to be there.
If you are looking to migrate from Jadu, or are trying to get to grips with the migration system then please get in touch!
Join me next week where we will look at how I generated the Jadu redirects for the newly created Drupal content.
Add new comment