Drupal

I've talked a lot about the Batch API in Drupal recently, and I've mentioned that it is built upon the Queue API, but I haven't gone any deeper than that. I wrote about the Queues API in Drupal 7, but thought I would bring my understanding up to date.

A queue is a data construct that uses a "first in, first out" (or FIFO) flow where items are processed in the order that they were added to the queue with the first item added being processed first. This system has a lot of different uses, but is most important when it comes to asynchronous data processing. Drupal and many modules make use of the queue system to process information behind the scenes.

The difference between a queue and a batch is that the batch is for time sensitive things where the user is expecting something to happen. A queue, on the other hand, is more for data processing that needs to happen behind the scenes or without any user triggering the process.

DrupalCamp Scotland returned after a small hiatus of 5 years on the 25th October 2024 and saw nearly 50 people attend the university of Edinburgh Paterson's Land building for a day of talks and sessions. I had the honor of being invited to speak at the conference, which was the first physical speaking session I've had since 2019.

I arrived early to the conference on a sunny Friday morning after driving up the night before. After a cup of coffee and a lovely chat with a few people we started the conference.

This is the sixth article in a series of articles about the Batch API in Drupal. The Batch API is a system in Drupal that allows data to be processed in small chunks in order to prevent timeout errors or memory problems.

So far in this series we have looked at creating a batch process using a form, followed by creating a batch class so that batches can be run through Drush, using the finished state to control batch processing, processing CSV files through a batch process and finally adding to running batch processes. These articles give a good grounding of how to use the Drupal Batch API.

In this article we will look at how the Batch API is used within Drupal. The Batch API in Drupal is either used to perform a task, which I will call "direct", or to pass on the batch operations to a hook, which I will call "indirect". These aren't official terms you understand, I'm just using them here to separate how Drupal uses the Batch API.

This is the fifth article in a series of articles about the Batch API in Drupal. The Batch API is a system in Drupal that allows data to be processed in small chunks in order to prevent timeout errors or memory problems.

So far in this series we have looked at creating a batch process using a form, followed by creating a batch class so that batches can be run through Drush, using the finished state to control batch processing and then processing CSV files through a batch process. All of these articles give a good grounding of how to use the Drupal Batch API.

In this article we will take a closer look at how the batch system processes items by creating a batch run inside an already running batch process. This will show how batch systems run and what happens when you try to add additional operations to a running batch.

This is the fourth article in a series of articles about the Batch API in Drupal. The Batch API is a system in Drupal that allows data to be processed in small chunks in order to prevent timeout errors or memory problems.

So far in this series we have looked at creating a batch process using a form, followed by creating a batch class so that batches can be run through Drush and then using the finished state to control batch processing. All of these articles go together to form a basis of batch processing in Drupal.

In this article we will look at bringing these concepts together to perform a task that is quite common on websites, processing Comma Separated Value or CSV files, which we will do using the Drupal Batch API.

This is the third article in a series of articles about the Batch API in Drupal. The Batch API is a system in Drupal that allows data to be processed in small chunks in order to prevent timeout errors or memory problems.

So far in this series we have looked at creating a batch process using a form and then creating a batch class so that batches can be run through Drush. Both of these examples used the Batch API to run a set number of items through a set number of process function callbacks. When setting up the batch run we created a list of items that we wanted to process and then split this list up into chunks, each chunk being sent to a batch process callback.

There is another way to set up the Batch API that will run the same number of operations without defining how many times we want to run them first. This is possible by using the "finished" setting in the batch context.

This is the second part of a series of articles looking at the Batch API in Drupal. The Batch API is a system in Drupal that allows data to be processed in small chunks in order to prevent timeout errors or memory problems.

In the previous article we looked at how to setup the batch process using a form, with the batch methods being contained within the structure of the form class. When the form was submitted the batch process ran through 1,000 items and then printed out a result at the end.

Whilst there is nothing wrong with running the Batch API with everything in a form class, it is normally better to abstract the batch processing code into a separate class.

The Batch API is a powerful feature in Drupal that allows complex or time consuming tasks to be split into smaller parts.

For example, let's say you wanted to run a function that would go through every page on you Drupal site and perform an action. This might be removing specific authors or removing links in text. You might create a small loop that just loads all pages and performs the action on those pages.

This is normally fine on sites that have a small number of pages, like under 100 pages. But what happens when the site has 10,000 pages, or a million? Your little loop will soon hit the limits of PHP execution times or memory limits and cause the page to be terminated. How do you know how far your loop progressed through the data? What happens if you tried to restart the loop?

Drupal's migration system allows the use of a number of different plugins to perform source, processing, and destination functions. 

Process plugins are responsible for copying and sometimes manipulating data into the destination. There are a number of different process plugins that allow you to get data in different ways and and apply it to your destination fields.

Both the core Migrate module and the excellent Migrate Plus module contain a number of different process plugins that you can use to process your data in different ways.

April 23rd, 2024 saw the first LocalGov Drupal Camp, held at the Birmingham Council buildings in Birmingham city center.

It's been ages since I attended an in person Drupal Camp in the UK, so when I saw that the LocalGov Drupal people were organising one just down the road for me I jumped at the chance to grab a ticket.

LocalGov Drupal is a distribution that combines Drupal, some configuration, some contributed modules, and some glue code with the aim of making it easier for councils to generate sites.