Queues API

Creating queues using the core queue classes in Drupal is fairly straightforward. You just need a mechanism of adding data to the queue and a worker to process that data.

As the data you add to the queue is serialised you can add pretty much any data you want to the queue, so the only limitation is rebuilding the data once you pull it out of the queue.

There are some situations where the core Drupal queue system needs to be altered in some way. You might want to separate the data into different tables, or have a different logic for creating or storing the queue items, or even integrate with a third party queue system for manage the queues.

When writing data to the queue database system Drupal will convert the information to a string using the PHP serialize() function. When the information is pulled out of the database the queue system will unserialize() the data to convert it back into the original information.

When you first start using the queue system you will probably use an array or the PHP stdClass object to store the information on your queue. Whilst this works, the information they contain is pretty free form and makes testing or working with the data a little cumbersome.

A much better way of storing data is by creating an object of a known type and using that as the storage medium for the queue.

I've talked a lot about the Batch API in Drupal recently, and I've mentioned that it is built upon the Queue API, but I haven't gone any deeper than that. I wrote about the Queues API in Drupal 7, but thought I would bring my understanding up to date.

A queue is a data construct that uses a "first in, first out" (or FIFO) flow where items are processed in the order that they were added to the queue with the first item added being processed first. This system has a lot of different uses, but is most important when it comes to asynchronous data processing. Drupal and many modules make use of the queue system to process information behind the scenes.

The difference between a queue and a batch is that the batch is for time sensitive things where the user is expecting something to happen. A queue, on the other hand, is more for data processing that needs to happen behind the scenes or without any user triggering the process.

The Drupal 7 Queues API is a few feature of Drupal and provides a first in first out data structure that is used internally by Drupal itself and can be completely customised.

The Queues API isn't just a part of the codebase of Drupal, it is used internally as part of several different processes. The Batch API, which allows lots of things to be done at once, is built upon the Queues API and provides some customised queue classes. I won't be going into the batch API in this post, but I might cover it in later posts.

The new cron system in Drupal 7 uses the Queues API heavily. It works by allowing other modules to create queue items during their normal hook_cron() calls, which are then run afterwards. Each module that wants to include an item in the system cron queue can do so by implementing a hook called hook_cron_queue_info(), which tells cron what function to callback when the queue items are retrieved.