In the last article in this series I looked at creating a link directory on a Drupal site. In that article I looked at how I set up the links and took screenshots of the sites using a headless Chromium browser as the links were added.
The issue I had was that when I used headless Chromium to take screenshots of the sites the success rate was not very high. In these days of AI attacks, site captcha checks, and cookie popups it turned out to be quite difficult to take a clean screenshot of a site without being blocked either by a CDN or a cookie popup. In fact, most of the time the screenshot would be just a CDN error page.
I therefore looked for a different mechanism. Since I wanted to take a screenshot of a website it made sense to me to use a browser to do this, and because I am already using a browser why not get the browser I'm using to take the screenshot. After a bit of research I realised that creating browser extensions to do this was actually pretty simple. Plus once the screenshot has been taken I can post this to the Drupal site using a REST resource.
The only niggle was that I needed the screenshot to be at a set dimension, since all the link images on the site also have that dimension. That turned out to be slightly more challenging.
In this article we will look at setting up a rest resource to generate (or update) links, and then creating a Chrome extension to take a screenshot of a site at a set resolution.
First, let's look at creating the REST resource in Drupal.
Creating A REST Resource
This needs to accept the data from the Chrome extension and generate a Link content entity using that data.
As we are accepting data this REST resource needs to accept only POST requests, which adds a little bit of complexity to the setup of the system. In order to accept an incoming POST request Drupal must be able to authenticate you as a user, which can be done in a number of different ways.
The core of Drupal only accepts cookie authentication, which is sort of complicated for a REST resource as it needs you to authenticate using your username and password, collect the cookie from the page response, and then send that cookie every time you make a request (assuming that the cookie is still authenticates you).
It's much simpler to use a contributed module to do this, and I oped to use the Key Auth module. Using this module you can set each user on your site to have a key, and that key can be passed in with a request in a number of ways to authenticate that user onto the site. The keys can be easily rotated and have permissions associated with them so it's easy to lock down the keys to certain users, or stop users from using keys.
The authentication mechanism is set up secondarily to the REST plugin, so let's put that together first.
namespace Drupal\link_directory\Plugin\rest\resource;
use Drupal\Core\StringTranslation\TranslatableMarkup;
use Drupal\file\FileInterface;
use Drupal\link_directory\Service\LinkManagerServiceInterface;
use Drupal\rest\Attribute\RestResource;
use Drupal\rest\ModifiedResourceResponse;
use Drupal\rest\Plugin\ResourceBase;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\HttpFoundation\Request;
/**
* REST service for receiving link data.
*/
#[RestResource(
id: "link_directory_receive_link",
label: new TranslatableMarkup("Receive Link"),
uri_paths: [
"create" => "/api/links/[REDACTED]",
]
)]
class ReceiveLink extends ResourceBase {
protected LinkManagerServiceInterface $linkManagerService;
public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
$instance = new static(
$configuration,
$plugin_id,
$plugin_definition,
$container->getParameter('serializer.formats'),
$container->get('logger.factory')->get('rest')
);
$instance->linkManagerService = $container->get('link_directory.link_manager');
return $instance;
}
public function post(Request $request) {
$payload = $request->getContent();
try {
$data = json_decode($payload, TRUE, flags: JSON_THROW_ON_ERROR);
}
catch (\Exception $e) {
return new ModifiedResourceResponse([], 422);
}
// Extract information from the data.
$link = $data['link'];
$title = $data['title'];
$fileData = $data['imagedata'];
// Create a Link node object.
$node = $this->linkManagerService->createLinkObject($link, $title);
// Turn off image update.
$node->set('field_link_image_update', TRUE);
// Remove head header from the base 64 string.
$fileData = substr($fileData, strpos($fileData, ',') + 1);
$file = $this->linkManagerService->createFileFromBase64Data($fileData, hash('sha1', $link));
if ($file instanceof FileInterface) {
$node->set('field_link_image', [
'target_id' => $file->id(),
'alt' => $title,
]);
}
// Save the node.
$node->save();
// Built response payload.
$return = [
'message' => 'Link created!',
'url' => $node->toUrl()->setAbsolute()->toString(),
];
// Do not cache the response.
return new ModifiedResourceResponse($return, 201);
}
}
Note that I have removed quite a bit of code here that checks the incoming request for validity and responds with errors when something is wrong. I have removed this to shorten the example, but if you use this code then please add some sensible error checking.
With the Drupal REST plugin in place we can open the REST administration interface (powered by the REST UI module) to enable and configure it.
Here, we set the authentication provider to be key_auth, which is how we will authenticate with the system. Each user can generate a key and use that to push data to the receive link endpoint, which authenticates them with the request. That means that when a link is created in directory we can tell who created the link.
The mechanism to make a request with this system is slightly different to a GET resource. In order to authenticate with the site we first need to make a request to the endpoint /session/token, sending the key that we want to authenticate with. This will return a session token, which we then need to append to subsequent requests in order for them to be authenticated correctly. So whilst the GET request is a single request, a POST request requires two requests, one to authenticate, and another to make the request.
SecKit Bypass
One slight niggle before we can move on is bypassing the security provided by the SecKit module on the site. This module is used to add various security hardening options to Drupal, mitigating a number of different vulnerabilities. The main protection mechanism we are concerned about here is the Cross-site Request Forgery (CSRF) protection, which will look for the Origin header and reject the request if it isn't recognised as being from a set number of websites.
When we make a request from a Chrome extension it will come from a chrome:// address, with a unique ID that is different for every browser that the extension is installed into. Clearly, we can't keep on top of updating the allowed list every time we install the extension, so we need another mechanism to bypass this security layer, but still have it active.
The way around this is to decorate the \Drupal\seckit\EventSubscriber\SecKitEventSubscriber event subscriber from the SecKit module so that we can look for a request from the extension and allow it to proceed. To decorate a service we just need to create a service in our modules services.yml file and add the "decorates" directive, pointing to the service we want to decorate.
link_directory.decorate_seckit_event:
class: Drupal\link_directory\EventSubscriber\LinkDirectorySecKitEventSubscriber
decorates: seckit.subscriber
arguments: ['@logger.channel.seckit', '@config.factory', '@extension.list.module']
tags:
- { name: event_subscriber }
With this in place, Drupal will load our LinkDirectorySecKitEventSubscriber class instead of the original SecKitEventSubscriber class, meaning that we can do what we need to allow the request in.
The mechanism I came up with was to add a "Hashbangcode-Extension" header to the incoming request from the extension, which is then used to check if the incoming request is from the extension. Since the extension will only ask for two endpoints on the site we can also check that one of these two paths is being requested before allowing the request in.
class LinkDirectorySecKitEventSubscriber extends SecKitEventSubscriber {
public function seckitOrigin($event) {
$hashbangcodeExtension = $this->request->headers->get('Hashbangcode-Extension');
if ($hashbangcodeExtension === 'link-it') {
$pathInfo = $this->request->getPathInfo();
if ($pathInfo == '/session/token' || $pathInfo == '/api/links/[REDACTED]') {
// Allow the request if it had the correct header and is pointed
// at the correct paths.
$this->request->headers->set('Origin', 'null');
return;
}
}
parent::seckitOrigin($event);
}
}
Most importantly, if we don't recognise the request then we must call the parent::seckitOrigin() method to allow the SecKit module to run the normal checks on the request. It's important when poking holes in a security layer to always allow the default behaviour to continue when we don't need the bypass.
Now that we have the REST resource in place, and allowed the request to be accepted by the site, we can create a Chrome extension.
Creating A Chrome Extension
This could be an article all on it's own, so I'll keep it somewhat brief. In basic terms, a Chrome extension is essentially a small website, created using normal HTML, JavaScript and CSS files. If you are familiar with writing JavaScript then Chrome extensions will be second nature really.
I called the extension "Link it!", and created a directory to store the source files in.
The extension has a manifest.json file that gives some instructions to the browser about what the extension is called, what actions to perform, and what permissions the extension has. This is the manifest.json file for the Link It! extension.
{
"name": "Link Directory Link It",
"version": "1.0.0",
"description": "Takes screenshots for the link directory.",
"manifest_version": 3,
"author": "Philip Norton",
"permissions": ["debugger", "tabs", "activeTab", "storage"],
"action":{
"default_popup": "index.html",
"default_title": "Link it!"
},
"background": {
"service_worker": "background.js"
},
"host_permissions": [
"https://www.hashbangcode.com/*",
"https://hashbangcode.ddev.site/*"
],
"options_page": "options.html"
}
There are a couple of things to pay attention to here:
- permissions : This is a list of the permissions associated with the extension. In this case I need:
- debugger : The screenshot mechanism is run using the Chrome debugger, so we need to allow permission to this system.
- tabs, activeTab : As we want to take a screenshot of the current, active, tab we need to give permission to access this information.
- storage : I want to store the access key in the extension, so it needs access to the storage permission to do this.
- action : This is used to create the extension icon in Chrome, and tells the browser what file to load initially.
- background : The screenshot is taken using a service_worker, and this option tells Chrome what script file to load as part of this service.
- host_permissions : In order to make a request to an endpoint we need to tell Chrome what endpoints we are allowing. Here I'm allowing the main site, and the ddev instance for debugging and development purposes.
- options_page : I'm not showing this in the examples below, but the user key for authentication is saved using an options page, which is just another HTML page that contains a small script.
The file index.html is just a plan HTML file, here is the contents of the body element of that file.
<button id="snap">Grab Link Data</button>
<div id="snap-status"></div>
<img id="result-image" style="width: 100%; margin-top: 10px;" />
<input type="hidden" id="image-data" />
<input type="text" id="title" name="title" class="hidden" />
<input type="text" id="linkurl" name="linkurl" class="hidden" />
<button id="send" class="hidden">Link It!</button>
<p id="send-status" class="hidden">Success!</p>
<button id="go-to-options">options</button>
<script src="script.js"></script>
Note that I have stripped out the rest of the HTML file here since that's pretty standard. It has the usual html, head, and body elements, including some styles add some simple layout to the extension.
When we click the button in Chrome we are shown this dialog, with some of the options hidden from view until they are needed.
The included script.js file contains all of the event listeners for the extension to react to user input. I won't show them all here, but the main one is listening to the click event on the "Grab Link Data" button. When the user clicks this button the script will pick up the current tab, use the service worker to take a screenshot of the page (with set dimensions), collect any information about the page we can, and present the results to the user.
document.getElementById('snap').addEventListener('click', async () => {
const sendStatus = document.getElementById('send-status');
sendStatus.classList.add('hidden');
sendStatus.innerHTML = '';
const [tab] = await chrome.tabs.query({ active: true, currentWindow: true });
const status = document.getElementById('snap-status');
status.innerText = "Capturing...";
// Generate screenshot of the page using the service worker.
chrome.runtime.sendMessage({
action: "take_screenshot",
tabId: tab.id,
width: 1280,
height: 960
}, (response) => {
if (response?.success) {
status.innerHTML = "<br>";
document.getElementById('result-image').src = response.dataUrl;
document.getElementById('image-data').value = response.dataUrl;
} else {
status.innerText = "Error: " + (response?.error || "Unknown");
}
});
const titlefield = document.getElementById('title');
titlefield.classList.remove('hidden');
titlefield.value = tab.title;
const linkurl = document.getElementById('linkurl');
linkurl.classList.remove('hidden');
linkurl.value = tab.url;
const send = document.getElementById('send');
send.classList.remove('hidden');
});
Triggering the sendCommand() function (part of the Chrome debugger system) will run the following code in the background.js file. This essentially runs the asynchronous function captureExactSize() and then respond to the output of that function.
chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
if (message.action === "take_screenshot") {
// We must call the async function and handle the response
captureExactSize(message.tabId, message.width, message.height)
.then((dataUrl) => sendResponse({ success: true, dataUrl }))
.catch((error) => sendResponse({ success: false, error: error.message }));
return true; // Keeps the message channel open for the async response.
}
});
The captureExactSize() function will attempt to resize the browser window to a particular size, ensuring that any scale factors are set correctly when doing so. I say "attempt" as there were actually a few different parameters available to resize the browser window, and although they are all documented, not all of them have the intended effect.
Once the screen has been resized we then wait for 100ms to ensure that the site has settled down before taking a screenshot of the site. We don't need to wait longer than 100ms since we have already loaded the page, this is just the act of resizing the currently selected tab.
async function captureExactSize(tabId, width, height) {
const target = { tabId };
// 1. Attach the debugger
await chrome.debugger.attach(target, "1.3");
try {
// 2. Force the page to render at the specific dimensions
// deviceScaleFactor: 1 ensures a 1:1 pixel ratio (not retina/high-dpi)
await chrome.debugger.sendCommand(target, "Emulation.setDeviceMetricsOverride", {
width: width,
height: height,
deviceScaleFactor: 1,
mobile: false,
screenWidth: width, // Explicitly set the "monitor" size
screenHeight: height,
viewSize: { width: width, height: height }
});
// Set the page scaling factor.
await chrome.debugger.sendCommand(target, "Emulation.setPageScaleFactor", {
pageScaleFactor: 1
});
// 3. Give the browser a "tick" to reflow the layout
await new Promise(resolve => setTimeout(resolve, 100));
// 4. Capture the screenshot
const { data } = await chrome.debugger.sendCommand(target, "Page.captureScreenshot", {
format: "png",
clip: {
x: 0,
y: 0,
width: width,
height: height,
scale: 1,
},
fromSurface: true,
captureBeyondViewport: true,
});
// 5. Reset dimensions
await chrome.debugger.sendCommand(target, "Emulation.clearDeviceMetricsOverride");
return `data:image/png;base64,${data}`;
} finally {
// Always detach the debugger, even if an error occurs
await chrome.debugger.detach(target);
}
}
After we have the screenshot data we return this as a base64 encoded string, which allows us to both render the resulting screenshot easily in the extension window, and send the information to the REST resource without having to convert the data to a different format.
If you want more information on what code is run here then the following pages from the Chrome docs might be of interest to you
- Page reference : https://chromedevtools.github.io/devtools-protocol/tot/Page/
- Emulation reference : https://chromedevtools.github.io/devtools-protocol/tot/Emulation/
The resulting screenshot and data are presented to the user, giving them the option to review the result, and perhaps tweak the information discovered. They can then click the "Link it!" button to send the data to the REST endpoint of the site to create a link.
Rather than post lost more code about how this is done, I've extracted the part of the code responsible for sending the request to the site.
const tokenUrl = 'https://www.hashbangcode.com/session/token';
const linkCreateUrl = 'https://www.hashbangcode.com/api/links/[REDACTED]';
// Fetch the token.
const tokenResponse = await fetch(tokenUrl, {
method: 'GET',
headers: {
'Content-Type': 'text/plain',
'Hashbangcode-Extension': 'link-it',
}
});
// Extract the token.
const tokenResult = await tokenResponse.text();
// Send link data with our key and csrf token.
const response = await fetch(linkCreateUrl, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Hashbangcode-Extension': 'link-it',
'X-CSRF-Token': tokenResult,
'api-key': apiKey,
},
body: JSON.stringify(data)
});
// Get the response.
const result = await response.json();
As you can see, we first make a request to the /session/token endpoint to get a token, which is then sent with the main create link request to authenticate the request. Remember that we needed to bypass the SecKit module in our request. As a result we set a header called "Hashbangcode-Extension" and set this to "link-it", which is understood by the site and allows the request to proceed.
With all of that in place we now have a extension we can use. Rather than upload what is essentially a private extension to the Google Web Store I have instead opted to install it locally. You can do this by loading the extensions page and selecting the options "Load unpacked" from the top of the page. This will open a dialog window that you just point to the location of the extension on your disk.
If the extension was loaded correctly then you will see something like this in the list of extensions. You can now treat this just like any other browser extension on the browser.
Let's have a look at how the extension can be used.
Using The Link It! Extension
What I'm going to do here is take a picture of the main php.net site using the Linkt It! extension and send this to the link directory.
With the extension installed we see an icon in our extensions list. We can pin the icon the same way as normal Chrome extensions to make it always visible.
Clicking the Link It! icon will popup this little window, which is the default display of the index.html file (with some of the fields hidden as they don't have data in them).
Clicking on "Grab Link Data" will resize the window and take a picture of the site in that new viewport size. The data is then presented to the user, along with options to be able to change the title and URL before submitting them to the link directory.
If we click the "Linkt It!" button it will trigger the requests to the site and either generate or update the link on the site. The result is a JSON response that contains a URL back to the link we just created, which we can click on to see the link in the directory (to add a description and tags).
If you want to look at the full source code of the Link It! extension then you can find it on CodeBerg. You won't be able to push anything to the site, but it shows how to create an extension, create an options page for that extension, and take a screenshot of the currently active tab.
Conclusion
I've been using this extension for a few weeks now, and it's been working very well. It was quite easy to put together all of this code, it just took a few evenings reading the Chrome internal documentation to resize windows and take screenshots.
What seems to be hit and miss is actually getting Chrome to take a decent screenshot of the page. Sometimes when resizing the window Chrome will corrupt the resulting screenshot by wrapping the page around itself. This typically happens when the window you are taking the screenshot with is smaller than the size of the screenshot (1280x960), but it doesn't always happen.
Here is an example of the problem.
I've tried quite a few mechanisms to try and solve this, but not really had much success, although it doesn't happen every time. It appears to be something that Chrome does due to the difference between the screenshot size and the window size. What Chrome is doing here is attempting to fill in any blank space in the screenshot, but it is doing this by just showing the site again in that blank space, causing a tiling effect.
This is why previewing the image was essential as it clearly shows if things went wrong, and gives you the opportunity to have another go. As it doesn't happen every time giving it another big try tends to get things working.
You might be wondering if the extension works in other browsers like Vivaldi, or Firefox? The answer is yes, quite surprisingly it works in just the same way. Even with non-Chrome based browsers it works very well.
Other than that, there are a couple of minor things to watch out for. For example, when you take a screenshot of the site whilst logged in you often get your profile picture included in the image. Not a big problem, but I've gotten used to opening an incognito window and taking the screenshot from there if I happen to be logged in at the time.
This has been a fun project to work through, with a number of interesting challenges to overcome. Creating the extension to take screenshots of sites took a surprising short amount of time to do, even though attempting to fix the issues did take a little while.
As for the link directory itself, there will be a part 3 in the not too distant future where I attempt to test all of the links on the site to detect broken links. This is an essential part of maintaining a link directory as it keeps the directory up to date without stumbling across broken links.
Add new comment