Note: This post is over a year old and so the information contained here might be out of date. If you do spot something please leave a comment and I will endeavour to correct.
A robots.txt file tells search engines spiders what pages or files they should or shouldn't request from your site. It is more of a way of preventing your site from being overloaded by requests rather than a secure mechanism to prevent access. It really shouldn't be used as a way of preventing access to your site, and the chances are that some search engine spiders will access the site anyway. If you do need to prevent access then think about using noindex directives within the page itself, or even password protecting the page.
I was writing some code on a Drupal site that detects if a page is being published and I realised that this state isn't as clear cut as you might expect. Drupal stores the published state of a page as the 'status', with 0 being unpublished and 1 being published. With revisions being turned on by default since Drupal 8 it is possible to see past states of the page when saving the page. The issue is that there is nothing in the current state of the page stating that this is the first time it is being published.
A recent Wired article about the Parler data hack talked about how a hacker group was able to steal publicly available information from the Parler website using an Insecure Direct Object Reference (IDOR) or enumeration attack. This type of attack involves a hacker looking at the structure of the site and attempting to guess the next available resource by looking at the URL. Apparently, terabytes of Parler's data was downloaded by simply enumerating through the ID's of their publicly available posts.
For the last few years I have been tagging articles as I write them on this site. This tagging has largely been to tie posts together in an aggregated list of other posts tagged with that term. I recently wondered if I could use those tags to show related content below each article. I have struggled with this feature on client websites in the past and it either boils down to a manually curated list or some sort of complex content analysis and Solr search.
Pushing data from Drupal into Solr is a really convenient way of creating a robust and extensible search solution. The Search API module has a number of different fields available that can be used to integrate with all sorts of fields, but what isn't included is computed fields or other data.
Thankfully, adding a custom field into the Search API system doesn't need a lot of code. A single class, with an optional hook, is all that's needed to het everything working.
Drupal's login forms are protected by a protection mechanism that prevents brute force attacks. This means that if an attacker attempts to repeatedly guess a user's password to gain entry to their account they will be blocked before being successful. This system has been a part of Drupal for many years and so is battle tested.
