Help! My search page is getting hammered by an AI bot!
Help! My search page is getting hammered by an AI bot!
Antje Lorch (Ifrik)
AI bots are bringing websites down with never ending search queries.
What can you as a site builder do about that? Which modules can help?
And how do you get a better search experience at the same time.
Prerequisite
Some experience in site building, esp. with setting up faceted search
pages is useful for this session.
Outline
I will show what site builders can do with contrib modules such as Solr
Search, Facets 3 and Antibot to mitigate the effect of AI bots crawling
a Drupal website - and to optimize the search for users at the same
time. We will go through some of the configuration step-by-step to
understand the reasoning behind it, and through some simple debugging steps.
Additionally we will look at measures that can be done on the server.
This is followed by a discussion of what other measures site builders
and developers take, and what of these could be formulated as feature
requests for contrib modules.
Learning Objectives
Site builders will be able to configure existing or new search pages to
reduce the negative effects of AI bots and to improve the UX for
visitors at the same time. They should also be able to communicate to
their hosting providers what kind of measures can be taken on the
server. Module developers should get an insight in how their modules can
help site builders with this specific problem.
Experience level
Intermediate