The SearchStax Site Search solution’s crawler add-on can be reconfigured at will to improve its results. Pages that can no longer be reached by the modified crawler are automatically deleted from the index.
This is transparent to the user except in one situation. The crawler has a built-in safeguard against deleting records when the deletions exceed 20% of the index. Such a massive change might indicate an accidental crawler misconfiguration that could take a production system offline.
If you want to go ahead and delete the unreachable pages from the index, you have these options:
- The easiest way to clear a crawled index is to delete the crawler and create a new one.
- SearchStax Support can adjust the deletion threshold for your crawler. The unwanted pages will be deleted on the next crawl.
- You can manually wipe the index by using an API call like this one:
curl -X POST --location "<Site Search update URL>" \ -H "Content-Type: application/json" \ -d '{ "delete": { "query": "*:*" } }' \ --basic --user <Site Search admin>:<admin password>
Bear in mind that it might take five minutes before the search engine commits the deletes. Until then, you won’t see any change in the search results.
Questions?
Do not hesitate to contact the SearchStax Support Desk.