CRAWLER

Quickly Index Your Site Content and Start Searching

Index your website pages, PDFs, Office documents and more with SearchStax Site Search Crawler.

CRAWLER

Quickly Index Your Site Content and Start Searching

Index your website pages, PDFs, Office documents and more with SearchStax Site Search Crawler.

Easily ingest data

The Site Search Crawler can automatically discover all web pages, content and documents on your website by crawling through your site’s links and internal navigation as well as XML sitemaps generated by your CMS.

The Site Search Crawler can automatically discover all web pages, content and documents on your website by crawling through your site’s links and internal navigation as well as XML sitemaps generated by your CMS.

Crawl across your website

Site Search Crawler can index any public-facing content on your website - regardless of what content management system you’re using.

Popular CMSs and DXPs

Beyond webpages

Build search experiences fast with Crawler

Start now and schedule for later

You can start crawling your site immediately so you can get up and running quickly. Then schedule daily crawls to keep your site search indexes up-to-date.

Data extraction

Site Search Crawler extracts page content, titles and metadata automatically. You can also:

Add your own metadata fields (like OG tags, author info and publish date)
Use Xpath to extract data from specific sections of your site
Use CSS selectors to pull data from specific HTML tags
Define exclusions to avoid crawling certain portions of your site
Respect robots.txt to ensure only the right content goes into your index

Start now and schedule for later

You can start crawling your site immediately so you can get up and running quickly. Then schedule daily crawls to keep your site search indexes up-to-date.

Data extraction

Site Search Crawler extracts page content, titles  and metadata automatically. You can also:

Add your own metadata fields (like OG tags, author info and publish date)
Use Xpath to extract data from specific sections of your site
Use CSS selectors to pull data from specific HTML tags
Define exclusions to avoid crawling certain portions of your site
Respect robots.txt to ensure only the right content goes into your index

Gear up with marketer-friendly tech

Site Search Crawler enables marketing teams to create engaging search experiences across their entire digital ecosystem.

Click to crawl

Simply enter a starter URL and Crawler will start populating your search index.

Daily updates

As your content continually changes, Crawler will keep your search results fresh with a daily crawl.

Clear visibility

Easily monitor your crawls to ensure pages are indexed so your site health and search quality stay high.

There are more Site Search features to explore

Connectors

Connect to any web CMS, DXP or custom HTML site with our Crawler, pre-built connectors and APIs.

Explore Connectors

Search Experience

Give your visitors instant access to exactly the information they’re looking for.

Search & Discovery

Boost and Promote

Boost your top performing content on your search results page with easy-to-use tools for marketers

Explore Optimizations

Deliver better website search experiences and gain insights with SearchStax. Let's talk.