How to crawl a website

 

 

To start crawling a website, go to any of the dropdown menus highlighted in the screenshot within the app and select ‘Crawl Website’.

 

 

This will take you to the following screen, where you can start the crawl by entering your website url, either with the http/https protocol, or without.

 

 

If you are in your personal account, we will only crawl up to 50 pages. Please upgrade to a paid plan to crawl up to 5,000 pages at a time.

 

If you have reached the sitemaps limit in your account, you will need to do one of the following:

  • Delete / archive existing sitemaps to bring yourself under the sitemaps limit
  • Upgrade to an organization account if you only have a personal account
  • Upgrade your subscription plan if you are in an organization account

 

View more ways to generate a visual sitemap with our visual sitemap generators.