github.com/bcampbell/scrapeomat@v0.0.0-20220820232205-23e64141c89e/cmd/sitemapwalker/README.md (about)

     1  # sitemapwalker
     2  
     3  Tool to scan through sitemap files looking for article links.
     4  
     5  Look at a sites robots.txt to find sitemap files.
     6  (or `http[s]://<sitename>/sitemap.xml` is a common one...)
     7  
     8