diff --git a/seed-lists/README.md b/seed-lists/README.md index 8f75fbd..9d6f010 100644 --- a/seed-lists/README.md +++ b/seed-lists/README.md @@ -79,7 +79,7 @@ Seeds supplied by Kelly L. Smith, Government Information Librarian and Librarian * Hermann-Wu-nps-20241209.txt - NPS seeds submitted by Hermann-Wu ### Seeds sourced from Web resources -The End of Term Web Archive team compiled a list of sources on the Web from which to source seeds: +The End of Term Web Archive team and other contributors compiled a list of sources on the Web from which to source seeds: * US_Digital_Registry.csv - CSV file generated on 9/11/2024 by Praneeth Rikka at UNT from the data at the [Touchpoints U.S. Digital Registry](https://touchpoints.app.cloud.gov/registry). * Military-Departments-A-Z-List.csv - CSV file generated on 9/11/2024 by Lauren Ko at UNT from the data of the [U.S. Department of Defense's A-Z List](https://www.defense.gov/Resources/Military-Departments/A-Z-List/). @@ -91,3 +91,4 @@ The End of Term Web Archive team compiled a list of sources on the Web from whic * usagov.csv - Seeds scraped from https://www.usa.gov/agency-index/ by Jake Abrams, Founder, CivicsUS, LLC. * CDC html URLs from sitemap data - 20241201.csv - file of about 46,000 .html URLs created by parsing the CDC's sitemap file at https://www.cdc.gov/wcms-auto-sitemap-index.xml, which then pointed to other sitemaps, which pointed to .html files. * CDC found PDFs 20241209 cleaned single file.csv - .gov PDF links obtained from webpages found on the US CDC website. It contains 46,873 links, with the format: the source HTML file containing the PDF link; the time in UTC in which the accessibility of the PDF file was confirmed; and a URL pointing to the PDF file itself. PDF links are deduped when multiple pages point to the same PDF, and link fragments are removed. All the PDF files have had their accessibility and content type verified with a HTTP HEAD request on Dec. 09 2024. +* sitemaps.txt - List of federal website sitemap URLs discovered programmatically by Bentley Hensel via robots.txt files and common sitemap URL paths. URLs scraped from these sitemaps are organized by files named by hostname in the sitemap-url-seeds directory. This directory contains 2,749 files listing more than 56 million URLs as well as `_report_6_FINAL.txt` that gives statistics about the file sizes and URL counts in each of the files. URLs from the sitemap-url-seeds files will NOT be loaded into the eth2024_bulk Nomination Tool instance due to size.