# robots.txt for http://www.wvgs.wvnet.edu/ # allow all user agents User-agent: SemrushBot Disallow: / User-agent: Bingbot Disallow: / User-agent: dotbot Disallow: / User-agent: Python-urllib Disallow: / User-agent: AhrefsBot Disallow: / User-agent: * # but disallow the following directories to all agents # as directories contain out-of-date and test files Disallow: /www/DevonianShales_Draft/ Disallow: /www/www_idle/ Disallow: /www/declines/ Disallow: /www/resources/ Disallow: /www/test/ Disallow: /www/MRCI/ Disallow: /www/MUDvnnSh/MUDvnnSh_AllReportParts.htm Disallow: /www/MUDvnnSh/OlderAndDraftMaterials/ Disallow: /gis/og/MUDvnnSh_FlowUnitsTest/index.html Disallow: /gis/og/MUDvnnSh_FlowUnitsTest/ Disallow: /www/gis/og/MUDvnnSh_FlowUnitsTest/index.html Disallow: /www/gis/og/MUDvnnSh_FlowUnitsTest/ Disallow: /www/museum/musedisplays.htm Disallow: /www/museum/museum_new.htm Disallow: /www/museum/WVGESMuseumSpecimenCatalog.pdf Disallow: /www/museum/museunknownfossil1.htm Disallow: /www/museum/Dimetrodon_ColoringPage.pdf Disallow: /www/museum/GroundSloth_ColoringPage.pdf Disallow: /www/museum/Smilodon_ColoringPage.pdf Disallow: /www/mids/ Disallow: /oginfo/pipeline2/ # references include: http://www.w3.org/robots.txt and # http://www.w3.org/TR/html4/appendix/notes.html#recs # file created by SEP on June 21, 2001; modified by SEP on September 30, 2002; October 29, 2004; February 3, 2006; July 28, 2017; June 28, 2018 (removed Disallow for /www/MUDvnnSh/); March 18, 2021; June 17, 2021 (added MRCI disallow); July 6, 2021 (added SemrushBot code)