Re: [Savannah-hackers-public] Please disallow www-commits in robots.txt

[Thérèse Godefroy]

Le 10/05/2023 à 19:14, Alfred M. Szmidt a écrit :
>     I was searching for an article with DuckDuckGo, and guess what
>     appeared on top of the results...
>     https://lists.gnu.org/archive/html/www-commits/2023-05/msg00082.html
>     and
>     https://lists.gnu.org/archive/html/www-commits/2023-05/msg00062.html
>     !!
>
> What is the problem?
>
>     I understand it wouldn't be practical to make this list private,
>     but it could at least be off-limits for crawlers.
>
> How is www-commits different from anything else?

Because it registers every single commit to www, including to working
directories that webmasters have disallowed, for instance
*/po/, /server/staging/, */workshop/, /prep/gnumaint/, etc.

Please see https://www.gnu.org/robots.txt.