Re: [Savannah-hackers-public] Please disallow www-commits in robots.txt

[Ian Kelling]

Thérèse Godefroy <godef.th@free.fr> writes:

> Hello,
>
> I was searching for an article with DuckDuckGo, and guess what appeared
> on top of the results...
> https://lists.gnu.org/archive/html/www-commits/2023-05/msg00082.html
> and
> https://lists.gnu.org/archive/html/www-commits/2023-05/msg00062.html !!
>
> I understand it wouldn't be practical to make this list private, but it
> could at least be off-limits for crawlers.
>
> Thanks!
>
> Thérèse

robots.txt isn't about just not making things public, it is perfectly
valid to use for SEO and helping google turn up the best results for
your site and avoiding non-canonical duplicate pages. I've seen
www-commits list archive pages come up in searches before and they were
not what I wanted. In that light, this request sounds like a good idea
to me. I will implement it. We can change it back if webmasters agree
otherwise.

-- 
Ian Kelling | Senior Systems Administrator, Free Software Foundation
GPG Key: B125 F60B 7B28 7FF6 A2B7  DF8F 170A F0E2 9542 95DF
https://fsf.org | https://gnu.org