Re: Deleting duplicates from nnml:mail.misc

[Alex Schroeder]

Alex Schroeder <alex@gnu.org> writes:

> Is my nnml:mail.misc of about 60000 messages full of duplicates?

I think I found a way of doing this within Emacs. I'm sure this could be
made into a nice package, but hope to use it just once. And it takes
long seconds to build up the data structures.

Anyway, here's what I do:

;; get all the headers from the overview file
(setq asc:headers (nnheader-parse-overview-file
                   "/Volumes/Extern/Archives/Mail/mail/misc/.overview"))

;; how many mails in total?
(length asc:headers)

;; create an alist with key Message-ID and value being a list of
;; article numbers sharing this Message-Id (in other words, if there
;; are more than one article number, these are potential duplicates)
;; Example list item: ("<m28uxxr3jk.fsf@gnu.org>" 62335 62329)
(setq asc:duplicates nil)
(dolist (hdr asc:headers)
  (let ((cell (assoc (aref hdr 4) asc:duplicates)))
    (if cell
        (setcdr cell (cons (aref hdr 0) (cdr cell)))
      (setq asc:duplicates (cons (list (aref hdr 4)
                                       (aref hdr 0))
                                 asc:duplicates)))))

;; how many unique Message-IDs?
(length asc:duplicates)

;; look at an example entry
(car asc:duplicates)

;; check how many entries refer to more than one article number
(let ((count 0))
  (dolist (item asc:duplicates)
    (when (> (length item) 2)
      (setq count (1+ count))))
  count)

;; our todo list are the Message-IDs with more than one article number
(setq asc:todo (remove-if (lambda (item) (= (length item) 2)) asc:duplicates))

;; look at the first todo item
(car asc:todo)

;; if you want to check upon those numbers...
;; (gnus-summary-goto-article 62335)
;; (gnus-summary-goto-article 62329)

;; how many todo items?
(length asc:todo)

;; let's get a flat list of article numbers -- using nconc means
;; trashing asc:duplicates!
(setq asc:articles (apply 'nconc
                          (mapcar 'cdr
                                  (remove-if (lambda (item) (= (length item) 2))
                                             asc:duplicates))))

;; how many articles to look at?
(length asc:articles)

;; read them all
(gnus-group-read-group t t "nnml+mail:mail.misc" asc:articles)