Cleaning Out Duplicate IMAP messages

There are a number of ways to end up with a lot of duplicate messages in an IMAP folder, and while IMAP tends to handle very large stores gracefully, it is possible to hose things. On my 32 bit server and with Mulberry as a client things get weird after about 15,000 messages in a single folder.

Google does some odd things and at one point a periodic check of my gMail account resulted in about 70,000 messages in a single folder, which definitely caused some chaos.

I thought that was pretty impressive, but my girlfriend just managed to get 144,000 messages in a single folder. Woo Hoo!!! High Score.

Anyway, things like the dedup plugins for Thunderbird can just make things worse at that point as they seem to fail gracelessly on very large message counts.

I found that Rick Sander’s perl scripts are the best way out of this difficult situation. delIMAPdups.pl solves the problem without running out of memory or munging files. I haven’t had any lost data and just tested by clearing about 1400 dups out of a directory of 15,000 messages (my 2009 store to date).
/.delIMAPdups.pl -S example.com:993/user/pass -m INBOX.2009 -p
-m is the mailbox to expunge
-p is purge
-S means use SSL