Gabriele.vivinetto: Created page with "* Questa operazione consente di convertire una gerarchia di cartelle messaggi da Thunderbird direttamente in formato Maildir, trasportabile direttamente su un server IMAP * Q..."

2016-09-21T14:55:48Z

Created page with "* Questa operazione consente di convertire una gerarchia di cartelle messaggi da Thunderbird direttamente in formato Maildir, trasportabile direttamente su un server IMAP * Q..."

New page

* Questa operazione consente di convertire una gerarchia di cartelle messaggi da Thunderbird direttamente in formato Maildir, trasportabile direttamente su un server IMAP

* Questa operazione è possibile anche con cartelli Imap scaricate in Thunderbird Offline

* Avendo a disposizione il file MBOX di thunderbird, usare [http://perfectmaildir.home-dn.net/perfect_maildir/perfect_maildir.pl questo script]:

<pre>
#!/usr/bin/perl -w

# "Simple but Perfect" mbox to Maildir converter v0.3
# Copyright (C) 2001-2003 Philip Mak <pmak@aaanime.net>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

use strict;
use Date::Parse qw( str2time );

#### Settings
# This will be used to set the file time (needed for courier-imap and some others)
# $datestyle = "date": extract date from the "Date: " header
# $datestyle = "from": extract date from the "From " mbox header
my $datestyle = "from";

# Use maildir++ format (append the message size to the filename)
my $maildirplus = 0;
####

# Get the hostname
my $hostname = `hostname`;
chomp ($hostname);

# check for valid arguments
my ($maildir) = @ARGV;
if (!$maildir) {
print STDERR "Usage: perfect_maildir ~/Maildir < mbox\n";
exit 1;
} elsif (! -d $maildir) {
print STDERR "Cannot open $maildir\n";
exit 1;
}

# check for writable maildir
unless (-w "$maildir/cur") {
print STDERR "Cannot write to $maildir/cur\n";
exit 1;
}
unless (-w "$maildir/new") {
print STDERR "Cannot write to $maildir/new\n";
exit 1;
}

my $num = 0;
my $time = time;
my $date;
my $delivered_time;

repeat:

# read header
my $headers = '';
my $flags = '';
my $subject = '';
while (my $line = <STDIN>) {
# detect end of headers
last if $line eq "\n";

if ($datestyle eq "from") {
# Get date from the "From " line (this should appears here for the first message only)
$date = $1 if $line =~ /^From [^ ^\t]+[ \t]+(.{24})/;
} elsif ($datestyle eq "date") {
# Get date from the "Date: " header
$date = $1 if $line =~ /^Date: (.*)$/;
}
# strip "From" line from header
$headers .= $line unless $line =~ /^From ./;

# detect flags
$flags .= $1 if $line =~ /^Status: ([A-Z]+)/;
$flags .= $1 if $line =~ /^X-Status: ([A-Z]+)/;
$subject = $1 if $line =~ /^Subject: (.*)$/;
}

$num++;

if ($datestyle =~ /(from|date)/) {
$delivered_time = str2time("$date");
} else {
$delivered_time = $time;
}

# open output file
my $file;
if ($flags =~ /O/) {
$file = sprintf( "%s%05d%s", "$maildir/cur/$delivered_time.", $num, ".$hostname" );
my $extra = '';
$extra .= 'F' if $flags =~ /F/; # flagged
$extra .= 'R' if $flags =~ /A/; # replied
$extra .= 'S' if (($flags =~ /R/) || ($flags =~ /O/)); # seen
$extra .= 'T' if $flags =~ /D/; # trashed
$file .= ":2,$extra" if $extra;
} else {
$file = sprintf( "%s%05d%s", "$maildir/new/$delivered_time.", $num, ".$hostname" );
}

# filter out the "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA" message or the message doesn't exists
if (($num == 1 and $subject eq "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA") || (!$headers)) {
$file = '/dev/null';
$num--;
}

open(FILE, ">$file");
print FILE "$headers\n";
while (my $line = <STDIN>) {
if ($datestyle eq "from") {
# Get date from the "From " line (this should appears here for the first message only)
$date = $1 if $line =~ /^From [^ ^\t]+[ \t]+(.{24})/;
}
# End of current message
last if ($line =~ /^From ./);

# unescape "From"
$line =~ s/^>From (.)/From $1/;

print FILE $line;
}
close(FILE);

utime( $time, $delivered_time, $file ) if ($datestyle =~ /(from|date)/);

if ($maildirplus) {
my $size = -s $file;
my $mdplusfile = $file;
$mdplusfile =~ s/\.$hostname/.$hostname,S=$size/;
rename $file,$mdplusfile;
}

goto repeat unless eof(STDIN);

my $elapsed = time - $time;
print "Inserted $num messages into maildir $maildir in $elapsed seconds\n";
</pre>

* Creare una maildir vuota con
maildirmake ./cartella

* Convertire il file MBOX con
./perfect_maildir.pl ./cartella < FILEMBOX

* Nei riferimenti lo script per convertire un'intera gerarchia di cartelle

=Riferimenti=
*[https://blogs.gnome.org/muelli/2012/11/converting-mailman-archives-mboxes-to-maildir/ Converting Mailman archives (mboxes) to maildir – muellis blog]

= Converting Mailman archives (mboxes) to maildir =

I wanted to search discussions on mailing lists and view conversations. I didn’t want to use some webinterface because that wouldn’t allow me to search quickly and offline. So making my mail client aware of these emails seemed to be the way to go. Fortunately, the GNOME mailinglists are mbox archived. So you download the entire traffic in a [[standardised mbox]].

But how to properly get this into your email clients then? I think Thunderbird can import mbox natively. But I wanted to access it from other clients, too, so I needed to make my server aware of these emails. Of course, I configured my mailserver to use [[maildir]], so some conversion was needed.

I will present my experiences dealing with this problem. If you want to do similar things, or even only want to import the mbox directly, this post might be for you.

=== The archives ===

First, we need to get all the archives. As I had to deal with a couple of mailinglists and more than a couple of month, I couldn’t be arsed to click every single mbox file manually.

The following script scrapes the mailman page. It makes use of the interesting [[Splinter library]], basically a wrapper around selenium and other browsers for Python.

<pre>
#!/usr/bin/env python

import getpass
from subprocess import Popen, list2cmdline
import sys

import splinter

def fill_password(b, username=None, password=None):
if not username:
username = getpass.getpass('username: ')
if not password:
password = getpass.getpass('password: ')

b.fill('username', username)
b.fill('password', password)
b.find_by_name('submit').click()

def main(url, username=None):
b = splinter.Browser()

try:
#url = 'https://mail.gnome.org/mailman/private/board-list/'
b.visit(url)

if 'Password' in b.html:
fill_password(b, username=username)

links = [l['href'] for l in b.find_link_by_partial_text('Text')]

cookie = b.driver.get_cookies()[0]
cookie_name = cookie['name']
cookie_value = cookie['value']
cookie_str = "Cookie: {name}={value}".format(name=cookie_name, value=cookie_value)
wget_cookie_arg = '--header={0}'.format(cookie_str)
#print wget_cookie_arg

b.quit()

for link in links:
#print link
cmd = ['wget', wget_cookie_arg, link]
print list2cmdline(cmd)
# pipe that to "parallel -j 8"

except:
b.quit()

if __name__ == '__main__':
site = sys.argv[1]
user = sys.argv[2]

if site.startswith('http'):
url=site
else:
url = 'https://mail.gnome.org/mailman/private/{0}'.format(site)

main(username=user, url=url)

</pre>

You can [[download the thing, too]].

I use splinter because handling cookies is not fun as well as parsing the web page. So I just use whatever is most convenient for me, I wanted to get things done, after all. The script will print a line for each link it found, nicely prefixed with <code>wget</code> and its necessary arguments for the authorization cookie. You can pipe that to <code>sh</code> but if you want to download many month, you want to do it in parallel. And fortunately, [[there is an app for that]]!

=== Conversion to maildir ===

After having received the mboxes, it turned out to be a good idea nonetheless to convert to maildir; if only to extract properly formatted mails only and remove duplicates.

I came around [[mb2md-3.20.pl from 2004]] quite soon, but it is broken. It cannot parse the mboxes I have properly. It will create broken mails with header lingering around as it seems to be unable to detect the beginning of new mails reliably. It took me a good while to find the problem though. So again, be advised, [[do not use mb2md 3.20]].

As I use mutt myself I found [[this blog article]] promising. It uses mutt to create a mbox out of a maildir. I wanted it the other way round, so after a few trial and errors, I figured that the following would do what I wanted:

<pre>
mutt -f mymbox -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s/tmp/mymuttmaildir"'

</pre>

where “<code>mymbox</code>” is your source file and “<code>/tmp/mymuttmaildir</code>” the target directory.

This is a bit lame right? We want to have parameters, because we want to do some batch processing on many archive mboxes.

The problem is, though, that the parameters are very deep inside the quotes. So just doing something like

<pre>
mutt -f $source -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s$target"'

</pre>

wouldn’t work, because the $target would be interpreted as a raw string due to the single quotes. And I couldn’t find a way to make it work so I decided to make it work with the language that I like the most: Python. So an hour or so later I came up with the following which works (kinda):

<pre>
import os
import subprocess
source = os.environ['source']
destination = os.environ['destination']

conf = 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s{0}"'.format(destination)

cmd = ['mutt', '-f', source, '-e', conf]
subprocess.call(cmd)

</pre>

But well, I shouldn’t become productive just yet by doing real work. Mutt apparently expects a terminal. It would just prompt me with “No recipients were specified.”.

So alright, this unfortunately wasn’t what I wanted. I you don’t need batch processing though, you might very well go with mutt doing your mbox to maildir conversion (or vice versa).

Damnit, another two hours or more wasted on that. I was at the point of just doing the conversion myself. Shouldn’t be too hard after all, right? While researching I found that Python’s stdlib has some email related functions *yay*. [[Some dude on the web]] wrote something close to what I needed. I beefed it up a very little bit and landed with the following:

<pre>
#!/usr/bin/env python

# http://www.hackvalue.nl/en/article/109/migrating%20from%20mbox%20to%20maildir

import datetime
import email
import email.Errors
import mailbox
import os
import sys
import time

def msgfactory(fp):
try:
return email.message_from_file(fp)
except email.Errors.MessageParseError:
# Don't return None since that will
# stop the mailbox iterator
return ''
dirname = sys.argv[1]
inbox = sys.argv[2]
fp = open(inbox, 'rb')
mbox = mailbox.UnixMailbox(fp, msgfactory)

try:
storedir = os.mkdir(dirname, 0750)
os.mkdir(dirname + "/new", 0750)
os.mkdir(dirname + "/cur", 0750)
except:
pass

count = 0
for mail in mbox:
count+=1
#hammertime = time.time() # mail.get('Date', time.time())
hammertime = datetime.datetime(*email.utils.parsedate(mail.get('Date',''))[:7]).strftime('%s')
hostname = 'mb2mdpy'
filename = dirname + "/cur/%s%d.%s:2,S" % (hammertime, count, hostname)
mail_file = open(filename, 'w+')
mail_file.write(mail.as_string())

print "Processed {0} mails".format(count)

</pre>

And it seemed to work well! It recovered many more emails than the Perl script (hehe) but the generated maildir wouldn’t work with my IMAP server. I was confused. The mutt maildirs worked like charm and I couldn’t see any difference to mine.

I <code>scp</code>ed the file onto my <code>.maildir/</code> on my server, which takes quite a while because <code>scp</code> isn’t all too quick when it comes to many small files. Anyway, it wouldn’t necessarily work for some reason which is way beyond me. Eventually I <code>straced</code> the IMAP server and figured that it was desperately looking for a <code>tmp/</code> folder. Funnily enough, it didn’t need that for other maildirs to work. Anyway: Lesson learnt: If your <code>[[dovecot]]</code> doesn’t play well with your maildir and you have no clue how to make it log more verbosely, check whether you need a <code>tmp/</code> folder.

But I didn’t know that so I investigated a bit more and I found [[another PERL script]] which converted the emails fine, too. For some reason it put my mails in “<code>.new/</code>” and not in “<code>.cur/</code>“, which the other tools did so far. Also, it would leave the messages as unread which I don’t like.

Fortunately, one (more or less) only needs to rename the files in a maildir to end in <code>S</code> for “seen”. While this sounds like a simple

<pre>
for f in maildir/cur/*; do mv ${f} ${f}:2,S

</pre>

it’s not so easy anymore when you have to move the directory as well. But that’s easily being worked around by shuffling the directories around.

Another, more annoying problem with that is “[[Argument list too long]]” when you are dealing with a lot of files. So a solution must involve “<code>find</code>” and might look something like this: <code>find ${CUR} -type f -print0 | xargs -i -0 mv '{}' '{}':2,S</code>

=== Duplicates ===

There was, however, a very annoying issue left: Duplicates. I haven’t investigated where the duplicates came from but it didn’t matter to me as I didn’t want duplicates even if the downloaded mbox archive contained them. And in my case, I’m quite confident that the mboxes are messed up. So I wanted to get rid of duplicates anyway and decided to use a hash function on the file content to determine whether two file are the same or not. I used <code>sha1sum</code> like this:

<pre>
$ find maildir/.board-list/ -type f -print0 | xargs -0 sha1sum | head
c6967e7572319f3d37fb035d5a4a16d56f680c59 maildir/.board-list/cur/1342797208.000031.mbox:2,
2ea005ec0e7676093e2f488c9f8e5388582ee7fb maildir/.board-list/cur/1342797281.000242.mbox:2,
a4dc289a8e3ebdc6717d8b1aeb88959cb2959ece maildir/.board-list/cur/1342797215.000265.mbox:2,
39bf0ebd3fd8f5658af2857f3c11b727e54e790a maildir/.board-list/cur/1342797210.000296.mbox:2,
eea1965032cf95e47eba37561f66de97b9f99592 maildir/.board-list/cur/1342797281.000114.mbox:2,

</pre>

and if there were two files with the same hash, I would delete one of them. Probably like so:

<pre>
#!/usr/bin/env python
import os
import sys

hashes = []
for line in sys.stdin.readlines():
hash, fname = line.split()
if hash in hashes:
os.unlink(fname)
else:
hashes.append(hash)

</pre>

But it turns out that the following snippet works, too:

<pre>
find /tmp/maildir/ -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

</pre>

So it’ll check the files for the same contents via a . In order to make <code>uniq</code> detect equal lines, we need to give it sorted input. Hence the <code>sort</code>. We cannot, however, check the whole lines for equality as the filename will show up in the line and it will of course be different. So we only compare the size of the hex representation of the hash, in this case 40 bytes. If we found such a duplicate hash, we cut off the hash, take the filename, which is the remainder of the line, and delete the file.

Phew. What a trip so far. Let’s put it all together:

=== The final thing ===

<pre>

LIST=board-list

umask 077

DESTBASE=/tmp/perfectmdir

LISTBASE=${DESTBASE}/.${LIST}

CUR=${LISTBASE}/cur
NEW=${LISTBASE}/new
TMP=${LISTBASE}/tmp

mkdir -p ${CUR}
mkdir -p ${NEW}
mkdir -p ${TMP}

for f in /tmp/${LIST}/*; do /tmp/perfect_maildir.pl ${LISTBASE} < ${f} ; done
mv ${CUR} ${CUR}.tmp
mv ${NEW} ${CUR}
mv ${CUR}.tmp ${NEW}
find ${CUR} -type f -print0 | xargs -i -0 mv '{}' '{}':2,S
find ${CUR} -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

</pre>

And that’s handling email in 2012…

=Riferimenti=

Convertire Cartelle Locali di Thunderbird in Maildir - Revision history

Gabriele.vivinetto: Created page with "* Questa operazione consente di convertire una gerarchia di cartelle messaggi da Thunderbird direttamente in formato Maildir, trasportabile direttamente su un server IMAP * Q..."