Edit File: MailboxFormat.mbox.txt
Mbox Mailbox Format =================== Contents 1. Mbox Mailbox Format 1. Locking 1. Dotlock 2. Deadlocks 2. Directory Structure 3. Dovecot's Metadata 4. Dovecot's Speed Optimizations 5. From Escaping 6. Mbox Variants 7. References Usually UNIX systems are configured by default to deliver mails to '/var/mail/username' or '/var/spool/mail/username' mboxes. In IMAP world these files are called INBOX mailboxes. IMAP protocol supports multiple mailboxes however, so there needs to be a place for them as well. Typically they're stored in '~/mail/' or '~/Mail/' directories. The mbox file contains all the messages of a single mailbox. Because of this, the mbox format is typically thought of as a slow format. However with Dovecot's indexing this isn't true. Only expunging messages from the beginning of a large mbox file is slow with Dovecot, most other operations should be fast. Also because all the mails are in a single file, searching is much faster than with maildir. Modifications to mbox may require moving data around within the file, so interruptions (eg. power failures) can cause the mbox to break more or less badly. Although Dovecot tries to minimize the damage by moving the data in a way that data should never get lost (only duplicated), mboxes still aren't recommended to be used for important data. Locking ------- Locking is a mess with mboxes. There are multiple different ways to lock a mbox, and software often uses incompatible locking. See <MboxLocking.txt> for how to check what locking methods some commonly used programs use. There are at least four different ways to lock a mbox: * *dotlock*: 'mailboxname.lock' file created by almost all software when writing to mboxes. This grants the writer an exclusive lock over the mbox, so it's usually not used while reading the mbox so that other processes can also read it at the same time. So while using a dotlock typically prevents actual mailbox corruption, it doesn't protect against read errors if mailbox is modified while a process is reading. * *flock*: 'flock()' system call is quite commonly used for both read and write locking. The read lock allows multiple processes to obtain a read lock for the mbox, so it works well for reading as well. The one downside to it is that it doesn't work if mailboxes are stored in NFS. * *fcntl*: Very similar to *flock*, also commonly used by software. In some systems this 'fcntl()' system call is compatible with 'flock()', but in other systems it's not, so you shouldn't rely on it.*fcntl* works with NFS if you're using lockd daemon in both NFS server and client. * *lockf*: POSIX 'lockf()' locking. Because it allows creating only exclusive locks, it's somewhat useless so Dovecot doesn't support it. With Linux 'lockf()' is internally compatible with 'fcntl()' locks, but again you shouldn't rely on this. Dotlock ------- Another problem with dotlocks is that if the mailboxes exist in '/var/mail/', the user may not have write access to the directory, so the dotlock file can't be created. There are a couple of ways to work around this: * Give a mail group write access to the directory and then make sure that all software requiring access to the directory runs with the group's privileges. This may mean making the binary itself setgid-mail, or using a separate dotlock helper program which is setgid-mail. With Dovecot this can be done by setting 'mail_privileged_group = mail'. * Set sticky bit to the directory ('chmod +t /var/mail'). This makes it somewhat safe to use, because users can't delete each others mailboxes, but they can still create new files (the dotlock files). The downside to this is that users can create whatever files they wish in there, such as a mbox for newly created user who hadn't yet received mail. Deadlocks --------- If multiple lock methods are used, which is usually the case since dotlocks aren't typically used for read locking, the order in which the locking is done is important. Consider if two programs were running at the same time, both use dotlock and fcntl locking but in different order: * Program A: fcntl locks the mbox * Program B at the same time: dotlocks the mbox * Program A continues: tries to dotlock the mbox, but since it's already dotlocked by B, it starts waiting * Program B continues: tries to fcntl lock the mbox, but since it's already fcntl locked by A, it starts waiting Now both of them are waiting for each others locks. Finally after a couple of minutes they time out and fail the operation. Directory Structure ------------------- By default, when listing mailboxes, Dovecot simply assumes that all files it sees are mboxes and all directories mean that they contain sub-mailboxes. There are two special cases however which aren't listed: * '.subscriptions' file contains IMAP's mailbox subscriptions. * '.imap/' directory contains Dovecot's index files. Because it's not possible to have a file which is also a directory, it's not normally possible to create a mailbox and child mailboxes under it. However if you really want to be able to have mailboxes containing both messages and child mailboxes under mbox, then Dovecot can be configured to do this, subject to certain provisos; see <MboxChildFolders.txt>. Dovecot's Metadata ------------------ Dovecot uses C-Client (ie. UW-IMAP, Pine) compatible headers in mbox messages to store metadata. These headers are: * X-IMAPbase: Contains UIDVALIDITY, last used UID and list of used keywords * X-IMAP: Same as X-IMAPbase but also specifies that the message is a "pseudo message" * X-UID: Message's allocated UID * Status: R (\Seen) and O (non-\Recent) flags * X-Status: A (\Answered), F (\Flagged), T (\Draft) and D (\Deleted) flags * X-Keywords: Message's keywords * Content-Length: Length of the message body in bytes Whenever any of these headers exist, Dovecot treats them as its own private metadata. It does sanity checks for them, so the headers may also be modified or removed completely. None of these headers are sent to IMAP/POP3 clients when they read the mail. *The MTA, MDA or LDA should strip all these headers _case-insensitively_ before writing the mail to the mbox.* Only the first message contains the X-IMAP or X-IMAPbase header. The difference is that when all the messages are deleted from mbox file, a "pseudo message" is written to the mbox which contains X-IMAP header. This is the "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA" message which you hate seeing when using non-C-client and non-Dovecot software. This is however important to prevent abuse, otherwise the first mail which is received could contain faked X-IMAPbase header which could cause trouble. If message contains X-Keywords header, it contains a space-separated list of keywords for the mail. Since the same header can come from the mail's sender, only the keywords are listed in X-IMAP header are used. The UID for a new message is calculated from "last used UID" in X-IMAP header + 1. This is done always, so fake X-UID headers don't really matter. This is also why the pseudo message is important. Otherwise the UIDs could easily grow over 2^31 which some clients start treating as negative numbers, which then cause all kinds of problems. Also when 2^32 is exceeded, Dovecot will also start having some problems. Content-Length is used as long as another valid mail starts after that many bytes. Because the byte count must be exact, it's quite unlikely that abusing it can cause messages to be skipped (or rather appended to the previous message's body). Status and X-Status headers are trusted completely, so it's pretty good idea to filter them in LDA if possible. Dovecot's Speed Optimizations ----------------------------- Updating messages' flags and keywords can be a slow operation since you may have to insert a new header (Status, X-Status, X-Keywords) or at least insert data in the header's value. Some mbox MUAs do this simply by rewriting all of the mbox after the inserted data. If the mbox is large, this can be very slow. Dovecot optimizes this by always leaving some space characters after some of its internal headers. It can use this space to move only minimal amount of data necessary to get the necessary data inserted. Also if data is removed, it just grows these spaces areas. 'mbox_lazy_writes' setting works by adding and/or updating Dovecot's metadata headers only after closing the mailbox or when messages are expunged from the mailbox. C-Client works the same way. The upside of this is that it reduces writes because multiple flag updates to same message can be grouped, and sometimes the writes don't have to be done at all if the whole message is expunged. The downside is that other processes don't notice the changes immediately (but other Dovecot processes do notice because the changes are in index files). 'mbox_dirty_syncs' setting tries to avoid re-reading the mbox every time something changes. Whenever the mbox changes (ie. timestamp or size), it first checks if the mailbox's size changed. If it didn't, it most likely meant that only message flags were changed so it does a full mbox read to find it. If the mailbox shrunk, it means that mails were expunged and again Dovecot does a full sync. Usually however the only thing besides Dovecot that modifies the mbox is the LDA which appends new mails to the mbox. So if the mbox size was grown, Dovecot first checks if the last known message is still where it was last time. If it is, Dovecot reads only the newly added messages and goes into a "dirty mode". As long as Dovecot is in dirty mode, it can't be certain that mails are where it expects them to be, so whenever accessing some mail, it first verifies that it really is the correct mail by finding its X-UID header. If the X-UID header is different, it fallbacks to a full sync to find the mail's correct position. The dirty mode goes away after a full sync. If 'mbox_lazy_writes' was enabled and the mail didn't yet have X-UID header, Dovecot uses MD5 sum of a couple of headers to compare the mails. 'mbox_very_dirty_syncs' does the same as 'mbox_dirty_syncs', but the dirty state is kept also when opening the mailbox. Normally opening the mailbox does a full sync if it had been changed outside Dovecot. From Escaping ------------- In mboxes a new mail always begins with a "From " line, commonly referred to as From_-line. To avoid confusion, lines beginning with "From " in message bodies are usually prefixed with '>' character while the message is being written to in mbox. Dovecot doesn't currently do this escaping however. Instead it prevents this confusion by adding Content-Length headers so it knows later where the next message begins. Dovecot doesn't either remove the '>' characters before sending the data to clients. Both of these will probably be implemented later. Mbox Variants ------------- There are a few minor variants of this format: *mboxo* is the name of original mbox format originated with Unix System V. Messages are stored in a single file, with each message beginning with a line containing "From SENDER DATE". If "From " (case-sensitive, with the space) occurs at the beginning of a line anywhere in the email, it is escaped with a greater-than sign (to ">From "). Lines already quoted as such, for example ">From " or ">>>From " are *not* quoted again, which leads to irrecoverable corruption of the message content. *mboxrd* was named for Raul Dhesi in June 1995, though several people came up with the same idea around the same time. An issue with the mboxo format was that if the text ">From " appeared in the body of an email (such as from a reply quote), it was not possible to distinguish this from the mailbox format's quoted ">From ". mboxrd fixes this by always quoting already quoted "From " lines (e.g. ">From ", ">>From ", ">>>From ", etc.) as well, so readers can just remove the first ">" character. This format is used by qmail and getmail (>=4.35.0). *mboxcl* format was originated with Unix System V Release 4 mail tools. It adds a Content-Length field which indicates the number of bytes in the message. This is used to determine message boundaries. It still quotes "From " as the original mboxo format does (and *not* as mboxrd does it). *mboxcl2* is like mboxcl but does away with the "From " quoting. *MMDF* (Multi-channel Memorandum Distribution Facility mailbox format) was originated with the MMDF daemon. The format surrounds each message with lines containing four control-A's. This eliminates the need to escape From: lines. Dovecot currently uses mboxcl2 format internally, but it's planned to move to combination of mboxrd and mboxcl. *How a message is read stored in mbox extension ?* * An email client reader scans throughout mbox file looking for From_ lines. * Any From_ line marks the beginning of a message. * Once the reader finds a message, it extracts a (possibly corrupted) envelope sender and delivery date out of the From_ line. * It then reads until the next From_ line or scans till the end of file, whenever From_ comes first. * It removes the last blank line and deletes the quoting of >From_ lines and >>From_ lines and so on. References ---------- * Wikipedia [http://en.wikipedia.org/wiki/Mbox] * Qmail mbox [http://www.qmail.org/man/man5/mbox.html] * Mbox family [http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html] * CommuniGatePro mbox [http://www.communigate.com/CommuniGatePro/Mailboxes.html#mbox] * MBOX File Viewer [http://www.freeviewer.org/mbox/] (This file was created from the wiki on 2018-02-28 04:42)