The Book of IMAP: Building a Mail Server with Courier and Cyrus, by Peer Heinlein and Peer Hartleben, is a quality resource for any serious mail administrator. The approach taken is direct, but at the same time it’s very expansive, setting this book apart from most others I have read. It’s packed full of rich examples which are used to solidify the topic being covered. At several places the authors reach out to explain when the subject is addressing ambiguous or otherwise undocumented information which is to great advantage to the reader and worthy of recognition.
The technology covered mainly pertains to the Courier and Cyrus daemons, hence the title, but the book also provides many general tips along the way to squeeze out every ounce of performance possible including file system particulars and kernel tweaks. Various file systems are discussed and the advantages provided by them are weighed out. Common bottlenecks are identified which can streamline the performance yielding more efficiency. Each server and the constituent parts are broken down and thoroughly described as well as the respective tools distributed. This makes for a great comparative analysis of each server with specific emphasis given to the their individual distinctive qualities.
The range covered is broad and sweeping including basic topics like web-mail clients and basic protocol structure, up to more advanced subjects such as load balancing, proxy configurations, and clustered configurations. The protocols POP3 and IMAP are articulated in the appendix and also used frequently throughout the text affording a transparent depiction of the underlying mechanics involved. The book discusses at great length additional setups which extend the native functionality by use of NFS and relational databases including MySQL and PostgreSQL especially as related to authentication. Security is consistently addressed and each authentication method is explained adequately. Utilizing encrypted tunnels is covered sufficiently all the way down to the generation of the SSL certificate itself.
The focal point is building a server from the ground up so each configuration file directive is explored exhaustively. The structure of each server is comprehensively covered as well as associated modules that extend its capabilities. Invaluable information is dispensed when this book explains situations and issues that can arise when migrating an existing server in a production setup to a new server. General administrative tips are expounded upon including quotas, virtual domains, and mail filtering techniques by use of Sieve. All angles are addressed equally well.
One drawback is the frequent examples given use a SuSE Linux architecture which the authors seem to be most familiar with. While there is some coverage for Debian and Red Hat, more would have been better, along with other UNIX flavors. Some topics such as subscriptions and a few other implementation details tended to be a bit drawn out and could have been made to be more concise in my opinion. There were a few syntax and typographical flaws, but very minor ones.
One aspect I enjoyed most about this book is how often real world scenarios are put forth. The overall content is great actionable information mostly gained by first-hand experience. I will definitely be seeking out more titles from No Starch Press to add to my bookshelf in the near future.
(Reviewer Ryan Masters is a system administrator and web developer.)
I use IMAP and Gmail and I think IMAP has to evolve…It shouldn’t limit itself to just a simple remote folder and message manager.
1. It’s slow. It wasn’t designed for our gigabyte email accounts. When I have to sync my computer with my server, it synces one mail after the other. I have hundreds of thousands of email, it takes a huge amount of time. It should create a single zipped file of the headers and send it to my computer, it would be much faster than unzipped individual files. With numerous three-way TCP/IP connections, syncing with IMAP is just too slow.
2. Remote search is weak. Even with Dovecot. Try performing an advanced search and not only is it slow but often you don’t get what you’re searching because your IMAP server doesn’t index attachments, sometimes it only searches headers and discards bodies.
3. No support for tags. While some MUA and webmails now migrated from folders to tags or labels, IMAP still uses folders, so for instance in Gmail I have “Projects” and “To do” tags, if a message has both labels, in IMAP it is physically duplicated.
4. No support for calendar/address book/RSS. I know IMAP is not a calendar server application but nowadays, many MUAs have calendar, RSS and address book integrated and it’s essential to be able to sync a calendar, feeds and an address book without the need to set up an LDAP server.
5. Be in par with new needs. IMAP should offer all one expects from a mail protocol these days. I think when you use Gmail and IMAP, you shouldn’t lose functionality, power, features and speed. It’s time to dramatically upgrade the IMAP protocol, in the meantime, some people are migrating to web applications that have fast, synced, database-indexed emailing, calendar, RSS and address book.
Edited 2008-08-26 06:35 UTC
I am not sure whether you know what you are talking about.
Here’s is one for you to try on YOUR machine: zip up a few hundred thousand mini-KB files and report back to us how long it took you. Then delete a random 50000 files and ad 70000 new one into that zip-file; then report back and tell us how long that took. Then do your math and let us know what will happen if the server has to do that to any number of people typically accessing their mail at a time… the server would explode
No comment.
Back to topic. I think the problem you’re exposing is that IMAP servers work with *files*. Obviously, email servers would cripple if they had to zip individual small files. And they already cripple just retrieving individual messages. How many times do we get timeout error messages from Squirrelmail when we have too many messages in an IMAP folder? The solution is retrieving relevant messages from a *database*, storing them into a temporary file (single file), zipping this file and sending it to the MUA. This would solve our problem and servers would be happy.
There already are imap servers that use databases,
one of them is archiveopteryx http://www.archiveopteryx.org/
It uses Postgresql for backend storage, this means that you can make use of the extremly quick Postgresql full text search engine to search your mail.
The database is only used for backend storage. DBMail works this way for instance. But the client-server communication doesn’t benefit from it.
i haven’t done a start from scratch sync for a while, so i dunno if this is already the case.
However instead of the zip option perhaps being able to download multiple emails at the same time (parrallel download)
I do agree that IMAP and also Microsoft’s implementation a la Exchange needs improving with the points you have listed. Email is a standard, many companies are using it more than telephone conversations. As commented before many people are reaching multi GB sizes, many people live in their email system, in some cases i wonder if the email system is becoming an operating system in itself.
Google Mail could use IMAP-tags:
http://www.ii.com/internet/messaging/imap/isps/#keywords
You’re measuring the usage of a single IMAP client with a single IMAP server, and complaining that everything related to IMAP sucks? Yeah, that makes sense.
GMail uses IMAP over SSL, so you have an encryption/decryption process on top of the transport., slowing things down, depending on your CPU.
You don’t specify which IMAP client you are using. Accessing GMail from KMail is nice and snappy, as KMail does multi-threaded message downloads/folder syncs.
Perhaps you should investigate other IMAP clients?
I have 400 MB in my GMail account. Connecting to it from a brand new PC takes less than 5 sec to bring up the inbox (only new messages) and less than a minute to bring up the archive (just under 10,000 messages).
IOW, get a better IMAP client.
Get a better IMAP server. Remote search against Cyrus servers is very fast, even against my 1.5 GB work account. I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.
Zimbra’s Cyrus implementation in their OSS version is also quite speedy. And their Network edition is even faster.
Can’t comment on this, as I don’t use tagging in any of my IMAP clients.
Get a better IMAP server. For instance, Kolab uses Cyrus IMAP to store e-mail, contacts, and calendars in IMAP folders.
It does. Since when is calendars considered part of a “mail” protocol? Or even a “message” protocol”?
Edited 2008-08-26 23:20 UTC
This is a huge amount of time. Searching messages on a web-based forum takes one second. I shouldn’t expect an IMAP server to be slower. In both cases you’re searching for a phrase in a list of messages. The IMAP server architecture makes the whole process slow. Five minutes is way too much to wait for.
Thunderbird also has an extension that uses folders to store this information but I disagree with this practice. Folders should be used only for messages, and I would use one single large folder in the first place, each message would be labeled and indexed.
Since when people started using email, calendar, address book and collaboration tools at the same time in the same application (this is why Zimbra, Lotus Notes and MS Outlook are so popular in companies). Needs are evolving and IMAP needs to extend its functionalities to meet these needs, especially for companies.
Well, to each their own. I don’t consider 5 minutes to search through several hundred thousand messages, and 1.5 GB of mixed text/binary data to be a long time. Maybe it’s because I run my searches in the background while doing other things. Especially when you consider our IMAP server is also the webmail interface (squirrelmail), and is only a 1.6 GHz Opteron system with 4 GB of RAM, server just under 1500 users, about 500-ish logged in concurrently.
If we split that out into separate IMAP, web, auth, etc systems, and put in multiple CPUs and more RAM, the search time would drop. But, for now, we don’t need it.
So, do you want a groupware server that does everything, or do you want an IMAP server that just does mail? Make up your mind.
See above.
Zimbra, Lotus Notes, Exchange, and other similar groupware servers don’t use IMAP for everything. In fact, they don’t use IMAP for anything except access to mail folders from non-native clients. They each use their own custom protocol to communicate between the client and the server. You can’t compare these to standard IMAP servers.
The Zimbra client, for example, communicates with the Zimbra server using SOAP calls for everything (mail, contacts, calendar, documents, notes, etc). However, they also provide an IMAP interface to the message store, an iCal interface to the calendars, a WebDAV interface to the documents storage, and an LDAP interface to the global addressbook.
IMAP is for accessing messages, nothing more. And it does that quite nicely. You just have to use the right server and client (especially the right server).
For examples of how to do IMAP wrong, just look at the FirstClass server, and MS Outlook.
I don’t know any reaonable company that would accept it. My Gmail that I use for personal purpose does this job in half a second with the web-based interface.
Even if we want to limit IMAP strictly to email purpose, it still has the huge problem of lack of scalability. IMAP server are too slow when you reach a critical size.
See my reply below. Seems my IMAP client was the bottleneck, not the IMAP server.
What’s “critical size”? And what hardware is being used?
Correction, get a better IMAP client. Doing searches from within Squirrelmail 1.5.0, it takes under 30 seconds to search through my 1.5 GB mailbox (body search). Using KMail 3.5.9, though, it takes just about 5 minutes to do the same search, against the same IMAP server. Not sure what’s going on there.