fix it. tell us how you fixed it. tell us what was actually wrong. tell us what will happen if it happens again before ‘10. live by it. simple things that are hard to do. push up your quarterly to address this, and to get the bad news out soon, and where it will serve you best.

“my isp sucks less, should not be our goal.”

I have my own view about whether you should EVER use a service that is free (even if you pay, you are still paying for a service tuned for “free”) for a number of reasons, but I am in no position to judge right now :-(

good luck,

Mark

Both in August and last week we were told “rebuilding indexes” or “reindexing” were the cause for the last 2-3 days of recovery during the outage. That means if the reindexing problem could be addressed service would have been restored and available 2-3 days EARLIER.

I don’t know your architecture and, as I said above, I don’t know the software you are using. That said, it seems as though it should be possible to do get around this bottle neck, or at least delay the impact somehow.

In reading about Dovecot it seems that it supports standard maildir mode as well as its modified internal folder model. Perhaps there is a way to go to (the potentially) slower standard maildir model while rebuilding indexes in a more controlled manner allowing service to be restored and indexes rebuilt at the same time. Yes, at completion, the indexes wouldn’t be 100%, but based on comments in the video perhaps they would be clean and could be updated at the time of login with minimal impact, assuming of course that dovecot is smart enough to update and not rebuild in that scenario.

The other suggestion is that you find a way to move affected users in a down cluster to another cluster for immediate restoration of email functionality. Users could then send and receive current email while the failed cluster is repaired. They would not have access to old messages. This would be “degraded” service but not a full outage. It would still be bad, but this would take the sting out.

Lastly, if you implement bounces (preferably timeouts during delivery so messages stay in queues and don’t get lost assuming you can deliver them within 5-7 days), please make it adjustable (if possible) with a setting in MAC or reseller interface. I can certainly understand why some would want bounce notifications, but probably not everyone will. I am on several lists that auto unsubscribe you on bounces, so for myself personally I prefer not to have them bounced. Of course, my users often feel differently. :)

I suspect that if you have a mail server in-line with the rest of your system as the front line, it can return a timeout delivery warning message and hold the message in the queue until it can be delivered. It sounds like you are already doing a lot of that right now, short of the bounce message. I believe typical configurations are 4 hours for delivery timeout warning message and 5 days for delivery failure. I’d probably bump up the 5 days to 15 to allow for crazy problems, but you get the idea.

These are just a few thoughts and observations.

I look forward to hearing about what changes you make.

http://www.linkedin.com/e/gis/1012737

This is an independant place where we resellers can discuss matters of interest to us. After this latest outage, I thought it would be a good idea for us to have a truly independent forum for us to discuss things.

This includes what we are doing to recover from outages, keep our customers, and/or improve our businesses. It also includes items related to our types of business where Tucows/OpenSRS provides some of all the services we rely. Topics do not have to include Tucows or OpenSRS specifically.

Joining the group is easy, especially if you are already on LinkedIn, using the link: http://www.linkedin.com/e/gis/1012737