John Hawkinson
2018-09-15 15:23:55 UTC
Good morning, mutt-dev:
At the beginning of this year, our Exchange servers were updated to Exchange 2013 (15.0.1365.7), at which point I and other MIT mutt users began to suffer greatly from mutt connections to the Exchange IMAP server timing out after 5 minutes if mutt was anywhere other than the index (e.g. reading or composing a message).
This was extremely painful, and I'm super-embarassed to report that at the time, although I spent a while mucking with the "timeout" and "mail_check" parameters to no avail, somehow the "imap_keepalive" parameter escaped my notice. Reducing it from it's 300-second default to 60-seconds seems to solve the vast majority of those problems, and I am kicking myself for suffering through months of aborted connections and lost deletion state unnecessarily.
I wonder if there's any appetite for reducing the default "imap_keepalive," either uncondionally all the time (I'm not sure whether our Exchange short timeout is a site-specific configuration headache or a standard one, and I also haven't experimeted to determine the largest timeout that safely works), or adaptively in response to servers identifying themselves as:
6< * OK The Microsoft Exchange IMAP4 service is ready.
6> a0000 CAPABILITY
6< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=NTLM AUTH=GSSAPI STARTTLS UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
6< a0000 OK CAPABILITY completed.
In any case, defaults aside, it appears there are a few other places where IMAP keepalives aren't being sent. The primary one that I hit is when an external interactive viewer is launched (e.g. w3m for html email), mutt calls mutt_wait_filter() on the pid, which calls waitpid() and blocks. But instead it could call imap_wait_keepalive() on the pid, just as _mutt_system() does. This appears to solve the problem of dieing connections during external viewers. A patch for this is attached.
It does appear there are a handful of other places where waitpid() is called directly that could potentially be problems: (1) In mutt_tunnel.c; I don't use this feature, so I'm loathe to touch it. (2) In sendlib.c; as long as sending mail doesn't take too long, this shouldn't come up, and this code already uses alarm() around wait4pid(), so likely would not play nice with imap_wait_keepalive() which also uses alarm() -- at least not without careful thought.
Anyhow, patch for the simpler mutt_wait_filter() case attached.
p.s.: While running mutt under lldb before I realized "imap_timeout"'s existence, it seemed that mutt's choice to ignore SIGPIPE leads to it failing to report that the IMAP connection had closed and it can sit unresponsive to user input for many seconds, when instead it should be able to report closed connections earlier from SIGPIPE. I did not run this down, and presumably closed connections will be much more rare in my life going forward.
Thanks.
--***@mit.edu
John Hawkinson
From 998aee796ba1a5459e28aea047ff7b5452836e51 Mon Sep 17 00:00:00 2001
From: John Hawkinson <***@mit.edu>
Date: Sat, 15 Sep 2018 00:40:34 -0400
Subject: [PATCH 1/1] mutt_wait_filter(): Use imap_wait_keepalive() not
waitpid()
Otherwise we do not maintain IMAP connections while an external viewer
is open (e.g. HTML email), and servers with short timeouts can close
the connection when we come back.
---
filter.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/filter.c b/filter.c
index 331d5425..67dcf72f 100644
--- a/filter.c
+++ b/filter.c
@@ -23,6 +23,10 @@
#include "mutt.h"
#include "mutt_curses.h"
+#ifdef USE_IMAP
+# include "imap.h"
+#endif
+
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
@@ -183,7 +187,12 @@ int mutt_wait_filter (pid_t pid)
{
int rc;
- waitpid (pid, &rc, 0);
+#ifndef USE_IMAP
+ /* wait for the (first) child process to finish */
+ waitpid (pid, &rc, 0);
+#else
+ rc = imap_wait_keepalive (pid);
+#endif
mutt_unblock_signals_system (1);
rc = WIFEXITED (rc) ? WEXITSTATUS (rc) : -1;
At the beginning of this year, our Exchange servers were updated to Exchange 2013 (15.0.1365.7), at which point I and other MIT mutt users began to suffer greatly from mutt connections to the Exchange IMAP server timing out after 5 minutes if mutt was anywhere other than the index (e.g. reading or composing a message).
This was extremely painful, and I'm super-embarassed to report that at the time, although I spent a while mucking with the "timeout" and "mail_check" parameters to no avail, somehow the "imap_keepalive" parameter escaped my notice. Reducing it from it's 300-second default to 60-seconds seems to solve the vast majority of those problems, and I am kicking myself for suffering through months of aborted connections and lost deletion state unnecessarily.
I wonder if there's any appetite for reducing the default "imap_keepalive," either uncondionally all the time (I'm not sure whether our Exchange short timeout is a site-specific configuration headache or a standard one, and I also haven't experimeted to determine the largest timeout that safely works), or adaptively in response to servers identifying themselves as:
6< * OK The Microsoft Exchange IMAP4 service is ready.
6> a0000 CAPABILITY
6< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=NTLM AUTH=GSSAPI STARTTLS UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
6< a0000 OK CAPABILITY completed.
In any case, defaults aside, it appears there are a few other places where IMAP keepalives aren't being sent. The primary one that I hit is when an external interactive viewer is launched (e.g. w3m for html email), mutt calls mutt_wait_filter() on the pid, which calls waitpid() and blocks. But instead it could call imap_wait_keepalive() on the pid, just as _mutt_system() does. This appears to solve the problem of dieing connections during external viewers. A patch for this is attached.
It does appear there are a handful of other places where waitpid() is called directly that could potentially be problems: (1) In mutt_tunnel.c; I don't use this feature, so I'm loathe to touch it. (2) In sendlib.c; as long as sending mail doesn't take too long, this shouldn't come up, and this code already uses alarm() around wait4pid(), so likely would not play nice with imap_wait_keepalive() which also uses alarm() -- at least not without careful thought.
Anyhow, patch for the simpler mutt_wait_filter() case attached.
p.s.: While running mutt under lldb before I realized "imap_timeout"'s existence, it seemed that mutt's choice to ignore SIGPIPE leads to it failing to report that the IMAP connection had closed and it can sit unresponsive to user input for many seconds, when instead it should be able to report closed connections earlier from SIGPIPE. I did not run this down, and presumably closed connections will be much more rare in my life going forward.
Thanks.
--***@mit.edu
John Hawkinson
From 998aee796ba1a5459e28aea047ff7b5452836e51 Mon Sep 17 00:00:00 2001
From: John Hawkinson <***@mit.edu>
Date: Sat, 15 Sep 2018 00:40:34 -0400
Subject: [PATCH 1/1] mutt_wait_filter(): Use imap_wait_keepalive() not
waitpid()
Otherwise we do not maintain IMAP connections while an external viewer
is open (e.g. HTML email), and servers with short timeouts can close
the connection when we come back.
---
filter.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/filter.c b/filter.c
index 331d5425..67dcf72f 100644
--- a/filter.c
+++ b/filter.c
@@ -23,6 +23,10 @@
#include "mutt.h"
#include "mutt_curses.h"
+#ifdef USE_IMAP
+# include "imap.h"
+#endif
+
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
@@ -183,7 +187,12 @@ int mutt_wait_filter (pid_t pid)
{
int rc;
- waitpid (pid, &rc, 0);
+#ifndef USE_IMAP
+ /* wait for the (first) child process to finish */
+ waitpid (pid, &rc, 0);
+#else
+ rc = imap_wait_keepalive (pid);
+#endif
mutt_unblock_signals_system (1);
rc = WIFEXITED (rc) ? WEXITSTATUS (rc) : -1;
--
2.12.2
2.12.2