IMAP and protocol sniffing

My recent columns (here and here) about struggling with various combinations of the Treo 600, Notes, and Outlook using IMAP clients spurred some very practical reader feedback, a lot of it focused on my reference to protocol sniffing -- something that perhaps shouldn't be a last resort as I described in my column:

The next step is setting up a protocol sniffer to see what's really going on at the lowest level, even though everyone knows that pulling out the protocol sniffer is the IT equivalent of the 99-yard Hail Mary pass with no time on the clock.

In my experience at least, command line protocol sniffers can be hard to use if you don't use them regularly (I've always struggled with ethereal and tcpdump for some reason). Bill Campbell of Celestial Systems turned me on to tcpflow, a protocol sniffer that can be used to debug IMAP and anything else that transmits data via a TCP connection. I found it to be immediately useful.

The tcpflow man page explains how it works:

tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis or debugging. A program like tcpdump(4) shows a summary of packets seen on the wire, but usually doesn't store the data that's actually being transmitted. In contrast, tcpflow reconstructs the actual data streams and stores each flow in a separate file for later analysis. tcpflow understands TCP sequence numbers and will correctly reconstruct data streams regardless of retransmissions or out-of-order delivery.

I decided to run tcpflow on my Iaptop to see the IMAP conversation on port 143 (sniffing the wireless interface on my PowerBook) -- very easy to read the output:

/usr/local/bin/tcpflow -i en1 -c port 143

/usr/local/bin/tcpflow[796]: listening on en1
192.168.122.035.51209-064.095.097.093.00143: 11 NOOP

064.095.097.093.00143-192.168.122.035.51209: 11 OK Completed

192.168.122.035.51197-066.033.217.004.00143: 9 SELECT INBOX

066.033.217.004.00143-192.168.122.035.51197: * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\Draft \Answered \Flagged \Deleted \Seen)] Limited
* 244 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1066412754] Ok
9 OK [READ-WRITE] Ok

192.168.122.035.51197-066.033.217.004.00143: 10 UID FETCH 1740:* (UID FLAGS)

066.033.217.004.00143-192.168.122.035.51197: 10 OK FETCH completed.

192.168.122.035.51197-066.033.217.004.00143: 11 UID FETCH 1:1739 (UID FLAGS)

066.033.217.004.00143-192.168.122.035.51197: 8 FLAGS (\Seen))
* 36 FETCH (UID 1171 FLAGS (\Seen))
* 37 FETCH (UID 1174 FLAGS (\Seen))
* 38 FETCH (UID 1178 FLAGS (\Answered \Seen))
* 39 FETCH (UID 1181 FLAGS (\Seen))
* 40 FETCH (UID 1183 FLAGS (\Answered \Seen))
* 41 FETCH (UID 1184 FLAGS (\Seen))
* 42 FETCH (UID 1186 FLAGS (\Seen))
* 43 FETCH (UID 1187 FLAGS (\Answered \Seen))
* 44 FETCH (UID 1189 FLAGS (\Seen))
* 45 FETCH (UID 1199 FLAGS (\Seen))
* 46 FETCH (UID 1200 FLAGS (\Seen))
* 47 FETCH (UID 1201 FLAGS (\Seen))
* 48 FETCH (UID 1202 FLAGS (\Seen))
* 49 FETCH (UID 1203 FLAGS (\Seen))
* 50 FETCH (UID 1205 FLAGS (\Seen))
* 51 FETCH (UID 1212 FLAGS (\Seen))
* 52 FETCH (UID 1215 FLAGS (\Seen))
* 53 FETCH (UID 1216 FLAGS (\Seen))
* 54 FETCH (UID 1219 FLAGS (\Seen))
* 55 FETCH (UID 1221 FLAGS (\Seen))
* 56 FETCH (UID 1222 FLAGS (\Answered \Seen))
* 57 FETCH (UID 1223 FLAGS (\Answered \Seen))
* 58 FETCH (UID 1227 FLAGS (\Answered \Seen))
* 59 FETCH (UID 1228 FLAGS (\Answered \Seen))
* 60 FETCH (UID 1232 FLAGS (\Seen))
* 61 FETCH (UID 1235 FLAGS (\Seen))
* 62 FETCH (UID 1238 FLAGS (\Seen))
* 63 FETCH (UID 1240 FLAGS (\Seen))
* 64 FETCH (UID 1246 FLAGS (\Seen))
* 65 FETCH (UID 1247 FLAGS (\Answered \Seen))
* 66 FETCH (UID 1249 FLAGS (\Answered \Seen))
* 67 FETCH (UID 1250 FLAGS (\Seen))
* 68 FETCH (UID 1251 FLAGS (\Answered \Seen))
* 69 FETCH (UID 1253 FLAGS (\
066.033.217.004.00143-192.168.122.035.51197: * 1 FETCH (UID 1024 FLAGS (\Seen))
* 2 FETCH (UID 1026 FLAGS (\Seen))
* 3 FETCH (UID 1030 FLAGS (\Seen))
* 4 FETCH (UID 1031 FLAGS (\Seen))
* 5 FETCH (UID 1106 FLAGS (\Answered \Seen))
* 6 FETCH (UID 1107 FLAGS (\Seen))
* 7 FETCH (UID 1112 FLAGS (\Seen))
* 8 FETCH (UID 1113 FLAGS (\Seen))
* 9 FETCH (UID 1116 FLAGS (\Answered \Seen))
* 10 FETCH (UID 1118 FLAGS (\Answered \Seen))
* 11 FETCH (UID 1119 FLAGS (\Seen))
* 12 FETCH (UID 1121 FLAGS (\Seen))
* 13 FETCH (UID 1122 FLAGS (\Seen))
* 14 FETCH (UID 1123 FLAGS (\Answered \Seen))
* 15 FETCH (UID 1124 FLAGS (\Seen))
* 16 FETCH (UID 1129 FLAGS (\Seen))
* 17 FETCH (UID 1132 FLAGS (\Seen))
* 18 FETCH (UID 1134 FLAGS (\Seen))
* 19 FETCH (UID 1135 FLAGS (\Seen))
* 20 FETCH (UID 1138 FLAGS (\Answered \Seen))
* 21 FETCH (UID 1140 FLAGS (\Seen))
* 22 FETCH (UID 1141 FLAGS (\Answered \Seen))
* 23 FETCH (UID 1143 FLAGS (\Seen))
* 24 FETCH (UID 1144 FLAGS (\Seen))
* 25 FETCH (UID 1146 FLAGS (\Seen))
* 26 FETCH (UID 1147 FLAGS (\Seen))
* 27 FETCH (UID 1148 FLAGS (\Seen))
* 28 FETCH (UID 1153 FLAGS (\Answered \Seen))
* 29 FETCH (UID 1155 FLAGS (\Seen))
* 30 FETCH (UID 1156 FLAGS (\Answered \Seen))
* 31 FETCH (UID 1157 FLAGS (\Seen))
* 32 FETCH (UID 1161 FLAGS (\Answered \Seen))
* 33 FETCH (UID 1165 FLAGS (\Seen))
* 34 FETCH (UID 1166 FLAGS (\Answered \Seen))
* 35 FETCH (UID 116
066.033.217.004.00143-192.168.122.035.51197: Seen))
* 70 FETCH (UID 1266 FLAGS (\Seen))
* 71 FETCH (UID 1269 FLAGS (\Answered \Seen))
* 72 FETCH (UID 1270 FLAGS (\Answered \Seen))
* 73 FETCH (UID 1278 FLAGS (\Answered \Seen))
* 74 FETCH (UID 1279 FLAGS (\Seen))
* 75 FETCH (UID 1281 FLAGS (\Answered \Seen))
* 76 FETCH (UID 1285 FLAGS (\Seen))
* 77 FETCH (UID 1289 FLAGS (\Seen))
* 78 FETCH (UID 1299 FLAGS (\Answered \Seen))
* 79 FETCH (UID 1301 FLAGS (\Seen))
* 80 FETCH (UID 1307 FLAGS (\Answered \Seen))
* 81 FETCH (UID 1308 FLAGS (\Answered \Seen))
* 82 FETCH (UID 1309 FLAGS (\Seen))
* 83 FETCH (UID 1314 FLAGS (\Seen))
* 84 FETCH (UID 1315 FLAGS (\Seen))
* 85 FETCH (UID 1317 FLAGS (\Seen))
* 86 FETCH (UID 1318 FLAGS (\Answered \Seen))
* 87 FETCH (UID 1323 FLAGS (\Answered \Seen))
* 88 FETCH (UID 1324 FLAGS (\Seen))
* 89 FETCH (UID 1327 FLAGS (\Answered \Seen))
* 90 FETCH (UID 1331 FLAGS (\Seen))
* 91 FETCH (UID 1332 FLAGS (\Answered \Seen))
* 92 FETCH (UID 1333 FLAGS (\Seen))
* 93 FETCH (UID 1334 FLAGS (\Seen))
* 94 FETCH (UID 1340 FLAGS (\Answered \Seen))
* 95 FETCH (UID 1341 FLAGS (\Seen))
* 96 FETCH (UID 1349 FLAGS (\Seen))
* 97 FETCH (UID 1350 FLAGS (\Answered \Seen))
* 98 FETCH (UID 1355 FLAGS (\Answered \Seen))
* 99 FETCH (UID 1356 FLAGS (\Seen))
* 100 FETCH (UID 1372 FLAGS (\Seen))
* 101 FETCH (UID 1374 FLAGS (\Answered \Seen))
* 102 FETCH (UID 1380 FL
066.033.217.004.00143-192.168.122.035.51197: AGS (\Answered \Seen))
* 103 FETCH (UID 1384 FLAGS (\Seen))
* 104 FETCH (UID 1386 FLAGS (\Answered \Seen))
* 105 FETCH (UID 1389 FLAGS (\Answered \Seen))
* 106 FETCH (UID 1390 FLAGS (\Seen))
* 107 FETCH (UID 1392 FLAGS (\Answered \Seen))
* 108 FETCH (UID 1397 FLAGS (\Answered \Seen))
* 109 FETCH (UID 1398 FLAGS (\Answered \Seen))
* 110 FETCH (UID 1400 FLAGS (\Seen))
* 111 FETCH (UID 1401 FLAGS (\Seen))
* 112 FETCH (UID 1407 FLAGS (\Answered \Seen))
* 113 FETCH (UID 1417 FLAGS (\Answered \Seen))
* 114 FETCH (UID 1418 FLAGS (\Seen))
* 115 FETCH (UID 1420 FLAGS (\Seen))
* 116 FETCH (UID 1423 FLAGS (\Seen))
* 117 FETCH (UID 1424 FLAGS (\Seen))
* 118 FETCH (UID 1427 FLAGS (\Seen))
* 119 FETCH (UID 1428 FLAGS (\Seen))
* 120 FETCH (UID 1429 FLAGS (\Answered \Seen))
* 121 FETCH (UID 1432 FLAGS (\Seen))
* 122 FETCH (UID 1433 FLAGS (\Seen))
* 123 FETCH (UID 1440 FLAGS (\Seen))
* 124 FETCH (UID 1442 FLAGS (\Seen))
* 125 FETCH (UID 1443 FLAGS (\Seen))
* 126 FETCH (UID 1453 FLAGS (\Seen))
* 127 FETCH (UID 1461 FLAGS (\Answered \Seen))
* 128 FETCH (UID 1472 FLAGS (\Seen))
* 129 FETCH (UID 1473 FLAGS (\Answered \Seen))
* 130 FETCH (UID 1478 FLAGS (\Answered \Seen))
* 131 FETCH (UID 1479 FLAGS (\Answered \Seen))
* 132 FETCH (UID 1480 FLAGS (\Answered \Seen))
* 133 FETCH (UID 1482 FLAGS (\Answered \Seen))
* 134 FETCH (UID 1483 FLAG
066.033.217.004.00143-192.168.122.035.51197: S (\Answered \Seen))
* 135 FETCH (UID 1484 FLAGS (\Answered \Seen))
* 136 FETCH (UID 1488 FLAGS (\Answered \Seen))
* 137 FETCH (UID 1493 FLAGS (\Seen))
* 138 FETCH (UID 1495 FLAGS (\Answered \Seen))
* 139 FETCH (UID 1496 FLAGS (\Answered \Seen))
* 140 FETCH (UID 1497 FLAGS (\Answered \Seen))
* 141 FETCH (UID 1500 FLAGS (\Seen))
* 142 FETCH (UID 1501 FLAGS (\Answered \Seen))
* 143 FETCH (UID 1502 FLAGS (\Answered \Seen))
* 144 FETCH (UID 1504 FLAGS (\Answered \Seen))
* 145 FETCH (UID 1505 FLAGS (\Answered \Seen))
* 146 FETCH (UID 1506 FLAGS (\Seen))
* 147 FETCH (UID 1509 FLAGS (\Seen))
* 148 FETCH (UID 1510 FLAGS (\Answered \Seen))
* 149 FETCH (UID 1512 FLAGS (\Answered \Seen))
* 150 FETCH (UID 1513 FLAGS (\Seen))
* 151 FETCH (UID 1514 FLAGS (\Seen))
* 152 FETCH (UID 1515 FLAGS (\Seen))
* 153 FETCH (UID 1517 FLAGS (\Answered \Seen))
* 154 FETCH (UID 1518 FLAGS (\Seen))
* 155 FETCH (UID 1523 FLAGS (\Answered \Seen))
* 156 FETCH (UID 1526 FLAGS (\Answered \Seen))
* 157 FETCH (UID 1527 FLAGS (\Seen))
* 158 FETCH (UID 1533 FLAGS (\Seen))
* 159 FETCH (UID 1535 FLAGS (\Answered \Seen))
* 160 FETCH (UID 1543 FLAGS (\Seen))
* 161 FETCH (UID 1544 FLAGS (\Seen))
* 162 FETCH (UID 1547 FLAGS (\Answered \Seen))
* 163 FETCH (UID 1552 FLAGS (\Seen))
* 164 FETCH (UID 1555 FLAGS (\Seen))
* 165 FETCH (UID 1558 FLAGS (\Seen))
* 166 FE
066.033.217.004.00143-192.168.122.035.51197: TCH (UID 1560 FLAGS (\Answered \Seen))
* 167 FETCH (UID 1567 FLAGS (\Answered \Seen))
* 168 FETCH (UID 1568 FLAGS (\Seen))
* 169 FETCH (UID 1569 FLAGS (\Seen))
* 170 FETCH (UID 1577 FLAGS (\Seen))
* 171 FETCH (UID 1579 FLAGS (\Answered \Seen))
* 172 FETCH (UID 1581 FLAGS (\Seen))
* 173 FETCH (UID 1585 FLAGS (\Seen))
* 174 FETCH (UID 1587 FLAGS (\Seen))
* 175 FETCH (UID 1591 FLAGS (\Seen))
* 176 FETCH (UID 1596 FLAGS (\Seen))
* 177 FETCH (UID 1598 FLAGS (\Seen))
* 178 FETCH (UID 1599 FLAGS (\Seen))
* 179 FETCH (UID 1600 FLAGS (\Seen))
* 180 FETCH (UID 1601 FLAGS (\Seen))
* 181 FETCH (UID 1602 FLAGS (\Seen))
* 182 FETCH (UID 1604 FLAGS (\Seen))
* 183 FETCH (UID 1608 FLAGS (\Seen))
* 184 FETCH (UID 1610 FLAGS (\Seen))
* 185 FETCH (UID 1611 FLAGS (\Seen))
* 186 FETCH (UID 1612 FLAGS (\Seen))
* 187 FETCH (UID 1615 FLAGS (\Seen))
* 188 FETCH (UID 1624 FLAGS (\Seen))
* 189 FETCH (UID 1627 FLAGS (\Seen))
* 190 FETCH (UID 1628 FLAGS (\Seen))
* 191 FETCH (UID 1629 FLAGS (\Answered \Seen))
* 192 FETCH (UID 1630 FLAGS (\Seen))
* 193 FETCH (UID 1631 FLAGS (\Seen))
* 194 FETCH (UID 1637 FLAGS (\Seen))
* 195 FETCH (UID 1639 FLAGS (\Seen))
* 196 FETCH (UID 1647 FLAGS (\Seen))
* 197 FETCH (UID 1648 FLAGS (\Seen))
* 198 FETCH (UID 1651 FLAGS (\Answered \Seen))
* 199 FETCH (UID 1655 FLAGS (\Answered \Seen))
* 200 FETCH (UID 1660 FL
066.033.217.004.00143-192.168.122.035.51197: AGS (\Answered \Seen))
* 201 FETCH (UID 1661 FLAGS (\Seen))
* 202 FETCH (UID 1662 FLAGS (\Seen))
* 203 FETCH (UID 1665 FLAGS (\Seen))
* 204 FETCH (UID 1667 FLAGS (\Answered \Seen))
* 205 FETCH (UID 1668 FLAGS (\Answered \Seen))
* 206 FETCH (UID 1670 FLAGS (\Answered \Seen))
* 207 FETCH (UID 1671 FLAGS (\Seen))
* 208 FETCH (UID 1672 FLAGS (\Seen))
* 209 FETCH (UID 1675 FLAGS (\Answered \Seen))
* 210 FETCH (UID 1676 FLAGS (\Seen))
* 211 FETCH (UID 1677 FLAGS (\Answered \Seen))
* 212 FETCH (UID 1678 FLAGS (\Seen))
* 213 FETCH (UID 1681 FLAGS (\Seen))
* 214 FETCH (UID 1682 FLAGS (\Answered \Seen))
* 215 FETCH (UID 1686 FLAGS (\Seen))
* 216 FETCH (UID 1688 FLAGS (\Seen))
* 217 FETCH (UID 1690 FLAGS (\Answered \Seen))
* 218 FETCH (UID 1691 FLAGS (\Seen))
* 219 FETCH (UID 1692 FLAGS (\Seen))
* 220 FETCH (UID 1695 FLAGS (\Seen))
* 221 FETCH (UID 1696 FLAGS (\Seen))
* 222 FETCH (UID 1697 FLAGS (\Answered \Seen))
* 223 FETCH (UID 1699 FLAGS (\Seen))
* 224 FETCH (UID 1700 FLAGS (\Seen))
* 225 FETCH (UID 1701 FLAGS (\Seen))
* 226 FETCH (UID 1704 FLAGS (\Seen))
* 227 FETCH (UID 1705 FLAGS (\Seen))
* 228 FETCH (UID 1706 FLAGS (\Seen))
* 229 FETCH (UID 1708 FLAGS (\Answered \Seen))
* 230 FETCH (UID 1709 FLAGS (\Seen))
* 231 FETCH (UID 1713 FLAGS (\Seen))
* 232 FETCH (UID 1714 FLAGS (\Seen))
* 233 FETCH (UID 1715 FLAGS (\Seen))

066.033.217.004.00143-192.168.122.035.51197: * 234 FETCH (UID 1716 FLAGS (\Seen))
* 235 FETCH (UID 1718 FLAGS (\Seen))
* 236 FETCH (UID 1722 FLAGS (\Seen))
* 237 FETCH (UID 1723 FLAGS (\Seen))
* 238 FETCH (UID 1729 FLAGS (\Seen))
* 239 FETCH (UID 1731 FLAGS (\Seen))
* 240 FETCH (UID 1732 FLAGS (\Seen))
* 241 FETCH (UID 1735 FLAGS (\Seen))
* 242 FETCH (UID 1736 FLAGS (\Seen))
* 243 FETCH (UID 1738 FLAGS (\Seen))
* 244 FETCH (UID 1739 FLAGS (\Seen))
11 OK FETCH completed.

192.168.122.035.51197-066.033.217.004.00143: 12 NOOP

066.033.217.004.00143-192.168.122.035.51197: 12 OK NOOP completed

192.168.122.035.51197-066.033.217.004.00143: 13 CLOSE

066.033.217.004.00143-192.168.122.035.51197: 13 OK mailbox closed.

Posted by Chad Dickerson at May 7, 2004 09:19 AM