[12:27:12] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#2030874 (10Aklapper) [12:27:50] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1336469 (10Aklapper) [12:28:39] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2030881 (10fgiunchedi) 3NEW [12:40:25] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2030906 (10Aklapper) p:5Triage>3High [12:43:37] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2030908 (10ArielGlenn) From yesterday's log I see: 2016-02-15 20:30:57 exim 4.82 daemon started: pid=4631, -q10m, not listening for SMTP This may have occurred during the phab maintenance w... [12:47:44] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2030910 (10ArielGlenn) Yes, checking the syslog confirms that this is due to the one puppet run done at that time. But why restarting the service 'just worked'... no idea. [12:55:45] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2030921 (10ArielGlenn) Feb 15 20:31:13 iridium puppet-agent[2616]: (/Stage[main]/Exim4/File[/etc/default/exim4]/content) +++ /tmp/puppet file20160215-2616-1b6m86a#0112016-02-15 20:31:13.66... [13:00:35] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2030935 (10fgiunchedi) likely not related but each mail from iridium also generates an exim mainlog entry for `localhost.localdomain` not found ``` 2016-02-16 12:36:04 no IP address found... [13:15:55] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2030990 (10fgiunchedi) looks like it is back now, additionally mx1001 / mx2001 ipv6 addresses are firewalled off iridium, in this case exim falls back to ipv4 after 5 minutes ``` root@mx10... [13:19:10] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2031003 (10fgiunchedi) p:5High>3Normal [13:30:20] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2031048 (10ArielGlenn) here's what's on iridum, from manifests/role/mail.pp role::mail::mx ferm::service { 'exim-smtp': proto => 'tcp', port => '25', } iptable... [13:32:55] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2031058 (10ArielGlenn) ip6tables shows no such similar rule so I guess the ferm rule is wrong. if that's the case other hosts could be impacted too. [13:44:22] 6Phabricator, 10Phabricator-Sprint-Extension, 10Phragile, 6TCB-Team: Projects V3 does not expose maniphest.points through the API - https://phabricator.wikimedia.org/T126982#2031076 (10Jakob_WMDE) @epriestley thanks, I'm looking into it! [14:00:24] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2031132 (10ArielGlenn) nope, looking at wrong tab. here's the class on iridium. ferm/conf.d/10_phabmain-smtp # Autogenerated by puppet. DO NOT EDIT BY HAND! # # &R_SERVICE(tcp, 25, (@re... [14:34:00] 6Phabricator: Improve Phabricator welcome panel - https://phabricator.wikimedia.org/T126865#2031284 (10Aklapper) Better now? :) See https://phabricator.wikimedia.org/ [14:41:16] 6Phabricator, 6Operations: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2031318 (10ArielGlenn) https://gerrit.wikimedia.org/r/#/c/270960/ why did this not show up on the ticket when the task number is in the commit? Anyways.. [14:59:05] 6Phabricator: Decide what to do with open WMF "Growth" Team tasks in Phabricator - https://phabricator.wikimedia.org/T127072#2031404 (10Aklapper) 3NEW [17:28:34] 6Phabricator, 6Operations, 5Patch-For-Review: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2032101 (10Aklapper) [19:11:07] 6Phabricator: Plan announcement of new major phabricator upgrade - https://phabricator.wikimedia.org/T127019#2032563 (10mmodell) I can post an announcement on the front page of phabricator but the home page is customized by most heavy users so they likely would not see the message. [19:32:42] 6Phabricator, 6Operations, 5Patch-For-Review: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2032646 (10Dzahn) >>! In T127053#2031318, @ArielGlenn wrote: > https://gerrit.wikimedia.org/r/#/c/270960/ why did this not show up on the ticket when the task number i... [19:35:03] 6Phabricator, 6Operations, 5Patch-For-Review: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2032649 (10mmodell) From what I can remember, iridium hasn't had full ipv6 support until recently. I guess it's taken more work due to it being a somewhat special case... [20:02:42] mail for phabricator user Philippe-WMF is piling up in the queue, should we disable the phab user? [20:08:22] there should probably be some policy about disabling WMF-only accounts who leave [20:08:35] of course, you'd have to actually be notified when people leave [20:08:43] (seriously, good luck with that bit) [20:08:53] but yes, I think that one is safe to disable [20:12:51] the main problem is that the wikimedia.org email account gets disabled and phabricator just keeps trying to deliver their mail, it doesn't seem to ever give up [20:13:14] 6Phabricator, 6Operations, 5Patch-For-Review: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2032824 (10Dzahn) The change above looks good and i would have merged it but i did not because puppet is disabled in iridium, with: (Reason: 'more testing needed before... [20:20:19] there are enough mediums outside of phab for a user to escalate if they need to [20:20:29] plus that user wno't have access to that email anyway [20:20:35] so it's a necessity [20:20:38] I would [20:24:34] twentyafterfour: Disable and add to the list on https://phabricator.wikimedia.org/T123139 [20:35:05] 6Phabricator, 6Operations, 5Patch-For-Review: iridium / phabricator not accepting email via smtp - https://phabricator.wikimedia.org/T127053#2032943 (10ArielGlenn) That's why I commented on the changeset to not merge it. Puppet will remain disabled over there til this https://gerrit.wikimedia.org/r/#/c/2695... [21:18:37] 6Phabricator, 7Mail, 7WorkType-Maintenance: Disable Phab accounts with invalid e-mails that bounce - https://phabricator.wikimedia.org/T123139#2033090 (10mmodell)