[00:00:31] Krenair: It kinda seems it's random [00:00:38] some do, some don't [00:01:04] well [00:01:08] somehow I'm not surprised [00:01:10] we should make a list [00:53:44] Reedy, and, you know there's a few other tickets lying around wikimedia-site-requests where your help would be appreciated [00:54:04] Examples? [00:54:09] Too many tickets :) [00:55:08] Reedy, like all of them [00:55:44] you may have some useful input at https://phabricator.wikimedia.org/T126330 [00:59:46] Be nice to get UserThrottle deployed [01:00:50] uh [01:00:53] throttleoverride [01:11:30] 06Operations, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862482 (10DStrine) p:05Triage>03Unbreak! a:03Jgreen [01:15:55] 06Operations, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862486 (10DStrine) I'm putting this back to Unbreak Now. This justifies immediate attention at someone's earliest convenience. Major Gifts' needs should be high priority during the... [01:24:52] 06Operations, 10Fundraising-Backlog, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862518 (10DStrine) [01:36:39] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:37:22] 06Operations, 10DNS, 10Traffic, 07Mobile: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#2862570 (10Krenair) [01:44:25] 06Operations, 10DNS, 10Traffic, 07Mobile: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#2862589 (10Krenair) It doesn't make much sense to add labtestwikitech given T120527. The rest of these can probably just be done easily, might want to be careful with login and zero but... [01:53:13] 06Operations, 06Labs, 10Labs-Infrastructure, 06Reading-Web-Backlog, and 2 others: https://wikitech.m.wikimedia.org/ serves wikimedia.org portal - https://phabricator.wikimedia.org/T120527#2862604 (10Krenair) From the sounds of what @MaxSem wrote on T87633, it actually can't support this, and we don't need... [01:56:23] 06Operations, 10Fundraising-Backlog, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862607 (10Krenair) p:05Unbreak!>03High >>! In T152641#2862486, @DStrine wrote: > I'm putting this back to Unbreak Now. It's not a task that can hold UBN... [02:04:40] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [02:12:32] 06Operations, 10Fundraising-Backlog, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2855852 (10awight) @Krenair wrote, > It only affects (a subset of) staff. That's likely not enough people to qualify for 'drop everything else and work on thi... [02:22:58] 06Operations, 10Fundraising-Backlog, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862648 (10Krenair) I'm an experienced community member here, don't bother trying to belittle me that way. [02:24:39] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:52:39] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [02:53:58] 06Operations, 10Fundraising-Backlog, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862716 (10Jgreen) benefactors@ already exists as either an alias or account under the OIT administered part of the system, which preempts the Ops-administere... [02:57:20] (03PS1) 10Tim Landscheidt: Tools: Undo obsolete /var/mail customization [puppet] - 10https://gerrit.wikimedia.org/r/326306 [02:59:33] (03CR) 10Tim Landscheidt: "Tested on Toolsbeta; random piece of trivia: On Debian, package scripts (postinst, etc.) are stored beneath /var/lib/dpkg/info." [puppet] - 10https://gerrit.wikimedia.org/r/326306 (owner: 10Tim Landscheidt) [03:03:26] (03PS1) 10Tim Landscheidt: Tools: Update list of host aliases for mail relay [puppet] - 10https://gerrit.wikimedia.org/r/326308 [03:09:56] 06Operations, 10Fundraising-Backlog, 10Mail, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862753 (10awight) We're pursuing a workaround in parallel, simply configuring the Gmail account to forward to Zendesk. [03:33:49] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz] [03:34:16] Request from 90.180.83.194 via cp3042 cp3042, Varnish XID 616246274 [03:34:19] Error: 503, Backend fetch failed at Sun, 11 Dec 2016 03:33:58 GMT [03:42:59] PROBLEM - puppet last run on mx2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:43:09] Danny_B, is it recurring? [03:43:11] or a one-off? [03:43:37] now ok, was recurring in time when i reported it [03:43:55] across multiple URLs? [03:44:57] didn't test [04:01:50] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:08:29] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=392.20 Read Requests/Sec=1520.70 Write Requests/Sec=54.00 KBytes Read/Sec=34134.80 KBytes_Written/Sec=1163.20 [04:10:59] RECOVERY - puppet last run on mx2001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:14:29] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=3.30 Read Requests/Sec=66.60 Write Requests/Sec=9.40 KBytes Read/Sec=504.80 KBytes_Written/Sec=515.20 [05:31:53] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862837 (10CaitVirtue) [05:50:35] (03PS1) 10Tim Landscheidt: Labs: Set SYS_UID_MAX and SYS_GID_MAX to 499 [puppet] - 10https://gerrit.wikimedia.org/r/326311 (https://phabricator.wikimedia.org/T45795) [05:51:24] (03CR) 10jenkins-bot: [V: 04-1] Labs: Set SYS_UID_MAX and SYS_GID_MAX to 499 [puppet] - 10https://gerrit.wikimedia.org/r/326311 (https://phabricator.wikimedia.org/T45795) (owner: 10Tim Landscheidt) [05:52:30] 06Operations, 10ops-eqiad: labsdb1001: Swap eth0 cable - https://phabricator.wikimedia.org/T137555#2371671 (10Krenair) I see the warning is still active though [06:20:21] (03PS1) 10Tim Landscheidt: Labs: Remove obsolete code [puppet] - 10https://gerrit.wikimedia.org/r/326312 [06:21:09] (03CR) 10Tim Landscheidt: "Since 47c33658494a580a8fd53265bad7b2f171bd4900 (Tue Jun 30 16:08:48 2015 +0100)." [puppet] - 10https://gerrit.wikimedia.org/r/326312 (owner: 10Tim Landscheidt) [06:54:59] PROBLEM - puppet last run on elastic2021 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[acct],Package[htop] [07:02:09] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [07:22:59] RECOVERY - puppet last run on elastic2021 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:23:09] PROBLEM - puppet last run on mw1275 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:29:09] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:51:09] RECOVERY - puppet last run on mw1275 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:39:49] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:07:49] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [10:20:09] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:34:57] 06Operations, 10Mail, 10MediaWiki-Email, 10Wikimedia-General-or-Unknown, and 3 others: Email server's DMARC config prevents users from sending emails via Special:EmailUser - https://phabricator.wikimedia.org/T66795#2862931 (10AnnaMariaKoshka) @Legoktm - I have sent 3 emails with copy to me. Two of these em... [10:42:49] 06Operations, 10Mail, 10MediaWiki-Email, 10Wikimedia-General-or-Unknown, and 3 others: Email server's DMARC config prevents users from sending emails via Special:EmailUser - https://phabricator.wikimedia.org/T66795#2862943 (10AnnaMariaKoshka) Sorry - it was only a time delay! [10:49:09] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [10:54:49] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [11:12:09] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:14:05] (03CR) 10Merlijn van Deen: [C: 031] "lgtm; http://tools-puppet-compiler.wmflabs.org/326306000/ also looks good." [puppet] - 10https://gerrit.wikimedia.org/r/326306 (owner: 10Tim Landscheidt) [11:22:18] (03CR) 10Dereckson: [C: 031] "Please schedule this next week in SWAT, this is the last week available before code freeze." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323236 (https://phabricator.wikimedia.org/T121853) (owner: 10Odder) [11:22:50] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [11:40:09] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [12:03:49] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:31:49] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [12:40:09] PROBLEM - puppet last run on sca2004 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [12:40:49] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:07:59] RECOVERY - puppet last run on sca2004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:08:49] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [13:27:22] 06Operations, 10Mail, 10MediaWiki-Email, 10Wikimedia-General-or-Unknown, and 3 others: Email server's DMARC config prevents users from sending emails via Special:EmailUser - https://phabricator.wikimedia.org/T66795#2863149 (10AnnaMariaKoshka) @Legoktm - after some time of tesing - it doesnt work. If I se... [14:02:09] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [14:09:19] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:30:19] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:38:19] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [14:52:19] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 10.192.48.44 on port 6479 [14:53:19] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 4633223 keys, up 41 days 6 hours - replication_delay is 0 [15:25:24] (03PS1) 10MarcoAurelio: Removing 'technican' user group from tr.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326354 (https://phabricator.wikimedia.org/T152911) [15:27:09] PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:29:17] (03CR) 10MarcoAurelio: [C: 04-1] "Gotta do I6a02318b first, then this one. The 'technican' group is being removed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio) [15:29:35] (03PS8) 10MarcoAurelio: Rename 'technican' and 'technician' to 'interface-editor' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) [15:31:55] (03CR) 10Brian Wolff: [C: 031] "FWIW, I think this brings a lot of sanity to stuff." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio) [15:42:27] (03PS9) 10MarcoAurelio: Rename 'technician' to 'interface-editor' on trwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) [15:44:57] (03CR) 10MarcoAurelio: "On this PS9 I did remove the references to 'technican' at tr.wikiquote since it's being removed at I6a02318b." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio) [15:56:09] RECOVERY - puppet last run on analytics1047 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:59:40] (03CR) 10Dereckson: [C: 031] DNS configuration for arbcom_cs.wikipedia.org [dns] - 10https://gerrit.wikimedia.org/r/323851 (https://phabricator.wikimedia.org/T151731) (owner: 10MarcoAurelio) [16:00:31] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2863366 (10DStrine) @JGulingan This might be going back to you or someone in IT. We need someone to help clear this up ASAP. [16:00:42] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2863367 (10DStrine) p:05High>03Unbreak! [16:04:42] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2863368 (10DStrine) @Krenair As far as I can tell you aren't part of ops FR-tech or OIT. Please leave this conversation. Also @awight is righ... [16:08:00] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2855852 (10Reedy) I'm kinda confused when you have a dedicated ops person why you didn't contact them directly before... Or OIT as appropriate... [16:12:06] (03CR) 10Odder: "Schedule for the SWAT window beginning at 00:00 UTC on Tuesday, 13 December." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323236 (https://phabricator.wikimedia.org/T121853) (owner: 10Odder) [16:12:59] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:13:59] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2863376 (10Reedy) >>! In T152641#2863368, @DStrine wrote: > @Krenair As far as I can tell you aren't part of ops FR-tech or OIT. Please leave... [16:18:01] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2863379 (10Reedy) >>! In T152641#2863368, @DStrine wrote: > I'm about to start calling ops and OIT on a Sunday. Major Girfts is getting bounced... [16:19:50] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2863380 (10DStrine) We now have a regular oit ticket and a direct email to the team. I'll see what that does in the next few hours [16:30:19] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:34:49] 06Operations, 10DNS, 10Traffic, 07Mobile, 13Patch-For-Review: OTRS-Wiki link to mobile website - https://phabricator.wikimedia.org/T152870#2863395 (10Framawiki) It could be great, of course. Thanks [16:35:10] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Puppet has 22 failures. Last run 2 minutes ago with 22 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [16:39:03] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2855852 (10MZMcBride) >>! In T152641#2862643, @awight wrote: > That's technically incorrect: if MBeat's team can't keep up with their workload... [16:41:59] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:50:29] PROBLEM - puppet last run on analytics1051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:58:19] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:02:09] PROBLEM - puppet last run on poolcounter1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:03:09] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:04:00] 06Operations, 10Fundraising-Backlog, 10Mail, 06Office-IT, 10fundraising-tech-ops: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2863426 (10JGulingan) Hi all, I didn't know that benefactors@ was an actual account that was in our system. There might be a temporary work ar... [17:19:29] RECOVERY - puppet last run on analytics1051 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:31:09] RECOVERY - puppet last run on poolcounter1002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:13:38] (03PS1) 10Dereckson: Enable Translate on no.wikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326359 (https://phabricator.wikimedia.org/T152490) [18:24:55] (03PS1) 10Dereckson: Set language links order for no.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326360 (https://phabricator.wikimedia.org/T148021) [18:30:19] (03CR) 10Tim Landscheidt: [C: 04-1] "Come to think of it, this also applies to production hosts; cf. modules/admin/data/data.yaml with uids < 1000." [puppet] - 10https://gerrit.wikimedia.org/r/326311 (https://phabricator.wikimedia.org/T45795) (owner: 10Tim Landscheidt) [18:32:59] PROBLEM - Host eeden is DOWN: PING CRITICAL - Packet loss = 100% [18:33:09] PROBLEM - Host ns2-v4 is DOWN: PING CRITICAL - Packet loss = 100% [18:38:00] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:41:29] RECOVERY - Host eeden is UP: PING OK - Packet loss = 0%, RTA = 83.72 ms [18:41:29] RECOVERY - Host ns2-v4 is UP: PING OK - Packet loss = 0%, RTA = 84.01 ms [19:06:00] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:09:29] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:19:59] PROBLEM - puppet last run on elastic1040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:35:19] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:37:29] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:47:00] RECOVERY - puppet last run on elastic1040 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:55:12] (03Abandoned) 10Dereckson: Namespace configuration for sk.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/296270 (https://phabricator.wikimedia.org/T138779) (owner: 10Dereckson) [20:01:28] 06Operations, 10ops-esams, 10DNS, 10Traffic, 10netops: eeden ethernet outage - https://phabricator.wikimedia.org/T146391#2863685 (10elukey) Happened again today (UTC timezone): ``` 18:32 PROBLEM - Host eeden is DOWN: PING CRITICAL - Packet loss = 100% 18:33 PROBLEM - Host ns2-v... [20:04:19] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:16:29] PROBLEM - Router interfaces on cr1-eqdfw is CRITICAL: CRITICAL: host 208.80.153.198, interfaces up: 33, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-0/0/1: down - Transit: NTT (service ID 253065) {#11401} [10Gbps]BR [22:19:29] RECOVERY - Router interfaces on cr1-eqdfw is OK: OK: host 208.80.153.198, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 [22:25:19] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:53:19] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:00:30] (03Draft1) 10Paladox: Gerrit: Install filebeat on gerrit server [puppet] - 10https://gerrit.wikimedia.org/r/326374 (https://phabricator.wikimedia.org/T141324) [23:00:36] (03PS2) 10Paladox: Gerrit: Install filebeat on gerrit server [puppet] - 10https://gerrit.wikimedia.org/r/326374 (https://phabricator.wikimedia.org/T141324) [23:01:37] (03CR) 10Paladox: "I've tested filebeat locally with elasticsearch + kibana + logstash" [puppet] - 10https://gerrit.wikimedia.org/r/326374 (https://phabricator.wikimedia.org/T141324) (owner: 10Paladox) [23:10:10] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:24:54] (03CR) 10Dereckson: [C: 031] "I concur, we can deploy this, especially as trusty is mainly used by current tools hosts." [puppet] - 10https://gerrit.wikimedia.org/r/297975 (https://phabricator.wikimedia.org/T139738) (owner: 10Dereckson) [23:30:11] 06Operations, 07Puppet, 06Labs, 10Phabricator, and 2 others: Install Arcanist in toollabs::dev_environ - https://phabricator.wikimedia.org/T139738#2863925 (10Dereckson) As there is a consensus to merge this right now and take care of Precise later if needed, this change has been scheduled to [[ https://wi... [23:39:09] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:41:16] (03PS2) 10Dereckson: Reconfigure interface editor group on ur.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/285209 (https://phabricator.wikimedia.org/T133564) [23:42:18] (03CR) 10Dereckson: "PS2: rebased" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/285209 (https://phabricator.wikimedia.org/T133564) (owner: 10Dereckson) [23:45:46] (03Abandoned) 10Dereckson: Set site title for ma.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/291601 (https://phabricator.wikimedia.org/T136514) (owner: 10Dereckson) [23:59:31] 06Operations, 07Puppet, 06Labs, 10Phabricator, and 2 others: Install Arcanist in toollabs::dev_environ - https://phabricator.wikimedia.org/T139738#2863940 (10Dereckson) p:05Triage>03Normal