[00:00:17] We don't want the subscribe at all-projects, just at the meta level, right? [00:00:25] yeh [00:00:28] if you want [00:00:31] i will move it now [00:00:43] to the mediawiki project [00:01:29] Well, first we want to remove all of the allowSuperprojects we have everywhere [00:01:34] So they inherit and don't just override [00:01:59] Ok [00:02:15] ostriches one problem do we have to add subscribe to mediawiki/extensions project [00:02:19] mediawiki/skins project [00:02:25] since the mediawiki project is locked [00:04:55] ostriches https://gerrit.wikimedia.org/r/#/c/326201/ [00:05:10] i will clean up All-Projects once we see if ^^ works. [00:05:20] then i can do it to mediawiki/skins too :) [00:07:51] 06Operations, 10Wikimedia-Logstash: Get 5xx logs into kibana/logstash - https://phabricator.wikimedia.org/T149451#2861108 (10fgiunchedi) Indeed feeding the firehose into logstash directly isn't practical. I checked kafkatee and the filtering happens on output via `grep`, we could replicate the setup we have on... [00:09:15] I've cleaned up it from All-Projects here https://gerrit.wikimedia.org/r/#/c/326200/ [00:09:18] ostriches ^^ [00:09:19] :) [00:10:23] https://gerrit.wikimedia.org/r/#/c/326202/ - will need this on all skins/extensions/vendor [00:10:50] Oh yep [00:10:54] will add it to skins now [00:11:44] I've got patches to everything in skins. I'm just going to push directly rather than spam gerrit with mass conf changes [00:11:59] oh [00:12:00] ok [00:12:40] ostriches im wondering to get submodules to auto update do we have to do a test change [00:12:45] like in extensions/Example [00:13:07] 06Operations, 10DBA, 10MediaWiki-Database: db1028 increased lag after extensions/CentralAuth/maintenance/populateLocalAndGlobalIds.php - https://phabricator.wikimedia.org/T152761#2861123 (10kaldari) @jcrespo: OK, I've [[ https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20161212T1830 | scheduled... [00:13:18] Once we remove all the bogus config. [00:13:23] Ok [00:13:25] Then do a test change [00:13:32] Ok :) [00:13:37] Let's do skins first, done pushing [00:13:41] Ok [00:13:47] which repo should we do it on? [00:14:04] in skins [00:14:13] mediawiki/skins/Example? [00:14:19] Ok [00:14:20] thanks [00:14:36] Oh, we need the subscribe first [00:15:09] https://gerrit.wikimedia.org/r/#/c/326201/ but for skins [00:15:32] ostriches https://gerrit.wikimedia.org/r/#/c/326206/ [00:15:39] oh [00:16:01] ostriches do i do that for skins? [00:16:05] or are you going to force push. [00:16:21] I already updated all the individual skins [00:16:30] oh :) [00:16:45] We just need the subscribe bit for mediawiki/skins itself [00:16:52] ok [00:16:58] submiting now [00:18:19] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:18:32] ostriches https://gerrit.wikimedia.org/r/#/c/326207/ for skins :) [00:20:04] Ok, now we've got this configured for skins, let's try submitting yours. [00:20:09] Ok [00:20:15] https://gerrit.wikimedia.org/r/#/c/326206/ [00:20:16] :) [00:20:29] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:30:11] Nope. [00:30:22] I think I know what's up.... I think the parent has to subscribe to each child explicitly [00:31:03] Oh [00:31:12] I have an idea, one sec [00:31:21] Ok [00:34:42] Like https://gerrit.wikimedia.org/r/#/c/326212/ [00:34:42] ? [00:36:27] yeh maybe [00:36:40] Actually nope [00:36:50] [subscribe ""] [00:37:16] ostriches https://github.com/gerrit-review/gerrit/blob/c62f9fe5111af422d4e93cf48e1cb0dfe6a561dd/Documentation/user-submodules.txt#L88 [00:41:52] Meh, I can't figure it out right now :\ [00:41:59] I'm gonna figure it out later [00:42:40] Ok [00:45:04] ostriches it's strange that it works for me [00:45:18] https://gerrit.git.wmflabs.org/r/#/c/38/ [00:46:19] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [00:46:49] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:05:49] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:14:50] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [01:26:51] (03PS1) 10Arlolra: Enable Parsoid's linter on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/326232 [01:34:49] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:16:43] (03PS1) 10Mattflaschen: Enable GuidedTour on metawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326235 (https://phabricator.wikimedia.org/T152656) [02:17:13] i have no idea why submodules are not working on prod yet they work on test install now [02:17:16] ostriches ^^ [02:19:19] PROBLEM - HHVM rendering on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:19:27] So… should I do a commit to mediawiki/extensions.git that brings all the submodules up to date with master? [02:19:56] It's now quite old (nothing except manual changes since Tuesday). [02:20:09] RECOVERY - HHVM rendering on mw1289 is OK: HTTP OK: HTTP/1.1 200 OK - 70819 bytes in 0.173 second response time [02:20:29] PROBLEM - puppet last run on druid1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:21:46] James_F yeh, it has to be manual for now, but i have been looking at this and got it working on my test install just carn't seem to get it to do it on prod. [02:21:50] ostriches https://gerrit.wikimedia.org/r/#/c/326237/ [02:22:00] ^^ im going to see if that will work. [02:41:06] theres some replys here https://groups.google.com/forum/#!topic/repo-discuss/KzLJiNqu2AM [02:46:32] (03CR) 10Subramanya Sastry: [C: 031] Enable Parsoid's linter on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/326232 (owner: 10Arlolra) [02:48:29] RECOVERY - puppet last run on druid1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:17:27] (03CR) 10Jforrester: [C: 031] "Product sign-off in case anyone asks. :-)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326235 (https://phabricator.wikimedia.org/T152656) (owner: 10Mattflaschen) [03:38:59] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:57:09] PROBLEM - puppet last run on db1086 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:07:59] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:08:29] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=350.00 Read Requests/Sec=3188.40 Write Requests/Sec=716.90 KBytes Read/Sec=20618.00 KBytes_Written/Sec=8437.60 [04:19:29] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=61.80 Read Requests/Sec=0.10 Write Requests/Sec=0.40 KBytes Read/Sec=0.80 KBytes_Written/Sec=3.60 [04:21:09] !log deployed hotfix for T152726, restarted apache on iridium [04:21:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:21:22] T152726: Phabricator upgrade broke milestone tag completion - https://phabricator.wikimedia.org/T152726 [04:26:09] RECOVERY - puppet last run on db1086 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [05:18:50] 06Operations, 06Commons, 06Multimedia, 10media-storage, 15User-Josve05a: Specific revisions of multiple files missing from Swift - 404 Not Found returned - https://phabricator.wikimedia.org/T124101#2861469 (10Srittau) This is a current version: https://commons.wikimedia.org/wiki/File:Burbuja_(1496994920)... [05:32:39] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:50:19] PROBLEM - puppet last run on es1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:00:39] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:19:19] RECOVERY - puppet last run on es1016 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:21:03] 06Operations, 10DBA, 10MediaWiki-Database: db1028 increased lag after extensions/CentralAuth/maintenance/populateLocalAndGlobalIds.php - https://phabricator.wikimedia.org/T152761#2861482 (10Marostegui) ok - I have downtimed db1028 from Monday 12th Dec starting at 18:30:00 UTC until Tuesday 13th finishing at... [07:01:39] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:11:19] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:29:39] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:40:19] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:41:22] 06Operations, 10MediaWiki-API, 10Parsoid, 10RESTBase, and 6 others: HHVM request timeouts not working; support lowering the API request timeout per request - https://phabricator.wikimedia.org/T97192#2861483 (10Joe) Just for the record, the reason requests piled up in T151702 is because of a low-level deadl... [08:09:19] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:20] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:43:51] (03PS1) 10Urbanecm: Alias from WP to NS_PROJECT in kuwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326246 (https://phabricator.wikimedia.org/T152815) [08:54:19] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 2 minutes ago with 21 failures. Failed resources (up to 3 shown): Package[quickstack],Service[puppet],Service[rsyslog],Exec[ip addr add 2620:0:861:103:10:64:32:28/64 dev eth0] [08:54:21] (03PS1) 10Ema: dstat_varnishstat: remove varnish 3 compatibility code [puppet] - 10https://gerrit.wikimedia.org/r/326247 (https://phabricator.wikimedia.org/T150660) [09:03:51] (03PS2) 10Dereckson: Enable GuidedTour on metawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326235 (https://phabricator.wikimedia.org/T152656) (owner: 10Mattflaschen) [09:04:20] (03CR) 10Dereckson: [C: 031] "PS2: more context" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326235 (https://phabricator.wikimedia.org/T152656) (owner: 10Mattflaschen) [09:11:19] PROBLEM - puppet last run on labstore1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:23:20] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [09:39:19] RECOVERY - puppet last run on labstore1005 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [09:43:29] PROBLEM - puppet last run on mw1290 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:12:29] RECOVERY - puppet last run on mw1290 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [10:38:39] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:45:50] (03CR) 10Mobrovac: [C: 031] Enable Parsoid's linter on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/326232 (owner: 10Arlolra) [10:48:19] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:51:49] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0] [10:59:49] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [11:06:39] PROBLEM - puppet last run on mw2156 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:07:39] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [11:16:20] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [11:16:49] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:31:59] PROBLEM - parsoid on wtp1010 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:33:49] RECOVERY - parsoid on wtp1010 is OK: HTTP OK: HTTP/1.1 200 OK - 1014 bytes in 0.011 second response time [11:34:39] RECOVERY - puppet last run on mw2156 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [11:44:49] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [11:54:19] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:01:39] PROBLEM - puppet last run on cp3032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:22:20] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [12:29:40] RECOVERY - puppet last run on cp3032 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:06:54] Seen multiple instances on commons today people uploading 500MB+ images with embedded data; Would it be possible to catch those kinds at the upload step? [13:07:24] AzaToth: It's been discussed about [13:07:25] Yes, but umm somebody would have to implement that [13:07:39] There's a thread on COM:AN I think right now [13:07:40] was really slow trying to get onto the page with the image to delete it :-P [13:07:44] ah [13:07:51] oh [13:08:00] Because its low resolution, we show the original [13:08:06] in its 500mb of glory [13:08:09] yea [13:08:11] I felt it [13:08:14] that's icky [13:08:40] PROBLEM - puppet last run on sca2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:08:40] * bawolff imagines someone on metered internet... [13:08:49] Is this getting really common? [13:09:19] seems to speed up [13:09:30] https://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard#Influx_of_files_with_embedded_data_.28CSD.23F9.29 [13:09:52] I wonder if we can block "new" users from uploading large files [13:09:53] AzaToth: there's an abusefilter that should be preventing those [13:09:57] as they seems to figure out this is something they can do [13:09:57] Reedy: ^ [13:10:01] lol [13:10:11] MatmaRex: when was that set in effect? [13:10:23] AzaToth: a couple days ago [13:10:28] WikipediaZero is for porn. [13:10:35] filter 160 and 162 [13:10:41] MatmaRex: https://commons.wikimedia.org/w/index.php?title=Special:Undelete&target=File%3ACom.rockstargames.bully.part2.png [13:10:46] that one was uploaded today [13:11:58] well, in that case, it sounds like the filter for PNGs is broken [13:11:59] Maybe we could have a test where the size limit changes depending on width*height*frames [13:12:06] oh wait [13:12:14] the one for PNGs only marks them, it doesn't prevent uploading [13:12:19] lol [13:12:19] k [13:13:04] so if folks who wrote it are happy with it now, it should probably be changed to preventing [13:13:15] bawolff: yeah, that's now it's done [13:13:24] ah [13:17:10] how* [13:25:51] 06Operations, 10Mail: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2861660 (10Krenair) This does not remotely qualify for UBN status. Why should this go to ZenDesk instead of OTRS? Additionally, ETAs are not normally given. [13:26:00] 06Operations, 10Mail: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2861661 (10Krenair) p:05Unbreak!>03Triage [13:29:27] (03CR) 10Volans: "LGTM, nitpicking over 2 minor details inline." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/325975 (https://phabricator.wikimedia.org/T147426) (owner: 10Filippo Giunchedi) [13:37:39] RECOVERY - puppet last run on sca2004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:55:53] grrrit-wm1: nick [13:56:17] grrrit-wm1: force-restart [13:56:19] Re-connecting to Gerrit and IRC. [13:57:00] re-connected to Gerrit and IRC. [14:24:21] 06Operations, 06Labs, 13Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#1511839 (10AlexMonk-WMF) Yes. Let's open a separate task (maybe a subtask of T142815) for that? [14:25:13] 06Operations, 06Labs, 10Striker, 07LDAP: Store Wikimedia unified account name (SUL) in LDAP directory - https://phabricator.wikimedia.org/T148048#2861689 (10AlexMonk-WMF) [14:25:16] 06Operations: Enhance account handling (meta bug) - https://phabricator.wikimedia.org/T142815#2547150 (10AlexMonk-WMF) [14:25:43] 06Operations: Require/track Phabricator username - https://phabricator.wikimedia.org/T142830#2547553 (10AlexMonk-WMF) Not all NDA statuses are stored in Phabricator [14:33:49] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:37:39] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:46:59] 06Operations: Sometimes no js is loading on commons - https://phabricator.wikimedia.org/T152839#2861692 (10Steinsplitter) [14:48:05] (not sure if ops is the right product) [14:56:13] Steinsplitter: Checked JS console when it errors? [14:58:44] Reedy: i see nothing special in it. [14:58:59] Be useful to know if it's a HTTP error, a timeout or wahtever [14:59:07] https://www.irccloud.com/pastebin/XnwDHUkZ/console.log [15:02:49] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [15:04:39] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [15:45:15] 06Operations, 10MediaWiki-extensions-UniversalLanguageSelector, 07I18n: Noto Naksh Arabic fonts installation on Sindhi Wikipedia - https://phabricator.wikimedia.org/T152840#2861726 (10Aklapper) [16:06:50] PROBLEM - puppet last run on logstash1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:32:39] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [16:33:49] RECOVERY - puppet last run on logstash1004 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:41:35] 07Puppet, 10Beta-Cluster-Infrastructure: deployment-eventlogging03 has puppet failure due to missing class - https://phabricator.wikimedia.org/T152842#2861773 (10Krenair) [16:42:05] 07Puppet, 10Beta-Cluster-Infrastructure: deployment-eventlogging03 has puppet failure due to missing class - https://phabricator.wikimedia.org/T152842#2861788 (10Krenair) Due to https://gerrit.wikimedia.org/r/#/c/325948/1 which claimed this class was unused (it's not) - T152621 [16:42:58] (03PS18) 10Paladox: Add support for searching gerrit using bug:T1 [puppet] - 10https://gerrit.wikimedia.org/r/308753 (https://phabricator.wikimedia.org/T85002) [16:48:59] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 10.192.48.44 on port 6479 [16:49:59] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 4683723 keys, up 40 days 8 hours - replication_delay is 0 [16:59:39] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:14:49] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:37:20] 06Operations, 10Mail: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2861836 (10MBeat33) The OTRS queue already generally forwards anything donation-related to Zendesk, so this fits with established practice. @Krenair can you explain more about UBN status? Fwd-ing benefactors@... [17:43:49] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:05:43] 06Operations, 10Mail: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2861845 (10Krenair) Yes, this seems inappropriate for UBN priority for several reasons. At first glance: * This is not a bug, it is essentially a feature request (i.e., you are asking for something new) * You... [19:34:39] PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:59:29] PROBLEM - MariaDB Slave Lag: s1 on db1047 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 317.34 seconds [20:02:39] RECOVERY - puppet last run on labnodepool1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:06:39] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:32:29] PROBLEM - puppet last run on analytics1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:34:39] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:50:49] 06Operations, 10Mail: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862031 (10CaitVirtue) Apologies if we're not totally up on the standard protocol here and didn't give enough context. This request is in conjunction with the annual fundraiser going on right now, where we ex... [20:52:59] PROBLEM - MD RAID on thumbor1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:53:50] RECOVERY - MD RAID on thumbor1002 is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0 [21:01:29] RECOVERY - puppet last run on analytics1050 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [21:04:29] RECOVERY - MariaDB Slave Lag: s1 on db1047 is OK: OK slave_sql_lag Replication lag: 27.48 seconds [21:12:49] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:15:21] 06Operations, 10Mail: Create email alias for benefactors@ - https://phabricator.wikimedia.org/T152641#2862072 (10Krenair) I'm not sure if operations will need any more details (and I don't fully understand the mail rules that allow ZenDesk to handle wikimedia mail), this is essentially waiting for one of them... [21:40:50] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:39:59] PROBLEM - puppet last run on sca2004 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [23:07:59] RECOVERY - puppet last run on sca2004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:52:28] (03PS1) 10Reedy: Add otrs-wiki.m.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/326291 (https://phabricator.wikimedia.org/T152870) [23:58:17] Reedy, how many other wikis do we have without existing mobile domains? [23:58:33] Not sure [23:58:41] I noticed stewards had them [23:59:30] arbcom don't