[00:03:17] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [00:05:45] 6operations, 6Labs, 10Labs-Infrastructure, 10Wikimedia-Apache-configuration, 10wikitech.wikimedia.org: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#1384434 (10Dzahn) The whole VM has 40G in / but /var/lib/mysql alone is 24G already, plus the images and dump files, we'll just ne... [00:08:57] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected [00:10:15] 6operations, 10RESTBase, 10RESTBase-Cassandra: begin testing Cassandra 2.1.6 - https://phabricator.wikimedia.org/T101745#1384447 (10GWicke) We have moved from 2.1.6 to the pre-release 2.1.7 deb: http://people.apache.org/~jake/cassandra_2.1.7_all.deb So far things are looking good: - metrics are working -... [00:11:36] 6operations, 10Traffic, 5HTTPS-by-default, 5Patch-For-Review, 7Pybal: Add support for setting weight=0 when depooling - https://phabricator.wikimedia.org/T86650#1384451 (10BBlack) Yes, and that patch only exists in kernel v3.11 and higher. Luckily, even on the LVS balancers that are still precise, we up... [00:16:33] (03PS1) 10Thcipriani: Make varnish template compatible with labs [puppet] - 10https://gerrit.wikimedia.org/r/219502 (https://phabricator.wikimedia.org/T102570) [00:17:07] RECOVERY - puppet last run on mw2060 is OK Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:26:21] (03PS1) 10GWicke: Don't start cassandra on boot or via puppet [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/219503 (https://phabricator.wikimedia.org/T103134) [00:26:31] (03PS1) 10Ori.livneh: mediawiki: add python-pygments package [puppet] - 10https://gerrit.wikimedia.org/r/219504 [00:26:48] (03CR) 10Ori.livneh: [C: 032 V: 032] mediawiki: add python-pygments package [puppet] - 10https://gerrit.wikimedia.org/r/219504 (owner: 10Ori.livneh) [00:26:57] legoktm: like that ^^ [00:28:28] ori: so that's pygments 1.6? the current release is 2.0.2 [00:28:46] * legoktm looks at changelog [00:29:23] 6operations, 10Traffic, 5HTTPS-by-default, 5Patch-For-Review, 7Pybal: Add support for setting weight=0 when depooling - https://phabricator.wikimedia.org/T86650#1384491 (10BBlack) That made me go and review all of this, though, and there is one minor issue: the flag for doing smart things on weight=0 exi... [00:32:48] legoktm: I tested with both, but we can backport 2.0.2 [00:33:07] legoktm: also, pygments is pure python and has no dependencies, so we can bundle it, yeah. [00:33:57] (03PS2) 10BBlack: Make varnish template compatible with labs [puppet] - 10https://gerrit.wikimedia.org/r/219502 (https://phabricator.wikimedia.org/T102570) (owner: 10Thcipriani) [00:34:37] (03CR) 10BBlack: [C: 032 V: 032] Make varnish template compatible with labs [puppet] - 10https://gerrit.wikimedia.org/r/219502 (https://phabricator.wikimedia.org/T102570) (owner: 10Thcipriani) [00:34:53] legoktm: in fact, it's distributed as a single wheel file: https://pypi.python.org/pypi/Pygments this should be easy [00:35:01] yay wheels [00:35:18] I think bundling is a better idea for now, especially given how often we've had to patch geshi [00:36:41] also, the syntaxhighlight bugs need to be triaged...I bet https://phabricator.wikimedia.org/T41643 is fixed with pygments, so is the request for Mathematic support (in 2.0+) [00:36:45] Mathmatica* [00:37:28] yeah, but let's get it merged first [00:37:42] i'd also like to do the bundling of pygments in a follow-up, too [00:50:33] legoktm: if i want a config var that is assigned a default value in extension.json to have some value relative to $IP, how do i do that? [00:50:43] do i initialize it in an extension setup function instead? [00:56:56] ori: use a setup function for now, https://phabricator.wikimedia.org/T100956 is the bug for it. I have a general idea of it in my head, but not much time to implement it [01:32:25] 6operations, 10Traffic, 7HTTPS: implement Public Key Pinning (HPKP) for Wikimedia domains - https://phabricator.wikimedia.org/T92002#1384558 (10BBlack) We've chatted about this off and on via IRC. My current thinking (which isn't all that different from above is): - **Definitely** - we should turn on H... [02:18:37] !log l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 07m 02s) [02:18:46] Logged the message, Master [02:21:31] !log LocalisationUpdate completed (1.26wmf10) at 2015-06-20 02:21:30+00:00 [02:21:35] Logged the message, Master [04:19:34] 6operations, 6Labs, 10Labs-Infrastructure, 10Wikimedia-Apache-configuration, 10wikitech.wikimedia.org: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#1384634 (10Andrew) wikitech-static was syncing a ton of unused image files from silver which I just cleaned up a few days ago -- t... [04:27:14] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 20 04:27:14 UTC 2015 (duration 27m 13s) [04:27:19] Logged the message, Master [04:36:33] (03CR) 10Krinkle: [C: 04-1] Enable browse prototype on English Wikipedia (035 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/219451 (https://phabricator.wikimedia.org/T101155) (owner: 10Jdlrobson) [06:30:27] PROBLEM - puppet last run on mw2086 is CRITICAL puppet fail [06:30:38] PROBLEM - puppet last run on cp1071 is CRITICAL puppet fail [06:31:54] 6operations, 10ops-eqiad, 10Traffic: eqiad: investigate thermal issues with some cp10xx machines - https://phabricator.wikimedia.org/T103226#1384806 (10BBlack) 3NEW a:3Cmjohnson [06:32:36] PROBLEM - puppet last run on labsdb1003 is CRITICAL Puppet has 1 failures [06:32:37] PROBLEM - puppet last run on cp1056 is CRITICAL Puppet has 2 failures [06:32:46] PROBLEM - puppet last run on cp3014 is CRITICAL Puppet has 1 failures [06:33:16] PROBLEM - puppet last run on cp4008 is CRITICAL Puppet has 1 failures [06:33:16] PROBLEM - puppet last run on cp3037 is CRITICAL Puppet has 1 failures [06:33:47] PROBLEM - puppet last run on iron is CRITICAL Puppet has 1 failures [06:33:56] PROBLEM - puppet last run on analytics1030 is CRITICAL Puppet has 1 failures [06:34:07] PROBLEM - puppet last run on rhodium is CRITICAL Puppet has 1 failures [06:34:09] PROBLEM - puppet last run on db1002 is CRITICAL Puppet has 1 failures [06:34:17] PROBLEM - puppet last run on db2064 is CRITICAL Puppet has 1 failures [06:34:17] PROBLEM - puppet last run on subra is CRITICAL Puppet has 1 failures [06:34:28] PROBLEM - puppet last run on cp3042 is CRITICAL Puppet has 1 failures [06:34:37] PROBLEM - puppet last run on mw1069 is CRITICAL Puppet has 1 failures [06:34:47] PROBLEM - puppet last run on db1015 is CRITICAL Puppet has 1 failures [06:34:56] PROBLEM - puppet last run on ms-fe2001 is CRITICAL Puppet has 1 failures [06:34:56] PROBLEM - puppet last run on wtp2015 is CRITICAL Puppet has 1 failures [06:34:57] PROBLEM - puppet last run on mw1173 is CRITICAL Puppet has 1 failures [06:35:06] PROBLEM - puppet last run on mw1170 is CRITICAL Puppet has 1 failures [06:35:16] PROBLEM - puppet last run on mw1123 is CRITICAL Puppet has 1 failures [06:35:27] PROBLEM - puppet last run on mw1092 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw1153 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw1176 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw2093 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw2003 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw2095 is CRITICAL Puppet has 1 failures [06:35:46] PROBLEM - puppet last run on mw2045 is CRITICAL Puppet has 1 failures [06:36:06] PROBLEM - puppet last run on mw2184 is CRITICAL Puppet has 1 failures [06:36:16] PROBLEM - puppet last run on mw2036 is CRITICAL Puppet has 1 failures [06:36:36] PROBLEM - puppet last run on mw2050 is CRITICAL Puppet has 1 failures [06:36:37] PROBLEM - puppet last run on mw2134 is CRITICAL Puppet has 1 failures [06:36:37] PROBLEM - puppet last run on mw2079 is CRITICAL Puppet has 1 failures [06:36:37] PROBLEM - puppet last run on mw2017 is CRITICAL Puppet has 1 failures [06:36:38] PROBLEM - puppet last run on mw2023 is CRITICAL Puppet has 1 failures [06:47:07] RECOVERY - puppet last run on db1015 is OK Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:47:26] RECOVERY - puppet last run on cp4008 is OK Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:47:26] RECOVERY - puppet last run on cp3037 is OK Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:47:56] RECOVERY - puppet last run on iron is OK Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:47:57] RECOVERY - puppet last run on analytics1030 is OK Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:47:57] RECOVERY - puppet last run on mw1176 is OK Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:48:07] RECOVERY - puppet last run on rhodium is OK Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:48:16] RECOVERY - puppet last run on cp1071 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:17] RECOVERY - puppet last run on db1002 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:26] RECOVERY - puppet last run on db2064 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:27] RECOVERY - puppet last run on labsdb1003 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:27] RECOVERY - puppet last run on subra is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:27] RECOVERY - puppet last run on cp1056 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:37] RECOVERY - puppet last run on cp3042 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:37] RECOVERY - puppet last run on cp3014 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:46] RECOVERY - puppet last run on mw1069 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:57] RECOVERY - puppet last run on ms-fe2001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:06] RECOVERY - puppet last run on wtp2015 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:06] RECOVERY - puppet last run on mw2134 is OK Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:49:07] RECOVERY - puppet last run on mw2079 is OK Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:49:07] RECOVERY - puppet last run on mw2023 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:07] RECOVERY - puppet last run on mw2017 is OK Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:49:07] RECOVERY - puppet last run on mw1173 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:16] RECOVERY - puppet last run on mw1170 is OK Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:49:26] RECOVERY - puppet last run on mw1123 is OK Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:49:38] RECOVERY - puppet last run on mw1092 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:46] RECOVERY - puppet last run on mw1153 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:47] RECOVERY - puppet last run on mw2093 is OK Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:49:47] RECOVERY - puppet last run on mw2086 is OK Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:49:47] RECOVERY - puppet last run on mw2095 is OK Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:49:47] RECOVERY - puppet last run on mw2003 is OK Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:49:47] RECOVERY - puppet last run on mw2045 is OK Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:50:07] RECOVERY - puppet last run on mw2184 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:50:19] RECOVERY - puppet last run on mw2036 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:50:38] RECOVERY - puppet last run on mw2050 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [08:04:58] PROBLEM - Host cp2006 is DOWN: PING CRITICAL - Packet loss = 100% [08:38:07] PROBLEM - puppet last run on ms-fe3001 is CRITICAL Puppet has 1 failures [08:51:52] 10Ops-Access-Requests, 6operations, 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384949 (10hashar) 5Open>3Resolved a:3hashar @Sniedzielski you should now have access to the labs `integration` project :-} [08:53:56] RECOVERY - puppet last run on ms-fe3001 is OK Puppet is currently enabled, last run 52 seconds ago with 0 failures [08:58:57] RECOVERY - Host cp2006 is UPING OK - Packet loss = 0%, RTA = 42.94 ms [09:03:26] 6operations, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1384964 (10hashar) >>! In T102566#1383626, @Tgr wrote: > [snip] we need a drop-in fix for people who are on old/weird OSes... [09:06:34] 6operations, 10Traffic: update the multicast purging documentation - https://phabricator.wikimedia.org/T82096#1384973 (10hashar) One might also want to describe the related MediaWiki configuration via `$wgHTCPRouting`. [09:20:17] PROBLEM - puppet last run on mw2018 is CRITICAL puppet fail [09:37:46] RECOVERY - puppet last run on mw2018 is OK Puppet is currently enabled, last run 58 seconds ago with 0 failures [10:33:18] PROBLEM - High load average on ms-be1002 is CRITICAL - load average: 242.61, 162.89, 79.16 [10:36:29] 6operations, 10ops-eqiad, 7Database: Disk issue on db1028 - https://phabricator.wikimedia.org/T103230#1385038 (10jcrespo) 3NEW [10:38:08] PROBLEM - puppet last run on cp2016 is CRITICAL puppet fail [10:38:25] ^I've seen this, degradation seems not critical yet, plus it is a slave [10:42:38] MariaDB PRODUCTION s7 localhost (none) > SET GLOBAL innodb_flush_log_at_trx_commit = 0; just temporarelly [10:43:58] 6operations, 10ops-eqiad, 7Database: Disk issue on db1028 - https://phabricator.wikimedia.org/T103230#1385051 (10jcrespo) Setting temporarily SET GLOBAL innodb_flush_log_at_trx_commit = 0; to help with replication. [10:55:48] RECOVERY - puppet last run on cp2016 is OK Puppet is currently enabled, last run 15 seconds ago with 0 failures [11:26:14] 6operations, 10Wikimedia-Bugzilla, 5Patch-For-Review: redirect old-bugzilla to static-bugzilla - https://phabricator.wikimedia.org/T103190#1385087 (10Aklapper) [11:26:16] 6operations, 10Wikimedia-Bugzilla: remove Bugzilla installation remnants from zirconium and repos - https://phabricator.wikimedia.org/T103193#1385086 (10Aklapper) [12:45:58] 10Ops-Access-Requests, 6operations, 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1385135 (10Krenair) I had thought these requests would require production access as well? Was that wrong? [12:50:18] (03CR) 10Mobrovac: [C: 031] Don't start cassandra on boot or via puppet [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/219503 (https://phabricator.wikimedia.org/T103134) (owner: 10GWicke) [13:17:27] db1047 is laging also due to RAID policy having failed back to writethrough (not critical also, not for production) [14:01:17] PROBLEM - puppet last run on db2017 is CRITICAL Puppet has 1 failures [14:06:03] ^checking [14:11:46] RECOVERY - puppet last run on db2017 is OK Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:11:57] spurious proxy error from puppet? [14:12:36] The proxy server received an invalid response from an upstream server. [14:17:57] <_joe_> jynus: no i think strontium had an hiccup [14:18:13] <_joe_> strontium is our second backend for the puppetmaster [15:01:17] PROBLEM - puppet last run on mw2131 is CRITICAL puppet fail [15:18:37] RECOVERY - puppet last run on mw2131 is OK Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:23:36] PROBLEM - YARN NodeManager Node-State on analytics1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:25:16] RECOVERY - YARN NodeManager Node-State on analytics1016 is OK YARN NodeManager analytics1016.eqiad.wmnet:8041 Node-State: RUNNING [16:30:27] PROBLEM - YARN NodeManager Node-State on analytics1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:33:56] RECOVERY - YARN NodeManager Node-State on analytics1016 is OK YARN NodeManager analytics1016.eqiad.wmnet:8041 Node-State: RUNNING [17:12:57] PROBLEM - puppet last run on mw2006 is CRITICAL puppet fail [17:14:27] (03PS1) 10Giuseppe Lavagetto: appservers: re-move to ganglia_new [puppet] - 10https://gerrit.wikimedia.org/r/219585 [17:15:19] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] appservers: re-move to ganglia_new [puppet] - 10https://gerrit.wikimedia.org/r/219585 (owner: 10Giuseppe Lavagetto) [17:21:27] PROBLEM - YARN NodeManager Node-State on analytics1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:24:47] RECOVERY - YARN NodeManager Node-State on analytics1016 is OK YARN NodeManager analytics1016.eqiad.wmnet:8041 Node-State: RUNNING [17:32:17] RECOVERY - puppet last run on mw2006 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [17:42:15] 6operations, 10Traffic, 7HTTPS: implement Public Key Pinning (HPKP) for Wikimedia domains - https://phabricator.wikimedia.org/T92002#1385234 (10konklone) Are you considering doing a phase where you do the `Public-Key-Pins-Report-Only` header first, to see what the likely issues would be? [18:47:17] (03CR) 10Alexandros Kosiaris: "Let's do the file_line to keep consistency. I 'll amend the augeas one in ganglia web as well. Using file_line back then did not even cros" [puppet] - 10https://gerrit.wikimedia.org/r/217724 (https://phabricator.wikimedia.org/T102092) (owner: 10Filippo Giunchedi) [20:13:29] 10Ops-Access-Requests, 6operations, 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1385353 (10hashar) >>! In T103192#1385135, @Krenair wrote: > I had thought these requests would require production access as well? W... [20:22:47] PROBLEM - YARN NodeManager Node-State on analytics1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:24:31] get yur knitting needles [20:29:46] RECOVERY - YARN NodeManager Node-State on analytics1016 is OK YARN NodeManager analytics1016.eqiad.wmnet:8041 Node-State: RUNNING [21:05:37] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [21:09:17] PROBLEM - puppet last run on cp4004 is CRITICAL puppet fail [21:14:08] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60552 bytes in 0.293 second response time [21:15:52] 6operations, 10RESTBase, 7Monitoring, 5Patch-For-Review: Detailed cassandra monitoring: metrics and dashboards done, need to set up alerts - https://phabricator.wikimedia.org/T78514#1385425 (10GWicke) Metrics have been less than reliable recently. We have observed several instances of metric reporting stop... [21:19:37] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [21:22:27] PROBLEM - YARN NodeManager Node-State on analytics1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [21:24:06] RECOVERY - YARN NodeManager Node-State on analytics1016 is OK YARN NodeManager analytics1016.eqiad.wmnet:8041 Node-State: RUNNING [21:26:57] RECOVERY - puppet last run on cp4004 is OK Puppet is currently enabled, last run 53 seconds ago with 0 failures [21:42:26] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60556 bytes in 0.309 second response time [21:47:47] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [21:48:20] I wonder what's up with gitblit [22:03:27] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60577 bytes in 0.185 second response time [22:08:09] 6operations, 10Traffic, 7HTTPS: implement Public Key Pinning (HPKP) for Wikimedia domains - https://phabricator.wikimedia.org/T92002#1101271 (10BBlack) It might be a good idea for the initial deploy for a day or two, yeah. [22:08:56] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [22:19:16] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60556 bytes in 5.071 second response time [22:24:46] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [22:24:46] bblack: awake by a chance ? [22:24:56] PROBLEM - Host ms-be1002 is DOWN: PING CRITICAL - Packet loss = 100% [22:26:35] 6operations, 6Security-Team: Production cluster can't access labs cluster - https://phabricator.wikimedia.org/T95714#1385467 (10Matanya) @yuvipanda can we work together to make this happen ? [22:48:13] matanya: briefly [22:48:29] bblack: mind kicking gitblit ? [22:48:45] (not critical) [22:48:48] is it just some known normal thing? restart some service when it starts complaining? [22:49:09] chad does that once in a while, but it can wait [22:49:16] looking at it [22:49:58] thanks [22:50:07] !log restarted gitblit java service on antimony [22:50:12] Logged the message, Master [22:50:21] the other question i had was regards https://phabricator.wikimedia.org/T95714 [22:51:01] anything else needed apart from a labs project limited to trusted users from your point of view ? [22:51:59] I don't know, it's not my area and I'm probably not even aware of whatever intricate concerns there are there [22:52:40] my gut feeling on browsing those tickets, though, is that: (a) it does make general sense for the prod clusters to treat labs like the rest of the internet. if there are things in prod that should be able to access random sites, I don't see why labs wouldn't be among them. [22:53:22] but (b) for the specific case of wgCopyUploadsDomains, it probably does not make sense to just blanket-enable wmflabs.org. That's a somewhat restrictive list, and *.wmflabs.org isn't very restricted at all. [22:54:08] thanks, those are good points [22:54:16] I'd think you'd want specific projects to meet standards, make a request, and be enabled one by one, like external sites do. [22:54:50] i'll wish you a good weekend at this point, and stop bugging you, thank you very much for your help [22:54:55] np [23:01:27] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60556 bytes in 0.180 second response time [23:18:27] PROBLEM - puppet last run on mw2129 is CRITICAL puppet fail [23:37:46] RECOVERY - puppet last run on mw2129 is OK Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:40:27] (03PS1) 10Gage: puppet.conf: remove obsolete ca_md setting [puppet] - 10https://gerrit.wikimedia.org/r/219600