[00:07:08] RECOVERY - citoid endpoints health on sca1001 is OK: All endpoints are healthy [00:10:56] (03PS11) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [00:12:48] PROBLEM - citoid endpoints health on sca1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:43:46] YuviPanda: Not sure of what? I was at dinner [01:39:49] PROBLEM - puppet last run on mw2060 is CRITICAL puppet fail [01:54:00] Coren|MX: yurik was asking about maps [01:56:58] PROBLEM - LVS HTTPS IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out [01:58:49] RECOVERY - LVS HTTPS IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 18416 bytes in 1.021 second response time [02:03:05] !log LocalisationUpdate failed (1.26wmf14) at 2015-07-19 02:03:05+00:00 [02:03:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:07:15] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 19 02:07:15 UTC 2015 (duration 7m 14s) [02:07:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:08:18] RECOVERY - puppet last run on mw2060 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [02:23:55] !log l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 07m 04s) [02:24:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:27:35] !log LocalisationUpdate completed (1.26wmf14) at 2015-07-19 02:27:35+00:00 [02:27:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:47:44] (03CR) 10Matanya: "@glaisher, this namespace is empty, so no worries." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225515 (https://phabricator.wikimedia.org/T106188) (owner: 10Matanya) [03:01:32] (03PS1) 10Ori.livneh: Undeploy VectorBeta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225668 [03:35:49] PROBLEM - puppet last run on mw1252 is CRITICAL Puppet has 1 failures [03:36:08] PROBLEM - puppet last run on mw1212 is CRITICAL Puppet has 1 failures [03:36:09] PROBLEM - puppet last run on mw1216 is CRITICAL Puppet has 1 failures [03:43:00] PROBLEM - puppet last run on mw2149 is CRITICAL puppet fail [04:02:20] RECOVERY - puppet last run on mw1252 is OK Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:02:39] RECOVERY - puppet last run on mw1212 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:02:39] RECOVERY - puppet last run on mw1216 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:06:09] PROBLEM - puppet last run on mw2038 is CRITICAL Puppet has 1 failures [04:11:29] RECOVERY - puppet last run on mw2149 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:32:38] RECOVERY - puppet last run on mw2038 is OK Puppet is currently enabled, last run 57 seconds ago with 0 failures [04:37:06] 6operations, 10Labs-Vagrant: Backport Vagrant 1.7+ from Debian experimental to our Trusty apt repo - https://phabricator.wikimedia.org/T93153#1463046 (10bd808) @akosiaris and I talked over difficulty that backporting and continuing to maintain all of the gem packages would be and he grudgingly agreed that we s... [05:10:10] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 19 05:10:10 UTC 2015 (duration 10m 9s) [05:10:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [05:11:26] (03CR) 10Glaisher: Fix fdcwiki's wgMetaNamespace to not be Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225515 (https://phabricator.wikimedia.org/T106188) (owner: 10Matanya) [05:19:51] (03PS2) 10Alex Monk: Set up Norwegian language aliases for remaining sites [puppet] - 10https://gerrit.wikimedia.org/r/225646 (https://phabricator.wikimedia.org/T86924) [05:24:37] 6operations, 10Wikimedia-Apache-configuration, 5Patch-For-Review: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1463085 (10Glaisher) [05:29:48] PROBLEM - puppet last run on sca1001 is CRITICAL Puppet has 23 failures [05:33:17] (03PS2) 10Alex Monk: Add basic contact form for stewards [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225509 (https://phabricator.wikimedia.org/T98625) [05:37:25] (03PS4) 10Alex Monk: Enable IP user page creation on fawiki's Draft ns [mediawiki-config] - 10https://gerrit.wikimedia.org/r/223497 (https://phabricator.wikimedia.org/T105118) (owner: 10Ebrahim) [06:31:29] PROBLEM - puppet last run on db1067 is CRITICAL Puppet has 1 failures [06:31:49] PROBLEM - puppet last run on mw2126 is CRITICAL Puppet has 1 failures [06:31:49] PROBLEM - puppet last run on mc2015 is CRITICAL Puppet has 1 failures [06:32:08] PROBLEM - puppet last run on ms-be1010 is CRITICAL Puppet has 1 failures [06:32:29] PROBLEM - puppet last run on mw2158 is CRITICAL Puppet has 1 failures [06:32:29] PROBLEM - puppet last run on db2055 is CRITICAL Puppet has 1 failures [06:32:30] PROBLEM - puppet last run on wtp2017 is CRITICAL Puppet has 1 failures [06:32:30] PROBLEM - puppet last run on mw2207 is CRITICAL Puppet has 1 failures [06:32:38] PROBLEM - puppet last run on mw2073 is CRITICAL Puppet has 1 failures [06:32:38] PROBLEM - puppet last run on mw2018 is CRITICAL Puppet has 1 failures [06:32:49] PROBLEM - puppet last run on mw2050 is CRITICAL Puppet has 1 failures [06:33:00] PROBLEM - puppet last run on mw2129 is CRITICAL Puppet has 1 failures [06:33:09] PROBLEM - puppet last run on mw1158 is CRITICAL Puppet has 1 failures [06:33:38] PROBLEM - puppet last run on mw1135 is CRITICAL Puppet has 1 failures [06:33:48] PROBLEM - puppet last run on mw2045 is CRITICAL Puppet has 1 failures [06:56:09] RECOVERY - puppet last run on db1067 is OK Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:56:39] RECOVERY - puppet last run on ms-be1010 is OK Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:57:09] RECOVERY - puppet last run on mw2158 is OK Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:57:18] RECOVERY - puppet last run on db2055 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:18] RECOVERY - puppet last run on wtp2017 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:19] RECOVERY - puppet last run on mw2207 is OK Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:57:19] RECOVERY - puppet last run on mw2073 is OK Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:57:19] RECOVERY - puppet last run on mw2018 is OK Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:57:29] RECOVERY - puppet last run on mw2050 is OK Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:57:40] RECOVERY - puppet last run on mw2129 is OK Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:57:48] RECOVERY - puppet last run on mw1158 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:18] RECOVERY - puppet last run on mw1135 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:29] RECOVERY - puppet last run on mw2126 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:29] RECOVERY - puppet last run on mw2045 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:29] RECOVERY - puppet last run on mc2015 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:58] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:49:06] (03CR) 10John Vandenberg: [C: 031] Set up Norwegian language aliases for remaining sites [puppet] - 10https://gerrit.wikimedia.org/r/225646 (https://phabricator.wikimedia.org/T86924) (owner: 10Alex Monk) [08:00:59] PROBLEM - puppet last run on sca1001 is CRITICAL Puppet has 36 failures [09:04:49] (03PS12) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [09:25:39] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 18 seconds ago with 0 failures [10:04:39] PROBLEM - puppet last run on wtp2003 is CRITICAL Puppet has 1 failures [10:29:49] PROBLEM - puppet last run on sca1001 is CRITICAL puppet fail [10:31:08] RECOVERY - puppet last run on wtp2003 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [11:10:38] PROBLEM - Hadoop NodeManager on analytics1037 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [11:17:48] RECOVERY - citoid endpoints health on sca1001 is OK: All endpoints are healthy [11:17:49] RECOVERY - citoid endpoints health on sca1002 is OK: All endpoints are healthy [11:23:19] PROBLEM - citoid endpoints health on sca1001 is CRITICAL: /api is CRITICAL: Test bad PMCID returned the unexpected status 200 (expecting: 404) [11:23:19] PROBLEM - citoid endpoints health on sca1002 is CRITICAL: /api is CRITICAL: Test bad PMCID returned the unexpected status 200 (expecting: 404) [11:24:59] PROBLEM - Hadoop NodeManager on analytics1034 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [11:30:49] RECOVERY - Hadoop NodeManager on analytics1034 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [11:31:28] RECOVERY - Hadoop NodeManager on analytics1037 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [11:57:42] (03CR) 10Hashar: "T92756 got closed and hence labs/private is auto rebased by puppetmaster::gitsync :)" [puppet] - 10https://gerrit.wikimedia.org/r/224213 (owner: 10BBlack) [12:27:00] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:01:19] PROBLEM - puppet last run on sca1001 is CRITICAL Puppet has 43 failures [13:34:41] (03PS6) 10Thcipriani: Add service deploy via scap [tools/scap] - 10https://gerrit.wikimedia.org/r/224374 [14:17:19] PROBLEM - puppet last run on cp3018 is CRITICAL puppet fail [14:27:58] akosiaris: thanks for the vagrant deb stuff [14:34:05] (03CR) 10Dzahn: "i was about to merge this, then noticed we have:" [puppet] - 10https://gerrit.wikimedia.org/r/224588 (https://phabricator.wikimedia.org/T104992) (owner: 10Muehlenhoff) [14:34:34] (03PS2) 10Dzahn: statistics: don't let Apache listen on 443 anymore [puppet] - 10https://gerrit.wikimedia.org/r/224642 [14:36:06] (03CR) 10Dzahn: [C: 032] "none of the sites-enabled has 443 anymore" [puppet] - 10https://gerrit.wikimedia.org/r/224642 (owner: 10Dzahn) [14:40:15] (03PS2) 10Alexandros Kosiaris: maps: LVS IP for kartotherian, tilerator and maps [dns] - 10https://gerrit.wikimedia.org/r/225640 (https://phabricator.wikimedia.org/T105074) [14:42:20] (03PS1) 10Alexandros Kosiaris: new_wmf_service.py: Remove extraneous debug statement [puppet] - 10https://gerrit.wikimedia.org/r/225692 [14:42:58] (03CR) 10Alexandros Kosiaris: [C: 032] maps: LVS IP for kartotherian, tilerator and maps [dns] - 10https://gerrit.wikimedia.org/r/225640 (https://phabricator.wikimedia.org/T105074) (owner: 10Alexandros Kosiaris) [14:44:44] (03CR) 10Dzahn: "replaced file contents with a "Listen 80", restarted Apache" [puppet] - 10https://gerrit.wikimedia.org/r/224642 (owner: 10Dzahn) [14:45:39] RECOVERY - puppet last run on cp3018 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [14:47:24] (03CR) 10Ori.livneh: [C: 031] "Nice :) Thanks." [puppet] - 10https://gerrit.wikimedia.org/r/225499 (owner: 10Andrew Bogott) [14:58:29] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [15:01:21] (03PS1) 10Ori.livneh: disable joal's SSH key [puppet] - 10https://gerrit.wikimedia.org/r/225695 [15:01:37] (03CR) 10Ori.livneh: [C: 032 V: 032] disable joal's SSH key [puppet] - 10https://gerrit.wikimedia.org/r/225695 (owner: 10Ori.livneh) [15:01:59] ori: someones quick :) [15:02:27] his laptop was stolen :( [15:02:44] yeah I saw the email :/ [15:03:40] I was about to ask them to aggressively phone or irc than email but you beat me to it :) [15:13:28] (03PS1) 10Alexandros Kosiaris: Introduce kartotherian service puppet module and role [puppet] - 10https://gerrit.wikimedia.org/r/225696 [15:16:05] (03PS5) 10Yuvipanda: cold-migrate: A few style updates + some docs [puppet] - 10https://gerrit.wikimedia.org/r/225499 (owner: 10Andrew Bogott) [15:16:18] (03CR) 10Yuvipanda: [C: 032 V: 032] cold-migrate: A few style updates + some docs [puppet] - 10https://gerrit.wikimedia.org/r/225499 (owner: 10Andrew Bogott) [15:26:26] (03PS1) 10Dzahn: planet: update Apache config for 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/225700 [15:27:04] (03PS2) 10Dzahn: planet: update Apache config for 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/225700 [15:29:08] (03PS1) 10Alexandros Kosiaris: Assign role::kartotherian to maps role [puppet] - 10https://gerrit.wikimedia.org/r/225701 [15:31:28] (03CR) 10Dzahn: [C: 032] planet: update Apache config for 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/225700 (owner: 10Dzahn) [15:38:04] (03PS1) 10MaxSem: role::maps::master: Import waterlines on init and then weekly [puppet] - 10https://gerrit.wikimedia.org/r/225702 [15:38:33] (03PS2) 10MaxSem: role::maps::master: Import waterlines on init and then weekly [puppet] - 10https://gerrit.wikimedia.org/r/225702 [15:39:31] (03PS2) 10Alexandros Kosiaris: Assign role::kartotherian to maps role [puppet] - 10https://gerrit.wikimedia.org/r/225701 [15:39:33] (03PS1) 10Alexandros Kosiaris: Setup LVS for kartotherian service on maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225703 [15:42:31] (03PS3) 10Alexandros Kosiaris: Assign role::kartotherian to maps role [puppet] - 10https://gerrit.wikimedia.org/r/225701 [15:42:33] (03PS2) 10Alexandros Kosiaris: Setup LVS for kartotherian service on maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225703 [15:42:35] (03PS2) 10Alexandros Kosiaris: Introduce kartotherian service puppet module and role [puppet] - 10https://gerrit.wikimedia.org/r/225696 [15:48:55] (03PS3) 10Alexandros Kosiaris: Introduce kartotherian service puppet module and role [puppet] - 10https://gerrit.wikimedia.org/r/225696 [15:49:04] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Introduce kartotherian service puppet module and role [puppet] - 10https://gerrit.wikimedia.org/r/225696 (owner: 10Alexandros Kosiaris) [15:52:53] (03PS4) 10Alexandros Kosiaris: Assign role::kartotherian to maps role [puppet] - 10https://gerrit.wikimedia.org/r/225701 [15:52:56] (03PS3) 10Alexandros Kosiaris: Setup LVS for kartotherian service on maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225703 [15:53:45] (03PS5) 10Alexandros Kosiaris: Assign role::kartotherian to maps role [puppet] - 10https://gerrit.wikimedia.org/r/225701 [15:53:59] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Assign role::kartotherian to maps role [puppet] - 10https://gerrit.wikimedia.org/r/225701 (owner: 10Alexandros Kosiaris) [16:00:19] PROBLEM - puppet last run on maps-test2002 is CRITICAL Puppet has 1 failures [16:03:22] (03PS1) 10Alex Monk: Sync arbcom_enwiki logo with their local Vector CSS per request [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225705 [16:04:01] PROBLEM - puppet last run on maps-test2004 is CRITICAL puppet fail [16:04:31] PROBLEM - puppet last run on maps-test2001 is CRITICAL Puppet has 1 failures [16:04:43] PROBLEM - puppet last run on maps-test2003 is CRITICAL puppet fail [16:06:02] PROBLEM - kartotherian on maps-test2001 is CRITICAL: Connection refused [16:06:24] (03PS1) 10Alexandros Kosiaris: kartotherian/deploy: HTTPS, not HTTP [puppet] - 10https://gerrit.wikimedia.org/r/225707 [16:06:56] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] kartotherian/deploy: HTTPS, not HTTP [puppet] - 10https://gerrit.wikimedia.org/r/225707 (owner: 10Alexandros Kosiaris) [16:07:52] PROBLEM - kartotherian on maps-test2002 is CRITICAL: Connection refused [16:11:41] (03PS2) 10Alex Monk: Sync arbcom_enwiki logo with their local Vector CSS per request [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225705 [16:11:53] RECOVERY - puppet last run on maps-test2002 is OK Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:15:51] RECOVERY - puppet last run on maps-test2001 is OK Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:23:41] RECOVERY - puppet last run on maps-test2003 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:32] RECOVERY - puppet last run on maps-test2004 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:48] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.69% of data above the critical threshold [500.0] [16:35:09] PROBLEM - kartotherian on maps-test2004 is CRITICAL: Connection refused [16:36:08] PROBLEM - kartotherian on maps-test2003 is CRITICAL: Connection refused [16:39:32] (03PS2) 10Alexandros Kosiaris: new_wmf_service.py: Remove extraneous debug statement [puppet] - 10https://gerrit.wikimedia.org/r/225692 [16:39:38] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] new_wmf_service.py: Remove extraneous debug statement [puppet] - 10https://gerrit.wikimedia.org/r/225692 (owner: 10Alexandros Kosiaris) [16:45:08] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [16:51:40] (03PS4) 10Alexandros Kosiaris: Setup LVS for kartotherian service on maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225703 [16:51:42] (03PS1) 10Alexandros Kosiaris: maps: Setup LVS for public entrypoint [puppet] - 10https://gerrit.wikimedia.org/r/225716 [16:56:37] 6operations, 10Wikimedia-Site-requests: Extension RSS fails to connect to feeds - https://phabricator.wikimedia.org/T90513#1463739 (10Krenair) [16:57:14] 6operations, 10Wikimedia-Site-requests: Extension RSS fails to connect to feeds - https://phabricator.wikimedia.org/T90513#1463741 (10Krenair) I found the problem - this is in our config: ```lang=php if ( $wmgUseRSSExtension ) { include( "$IP/extensions/RSS/RSS.php" ); // Only enable the proxy on foundationw... [16:58:29] 6operations, 10Wikimedia-Site-requests: Extension RSS fails to connect to feeds - https://phabricator.wikimedia.org/T90513#1463753 (10Krenair) ```b9084a00 (Kunal Mehta 2014-09-23 18:51:01 -0700 992) // Only enable the proxy on foundationwiki since the blog b9084a00 (Kunal Mehta 2014-09-23 18:51:01 -0700 99... [17:00:02] 6operations, 10Wikimedia-Site-requests: Extension RSS fails to connect to feeds - https://phabricator.wikimedia.org/T90513#1463759 (10Krenair) Caused by this commit: https://gerrit.wikimedia.org/r/#/c/162532/2 [17:02:13] (03CR) 10LFaraone: [C: 031] Sync arbcom_enwiki logo with their local Vector CSS per request [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225705 (owner: 10Alex Monk) [17:05:08] (03PS1) 10Alex Monk: Revert "Only use the RSS proxy on foundationwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225717 (https://phabricator.wikimedia.org/T90513) [17:05:11] (03CR) 10jenkins-bot: [V: 04-1] Revert "Only use the RSS proxy on foundationwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225717 (https://phabricator.wikimedia.org/T90513) (owner: 10Alex Monk) [17:11:18] PROBLEM - Hadoop NodeManager on analytics1035 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:11:32] (03CR) 10Alex Monk: [C: 032] Sync arbcom_enwiki logo with their local Vector CSS per request [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225705 (owner: 10Alex Monk) [17:11:38] (03Merged) 10jenkins-bot: Sync arbcom_enwiki logo with their local Vector CSS per request [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225705 (owner: 10Alex Monk) [17:14:16] !log krenair Synchronized w/static/images/project-logos/arbcom_enwiki.png: https://gerrit.wikimedia.org/r/#/c/225705/ (duration: 00m 12s) [17:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:15:08] RECOVERY - Hadoop NodeManager on analytics1035 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:20:57] RECOVERY - kartotherian on maps-test2001 is OK: HTTP OK: HTTP/1.1 200 OK - 486 bytes in 1.308 second response time [17:21:05] !log krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/225705/ (duration: 00m 12s) [17:21:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:24:42] well that didn't work. [17:25:40] (03PS1) 10Alex Monk: Fix the size of that last logo change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225718 [17:26:39] (03CR) 10Alex Monk: "Source is https://upload.wikimedia.org/wikipedia/commons/thumb/0/0e/Scale_of_justice_2.svg/204px-Scale_of_justice_2.svg.png by the way." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225705 (owner: 10Alex Monk) [17:26:48] (03CR) 10Alex Monk: [C: 032] Fix the size of that last logo change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225718 (owner: 10Alex Monk) [17:26:54] (03Merged) 10jenkins-bot: Fix the size of that last logo change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225718 (owner: 10Alex Monk) [17:27:53] !log krenair Synchronized w/static/images/project-logos/arbcom_enwiki.png: https://gerrit.wikimedia.org/r/#/c/225718/ (duration: 00m 12s) [17:27:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:31:27] PROBLEM - puppet last run on sca1001 is CRITICAL puppet fail [17:31:48] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL Anomaly detected: 10 data above and 2 below the confidence bounds [17:32:52] (03PS5) 10Alexandros Kosiaris: Setup LVS for kartotherian service on maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225703 [17:33:02] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Setup LVS for kartotherian service on maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225703 (owner: 10Alexandros Kosiaris) [17:34:20] (03PS2) 10Ori.livneh: Undeploy VectorBeta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225668 [17:34:27] (03CR) 10Ori.livneh: [C: 032] Undeploy VectorBeta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225668 (owner: 10Ori.livneh) [17:34:33] (03Merged) 10jenkins-bot: Undeploy VectorBeta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225668 (owner: 10Ori.livneh) [17:34:38] ori: nice [17:37:45] !log ori Synchronized wmf-config: Ib508a440: Undeploy VectorBeta (Task: T87489) (duration: 00m 13s) [17:37:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:38:52] (03PS21) 10BryanDavis: Add role::mediawiki_vagrant_lxc [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) [17:40:03] (03PS1) 10Alexandros Kosiaris: kartotherian: add conftool LVS stanza [puppet] - 10https://gerrit.wikimedia.org/r/225721 [17:40:07] PROBLEM - puppet last run on lvs2006 is CRITICAL puppet fail [17:40:19] PROBLEM - puppet last run on lvs2003 is CRITICAL puppet fail [17:40:20] ^ known, ignore [17:40:40] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] kartotherian: add conftool LVS stanza [puppet] - 10https://gerrit.wikimedia.org/r/225721 (owner: 10Alexandros Kosiaris) [17:43:48] RECOVERY - puppet last run on lvs2006 is OK Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:44:08] RECOVERY - puppet last run on lvs2003 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [17:47:58] legoktm: https://gerrit.wikimedia.org/r/#/c/225669/ [17:49:58] (03PS13) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [17:50:44] (03PS14) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [17:52:54] ori: do you like https://gerrit.wikimedia.org/r/#/c/224129/ [17:53:04] 6operations, 10Labs-Vagrant, 10Wikimania-Hackathon-2015: Backport Vagrant 1.7+ from Debian experimental to our Trusty apt repo - https://phabricator.wikimedia.org/T93153#1463879 (10bd808) [17:57:56] (03PS1) 10Alex Monk: Stop meta bureaucrats from removing sysop or bureaucrat [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225722 (https://phabricator.wikimedia.org/T106291) [17:57:57] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [17:58:48] (03CR) 10BryanDavis: "It's alive! https://vagrant-lxc-demo.wmflabs.org/wiki/Main_Page" [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) (owner: 10BryanDavis) [17:59:16] akosiaris, YuviPanda: I got it working! https://vagrant-lxc-demo.wmflabs.org/wiki/Main_Page [17:59:17] bd808: nice! [18:00:37] very cool [18:01:18] (03PS1) 10Alexandros Kosiaris: maps: Add conftool data, maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225723 [18:01:21] bd808: yay! [18:01:49] (03PS2) 10Alex Monk: Revert "Only use the RSS proxy on foundationwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225717 (https://phabricator.wikimedia.org/T90513) [18:03:16] mutante: yeah, lgtm [18:03:23] (03PS3) 10Ori.livneh: grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:03:29] (03CR) 10Ori.livneh: [C: 031] grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:03:59] (03CR) 10Dzahn: [C: 032] grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:04:15] (03CR) 10Alexandros Kosiaris: [C: 032] maps: Add conftool data, maps cluster [puppet] - 10https://gerrit.wikimedia.org/r/225723 (owner: 10Alexandros Kosiaris) [18:04:48] (03CR) 10Krinkle: "Hold on. This needs to be changed to /db/home. db/grafana is a problem because it is prone to be overwritten. In fact that happened and th" [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:04:59] (03PS4) 10Dzahn: grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:05:44] (03CR) 10Dzahn: "ok, not really merged yet because another rebase was needed" [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:05:46] (03CR) 10Yuvipanda: Add role::mediawiki_vagrant_lxc (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) (owner: 10BryanDavis) [18:05:55] bd808: looks good generally, minor nits / questions [18:08:51] (03PS1) 10Alexandros Kosiaris: Add the maps cluster to hieradata [puppet] - 10https://gerrit.wikimedia.org/r/225724 [18:09:52] (03CR) 10Alexandros Kosiaris: [C: 032] Add the maps cluster to hieradata [puppet] - 10https://gerrit.wikimedia.org/r/225724 (owner: 10Alexandros Kosiaris) [18:10:12] (03CR) 10Dzahn: "talked briefly to Andrew. this mount is empty and most likely not used. we can go ahead" [puppet] - 10https://gerrit.wikimedia.org/r/224588 (https://phabricator.wikimedia.org/T104992) (owner: 10Muehlenhoff) [18:12:42] (03PS2) 10Dzahn: Enable packet filter for stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/224588 (https://phabricator.wikimedia.org/T104992) (owner: 10Muehlenhoff) [18:13:49] (03CR) 10Dzahn: [C: 032] Enable packet filter for stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/224588 (https://phabricator.wikimedia.org/T104992) (owner: 10Muehlenhoff) [18:17:29] (03PS22) 10BryanDavis: Add role::mediawiki_vagrant_lxc [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) [18:17:40] (03CR) 10BryanDavis: Add role::mediawiki_vagrant_lxc (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) (owner: 10BryanDavis) [18:18:16] (03CR) 10Dzahn: "applied. http, rsync and the basics allowed. sites stats.wm, datasets.wm, metrics.wm still up as normal" [puppet] - 10https://gerrit.wikimedia.org/r/224588 (https://phabricator.wikimedia.org/T104992) (owner: 10Muehlenhoff) [18:18:47] !log ori Synchronized php-1.26wmf14/includes/site/SiteSQLStore.php: I0e5f2d3b2: Use CACHE_ACCEL for SiteLists if on HHVM (duration: 00m 12s) [18:18:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:21:53] Anyone deploying? [18:22:32] (03CR) 10Dzahn: Remove several dead domains from redirects (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/225041 (https://phabricator.wikimedia.org/T105981) (owner: 10Glaisher) [18:22:54] (03PS1) 10Yuvipanda: ores: Disable nginx cache temporarily [puppet] - 10https://gerrit.wikimedia.org/r/225728 [18:23:10] (03PS2) 10Yuvipanda: ores: Disable nginx cache temporarily [puppet] - 10https://gerrit.wikimedia.org/r/225728 [18:23:17] (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Disable nginx cache temporarily [puppet] - 10https://gerrit.wikimedia.org/r/225728 (owner: 10Yuvipanda) [18:25:22] (03CR) 10Hoo man: [C: 032] Enable IP user page creation on fawiki's Draft ns [mediawiki-config] - 10https://gerrit.wikimedia.org/r/223497 (https://phabricator.wikimedia.org/T105118) (owner: 10Ebrahim) [18:25:48] (03Merged) 10jenkins-bot: Enable IP user page creation on fawiki's Draft ns [mediawiki-config] - 10https://gerrit.wikimedia.org/r/223497 (https://phabricator.wikimedia.org/T105118) (owner: 10Ebrahim) [18:26:25] RECOVERY - kartotherian on maps-test2004 is OK: HTTP OK: HTTP/1.1 200 OK - 486 bytes in 0.101 second response time [18:26:44] RECOVERY - kartotherian on maps-test2003 is OK: HTTP OK: HTTP/1.1 200 OK - 486 bytes in 0.099 second response time [18:26:44] RECOVERY - kartotherian on maps-test2002 is OK: HTTP OK: HTTP/1.1 200 OK - 486 bytes in 0.101 second response time [18:29:01] !log hoo Synchronized wmf-config/CommonSettings.php: Enable IP user page creation on fawiki's Draft ns (duration: 00m 11s) [18:29:05] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected [18:29:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:37:05] PROBLEM - Check correctness of the icinga configuration on neon is CRITICAL: Icinga configuration contains errors [18:39:49] ^ Error: Could not find any hostgroup matching 'maps_codfw' [18:43:21] akosiaris: icinga misses a host group for maps_codfw [18:43:39] puppet_hostgroups.cfg doesnt have the groups anymore though.. hmm [18:45:14] mutante: yes I know. I am fixing it. thanks [18:45:46] akosiaris: ok, cool. should be @monitoring::group in the role i suppose [18:49:44] (03PS1) 10Alexandros Kosiaris: Add icinga groups for maps boxes [puppet] - 10https://gerrit.wikimedia.org/r/225730 [18:50:49] (03CR) 10Dzahn: [C: 031] Add icinga groups for maps boxes [puppet] - 10https://gerrit.wikimedia.org/r/225730 (owner: 10Alexandros Kosiaris) [18:51:11] (03CR) 10Alexandros Kosiaris: [C: 032] Add icinga groups for maps boxes [puppet] - 10https://gerrit.wikimedia.org/r/225730 (owner: 10Alexandros Kosiaris) [18:51:21] citoid endpoints /api are 200 but monitoring expected 404 [18:52:35] 6operations, 6Commons: Please whitelist the domain "archive.org" for sideloading on Commons - https://phabricator.wikimedia.org/T106293#1463950 (1080686) 3NEW [18:54:03] 6operations, 6Commons: Please whitelist the domain "archive.org" for sideloading on Commons - https://phabricator.wikimedia.org/T106293#1463958 (10brion) We'd like this to upload some video files that were recorded for Wikimania... [18:54:25] 6operations, 6Commons, 6Release-Engineering: Please whitelist the domain "archive.org" for sideloading on Commons - https://phabricator.wikimedia.org/T106293#1463959 (10brion) [18:55:43] 6operations, 6Commons, 6Release-Engineering: Please whitelist the domain "archive.org" for sideloading on Commons - https://phabricator.wikimedia.org/T106293#1463962 (10Fuzheado) "It's not already?" = Probably should do it right away [18:58:06] 6operations, 6Commons, 6Release-Engineering: Please whitelist the domain "archive.org" for sideloading on Commons - https://phabricator.wikimedia.org/T106293#1463971 (10Krenair) Please read https://meta.wikimedia.org/wiki/Requesting_wiki_configuration_changes [18:58:34] RECOVERY - Check correctness of the icinga configuration on neon is OK: Icinga configuration is correct [18:59:31] Krenair: what bit specifically you want us to read? [18:59:47] brion, I wanted the creator to read it [19:00:13] operations do not handle our mediawiki config requests [19:00:16] Krenair: he's sitting next to me [19:00:29] who does? can i ping them on irc in a hurry? [19:00:36] I do. [19:00:42] Other people do sometimes. [19:00:42] :) [19:01:05] so is that a "yes I can merge your config patch as soon as it's ready" or a "....?" [19:01:25] I don't think we've really required community consensus for CopyUploadDomains additions before, but it's not an ops task [19:01:57] well that's already fixed [19:02:01] anything else we need to do? [19:02:08] is there a patch? [19:02:17] there is about to be [19:02:23] took me a few minutes to clone the repo [19:02:45] i can sync it [19:03:13] thx [19:04:00] Thanks ori. [19:04:02] * Krenair goes to get lunch [19:04:07] thanks Krenair & ori! [19:04:11] sorry about filing it incorrectly :D [19:04:15] (03PS1) 10Brion VIBBER: Whitelist *.archive.org for wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225731 (https://phabricator.wikimedia.org/T106293) [19:05:23] (03CR) 10Ori.livneh: [C: 032] Whitelist *.archive.org for wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225731 (https://phabricator.wikimedia.org/T106293) (owner: 10Brion VIBBER) [19:06:04] woo [19:06:26] !log ori Synchronized wmf-config/InitialiseSettings.php: I189d748a: Whitelist *.archive.org for wgCopyUploadsDomains (T106293) (duration: 00m 13s) [19:06:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:09:19] (03PS1) 10Ori.livneh: Follow-up for I189d748: whitelist 'archive.org' too [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225732 [19:09:30] (03CR) 10Ori.livneh: [C: 032] Follow-up for I189d748: whitelist 'archive.org' too [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225732 (owner: 10Ori.livneh) [19:09:35] (03Merged) 10jenkins-bot: Follow-up for I189d748: whitelist 'archive.org' too [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225732 (owner: 10Ori.livneh) [19:10:21] !log ori Synchronized wmf-config/InitialiseSettings.php: Ic0573f26: Follow-up for I189d748: whitelist 'archive.org' too (duration: 00m 12s) [19:15:45] PROBLEM - Restbase endpoints health on restbase1004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:17:34] RECOVERY - Restbase endpoints health on restbase1004 is OK: All endpoints are healthy [19:29:15] PROBLEM - puppet last run on sca1001 is CRITICAL Puppet has 62 failures [19:41:23] (03CR) 10Mjbmr: "so, we can upload from any url now?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225731 (https://phabricator.wikimedia.org/T106293) (owner: 10Brion VIBBER) [20:13:35] PROBLEM - Apache HTTP on mw1160 is CRITICAL - Socket timeout after 10 seconds [20:15:16] RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.086 second response time [20:21:55] (03PS1) 10Alex Monk: Try (again) to fix size of arbcom_enwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225822 [20:22:06] lfaraone, ^ [20:27:34] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [20:41:54] (03PS2) 10Alexandros Kosiaris: maps: Give rights to maps-admins [puppet] - 10https://gerrit.wikimedia.org/r/225566 [20:42:40] (03CR) 10jenkins-bot: [V: 04-1] maps: Give rights to maps-admins [puppet] - 10https://gerrit.wikimedia.org/r/225566 (owner: 10Alexandros Kosiaris) [20:45:24] (03PS3) 10Alexandros Kosiaris: maps: Give rights to maps-admins [puppet] - 10https://gerrit.wikimedia.org/r/225566 [20:47:52] (03CR) 10Alexandros Kosiaris: [C: 032] maps: Give rights to maps-admins [puppet] - 10https://gerrit.wikimedia.org/r/225566 (owner: 10Alexandros Kosiaris) [20:48:17] (03PS2) 10Alexandros Kosiaris: maps: Tune postgresql [puppet] - 10https://gerrit.wikimedia.org/r/225576 [20:48:37] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] maps: Tune postgresql [puppet] - 10https://gerrit.wikimedia.org/r/225576 (owner: 10Alexandros Kosiaris) [20:51:04] (03CR) 10Alex Monk: [C: 032] Try (again) to fix size of arbcom_enwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225822 (owner: 10Alex Monk) [20:51:10] (03Merged) 10jenkins-bot: Try (again) to fix size of arbcom_enwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225822 (owner: 10Alex Monk) [20:52:30] !log krenair Synchronized w/static/images/project-logos/arbcom_enwiki.png: https://gerrit.wikimedia.org/r/#/c/225822/ (duration: 00m 12s) [20:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:53:41] lfaraone, you can remove your css now [20:56:29] (03PS1) 10Alexandros Kosiaris: Fix typo introduced in 4efae00 [puppet] - 10https://gerrit.wikimedia.org/r/225824 [21:33:53] 6operations, 5Patch-For-Review: Ferm rules for abacist - https://phabricator.wikimedia.org/T104992#1464159 (10Dzahn) per Gerrit change: " role::abacist only requires access to redis on localhost" [21:34:00] 6operations, 5Patch-For-Review: Ferm rules for abacist - https://phabricator.wikimedia.org/T104992#1464160 (10Dzahn) 5Open>3Resolved [21:36:12] akosiaris, are you adding tmux? [21:40:16] (03PS3) 10Dzahn: Set up Norwegian language aliases for remaining sites [puppet] - 10https://gerrit.wikimedia.org/r/225646 (https://phabricator.wikimedia.org/T86924) (owner: 10Alex Monk) [21:42:16] (03CR) 10Dzahn: [C: 032] Set up Norwegian language aliases for remaining sites [puppet] - 10https://gerrit.wikimedia.org/r/225646 (https://phabricator.wikimedia.org/T86924) (owner: 10Alex Monk) [21:54:16] 6operations, 6Labs, 10Labs-Infrastructure: rename holmium to labdns1002 - https://phabricator.wikimedia.org/T106303#1464173 (10Dzahn) 3NEW [21:54:28] 6operations, 6Labs, 10Labs-Infrastructure: rename holmium to labdns1002 - https://phabricator.wikimedia.org/T106303#1464183 (10Dzahn) [21:54:31] 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1460393 (10Dzahn) [22:04:07] 6operations, 10Wikimedia-Apache-configuration, 5Patch-For-Review: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1464185 (10Dzahn) 5Open>3Resolved a:3Dzahn ``` [terbium:~] $ apache-fast-test nb.urls mw1033 testing 6 urls on 1 servers, totalling 6 requests spawning threads.. http://n... [22:04:48] 6operations, 10Wikimedia-Apache-configuration: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1464188 (10Dzahn) [22:06:49] 6operations, 10Wikimedia-Apache-configuration: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1464189 (10Dzahn) 5Resolved>3Open [22:08:58] eh, what "git review" uses SSL_v3 , but did not before [22:09:02] File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 70, in [22:09:09] ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD, [22:09:17] AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3' [22:10:10] 6operations, 10Wikimedia-Apache-configuration: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1464193 (10Dzahn) nb.wikinews.org -> no.wikinews.org is missing indeed [22:11:15] * mutante removes python 2.7 [22:16:54] 6operations, 10Wikimedia-Apache-configuration: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1464204 (10Krenair) Although https://nb.wikinews.org doesn't actually exist, it just redirects to incubator. [22:28:08] (03PS1) 10Gergő Tisza: [WIP] Version update: 7.4.3 -> 7.6.2 [software/sentry] - 10https://gerrit.wikimedia.org/r/225827 (https://phabricator.wikimedia.org/T105374) [22:34:23] 6operations, 10Wikimedia-Apache-configuration: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1464226 (10Dzahn) @Krenair yes, we should add that redirect as well [22:47:04] (03PS1) 10Mjbmr: Add South Azeri (azb) subdomain [dns] - 10https://gerrit.wikimedia.org/r/225830 (https://phabricator.wikimedia.org/T106305) [22:50:35] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 895.577154161 [22:53:46] (03PS1) 10Alex Monk: Also add redirect for nowikinews [puppet] - 10https://gerrit.wikimedia.org/r/225831 (https://phabricator.wikimedia.org/T86924) [22:53:52] (03CR) 10jenkins-bot: [V: 04-1] Also add redirect for nowikinews [puppet] - 10https://gerrit.wikimedia.org/r/225831 (https://phabricator.wikimedia.org/T86924) (owner: 10Alex Monk) [23:20:38] (03PS2) 10Gergő Tisza: [WIP] Version update: 7.4.3 -> 7.6.2 [software/sentry] - 10https://gerrit.wikimedia.org/r/225827 (https://phabricator.wikimedia.org/T105374) [23:21:13] (03PS3) 10Gergő Tisza: Version update: 7.4.3 -> 7.6.2 [software/sentry] - 10https://gerrit.wikimedia.org/r/225827 (https://phabricator.wikimedia.org/T105374) [23:25:36] (03PS4) 10Gergő Tisza: Version update: 7.4.3 -> 7.6.2 [software/sentry] - 10https://gerrit.wikimedia.org/r/225827 (https://phabricator.wikimedia.org/T105374) [23:26:18] (03PS1) 10Mjbmr: Configuration for South Azerbaijani Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225835 (https://phabricator.wikimedia.org/T106305) [23:37:55] (03PS5) 10Gergő Tisza: Version update: 7.4.3 -> 7.6.2 [software/sentry] - 10https://gerrit.wikimedia.org/r/225827 (https://phabricator.wikimedia.org/T105374) [23:38:04] (03PS1) 10GWicke: Don't killall $DAEMON [puppet] - 10https://gerrit.wikimedia.org/r/225836 [23:59:06] (03PS2) 10Dzahn: Also add redirect for nowikinews [puppet] - 10https://gerrit.wikimedia.org/r/225831 (https://phabricator.wikimedia.org/T86924) (owner: 10Alex Monk) [23:59:48] (03CR) 10Dzahn: [C: 032] Also add redirect for nowikinews [puppet] - 10https://gerrit.wikimedia.org/r/225831 (https://phabricator.wikimedia.org/T86924) (owner: 10Alex Monk)