[00:00:12] (03PS6) 10Dzahn: install fonts-unfonts-core, not just fonts-unfonts-extra [puppet] - 10https://gerrit.wikimedia.org/r/194828 (https://phabricator.wikimedia.org/T91685) [00:00:37] RECOVERY - check if wikidata.org dispatch lag is higher than 2 minutes on wikidata is OK: HTTP OK: HTTP/1.1 200 OK - 1449 bytes in 0.241 second response time [00:00:38] (03PS7) 10Dzahn: install fonts-unfonts-core, not just fonts-unfonts-extra [puppet] - 10https://gerrit.wikimedia.org/r/194828 (https://phabricator.wikimedia.org/T91685) [00:01:54] (03PS1) 10Hoo man: Fix $wgWBRepoSettings['localClientDatabases'] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194984 [00:02:48] greg-g: Now that we decided that we ignore the days... can I fix my previous change? :P [00:03:12] It didn't break anything, but it also doesn't work, and I failed to see that [00:04:05] hoo: don't tell anyone [00:04:53] 6operations: Decommission svn.wikimedia.org server (import SVN into Phabricator) - https://phabricator.wikimedia.org/T86655#1097277 (10Dzahn) now we only need to have consenus that the actual cloning with the svn protocol can be disabled [00:05:17] (03PS2) 10Hoo man: Fix $wgWBRepoSettings['localClientDatabases'] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194984 [00:05:47] (03CR) 10Hoo man: [C: 032] Fix $wgWBRepoSettings['localClientDatabases'] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194984 (owner: 10Hoo man) [00:05:52] (03Merged) 10jenkins-bot: Fix $wgWBRepoSettings['localClientDatabases'] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194984 (owner: 10Hoo man) [00:06:57] !log hoo Synchronized wmf-config/Wikibase.php: Turns out trim is actually needed... (duration: 00m 05s) [00:07:03] Logged the message, Master [00:07:51] 6operations, 10ops-eqiad: mw1062 needs a disk replacement - https://phabricator.wikimedia.org/T86542#1097279 (10Dzahn) help, does anyone else see which this host is not back in normal rotation like any other server? why is nothing showing up in the ganglia graph even though this host should be enabled [00:08:49] 6operations, 10ops-eqiad: mw1062 needs a disk replacement - https://phabricator.wikimedia.org/T86542#1097281 (10Dzahn) a:5Dzahn>3None [00:16:25] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Access to stat1003 for Niklas and Kartik - https://phabricator.wikimedia.org/T91625#1097301 (10Dzahn) a:3RobH looks all good, but due to the 3-day waiting rule for access requests it has to wait for merge until next week.. handing over to RobH who wil... [00:22:13] 10Ops-Access-Requests, 6operations: RESTBase deploy access and shell on Cassandra cluster for eevans - https://phabricator.wikimedia.org/T91134#1097313 (10Dzahn) adding @fgiunchedi because he worked on this on T89366 and @robh because he will be on Clinic Duty next week do you guys know which admin groups spe... [00:23:36] 10Ops-Access-Requests, 6operations: RESTBase deploy access and shell on Cassandra cluster for eevans - https://phabricator.wikimedia.org/T91134#1097314 (10Dzahn) @gwicke: seems we currenlty have: restbase-roots and cassandra-test-roots but not cassandra-roots. are we talking about one of these groups or do we... [00:27:56] 10Ops-Access-Requests, 6operations: RESTBase deploy access and shell on Cassandra cluster for eevans - https://phabricator.wikimedia.org/T91134#1097335 (10GWicke) @dzahn, restbase-roots and cassandra-test-roots are fine, that's what mobrovac & I have as well. Can reshuffle the groups some other time, there's n... [00:38:05] !log Set wb_changes_dispatch.chd_disabled = 1 for all closed wikis on wikidata [00:38:10] Logged the message, Master [00:46:32] !log depooled cp4014 in pybal [00:46:38] Logged the message, Master [00:48:57] PROBLEM - Host cp4014 is DOWN: PING CRITICAL - Packet loss = 100% [00:51:07] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [00:51:35] (03PS1) 10Dzahn: add eevans to restbase/cassandra roots and deployers [puppet] - 10https://gerrit.wikimedia.org/r/194987 (https://phabricator.wikimedia.org/T91134) [00:52:16] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 1 failures [00:52:16] (03CR) 10Dzahn: [C: 04-1] add eevans to restbase/cassandra roots and deployers [puppet] - 10https://gerrit.wikimedia.org/r/194987 (https://phabricator.wikimedia.org/T91134) (owner: 10Dzahn) [00:52:30] (03CR) 10jenkins-bot: [V: 04-1] add eevans to restbase/cassandra roots and deployers [puppet] - 10https://gerrit.wikimedia.org/r/194987 (https://phabricator.wikimedia.org/T91134) (owner: 10Dzahn) [00:53:54] 6operations: Decommission svn.wikimedia.org server (import SVN into Phabricator) - https://phabricator.wikimedia.org/T86655#1097401 (10demon) >>! In T86655#1097277, @Dzahn wrote: > now we only need to have consenus that the actual cloning with the svn protocol can be disabled -1 [00:55:00] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase deploy access and shell on Cassandra cluster for eevans - https://phabricator.wikimedia.org/T91134#1097413 (10Dzahn) @eevans Hi! We are preparing your shell access request. Please read and sign L3 and provide us with a SSH public key (but not th... [00:55:40] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase deploy access and shell on Cassandra cluster for eevans - https://phabricator.wikimedia.org/T91134#1097416 (10Dzahn) a:3RobH [01:03:17] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [01:09:59] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:10:37] RECOVERY - Host cp4014 is UP: PING OK - Packet loss = 0%, RTA = 78.08 ms [01:12:57] PROBLEM - Varnish HTTP upload-backend on cp4014 is CRITICAL: Connection refused [01:13:17] PROBLEM - RAID on cp4014 is CRITICAL: Connection refused by host [01:13:17] PROBLEM - DPKG on cp4014 is CRITICAL: Connection refused by host [01:13:26] PROBLEM - HTTPS on cp4014 is CRITICAL: Return code of 255 is out of bounds [01:13:37] PROBLEM - Varnish HTTP upload-frontend on cp4014 is CRITICAL: Connection refused [01:13:47] PROBLEM - Varnishkafka log producer on cp4014 is CRITICAL: Connection refused by host [01:13:47] PROBLEM - configured eth on cp4014 is CRITICAL: Connection refused by host [01:13:47] PROBLEM - salt-minion processes on cp4014 is CRITICAL: Connection refused by host [01:13:47] PROBLEM - dhclient process on cp4014 is CRITICAL: Connection refused by host [01:13:47] PROBLEM - Varnish HTCP daemon on cp4014 is CRITICAL: Connection refused by host [01:13:48] PROBLEM - Disk space on cp4014 is CRITICAL: Connection refused by host [01:13:48] PROBLEM - Varnish traffic logger on cp4014 is CRITICAL: Connection refused by host [01:13:49] PROBLEM - puppet last run on cp4014 is CRITICAL: Connection refused by host [01:13:57] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [01:14:11] 6operations, 10Datasets-General-or-Unknown, 6Services, 10hardware-requests: Hardware for HTML / zim dumps - https://phabricator.wikimedia.org/T91853#1097489 (10GWicke) 3NEW [01:14:36] 6operations, 10Datasets-General-or-Unknown, 6Services, 10hardware-requests: Hardware for HTML / zim dumps - https://phabricator.wikimedia.org/T91853#1097498 (10GWicke) [01:42:21] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [01:50:07] 6operations, 10Parsoid, 6Services: Lets consider upgrading our nodejs installs to iojs (once decent Debian packages are ready) - https://phabricator.wikimedia.org/T91855#1097544 (10GWicke) 3NEW [01:51:21] !log repooled cp4014 in pybal [01:51:28] Logged the message, Master [01:54:03] 6operations, 10Parsoid, 6Services: Lets consider upgrading our nodejs installs to iojs (once decent Debian packages are ready) - https://phabricator.wikimedia.org/T91855#1097554 (10GWicke) [02:03:55] !log l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 04s) [02:04:02] Logged the message, Master [02:05:03] !log LocalisationUpdate completed (1.25wmf19) at 2015-03-07 02:03:59+00:00 [02:05:08] Logged the message, Master [02:07:13] !log l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 01s) [02:07:18] Logged the message, Master [02:08:20] !log LocalisationUpdate completed (1.25wmf20) at 2015-03-07 02:07:16+00:00 [02:08:25] Logged the message, Master [02:17:55] (03PS1) 10BBlack: disable quickstack installation/update on jessie for now [puppet] - 10https://gerrit.wikimedia.org/r/195004 [02:18:57] (03CR) 10BBlack: [C: 032] disable quickstack installation/update on jessie for now [puppet] - 10https://gerrit.wikimedia.org/r/195004 (owner: 10BBlack) [02:21:21] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:40] !log manually finished global rename for Just.isabella on commonswiki [02:25:47] Logged the message, Master [02:29:40] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:44:25] PROBLEM - puppet last run on labstore1002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:48:45] RECOVERY - puppet last run on labstore1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:04:25] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: puppet fail [03:06:50] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Mar 7 03:05:47 UTC 2015 (duration 5m 46s) [03:07:00] Logged the message, Master [03:23:24] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:07:15] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 2 below the confidence bounds [05:13:52] (03PS1) 10Mjbmr: Enable NewUserMessage extension for fawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/195014 (https://phabricator.wikimedia.org/T91861) [05:23:53] 6operations, 7HTTPS: Replace SHA1 certificates with SHA256 - https://phabricator.wikimedia.org/T73156#1097690 (10Chmarkine) Chrome 41 has been released. Are there any plans for replacing all the remaining SHA1 certificates? [05:24:15] 6operations, 7HTTPS: Replace SHA1 certificates with SHA256 - https://phabricator.wikimedia.org/T73156#1097691 (10Chmarkine) [05:49:15] (03PS1) 10Mjbmr: Add previous project alias namespace for fawikibooks (was removed on T60655) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/195017 [06:02:06] (03CR) 10Krinkle: [C: 031] install fonts-unfonts-core, not just fonts-unfonts-extra [puppet] - 10https://gerrit.wikimedia.org/r/194828 (https://phabricator.wikimedia.org/T91685) (owner: 10Dzahn) [06:04:44] (03PS2) 10Krinkle: Add "composer test" command to lint files and run tests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/189148 (https://phabricator.wikimedia.org/T85947) (owner: 10Legoktm) [06:11:02] (03CR) 10Krinkle: "I assume the PrivateSettings exclude is because of the symlink bug, not because of paranoia as the file isn't in version control." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/189148 (https://phabricator.wikimedia.org/T85947) (owner: 10Legoktm) [06:11:34] (03CR) 10Krinkle: [C: 031] Add "composer test" command to lint files and run tests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/189148 (https://phabricator.wikimedia.org/T85947) (owner: 10Legoktm) [06:29:16] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:45] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:45] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:04] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:15] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 3 failures [06:30:15] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:24] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:25] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:16] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:15] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:45:25] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:45:36] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:45:55] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:45:56] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:55] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:47:05] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:47:06] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:51:04] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:59:25] (03PS1) 10KartikMistry: Syntax: Fixed yaml [puppet] - 10https://gerrit.wikimedia.org/r/195023 [07:14:25] RECOVERY - Disk space on fluorine is OK: DISK OK [07:44:35] (03CR) 10Tim Landscheidt: "Why not set this up in role::labs::instance? With the previous "if $::realm != 'labs'", the file resource would only be defined once." [puppet] - 10https://gerrit.wikimedia.org/r/194858 (https://phabricator.wikimedia.org/T63897) (owner: 10coren) [08:32:55] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet has 1 failures [08:50:55] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [09:36:17] (03CR) 10Cwek: [C: 031] Add Draft namespace on zhwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/193827 (https://phabricator.wikimedia.org/T91223) (owner: 10Gerrit Patch Uploader) [09:37:16] 6operations, 10Datasets-General-or-Unknown, 6Services, 10hardware-requests: Hardware for HTML / zim dumps - https://phabricator.wikimedia.org/T91853#1097830 (10Kelson) [09:37:17] 6operations, 10Datasets-General-or-Unknown: Mirror more Kiwix downloads directories - https://phabricator.wikimedia.org/T57503#1097829 (10Kelson) [09:53:45] PROBLEM - HHVM rendering on mw1119 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50943 bytes in 0.158 second response time [09:54:04] PROBLEM - Apache HTTP on mw1119 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50924 bytes in 0.062 second response time [10:13:27] !log started checkLocalUser.php and checkLocalNames.php scripts (CentralAuth) [10:13:35] Logged the message, Master [10:44:13] (03CR) 10Liuxinyu970226: "Shouldn't we also add VisualEditor support to it? @Jdforrester" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/193827 (https://phabricator.wikimedia.org/T91223) (owner: 10Gerrit Patch Uploader) [11:11:24] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [11:29:15] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [11:31:45] PROBLEM - dhclient process on rhenium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:31:45] PROBLEM - salt-minion processes on rhenium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:32:45] RECOVERY - dhclient process on rhenium is OK: PROCS OK: 0 processes with command name dhclient [11:32:45] RECOVERY - salt-minion processes on rhenium is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [12:55:54] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.071 second response time [12:59:15] PROBLEM - Apache HTTP on mw1119 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50924 bytes in 0.057 second response time [13:23:24] PROBLEM - check if wikidata.org dispatch lag is higher than 2 minutes on wikidata is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1454 bytes in 0.343 second response time [13:32:55] (03PS1) 10Hashar: contint: disable hhvm stacktraces / map [puppet] - 10https://gerrit.wikimedia.org/r/195035 (https://phabricator.wikimedia.org/T64788) [13:54:35] 6operations, 6MediaWiki-Core-Team, 6Multimedia, 6Parsoid-Team, and 3 others: Prepare Platform April 2015 quarterly review presentation - https://phabricator.wikimedia.org/T91803#1098059 (10Qgil) [14:07:36] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [14:21:05] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [14:35:39] (03CR) 10Krinkle: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/194828 (https://phabricator.wikimedia.org/T91685) (owner: 10Dzahn) [14:36:14] andrewbogott_afk: puppet runs every 20 minutes by cron in a puppet-run wrapper script [14:53:44] RECOVERY - check if wikidata.org dispatch lag is higher than 2 minutes on wikidata is OK: HTTP OK: HTTP/1.1 200 OK - 1445 bytes in 0.150 second response time [14:54:47] (03CR) 10Alexandros Kosiaris: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/194987 (https://phabricator.wikimedia.org/T91134) (owner: 10Dzahn) [15:01:05] (03CR) 10Alexandros Kosiaris: "jenkins says:" [puppet] - 10https://gerrit.wikimedia.org/r/194987 (https://phabricator.wikimedia.org/T91134) (owner: 10Dzahn) [15:07:35] (03PS9) 10Alexandros Kosiaris: Puppet module for the zotero service [puppet] - 10https://gerrit.wikimedia.org/r/194495 (https://phabricator.wikimedia.org/T89867) [15:11:54] (03PS1) 10Hoo man: Increase number of Wikidata dispatchers by 1 [puppet] - 10https://gerrit.wikimedia.org/r/195040 [15:12:33] akosiaris: ^ Any chance you can push that? We have ~350 edits/minute steadily right now and that seems to be on the edge for the current number of dispatchers [15:12:41] terbium has the reserves to run one more instance [15:15:51] hoo: need some oddly time-zoned help? [15:15:55] * YuviPanda looks [15:16:07] hoo: seems simple enough [15:16:15] ah [15:16:24] * YuviPanda steps away from gerrit, akosiaris’ got it [15:16:28] YuviPanda: hi :) The time isn't odd... at least here it's not [15:16:39] oh, right. [15:16:44] oh wow, it’s almost 9PM here. [15:16:48] * YuviPanda attempts to venture out for food [15:18:11] (03CR) 10Alexandros Kosiaris: [C: 032] Increase number of Wikidata dispatchers by 1 [puppet] - 10https://gerrit.wikimedia.org/r/195040 (owner: 10Hoo man) [15:19:33] hoo: merged... I am gonna keep my eye on it for a while [15:19:50] akosiaris: So will I, thanks :) [15:28:40] (03PS1) 10Alexandros Kosiaris: Include the zotero role in the sca role [puppet] - 10https://gerrit.wikimedia.org/r/195041 (https://phabricator.wikimedia.org/T89869) [15:29:28] 6operations, 10Citoid: Puppetize zotero - https://phabricator.wikimedia.org/T89867#1098147 (10akosiaris) Proxy works fine as well in labs. To be merged Monday [15:30:21] 6operations, 10Citoid: Assign hardware for the zotero service - https://phabricator.wikimedia.org/T89869#1098148 (10akosiaris) https://gerrit.wikimedia.org/r/195041 has the hardware assignment via the temporary sca role class [15:40:26] 6operations, 10Wikimedia-Mailing-lists: Let public archives be indexed and archived - https://phabricator.wikimedia.org/T90407#1098150 (10JohnLewis) There is actually a policy (or guideline or procedure, what you want to call it) that ops follow regarding removal of mailman archive content which is more or les... [15:54:29] 6operations, 10Citoid: Update the citoid/deploy branch to not contain zotero deploy - https://phabricator.wikimedia.org/T89872#1098160 (10QChris) >>! In T89872#1095559, @chasemp wrote: >>>! In T89872#1095399, @akosiaris wrote: >> Change is at https://gerrit.wikimedia.org/r/#/c/194548/ (isn't a bot supposed to... [16:34:51] 6operations: Create wikimania2016 wiki - https://phabricator.wikimedia.org/T85374#1098181 (10Glaisher) [16:50:25] (03CR) 10Jforrester: "Chinese variants support is far off, so this might not be worthwhile, but happy to do so if you want." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/193827 (https://phabricator.wikimedia.org/T91223) (owner: 10Gerrit Patch Uploader) [17:15:15] (03PS3) 10Nemo bis: Hide "prefershttps" preference on HSTS domains (ru): it has no effect [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194856 (https://phabricator.wikimedia.org/T91352) [17:15:29] (03PS4) 10Nemo bis: Hide "prefershttps" preference on HSTS domains (ru): it has no effect [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194856 (https://phabricator.wikimedia.org/T91352) [17:17:25] (03PS5) 10Nemo bis: Hide "prefershttps" preference on HSTS domains (ru): it has no effect [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194856 (https://phabricator.wikimedia.org/T91352) [19:49:25] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 14 data above and 9 below the confidence bounds [19:53:25] PROBLEM - puppet last run on es2007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:10:15] RECOVERY - puppet last run on es2007 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:22:45] (03PS1) 10Nemo bis: Just use "en" as language code for WMCA wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/195064 [20:34:05] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [20:36:35] (03CR) 10Chmarkine: [C: 031] Hide "prefershttps" preference on HSTS domains (ru): it has no effect [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194856 (https://phabricator.wikimedia.org/T91352) (owner: 10Nemo bis) [20:45:55] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: puppet fail [21:04:55] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [21:06:50] 6operations, 10Wikimedia-General-or-Unknown: Icinga has httpauth on (not accessible for public) - https://phabricator.wikimedia.org/T62112#1098490 (10scfc) IIRC, it could also be fixed by upgrading `neon` (?) to Trusty, and as those upgrades always seem imminent and I think they have more value than shepherdin... [21:49:16] 7Puppet, 6operations: Resource attributes are quoted inconsistently - https://phabricator.wikimedia.org/T91908#1098540 (10scfc) 3NEW [21:53:27] 7Puppet, 6operations: Resource attributes are quoted inconsistently - https://phabricator.wikimedia.org/T91908#1098551 (10Matanya) [[ http://docs.puppetlabs.com/guides/style_guide.html#quoting | upstream guide ]] is not forcing it, and when i wrote our style guide, i was for forcing it, for consistency reasons... [22:55:45] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 1 failures [23:13:35] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:14] RECOVERY - HHVM rendering on mw1119 is OK: HTTP OK: HTTP/1.1 200 OK - 68359 bytes in 0.147 second response time [23:53:34] PROBLEM - HHVM rendering on mw1119 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50943 bytes in 0.145 second response time