[01:24:43] <icinga-wm>	 PROBLEM - puppet last run on mw2046 is CRITICAL: CRITICAL: Puppet has 1 failures
[01:29:33] <icinga-wm>	 PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago
[01:53:33] <icinga-wm>	 RECOVERY - puppet last run on mw2046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:05:44] <icinga-wm>	 PROBLEM - puppet last run on mw2204 is CRITICAL: CRITICAL: puppet fail
[02:07:17] <andrewbogott>	 Krenair, Ori turned nutcracker back on a few days ago.  We both were sure that it would be harmless but not I suspect that that’s the cause of the session losses.
[02:07:31] <andrewbogott>	 I don’t have time to pay attention to it, though, unless it hits crisis levels.
[02:11:04] <icinga-wm>	 PROBLEM - puppet last run on db2040 is CRITICAL: CRITICAL: puppet fail
[02:19:47] <wikibugs>	 6operations, 6Services, 3Discovery-Maps-Sprint: Kartotherian does not start in producton - https://phabricator.wikimedia.org/T115074#1724172 (10Yurik) 5Open>3Resolved a:3Yurik I had to refactor the code to avoid 10+ different singleton instances of mapnik, and now it starts ok.
[02:23:49] <krrrit-wm>	 (03PS1) 10Ori.livneh: Drop Grafana 1.x; promote Grafana 2.x [puppet] - 10https://gerrit.wikimedia.org/r/246136 
[02:32:44] <icinga-wm>	 PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds
[02:34:24] <icinga-wm>	 RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212
[02:34:53] <icinga-wm>	 RECOVERY - puppet last run on mw2204 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[02:35:11] <krrrit-wm>	 (03PS2) 10Ori.livneh: Drop Grafana 1.x; promote Grafana 2.x [puppet] - 10https://gerrit.wikimedia.org/r/246136 
[02:36:00] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Drop Grafana 1.x; promote Grafana 2.x [puppet] - 10https://gerrit.wikimedia.org/r/246136 (owner: 10Ori.livneh)
[02:36:45] <icinga-wm>	 RECOVERY - puppet last run on db2040 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[02:37:08] <logmsgbot>	 !log l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 10s)
[02:37:13] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:40:43] <logmsgbot>	 !log l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-14 02:40:43+00:00
[02:40:48] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:46:17] <wikibugs>	 6operations, 7Graphite, 5Patch-For-Review: Upgrade to Grafana v2.x - https://phabricator.wikimedia.org/T104738#1724187 (10ori) 5Open>3Resolved
[02:48:10] <wikibugs>	 6operations, 7Graphite, 5Patch-For-Review: Upgrade to Grafana v2.x - https://phabricator.wikimedia.org/T104738#1425863 (10ori) There are now two vhosts:  * [[ https://grafana.wikimedia.org/ | grafana.wikimedia.org ]] provides a read-only, publicly-accessible view of our dashboards. * [[ https://grafana-admin...
[02:55:04] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 2 below the confidence bounds
[03:00:23] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 2 below the confidence bounds
[03:05:47] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 3 below the confidence bounds
[03:07:54] <logmsgbot>	 !log l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 37s)
[03:08:08] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:14:22] <logmsgbot>	 !log l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-14 03:14:22+00:00
[03:14:23] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 14 03:14:22 UTC 2015 (duration 14m 21s)
[03:14:31] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:14:36] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:15:57] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 3 below the confidence bounds
[03:20:59] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 3 below the confidence bounds
[03:29:15] <krrrit-wm>	 (03PS2) 10EBernhardson: Define cirrussearch shards/replicas per datacenter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244221 
[03:36:28] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 2 below the confidence bounds
[03:37:44] <krrrit-wm>	 (03Abandoned) 10EBernhardson: Update CirrusSearch config to define clusters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/237282 (https://phabricator.wikimedia.org/T109734) (owner: 10EBernhardson)
[03:45:16] <krrrit-wm>	 (03PS1) 10EBernhardson: Revert "cirrus tests: skip if full MediaWiki install is not availale" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246150 
[03:48:18] <krrrit-wm>	 (03PS2) 10EBernhardson: Revert "cirrus tests: skip if full MediaWiki install is not availale" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246150 
[03:49:10] <krrrit-wm>	 (03Abandoned) 10EBernhardson: Define cirrussearch shards/replicas per datacenter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244221 (owner: 10EBernhardson)
[04:00:16] <krrrit-wm>	 (03PS3) 10EBernhardson: Revert "cirrus tests: skip if full MediaWiki install is not availale" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246150 
[04:01:15] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Revert "cirrus tests: skip if full MediaWiki install is not availale" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246150 (owner: 10EBernhardson)
[04:01:22] <krrrit-wm>	 (03Merged) 10jenkins-bot: Revert "cirrus tests: skip if full MediaWiki install is not availale" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246150 (owner: 10EBernhardson)
[04:02:08] <icinga-wm>	 RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected
[04:39:19] <krrrit-wm>	 (03PS1) 10Ori.livneh: grafana: enable gzip [puppet] - 10https://gerrit.wikimedia.org/r/246153 
[04:42:25] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] grafana: enable gzip [puppet] - 10https://gerrit.wikimedia.org/r/246153 (owner: 10Ori.livneh)
[05:18:56] <krrrit-wm>	 (03Abandoned) 10KartikMistry: Fix nbwiki to nowiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244736 (owner: 10Amire80)
[05:23:45] <krrrit-wm>	 (03CR) 10EBernhardson: "didn't hear from anyone either way, so i've gone ahead and implemented the change so that all ElasticaWrite job failures are handled inter" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243744 (owner: 10EBernhardson)
[05:24:48] <icinga-wm>	 PROBLEM - puppet last run on mw2053 is CRITICAL: CRITICAL: Puppet has 1 failures
[05:51:48] <icinga-wm>	 RECOVERY - puppet last run on mw2053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:02:27] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1724286 (10Nemo_bis) Well, gerrit and Phabricator emails are certainly very bad. Multiple bugs have been reported in the last few years with actionable suggestions (some of whi...
[06:30:19] <icinga-wm>	 PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:30:48] <icinga-wm>	 PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:07] <icinga-wm>	 PROBLEM - puppet last run on nobelium is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:08] <icinga-wm>	 PROBLEM - puppet last run on db2056 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:57] <icinga-wm>	 PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:32:07] <icinga-wm>	 PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:32:38] <icinga-wm>	 PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: puppet fail
[06:32:59] <icinga-wm>	 PROBLEM - puppet last run on mw2050 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:09] <icinga-wm>	 PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:18] <icinga-wm>	 PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:50:38] <icinga-wm>	 PROBLEM - Kafka Broker Server on kafka1012 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args kafka.Kafka /etc/kafka/server.properties
[06:51:27] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1020 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [10.0]
[06:52:48] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1022 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [10.0]
[06:53:28] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1013 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [10.0]
[06:54:08] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1018 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [10.0]
[06:54:47] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1014 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [10.0]
[06:55:59] <icinga-wm>	 RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[06:56:18] <icinga-wm>	 RECOVERY - puppet last run on db2056 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[06:56:47] <icinga-wm>	 RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[06:56:59] <icinga-wm>	 RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[06:57:17] <icinga-wm>	 RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[06:57:18] <icinga-wm>	 RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:58] <icinga-wm>	 RECOVERY - puppet last run on nobelium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:18] <icinga-wm>	 RECOVERY - puppet last run on mw2050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:19] <icinga-wm>	 RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:58] <icinga-wm>	 RECOVERY - Kafka Broker Server on kafka1012 is OK: PROCS OK: 1 process with command name java, args kafka.Kafka /etc/kafka/server.properties
[06:59:37] <icinga-wm>	 RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:59:38] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1022 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [10.0]
[07:01:58] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1013 is CRITICAL: CRITICAL: 87.50% of data above the critical threshold [10.0]
[07:02:39] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1018 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [10.0]
[07:04:47] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1022 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [10.0]
[07:04:59] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on kafka1014 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [10.0]
[07:08:08] <icinga-wm>	 RECOVERY - Kafka Broker Under Replicated Partitions on kafka1022 is OK: OK: Less than 1.00% above the threshold [1.0]
[07:10:27] <icinga-wm>	 RECOVERY - Kafka Broker Under Replicated Partitions on kafka1013 is OK: OK: Less than 1.00% above the threshold [1.0]
[07:10:59] <icinga-wm>	 RECOVERY - Kafka Broker Under Replicated Partitions on kafka1018 is OK: OK: Less than 1.00% above the threshold [1.0]
[07:11:39] <icinga-wm>	 RECOVERY - Kafka Broker Under Replicated Partitions on kafka1014 is OK: OK: Less than 1.00% above the threshold [1.0]
[07:11:47] <icinga-wm>	 RECOVERY - Kafka Broker Under Replicated Partitions on kafka1020 is OK: OK: Less than 1.00% above the threshold [1.0]
[07:43:18] <icinga-wm>	 PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds
[07:44:57] <icinga-wm>	 RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212
[08:20:54] <krrrit-wm>	 (03PS1) 10Dereckson: Add *.unesco.org to server-side upload whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246167 (https://phabricator.wikimedia.org/T115338) 
[08:25:29] <krrrit-wm>	 (03PS2) 10Dereckson: Add *.unesco.org to server-side upload whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246167 (https://phabricator.wikimedia.org/T115338) 
[08:30:14] <krrrit-wm>	 (03PS1) 10Dereckson: Enable WikidataPageBanner on fr.wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246169 (https://phabricator.wikimedia.org/T115023) 
[08:55:30] <krrrit-wm>	 (03PS1) 10Dereckson: Configure default Echo subscriptions user options on he.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) 
[09:00:11] <krrrit-wm>	 (03CR) 10Dereckson: "See https://phabricator.wikimedia.org/P2193 for a compact file which illustrates how that works." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) (owner: 10Dereckson)
[10:12:43] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1724655 (10Aklapper)
[11:48:48] <icinga-wm>	 PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds
[11:50:27] <icinga-wm>	 RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212
[11:53:38] <icinga-wm>	 PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 1 failures
[11:56:29] <wikibugs>	 6operations, 6Labs, 10wikitech.wikimedia.org: Nutcracker having issues on wikitech - https://phabricator.wikimedia.org/T115457#1724804 (10Krenair) 3NEW
[12:07:11] <logmsgbot>	 !log krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/inspectors/ve.ui.MWLinkAnnotationInspector.js: https://gerrit.wikimedia.org/r/#/c/246205/ (duration: 01m 13s)
[12:07:15] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[12:08:20] <Krenair>	 Why does it take so long to connect to mira?
[12:09:34] <Krenair>	 debug1: Connecting to mira [2620:0:860:102:10:192:16:132] port 22.
[12:09:35] <Krenair>	 debug1: connect to address 2620:0:860:102:10:192:16:132 port 22: Connection timed out
[12:09:35] <Krenair>	 debug1: Connecting to mira [10.192.16.132] port 22.
[12:09:35] <Krenair>	 debug1: Connection established.
[12:09:35] <Krenair>	 Ugh.
[12:11:21] <Krenair>	 I bet this is the firewall resolving trusted hosts only to IPv4 addresses.
[12:13:38] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1724836 (10Elitre) >>! In T115416#1723815, @greg wrote: > Btw, I just got this when in gmail's web UI to mark them not as spam: > {F2717835} >  > Anyone else?  Didn't get that...
[12:14:47] <Krenair>	 And yet it works in reverse. Interesting.
[12:15:07] <Krenair>	 Oh, right. Because no firewall for tin. Okay.
[12:18:52] <krrrit-wm>	 (03PS1) 10Alex Monk: Fix scap firewall rules to allow connecting to mira over IPv6 [puppet] - 10https://gerrit.wikimedia.org/r/246206 
[12:20:28] <krrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "Thanks. This is really a ferm bug that I need to fix at some point..." [puppet] - 10https://gerrit.wikimedia.org/r/246206 (owner: 10Alex Monk)
[12:20:37] <icinga-wm>	 RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[12:22:20] <Krenair>	 paravoid, er, so I wrote that
[12:22:29] <paravoid>	 wrote what?
[12:22:30] <Krenair>	 And then realised that other puppet manifests use something else to allow the same thing
[12:22:51] <Krenair>	 We have deployment_hosts in special_hosts
[12:22:58] <Krenair>	     ferm::rule { 'deployment-ssh':
[12:22:58] <Krenair>	         ensure => present,
[12:22:58] <Krenair>	         rule   => 'proto tcp dport ssh saddr $DEPLOYMENT_HOSTS ACCEPT;',
[12:22:58] <Krenair>	     }
[12:23:02] <Krenair>	 is what the mediawiki role uses
[12:23:29] <paravoid>	 yeah I guess that's equivalent
[12:23:33] <Krenair>	 should I change it to that, or...?
[12:23:38] <paravoid>	 yeah sure, that works
[12:28:06] <krrrit-wm>	 (03PS1) 10Alex Monk: Use the same ferm rule for deployment hosts as we do for mediawiki etc. [puppet] - 10https://gerrit.wikimedia.org/r/246207 
[12:28:47] <paravoid>	 I have to go now, I'll merge it later
[12:28:49] <paravoid>	 sorry
[12:28:53] <Krenair>	 ok, no problem
[12:32:03] <logmsgbot>	 !log krenair@tin Synchronized README: testing https://gerrit.wikimedia.org/r/246206 (duration: 00m 17s)
[12:32:07] <Krenair>	 much better
[12:32:09] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[12:55:40] <hashar>	 !log ERROR: Could not connect to SMTP host: polonium.wikimedia.org, port: 25  (from labs instances)
[12:55:44] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[12:59:37] <Krenair>	 hashar, yeah polonium was replaced I think?
[12:59:54] <Krenair>	 mx100[12]?
[13:00:00] <Krenair>	 oh, just mw1001
[13:00:03] <Krenair>	 mx1001*
[13:00:09] <hashar>	 hieradata/labs/sentry/common.yaml:sentry::smtp_host: 'polonium.wikimedia.org'
[13:00:09] <hashar>	 hieradata/role/common/sentry.yaml:sentry::smtp_host: 'polonium.wikimedia.org'
[13:00:13] <hashar>	 still defined at some place
[13:00:19] * Krenair facepalms
[13:01:00] <hashar>	 and in Jenkins that is hardcoded in the definition (Jenkins is not pauperized), but that part we don't mind suffering disruption and having to fix it
[13:01:35] <Krenair>	 ah it was mx2001 in codfw
[13:01:38] <hashar>	 so yeah apparently $mail_smarthost    'labs'       => [ 'mx1001.wikimedia.org', 'mx2001.wikimedia.org' ],
[13:01:54] <Krenair>	 # spare
[13:01:54] <Krenair>	 node 'polonium.wikimedia.org' {
[13:03:07] <hashar>	 maybe I should get jenkins to send its mail to localhost  and get a smtp relay on gallium
[13:03:16] <JohnFLewis>	 to my knowledge polonium is mx1001 and lead has become mx2001
[13:03:30] <JohnFLewis>	 which is nice, because one is eqiad and one is codfw now :)
[13:04:46] <hashar>	 and we also have  wiki-mail.wikimedia.org  pointing to mx1001 :D
[13:05:06] <Krenair>	 should probably point stuff at that...
[13:05:39] <hashar>	 unless it is legacy
[13:07:51] <paravoid>	 if people don't use $mail_smarthost they're on their own as far as I care
[13:10:43] <wikibugs>	 6operations, 7Graphite, 5Patch-For-Review: Upgrade to Grafana v2.x - https://phabricator.wikimedia.org/T104738#1724974 (10BBlack) Thanks! log-scale y-axis now works :)
[13:11:27] <paravoid>	 I don't understand why sentry should have its own hiera config for the smarthost
[13:11:44] <paravoid>	 and as for jenkins, I don't understand why it needs to use SMTP and can't use sendmail()
[13:11:57] <paravoid>	 gallium does run an exim, as all hosts do
[13:12:02] <ottomata>	 !log restarted diamond, puppet didn't seem to after it removed the TcpConnStates from most hosts
[13:12:06] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[13:13:55] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1724991 (10Andrew) This will need some misc database support as well --  adding Jaime to the ticket.  We could potentially have the dbs all run on the c...
[13:15:12] <paravoid>	 (and yes, wiki-mail is legacy -- it actually never was supposed for what people used it for)
[13:16:41] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1725004 (10coren) Have you coordinated with @papaul about the currently unknown-function Cisco servers that used to be Labs and that currently live in D...
[13:19:55] <krrrit-wm>	 (03PS1) 10Ottomata: Turn on varnish reqstats diamond collector on eqiad mobile caches [puppet] - 10https://gerrit.wikimedia.org/r/246215 (https://phabricator.wikimedia.org/T83580) 
[13:22:45] <krrrit-wm>	 (03CR) 10Ottomata: [C: 032] Turn on varnish reqstats diamond collector on eqiad mobile caches [puppet] - 10https://gerrit.wikimedia.org/r/246215 (https://phabricator.wikimedia.org/T83580) (owner: 10Ottomata)
[13:23:10] <krrrit-wm>	 (03PS4) 10Ori.livneh: Add dblist to many paths [puppet] - 10https://gerrit.wikimedia.org/r/244743 (owner: 10Reedy)
[13:23:31] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Add dblist to many paths [puppet] - 10https://gerrit.wikimedia.org/r/244743 (owner: 10Reedy)
[13:30:08] <krrrit-wm>	 (03PS1) 10Ori.livneh: decom grafana-test and tessera vhosts [dns] - 10https://gerrit.wikimedia.org/r/246218 
[13:30:31] <ottomata>	 !log kafka-preferred-replica election after kafka1012
[13:30:35] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[13:32:45] <hashar>	 ori: have you upgraded grafana already? :D
[13:33:01] <krrrit-wm>	 (03CR) 10Luke081515: [C: 031] Configure default Echo subscriptions user options on he.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) (owner: 10Dereckson)
[13:35:26] <krrrit-wm>	 (03PS1) 10Andrew Bogott: Fixed the CST paging hours. Previously AM/PM were reversed. [puppet] - 10https://gerrit.wikimedia.org/r/246219 
[13:37:23] <krrrit-wm>	 (03PS2) 10Andrew Bogott: Fixed the CST paging hours. Previously AM/PM were reversed. [puppet] - 10https://gerrit.wikimedia.org/r/246219 
[13:38:22] <krrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Fixed the CST paging hours. Previously AM/PM were reversed. [puppet] - 10https://gerrit.wikimedia.org/r/246219 (owner: 10Andrew Bogott)
[13:38:56] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Move base::debdeploy into the base class [puppet] - 10https://gerrit.wikimedia.org/r/246220 
[13:44:10] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Move the base::firewall include into the impala role [puppet] - 10https://gerrit.wikimedia.org/r/246221 
[13:50:27] <icinga-wm>	 PROBLEM - etherpad.wikimedia.org HTTP on etherpad1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:51:54] <krrrit-wm>	 (03PS1) 10Muehlenhoff: tor: Move the ferm rules into the role [puppet] - 10https://gerrit.wikimedia.org/r/246223 
[13:52:07] <icinga-wm>	 RECOVERY - etherpad.wikimedia.org HTTP on etherpad1001 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 522 bytes in 0.002 second response time
[13:56:39] <krrrit-wm>	 (03PS1) 10Muehlenhoff: wdqs: Move the ferm rules into the role [puppet] - 10https://gerrit.wikimedia.org/r/246224 
[13:58:34] <wikibugs>	 6operations, 10ops-codfw: power off Codfw-Cisco Servers - https://phabricator.wikimedia.org/T115372#1725144 (10RobH) Also please boot each and ensure the disks were wiped.  If not, please wipe the disks.  (Wipe in software, not the degausser.)
[14:02:24] <wikibugs>	 6operations, 7Monitoring: non sms alternatives - https://phabricator.wikimedia.org/T114651#1702155 (10RobH) I wouldn't want to rely on data instead of SMS alerts.  Some parts of apartments have horrible data, but SMS works.  Thus, I can easily be in a situation where my phone isn't on data, but if I get an SMS...
[14:02:25] <wikibugs>	 6operations, 7Monitoring: non sms alternatives - https://phabricator.wikimedia.org/T114651#1725159 (10RobH)
[14:02:28] <wikibugs>	 6operations, 7Monitoring: non sms alternatives - https://phabricator.wikimedia.org/T114651#1702155 (10RobH)
[14:06:11] <wikibugs>	 6operations, 7Monitoring: non sms alternatives - https://phabricator.wikimedia.org/T114651#1725164 (10coren) The reason why I like data-based altering (in addition to/with SMS) is that it tends to work better when you have intermittent coverage (because of the way it works, with positive ack, means that you'll...
[14:06:30] <wikibugs>	 6operations, 5Patch-For-Review, 7Technical-Debt: Retire Torrus - https://phabricator.wikimedia.org/T87840#1725165 (10Dzahn) As @akosiaris pointed out yesterday the aggregated graphs don't seem to work currently.  That would mean we don't have this feature anyways and the others have been replaced by librenams.
[14:12:59] <ori>	 hashar: yep
[14:13:26] <hashar>	 ori: that is great, would you mind restoring the previous landing page? Not sure where it went / how it was named though
[14:13:41] <hashar>	 I had a use case for Grafana 2.x  so it is good to see it being upgraded
[14:14:41] <hashar>	 ori: there is some work going on to add ElasticSearch as a backend. So in theory we could create boards for our logstash data
[14:15:38] <hashar>	 ah http://docs.grafana.org/v2.1/reference/dashlist/
[14:15:40] <mutante|1way>	 !log mw1157 - deleted puppet lock file, fix puppet run. ("already running" but didnt since 18h)
[14:15:48] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[14:15:57] <wikibugs>	 6operations, 7Graphite, 5Patch-For-Review: Upgrade to Grafana v2.x - https://phabricator.wikimedia.org/T104738#1725170 (10hashar) Thank you a ton @ori  We should get http://docs.grafana.org/v2.1/reference/dashlist/ among others :-}
[14:17:45] <icinga-wm>	 RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:18:18] <krrrit-wm>	 (03PS1) 10Ori.livneh: decom tessera [puppet] - 10https://gerrit.wikimedia.org/r/246227 
[14:20:13] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] decom tessera [puppet] - 10https://gerrit.wikimedia.org/r/246227 (owner: 10Ori.livneh)
[14:20:22] <krrrit-wm>	 (03PS1) 10Ottomata: Enable diamond reqstats collector for all mobile varnishes [puppet] - 10https://gerrit.wikimedia.org/r/246229 (https://phabricator.wikimedia.org/T83580) 
[14:21:10] <krrrit-wm>	 (03PS2) 10Ottomata: Enable diamond reqstats collector for all mobile varnishes [puppet] - 10https://gerrit.wikimedia.org/r/246229 (https://phabricator.wikimedia.org/T83580) 
[14:21:40] <krrrit-wm>	 (03PS1) 10Ori.livneh: misc varnish: drop tessera.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/246230 
[14:22:26] <krrrit-wm>	 (03PS1) 10Alex Monk: ldaplist: Add support for projects/projectroles [puppet] - 10https://gerrit.wikimedia.org/r/246231 
[14:24:45] <ori>	 !log cleaned up tessera leftovers on graphite1001
[14:24:48] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[14:25:03] <krrrit-wm>	 (03CR) 10Ottomata: [C: 032] Enable diamond reqstats collector for all mobile varnishes [puppet] - 10https://gerrit.wikimedia.org/r/246229 (https://phabricator.wikimedia.org/T83580) (owner: 10Ottomata)
[14:25:58] <krrrit-wm>	 (03PS2) 10Ori.livneh: misc varnish: drop tessera.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/246230 
[14:26:41] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] misc varnish: drop tessera.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/246230 (owner: 10Ori.livneh)
[14:27:05] <wikibugs>	 6operations, 6Analytics-Kanban, 10netops, 5Patch-For-Review: Puppetize a server with a role that sets up Cassandra on Analytics machines [13 pts] {slug} - https://phabricator.wikimedia.org/T107056#1725187 (10Dzahn)
[14:27:12] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] decom grafana-test and tessera vhosts [dns] - 10https://gerrit.wikimedia.org/r/246218 (owner: 10Ori.livneh)
[14:28:00] <mutante|1way>	 wikibugs: talk 
[14:28:19] <hashar>	 wikibugs: ping
[14:28:30] <ori>	 wikibugs: yo
[14:28:55] <logmsgbot>	 wikibugs: sup
[14:31:34] <mutante|1way>	 works on other channel
[14:31:36] <mutante|1way>	 < wikibugs> Wikibugs: Wikibugs should not say ", and 0 others" - https://phabricator.wikimedia.org/T90324#1725196 (Dzahn) bla bla bla (testing wikibugs)
[14:40:52] <icinga-wm>	 PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 1 failures
[14:42:05] <yuvipanda>	 mutante|1way: what's the alert text?
[14:42:11] <yuvipanda>	 (icinga is being slow now)
[14:42:12] <krrrit-wm>	 (03PS1) 10Ori.livneh: Set up regular backups for the Grafana database [puppet] - 10https://gerrit.wikimedia.org/r/246236 
[14:42:16] <ori>	 akosiaris: ^
[14:42:38] <mutante|1way>	 yuvipanda: CRITICAL: Puppet has 1 failures 
[14:43:05] <yuvipanda>	 mutante|1way: no, for the tools-shadow bug you re-opened?
[14:43:43] <mutante|1way>	 yuvipanda: CRITICAL: master class instances not spread out enough  
[14:43:55] <yuvipanda>	 hmm
[14:43:57] <yuvipanda>	 not sure
[14:44:00] <yuvipanda>	 why that's the case
[14:44:02] <yuvipanda>	 I'll check later
[14:44:04] <mutante|1way>	 i just opened that one because the icinga still had a comment
[14:44:07] <krrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Set up regular backups for the Grafana database [puppet] - 10https://gerrit.wikimedia.org/r/246236 (owner: 10Ori.livneh)
[14:44:07] <yuvipanda>	 yeah
[14:44:07] <mutante|1way>	 that linked to this ticket
[14:44:10] <mutante|1way>	 ok, cool
[14:44:14] <ori>	 thanks!
[14:44:15] <yuvipanda>	 that was fixed a few weeks ago
[14:44:48] <mutante|1way>	 it seems different for labcontrol1001 vs. 1002
[14:44:59] <mutante|1way>	 the time since it became crit i mean
[14:47:19] <krrrit-wm>	 (03PS1) 10Alexandros Kosiaris: ldap.conf: Remove openldap unused parameters [puppet] - 10https://gerrit.wikimedia.org/r/246242 
[14:47:28] <krrrit-wm>	 (03CR) 10Dzahn: [C: 032] Format Tmax in slow queries page as a number [software/tendril] - 10https://gerrit.wikimedia.org/r/244964 (owner: 10Alex Monk)
[14:47:47] <krrrit-wm>	 (03CR) 10Dzahn: [V: 032] Format Tmax in slow queries page as a number [software/tendril] - 10https://gerrit.wikimedia.org/r/244964 (owner: 10Alex Monk)
[14:50:07] <krrrit-wm>	 (03PS3) 10Andrew Bogott: Openstack: Don't notify keystone when the keystone policy changes [puppet] - 10https://gerrit.wikimedia.org/r/244349 
[14:51:06] <krrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Openstack: Don't notify keystone when the keystone policy changes [puppet] - 10https://gerrit.wikimedia.org/r/244349 (owner: 10Andrew Bogott)
[14:55:02] <krrrit-wm>	 (03PS2) 10Dzahn: Include base::firewall into the planet role [puppet] - 10https://gerrit.wikimedia.org/r/245970 (owner: 10Muehlenhoff)
[14:57:00] <mutante|1way>	 legoktm: could you fix wikibugs. it stopped talking here but not in other channels
[14:57:17] <legoktm>	 hello
[14:57:20] <legoktm>	 wikibugs: !
[14:57:42] <wikibugs>	 6operations, 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1725253 (10Legoktm)
[14:57:47] <mutante|1way>	 eh :)
[14:57:53] <legoktm>	 seems to work fine?
[14:57:57] <mutante|1way>	 now it does :p
[14:58:18] <mutante|1way>	 fixed by asking :) thx
[14:58:25] <legoktm>	 yw :)
[15:00:04] <jouncebot>	 anomie ostriches thcipriani marktraceur Krenair: Dear anthropoid, the time has come. Please deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151014T1500).
[15:00:04] <jouncebot>	 MatmaRex: A patch you scheduled for Morning SWAT (Max 8 patches) is about to be deployed. Please be available during the process.
[15:00:42] <krrrit-wm>	 (03CR) 10Dzahn: [C: 032] Include base::firewall into the planet role [puppet] - 10https://gerrit.wikimedia.org/r/245970 (owner: 10Muehlenhoff)
[15:01:07] <MatmaRex>	 hi.
[15:02:42] <thcipriani>	 MatmaRex: hiya, I'll SWAT today.
[15:04:49] <krrrit-wm>	 (03PS2) 10Dzahn: tor: Move the ferm rules into the role [puppet] - 10https://gerrit.wikimedia.org/r/246223 (owner: 10Muehlenhoff)
[15:05:00] <Krinkle>	 !log Statsv and eventlogging-navtiming seems to have gone down 7 hours ago
[15:05:04] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:08:00] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Move dnsrecursor to the role keyword [puppet] - 10https://gerrit.wikimedia.org/r/246244 
[15:09:45] <krrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 031] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/246244 (owner: 10Muehlenhoff)
[15:11:12] <logmsgbot>	 !log thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files [[gerrit:246226]] (duration: 00m 18s)
[15:11:15] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:11:35] <thcipriani>	 MatmaRex: ^ Check if possible please.
[15:11:58] <MatmaRex>	 thcipriani: it only affects commons and it's still on wmf.2
[15:12:11] <krrrit-wm>	 (03CR) 10Paladox: "Not really a setting can be called what ever they like as long as they follow the mediawiki convention on how they are set out. The settin" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244739 (owner: 10Paladox)
[15:12:36] <thcipriani>	 MatmaRex: kk, going ahead with 2
[15:14:58] <ori>	 !log restarted navtiming and statsv services on hafnium
[15:15:02] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:19:55] <logmsgbot>	 !log thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files [[gerrit:246225]] (duration: 00m 17s)
[15:19:59] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:20:11] <thcipriani>	 ^ MatmaRex check please
[15:20:43] <MatmaRex>	 thcipriani: thanks, looking
[15:24:16] <MatmaRex>	 thcipriani: thanks, it works as expected
[15:24:26] <thcipriani>	 MatmaRex: nice. Thanks for checking!
[15:24:57] <thcipriani>	 dcausse: ping for SWAT, looks like the bot missed your patch.
[15:25:05] <dcausse>	 thcipriani: hi
[15:26:45] <thcipriani>	 dcausse: howdy, so is this just a maintenance script update? Don't need me to run anything, right?
[15:27:12] <dcausse>	 thcipriani: yes no need to run anything, it's for the new codfw elastic cluster
[15:27:24] <thcipriani>	 dcausse: cool, just making sure :)
[15:31:48] <logmsgbot>	 !log thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: SWAT: Split connection to source and target [[gerrit:246243]] (duration: 00m 18s)
[15:31:52] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:32:05] <thcipriani>	 ^ dcausse sync'd! check if possible
[15:32:36] <dcausse>	 thcipriani: looks good, thanks!
[15:32:52] <thcipriani>	 dcausse: cool, thank you!
[15:45:49] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1725452 (10bd808) New fun today, one message from {T115440} marked as a possible phishing scam. {F2720728}
[15:49:37] <wikibugs>	 6operations: audit all SSL certificates expiry on ops tracking gcal - https://phabricator.wikimedia.org/T112542#1725479 (10RobH)
[15:49:38] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Track/notify cert expiries better - https://phabricator.wikimedia.org/T112521#1725478 (10RobH)
[15:50:49] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Track/notify cert expiries better - https://phabricator.wikimedia.org/T112521#1637061 (10RobH) We've recently changed the policy on ordering ssl certificates to put in the ssl_renewals alias for all ssl cert info.  Additionally, we should see if we can change the backend conta...
[15:50:53] <krrrit-wm>	 (03PS1) 10Ori.livneh: grafana: provide the ability to provision dashboards via Puppet [puppet] - 10https://gerrit.wikimedia.org/r/246252 
[15:51:06] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1725485 (10greg) FWIW, I also had a ton of Shinken alerts in my spam folder as well.
[15:52:10] <krrrit-wm>	 (03PS2) 10Ori.livneh: grafana: provide the ability to provision dashboards via Puppet [puppet] - 10https://gerrit.wikimedia.org/r/246252 
[15:52:52] <wikibugs>	 6operations, 6Phabricator, 7Database, 5Patch-For-Review, 7WorkType-Maintenance: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. - https://phabricator.wikimedia.org/T109279#1725487 (10mmodell) Thanks. The graphs...
[15:52:59] <wikibugs>	 6operations, 6Phabricator, 7Database, 5Patch-For-Review, 7WorkType-Maintenance: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. - https://phabricator.wikimedia.org/T109279#1725489 (10mmodell) 5Open>3Resolved
[15:53:04] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] grafana: provide the ability to provision dashboards via Puppet [puppet] - 10https://gerrit.wikimedia.org/r/246252 (owner: 10Ori.livneh)
[15:53:56] <wikibugs>	 6operations, 6Phabricator, 7Database, 5Patch-For-Review, 7WorkType-Maintenance: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. - https://phabricator.wikimedia.org/T109279#1582035 (10mmodell)
[16:00:12] <wikibugs>	 6operations: Create roles for test systems and spares - https://phabricator.wikimedia.org/T115489#1725516 (10MoritzMuehlenhoff) 3NEW a:3MoritzMuehlenhoff
[16:14:24] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Allocate labs subnet in dallas - https://phabricator.wikimedia.org/T115491#1725570 (10Andrew) 3NEW a:3mark
[16:15:59] <wikibugs>	 6operations, 7Graphite, 7Monitoring, 5Patch-For-Review: deprecate gdash - https://phabricator.wikimedia.org/T104365#1725581 (10ori) Grafana is now set up to load JSON dashboard definition files from a local directory which the Grafana Puppet module manages. There is a `grafana::dashboard` resource that mak...
[16:17:11] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Allocate subnet for labs test cluster instances - https://phabricator.wikimedia.org/T115492#1725583 (10Andrew) 3NEW a:3mark
[16:21:02] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Allocate subnet for labs test cluster instances - https://phabricator.wikimedia.org/T115492#1725608 (10coren) Given that we are looking to do vlan config with Neutron, I think it'd be wise to allocate a very large subnet fo...
[16:25:57] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1725615 (10Andrew)
[16:26:03] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1725616 (10RobH) Ok, allocations for this:  * WMF3763 (previously named ssl2002) - 8GB System * WMF3810 (previously  named ssl2003) - 8GB System * WMF58...
[16:32:42] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1725645 (10Andrew) Please cable up two nics for:  wmf5850 (labtestvirt2001) WMF5835 (labtestnet2001)  The second nic will be on the instance vlan.  The...
[16:43:00] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 7 below the confidence bounds
[16:44:11] <icinga-wm>	 PROBLEM - puppet last run on wtp2014 is CRITICAL: CRITICAL: Puppet has 1 failures
[16:49:41] <icinga-wm>	 RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected
[17:00:04] <jouncebot>	 moritzm: Dear anthropoid, the time has come. Please deploy Operations (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151014T1700).
[17:09:50] <icinga-wm>	 RECOVERY - puppet last run on wtp2014 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[17:19:11] <icinga-wm>	 PROBLEM - etherpad.wikimedia.org HTTP on etherpad1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:19:30] <mobrovac>	 aha, it's not just me
[17:19:40] <mobrovac>	 akosiaris: ^^^
[17:31:11] <icinga-wm>	 RECOVERY - etherpad.wikimedia.org HTTP on etherpad1001 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 522 bytes in 0.002 second response time
[17:32:24] <wikibugs>	 6operations, 7HHVM, 5Patch-For-Review: Package and deploy HHVM 3.6.5+dfsg1-1+wm7 - https://phabricator.wikimedia.org/T112640#1725840 (10bd808)
[17:34:32] <wikibugs>	 6operations, 6Discovery, 5codfw-rollout: [EPIC] Set up a CirrusSearch cluster in codfw (Dallas, Texas) - https://phabricator.wikimedia.org/T105703#1725853 (10Smalyshev)
[17:42:05] <krrrit-wm>	 (03CR) 10Eranroz: Configure default Echo subscriptions user options on he.wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) (owner: 10Dereckson)
[17:56:23] <krrrit-wm>	 (03PS2) 10Chad: Load some more extensions directly through wfLoadExtension() (B-D) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/232966 (owner: 10Legoktm)
[17:56:35] <ostriches>	 legoktm: Any objections to ^?
[17:57:01] <legoktm>	 ostriches: nope! :)
[17:57:24] <krrrit-wm>	 (03CR) 10Chad: [C: 032] Load some more extensions directly through wfLoadExtension() (B-D) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/232966 (owner: 10Legoktm)
[17:57:31] <krrrit-wm>	 (03Merged) 10jenkins-bot: Load some more extensions directly through wfLoadExtension() (B-D) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/232966 (owner: 10Legoktm)
[17:57:39] <ostriches>	 legoktm: CX is spamming the warning log about using the deprecated extension loading format.
[17:57:40] <ostriches>	 :)
[17:57:50] <RoanKattouw>	 legoktm: Did I do https://gerrit.wikimedia.org/r/#/c/246276/1 right?
[17:58:21] <legoktm>	 RoanKattouw: yes
[17:58:49] <logmsgbot>	 !log demon@tin Synchronized wmf-config/extension-list: (no message) (duration: 00m 17s)
[17:58:53] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[17:58:57] <legoktm>	 RoanKattouw: if you look at https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/6019/console, it shows you that Flow will no longer run mwext-qunit
[17:59:06] <krrrit-wm>	 (03PS1) 10Ori.livneh: grafana: override the default home dashboard with something custom [puppet] - 10https://gerrit.wikimedia.org/r/246278 
[17:59:13] <logmsgbot>	 !log demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
[17:59:16] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[17:59:25] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] grafana: override the default home dashboard with something custom [puppet] - 10https://gerrit.wikimedia.org/r/246278 (owner: 10Ori.livneh)
[17:59:49] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1725915 (10Niedzielski) I filed a similar ticket[0] back in August for Phabricator and mailing list messages. The issue actually seemed to have resolved itself for the past mon...
[18:00:05] <jouncebot>	 twentyafterfour: Dear anthropoid, the time has come. Please deploy MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151014T1800).
[18:00:20] <twentyafterfour>	 jouncebot: ok fine
[18:00:47] <krrrit-wm>	 (03PS1) 1020after4: group1 wikis to 1.27.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246279 
[18:01:10] <Luke081515>	 A database query error has occurred. This may indicate a bug in the software. <-- at dewiki
[18:01:29] <Luke081515>	 I can't see any page
[18:01:47] <krrrit-wm>	 (03PS1) 10Chad: Use $wgFlaggedRevsTags instead of $wgFlaggedRevTags [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246281 
[18:01:47] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1725921 (10RobH) @Andrew:  Chatting with @Mark @ lunch, we need to clear the allocation of the 32GB misc machine with him.  (It was a passing comment, s...
[18:01:57] <bd808>	 "Error: 1066 Not unique table/alias: 'page_props' (10.64.48.25)"
[18:01:58] <RoanKattouw>	 legoktm: Awesome. Let's see if I have the rights to deploy that
[18:01:59] <thedj>	 ehm. i just got 0 content in the main page....
[18:02:04] <greg-g>	 :(
[18:02:27] <greg-g>	 twentyafterfour: revert
[18:03:04] <krrrit-wm>	 (03CR) 1020after4: [C: 032] group1 wikis to 1.27.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246279 (owner: 1020after4)
[18:03:07] <legoktm>	 RoanKattouw: I just did :)
[18:03:11] <krrrit-wm>	 (03Merged) 10jenkins-bot: group1 wikis to 1.27.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246279 (owner: 1020after4)
[18:03:20] <legoktm>	     303 Compilation failed: two named subpatterns have the same name at offset 263 in /srv/mediawiki/php-1.27.0-wmf.2/includes/MagicWord.php on line 960
[18:03:29] <wikibugs>	 6operations: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1725929 (10Luke081515) 3NEW
[18:03:31] <ostriches>	 Yep.
[18:04:14] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1725936 (10Krenair) >>! In T115416#1725915, @Niedzielski wrote: > I filed a similar ticket[0] back in August for Phabricator and mailing list messages. The issue actually seeme...
[18:04:16] <wikibugs>	 6operations: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1725938 (10greg) {F2721219}
[18:04:31] <bd808>	 dewiki is broken folks. anybody working on figuring out why?
[18:04:33] <legoktm>	 Function: RandomPage::selectRandomPageFromDB
[18:04:33] <legoktm>	 Error: 1066 Not unique table/alias: 'page_props' (10.64.48.27)
[18:04:50] <Luke081515>	 greg-g postet a screenshot of commons, same here
[18:04:51] <ori>	 revert the deploy
[18:04:52] <bd808>	 https://de.wikipedia.org/wiki/Main_Page is acked up too
[18:04:55] <ori>	 now
[18:04:57] <Luke081515>	 *posted
[18:05:09] <ostriches>	 group1 wiki deploy, I'd bet.
[18:05:17] <greg-g>	 twentyafterfour: revert
[18:05:18] <Luke081515>	 I wonder, wikipedias are group2
[18:05:19] <krrrit-wm>	 (03PS1) 10Ori.livneh: Revert "group1 wikis to 1.27.0-wmf.3" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246283 
[18:05:23] <legoktm>	 no, the errors are on wmf2
[18:05:39] <aude>	 what was the issue?
[18:05:49] <Luke081515>	 you can't see any page
[18:05:50] <twentyafterfour>	 I didn't deploy it yet
[18:05:54] <krrrit-wm>	 (03PS1) 10Ori.livneh: Follow-up for I5667f712: make the top panel transparent [puppet] - 10https://gerrit.wikimedia.org/r/246284 
[18:05:58] <bd808>	 maybe https://gerrit.wikimedia.org/r/#/c/232966/ ?
[18:05:58] <Krenair>	 I don't think it's just dewiki
[18:06:02] <thedj>	 en.wiki too
[18:06:06] <krrrit-wm>	 (03CR) 10Greg Grossmeier: "Krenair: Please remove your -2 on this patch. Chris S and I agree this is OK to do. Thanks." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236231 (https://phabricator.wikimedia.org/T110619) (owner: 10MarcoAurelio)
[18:06:06] <andre__>	 en.wiki also
[18:06:08] <krrrit-wm>	 (03PS1) 10Legoktm: Revert "Load some more extensions directly through wfLoadExtension() (B-D)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246285 
[18:06:11] <twentyafterfour>	 it's not related to any change I made
[18:06:13] <Shanmugamp7>	 mw and meta as well 
[18:06:16] <krrrit-wm>	 (03CR) 10Legoktm: [C: 032] Revert "Load some more extensions directly through wfLoadExtension() (B-D)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246285 (owner: 10Legoktm)
[18:06:20] <greg-g>	 what the
[18:06:22] <krrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Load some more extensions directly through wfLoadExtension() (B-D)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246285 (owner: 10Legoktm)
[18:06:24] <ostriches>	 legoktm: Already deploying.
[18:06:33] <logmsgbot>	 !log demon@tin Synchronized wmf-config: (no message) (duration: 00m 19s)
[18:06:37] <ori>	 yeah
[18:06:37] <ori>	 legoktm: 
[18:06:37] <ori>	 is right
[18:06:40] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[18:06:41] <krrrit-wm>	 (03CR) 10Alex Monk: "I don't agree that this is OK to do with the extension in it's current state." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236231 (https://phabricator.wikimedia.org/T110619) (owner: 10MarcoAurelio)
[18:06:47] <legoktm>	 ok
[18:06:53] <ori>	 legoktm: thanks
[18:06:54] <Luke081515>	 reverted?
[18:06:58] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Follow-up for I5667f712: make the top panel transparent [puppet] - 10https://gerrit.wikimedia.org/r/246284 (owner: 10Ori.livneh)
[18:07:09] <_joe_>	 are we ok?
[18:07:11] <Luke081515>	 works now! Thanks
[18:07:17] <bd808>	 https://de.wikipedia.org/wiki/Main_Page back after ?action=purge
[18:07:17] <ori>	 yes
[18:07:19] <ostriches>	 _joe_: Ya
[18:07:21] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s1 on db1051 is CRITICAL: CRITICAL slave_sql_lag Seconds_Behind_Master: 381
[18:07:25] <legoktm>	 ok sorry about that
[18:07:25] <ostriches>	 legoktm: The heck?
[18:07:29] <legoktm>	 it should have been a no-op
[18:07:36] <greg-g>	 main page on commons is still empty for me
[18:07:37] <ori>	 apparently not
[18:07:37] <ostriches>	 Indeed.
[18:07:47] <_joe_>	 what the fuck guys
[18:07:52] <legoktm>	 greg-g: I purged it
[18:07:59] <greg-g>	 back
[18:08:02] <_joe_>	 it's the second release done during this week
[18:08:05] <greg-g>	 _joe_: we just missed you, wanted you back
[18:08:07] <_joe_>	 that breaks the site
[18:08:10] <ori>	 well
[18:08:13] <ostriches>	 This wasn't a release.
[18:08:14] <greg-g>	 _joe_: read first, it wasn't the deploy
[18:08:15] <ori>	 what about all the other pages?
[18:08:19] <_joe_>	 so much for "no breaking changes"
[18:08:23] <_joe_>	 whatever.
[18:08:28] * twentyafterfour didn't break it
[18:08:31] <_joe_>	 was it deployed?
[18:08:31] <twentyafterfour>	 :P
[18:08:34] <ori>	 presumably there are plenty of others
[18:08:36] <ori>	 that need a purge
[18:08:44] <ostriches>	 _joe_: It wasn't supposed to be a breaking change, it was supposed to be a no-op.
[18:08:47] <paravoid>	 hey
[18:08:54] <bblack>	 ?
[18:08:55] <paravoid>	 what's the TL;DR of what's going on?
[18:08:55] <ori>	 there is an unknown number of pages that have been blanked, no?
[18:09:02] <bd808>	 those 200s were cached by varnish. Not sure there is any way to purge anything added to cache in the last N minutes
[18:09:03] <bblack>	 the answer to a deploy problem is not purge the caches :P
[18:09:10] <greg-g>	 this was the change that broke it, right? https://gerrit.wikimedia.org/r/#/c/232966/
[18:09:13] <ori>	 paravoid: change to how extensions are loaded, deployed by ostriches, reverted by legoktm
[18:09:14] <Luke081515>	 yes, we need to purge
[18:09:15] <ostriches>	 paravoid: config change caused blank pages.
[18:09:16] <legoktm>	 um, there's also high s1 slave lag
[18:09:22] <Luke081515>	 otherwise, there is no content visible
[18:09:24] <ori>	 paravoid: site is now fine, but many pages were cached as blank
[18:09:30] <bblack>	 is there some way to identify the affected pages?
[18:09:36] <mutante|1way>	 we got paged for mariadb lag
[18:10:02] <wikibugs>	 6operations: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1725947 (10demon) 5Open>3Resolved a:3demon
[18:10:26] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure, 10hardware-requests, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1725949 (10Andrew) Naming!  wmf5850 (32G): labtestvirt2001 WMF5835 (16G): labtestnet2001 WMF3763 (8GB): labtestneutron2001 WMF3810 (8GB): labtestmetal20...
[18:10:29] <twentyafterfour>	 seemingly all kinds of pages (anything newly cached in the last few minutes I guess?)
[18:10:32] <bblack>	 is there a specific response header included in only the messed up blank responses I could varnish ban on, for instance?
[18:10:33] <ori>	 how blank was blank? blank-blank or no article contents?
[18:10:40] <ostriches>	 No article contents
[18:10:47] <bblack>	 is there other content? example?
[18:10:47] <jynus>	 revert
[18:10:51] <ostriches>	 And likely, just pages re-rendered during the downtime, if I had to guess.
[18:10:53] <andre__>	 No article contents. Sometimes (for me) loading the "Categories" box though.
[18:10:54] <ori>	 jynus: already reverted
[18:10:55] <twentyafterfour>	 empty page basically
[18:10:55] <jynus>	 it is the code deploy
[18:10:57] <bblack>	 I could ban small content-length maybe too
[18:10:58] <jynus>	 :--)
[18:11:03] <_joe_>	 so to answer bblack: nope
[18:11:04] <bd808>	 jynus: already reverted
[18:11:06] <legoktm>	 the skin loaded, but the article text was empty
[18:11:12] <twentyafterfour>	 uhm, blank pages now have content for me
[18:11:12] <bblack>	 example link?
[18:11:21] <twentyafterfour>	 so maybe teh blanks weren't cached?
[18:11:22] <ori>	 twentyafterfour: because people have been purging them
[18:11:25] <legoktm>	 I'm trying to find one...I think people already purged them
[18:11:26] <ori>	 they were cached
[18:11:28] <andre__>	 Example? For example when I load https://en.wikipedia.org/wiki/GNU_General_Public_License
[18:11:32] <ori>	 but there is probably a long tail
[18:11:36] <ori>	 ^^^ DONT PURGE THAT ONE
[18:11:42] <andre__>	    GNU General Public License
[18:11:42] <andre__>	    From Wikipedia, the free encyclopedia
[18:11:42] <andre__>	    Categories (++): (+)
[18:11:53] <andre__>	 is all the content I get even when bypassing browser cache
[18:11:56] <wikibugs>	 6operations: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1725959 (10greg) Caused by https://gerrit.wikimedia.org/r/#/c/232966/, reverted https://gerrit.wikimedia.org/r/#/c/246285/ 9 minutes later.
[18:12:10] <Luke081515>	 andre__: yes, good example
[18:12:12] <legoktm>	 that page also has <!-- Saved in parser cache with key enwiki:pcache:idhash:18938683-0!*!*!*!*!*!* and timestamp 20151014180514 and revision id 685524523 -->
[18:12:12] <robh>	 that link fully loads for me no blank.
[18:12:13] <Luke081515>	 for that error
[18:12:20] <legoktm>	 so it's in parsercache too
[18:12:23] <greg-g>	 jynus: wasn't the main deploy, was a config change, see that task ^
[18:12:46] <ori>	 we can use the parser cache hook handler to reject items cached during the outage
[18:12:50] <twentyafterfour>	 not blank here either
[18:12:55] <Krenair>	 That's the kind of patch I'd test on mw1017 before fully syncing...
[18:13:04] <paravoid>	 wait, so they're cached in parser cache as well?
[18:13:12] <paravoid>	 it's not just varnish-cached then?
[18:13:13] <ori>	 looks like it
[18:13:18] <bd808>	 it's a varnish thing so I would expect different results from different DCs
[18:13:18] <paravoid>	 awesome
[18:13:31] <paravoid>	 that's not how it works, bd808
[18:13:54] <bd808>	 all varnish fe nods have the same data paravoid?
[18:13:57] <bd808>	 *nodes
[18:14:07] <ori>	 the parser cache key changed
[18:14:11] <ori>	 so i think they are blank
[18:14:14] <ori>	 okay so plan:
[18:14:19] <bblack>	 all varnish fes fetch from the same eqiad bes ultimately
[18:14:23] <legoktm>	 ori: we could blacklist anything where the contents are just html comments and whitespace?
[18:14:24] <andre__>	 Well, I get a blank one for the GPL article on en.wp, and I'm via esams I guess (Or not, technical people might know better)
[18:14:28] <ori>	 parser cache hook, if article content is blank, rejhect parser cache entry, emit purge
[18:14:32] <paravoid>	 so we can get all the URLs requested during that time period from kafka
[18:14:34] <bblack>	 but there can be timing races involved, so yes fe at two sites can end up with different versions of content
[18:14:37] <krrrit-wm>	 (03PS7) 10Merlijn van Deen: toollabs: add composer to dev hosts [puppet] - 10https://gerrit.wikimedia.org/r/246072 (https://phabricator.wikimedia.org/T104789) 
[18:14:40] <paravoid>	 and issue purges for them, can't we?
[18:14:44] <ori>	 i think what i suggested would work
[18:14:49] <ori>	 that's the hook you hated, bblack :P
[18:14:49] <ostriches>	 ori++
[18:15:15] <ori>	 i did this for the hieroglyph extension once
[18:15:17] <ori>	 let me get the code
[18:15:24] <_joe_>	 ori: seems like that would work, yes. 
[18:15:31] <bblack>	 we could also do something dumb like purge all matching /wiki/ + content-length < X
[18:15:48] <twentyafterfour>	 more dumb stuff +1
[18:15:54] <ori>	 no, that'll match a lot of article redirects
[18:16:00] * twentyafterfour doesn't think we've done enough stupid already
[18:16:00] <bblack>	 + 200
[18:16:17] <apergos>	 do we cache redirects?  I guess so
[18:16:18] <apergos>	 sigh
[18:16:26] <bblack>	 we can match a ban on boolean of anything in the response headers, basically
[18:16:34] <bblack>	 200 + /wiki/ + content-length < X
[18:16:44] <paravoid>	 (please don't make comments if you don't have anything constructive to say)
[18:17:05] <paravoid>	 ori: are you implementing your fix?
[18:17:08] <_joe_>	 bblack: I looked at the resp headers from the gpl page before it was purged, yes I don't see anything else that could work
[18:17:30] <ostriches>	 legoktm: So which extension made it explode? :\
[18:17:31] <legoktm>	 I figured out where I fucked up.
[18:17:33] <jynus>	 LinksUpdate::incrTableUpdate FROM pagelinks
[18:17:35] <_joe_>	 bblack: ofc that is going to generate a lot of false positives
[18:17:42] <legoktm>	 		2497	+»       wfLoadExtension( 'Disambiguator' );
[18:17:42] <legoktm>	 	2497	2498	 »       require_once( "$IP/extensions/Disambiguator/Disambiguator.php" );
[18:17:47] <apergos>	 what's the typical content length of one of these empty-content pages? 
[18:17:52] <legoktm>	 I didn't remove the require_once for Disambiguator
[18:18:10] <ori>	 i'm on a really shitty connection
[18:18:11] <ori>	 https://dpaste.de/hMA8/raw
[18:18:13] <akosiaris>	 apergos: it probably varies quite a lot... language bar, article title...
[18:18:13] <ori>	 is the basic idea
[18:18:16] <ostriches>	 legoktm: We should do these 1 by 1 from now on, rather than batches :\
[18:18:19] <ori>	 that's the code from syntaxhighlight
[18:18:23] <apergos>	 ok scratch that then
[18:18:29] <ori>	 but /* some condition */ needs to be written in
[18:18:30] <krrrit-wm>	 (03CR) 10Legoktm: Load some more extensions directly through wfLoadExtension() (B-D) (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/232966 (owner: 10Legoktm)
[18:18:34] <ostriches>	 ori: Turning that into a legit patch
[18:18:36] <legoktm>	 yeah :/
[18:18:38] <ori>	 thanks
[18:18:48] <wikibugs>	 6operations: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1725971 (10Aklapper) Status update: Currently potential followup actions are being investigated.
[18:18:49] <paravoid>	 ok, so ostriches running point then?
[18:18:58] <greg-g>	 yeah
[18:19:02] <paravoid>	 good, thanks
[18:19:03] <_joe_>	 ok good
[18:19:04] <ori>	 yes, i am on too shitty a connection to commit to anything
[18:19:15] <greg-g>	 thanks opsen, go back to the beach
[18:19:35] <paravoid>	 lol
[18:19:38] <MaxSem>	 anything I can help with?
[18:19:39] <greg-g>	 :)
[18:19:43] <mark>	 it's just jealousy
[18:19:45] <ori>	 bblack: maybe i'm wrong and a ban would be better
[18:19:46] * andre__ takes a walk to the Village Pumps
[18:20:01] <Luke081515>	 andree__: Good Idea
[18:20:03] <greg-g>	 mark: yeah, we hated the idea of you all not working on our stuff, so we paged the heck out of you
[18:20:07] <Luke081515>	 At dewiki this is clear
[18:20:12] <ori>	 the hook would only run as those pages are requested from the app servers, so it would require a page load from a logged in user
[18:20:15] <legoktm>	 was the s1 slave lag related?
[18:20:18] <paravoid>	 greg-g: incident doc for this and yesterday's too, after all of this is settled
[18:20:30] <paravoid>	 greg-g: because I still don't understand how something like this could escape all of our QA
[18:20:34] <jynus>	 the lag was created by the 8 minute query I just pasted
[18:20:42] <ori>	 that'll take care of it eventually, but if a ban on content-length < XXX is not too hard that would be faster
[18:21:03] <ori>	 jynus: pasted where?
[18:21:09] <twentyafterfour>	 paravoid: there isn't much qa on config changes, honestly
[18:21:11] <Coren>	 ori: That's going to do a /lot/ of false positives for any value XXX
[18:21:26] <_joe_>	 20:17 < jynus> LinksUpdate::incrTableUpdate FROM pagelinks
[18:21:27] <paravoid>	 oh so this was just a mw-config change, ok
[18:21:28] <greg-g>	 paravoid: yeah, legoktm will write an incident report for this one, right legoktm ? :P
[18:21:31] <ori>	 Coren: yes, but false positives = backend request, and presumably a cheap one for the false positives
[18:21:39] <legoktm>	 greg-g, paravoid: yeah, I can do that
[18:21:54] <krrrit-wm>	 (03PS1) 10Merlijn van Deen: toollabs: install mailutils [puppet] - 10https://gerrit.wikimedia.org/r/246289 (https://phabricator.wikimedia.org/T114073) 
[18:22:05] <bblack>	 ori: have an idea on length limit?
[18:22:14] <bblack>	 (consider lang variants, skin, etc?)
[18:22:17] <twentyafterfour>	 paravoid: yes not a deployment ...just bad timing (I was about to sync group1 when it blew up)
[18:22:23] <Coren>	 ori: Yeah, I'm just woried that X needs to be fairly high if we want to avoid false negatives and much would be uncachable for a while.
[18:22:26] <bblack>	 I have a ban expression prepped for what we said above
[18:22:47] <_joe_>	 if anyone finds a page which presents the problem, please paste the link here
[18:22:54] <bblack>	 (and don't purge it heh)
[18:23:03] <_joe_>	 yep, exactly :)
[18:23:36] <paravoid>	 did this last from 17:58  to 18:06?
[18:24:01] <paravoid>	 trying to extrapolate from https://wikitech.wikimedia.org/wiki/Server_Admin_Log but all of the deploys say "(no message)" so it's hard to tell
[18:24:18] <ori>	 greg-g: why is legoktm on the hook for writing the postmortem?!
[18:24:22] <aude>	 paravoid: think so
[18:24:46] <legoktm>	 paravoid: 17:59 to 18:06
[18:24:48] <thedj>	 ehm. i just got 0 content in the main page....
[18:24:56] <bblack>	 where?
[18:25:03] <bblack>	 enwiki? from the us or eu?
[18:25:03] <paravoid>	 which main page?
[18:25:06] <paravoid>	 enwiki?
[18:25:06] <thedj>	 sry
[18:25:08] <thedj>	 no
[18:25:08] <greg-g>	 ori: his patch, right?
[18:25:30] <icinga-wm>	 RECOVERY - MariaDB Slave Lag: s1 on db1051 is OK: OK slave_sql_lag Seconds_Behind_Master: 0
[18:25:31] <krrrit-wm>	 (03PS1) 10Chad: Purge pages with blank content [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246290 
[18:25:37] <krrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Purge pages with blank content [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246290 (owner: 10Chad)
[18:25:38] <ostriches>	 ori, others: Let's not rush, but ^^^
[18:26:12] <ostriches>	 Starting with valid php would help
[18:26:21] <thedj>	 sorry, my screen session cached a key up and when connection recovered, i just happen to press enter and my last history entry was posted in IRC :)
[18:26:38] <legoktm>	 ostriches: isn't the newPP report cached?
[18:26:55] <bblack>	 does anyone have the content of a blank article still up in browser, or fetched via curl, etc? for an idea on content-length?
[18:27:06] <legoktm>	 I have one open in my browser
[18:27:09] <legoktm>	 ostriches: http://fpaste.org/279225/84720014/raw/
[18:27:15] <jynus>	 full query (NDA only): https://phabricator.wikimedia.org/P2197
[18:27:36] <akosiaris>	 bblack: https://en.wikipedia.org/wiki/Throw-in
[18:27:48] <wikibugs>	 6operations, 10hardware-requests: codfw/eqiad: (1) eventlogging node (per site) - https://phabricator.wikimedia.org/T90747#1726009 (10RobH)
[18:27:49] <wikibugs>	 6operations: deploy eventlog2001 services - https://phabricator.wikimedia.org/T93220#1726007 (10RobH)
[18:27:51] <wikibugs>	 6operations, 10hardware-requests: deploy eventlog2001 - https://phabricator.wikimedia.org/T90907#1726005 (10RobH) 5Resolved>3Open Service implementation never completed.  I'll need to followup with Ori about this at a later time and find out what it is waiting on.  (He may not have been the proper person t...
[18:28:01] <ostriches>	 legoktm: So if page /starts/ with <!-- new PP stuff
[18:28:04] <ostriches>	 Then ban
[18:28:07] <legoktm>	 yeah
[18:28:32] <legoktm>	 that will also ban empty articles, but they're empty, so it should be cheap anyways
[18:28:34] <thedj>	 there might be a linebreak in front of that...
[18:28:41] <bblack>	 so ~25K for a blank wiki article
[18:30:03] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Add roles for marking unpuppetised test systems and unused spares [puppet] - 10https://gerrit.wikimedia.org/r/246291 (https://phabricator.wikimedia.org/T115349) 
[18:31:31] <krrrit-wm>	 (03PS2) 10Chad: Purge pages with blank content beyond the PP limit report [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246290 
[18:32:22] <ori>	 ostriches: trim() it first
[18:32:25] <ori>	 i think there is some whitespace
[18:32:30] <paravoid>	 anyone remembers what the "exptime" is set to on parser-cache entries?
[18:32:35] <paravoid>	 now() + what?
[18:32:40] <bblack>	 I can do the purge with basically "200 OK" + "/wiki/" + "length < 30K" pretty easy, but if you need to fix parsercache anyways and it's going to emit purges, can just wait on that
[18:32:41] <ori>	 paravoid: it depends on the content
[18:32:43] <ostriches>	 ori: Ah, then I can avoid the || entirely
[18:33:21] <paravoid>	 the data suggests that exptime is creation time?!
[18:33:36] <krrrit-wm>	 (03PS3) 10Chad: Purge pages with blank content beyond the PP limit report [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246290 
[18:33:48] <ori>	 bblack: yeah, i think now that we have a pretty good way of identifying them in that hook, and that we know there isn't an easy way to pick them out in varnish, i think i'm leaning back towards that
[18:33:52] <ori>	 towards the hook, i mean
[18:34:12] <ostriches>	 ori, legoktm: PS3 should do it.
[18:34:15] <paravoid>	 (disregard the comment above)
[18:34:46] <wikibugs>	 6operations: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1726024 (10Aklapper) The software change in https://gerrit.wikimedia.org/r/246290 is the attempt to mass-purge the blank pages.
[18:35:06] <aude>	 twentyafterfour: how long do you think until this is all resolved and we deploy group1?
[18:35:12] <paravoid>	 ori: 21:32 < ori> paravoid: it depends on the content
[18:35:14] <paravoid>	 what does that mean?
[18:35:20] <twentyafterfour>	 aude: apparently next week
[18:35:20] <paravoid>	 I see at least some of them being 30 days
[18:35:35] <paravoid>	 for those, we can just query parsercache and find the relevant IDs
[18:35:36] <bblack>	 ori: ok
[18:35:36] <legoktm>	 depending on what's in the content, it changes the expiry
[18:35:50] <legoktm>	 using {{#time:}} etc lowers it a lot
[18:35:59] <paravoid>	 aha
[18:36:12] <ori>	 ostriches: lgtm; maybe add a wfDebugLog( 'T115505', page title ) ?
[18:36:22] <ori>	 so we can get a handle on the volume
[18:36:27] <ostriches>	 Yeah, good idea.
[18:36:32] <aude>	 twentyafterfour: ?
[18:36:32] <ostriches>	 Was just wondering how to keep an eye on it
[18:36:35] <ori>	 i'll enable the log channel in a separate patch
[18:36:35] <legoktm>	 lgtm as well
[18:36:54] <thedj>	 do all pages allow getText() ?
[18:36:58] <thedj>	 Specials ?
[18:37:04] <_joe_>	 ostriches: just to understand, is this going to clear pc?
[18:37:15] <ori>	 all pages for which the hook would be invoked
[18:37:18] <_joe_>	 (return false to rejectParserCache)?
[18:37:35] <ori>	 _joe_: it's going to discard the parser cache entry and emit a purge to varnish for pages which match the criterion of having blank content
[18:37:57] <Coren>	 ori: Only if it gets there - not for anonymous users.
[18:38:01] <_joe_>	 ori: I was just baffled by the double negation logic (rejectParserCache returns false => pc gets purged)
[18:38:18] <_joe_>	 I do understand what the patch does apart from that
[18:38:23] <ostriches>	 _joe_: That's general MW hook behavior
[18:38:29] <ori>	 it'll purge it for anons as well once a logged in user visits the page
[18:38:35] <ostriches>	 False => Stop what you're doing
[18:38:42] <ostriches>	 True => Keep on doin' what you're doin'
[18:38:50] <ori>	 what ostriches said
[18:39:32] <ostriches>	 And yeah, callers to pcache->get() will recache as they get false returns.
[18:39:42] <ostriches>	 Specifically Article & co, which is what we care about
[18:39:44] <_joe_>	 ok thanks :)
[18:41:59] <krrrit-wm>	 (03PS4) 10Chad: Purge pages with blank content beyond the PP limit report [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246290 
[18:42:06] <ostriches>	 ori: Now, with logging!
[18:42:13] <akosiaris>	 https://en.wikipedia.org/wiki/Allahabad_City_railway_station
[18:42:23] <akosiaris>	 that's the very first blank article on enwiki
[18:42:29] <greg-g>	 aude: is anything broken for you all right now, as it is?
[18:43:12] <krrrit-wm>	 (03PS1) 10Ori.livneh: Enable T115505 log channel for I1d7969a [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246293 
[18:43:37] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Enable T115505 log channel for I1d7969a [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246293 (owner: 10Ori.livneh)
[18:43:43] <krrrit-wm>	 (03Merged) 10jenkins-bot: Enable T115505 log channel for I1d7969a [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246293 (owner: 10Ori.livneh)
[18:44:05] <krrrit-wm>	 (03PS1) 10EBernhardson: Revert "Log messages sent to the 'warning' channel" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246294 
[18:44:36] <krrrit-wm>	 (03PS2) 10BryanDavis: Revert "Log messages sent to the 'warning' channel" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246294 (owner: 10EBernhardson)
[18:44:38] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Purge pages with blank content beyond the PP limit report [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246290 (owner: 10Chad)
[18:44:45] <ori>	 ostriches: gonna sync it to mw1017 first
[18:44:47] <krrrit-wm>	 (03Merged) 10jenkins-bot: Purge pages with blank content beyond the PP limit report [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246290 (owner: 10Chad)
[18:44:47] <legoktm>	 in MagicWord.php: $text = preg_replace( $regex, '', $text );
[18:44:54] <ostriches>	 ori: mmk
[18:44:57] <legoktm>	 preg_replace was issuing warnings and returning NULL
[18:44:59] <greg-g>	 ?
[18:45:04] <krrrit-wm>	 (03CR) 10BryanDavis: [C: 031] "Generated 207G of log data on fluorine in less than 24 hours." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246294 (owner: 10EBernhardson)
[18:45:04] <legoktm>	 which was casted into empty string
[18:45:10] <greg-g>	 what's going on?
[18:45:16] <ori>	 with what?
[18:45:29] <greg-g>	 we're still cleaning up?
[18:45:32] <mutante|1way>	 greg-g: it's not fixed yet
[18:45:36] <greg-g>	 kk
[18:45:44] <greg-g>	 sorry
[18:45:57] <ebernhardson>	 can i deploy a small config change or is that going to get in the way?  A new logging channel turned on yesterday is generating way too much data and needs to be turned off
[18:46:18] <ostriches>	 ebernhardson: Bad timing...
[18:46:19] <ori>	 wait a couple of minutes
[18:46:22] <ebernhardson>	 sure
[18:47:53] <krrrit-wm>	 (03PS1) 10Muehlenhoff: k8s: Move the ferm fules into the role [puppet] - 10https://gerrit.wikimedia.org/r/246295 
[18:51:32] <greg-g>	 ok, I need to go get food before more meetings, be back shortly
[18:54:07] <ostriches>	 ori: we good for full sync?
[18:55:39] <ori>	 nope
[18:55:40] <ori>	 it's not working
[18:55:42] <ori>	 trying to figure out why
[18:57:59] <robh>	 (not for the current fix obviously, ref: deployment freeze in topic)
[18:58:39] <_joe_>	 robh: both fixes I'd say
[18:58:49] <_joe_>	 (the one from erik b too)
[18:59:08] <robh>	 rephrase: not for the current repairs to broken things that caused the freeze to be implemented in the first place
[18:59:38] <robh>	 (i just wanted to put something in topic so when we see merges scroll past those folks are appropriately prodded ;)
[19:01:36] <ori>	 ostriches: haystack, needle :)
[19:02:08] <ostriches>	 ori: herp derp? Backwards?
[19:02:14] * ostriches smacks head
[19:03:16] <krrrit-wm>	 (03PS1) 10Chad: Get strpos() parameters correct [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246296 
[19:03:20] <ostriches>	 ori: ^^^
[19:06:56] <krrrit-wm>	 (03PS2) 10Ori.livneh: Get strpos() parameters correct [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246296 (owner: 10Chad)
[19:07:05] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Get strpos() parameters correct [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246296 (owner: 10Chad)
[19:07:11] <krrrit-wm>	 (03Merged) 10jenkins-bot: Get strpos() parameters correct [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246296 (owner: 10Chad)
[19:09:59] <logmsgbot>	 !log ori@tin Synchronized wmf-config/InitialiseSettings.php: I3a8a750bb3: Enable T115505 log channel for I1d7969a (duration: 00m 17s)
[19:10:04] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:11:06] <logmsgbot>	 !log ori@tin Synchronized wmf-config/CommonSettings.php: I1d7969a9d: Purge pages with blank content beyond the PP limit report, I38cba9d9b6a: Get strpos() parameters correct (duration: 00m 17s)
[19:11:11] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:11:37] <ori>	 ostriches: synced, and purging stuff
[19:11:46] <ori>	 see fluorine:/a/mw-log/T115505.log
[19:12:20] <ori>	 2,668 titles during the first minute
[19:12:30] <_joe_>	 ori: \o/
[19:12:57] <apergos>	 excellent
[19:13:41] <_joe_>	 ori: I expected less purges tbh
[19:14:29] <ostriches>	 ori: We should skip category pages.
[19:14:29] <ori>	 it's partly because of me, i made the check less restrictive than ostriches
[19:14:41] <ostriches>	 Many (most?) have empty content.
[19:14:53] <ori>	 i wasn't sure about the newlines and the exact formatting so instead of the exact match i changed it to search for 'NewPP' in the first 50 chars
[19:14:56] <_joe_>	 ori: isn't it purging all empty categories and blank pages?
[19:15:00] <ori>	 so it's matching https://en.wikipedia.org/wiki/User:Cyberbot_II/Status for example
[19:15:01] * legoktm will bbl for lunch, I'll write the report after that
[19:15:10] <ori>	 which just contains "enable"
[19:16:52] <ori>	 i gotta run too
[19:16:53] <ori>	 ostriches: skipping category pages sounds good, changing back to your exact string start matching might be too
[19:17:12] <ori>	 but if app server load is not too high it's probably ok to just leave it running for an hour or two and then just remove it
[19:17:46] <ori>	 logged in users get around
[19:18:30] <ostriches>	 ori: I'm going to just shorten the length to < 10. In the empty page case, it should be 6.
[19:18:38] <ori>	 that's a good idea
[19:19:13] <ori>	 may as well self-merge and sync if no one is around, it's safe. i gotta run
[19:19:17] <ori>	 thanks for the patch, ttyl
[19:20:27] <krrrit-wm>	 (03PS1) 10Chad: Slightly tighten check on empty pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246299 
[19:21:11] <krrrit-wm>	 (03CR) 10Chad: [C: 032] Slightly tighten check on empty pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246299 (owner: 10Chad)
[19:21:17] <krrrit-wm>	 (03Merged) 10jenkins-bot: Slightly tighten check on empty pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246299 (owner: 10Chad)
[19:21:53] <logmsgbot>	 !log demon@tin Synchronized wmf-config/CommonSettings.php: Tighten check on empty page purges (duration: 00m 18s)
[19:22:00] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:22:06] <ostriches>	 Much slower.
[19:23:55] <apergos>	 purge rate looks really good now
[19:39:53] <wikibugs>	 6operations, 10Wikimedia-General-or-Unknown: security@mediawiki.org : Create a public key and publish it on the public key servers - https://phabricator.wikimedia.org/T40860#1726134 (10Dzahn) The people having access to this key will have to match the people on the security@ mail alias. That includes "root" (a...
[19:41:10] <wikibugs>	 6operations, 10Analytics, 6Discovery, 10EventBus, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1726139 (10Eevans) I've expanded upon @gwicke's prototype a bit, progress here: https://github.com/wikimedia/restevent
[19:45:43] <bd808>	 ebernhardson: probably safe to push that logging change now
[19:48:32] <ebernhardson>	 just getting back from lunch, yea looks all clear
[19:48:40] <krrrit-wm>	 (03CR) 10EBernhardson: [C: 032] Revert "Log messages sent to the 'warning' channel" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246294 (owner: 10EBernhardson)
[19:48:44] <krrrit-wm>	 (03PS1) 10Muehlenhoff: wdqs: Remove superflous include [puppet] - 10https://gerrit.wikimedia.org/r/246307 
[19:49:07] <krrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Log messages sent to the 'warning' channel" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246294 (owner: 10EBernhardson)
[19:50:04] <logmsgbot>	 !log ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off the mw "warning" logging channel (duration: 00m 18s)
[19:50:09] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:51:16] <wikibugs>	 6operations, 7Mail: Google Mail marking Phabricator and Gerrit notification emails as spam - https://phabricator.wikimedia.org/T115416#1726149 (10daniel) You can whitelist mails from wikimedia.org via gmail's web interface: http://email.about.com/od/gmailtips/qt/et_whitelist.htm
[19:51:36] <krrrit-wm>	 (03PS1) 10Muehlenhoff: aqs: Include base::firewall in the role [puppet] - 10https://gerrit.wikimedia.org/r/246330 
[19:52:03] <krrrit-wm>	 (03PS3) 10Dzahn: tor: Move the ferm rules into the role [puppet] - 10https://gerrit.wikimedia.org/r/246223 (owner: 10Muehlenhoff)
[19:52:31] <krrrit-wm>	 (03CR) 10Dzahn: [C: 032] tor: Move the ferm rules into the role [puppet] - 10https://gerrit.wikimedia.org/r/246223 (owner: 10Muehlenhoff)
[19:52:33] <krrrit-wm>	 (03PS1) 10Ori.livneh: Follow-up for Ic9c47594ad: make the top panel transparent [puppet] - 10https://gerrit.wikimedia.org/r/246340 
[19:52:46] <robh>	 !log reenabled puppet agent on etherpad1001, was disabled for a few hours and no reason specified and no SAL entry
[19:52:51] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:52:53] <mutante|1way>	 akosiaris: ^
[19:54:12] <krrrit-wm>	 (03PS2) 10Dzahn: Add roles for marking unpuppetised test systems and unused spares [puppet] - 10https://gerrit.wikimedia.org/r/246291 (https://phabricator.wikimedia.org/T115349) (owner: 10Muehlenhoff)
[19:54:47] <krrrit-wm>	 (03CR) 10Dzahn: [C: 032] Add roles for marking unpuppetised test systems and unused spares [puppet] - 10https://gerrit.wikimedia.org/r/246291 (https://phabricator.wikimedia.org/T115349) (owner: 10Muehlenhoff)
[19:55:18] <krrrit-wm>	 (03PS2) 10Ori.livneh: Follow-up for Ic9c47594ad: make the top panel transparent [puppet] - 10https://gerrit.wikimedia.org/r/246340 
[19:55:23] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Follow-up for Ic9c47594ad: make the top panel transparent [puppet] - 10https://gerrit.wikimedia.org/r/246340 (owner: 10Ori.livneh)
[19:55:34] <akosiaris>	 robh: ah yes, that was me trying to make it less crappy. failed obviously. should have reenabled it 
[19:55:54] <ori>	 akosiaris: btw have you seen http://whiteboard.debian.net/?
[19:56:00] <ori>	 looks simple but nice
[19:56:49] <akosiaris>	 ori: yeah, I did. looks nice
[19:57:24] <akosiaris>	 I 'd like a replacement for etherpad, but tbh I am kind of afraid of any migration process....
[19:57:32] <ori>	 yeah, fair enough
[19:57:38] <akosiaris>	 then again, if users just migrate anyway
[19:57:46] <ori>	 what is the issue with etherpad, do we know?
[19:58:49] <akosiaris>	 I have theories. So, but tailing the logs, it seems like the code (single threaded async, remember) is eating up quite some time somewhere to do something and is effectively blocked and not answering requests
[19:59:07] <akosiaris>	 I base that on the fact that suddenly the normal rate of logging stops
[19:59:17] <akosiaris>	 and suddenly, 20 secs later restarts
[19:59:21] <ori>	 hmmm!
[19:59:37] <mutante|1way>	 block pads with "spell" in the title :p
[19:59:47] <bd808>	 ebernhardson: I'm still seeing the log grow on fluorine.
[19:59:48] <akosiaris>	 lol... not the only problematic pad
[20:00:02] <robh>	 but its the one we dislike the most openly
[20:00:05] <jouncebot>	 gwicke cscott arlolra subbu bearND mdholloway: Respected human, time to deploy Services – Parsoid / OCG / Citoid / Mobileapps / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151014T2000). Please do the needful.
[20:00:19] <James_F>	 No deploys!
[20:00:27] <greg-g>	 bah
[20:00:30] <greg-g>	 jouncebot: reload
[20:00:32] <ori>	 it's a nodejs app right?
[20:00:33] <bearND>	 no deployment today
[20:00:36] <akosiaris>	 btw, there is an issue in etherpad's tracker where it says that pads larger than 700 or so lines are making the software croak and die 
[20:00:38] <akosiaris>	 ori: yes
[20:00:42] <ebernhardson>	 bd808: hmm, that should have done it :S
[20:00:48] <greg-g>	 that entry is no longer on [[deployments]]
[20:00:56] <ori>	 i wonder what it is _joe_ is using here: https://phabricator.wikimedia.org/T114558
[20:00:59] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Make spare role include base::firewall [puppet] - 10https://gerrit.wikimedia.org/r/246388 
[20:01:00] <bd808>	 jouncebot: refresh
[20:01:01] <jouncebot>	 I refreshed my knowledge about deployments.
[20:01:01] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Mark analytics1021 as a spare [puppet] - 10https://gerrit.wikimedia.org/r/246389 
[20:01:07] <ori>	 (task is about parsoid, but he connects to a nodejs instance and gets a backtrace from a running instance)
[20:01:11] <bd808>	 jouncebot: next
[20:01:14] <greg-g>	 jouncebot: next
[20:01:17] <greg-g>	 heh
[20:01:19] <ori>	 jouncebot: next
[20:01:19] <greg-g>	 nothing, good
[20:01:34] * greg-g 'll make schedule for next week later
[20:02:46] <JohnFLewis>	 greg-g: jouncebot should go "it's cold here..." ;)
[20:02:55] <akosiaris>	 ori: node debug I think
[20:03:01] <bd808>	 ebernhardson: looks like all the events are coming from a small number of hosts now so it must have mostly worked
[20:03:31] <krrrit-wm>	 (03PS8) 10Yuvipanda: toollabs: add composer to dev hosts [puppet] - 10https://gerrit.wikimedia.org/r/246072 (https://phabricator.wikimedia.org/T104789) (owner: 10Merlijn van Deen)
[20:03:41] <krrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] toollabs: add composer to dev hosts [puppet] - 10https://gerrit.wikimedia.org/r/246072 (https://phabricator.wikimedia.org/T104789) (owner: 10Merlijn van Deen)
[20:04:47] <akosiaris>	 ori: node debug -p <pid> - Connects to the process via the pid
[20:04:49] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Mark former bits clusters as spares [puppet] - 10https://gerrit.wikimedia.org/r/246390 (https://phabricator.wikimedia.org/T115489) 
[20:05:33] <ostriches>	 Feature request: Jouncebot to say "I know of nothing to deploy"
[20:06:26] <krrrit-wm>	 (03PS1) 10Muehlenhoff: Mark old MXes as spares [puppet] - 10https://gerrit.wikimedia.org/r/246392 (https://phabricator.wikimedia.org/T115489) 
[20:06:48] <ori>	 seems handy
[20:07:35] <ostriches>	 ori: Purge rather is like 20-30/min right now
[20:07:48] <ori>	 yeah, much better
[20:08:11] <ostriches>	 Eh, 50ish this minute, but yeah, 10s, not 100s or 1000s
[20:09:39] <ostriches>	 Also gonna skip TimedText ns. Only affects commons afaict and they're empty
[20:14:19] <krrrit-wm>	 (03CR) 10BBlack: [C: 04-1] Mark former bits clusters as spares [puppet] - 10https://gerrit.wikimedia.org/r/246390 (https://phabricator.wikimedia.org/T115489) (owner: 10Muehlenhoff)
[20:14:25] <krrrit-wm>	 (03CR) 10Faidon Liambotis: "The role spare idea sounds really good, but if I can bikeshed a little, I wonder how that should be ordered compared to the task for retur" [puppet] - 10https://gerrit.wikimedia.org/r/246392 (https://phabricator.wikimedia.org/T115489) (owner: 10Muehlenhoff)
[20:15:32] <krrrit-wm>	 (03CR) 10BBlack: "The bits cluster machines are meant to be re-used to expand the misc cluster into all pops here, but there's some software work left to be" [puppet] - 10https://gerrit.wikimedia.org/r/246390 (https://phabricator.wikimedia.org/T115489) (owner: 10Muehlenhoff)
[20:15:42] <krrrit-wm>	 (03CR) 10BBlack: [C: 04-1] Mark former bits clusters as spares [puppet] - 10https://gerrit.wikimedia.org/r/246390 (https://phabricator.wikimedia.org/T115489) (owner: 10Muehlenhoff)
[20:15:48] <krrrit-wm>	 (03PS1) 10Chad: Also skip TimedText namespace from empty purges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246398 
[20:15:53] <ostriches>	 ori: ^
[20:17:30] <krrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] Also skip TimedText namespace from empty purges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246398 (owner: 10Chad)
[20:17:50] <krrrit-wm>	 (03CR) 10Chad: [C: 032] Also skip TimedText namespace from empty purges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246398 (owner: 10Chad)
[20:17:56] <krrrit-wm>	 (03Merged) 10jenkins-bot: Also skip TimedText namespace from empty purges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246398 (owner: 10Chad)
[20:20:16] <logmsgbot>	 !log demon@tin Synchronized wmf-config/CommonSettings.php: Also exclude timedtext ns from purges (duration: 00m 18s)
[20:20:20] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:20:33] <hashar>	 !sal
[20:20:33] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Server_Admin_Log  https://tools.wmflabs.org/sal/production   See it and you will know all you need.
[20:22:04] <bd808>	 ebernhardson: stopped now with the last sync of settings. Must have been the classically horrible settings cache timestamp problem
[20:22:41] <ori>	 ostriches: lgtm
[20:24:13] <ebernhardson>	 bd808: i ran into that one last week as well, thought i really broke something ...turned out to just need a touch and re-sync
[20:25:26] <bd808>	 yeah. scap tries to fix that by touching on the deploy targets themselves after each sync but somehw it doesn't always work
[20:41:21] <krrrit-wm>	 (03PS1) 10Chad: Log OOM rate and HHVM-non-OOM error rate in statds for graphing [puppet] - 10https://gerrit.wikimedia.org/r/246409 
[20:41:49] <ostriches>	 bd808: jfdi ^, but dunno if I got the syntax right
[20:47:51] <krrrit-wm>	 (03PS1) 10Mforns: Add --percent-lost flag to refinery data check [puppet] - 10https://gerrit.wikimedia.org/r/246412 (https://phabricator.wikimedia.org/T113255) 
[20:58:20] <krrrit-wm>	 (03CR) 10BryanDavis: [C: 04-1] "comments inline" (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/246409 (owner: 10Chad)
[21:01:09] <ostriches>	 bd808: thx will amend
[21:03:17] <bd808>	 ostriches: we can totally test that in beta cluster when its close to see what happens
[21:03:42] <wikibugs>	 6operations, 5Patch-For-Review: Create roles for test systems and spares - https://phabricator.wikimedia.org/T115489#1726416 (10Dzahn) merged this https://gerrit.wikimedia.org/r/#/c/246291/   so these roles exist now
[21:08:13] <ostriches>	 bd808: '[type] == "hhvm" and [message] =~ "request has exceeded memory limit"'
[21:08:18] <ostriches>	 '[type] == "hhvm" and [message] =~ "(?:request has exceeded memory limit)"'
[21:09:13] <bd808>	 ostriches: !~ for the second one
[21:09:22] <ostriches>	 #til
[21:09:27] <bd808>	 http://logstash.net/docs/1.4.2/configuration#conditionals
[21:09:51] <icinga-wm>	 PROBLEM - citoid endpoints health on sca1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[21:10:06] <greg-g>	 ?
[21:10:09] <bd808>	 (the old docs are easier for me to read than then shiny new ones at elastic.co)
[21:10:15] <krrrit-wm>	 (03PS2) 10Chad: Log OOM rate and HHVM-non-OOM error rate in statds for graphing [puppet] - 10https://gerrit.wikimedia.org/r/246409 
[21:10:29] <krrrit-wm>	 (03CR) 10Chad: Log OOM rate and HHVM-non-OOM error rate in statds for graphing (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/246409 (owner: 10Chad)
[21:13:15] <krrrit-wm>	 (03CR) 10BryanDavis: "Let's cherry-pick this to deployment-puppetmaster, apply on deployment-logstash2 and then figure out where the graphite data goes in that " [puppet] - 10https://gerrit.wikimedia.org/r/246409 (owner: 10Chad)
[21:14:50] <icinga-wm>	 PROBLEM - zotero on sca1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:45:57] <greg-g>	 thanks for the report legoktm 
[21:46:04] <legoktm>	 yw
[21:47:09] <greg-g>	 twentyafterfour: just fyi re schedule for next week: https://wikitech.wikimedia.org/wiki/Deployments#Week_of_October_19th
[21:51:58] <wikibugs>	 6operations, 10Analytics, 10Analytics-Cluster, 10Fundraising Tech Backlog, 10Fundraising-Backlog: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1726599 (10atgo) Ping - what do we need to do to get 1:10 into pgheres?
[21:53:41] <icinga-wm>	 RECOVERY - zotero on sca1001 is OK: HTTP OK: HTTP/1.0 200 OK - 62 bytes in 0.014 second response time
[21:53:51] <icinga-wm>	 RECOVERY - citoid endpoints health on sca1001 is OK: All endpoints are healthy
[21:53:59] <akosiaris>	 !log restarted zotero on sca1001. complaining out of memory in logs
[21:54:03] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[21:54:17] <akosiaris>	 zotero(3)(+0000038): Translators: Looking for translators for http://twitter.com/
[21:54:24] <akosiaris>	 whois trying to translate twitter ?
[21:54:58] <akosiaris>	 er, cite twitter
[21:55:17] <akosiaris>	 and zotero(3)(+0000264): Translators: Looking for translators for http://example.com/
[21:55:27] <akosiaris>	 hmmm we need to kill this service....
[21:57:02] <YuviPanda>	 heh
[21:57:32] <wikibugs>	 6operations, 10Analytics, 10Analytics-Cluster, 10Fundraising Tech Backlog, 10Fundraising-Backlog: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1726618 (10Jgreen) >>! In T97676#1726599, @atgo wrote: > Ping - what do we need to do to get 1:10 into pgheres?...
[22:03:22] <icinga-wm>	 PROBLEM - puppet last run on mw2162 is CRITICAL: CRITICAL: puppet fail
[22:03:54] <wikibugs>	 6operations, 10Analytics, 10Analytics-Cluster, 10Fundraising Tech Backlog, 10Fundraising-Backlog: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1726630 (10Jgreen) One comment on the change from 1:100 to 1:10--this means we're collecting something like 10X t...
[22:10:18] <Krenair>	 akosiaris, that's the service that citoid uses IIRC
[22:12:38] <akosiaris>	 Krenair: yup it is
[22:12:44] <akosiaris>	 and it's crappy
[22:13:31] <akosiaris>	 remind me to describe it to you in person in January if you are interested in capturing facial expressions of people in dismay
[22:16:01] <Krenair>	 haha
[22:17:44] <wikibugs>	 6operations, 10Analytics, 10Analytics-Cluster, 10Fundraising Tech Backlog, 10Fundraising-Backlog: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1726675 (10ellery) 1:10 is already much better. Pgheres has a campaign field which is crucial. Are you able to ge...
[22:18:51] <icinga-wm>	 PROBLEM - citoid endpoints health on sca1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[22:23:40] <icinga-wm>	 PROBLEM - zotero on sca1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:23:47] <akosiaris>	 zotero(3)(+0006941): Translators: Looking for translators for http://example.com/thisurldoesntexist
[22:23:50] <akosiaris>	 grrrrrrrrrrr
[22:24:12] <akosiaris>	 someone is attacking zotero I think
[22:24:24] <akosiaris>	 for whatever definition of attacking...
[22:26:10] <akosiaris>	 !log restarted zotero on sca1002
[22:26:15] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:26:51] <icinga-wm>	 RECOVERY - zotero on sca1002 is OK: HTTP OK: HTTP/1.0 200 OK - 62 bytes in 0.010 second response time
[22:27:12] <icinga-wm>	 RECOVERY - citoid endpoints health on sca1002 is OK: All endpoints are healthy
[22:29:30] <wikibugs>	 6operations, 5Patch-For-Review: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1726731 (10greg) Just keeping interested people in the loop: Incident report here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20151014-MediaWiki
[22:30:41] <icinga-wm>	 RECOVERY - puppet last run on mw2162 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[22:34:53] <wikibugs>	 6operations, 10Wikimedia-General-or-Unknown, 5Patch-For-Review: Can't see any page, special:RandomPage gives databse error - https://phabricator.wikimedia.org/T115505#1726747 (10matmarex)
[22:45:17] <greg-g>	 akosiaris: a simple non-existent url causes it to fail?
[23:00:05] <jouncebot>	 RoanKattouw ostriches rmoen Krenair: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151014T2300).
[23:00:17] <Krenair>	 hmm
[23:00:22] <Krenair>	 jouncebot still not updated?
[23:00:27] <RoanKattouw>	 Deployment freeze, right?
[23:00:31] <Krenair>	 yep
[23:00:51] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 5 below the confidence bounds
[23:01:32] <bd808>	 jouncebot: refresh
[23:01:33] <jouncebot>	 I refreshed my knowledge about deployments.
[23:01:38] <bd808>	 jouncebot: next
[23:01:38] <jouncebot>	 In 111 hour(s) and 58 minute(s): Morning SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151019T1500)
[23:01:47] <bd808>	 that's better
[23:03:32] <ebernhardson>	 heh
[23:06:01] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 5 below the confidence bounds
[23:07:01] <MaxSem>	 1621552 Use of undefined constant NS_TIMEDTEXT - assumed 'NS_TIMEDTEXT' in /srv/mediawiki/wmf-config/CommonSettings.php on line 247
[23:07:11] <MaxSem>	 greg-g, ^^ needs fixage
[23:08:11] <greg-g>	 ugh
[23:08:15] <MaxSem>	 err, that's under exemption because https://gerrit.wikimedia.org/r/#/c/246398/1/wmf-config/CommonSettings.php
[23:09:00] <greg-g>	 as in, it's ok to sit a while?
[23:10:15] <MaxSem>	 I would say it makes logs unusable
[23:10:33] <MaxSem>	 also, it doesn't do what it's supposed to do
[23:10:56] <greg-g>	 ostriches: ^
[23:11:32] <ostriches>	 Bleh just hardcode 102
[23:11:53] <ostriches>	 Was afraid of that but thought itd load in time
[23:13:56] <krrrit-wm>	 (03PS1) 10MaxSem: Fix undefined constant [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246434 
[23:14:33] <MaxSem>	 ostriches, ^
[23:14:36] <ostriches>	 Lgtm merge away
[23:15:32] <ostriches>	 Although 102 might not be timedtext on other wikis
[23:15:37] <ostriches>	 Stupid namespaces
[23:16:23] <Krenair>	  /srv/mediawiki-staging/php-1.27.0-wmf.3/extensions/TimedMediaHandler/TimedMediaHandler.php:$wgTimedTextNS = 710;
[23:16:57] <Krenair>	  /srv/mediawiki-staging/php-1.27.0-wmf.3/extensions/TimedMediaHandler/TimedMediaHandler.hooks.php:			define( "NS_TIMEDTEXT", $wgTimedTextNS );
[23:17:02] <Krenair>	  /srv/mediawiki-staging/php-1.27.0-wmf.3/extensions/TimedMediaHandler/TimedMediaHandler.hooks.php:			define( "NS_TIMEDTEXT_TALK", $wgTimedTextNS +1 );
[23:17:12] <MaxSem>	 Krenair, /w/api.php?action=query&meta=siteinfo&format=json&siprop=namespaces
[23:17:40] <MaxSem>	 err on commons
[23:18:01] <icinga-wm>	 RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected
[23:18:01] <MaxSem>	 $ ack TimedTextNS
[23:18:01] <MaxSem>	 wmf-config/CommonSettings.php
[23:18:01] <MaxSem>	 661:		$wgTimedTextNS = 102;
[23:18:15] <ostriches>	 That ^
[23:18:26] <krrrit-wm>	 (03CR) 10MaxSem: [C: 032] Fix undefined constant [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246434 (owner: 10MaxSem)
[23:18:29] <Krenair>	 It's set to 102 on commosn
[23:18:29] <Krenair>	 102 is likely used for other things on other wikis though
[23:18:32] <krrrit-wm>	 (03Merged) 10jenkins-bot: Fix undefined constant [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246434 (owner: 10MaxSem)
[23:19:07] <krrrit-wm>	 (03PS1) 10Chad: Revert "Fix undefined constant" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246435 
[23:19:19] <krrrit-wm>	 (03CR) 10Chad: [C: 032 V: 032] Revert "Fix undefined constant" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246435 (owner: 10Chad)
[23:19:46] <MaxSem>	 remove that fucking ns at all?:P
[23:23:17] <krrrit-wm>	 (03PS1) 10MaxSem: Move shit around to make a define defined [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246438 
[23:23:26] <MaxSem>	 ostriches, ^
[23:23:26] <krrrit-wm>	 (03PS1) 10Chad: Only use ns 102 for purging on commonswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 
[23:24:21] <ostriches>	 Or that.
[23:24:35] <ostriches>	 Actually, NS_TIMEDTEXT won't be defined on any wiki without TMH installed.
[23:24:39] <ostriches>	 I think my patch is better.
[23:24:57] <MaxSem>	 pfft
[23:24:59] <MaxSem>	 okay
[23:25:20] <MaxSem>	 did I already mention how I hate TT? :P
[23:25:53] <krrrit-wm>	 (03Abandoned) 10MaxSem: Move shit around to make a define defined [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246438 (owner: 10MaxSem)
[23:27:08] <krrrit-wm>	 (03PS2) 10Chad: Only use ns 102 for purging on commonswiki and 710 elsewhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 
[23:33:34] <ostriches>	 MaxSem: ^?
[23:34:34] <krrrit-wm>	 (03CR) 10MaxSem: [C: 031] Only use ns 102 for purging on commonswiki and 710 elsewhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 (owner: 10Chad)
[23:34:47] <MaxSem>	 or you want me to deploy?
[23:34:52] <icinga-wm>	 PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds
[23:35:01] <icinga-wm>	 PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 2 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/).
[23:35:34] <Krenair>	 not gonna work...
[23:35:49] <Krenair>	 you're adding strings instead of integers
[23:36:26] <Krenair>	 wait, wtf php
[23:36:32] <Krenair>	 php > var_dump( in_array( 102, [ '102' ] ) );
[23:36:32] <Krenair>	 bool(true)
[23:36:40] <icinga-wm>	 RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212
[23:36:45] <RoanKattouw>	 Try in_array( 102, [ '102' ], true )
[23:36:48] * ostriches sighs
[23:36:57] <Krenair>	 maybe it will then
[23:36:59] <ostriches>	 I hate php.
[23:37:01] <Vito>	 https://phabricator.wikimedia.org/T115541 <-- is this related?
[23:37:22] <ostriches>	 I doubt it?
[23:38:04] <wikibugs>	 6operations, 7Performance: ERR_SPDY_PROTOCOL_ERROR while opening files at commons - https://phabricator.wikimedia.org/T115541#1726950 (10Catrope)
[23:38:26] <Krenair>	 When I find out who continually subscribes me to tasks that I can't view I am gonna be pissed.
[23:38:36] <krrrit-wm>	 (03PS3) 10Chad: Just check for NS_CATEGORY, TimedText will just have to be purged [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 
[23:38:52] <ostriches>	 MaxSem, Krenair, RoanKattouw: lol ^
[23:39:11] <icinga-wm>	 PROBLEM - puppet last run on mw2115 is CRITICAL: CRITICAL: Puppet has 1 failures
[23:39:16] <Krenair>	 Vito, I can load images on commons fine...
[23:39:19] <wikibugs>	 6operations, 7Performance: ERR_SPDY_PROTOCOL_ERROR while opening files at commons - https://phabricator.wikimedia.org/T115541#1726953 (10Vituzzu) Doing a bit more tests it seems to happen with chrome only, I'll try another machine.
[23:39:34] <Vito>	 Krenair: I'm doing some more tests
[23:39:49] <krrrit-wm>	 (03CR) 10MaxSem: Just check for NS_CATEGORY, TimedText will just have to be purged (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 (owner: 10Chad)
[23:39:56] <RoanKattouw>	 Krenair: Wait, I thought most of our policies were "CCed can view"?
[23:40:13] <Krenair>	 security ones are
[23:40:20] <Krenair>	 but that's no problem I can see all security tasks
[23:40:27] <krrrit-wm>	 (03PS4) 10Chad: Just check for NS_CATEGORY, TimedText will just have to be purged [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 
[23:41:01] <MaxSem>	 how many developers does it take to fix a warning? :P
[23:41:15] <Krenair>	 There's some fundraising/CL/procurement stuff around which I can't view. I'm betting one of those people are responsible.
[23:41:36] <krrrit-wm>	 (03CR) 10MaxSem: [C: 031] Just check for NS_CATEGORY, TimedText will just have to be purged [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 (owner: 10Chad)
[23:41:46] <Krenair>	 Of course, if we still had mysql_root_pass and co. I could have found out myself.
[23:41:47] <Vito>	 MaxSem: three, one to create the warning, one to propose the wrong solution and another one to fix it :p
[23:42:00] <krrrit-wm>	 (03CR) 10Chad: [C: 032] Just check for NS_CATEGORY, TimedText will just have to be purged [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 (owner: 10Chad)
[23:42:06] <krrrit-wm>	 (03Merged) 10jenkins-bot: Just check for NS_CATEGORY, TimedText will just have to be purged [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246439 (owner: 10Chad)
[23:42:11] <ostriches>	 MaxSem: If you're already on tin, go for it ^
[23:42:14] <ostriches>	 Otherwise, I can login
[23:42:57] <MaxSem>	 Vito, you forgot the forth: Rasmus Lerdorf wgo created this wonderful language with this wonderful concept of runtime warnings
[23:43:12] <wikibugs>	 6operations, 7Performance: ERR_SPDY_PROTOCOL_ERROR while opening files at commons - https://phabricator.wikimedia.org/T115541#1726963 (10Vituzzu) 5Open>3Resolved a:3Vituzzu It seems to work again, will reopen if it happens again.
[23:44:11] <ostriches>	 MaxSem: Five. Someone to bring me a beer when I finally put this to rest today.
[23:44:42] <Vito>	 MaxSem: afair in some African languages any number >4 is "many", so "many" is a good answer ^^
[23:44:53] <Krenair>	 you forgot the developer pointing out that the solution is wrong but not fixing it
[23:45:12] <MaxSem>	 six!
[23:45:57] <krrrit-wm>	 (03PS3) 10EBernhardson: Drop cirrussearch write jobs after 3 hours of failures [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243744 
[23:45:59] <krrrit-wm>	 (03PS1) 10EBernhardson: Enable config for all three search clusters, but only write to eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246443 (https://phabricator.wikimedia.org/T115434) 
[23:46:17] <Vito>	 and at least two or three lazy developers doing ot comments meanwhile
[23:46:26] <krrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Drop cirrussearch write jobs after 3 hours of failures [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243744 (owner: 10EBernhardson)
[23:46:38] <logmsgbot>	 !log demon@tin Synchronized wmf-config/CommonSettings.php: Nevermind on skipping timed text, extension config is weird (duration: 00m 17s)
[23:46:42] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[23:46:43] <ostriches>	 MaxSem, greg-g: ^
[23:46:45] <krrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Enable config for all three search clusters, but only write to eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246443 (https://phabricator.wikimedia.org/T115434) (owner: 10EBernhardson)
[23:46:47] <MaxSem>	 :)
[23:47:00] <icinga-wm>	 RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge.
[23:47:01] <legoktm>	 tl;dr: extension config is hard
[23:48:47] <ostriches>	 legoktm: Why do you think I only run vanilla MW? :D
[23:48:56] <krrrit-wm>	 (03PS4) 10EBernhardson: Drop cirrussearch write jobs after 3 hours of failures [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243744 
[23:48:58] <krrrit-wm>	 (03PS2) 10EBernhardson: Enable config for all three search clusters, but only write to eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246443 (https://phabricator.wikimedia.org/T115434) 
[23:49:18] <krrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Drop cirrussearch write jobs after 3 hours of failures [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243744 (owner: 10EBernhardson)
[23:49:38] <krrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Enable config for all three search clusters, but only write to eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246443 (https://phabricator.wikimedia.org/T115434) (owner: 10EBernhardson)