[13:33:33] <icinga-wm>	 PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 59.09% of data above the critical threshold [5000000.0]
[13:44:13] <icinga-wm>	 RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 50.00% above the threshold [1000000.0]
[14:15:54] <wikibugs>	 6Operations, 10DBA: Email spam from some MariaDB's logrotate - https://phabricator.wikimedia.org/T127638#2049492 (10jcrespo)
[14:20:11] <wikibugs>	 6Operations, 10DBA: Email spam from some MariaDB's logrotate - https://phabricator.wikimedia.org/T127638#2049496 (10jcrespo) Related reading: https://www.percona.com/blog/2014/11/12/log-rotate-and-the-deleted-mysql-log-file-mystery/ https://www.percona.com/blog/2013/04/18/rotating-mysql-slow-logs-safely/
[14:22:58] <grrrit-wm>	 (03PS1) 10BBlack: codfw: move cache_parsoid nodes to cache_upload [puppet] - 10https://gerrit.wikimedia.org/r/272318 (https://phabricator.wikimedia.org/T110472) 
[14:39:02] <jynus>	 !log restarting db2070 to test configuration change
[14:39:07] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[14:54:47] <twkozlowski>	 morning.
[14:55:12] <twkozlowski>	 Error: 503, Service Unavailable at Sun, 21 Feb 2016 14:54:29 GMT
[14:55:59] <twkozlowski>	 Request from 10.20.0.165 via cp3007 cp3007 ([10.20.0.107]:3128), Varnish XID 142017622
[14:58:30] <grrrit-wm>	 (03CR) 10Thiemo Mättig (WMDE): [C: 04-1] "+1 for adding the sandbox properties. -1 because this removes a property that is (currently) not deleted." (034 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/271971 (owner: 10Matěj Suchánek)
[15:00:55] <Krenair>	 twkozlowski, consistently getting this error?
[15:01:06] <twkozlowski>	 Krenair: No, just a one-off
[15:01:20] <twkozlowski>	 which is why I joned the channel hoping not to see a flood of warning messages
[15:08:45] <grrrit-wm>	 (03CR) 10Matěj Suchánek: "It has no usage, we are only waiting for ruwiki to update tteir template. No usage = no entries in the table." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/271971 (owner: 10Matěj Suchánek)
[15:17:03] <grrrit-wm>	 (03PS2) 10Matěj Suchánek: Update Wikidata property blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/271971 
[15:19:04] <grrrit-wm>	 (03CR) 10Matěj Suchánek: Update Wikidata property blacklist (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/271971 (owner: 10Matěj Suchánek)
[15:19:52] <icinga-wm>	 PROBLEM - Kafka Broker Replica Max Lag on kafka1013 is CRITICAL: CRITICAL: 60.87% of data above the critical threshold [5000000.0]
[15:20:13] <icinga-wm>	 PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 64.00% of data above the critical threshold [5000000.0]
[15:30:53] <icinga-wm>	 RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 50.00% above the threshold [1000000.0]
[15:34:03] <icinga-wm>	 RECOVERY - Kafka Broker Replica Max Lag on kafka1013 is OK: OK: Less than 50.00% above the threshold [1000000.0]
[16:43:00] <grrrit-wm>	 (03PS1) 10BBlack: parsoidcache: remove from LVS [puppet] - 10https://gerrit.wikimedia.org/r/272322 (https://phabricator.wikimedia.org/T110472) 
[16:43:02] <grrrit-wm>	 (03PS1) 10BBlack: decom cache_parsoid [puppet] - 10https://gerrit.wikimedia.org/r/272323 (https://phabricator.wikimedia.org/T110472) 
[17:30:37] <kaldari_>	 Just got a DB connection error on enwiki if anyone's interested: Error: 2013 Lost connection to MySQL server during query (10.64.32.21)
[17:33:59] <hoo>	 kaldari_: We have quite some of these (see tendril or error logs)
[17:34:06] <hoo>	 If it's one off... ignore :S
[17:34:19] <kaldari_>	 cool, I'm just paranoid :)
[17:38:15] <grrrit-wm>	 (03CR) 10Hoo man: [C: 04-1] "Otherwise the list looks fine to me. Can deploy this tomorrow." (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/271971 (owner: 10Matěj Suchánek)
[17:47:07] <grrrit-wm>	 (03PS3) 10Matěj Suchánek: Update Wikidata property blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/271971 
[17:53:14] <grrrit-wm>	 (03CR) 10Hoo man: [C: 031] Update Wikidata property blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/271971 (owner: 10Matěj Suchánek)
[21:03:33] <icinga-wm>	 PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 1 failures
[21:28:24] <icinga-wm>	 RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[21:50:24] <grrrit-wm>	 (03PS1) 10BryanDavis: Tools: Add dev packages needed to compile python-ldap [puppet] - 10https://gerrit.wikimedia.org/r/272415 (https://phabricator.wikimedia.org/T114388) 
[21:52:24] <grrrit-wm>	 (03PS2) 10BryanDavis: Tools: Add dev packages needed to compile python-ldap [puppet] - 10https://gerrit.wikimedia.org/r/272415 (https://phabricator.wikimedia.org/T114388)