[00:00:21] <grrrit-wm>	 (03PS2) 10BryanDavis: beta: honor log sampling and levels for logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181349 
[00:02:42] <grrrit-wm>	 (03CR) 10Legoktm: [C: 04-1] beta: honor log sampling and levels for logstash (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181349 (owner: 10BryanDavis)
[00:04:16] <grrrit-wm>	 (03PS3) 10BryanDavis: beta: honor log sampling and levels for logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181349 
[00:07:20] <grrrit-wm>	 (03CR) 10MaxSem: [C: 031] beta: honor log sampling and levels for logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181349 (owner: 10BryanDavis)
[00:07:25] <icinga-wm>	 RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures  
[00:08:02] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] beta: honor log sampling and levels for logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181349 (owner: 10BryanDavis)
[00:08:26] <icinga-wm>	 RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures  
[00:09:05] <icinga-wm>	 RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:19] <grrrit-wm>	 (03PS3) 10BryanDavis: monolog: honor log sampling for logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181350 
[00:34:41] <grrrit-wm>	 (03PS4) 10BryanDavis: monolog: honor log sampling and levels for logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181350 
[00:37:35] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 04-2] "Needs Icd14fc8c44ca9eef0f3f5cc4f1d1d8b68d517f07 on group0. (Should be in 1.25wmf14)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181350 (owner: 10BryanDavis)
[00:52:25] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 338 seconds  
[00:52:29] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 344 seconds  
[00:54:35] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds  
[00:54:46] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds  
[00:55:14] <Krenair>	 -0 seconds :D
[01:45:16] <Nemo_bis>	 Better than 0-, which would be worrying
[03:43:06] <icinga-wm>	 PROBLEM - very high load average likely xfs on ms-be2003 is CRITICAL: CRITICAL - load average: 106.43, 100.16, 98.41  
[03:57:16] <icinga-wm>	 PROBLEM - very high load average likely xfs on ms-be2003 is CRITICAL: CRITICAL - load average: 104.82, 100.57, 98.75  
[04:01:56] <icinga-wm>	 PROBLEM - very high load average likely xfs on ms-be2003 is CRITICAL: CRITICAL - load average: 100.50, 100.83, 99.30  
[04:18:16] <icinga-wm>	 PROBLEM - very high load average likely xfs on ms-be2003 is CRITICAL: CRITICAL - load average: 108.41, 102.34, 99.99  
[04:28:05] <icinga-wm>	 PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: Puppet has 1 failures  
[04:40:36] <icinga-wm>	 PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 1 failures  
[04:45:46] <icinga-wm>	 RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures  
[04:58:15] <icinga-wm>	 RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures  
[05:19:35] <icinga-wm>	 PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: puppet fail  
[05:39:25] <icinga-wm>	 RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures  
[06:28:16] <icinga-wm>	 PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: puppet fail  
[06:29:15] <icinga-wm>	 PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:35] <icinga-wm>	 PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 3 failures  
[06:29:46] <icinga-wm>	 PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 3 failures  
[06:29:46] <icinga-wm>	 PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:46] <icinga-wm>	 PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:56] <icinga-wm>	 PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:30:06] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 2 failures  
[06:44:15] <icinga-wm>	 RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures  
[06:45:35] <icinga-wm>	 RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures  
[06:45:55] <icinga-wm>	 RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures  
[06:46:06] <icinga-wm>	 RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures  
[06:46:25] <icinga-wm>	 RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures  
[06:46:25] <icinga-wm>	 RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures  
[06:46:35] <icinga-wm>	 RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures  
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures  
[08:04:36] <icinga-wm>	 PROBLEM - Apache HTTP on mw1113 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[08:04:46] <icinga-wm>	 PROBLEM - HHVM rendering on mw1113 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[11:32:27] <icinga-wm>	 PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: puppet fail  
[11:52:17] <icinga-wm>	 RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures  
[12:55:15] <icinga-wm>	 PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.01  
[13:05:35] <icinga-wm>	 RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0  
[15:19:07] <grrrit-wm>	 (03PS1) 10Springle: upgrade db1061 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/182419 
[15:19:27] <springle>	 !Log upgrade db1061 trusty
[15:19:33] <morebots>	 Logged the message, Master
[15:19:33] <springle>	 !log upgrade db1061 trusty
[15:21:07] <grrrit-wm>	 (03CR) 10Springle: [C: 032] upgrade db1061 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/182419 (owner: 10Springle)
[15:55:23] <grrrit-wm>	 (03Abandoned) 10Amire80: Open Special:ContentTranslation in the target wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/147918 (owner: 10Amire80)
[16:27:21] <icinga-wm>	 PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Puppet has 1 failures  
[16:43:51] <icinga-wm>	 RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures  
[16:57:57] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "This might make it too slow. Defering until we move check_sslxNN to be an nrpe check." [puppet] - 10https://gerrit.wikimedia.org/r/182306 (owner: 10Faidon Liambotis)
[17:34:18] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "/var/run/motd (or /run/motd.dynamic, depending on the release) isn't state. It's just a temp file to work around PAM motd limitations. pam" [puppet] - 10https://gerrit.wikimedia.org/r/182373 (owner: 10Faidon Liambotis)
[17:38:18] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "As I explain in the comments, these two vary between different OS releases. precise/trusty don't have 99footer (but our debian-installer c" [puppet] - 10https://gerrit.wikimedia.org/r/182374 (owner: 10Faidon Liambotis)
[17:40:21] <icinga-wm>	 PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.0199335548173  
[17:40:28] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "For some of them it doesn't matter, you're right (for others it does, e.g. the intention is to print roles in the start of the motd and th" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182376 (owner: 10Faidon Liambotis)
[17:45:04] <grrrit-wm>	 (03PS9) 10Yuvipanda: Refactor to not be a big ball of mud [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182164 
[17:45:21] <icinga-wm>	 RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0  
[17:45:57] <paravoid>	 I like that commit message :)
[17:46:02] <YuviPanda>	 Heh
[17:47:32] <YuviPanda>	 it was very prototype-y script-y code, rewriting it into something that can actually be called ‘production’ now
[18:30:45] <grrrit-wm>	 (03PS10) 10Yuvipanda: Refactor to not be a big ball of mud [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182164 
[18:46:32] <icinga-wm>	 PROBLEM - Ubuntu mirror in sync with upstream on carbon is CRITICAL: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 12 hours old.  
[18:47:41] <icinga-wm>	 RECOVERY - Ubuntu mirror in sync with upstream on carbon is OK: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 0 hours old.  
[18:55:35] <YuviPanda>	 didn’t even know we had that check
[21:14:13] <grrrit-wm>	 (03PS11) 10Yuvipanda: Refactor to not be a big ball of mud [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182164 
[21:37:12] <YuviPanda>	 valhallasw`cloud: you are right, the decorator is making things complexer. I shall remove
[21:38:11] <valhallasw`cloud>	 YuviPanda: not going for the class option after all?
[21:38:29] <YuviPanda>	 valhallasw`cloud: not yet. I’m just going to make them functions for now, and split them into files.
[21:38:34] <valhallasw`cloud>	 ah ok
[21:44:04] <YuviPanda>	 valhallasw`cloud: with description provided by docstring and name by function name, I’m not sure if they should be objects
[21:44:53] <grrrit-wm>	 (03PS12) 10Yuvipanda: Refactor to not be a big ball of mud [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182164 
[21:46:55] <valhallasw`cloud>	 YuviPanda: mmmm
[21:47:04] <valhallasw`cloud>	 that's dirty
[21:47:05] <valhallasw`cloud>	 I like it
[21:47:13] <YuviPanda>	 valhallasw`cloud: what’s dirty?
[21:47:18] <YuviPanda>	 __doc__ and name?
[21:47:19] <YuviPanda>	 noooo
[21:47:21] <YuviPanda>	 that’s not dirty :D
[21:47:35] <valhallasw`cloud>	 yes it is :P
[21:48:13] <valhallasw`cloud>	 if you kill the decorator and use rr.add_report(databases) instead I'll forgive you :P
[21:48:44] <YuviPanda>	 valhallasw`cloud: I might end up having to do that since I can’t really use the decorator when I’m splitting up the functions
[21:49:47] <YuviPanda>	 valhallasw`cloud: not dirty :P
[21:50:06] <valhallasw`cloud>	 YuviPanda: this is not what they're for :P
[21:50:16] <valhallasw`cloud>	 YuviPanda: the clean solution is a class, with two properties and a function :P
[21:50:55] <YuviPanda>	 noooooo :P
[21:50:58] <YuviPanda>	 well, maybe
[21:51:08] <YuviPanda>	 valhallasw`cloud: I don’t want to switch to a class that’ll never be used more than once.
[21:51:19] <YuviPanda>	 well, never be instantiated more than once, and has no instance state
[21:51:28] <YuviPanda>	 that just seems wrong
[21:51:33] <YuviPanda>	 and there has to be a more elegant solution
[21:51:36] <valhallasw`cloud>	 YuviPanda: you can instantiate it once for every database, if you want :P
[21:51:40] <YuviPanda>	 this one is elegantish
[21:51:47] <valhallasw`cloud>	 this one is magic
[21:51:48] <YuviPanda>	 valhallasw`cloud: why….
[21:51:55] <YuviPanda>	 valhallasw`cloud: __doc__ isn’t magic
[21:52:05] <YuviPanda>	 it’s just taking advantage of the dynamic nature of the language :D
[21:52:15] <YuviPanda>	 desc -> __doc__ maps cleanly
[21:52:30] <valhallasw`cloud>	 YuviPanda: no, it's abusing a language feature. __doc__ is not meant to be a description for a report, it's supposed to be documentation for a function
[21:52:50] <YuviPanda>	 the documentation of this function is helpfully used as a description of the report :P
[21:55:06] <YuviPanda>	 valhallasw`cloud: this is definitely better now than it was earlier in the day :)
[21:55:13] <YuviPanda>	 valhallasw`cloud: and still better than how it was before this patchset
[21:55:19] <valhallasw`cloud>	 *nod*
[21:56:49] <wikibugs>	 3ops-ulsfo: Dear ulsfo@rt.wikimedia.org, No Publication Fee for AASCIT Members - https://phabricator.wikimedia.org/T85678#952101 (10emailbot)
[21:57:26] <valhallasw`cloud>	 :D :( :'(
[22:02:33] <YuviPanda>	 valhallasw`cloud: I gave it a token
[22:06:00] <Krenair>	 YuviPanda, what happens when you file a ticket as "Private Issue"?
[22:06:00] <Krenair>	 you (author) can view it, and WMF-NDA members can view it?
[22:08:05] <YuviPanda>	 Krenair: not sure at all...
[22:13:37] <Nemo_bis>	 Krenair: did you already check the security bugs cc bug?
[22:21:51] <Krenair>	 Nemo_bis, how is that relevant?
[22:22:34] <Krenair>	 oh they're discussing the difference there, I think?
[22:27:14] <Nemo_bis>	 Upstream maintains that reporters are not special subscribers
[22:28:20] <valhallasw`cloud>	 Nemo_bis: sounds like a reasonable stance? Anything else can be solved with Harald rules anyway, I think?
[22:32:25] <Nemo_bis>	 I wasn't expressing an opinion
[22:33:41] <icinga-wm>	 PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 1 failures  
[22:51:11] <icinga-wm>	 RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures  
[22:57:32] <JohnFLewis>	 robh: around?
[23:17:09] <grrrit-wm>	 (03PS13) 10Yuvipanda: Refactor to not be a big ball of mud [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182164 
[23:19:35] <grrrit-wm>	 (03PS14) 10Yuvipanda: Refactor to not be a big ball of mud [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182164 
[23:25:32] <icinga-wm>	 PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: puppet fail  
[23:45:22] <icinga-wm>	 RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures