[00:00:04] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[00:00:04] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[00:07:43] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 00:07:35 UTC 2013
[00:07:53] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 00:07:44 UTC 2013
[00:08:04] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[00:08:04] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[00:08:13] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 00:08:02 UTC 2013
[00:08:23] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 00:08:14 UTC 2013
[00:09:03] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[00:09:04] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[00:09:13] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 00:09:03 UTC 2013
[00:09:13] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 00:09:10 UTC 2013
[00:10:09] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[00:10:09] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[00:11:01] <mutante>	 !log DNS update - push reverse IPv6 for zirconium
[00:11:11] <morebots>	 Logged the message, Master
[00:12:53] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 00:12:52 UTC 2013
[00:13:03] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[00:17:33] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 207 seconds
[00:18:13] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 239 seconds
[00:18:20] <gerrit-wm>	 New patchset: Lcarr; "ms1-4 are decommissioned" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45692
[00:18:57] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45692
[00:19:20] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 308 seconds
[00:19:29] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 316 seconds
[00:20:13] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds
[00:20:33] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds
[00:20:40] <LeslieCarr>	 the above icinga freshness checks flapping are because the machines are both in decomissioning and have classes on them so they are unhappy - removed the classes from ms1 and ms2
[00:21:08] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds
[00:21:17] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 4 seconds
[00:25:09] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45683
[00:26:45] <gerrit-wm>	 New patchset: Dzahn; "add interface_add_ip6_mapped for zirconium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45695
[00:27:29] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45664
[00:27:33] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45695
[00:27:41] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45666
[00:35:05] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[00:38:23] <mutante>	 !log DNS update - set non-autoconfig IPv6 and reverse for zirconium
[00:38:34] <morebots>	 Logged the message, Master
[00:46:26] <gerrit-wm>	 New review: preilly; "nice!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45688
[01:00:02] <gerrit-wm>	 New patchset: Pyoungmeister; "mariadb pacakges. but only for those who really want them" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45698
[01:02:43] <preilly>	 rfaulkner: can you fix sartoris.py:135:80: E501 line too long (95 characters)
[01:03:07] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45698
[01:03:58] <preilly>	 notpeter: you do know that we can hear you right?
[01:04:21] <notpeter>	 what am I saying?
[01:04:24] <notpeter>	 also, i don't care
[01:04:27] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[01:06:30] <dschoon>	 preilly: http://riemann.io/
[01:07:05] <mutante>	 !log planet is now an IPv6-enabled service
[01:07:15] <morebots>	 Logged the message, Master
[01:08:00] <notpeter>	 binasher: we have mariadb packages again
[01:08:21] <Susan>	 mutante: \o/
[01:08:22] <notpeter>	 and only for boxes that actually want them.
[01:12:17] <preilly>	 mutante: nice work
[01:13:27] <mutante>	 thanks. now i just need the wildcard SSL cert
[01:13:44] <rfaulkner>	 preilly: https://gerrit.wikimedia.org/r/45702
[01:14:57] <mutante>	 notpeter: i saw you working on db1047.. and found an oooold ticket.. i guess it can be closed..it was "Cannot reassign variable read_only at /etc/puppet/manifests/mysql.pp:149 on node db1047.eqiad.wmnet" ...in 2011 though
[01:15:27] <binasher>	 !log moved s[2-7] analytics slave cnames to pmtpa dbs
[01:15:39] <morebots>	 Logged the message, Master
[01:15:57] <mutante>	 notpeter: linked it to the upgrade ticket
[01:15:59] <notpeter>	 mutante: link?
[01:16:15] <mutante>	 https://rt.wikimedia.org/Ticket/Display.html?id=1661
[01:16:43] <notpeter>	 mutante: I'm asusming that that was a "puppet is broken" ticket
[01:16:47] <notpeter>	 it is no longer broken
[01:16:52] <notpeter>	 thus, can be closed
[01:17:23] <mutante>	 thanks, done:)
[01:17:56] <preilly>	 rfaulkner: New patchset: preilly; "Conform to PEP 8" [sartoris] (master) - https://gerrit.wikimedia.org/r/45703
[01:19:39] <preilly>	 Ryan_Lane: you need to rebase https://gerrit.wikimedia.org/r/#/c/45701/
[01:19:48] <rfaulkner>	 preilly: cool, pulled the latest
[01:20:20] <preilly>	 rfaulkner: thanks
[01:20:44] <preilly>	 binasher: https://gerrit.wikimedia.org/r/#/q/owner:+%2522preilly%2522,n,z
[01:20:55] <Ryan_Lane>	 preilly: rebased
[01:24:20] <preilly>	 Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/45705/1/sartoris/sartoris.py
[01:27:32] <rfaulkner>	 preilly: small fix, https://gerrit.wikimedia.org/r/45706
[01:27:52] <gerrit-wm>	 New patchset: Asher; "fix logging and clarified reporting per-shard" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45707
[01:36:16] <preilly>	 dschoon: https://gerrit.wikimedia.org/r/gitweb?p=sartoris.git;a=tree;
[01:36:36] <preilly>	 dschoon: or https://github.com/wikimedia/sartoris
[01:42:33] <gerrit-wm>	 New patchset: Asher; "cleanup for greater pep8 compliance" [operations/software] (master) - https://gerrit.wikimedia.org/r/45710
[01:44:43] <average_drifter>	 hello guys
[01:44:47] <average_drifter>	 I'm from WMF analytics
[01:44:50] <average_drifter>	 I need access to analytics1001.wikimedia.org
[01:45:20] <gerrit-wm>	 Change merged: Asher; [operations/software] (master) - https://gerrit.wikimedia.org/r/45710
[01:45:27] <dschoon>	 average_drifter: you need to talk to andrew :)
[01:45:35] <dschoon>	 he's out for the day, though
[01:45:47] <dschoon>	 if there's something you need, i can probably help
[01:47:52] <average_drifter>	 dschoon: just access to an01
[01:58:48] <preilly>	 Ryan_Lane: I was talking about pathogen
[01:59:09] <Ryan_Lane>	 ah. I'll have to try that
[02:00:07] <preilly>	 Ryan_Lane: set foldmethod=indent
[02:00:07] <preilly>	 set foldlevel=99
[02:01:15] <preilly>	 Ryan_Lane: take a look at: http://sontek.net/blog/detail/turning-vim-into-a-modern-python-ide
[02:07:01] <nagios-wm>	 PROBLEM - Puppet freshness on stat1001 is CRITICAL: Puppet has not run in the last 10 hours
[02:11:16] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[02:11:45] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[02:12:09] <preilly>	 dschoon: I just made a simple file: https://github.com/wikimedia/sartoris/blob/master/setup.py
[02:12:14] <preilly>	 dschoon: you can totally fix it
[02:12:24] <dschoon>	 yeep.
[02:13:45] <icinga-wm>	 PROBLEM - Puppet freshness on stat1001 is CRITICAL: Puppet has not run in the last 10 hours
[02:14:34] <preilly>	 dschoon: that's just a hack for Python 3
[02:14:44] <dschoon>	 isn't it always false?
[02:14:48] <dschoon>	 if False and ...:
[02:15:00] <preilly>	 dschoon: try it and you'll see
[02:15:12] <dschoon>	 mk.
[02:21:10] <Ryan_Lane>	 AaronSchulz, dschoon: http://etherpad.wmflabs.org/pad/p/weird-python
[02:27:21] <logmsgbot>	 !log LocalisationUpdate completed (1.21wmf8) at Fri Jan 25 02:27:20 UTC 2013
[02:27:33] <morebots>	 Logged the message, Master
[02:28:15] <dschoon>	 Ryan_Lane: huh, yeah. you were right. i had it backward. https://gist.github.com/4631211
[02:28:21] <dschoon>	 AaronSchulz ^^
[02:28:54] <Ryan_Lane>	 weird construct, right? :)
[02:29:08] <Ryan_Lane>	 the process killing is the only use case I've found so far
[02:30:25] <RoanKattouw>	 Hah, you can do while...else
[02:30:43] <RoanKattouw>	 ?
[02:33:20] <Ryan_Lane>	 RoanKattouw: yeah. it's interesting, right?
[02:47:23] <rfaulkner>	 preilly: https://gerrit.wikimedia.org/r/45717
[02:47:32] <preilly>	 dschoon: http://pastebin.mozilla.org/2084375
[02:47:45] <dschoon>	 yeah, i didn't run it through pep8.
[02:48:51] <preilly>	 dschoon: can you fix it?
[02:49:09] <dschoon>	 yeah; i'll make a note to get to it
[02:49:22] <preilly>	 dschoon: I'll just fix it
[02:49:34] <dschoon>	 sorry, spent enough time on it already :)
[02:53:01] <logmsgbot>	 !log LocalisationUpdate completed (1.21wmf7) at Fri Jan 25 02:53:00 UTC 2013
[02:53:11] <morebots>	 Logged the message, Master
[03:20:02] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[03:20:32] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[04:07:42] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 04:07:37 UTC 2013
[04:07:52] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 04:07:49 UTC 2013
[04:07:53] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[04:08:03] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[04:08:12] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 04:08:06 UTC 2013
[04:08:13] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 04:08:07 UTC 2013
[04:08:52] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[04:09:02] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 04:08:52 UTC 2013
[04:09:03] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[04:09:13] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 04:09:08 UTC 2013
[04:09:52] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[04:10:02] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[04:17:32] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 04:17:22 UTC 2013
[04:17:53] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[04:21:43] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 04:21:40 UTC 2013
[04:22:02] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[04:55:48] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours
[05:10:51] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[05:22:30] <nagios-wm>	 PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours
[05:22:31] <nagios-wm>	 PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours
[05:22:31] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[05:22:31] <nagios-wm>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[05:22:31] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: Puppet has not run in the last 10 hours
[05:22:31] <nagios-wm>	 PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours
[05:22:31] <nagios-wm>	 PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours
[05:22:32] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[06:19:23] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[06:26:23] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 06:26:22 UTC 2013
[06:26:44] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 06:26:32 UTC 2013
[06:26:54] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[06:27:03] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 06:26:56 UTC 2013
[06:27:23] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[06:27:54] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[06:35:43] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 06:35:34 UTC 2013
[06:35:53] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[06:36:38] <nagios-wm>	 PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours
[06:45:43] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 06:45:35 UTC 2013
[06:46:24] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[08:02:22] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[08:02:41] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[08:07:42] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 08:07:37 UTC 2013
[08:07:52] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 08:07:42 UTC 2013
[08:08:21] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[08:08:41] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[08:10:13] <nagios-wm>	 PROBLEM - Puppet freshness on db1031 is CRITICAL: Puppet has not run in the last 10 hours
[08:12:19] <nagios-wm>	 PROBLEM - Puppet freshness on db1037 is CRITICAL: Puppet has not run in the last 10 hours
[08:15:10] <nagios-wm>	 PROBLEM - Puppet freshness on db1012 is CRITICAL: Puppet has not run in the last 10 hours
[08:17:16] <nagios-wm>	 PROBLEM - Puppet freshness on db1015 is CRITICAL: Puppet has not run in the last 10 hours
[08:17:17] <nagios-wm>	 PROBLEM - Puppet freshness on db1014 is CRITICAL: Puppet has not run in the last 10 hours
[08:18:10] <nagios-wm>	 PROBLEM - Puppet freshness on db1023 is CRITICAL: Puppet has not run in the last 10 hours
[08:19:13] <nagios-wm>	 PROBLEM - Puppet freshness on db1030 is CRITICAL: Puppet has not run in the last 10 hours
[08:19:22] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 08:19:16 UTC 2013
[08:19:41] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[08:29:48] <nagios-wm>	 PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours
[08:32:48] <nagios-wm>	 PROBLEM - Puppet freshness on db1044 is CRITICAL: Puppet has not run in the last 10 hours
[08:32:48] <nagios-wm>	 PROBLEM - Puppet freshness on db1045 is CRITICAL: Puppet has not run in the last 10 hours
[08:35:48] <nagios-wm>	 PROBLEM - Puppet freshness on db1016 is CRITICAL: Puppet has not run in the last 10 hours
[10:02:44] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[10:02:44] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[10:08:13] <icinga-wm>	 PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours
[10:37:33] <icinga-wm>	 PROBLEM - Memcached on virt0 is CRITICAL: Connection refused
[10:39:07] <nagios-wm>	 PROBLEM - Memcached on virt0 is CRITICAL: Connection refused
[10:58:28] <nagios-wm>	 RECOVERY - Memcached on virt0 is OK: TCP OK - 0.032 second response time on port 11000
[10:58:33] <icinga-wm>	 RECOVERY - Memcached on virt0 is OK: TCP OK - 0.027 second response time on port 11000
[11:45:37] <Jarry1250>	 Does being "on duty" mean that LeslieCarr is online? Anyone else from ops?
[11:48:00] <gerrit-wm>	 New patchset: Hashar; "move PHP linter under `wikimedia` module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/29937
[11:48:00] <gerrit-wm>	 New patchset: Hashar; "refactor continuous integration manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43429
[11:48:01] <gerrit-wm>	 New patchset: Hashar; "wikimedia module placeholder" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43420
[11:48:20] <Jarry1250>	 (I don't get an "away" indicator on this client)
[11:48:41] <Jarry1250>	 In any case, what should I tell the people at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Users_reporting_site_time_issues_and_delay_in_visible_update_of_edits
[12:06:07] <gerrit-wm>	 New review: Hashar; "rebased, fixed a few linting issues" [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/29937
[12:07:50] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 12:07:43 UTC 2013
[12:07:50] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 12:07:46 UTC 2013
[12:08:08] <nagios-wm>	 PROBLEM - Puppet freshness on stat1001 is CRITICAL: Puppet has not run in the last 10 hours
[12:08:39] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[12:08:40] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[12:08:50] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 12:08:47 UTC 2013
[12:08:57] <gerrit-wm>	 New patchset: Hashar; "move PHP linter under `wikimedia` module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/29937
[12:08:58] <gerrit-wm>	 New patchset: Hashar; "refactor continuous integration manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43429
[12:08:58] <gerrit-wm>	 New patchset: Hashar; "wikimedia module placeholder" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43420
[12:08:59] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 12:08:49 UTC 2013
[12:09:39] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[12:09:40] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[12:09:43] <gerrit-wm>	 New review: Hashar; "PS4 re removes the init.pp placeholder" [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/43420
[12:10:32] <gerrit-wm>	 New review: Hashar; "rebased on latest production branch" [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/43429
[12:14:30] <icinga-wm>	 PROBLEM - Puppet freshness on stat1001 is CRITICAL: Puppet has not run in the last 10 hours
[13:05:22] <gerrit-wm>	 New review: Faidon; "The check is crap, but the change itself is sane." [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/43843
[13:05:23] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43843
[14:56:37] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours
[15:12:46] <icinga-wm>	 PROBLEM - Host ms-be1008 is DOWN: PING CRITICAL - Packet loss = 100%
[15:15:11] <nagios-wm>	 PROBLEM - Host ms-be1008 is DOWN: PING CRITICAL - Packet loss = 100%
[15:20:28] <cmjohnson1>	 paravoid: ms-be1008 raid setup....going to installer now
[15:21:02] <nagios-wm>	 RECOVERY - Host ms-be1008 is UP: PING WARNING - Packet loss = 44%, RTA = 26.69 ms
[15:23:16] <icinga-wm>	 RECOVERY - Host ms-be1008 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms
[15:23:26] <nagios-wm>	 PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:27] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:27] <nagios-wm>	 PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:27] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:27] <nagios-wm>	 PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours
[15:23:27] <nagios-wm>	 PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:27] <nagios-wm>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:28] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[15:30:56] <nagios-wm>	 PROBLEM - Packetloss_Average on oxygen is CRITICAL: CRITICAL: packet_loss_average is 8.22690244094 (gt 8.0)
[15:50:29] <paravoid>	 cmjohnson1: thanks
[15:50:49] <cmjohnson1>	 yw...do you want to check it b4 puppet run?
[15:53:13] <paravoid>	 nah
[15:54:42] <cmjohnson1>	 k
[15:58:03] <gerrit-wm>	 New patchset: Ottomata; "filters.oxygen.erb - disabling x-cs filter until we get the ok to disable the other IP based filters." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45769
[15:58:17] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45769
[16:04:32] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[16:04:41] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[16:07:43] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 16:07:39 UTC 2013
[16:07:52] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 16:07:42 UTC 2013
[16:08:31] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[16:08:32] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 16:08:29 UTC 2013
[16:08:41] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[16:08:44] <nagios-wm>	 RECOVERY - Packetloss_Average on oxygen is OK: OK: packet_loss_average is 0.0
[16:08:51] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 16:08:50 UTC 2013
[16:09:31] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[16:09:42] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[16:12:21] <cmjohnson1>	 paravoid
[16:12:24] <cmjohnson1>	 nable to install GRUB in /dev/sda         │         │
[16:12:24] <cmjohnson1>	   │          │ Executing 'grub-install /dev/sda' failed
[16:12:31] <icinga-wm>	 RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jan 25 16:12:25 UTC 2013
[16:12:42] <icinga-wm>	 PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours
[16:14:42] <paravoid>	 is the raid configured ok?
[16:15:14] <cmjohnson1>	 it is but I think i may know what it is...the new board may have the raid controller disabled
[16:16:22] <paravoid>	 ok
[16:21:31] <icinga-wm>	 PROBLEM - Host ms-be1008 is DOWN: PING CRITICAL - Packet loss = 100%
[16:27:38] <nagios-wm>	 PROBLEM - Host ms-be1008 is DOWN: PING CRITICAL - Packet loss = 100%
[16:31:02] <icinga-wm>	 RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri Jan 25 16:30:56 UTC 2013
[16:31:32] <icinga-wm>	 PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours
[16:32:02] <icinga-wm>	 RECOVERY - Host ms-be1008 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms
[16:33:30] <nagios-wm>	 RECOVERY - Host ms-be1008 is UP: PING OK - Packet loss = 0%, RTA = 26.51 ms
[16:37:23] <nagios-wm>	 PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours
[16:49:10] <paravoid>	 cmjohnson1: works now?
[16:51:09] <^demon>	 !log formey: mediawiki svn repository is now read-only. long live git.
[16:51:21] <morebots>	 Logged the message, Master
[16:52:29] <cmjohnson1>	 paravoid: still going
[16:53:44] <paravoid>	 ^demon: nice job :)
[16:54:54] <^demon>	 https://gerrit.wikimedia.org/r/#/c/45564/ can now be merged.
[17:04:03] <LeslieCarr>	 hrm paravoid there is something with either naggen or puppet that's making ms1 and ms2 still have the checks generated in the icinga files and those annoying puppet freshness checks -- have any ideas of where i should start looking ?
[17:05:21] <paravoid>	 uhm
[17:05:31] <paravoid>	 they both just contacted the puppetmaster
[17:05:42] <paravoid>	 so their exports were restored
[17:05:49] <paravoid>	 decom will run at some point again and remove them
[17:05:56] <paravoid>	 and then they'll run again and add themselves again
[17:06:03] <paravoid>	 until the end of time :-)
[17:06:12] <paravoid>	 kill puppet on the boxes or the boxes themselves
[17:08:26] <paravoid>	 LeslieCarr: ^
[17:08:42] <LeslieCarr>	 hehe ok
[17:11:02] <LeslieCarr>	 thanks
[17:17:04] <gerrit-wm>	 New review: Demon; "It's tomorrow :)" [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/45564
[17:18:24] <cmjohnson1>	 paravoid: same problem...not installing /sda
[17:19:37] <paravoid>	 I'll have a look
[17:19:57] <Coren>	 Is anyone physically adjacent to Ryan?
[17:20:14] <Ryan_Lane>	 Coren: no. I'm still at home
[17:20:29] <Ryan_Lane>	 we're talking in 40 minutes, right?
[17:20:40] <cmjohnson1>	 i am out of the console (paravoid)
[17:21:10] <Coren>	 Ryan_Lane: Err, that's okay with me but it was actually 20 minutes ago.  Long live timezone confusion!  :-)
[17:21:33] <Ryan_Lane>	 Coren: my calendar says 10 PST ;)
[17:21:54] <Coren>	 Gah!  Stupid irc client.  ^W is delete word, not close window!
[17:22:13] <Ryan_Lane>	 Coren: my calendar says 10 PST ;)
[17:22:40] <Coren>	 Ryan_Lane: Ah, Ion told me 17:00 UTC.  Either works for me anyways.  :-)
[17:22:59] <Coren>	 Ryan_Lane: Talk to you laters, then.  ~wave~
[17:23:10] <Ryan_Lane>	 cool. yep
[17:30:27] <gerrit-wm>	 New patchset: Reedy; "Disble QueryPage updates on frwiki like enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45783
[17:31:18] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45783
[17:31:50] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php
[17:32:01] <morebots>	 Logged the message, Master
[17:48:20] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours
[18:07:57] <gerrit-wm>	 New patchset: Reedy; "Add small, medium and large dblists to wikitags" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45785
[18:08:34] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45785
[18:08:56] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours
[18:09:50] <logmsgbot>	 !log reedy synchronized wmf-config/CommonSettings.php
[18:10:01] <morebots>	 Logged the message, Master
[18:11:17] <nagios-wm>	 PROBLEM - Puppet freshness on db1031 is CRITICAL: Puppet has not run in the last 10 hours
[18:13:14] <nagios-wm>	 PROBLEM - Puppet freshness on db1037 is CRITICAL: Puppet has not run in the last 10 hours
[18:15:37] <RD>	 Anybody aware of en.wiki (elsewhere?) main page rendering issues? https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Users_reporting_site_time_issues_and_delay_in_visible_update_of_edits
[18:15:45] <RD>	 quite a few emails coming to otrs too
[18:16:14] <nagios-wm>	 PROBLEM - Puppet freshness on db1012 is CRITICAL: Puppet has not run in the last 10 hours
[18:16:28] <Reedy>	 It would sound like it's stuff not being purged from squid in esams
[18:18:21] <nagios-wm>	 PROBLEM - Puppet freshness on db1015 is CRITICAL: Puppet has not run in the last 10 hours
[18:18:21] <nagios-wm>	 PROBLEM - Puppet freshness on db1014 is CRITICAL: Puppet has not run in the last 10 hours
[18:19:14] <nagios-wm>	 PROBLEM - Puppet freshness on db1023 is CRITICAL: Puppet has not run in the last 10 hours
[18:20:17] <nagios-wm>	 PROBLEM - Puppet freshness on db1030 is CRITICAL: Puppet has not run in the last 10 hours
[18:21:26] <gerrit-wm>	 New review: MZMcBride; "This is related to some bug, isn't it?" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45785
[18:21:56] <icinga-wm>	 RECOVERY - swift-account-reaper on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-reaper
[18:22:20] <gerrit-wm>	 New review: Reedy; "https://bugzilla.wikimedia.org/show_bug.cgi?id=43741" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45785
[18:22:57] <gerrit-wm>	 New review: MZMcBride; "Right-o. Thanks for working on this!" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45785
[18:24:56] <icinga-wm>	 PROBLEM - swift-account-reaper on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-reaper
[18:30:34] <gerrit-wm>	 New patchset: Reedy; "Bug 44349 - updateSpecialPages is run twice on small wikis" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45786
[18:31:14] <nagios-wm>	 PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours
[18:32:22] <gerrit-wm>	 New patchset: Reedy; "(bug 29692) Per-wiki namespace aliases shouldn't override (remove) global ones" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25737
[18:34:14] <nagios-wm>	 PROBLEM - Puppet freshness on db1045 is CRITICAL: Puppet has not run in the last 10 hours
[18:34:14] <nagios-wm>	 PROBLEM - Puppet freshness on db1044 is CRITICAL: Puppet has not run in the last 10 hours
[18:35:56] <Reedy>	 Importing 2013 Barack Obama Inauguration Ceremony (full).ogv... No comment file with extension txt found for /tmp/odder/2013 Barack Obama Inauguration Ceremony (full).ogv, using default comment. failed. (* Could not write file "mwstore&#58;//local-swift/local-public/a/a6/2013_Barack_Obama_Inauguration_Ceremony_(full).ogv" because it is larger than {{PLURAL:4294967296|one byte|4294967296 bytes}}.
[18:35:59] <Reedy>	 Saywhat?
[18:36:01] <Reedy>	 AaronSchulz: ^
[18:37:14] <nagios-wm>	 PROBLEM - Puppet freshness on db1016 is CRITICAL: Puppet has not run in the last 10 hours
[18:38:16] <Reedy>	 reedy@fenari:/home/wikipedia/common$ du --si /tmp/odder/2013\ Barack\ Obama\ Inauguration\ Ceremony\ \(full\).ogv
[18:38:16] <Reedy>	 5.1G    /tmp/odder/2013 Barack Obama Inauguration Ceremony (full).ogv
[18:38:37] <apergos>	 O_o
[18:39:15] <gerrit-wm>	 New review: Demon; "One minor nitpick, functionally ok." [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/45786
[18:40:32] <gerrit-wm>	 New patchset: Reedy; "Bug 44349 - updateSpecialPages is run twice on small wikis" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45786
[18:43:54] <gerrit-wm>	 New review: Reedy; "Is this still to go out?" [operations/mediawiki-config] (master); V: 0 C: -1;  - https://gerrit.wikimedia.org/r/43029
[18:44:38] <gerrit-wm>	 New patchset: Reedy; "Prevent MediaWiki maintenance scripts from running as privileged users" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44200
[18:45:08] <gerrit-wm>	 New patchset: Reedy; "(bug 44054) Enable Translate and TranslateNotifications on br.wikimedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44516
[18:45:35] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44516
[18:46:30] <gerrit-wm>	 New patchset: RobH; "barium & colby to become new dns servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45788
[18:47:40] <Reedy>	 !log Created translate tables on brwikimedia
[18:47:50] <morebots>	 Logged the message, Master
[18:49:34] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php
[18:49:45] <morebots>	 Logged the message, Master
[18:49:55] <cmjohnson1>	 robh: look for a ticket shortly for a cable order for eqiad..need 5' green and blue
[18:50:14] <RobH>	 np
[18:50:17] <gerrit-wm>	 Change abandoned: Reedy; "(no reason)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45150
[18:50:20] <gerrit-wm>	 Change abandoned: Reedy; "(no reason)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/45149
[18:51:16] <Reedy>	 Nemo_bis: Want to try https://gerrit.wikimedia.org/r/#/c/25737/ again?
[18:51:35] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44856
[18:52:11] <Nemo_bis>	 Reedy: yes!
[18:52:18] <Nemo_bis>	 I'll check if pt.wiki and friends break
[18:52:36] <Nemo_bis>	 tell me when you're going to sync, gotta prepare dinner soon
[18:52:37] <gerrit-wm>	 New review: RobH; "-stealth self review-" [operations/puppet] (production); V: 2 C: 2;  - https://gerrit.wikimedia.org/r/45788
[18:52:38] <gerrit-wm>	 Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45788
[18:53:09] <Reedy>	 I can do it now if you want?
[18:53:15] <Nemo_bis>	 ok
[18:53:28] <gerrit-wm>	 New patchset: Reedy; "(bug 29692) Per-wiki namespace aliases shouldn't override (remove) global ones" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25737
[18:53:33] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25737
[18:54:09] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php
[18:54:19] <morebots>	 Logged the message, Master
[18:54:36] <Nemo_bis>	 as of now, it still works
[18:54:39] <Reedy>	 And the surbey says
[18:54:45] <Reedy>	 Good
[18:54:55] <Nemo_bis>	 how long it takes to sync?
[18:54:59] <Reedy>	 It's done
[18:55:07] <Reedy>	 [18:54:09] <logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
[18:55:12] <Reedy>	 ^ That means it has finished
[18:55:18] <Nemo_bis>	 ah right :)
[18:55:31] <Reedy>	 See if anyone comes in later and complains ;)
[19:14:44] * Coren  complains.

[19:19:33] <woosters>	 coren - about what?
[19:19:41] <Coren>	 [13:55:31] <Reedy> See if anyone comes in later and complains ;)
[19:19:44] <Coren>	 :-)
[19:20:27] <woosters>	 :-)
[19:31:47] <duh>	 so otrs is getting flooded with emails asking why the main page is out of date…is this something that we can just say was due to the datacenter migration (was it?) and will hopefully be fixed soon?
[19:32:42] <Reedy>	 Ops are aware and are/were looking at it
[19:32:59] <Reedy>	 It would seem that the esams squids aren't recieving the purge requests from eqiad
[19:33:10] <Reedy>	 But yes, it's a leftover from the datacentre migration
[19:33:21] <duh>	 alright, ill just say it'll get fixed soon ;)
[19:33:50] <duh>	 thanks
[19:33:50] <RD>	 if its for sure related, could link the blog post about that duh
[19:33:53] <RD>	 shrug
[19:34:00] <duh>	 yeah was planning on it
[19:34:25] <Reedy>	 i'm presuming it's a "network issue" with how the purge packets are sent out
[19:34:31] <Reedy>	 and from eqiad they don't make it to pmtpa
[19:34:35] <Reedy>	 err, esams
[19:34:47] <RD>	 one of them metal box thingies had an issue
[19:34:54] <RD>	 that's what you reply with, duh
[19:34:57] <duh>	 :P
[19:36:03] <nagios-wm>	 PROBLEM - Host mw1001 is DOWN: PING CRITICAL - Packet loss = 100%
[19:37:38] <icinga-wm>	 PROBLEM - Host mw1001 is DOWN: PING CRITICAL - Packet loss = 100%
[19:39:56] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: Puppet has not run in the last 10 hours
[19:40:58] <Nemo_bis>	 we had such a complains also on it.quote
[19:41:10] <Nemo_bis>	 they said 16 January even
[19:41:18] <icinga-wm>	 RECOVERY - Host mw1001 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms
[19:41:26] <nagios-wm>	 RECOVERY - Host mw1001 is UP: PING OK - Packet loss = 0%, RTA = 26.70 ms
[19:43:19] <Nemo_bis>	 ah, pretty, en.wiki has a sitenotice even about broken cache
[19:43:49] <duh>	 the person who created that forgot to close the </center> tag and messed up a bunch of pages :P
[19:44:09] <duh>	 https://en.wikipedia.org/w/index.php?title=MediaWiki%3ASitenotice&diff=534860227&oldid=534859722
[19:44:19] <duh>	 well forgot to close the <!--
[19:44:44] <Nemo_bis>	 for 3 min
[19:45:11] <Nemo_bis>	 I think the purpose of the sitenotice is "hey, not all sitenotices are yellow and taking 75 % of your screen!"
[19:45:24] <duh>	 :P
[19:45:24] <Nemo_bis>	 "You forgot it, admit it"
[19:49:22] <Reedy>	 I saw that
[19:51:47] <nagios-wm>	 PROBLEM - Host solr1002 is DOWN: PING CRITICAL - Packet loss = 100%
[19:52:11] <icinga-wm>	 PROBLEM - Host solr1002 is DOWN: PING CRITICAL - Packet loss = 100%
[20:08:34] <nagios-wm>	 RECOVERY - Host solr1002 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms
[20:08:42] <icinga-wm>	 RECOVERY - Host solr1002 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms
[20:08:51] <icinga-wm>	 PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours
[20:13:24] <LeslieCarr>	 oi
[20:14:07] <LeslieCarr>	 so… i honestly am not certain the source of the purges in eqiad
[20:14:13] <LeslieCarr>	 i
[20:14:17] <LeslieCarr>	 m going to look around wikitech
[20:14:26] <LeslieCarr>	 if anyone knows, i'm awesome at the tracing the network issue ;)
[20:14:26] <RoanKattouw>	 LeslieCarr: What do you mean?
[20:14:37] <RoanKattouw>	 The literal source as in who's sending the purges?
[20:14:43] <LeslieCarr>	 so if purges aren't making it to esams
[20:14:43] <LeslieCarr>	 yeah
[20:14:57] <LeslieCarr>	 i want to make sure that there's not something in the layer 3 (IP) level
[20:15:00] <RoanKattouw>	 The purges should be coming from app servers going to a multicast IP
[20:15:17] <RoanKattouw>	 You can look at live purges by tcpdumping on an eqiad Squid, and you can generate purges using a command-line script
[20:15:19] <LeslieCarr>	 and then is there a relay to unicast again ?
[20:15:30] <RoanKattouw>	 There should be a relay to unicast for the transatlantic transport somewhere
[20:15:35] <LeslieCarr>	 because how else would we get across
[20:15:36] <LeslieCarr>	 ok
[20:15:39] <RoanKattouw>	 But I don't know exactly how that's set up
[20:15:40] <LeslieCarr>	 so that's my target ;)
[20:15:43] <Reedy>	 $wgHTCPMulticastAddress = '239.128.0.112';
[20:15:55] <RoanKattouw>	 I also don't know how the purges travel between eqiad and pmtpa, I'm guessing there's a relay there as well?
[20:16:00] <RoanKattouw>	 or maybe not?
[20:16:59] <RoanKattouw>	 I wouldn't be surprised if the relay was somehow unable to deal with the purges originating in eqiad rather than pmtpa
[20:17:11] <LeslieCarr>	 well between eqiad and pmtpa is all one network area
[20:17:15] <RoanKattouw>	 Right
[20:17:19] <LeslieCarr>	 so that's easy ;)
[20:17:20] <RoanKattouw>	 So that's just all native multicast?
[20:17:22] <LeslieCarr>	 yeah
[20:17:46] <LeslieCarr>	 hehe, just had the hammer idea - restart all the squids!
[20:17:50] <LeslieCarr>	 (j/k, fyi)
[20:18:01] <Reedy>	 Do it
[20:18:04] <Reedy>	 Then go to lunch
[20:18:04] <Reedy>	 ;)
[20:18:58] <LeslieCarr>	 yes!
[20:19:02] <LeslieCarr>	 turn off the phon
[20:19:02] <LeslieCarr>	 e
[20:20:04] <Reedy>	 might it be easier to just ask mark what the setup is?
[20:20:11] <Reedy>	 it's not too late for him now
[20:20:13] <RoanKattouw>	 Yeah
[20:20:16] <RoanKattouw>	 I can't find it in puppet either
[20:20:29] <RoanKattouw>	 There are other multicast relays in there, for Varnish and Squid logs
[20:20:37] <RoanKattouw>	 But I don't think I see this particular relay there
[20:20:43] <Reedy>	 guess you don't necesserily need him to come and fix it. but a pointer makes life easier
[20:20:57] <LeslieCarr>	 well it's friday
[20:21:00] <LeslieCarr>	 so ...
[20:21:22] <ottomata2>	 Haaaai mah friiiieeends
[20:21:34] <LeslieCarr>	 ah
[20:21:35] <LeslieCarr>	 hrm
[20:21:41] <ottomata2>	 anyone here know what packs 3 udp2log lines into a single UDP packet?
[20:21:52] <ottomata2>	 do we have patches on the frontend caches (vanrish, squid) to do that?
[20:21:55] <ottomata2>	 or do they do that by default?
[20:22:05] <ottomata2>	 (I betcha no one listening knows)
[20:22:10] <Reedy>	 ottomata2: code!
[20:22:12] <ottomata2>	 robla, do you have any historical info on this?
[20:23:00] <LeslieCarr>	 well having an ip of 239.128.0.112 helps - lemme try poking around from htere
[20:23:27] <LeslieCarr>	 wish i had my multicast book here ;)  i always forget about multicast stuff
[20:23:54] <RoanKattouw>	 LeslieCarr: Basically all i
[20:24:23] <RoanKattouw>	 *all I know is that you can see the traffic come in on a Squid by running tcpdump addr 239.128.0.112 or whatever
[20:25:30] <robla>	 ottomata2: I don't recall when that specific optimization was added....I imagine it's been a while
[20:26:24] <ottomata2>	 LeslieCarr, robla mentioned this when I was telling him about my 5% udp2log loss problem, perhaps your multicast purge problems are related to my 5% mutlicast data  loss problems
[20:26:30] <ottomata2>	 but it was added?
[20:26:31] <ottomata2>	 robla?
[20:26:36] <ottomata2>	 3 lines in a packet?
[20:26:45] <ottomata2>	 (or something like that)
[20:27:01] <ottomata2>	 i mean, there was a change to force frontend caches to fit more data into a single UDP packet?
[20:27:05] <LeslieCarr>	 but i don't think that there is actually a loss of packets
[20:27:14] <ottomata2>	 yeah, i htink you are probably right
[20:27:20] <RoanKattouw>	 It's probably just the relay being broken somehow
[20:27:24] <ottomata2>	 yeah
[20:27:43] <robla>	 the relay seems like a very worthy area of exploration
[20:28:29] <ottomata2>	 sorry to hijack your discussion Roan, hehe, these problems could very well be unrelated
[20:28:40] <RoanKattouw>	 no worries
[20:28:41] <ottomata2>	 but yeah, robla, i'm trying to eliminate udp2log as source of the problem
[20:28:43] <ottomata2>	 before messing witih oxygen
[20:30:30] <robla>	 ah, you're trying to figure out an HTCP purge issue?
[20:30:46] <RoanKattouw>	 robla: We suspect the HTCP US->Europe relay might be broken
[20:30:56] <RoanKattouw>	 There are reports of stale pages being served in Europe
[20:31:11] <robla>	 seems sensible
[20:31:31] <RoanKattouw>	 Given the timing, I suspect it may be related to the purges now originating from eqiad rather than pmtpa
[20:31:36] <RoanKattouw>	 Or maybe the thing just plain broke
[20:31:52] <Reedy>	 The former seems much more likely
[20:32:06] <RoanKattouw>	 No one really knows how this thing works except mark, and it doesn't seem to be in puppet, but Reedy dug up the multicast IP and Leslie is using that to investigate
[20:32:07] <Reedy>	 Mark apparently fixed it earlier this week (or attempted to)
[20:32:25] <RoanKattouw>	  06:48 Tim: on eqiad upload varnishes: purged /wikipedia/it/b/bc/Wiki.png at user request. varnishhtcpd appears to be totally broken.
[20:32:32] <RoanKattouw>	 That was Jan 22
[20:32:55] <LeslieCarr>	 hrm, is that related though ?
[20:33:01] <RoanKattouw>	 Can't find anything else related in the logs
[20:33:03] <RoanKattouw>	 I don't think so
[20:33:03] <LeslieCarr>	 because it's not upload, it's the squids
[20:33:07] <RoanKattouw>	 Yeah
[20:33:14] <RoanKattouw>	 I think Tim found and fixed that issue, he notified ops@
[20:33:14] <LeslieCarr>	 Reedy: have any irc logs from mark fixing ?
[20:33:38] <RoanKattouw>	 [Ops] varnishhtcpd completely broken
[20:33:50] <RoanKattouw>	 Apparently there's a deadlock bug in varnishtcpd
[20:33:58] <RoanKattouw>	 But again, that's Varnish, this is Squid, so probably unrelate
[20:33:59] <RoanKattouw>	 d
[20:34:11] <AaronSchulz>	 well tim made it single threaded to fix that
[20:34:25] <AaronSchulz>	 though it might overload a cpu or something, meh
[20:44:29] <Reedy>	 Can someone please run rm -rf /home/wikipedia/common/php-1.21wmf5 as root for me on fenari?
[20:44:42] <Reedy>	 MaxSem is keeping files to himself again
[20:45:16] <awjr>	 Ryan_Lane: is there any reason why it would not be good or at least OK to force https on pages handled by MobileFrontend? I think we'd need to force a redirect form http->https in MobileFrontend itself if we were to do this
[20:45:32] <LeslieCarr>	 but different issue :) so far i have discovered that mobile varnish caches are sending to that but not helpful
[20:47:26] <logmsgbot>	 !log reedy synchronized docroot
[20:47:37] <morebots>	 Logged the message, Master
[20:47:51] <logmsgbot>	 !log reedy synchronized live-1.5/
[20:48:02] <morebots>	 Logged the message, Master
[20:49:23] <LeslieCarr>	 RoanKattouw: what port's used for the purges again ?
[20:49:42] <LeslieCarr>	 ns0 is listening to that group which is weird
[20:50:09] <Reedy>	 $this->port = isset( $parts[1] ) ? $parts[1] : 80;
[20:50:16] <LeslieCarr>	 ah 4827
[20:51:34] <LeslieCarr>	 next step is more drastic
[20:51:47] <Reedy>	 Bigger hammer?
[20:52:03] <LeslieCarr>	 filter on the edges to log what is leaving the network via 4827
[20:52:09] <LeslieCarr>	 not filter in the block
[20:52:13] <LeslieCarr>	 filter in the log sense
[20:52:31] <LeslieCarr>	 more like a really annoying hammer ;)
[20:55:26] <LeslieCarr>	 ah, so this is interesting
[20:55:33] <LeslieCarr>	 so i haven't been able to find which is the magical relay
[20:56:33] <LeslieCarr>	 however it doesn't look like the purge requests sourced in eqiad aren't being relayed to tampa
[20:56:46] <LeslieCarr>	 so maybe there's some multicast shit going down
[20:58:08] <LeslieCarr>	 which is my problem ;)
[20:59:53] <Coren>	 Isn't "multicast shit" what happens when shit hits the fan?  Or is that broadcast shit?  :-)
[20:59:59] <RoanKattouw>	 LeslieCarr: I didn't quite understand that sentence. Are you saying the multicast packets from eqiad are *not* making it to pmtpa?
[21:00:15] <LeslieCarr>	 i am saying that from my first look it appears that they are not
[21:00:17] <RoanKattouw>	 Cause if the relay is in pmtpa ...
[21:00:22] <LeslieCarr>	 so if the magical relay is in tampa
[21:00:26] <LeslieCarr>	 damnit, you beat my typing to it!
[21:00:44] <LeslieCarr>	 Coren: that's broadcast shit ;)
[21:16:59] <ottomata>	 LeslieCarr, in my most recent test on an01
[21:17:03] <ottomata>	 NIC gets all packets
[21:17:06] <ottomata>	 udp2log does not
[21:17:10] <ottomata>	 or at least, does not write them to disk
[21:17:29] <LeslieCarr>	 ottomata: sorry but i'm on the multicast stuff now
[21:17:42] <ottomata>	 hokay, you mean you're not on MY multicast stuff?!
[21:17:45] <ottomata>	 all the time!
[21:17:48] <ottomata>	 PSHHH
[21:18:01] <LeslieCarr>	 :p
[21:18:15] <LeslieCarr>	 i just pruned my tree, bitch!
[21:18:24] <LeslieCarr>	 (ok that may have been too obscure of multicast joke)
[21:18:30] <ottomata>	 hahaha, i did not get it
[21:18:50] <ottomata>	 i'll start reading the mulitcast standards and then laugh one day
[21:19:06] <Reedy>	 well, it is udp..
[21:19:19] <ottomata>	 yes
[21:19:27] <ottomata>	 Reedy, you mean my comment?
[21:19:27] <ottomata>	 yes
[21:19:33] <ottomata>	 but we are seeing systematic 5% packet loss
[21:19:45] <ottomata>	 i'm trying to determine if loss is from network
[21:19:47] <ottomata>	 or from udp2log
[21:19:53] <Reedy>	 ottomata: no, multicast
[21:19:57] <ottomata>	 oh ok ok ok
[21:20:06] <Reedy>	 ottomata: You know the worst thing about udp jokes, right?
[21:20:07] <ottomata>	 nm, i will go back to telling my tale in the #analytics channel :p
[21:20:09] <ottomata>	 oh
[21:20:11] <ottomata>	 HAHAHHA
[21:20:11] <ottomata>	 I get it
[21:20:17] <Reedy>	 :D
[21:20:26] <ottomata>	 i mean, I get it…THIS TIME
[21:21:17] <Reedy>	 Though, Leslies joke might've been something completely different
[21:21:56] <LeslieCarr>	 it was
[21:21:59] <LeslieCarr>	 though that's a good one
[21:31:55] <Ryan_Lane>	 awjr: if you do what?
[21:32:06] <Ryan_Lane>	 awjr: you mean all mobilefrontend traffic being https?
[21:32:07] <Ryan_Lane>	 always?
[21:32:11] <Ryan_Lane>	 not just for logged-in users?
[21:32:15] <awjr>	 Ryan_Lane: correct
[21:32:24] <notpeter>	 can't lots of older devices not speak https?
[21:32:39] <Ryan_Lane>	 why would you ever have to do that?
[21:33:01] <awjr>	 Ryan_Lane: part of the problem right now is that when a use rlogs in on mobile, they are forced into https, but if they follow http links from say, google, or hit their back buttons and go back to http, they are not still logged in
[21:33:02] <Ryan_Lane>	 if you *have* to do that, you're doing something wrong
[21:33:21] <Ryan_Lane>	 there's a feature in mediawiki for this
[21:33:24] <awjr>	 oh?
[21:33:32] <awjr>	 that is handy
[21:33:32] <Ryan_Lane>	 it sets a cookie that says "I'm logged in via https"
[21:33:49] <Ryan_Lane>	 if you hit the site with http and it sees the cookie, it should redirect you to https
[21:33:54] <awjr>	 do you know what it is or where i can find it?
[21:34:02] <Ryan_Lane>	 talk to csteipp  about it
[21:34:09] <awjr>	 k thanks Ryan_Lane
[21:34:16] <csteipp>	 hmm?
[21:34:18] <Ryan_Lane>	 by no means should you consider putting all traffic on https
[21:34:26] <awjr>	 why not?
[21:34:33] <Ryan_Lane>	 because we aren't scaled for it
[21:34:34] <awjr>	 i mean, what's the harm?
[21:34:35] <awjr>	 ok
[21:34:40] <awjr>	 that makes sense
[21:34:46] <awjr>	 glad i asked :)
[21:35:04] <Nemo_bis>	 LeslieCarr: bawolff anticipates multicast investigations at https://bugzilla.wikimedia.org/show_bug.cgi?id=41130#c63
[21:35:21] <awjr>	 csteipp Ryan_Lane was mentioning there's some magic in mediawiki that will set a cookie when you log in via https that will redirect you to https if you're on http?
[21:36:10] <csteipp>	 awjr: Yeah, on desktop, I set an insecure cookie (so http can read it) that basically says, this user is logged in, and then I redirect to https if I see that cookie
[21:36:17] <awjr>	 notpeter: to answer your question, likely - but almost certainly not the device types that we are supporting for login (jquery is a requirement, so we don't surface login for devices that don't support jquery)
[21:36:21] <csteipp>	 Let me get you the commit...
[21:36:26] <awjr>	 thanks csteipp
[21:36:51] <notpeter>	 ah, cool!
[21:36:59] <notpeter>	 thanks for the clarification!
[21:37:23] <awjr>	 sure thang
[21:37:28] <csteipp>	 awjr: https://gerrit.wikimedia.org/r/#/c/24026/
[21:38:10] <awjr>	 thanks csteipp, that's great
[22:08:36] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: Puppet has not run in the last 10 hours
[22:08:57] <nagios-wm>	 PROBLEM - Puppet freshness on stat1001 is CRITICAL: Puppet has not run in the last 10 hours
[22:14:35] <icinga-wm>	 PROBLEM - Puppet freshness on stat1001 is CRITICAL: Puppet has not run in the last 10 hours
[22:29:08] <LeslieCarr>	 !log clearing multicast joins in order to troubleshoot multicast pruning
[22:29:19] <morebots>	 Logged the message, Mistress of the network gear.
[22:29:30] <LeslieCarr>	 analytics folks (dschoon ) this may have a temp log blip
[22:30:04] <dschoon>	 duly noted, mistress of network gear
[22:30:44] <LeslieCarr>	 :)
[22:30:53] <LeslieCarr>	 monitor stop
[22:30:54] <LeslieCarr>	 oops
[22:31:01] <LeslieCarr>	 wrong window ;)
[22:32:06] <notpeter>	 Reedy: you there?
[22:32:35] <Reedy>	 Yeah
[22:32:53] <Susan>	 I got a 503 error from upload.wikimedia.org, but it cleared itself on refresh.
[22:33:01] <Susan>	 It was one of those Varnish guru errors, I think.
[22:33:48] <Reedy>	 notpeter: wassup?
[22:33:56] <notpeter>	 Reedy: so, I feel like the issues is at the mc layer
[22:34:02] <Reedy>	 Oh?
[22:34:10] <notpeter>	 this is guessing at the moment, tbh....
[22:34:16] <notpeter>	 but
[22:34:25] <notpeter>	 db layer seems unlikely
[22:34:32] <notpeter>	 because if that was fucked, nothing would be happening
[22:34:43] <notpeter>	 and that's one of the few pieces that changed on tuesday
[22:34:49] <notpeter>	 and this problem def started on tuesday :)
[22:34:53] <Reedy>	 heh
[22:35:13] <notpeter>	 but, my php wizardry is bad
[22:35:42] <notpeter>	 so, my troubleshooting is not awesome
[22:35:50] <notpeter>	 but, I was wonering if anything came to mind for you on this front?
[22:36:23] <Reedy>	 What makes you think it's mc?
[22:36:38] <notpeter>	 that is where locks for jobrunning live, yes?
[22:36:53] <notpeter>	 or is that fuckin' ... whateverPool
[22:36:58] <notpeter>	 poolcounter
[22:37:01] <notpeter>	 that's the word I'm looking for
[22:37:59] <Reedy>	 Didn't realise the job queue used poolcounter..
[22:38:13] <notpeter>	 oh, I'm not sure it does
[22:38:20] <notpeter>	 I'm brainstorming here...
[22:38:28] <notpeter>	 I trust you completely on this
[22:39:02] <notpeter>	 as I said, I don't know the actual runJob script code
[22:40:02] <LeslieCarr>	 grrrr
[22:40:08] <LeslieCarr>	 i hate you so much multicast
[22:40:14] <LeslieCarr>	 i confirmed that it's broken ;)
[22:40:17] <LeslieCarr>	 as much as that helps
[22:41:00] <LeslieCarr>	 and that it shouldn't be broken ;)
[22:41:19] <Reedy>	 profiling is broken too with /0
[22:41:31] <Reedy>	 AaronSchulz: about?
[22:49:48] <mutante>	 i like how i see 2 possible target milestones in Bugzilla..  The choice is between "--" and "Mysterious future" :)  (at least in this component)
[23:07:54] <LeslieCarr>	 !log on cr1 and cr2 on eqiad allowing for unicast-reverse-path feasible-paths
[23:08:05] <morebots>	 Logged the message, Mistress of the network gear.
[23:09:13] <LeslieCarr>	 !!!
[23:09:14] <LeslieCarr>	 waiot
[23:09:20] <LeslieCarr>	 i think this is working
[23:09:34] <Reedy>	 yaay
[23:10:08] <LeslieCarr>	 no
[23:10:31] <LeslieCarr>	 i forgot that i had iperf running on a local to tampa location as well
[23:10:33] <LeslieCarr>	 goddamnit
[23:11:09] <LeslieCarr>	 wait
[23:11:11] <LeslieCarr>	 this is weird
[23:11:14] <LeslieCarr>	 so i createa  new multicast group
[23:11:17] <LeslieCarr>	 and bam, happy
[23:11:34] <LeslieCarr>	 wtf
[23:11:40] <LeslieCarr>	 i am going to be right back
[23:11:46] <LeslieCarr>	 this is the time when i wish i smoked cigarettes
[23:11:50] * Damianz  hands Leslie some Whisky

[23:16:52] <LeslieCarr>	 ok, dschoon using a multicast hammer
[23:17:15] <Damianz>	 Who gets hit at the same time?
[23:18:21] <dschoon>	 what's up?
[23:18:32] <LeslieCarr>	 so expect about 5 seconds (up to 30 if there's long reconverging times)
[23:18:50] <LeslieCarr>	 new multicast groups propogating, old ones not
[23:18:50] <LeslieCarr>	 going to clear all multicast routes and force the routers to re-do the trees
[23:19:19] <dschoon>	 do i need to do anything other than watch the fires start?
[23:40:06] <LeslieCarr>	 ip 239.128.0.112 port 4827 Ryan_Lane (purges are being sent there at least)
[23:47:57] <RobH>	 we dont run our own NTP server anywhere do we?
[23:49:37] <Reedy>	 RobH: I guess we do..
[23:49:37] <Reedy>	 class ntp::client(
[23:49:37] <Reedy>	   $servers =['linne.wikimedia.org', 'dobson.wikimedia.org'],
[23:49:39] <mutante>	 RobH: we do,
[23:49:41] <mutante>	 dobson
[23:50:20] <mutante>	 you should be able to set the time on any other server using  ntpdate dobson.wikimedia.org
[23:50:33] <Reedy>	 http://wikitech.wikimedia.org/view/NTP
[23:50:34] <RobH>	 ok, we need a cname for that
[23:50:37] <RobH>	 i dont wanna recall that
[23:50:39] <RobH>	 ntp.wikimedia.org
[23:50:44] <RobH>	 =P
[23:50:47] <mutante>	 (but you need to stop NTP the service to use ntpdate and then restart it)
[23:50:48] * RobH  goes to add it

[23:50:50] <Reedy>	 there doesn't seem to be a ntp.pp now
[23:51:01] <mutante>	 yea, having ntp. would be nice
[23:52:02] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db67 is CRITICAL: CRIT replication delay 218 seconds
[23:52:11] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db67 is CRITICAL: CRIT replication delay 233 seconds
[23:53:01] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db67 is OK: OK replication delay 0 seconds
[23:53:12] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db67 is OK: OK replication delay 0 seconds
[23:55:49] <binasher>	 Reedy: hey, i hear you were looking at the jobqueue breakage?
[23:56:33] <Reedy>	 I was, sort of yeah
[23:56:59] <mutante>	 !log installing mysql-client/common package upgrades on kaulen (bugzilla)
[23:57:10] <morebots>	 Logged the message, Master