[00:34:22] <grrrit-wm>	 (03PS1) 10Springle: rebalance group loads while db1050 out [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103326 
[00:36:06] <grrrit-wm>	 (03CR) 10Springle: [C: 032] rebalance group loads while db1050 out [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103326 (owner: 10Springle)
[00:37:10] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  'rebalance group loads while db1050 out'
[00:37:26] <morebots>	 Logged the message, Master
[00:43:14] <grrrit-wm>	 (03PS1) 10Springle: assign db1033 to s1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/103327 
[00:45:38] <grrrit-wm>	 (03CR) 10Springle: [C: 032] assign db1033 to s1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/103327 (owner: 10Springle)
[00:50:02] <springle>	 !log xtrabackup clone db1049 to db1033
[00:50:18] <morebots>	 Logged the message, Master
[01:59:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[01:59:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[02:07:23] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf7) at Mon Dec 23 02:07:23 UTC 2013
[02:07:42] <morebots>	 Logged the message, Master
[02:13:09] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf8) at Mon Dec 23 02:13:09 UTC 2013
[02:13:24] <morebots>	 Logged the message, Master
[02:26:35] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 23 02:26:35 UTC 2013
[02:26:52] <morebots>	 Logged the message, Master
[04:00:22] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[04:01:11] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical  
[05:00:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[05:00:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[05:17:31] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[05:54:35] <grrrit-wm>	 (03PS1) 10Springle: warm up db1033 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103339 
[05:56:28] <grrrit-wm>	 (03CR) 10Springle: [C: 032] warm up db1033 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103339 (owner: 10Springle)
[05:57:39] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  'warm up db1033 in s1'
[05:57:55] <morebots>	 Logged the message, Master
[06:18:31] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[06:24:58] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 031] "Using the "official" logstash packages works for me. I haven't looked at the startup scripts, etc that they provide, but we should be able" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103112 (owner: 10Faidon Liambotis)
[07:30:31] <icinga-wm>	 PROBLEM - MySQL Idle Transactions on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[07:30:31] <icinga-wm>	 PROBLEM - MySQL InnoDB on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[07:31:41] <icinga-wm>	 PROBLEM - MySQL Recent Restart on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[07:32:01] <icinga-wm>	 PROBLEM - mysqld processes on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[07:32:09] <springle>	 hmm
[07:32:11] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[07:32:21] <icinga-wm>	 PROBLEM - MySQL Processlist on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[07:34:31] <icinga-wm>	 RECOVERY - MySQL Recent Restart on db1016 is OK: OK 10541341 seconds since restart  
[07:34:51] <icinga-wm>	 RECOVERY - mysqld processes on db1016 is OK: PROCS OK: 1 process with command name mysqld  
[07:35:01] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1016 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[07:35:11] <icinga-wm>	 RECOVERY - MySQL Processlist on db1016 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics  
[07:35:21] <icinga-wm>	 RECOVERY - MySQL InnoDB on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[07:35:21] <icinga-wm>	 RECOVERY - MySQL Idle Transactions on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[07:43:44] <springle>	 !log taking down db1016 for xfs_check, locked up and: XFS (dm-0): xlog_space_left: head behind tail
[07:43:58] <paravoid>	 ouch
[07:44:01] <morebots>	 Logged the message, Master
[07:44:16] <springle>	 paravoid: know something about that?
[07:44:24] * springle  never seen before

[07:44:24] <paravoid>	 nope
[07:44:52] <paravoid>	 sounds like a kernel bug
[08:01:01] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1050 is CRITICAL: CRIT replication delay 45460 seconds  
[08:01:11] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1050 is CRITICAL: CRIT replication delay 45386 seconds  
[08:01:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[08:01:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[08:31:43] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] reprepro: import from elasticsearch/logstash apt [operations/puppet] - 10https://gerrit.wikimedia.org/r/103112 (owner: 10Faidon Liambotis)
[08:33:15] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "Ugh." [operations/puppet] - 10https://gerrit.wikimedia.org/r/103198 (owner: 10Yurik)
[08:33:57] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Zero: Keep things DRY - removed duplicate IDs [operations/puppet] - 10https://gerrit.wikimedia.org/r/103199 (owner: 10Yurik)
[08:35:10] <yurik>	 thanks paravoid, i will rebase the other patches
[08:36:17] <paravoid>	 which ones?
[08:41:56] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-2] "Per our meeting." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik)
[08:48:55] <yurik>	 paravoid, https://gerrit.wikimedia.org/r/#/c/102887/
[08:49:34] <paravoid>	 yup, I saw that
[08:51:08] <grrrit-wm>	 (03PS3) 10Yurik: Handle the change with netmapper and varnish 3.0.4 & later [operations/puppet] - 10https://gerrit.wikimedia.org/r/102887 
[08:52:11] <yurik>	 paravoid, actually - no rebase conflicts ^. Any objections to that patch?
[08:52:17] <paravoid>	 yes
[08:52:22] <paravoid>	 I'd like to hear from Brandon
[08:52:32] <paravoid>	 to see if it'd be possible to fix in the vmod itself
[08:53:51] <yurik>	 last time i spoke with him he said its beyond vmod - its the new feature of the varnish. But maybe he overlooked something
[08:54:50] <yurik>	 my only concern at this point is the fact that now i can't test anything in betalabs, because the varnish there is the latest, and I would have to downgrade it, which precludes me from ESI testing based on the patch you submitted
[08:55:02] <paravoid>	 ??
[08:55:10] <paravoid>	 I don't understand
[08:55:15] <yurik>	 beta cluster uses the same VCLs
[08:55:24] <paravoid>	 well, then don't
[08:55:33] <yurik>	 brandon migrated it to the 3.0.5+ alpha
[08:55:35] <paravoid>	 run a separate varnish instance from betalabs
[08:55:54] <paravoid>	 or wait until brandon wakes up, it's only a few hours
[08:56:07] <yurik>	 i thought he was out for 2 weeks?
[08:56:14] <paravoid>	 oh? is he?
[08:56:19] <yurik>	 that's what he said
[08:56:55] <yurik>	 checking log...
[08:57:22] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db69 is CRITICAL: CRIT replication delay 305 seconds  
[08:57:36] <yurik>	 sorry, no log - chatzilla doesn't log the new channels
[08:58:01] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db69 is CRITICAL: CRIT replication delay 342 seconds  
[09:03:01] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db69 is OK: OK replication delay -0 seconds  
[09:03:21] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db69 is OK: OK replication delay 0 seconds  
[09:04:37] <matanya>	 paravoid: https://gerrit.wikimedia.org/r/#/c/92288/ ... :)
[09:05:38] <grrrit-wm>	 (03PS5) 10Faidon Liambotis: removing cache clean up patch [operations/puppet] - 10https://gerrit.wikimedia.org/r/92288 (owner: 10Matanya)
[09:06:06] <grrrit-wm>	 (03PS6) 10Faidon Liambotis: removing cache clean up patch [operations/puppet] - 10https://gerrit.wikimedia.org/r/92288 (owner: 10Matanya)
[09:07:11] <grrrit-wm>	 (03PS7) 10Faidon Liambotis: varnish: removing temp cache fix for zero [operations/puppet] - 10https://gerrit.wikimedia.org/r/92288 (owner: 10Matanya)
[09:07:58] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] varnish: remove temp cache fix for zero [operations/puppet] - 10https://gerrit.wikimedia.org/r/92288 (owner: 10Matanya)
[09:09:39] <matanya>	 thanks paravoid 
[09:10:20] <yurik>	 paravoid, how about i do the same thing in zero.vcl as we did in mobile.vcl -- surround those if statements with the <% if cluster_options.fetch( "enable_esi", false ) -%>
[09:10:30] <paravoid>	 which statements?
[09:10:40] <paravoid>	 why can't you just setup a separate varnish instance to play with EIS
[09:10:43] <paravoid>	 *ESI
[09:10:53] <yurik>	 paravoid, https://gerrit.wikimedia.org/r/#/c/102887/3/templates/varnish/zero.inc.vcl.erb
[09:10:54] <paravoid>	 so that you don't have to go through a production review process to perform tests?
[09:12:18] <yurik>	 labs instance? can't do it very easily - all domain names are targeted there
[09:12:28] <yurik>	 so i would have to migrate to older varnish
[09:12:39] <yurik>	 which i would rather not do as that has high chance of killing betalabs
[09:12:43] <paravoid>	 wait, what?
[09:12:59] <yurik>	 http://en.m.wikipedia.beta.wmflabs.org/wiki/Main_Page
[09:13:06] <yurik>	 this is served by the new varnish
[09:13:32] <yurik>	 as well as all other *.m.*.beta  & *.zero.*.beta
[09:16:20] <paravoid>	 your idea kind of defeats the point of labs I think
[09:17:10] <paravoid>	 it's kinda silly to go via a production code review process to test VCL changes in labs
[09:24:03] <grrrit-wm>	 (03CR) 10Faidon Liambotis: Varnish: switch to W3C standard headers for ESI (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 (owner: 10Faidon Liambotis)
[09:26:42] <grrrit-wm>	 (03PS3) 10Faidon Liambotis: Varnish: switch to W3C standard headers for ESI [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 
[09:27:09] <grrrit-wm>	 (03PS4) 10Faidon Liambotis: Varnish: switch to W3C standard headers for ESI [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 
[09:30:12] <grrrit-wm>	 (03CR) 10Yurik: [C: 031] "No need to hold this off until backend, this way i will receive correct signals in the backend from varnish from the start" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 (owner: 10Faidon Liambotis)
[09:32:34] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Varnish: switch to W3C standard headers for ESI [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 (owner: 10Faidon Liambotis)
[09:34:55] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-2] "I don't see the point of an m->desktop redirect, as I pointed out on the bug report." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/101787 (owner: 10John F. Lewis)
[09:43:44] <grrrit-wm>	 (03Abandoned) 10Murfel: Update favicon spcom.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103249 (owner: 10Murfel)
[09:46:21] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms  
[09:47:36] <paravoid>	 that would be me
[09:53:01] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[09:54:33] <paravoid>	 !log rebooting cp1065 with Linux 3.2
[09:54:50] <morebots>	 Logged the message, Master
[09:55:11] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 1.04 ms  
[10:22:37] <grrrit-wm>	 (03PS2) 10Murfel: Update favicon spcom.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103203 
[10:38:11] <icinga-wm>	 RECOVERY - DPKG on lanthanum is OK: All packages OK  
[10:39:03] <paravoid>	 akosiaris: was that you?
[10:39:22] <akosiaris>	 paravoid: indirectly yes
[10:39:49] <akosiaris>	 !log uploaded php5_5.3.10-1ubuntu3.9+wmf1 to apt.wikimedia.org
[10:40:04] <morebots>	 Logged the message, Master
[11:02:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[11:02:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[11:11:31] <icinga-wm>	 PROBLEM - Disk space on labstore4 is CRITICAL: DISK CRITICAL - /srv is not accessible: Input/output error  
[11:11:42] <matanya>	 oh no ^
[11:17:44] <apergos>	 ugghh
[11:18:38] <akosiaris>	 that cannot be good
[11:18:45] <apergos>	 [2289504.725663] XFS (dm-0): xfs_log_force: error 5 returned.
[11:18:47] <apergos>	 ok yay
[11:18:50] <apergos>	 *oh yay
[11:19:48] <paravoid>	 that's a secondary error
[11:19:51] <paravoid>	 what's the primary one?
[11:20:25] <apergos>	 there's a lot of cruft in dmesg, I'm still looking
[11:20:56] <akosiaris>	 ffff880436b22000: 80 00 20 3c c3 04 a5 b2 00 00 00 01 00 00 00 00  .. <............
[11:20:56] <akosiaris>	 [2278571.852351] XFS (dm-0): Internal error xfs_dir2_data_reada_verify at line 226 of file /build/buildd/linux-lts-raring-3.8.0/fs/xfs/xfs_dir2_data.c.  Caller 0xffffffffa036e33f
[11:21:13] <akosiaris>	 and after the stacktrace [2278571.869978] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
[11:21:13] <akosiaris>	 [2278571.881017] XFS (dm-0): metadata I/O error: block 0x3c0082bb0 ("xfs_trans_read_buf_map") error 117 numblks 8
[11:21:53] <paravoid>	 [2288970.235855] XFS (dm-0): I/O Error Detected. Shutting down filesystem
[11:21:56] <paravoid>	 [2288970.238569] Buffer I/O error on device dm-0, logical block 85086316
[11:22:01] <paravoid>	 [2288970.163666] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
[11:22:04] <paravoid>	 yay
[11:22:09] <akosiaris>	 check-raid says ok... 
[11:22:18] <matanya>	 so much fun :/
[11:22:26] <apergos>	 it seems like this has been going on for awhile
[11:22:28] <paravoid>	 it's corrupted
[11:22:33] <apergos>	 [39175.745798] XFS (dm-0): Internal error xfs_dir2_data_reada_verify at line 226 of file /build/buildd/linux-lts-raring-3.8.0/fs
[11:22:42] <akosiaris>	 why are we using xfs again ? :P
[11:23:00] <paravoid>	 root@labstore4:~# mount | wc -l
[11:23:01] <paravoid>	 39302
[11:23:02] <paravoid>	 wtf.
[11:23:24] <paravoid>	 isn't that supposed to have a backup labstoer?
[11:23:32] <apergos>	 yes it is
[11:24:04] <apergos>	 don't know which one though
[11:36:07] <paravoid>	 text coren maybe?
[11:39:02] <apergos>	 so labstore3 would be the backup, we would have to move the labsnfs ip addr of10.0.0.45  (currently on labstore4) to labstore3 and do the startup
[11:39:09] <apergos>	 I'm not confident in getting that right myself
[11:39:31] <akosiaris>	 is that correct ? cause they see different sets of disks
[11:39:38] <akosiaris>	 at least according to check-raid
[11:40:04] <akosiaris>	 and I remember coren saying the share the disk shelves now
[11:40:25] <apergos>	 that is what I thought (they share them) 
[11:40:35] <apergos>	 so something could be different now (again)
[11:40:51] <apergos>	 texting is probably best
[11:41:02] <akosiaris>	 well one sees 25 logical and 36 physical and the other  12 logical, 12 physical
[11:42:30] <apergos>	 if labstore 3 is just missing some, that might be intentional
[11:42:46] <apergos>	 since it's supposed to be the standby host
[11:43:50] <apergos>	 the outputs of mount are ridiculous on both hosts
[13:24:33] <springle_>	 !log starting xtrabackup clone db1001 to db1016
[13:24:49] <morebots>	 Logged the message, Master
[13:39:25] <Betacommand>	 anyone with corens cellular number around?
[13:40:10] <paravoid>	 apergos was handling the issue afaik
[13:40:28] <apergos>	 he's been paged and voicemail left
[13:40:35] <Betacommand>	 apergos: thanks
[13:45:13] <Krenair>	 Can someone find out what happened to a seemingly missing password reset email?
[13:53:16] <Vito>	 https://tools.wmflabs.org/guc/?user=79.22.0.97 < WTF?
[13:53:35] <paravoid>	 Vito: #wikimedia-labs please
[14:03:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[14:03:41] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[14:34:41] <icinga-wm>	 PROBLEM - Host labstore4 is DOWN: PING CRITICAL - Packet loss = 100%  
[14:35:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:37:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:39:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:41:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:43:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:43:55] <icinga-wm>	 RECOVERY - Host labstore4 is UP: PING OK - Packet loss = 0%, RTA = 35.40 ms  
[14:44:26] <icinga-wm>	 RECOVERY - Disk space on labstore4 is OK: DISK OK  
[14:45:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:47:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:49:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:51:56] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:53:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:55:56] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:57:47] <apergos>	 ignre the lvs1003 issue please
[14:57:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[14:59:56] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 02:29:59 PM UTC  
[15:00:25] <icinga-wm>	 RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Mon Dec 23 15:00:17 UTC 2013  
[15:01:55] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 03:00:17 PM UTC  
[15:03:02] <apergos>	 that will be cleared up next puppet run
[15:28:46] <ottomata>	 qchris, this is part of the ssl discussion:
[15:28:47] <ottomata>	 https://rt.wikimedia.org/Ticket/Display.html?id=859
[15:28:49] <ottomata>	 can you read that?
[15:29:00] <qchris>	 Let me try :-)
[15:29:33] <icinga-wm>	 RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Mon Dec 23 15:29:26 UTC 2013  
[15:29:51] <qchris>	 ottomata: Thanks. I can read it :-)
[15:30:40] <cmjohnson1>	 !log labnet1001 relocating racks C4 to B3 eqiad
[15:30:56] <morebots>	 Logged the message, Master
[15:31:27] <ottomata>	 cool, the relevant bit there are towards the end of the discussion
[15:31:33] <ottomata>	 although i wish there were links to gerrit changes
[15:31:41] <ottomata>	 it doesn't really say why we still have them in the stream
[15:36:23] <ottomata>	 hiyaaa paravoid!  
[15:36:46] <ottomata>	 nuria wants to do some testing of varnish log changes for eventlogging, she's wondering if she could/should use the beta cluster
[15:36:49] <ottomata>	 who should she ask about that?
[15:37:53] <ottomata>	 oh maybe antoine, she'll figure it out I think
[15:44:23] <icinga-wm>	 PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Last successful Puppet run was Mon 23 Dec 2013 12:44:05 PM UTC  
[15:56:33] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[16:05:45] <ottomata>	 Jeff_Green: 
[16:05:49] <ottomata>	 i just merged your OCG comment
[16:05:51] <ottomata>	 s'ok?
[16:12:33] <icinga-wm>	 RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Mon Dec 23 16:12:30 UTC 2013  
[16:21:38] <Jeff_Green>	 ottomata: yes, thank you
[16:21:51] <Jeff_Green>	 i got interrupted and forgot
[16:24:19] <ottomata>	 k no probs
[16:44:29] <paravoid>	 ottomata: hey
[16:44:35] <paravoid>	 you forgot to close #6535
[16:45:13] <chrismcmahon>	 paravoid: does this change make sense to you?  https://gerrit.wikimedia.org/r/#/c/103376?  I was trying to use MobileFrontend on beta labs today and login would give me 503s
[16:45:13] <ottomata>	 thanks
[16:45:35] <ottomata>	 oh actually, i guess it isn't entirely closed, paravoid
[16:45:42] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/103375/
[16:45:48] <ottomata>	 see comment there ^
[16:46:45] <paravoid>	 chrismcmahon: hey, nice catch
[16:47:12] <paravoid>	 chrismcmahon: although the fix is incomplete, you need to make it on mobile-backend change too
[16:47:17] <paravoid>	 chrismcmahon: I'll fix it and merge, no worries
[16:48:00] <chrismcmahon>	 paravoid: great thanks, varnish params is not my area of expertise
[16:48:24] <paravoid>	 chrismcmahon: not that 35s is a reasonable amount of time to wait for a page...
[16:49:50] <chrismcmahon>	 paravoid:  something is going wrong with beta labs this morning, was getting stack traces from login, now getting 500 errors from Varnish, not sure way.   but before that I was getting 503s trying to log in.  whatever's going on is affecting page load times also
[16:52:00] <paravoid>	 chrismcmahon: I linked to https://bugzilla.wikimedia.org/show_bug.cgi?id=57026 in my comment
[16:52:03] <paravoid>	 and it has been reopened since
[16:52:19] <paravoid>	 with the cause identified and a proposed, yet unmerged, fix
[16:52:43] <paravoid>	 so it was probably this that is causing 57249
[16:53:33] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[16:55:17] <cmjohnson1>	 ottomata: analytics1001 is gone ...correct? I replaced it with virt1001 and removing puppet entries. Just want to confirm...thx
[16:56:53] <ottomata>	 cmjohnson1: it is still alive
[16:57:03] <ottomata>	 but afaik i think slated for repurposing
[16:57:26] <ottomata>	 i just sshed in anyway
[16:58:06] <cmjohnson1>	 ottomata: okay..yeah I haven't pushed dns changes yet 
[16:58:20] <cmjohnson1>	 just want to be clear 
[16:58:42] <ottomata>	 k
[16:58:43] <ottomata>	 cool
[16:59:41] <cmjohnson1>	 !log dns update
[16:59:56] <morebots>	 Logged the message, Master
[17:04:23] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[17:04:23] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[17:10:23] <ottomata>	 paravoid, can/should I just include admins::roots on vanadium?
[17:10:31] <ottomata>	 all I really need is groups::wikidev
[17:10:37] <paravoid>	 I already commented on the patchset
[17:10:40] <ottomata>	 oh sorry
[17:10:44] <paravoid>	 we've lost grrrit-wm
[17:10:54] <ottomata>	 ok thanks
[17:30:27] <cmjohnson1>	 gwicke: around?
[17:41:36] <James_F>	 cmjohnson1: Anything I can help with in gwicke's absence?
[17:48:16] <cmjohnson1>	 james_f did you get my last...i was disconnected 
[17:50:09] <James_F>	 cmjohnson1: No.
[17:51:02] <cmjohnson1>	 I want to take down cerium or xenon ...they're reporting failed raid but gwicke thinks it's a red herring but...still reports so I will either try to fix or replace disk
[17:51:18] * James_F  nods.

[17:52:03] <James_F>	 cmjohnson1: De-pooling should work fine, I think.
[17:52:30] <paravoid>	 there's no depooling, it's the cassandra test hosts
[17:52:46] <cmjohnson1>	 right...i just want approval
[17:52:54] <James_F>	 Oh, those.
[17:53:09] * James_F  looses track of what cerium is being used for this week. :-)

[17:53:22] <James_F>	 subbu should be able to tell for sure, but that sounds OK.
[17:54:55] <gwicke>	 cmjohnson1: don't worry about the raid
[17:55:18] <gwicke>	 I manually failed one of the disks each so that I could use them as linear stripe
[17:55:38] <cmjohnson1>	 okay...can i fail the drive so it stops reporting
[17:56:25] <gwicke>	 cmjohnson1: I did mdadm -f and -r
[17:56:36] <gwicke>	 the drive is used in lvm
[17:57:35] <gwicke>	 cmjohnson1: do you mean editing mdadm.conf or the like?
[17:58:06] <gwicke>	 I can disable reporting there if that helps
[17:58:37] <cmjohnson1>	 that would help
[18:02:25] <gwicke>	 cmjohnson1: it looks like AUTOCHECK=false in /etc/default/mdadm should do it
[18:03:10] <gwicke>	 changed that now, please let me know if mdadm continues to spam
[18:03:35] <cmjohnson1>	 cool..i will thanks for doing that
[18:04:11] <gwicke>	 sorry for the spam
[18:30:17] <ori-l>	 what's up with grrrt-wm?
[18:30:52] <ori-l>	 wow
[18:31:09] <apergos>	 nice job!
[18:31:10] <grrrit-wm>	 (03PS1) 10Yatinmaan: Added commons.ico with more resolutions. \n Buzilla Link - review/yatinmaan/103107 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103355 
[18:31:34] <grrrit-wm>	 (03CR) 10Aklapper: "Superseded by https://gerrit.wikimedia.org/r/#/c/103355/ according to comments in GCI task" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103107 (owner: 10Yatinmaan)
[18:31:35] <grrrit-wm>	 (03CR) 10Aklapper: [C: 04-1] "The commit message is currently broken and needs fixing. See https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103355 (owner: 10Yatinmaan)
[18:31:40] <grrrit-wm>	 (03PS1) 10Jgreen: comment out OCG icinga check for now [operations/puppet] - 10https://gerrit.wikimedia.org/r/103363 
[18:31:41] <grrrit-wm>	 (03CR) 10Jgreen: [C: 032 V: 031] comment out OCG icinga check for now [operations/puppet] - 10https://gerrit.wikimedia.org/r/103363 (owner: 10Jgreen)
[18:31:51] <grrrit-wm>	 (03CR) 10Qgil: [C: 031] "Thank you! Now everything looks good to me. Odder?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103203 (owner: 10Murfel)
[18:32:14] <grrrit-wm>	 (03PS1) 10Ottomata: Removing sudo_user otto from vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103372 
[18:32:15] <grrrit-wm>	 (03PS1) 10Ottomata: Giving Nuria access to vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103373 
[18:32:17] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Removing sudo_user otto from vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103372 (owner: 10Ottomata)
[18:32:20] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Giving Nuria access to vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103373 (owner: 10Ottomata)
[18:32:21] <grrrit-wm>	 (03PS1) 10Ottomata: Including admins::roots on vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103375 
[18:32:23] <grrrit-wm>	 (03CR) 10Ottomata: "Should admins::roots be everywhere?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103375 (owner: 10Ottomata)
[18:32:37] <grrrit-wm>	 (03PS1) 10Cmcmahon: varnish: adjust first_byte_timeout to 180s (MobileFrontend) [operations/puppet] - 10https://gerrit.wikimedia.org/r/103376 
[18:32:39] <grrrit-wm>	 (03PS1) 10Cmjohnson: Removing old analytics dns files (an1001,1002,1005,1006,1008). Replacing mgmt entries with virt100[1-3][7-8]. [operations/dns] - 10https://gerrit.wikimedia.org/r/103377 
[18:32:40] <grrrit-wm>	 (03PS2) 10Cmcmahon: varnish: adjust first_byte_timeout to 185s (MobileFrontend) [operations/puppet] - 10https://gerrit.wikimedia.org/r/103376 
[18:32:43] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] Removing old analytics dns files (an1001,1002,1005,1006,1008). Replacing mgmt entries with virt100[1-3][7-8]. [operations/dns] - 10https://gerrit.wikimedia.org/r/103377 (owner: 10Cmjohnson)
[18:32:45] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: install-server: further cleanups to apt-repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/103378 
[18:32:45] <grrrit-wm>	 (03PS2) 10Faidon Liambotis: install-server: further cleanups to apt-repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/103378 
[18:32:46] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032 V: 032] install-server: further cleanups to apt-repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/103378 (owner: 10Faidon Liambotis)
[18:32:51] <grrrit-wm>	 (03PS3) 10Faidon Liambotis: varnish: adjust first_byte_timeout to 185s (MobileFrontend) [operations/puppet] - 10https://gerrit.wikimedia.org/r/103376 (owner: 10Cmcmahon)
[18:32:54] <grrrit-wm>	 (03PS4) 10Faidon Liambotis: varnish: adjust first_byte_timeout for mobile too [operations/puppet] - 10https://gerrit.wikimedia.org/r/103376 (owner: 10Cmcmahon)
[18:32:57] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032 V: 032] varnish: adjust first_byte_timeout for mobile too [operations/puppet] - 10https://gerrit.wikimedia.org/r/103376 (owner: 10Cmcmahon)
[18:32:57] <grrrit-wm>	 (03CR) 10Qgil: "No, let's continue amending this change here, please. There is a chain of dependencies that needs to be resolved anyway. Yatinmaan, this c" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103107 (owner: 10Yatinmaan)
[18:33:00] <grrrit-wm>	 (03PS1) 10Cmjohnson: Changing dhcpd file entries to reflect server name change (analytics to virts) [operations/puppet] - 10https://gerrit.wikimedia.org/r/103380 
[18:33:01] <grrrit-wm>	 (03CR) 10Qgil: [C: 04-1] "Please, let's keep amending https://gerrit.wikimedia.org/r/#/c/103107/ . I have left some comments there about why it is good to amend pat" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103355 (owner: 10Yatinmaan)
[18:33:11] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "No, it shouldn't be everywhere, not until it's overhauled and fully automated. Just include groups::wikidev for now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/103375 (owner: 10Ottomata)
[18:33:14] <grrrit-wm>	 (03PS2) 10Cmjohnson: Updating puppet entries to reflect server name changes (analytics to virts) [operations/puppet] - 10https://gerrit.wikimedia.org/r/103380 
[18:33:18] <grrrit-wm>	 (03PS2) 10Ottomata: Including groups::wikidev on vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103375 
[18:33:19] <grrrit-wm>	 (03PS3) 10Ottomata: Including groups::wikidev on vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103375 
[18:33:20] <grrrit-wm>	 (03CR) 10Ottomata: "Ok, I changed this to just include groups::wikidev" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103375 (owner: 10Ottomata)
[18:33:20] <paravoid>	 ottomata1: analytics multicast should be fixed, can you check?
[18:33:21] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] "Ok, I changed this to just include groups::wikidev" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103375 (owner: 10Ottomata)
[18:33:22] <grrrit-wm>	 (03PS3) 10Cmjohnson: Updating puppet entries to reflect server name changes (analytics to virts) [operations/puppet] - 10https://gerrit.wikimedia.org/r/103380 
[18:33:23] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] Updating puppet entries to reflect server name changes (analytics to virts) [operations/puppet] - 10https://gerrit.wikimedia.org/r/103380 (owner: 10Cmjohnson)
[18:33:33] <grrrit-wm>	 (03PS1) 10Cmjohnson: Removing public ip for analytics1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/103382 
[18:33:33] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] Removing public ip for analytics1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/103382 (owner: 10Cmjohnson)
[18:33:36] <grrrit-wm>	 (03CR) 10Qgil: "Ok, so the 16x16 and 32x32 versions of the original favicon look more sharp than yours. Since the original ones are correct, you can just " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103355 (owner: 10Yatinmaan)
[18:33:43] <grrrit-wm>	 (03PS1) 10Ottomata: Adding Nuria so to admins::restricted [operations/puppet] - 10https://gerrit.wikimedia.org/r/103383 
[18:33:44] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Adding Nuria so to admins::restricted [operations/puppet] - 10https://gerrit.wikimedia.org/r/103383 (owner: 10Ottomata)
[18:34:30] <grrrit-wm>	 (03PS1) 10Ori.livneh: Revert "Hack: cron job to clean up tifs from /tmp on app servers" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103390 
[18:35:35] <ottomata1>	 paravoid, will do in just a sec
[18:35:50] <ottomata>	 (check multicast that is)
[18:37:36] <paravoid>	 hm, or not
[18:38:08] <paravoid>	 oh nevermind, it's probably fixed
[18:38:27] <ori-l>	 ottomata: nuria's account should have sudo to be meaningful (otherwise can't stop / start eventlogging services), paravoid's OK with it, so i'm submitting another patch on top of yours
[18:39:58] <ottomata>	 ok
[18:40:01] <ottomata>	 thanks ori
[18:40:29] <ottomata>	 ori that one has been merged so you'll need to submit a new change
[18:41:49] <grrrit-wm>	 (03PS1) 10Hoo man: Add the file namespace to Wikibase Client excludeNamespaces [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103394 
[18:48:08] <grrrit-wm>	 (03PS1) 10Ori.livneh: Add nuria to sudoers on vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103397 
[18:56:44] <grrrit-wm>	 (03PS2) 10Hoo man: Add file and translate NS to Wikibase Client excludeNamespaces [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103394 
[18:57:35] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Add nuria to sudoers on vanadium [operations/puppet] - 10https://gerrit.wikimedia.org/r/103397 (owner: 10Ori.livneh)
[18:58:20] <paravoid>	 ori-l, nuria: ^^
[18:59:24] <grrrit-wm>	 (03CR) 10Aaron Schulz: [C: 031] Revert "Hack: cron job to clean up tifs from /tmp on app servers" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103390 (owner: 10Ori.livneh)