[00:07:13] * AaronSchulz detects being added to a list [00:07:45] * ToAruShiroiNeko detects that AaronSchulz detects being added to a list [00:07:48] hello AaronSchulz [00:08:15] AaronSchulz do you know how the wmf wiki handles donation pages? [00:40:58] New patchset: preilly; "conclude the telenor montenegro test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17483 [00:41:14] binasher: can you handle this for me ^^ [00:41:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17483 [00:41:59] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17483 [01:41:40] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 231 seconds [01:41:40] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 231 seconds [01:42:43] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 279 seconds [01:42:43] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 279 seconds [01:48:34] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 632s [01:48:34] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 632s [01:54:07] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 11 seconds [01:54:07] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 11 seconds [01:54:16] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 21s [01:54:16] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 21s [01:54:25] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 5 seconds [01:54:25] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 5 seconds [03:09:43] PROBLEM - MySQL disk space on storage3 is CRITICAL: DISK CRITICAL - free space: /archive 163250 MB (3% inode=99%): [03:09:44] PROBLEM - MySQL disk space on storage3 is CRITICAL: DISK CRITICAL - free space: /archive 163250 MB (3% inode=99%): [03:14:04] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Fri Aug 3 03:13:32 UTC 2012 [03:14:05] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Fri Aug 3 03:13:32 UTC 2012 [03:22:01] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Fri Aug 3 03:21:55 UTC 2012 [03:22:02] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Fri Aug 3 03:21:55 UTC 2012 [04:42:25] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [04:42:25] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [05:25:28] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [05:25:29] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [05:50:12] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [05:50:13] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [06:29:12] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [06:29:12] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [06:44:50] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [06:44:51] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [06:58:56] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [06:58:56] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [07:13:56] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [07:13:56] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [07:58:05] !log GlusterFS upgrade of labs project storage seems to have been completed. Fixed the issues I had with it on 'beta' project, yeah!!!!! [07:58:16] Logged the message, Master [08:42:23] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [08:42:23] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [08:42:23] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [08:42:24] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [08:42:24] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [08:42:24] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [11:06:57] New patchset: Demon; "Further UI fixes to Gerrit, courtesy of Timo and Trevor" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17497 [11:07:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17497 [11:09:44] New patchset: Demon; "Further UI fixes to Gerrit, courtesy of Timo and Trevor" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17497 [11:10:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17497 [11:10:51] New review: Demon; "Added Timo to the review list since I want to make sure I got it all right." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/17497 [12:04:10] New review: Mark Bergsma; "Where are the config manifests?" [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/17342 [12:12:15] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16122 [12:13:31] Change abandoned: Mark Bergsma; "Packages purged." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17387 [12:16:01] New review: Hashar; "Thanks Mark!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17387 [12:25:28] New patchset: Matthias Mullie; "Bug 36772 - Article Feedback - Supporting feedback on help pages" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/17503 [12:35:55] http://bits.wikimedia.org/events.git returns a 204 ? :D [12:36:07] could be made just: HTTP_204.gif [12:36:14] easy rebase at least [12:36:56] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [12:37:24] "and I still don't like that it's a separate role class which duplicates most things" [12:37:29] mark: I am not sure what do there [12:37:35] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [12:37:49] why is that a separate class? [12:37:53] why not just use role::cache::bits? [12:38:54] cause most the configuration is different ? [12:39:04] we don't have the same varnish / bits server [12:39:05] no lvs [12:39:16] the common bits should remain common at least [12:39:25] so you could use inheritance there perhaps [12:39:39] if you split them off then production and labs are going to diverge, that's not really useful then [12:39:59] and why not have lvs? [12:40:16] there is no LVS support yet in labs :-D [12:40:25] that doesn't matter for varnish [12:40:30] the apaches are load balanced with a lame squid frontend :D [12:40:42] you can still install wikimedia-lvs-realserver [12:44:41] mark: I am not sure I want to get any lvs stuff on 'beta' for now [12:45:06] i thought the point was to stay as close as possible to production [12:45:13] but I can get backend_options and cluster_options abstracted [12:45:19] why would lvs harm anything? [12:45:21] yeah that is the point, get a copy of the production [12:45:43] then don't unnecessarily diverge from it... adding lvs configuration doesn't do anything [12:45:55] I have no idea what it could harm or not. I have just have 0 clue about our LVS system and I am afraid it is going to mess something in beta [12:46:05] it won't [12:46:08] we already have enough trouble keeping up with the various puppet changes and mediawiki changes :-) [12:46:24] just use role::cache::bits, and change the few hostnames you need to change for labs with $::realm [12:46:28] if it does nothing, it is file though :-] [12:46:32] ok [12:48:34] hehe $varnish_backends switch on $::site :-D [12:49:57] yeah [12:50:01] New review: Matthias Mullie; "Do not yet submit - awaiting confirmation from Fabrice" [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/17503 [12:50:14] I am not sure what $::site is set to on labs [12:50:16] New review: Aude; "This is a nice improvement with the logo. Minor suggestions include make the logo clickable (linked ..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/17497 [12:50:19] probably ptmpa / eqiad [12:50:20] pmtpa or eqiad [12:50:24] k [12:50:24] same as in production [12:50:30] so I need to add a $::realm around [12:50:34] yeah [12:50:53] brb [12:50:56] mark: you should talk with paravoid about puppet styling [12:51:07] yeah, we have notpeter submitting a module now [12:51:13] I think he prefers using additional classes instead of if( $::realm ) [12:51:32] ah [12:51:39] well, I'm not a fan of duplicating classes [12:52:10] but I prefer using declarative means (such as class inheritance/includes and overrides) rather than if/then blocks [12:57:11] is there a way to merge two hashes ? :-D [12:57:47] puppetlabs/stdlib seems to provide such a function [13:06:27] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [13:06:28] did an ugly if / else instead [13:06:42] mark: now with some if( $::realm ) https://gerrit.wikimedia.org/r/13304 ;) [13:06:43] a conditional is a declarative means [13:07:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [13:07:27] New review: Aude; "looks like the clickable logo is addressed in http://gerrit-dev.wmflabs.org/" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/17497 [13:07:39] hashar: [13:07:59] $varnish_backends = $::realm ? 'production' => ..., 'labs' => ... [13:08:02] that's declarative [13:09:51] what is the issue ? [13:09:56] don't use if/else [13:10:08] use a selector [13:10:23] for cluster_options the selector does not let me assign a hash [13:10:49] stuff like: $cluster_options = $::realm ? 'production' => { some cluster options }, returns a parse error [13:10:51] that's unfortunately true [13:10:54] stupid puppet [13:11:02] on 2.7.18 at least [13:11:13] maybe I should have added a comment there [13:11:23] or maybe there is a way to construct a hash [13:11:38] Hash[key => value, key => its value] [13:12:12] as I said, you can also use inheritance [13:12:18] perhaps that works for hashes [13:12:59] hmmm? parse error why? [13:13:09] I've done exactly that before [13:13:22] in puppet 2.6 [13:13:23] http://projects.puppetlabs.com/issues/5516 -- "hash can't be used in selectors" [13:13:26] broke in puppet 2.7 [13:13:27] argh. [13:13:33] argh. argh. [13:13:33] was very annoying [13:14:43] the 2.7 issue is https://projects.puppetlabs.com/issues/14301 -- "hashes can't be used in selectors (upgrading from 2.7.1 to 2.7.14)" [13:15:03] unfortunately when we upgraded, that wasn't so well known/documented yet [13:15:07] so I had to figure it out [13:15:25] was just getting stupid parse errors on things like the huge lvs.pp hashes [13:15:42] fortunately I could test it in labs [13:17:21] amazing [13:29:14] someone want to make me a gerrit account? (with prior SVN) [13:29:59] https://www.mediawiki.org/wiki/Project:Labsconsole_accounts?diff=568701&oldid=568571 [13:41:43] New review: Hashar; "err: Could not retrieve catalog from remote server: Error 400 on SERVER: $service_ips["bits"] is not..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/13304 [13:44:20] mark: so abstracting the two bits class lead the labs instance to setup LVS. It complains about lack of a hash for $service_ips['bits'][$::site] ;-D [13:44:39] yeah we should add those [13:45:55] it is there [13:45:58] but disabled :-) [13:45:59] # Labs has no LVS supports for now [13:46:00] if( $::realm == "production" ) { [13:46:01] include lvs::configuration [13:46:02] } [13:46:03] let me fix that [13:46:34] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [13:47:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [13:47:15] puppetmaster:self is nice :-) [13:47:25] would need to add some script to easily apply a patch from gerrit [13:50:35] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [13:51:14] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [13:55:07] * jeremyb waves hexmode... [13:55:24] hexmode: you have sudo on formey for moving accts svn -> git? [14:01:08] jeremyb: no [14:01:19] ;( [14:01:20] you want ^demon|away [14:01:24] i guess i should just ask puppet [14:01:28] hexmode: or any op [14:01:56] jeremyb: you just want a git account? [14:02:04] 03 13:29:14 < jeremyb> someone want to make me a gerrit account? (with prior SVN) [14:02:07] 03 13:29:59 < jeremyb> https://www.mediawiki.org/wiki/Project:Labsconsole_accounts?diff=568701&oldid=568571 [14:03:37] hexmode: here we go: "demon", "robla", "sumanah", "reedy" [14:03:40] plus ops [14:04:38] jeremyb: sumanah has been making sure they're processed in 24 hr iirc [14:04:52] hexmode: did you see her vacation email? [14:04:56] ;) [14:05:22] hexmode: http://lists.wikimedia.org/pipermail/wikitech-l/2012-July/061936.html [14:05:50] jeremyb: yes, but I'm just pointing out that you don't have to wait that long [14:06:01] i don't follow [14:06:11] > In my absence, here's whom to contact about: [14:06:16] > * Developer Access queue[1]: Varnent and JeremyB [14:06:33] heh [14:06:48] RGAGRGAEGHAEGAEG [14:06:55] * hexmode is slow and tired this morning [14:06:59] hexmode: haha [14:07:09] time to reboot bbiam [14:08:41] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [14:09:16] <^demon|away> jeremyb: /msg me the wiki username, the shell username, and e-mail address [14:09:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [14:09:28] ^demon|away: there's a diff link above [14:09:31] 03 14:02:04 < jeremyb> 03 13:29:59 < jeremyb> https://www.mediawiki.org/wiki/Project:Labsconsole_accounts?diff=568701&oldid=568571 [14:10:18] those service IPs are not good hashar [14:10:24] looks like the one that was there wasn't good either [14:10:26] let me allocate some [14:10:37] :) [14:10:45] shall I do that change? [14:10:49] sure :) [14:10:58] will rebase on top of your change and fix the resulting confligt [14:11:06] yes [14:11:13] I am eventually locked by https://bugzilla.wikimedia.org/show_bug.cgi?id=38524 : get_var is not defined [14:11:21] <^demon|away> jeremyb: Done, tell him to check his e-mail for the temporary password. [14:11:27] ^demon|away: danke! [14:11:33] New review: Hashar; "PS17 define some missing service IP in lvs.pp for labs, and convert some entries to hash having a 'p..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [14:11:38] mark: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Unknown function get_var at /etc/puppet/manifests/lvs.pp:766 on node i-0000031c.pmtpa.wmflabs [14:11:52] mark: I think that is hack that was not properly packaged / shipped in puppet on labs :( [14:13:09] actually [14:13:15] leave your lvs.pp change in, i'll merge that [14:13:30] ^demon|away: ugh, he's in [[special:userlist]] but not [[special:log/newusers]]. i guess that needs filing [14:14:03] <^demon|away> Complain to Ryan? I just did the command like I'm supposed to. [14:14:44] ^demon|away: ryan was my intended target. i thought maybe you were aware of it or something [14:14:49] <^demon|away> Nope. [14:14:54] <^demon|away> labsconsole is his baby :) [14:14:59] i know! [14:15:44] where is that get_var? [14:16:04] ah [14:16:05] mark: I have no idea :( [14:16:26] we should get rid of that [14:16:49] let's parameterize that class properly [14:19:50] ^demon|away: where can i get a list of svn users? [14:24:12] New patchset: Mark Bergsma; "Properly parameterize lvs::balancer, remove labs hack" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17514 [14:24:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17514 [14:25:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17514 [14:26:33] jeremyb: https://svn.wikimedia.org/users.php [14:26:44] Though, that may be based on userinfo.. [14:27:07] Reedy: well, *I* am on that list... [14:27:11] so it's not just svn [14:28:05] New patchset: Mark Bergsma; "Fix lvs::balancer::runcommand dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17515 [14:28:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17515 [14:28:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17515 [14:29:45] Reedy: i'm thinking about https://www.mediawiki.org/wiki/Developer_access/Archive#User:Paul_Oranje which says he had prior SVN but the [[special:log/newusers]] shows that i made it myself. does he still have svn lingering somewhere? i can't find it (USERINFO/LDAP/users.php) [14:30:39] maybe he's mistaken [14:30:48] <^demon|away> You could look at the authz list, that's a good place to start. [14:30:52] i agree. i just want to know how i should be checking [14:34:46] bastion1:~$ getent group | cut -d: -f -3 | egrep ':5[0-9]{2}$' [14:34:46] wikidev:*:500 [14:34:46] svn:*:550 [14:38:39] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 672s [14:38:40] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 672s [14:39:23] andrewbogott: have a few mins? [14:39:56] jeremyb: I will as soon as I finish my breakfast (5 minutes or so). [14:39:59] What's up [14:40:02] ? [14:41:33] andrewbogott: k, enjoy. some labs LDAP weirdness. maybe you have no answers and I just have to talk to ryan [14:42:14] * jeremyb goes to pour some OJ ;) [14:42:51] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [14:42:52] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [14:45:54] New patchset: Mark Bergsma; "Make all lvs::realserver invocations parameterized" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17520 [14:46:32] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/17520 [14:47:49] New patchset: Mark Bergsma; "Make all lvs::realserver invocations parameterized" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17520 [14:48:26] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/17520 [14:48:34] mark: nice :) [14:48:44] New patchset: Mark Bergsma; "Make all lvs::realserver invocations parameterized" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17520 [14:49:06] let's cleanup this mess now before people make even more workarounds around lvs in labs [14:49:11] jeremyb: You're probably right that I have no answers, but what are you seeing? [14:49:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17520 [14:51:32] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17520 [14:53:12] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 647s [14:53:12] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 647s [14:55:25] mark: ♥ [14:57:08] New patchset: Mark Bergsma; "Get rid of $lvs_realserver_ips completely, only the param is now supported" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17523 [14:57:11] this has annoyed me for a long time [14:57:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17523 [14:58:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17523 [15:00:42] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 660s [15:00:42] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 660s [15:02:12] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 220 seconds [15:02:12] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 220 seconds [15:02:12] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 221 seconds [15:02:12] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 221 seconds [15:02:21] PROBLEM - MySQL Replication Heartbeat on db1001 is CRITICAL: CRIT replication delay 229 seconds [15:02:21] PROBLEM - MySQL Replication Heartbeat on db1001 is CRITICAL: CRIT replication delay 229 seconds [15:02:30] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 238 seconds [15:02:30] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 238 seconds [15:02:39] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 247 seconds [15:02:39] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 247 seconds [15:02:57] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 265 seconds [15:02:57] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 265 seconds [15:03:45] New patchset: Mark Bergsma; "Remove stupid labs LVS workarounds, remove duplicate lvs::realserver invocation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17524 [15:04:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17524 [15:04:32] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17524 [15:04:54] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 616s [15:04:54] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 616s [15:05:30] PROBLEM - MySQL Slave Delay on db1001 is CRITICAL: CRIT replication delay 206 seconds [15:05:30] PROBLEM - MySQL Slave Delay on db1001 is CRITICAL: CRIT replication delay 206 seconds [15:05:48] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 294 seconds [15:05:48] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 294 seconds [15:05:57] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 232 seconds [15:05:57] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 232 seconds [15:06:06] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 239 seconds [15:06:07] PROBLEM - MySQL Slave Delay on db1043 is CRITICAL: CRIT replication delay 207 seconds [15:06:07] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 239 seconds [15:06:07] PROBLEM - MySQL Slave Delay on db1043 is CRITICAL: CRIT replication delay 207 seconds [15:06:33] PROBLEM - MySQL Slave Delay on db1033 is CRITICAL: CRIT replication delay 235 seconds [15:06:33] PROBLEM - MySQL Slave Delay on db1033 is CRITICAL: CRIT replication delay 235 seconds [15:07:00] RECOVERY - MySQL Slave Delay on db1001 is OK: OK replication delay 21 seconds [15:07:00] RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay 25 seconds [15:07:00] RECOVERY - MySQL Slave Delay on db1001 is OK: OK replication delay 21 seconds [15:07:00] RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay 25 seconds [15:07:18] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 0 seconds [15:07:18] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 0 seconds [15:07:27] RECOVERY - MySQL Slave Delay on db1043 is OK: OK replication delay 0 seconds [15:07:27] RECOVERY - MySQL Slave Delay on db1043 is OK: OK replication delay 0 seconds [15:07:54] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 631s [15:07:54] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 631s [15:08:03] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [15:08:03] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [15:08:12] RECOVERY - MySQL Replication Heartbeat on db1001 is OK: OK replication delay 0 seconds [15:08:12] RECOVERY - MySQL Replication Heartbeat on db1001 is OK: OK replication delay 0 seconds [15:08:46] New patchset: Mark Bergsma; "Make the labs service IPs follow the correct hash structure" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17525 [15:09:10] hm [15:09:21] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/17525 [15:09:21] could we maybe have nagios-wm not duplicated? [15:09:24] RECOVERY - MySQL Slave Delay on db1033 is OK: OK replication delay 0 seconds [15:09:25] RECOVERY - MySQL Slave Delay on db1033 is OK: OK replication delay 0 seconds [15:09:33] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay 0 seconds [15:09:33] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay 0 seconds [15:09:52] New patchset: Mark Bergsma; "Make the labs service IPs follow the correct hash structure" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17525 [15:10:18] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 1 seconds [15:10:18] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 1 seconds [15:10:18] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 1 seconds [15:10:18] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 1 seconds [15:10:21] mark: so, notpeter send a new iteration of his work, as a module [15:10:29] yeah [15:10:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17525 [15:10:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17525 [15:11:23] hashar: ok, rebase on this [15:11:33] mark: lets goooo [15:11:51] and then make some of those changes selectors [15:12:00] or use inheritance or something [15:12:11] what should I do with all the 'search_pool1' => {}, stuff ? [15:12:21] leave it I think? [15:12:25] k [15:12:29] you're not dealing with search servers now are you [15:12:36] so I don't think that will error [15:12:41] if it does we'll allocate ips for them [15:12:50] but I already did that change [15:12:51] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [15:12:57] you don't need to change lvs.pp now [15:13:03] ohh [15:13:04] h [15:13:15] well you said to leave them ? :-D [15:13:20] ah [15:13:23] the change you're doing now is fine [15:13:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [15:13:36] * hashar is happy [15:13:40] except I wish the order was the same as for production [15:13:59] i'll change your change [15:14:21] I have put all the undef at bottom to make it obvious they are not set [15:14:57] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [15:14:57] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 620s [15:14:57] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [15:14:57] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 620s [15:15:55] hiyaaa mutante [15:17:50] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [15:17:50] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [15:18:26] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [15:18:26] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [15:19:06] mutante: EU is missing you :-( [15:20:27] hashar: when are you moving? ;) [15:20:46] New patchset: Mark Bergsma; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [15:21:04] jeremyb: staying in France :-D [15:21:23] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/13304 [15:21:54] hashar: that's what you think [15:21:55] mark: applying on labs [15:22:21] doesn't work [15:22:27] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not parse for environment production: Syntax error at '{'; expected '}' at /etc/puppet/manifests/role/cache.pp:472 on node i-0000031c.pmtpa.wmflabs [15:22:52] yeah, fixing [15:23:03] i hate puppet for this [15:23:16] we should all have a pre commit hook to run puppet parser validate first :-D [15:23:21] (too lazy to set it up) [15:23:42] hashar: that means you need puppet locally! [15:23:54] I do have it [15:24:01] sudo gem install puppet [15:24:03] \O/ [15:24:04] hashar: better would be to just reject the change before adding it to the DB, etc. if it fails [15:24:09] does not need a puppetmaster to be setup locally [15:24:16] that is just doing syntax check [15:24:51] i still think it's a bit too onerous for every developer [15:25:00] onerous ? [15:26:02] too much burden [15:26:16] well that is just a command to run [15:26:26] better than waiting for gerrit to let you know something goes wrong [15:26:34] sure, people can do both [15:26:43] but i think gerrit should just do it earlier [15:26:51] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [15:26:51] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [15:27:06] anyway, minor issue ;) [15:29:40] New patchset: Mark Bergsma; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [15:30:19] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/13304 [15:31:00] New patchset: Mark Bergsma; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [15:31:37] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/13304 [15:32:08] lovely puppet [15:33:48] mark: what should we do re: style? [15:34:11] want to get more opinions? want to think about it more? [15:34:33] tabs vs spaces? [15:34:46] well that, and style guide in general [15:35:00] New patchset: Mark Bergsma; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [15:35:16] e.g. I'd like to introduce a rule that says "everything that goes to modules/ should be formatted according to $STYLE_GUIDE" [15:35:37] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/13304 [15:35:50] and personally I'd love $STYLE_GUIDE to be "official puppet style guide" [15:36:19] doing style cleanups at a later point will mess with blame history for a second time [15:36:28] mark: still at err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not parse for environment production: Syntax error at '{'; expected '}' at /etc/puppet/manifests/role/cache.pp:506 on node i-0000031c.pmtpa.wmflabs [15:36:29] :( [15:36:39] plus, I'm willing to work on it now :-) [15:36:39] yeah i'm still playing with it [15:36:57] paravoid: I can't help it, I hate some of the things in that style guide [15:37:14] the 2 space indentation being the worst of it [15:37:30] * jeremyb doesn't understand what's so bad about it [15:37:33] and I really don't expect that we'll be exchanging modules with them much, so in that light, I think: fuck that [15:37:42] but /me has resolved to not understand ;) [15:37:52]