[00:12:20] logmsgbot: Dead? [00:29:01] No morebots [01:09:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [02:26:28] !log LocalisationUpdate completed (1.23wmf10) at 2014-01-19 02:26:28+00:00 [02:37:39] !log LocalisationUpdate completed (1.23wmf11) at 2014-01-19 02:37:39+00:00 [02:40:54] Reedy: If I look on Tools, morebots has a running instance "production". Shouldn't this lead to it listening here? [02:50:07] scfc_de: it many need restarted [02:50:26] scfc_de: Ive seen a few IRC bots not always recover nicely [02:51:00] Betacommand: Okay, I'll give it a go. [02:51:59] scfc_de: Ive got one that should auto rejoin but for some reason like once a month it DC's and doesnt reconnect or die [02:56:18] !log Restarted morebots (production-logbot) [02:56:26] Logged the message, Master [02:56:31] Ah! [02:58:38] !log [09:59AEST] !log Updated kibana to bef3db2 [02:58:45] Logged the message, Master [03:11:21] (03PS1) 10Springle: repool db1028, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108294 [03:11:57] (03CR) 10Springle: [C: 032] repool db1028, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108294 (owner: 10Springle) [03:12:03] (03Merged) 10jenkins-bot: repool db1028, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108294 (owner: 10Springle) [03:12:20] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-01-19 03:12:20+00:00 [03:12:29] Logged the message, Master [03:12:53] !log springle synchronized wmf-config/db-eqiad.php 'repool db1028' [03:13:01] Logged the message, Master [03:17:36] (03PS1) 10Springle: depool db1024 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108296 [03:18:04] (03CR) 10Springle: [C: 032] depool db1024 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108296 (owner: 10Springle) [03:18:10] (03Merged) 10jenkins-bot: depool db1024 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108296 (owner: 10Springle) [03:19:03] !log springle synchronized wmf-config/db-eqiad.php 'depool db1024' [03:19:09] Logged the message, Master [03:58:52] !log ebernhardson synchronized php-1.23wmf11/extensions/Flow [03:58:59] Logged the message, Master [04:09:43] !log cherry-picked security fix to Flow for php-1.23wmf11 [04:09:50] Logged the message, Master [04:10:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [04:22:12] (03PS1) 10Springle: sideline db1024, RT 6672 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108301 [04:23:40] (03CR) 10Springle: [C: 032] sideline db1024, RT 6672 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108301 (owner: 10Springle) [04:30:13] (03PS1) 10Gerrit Patch Uploader: https://bugzilla.wikimedia.org/show_bug.cgi?id=56408 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108302 [04:30:15] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108302 (owner: 10Gerrit Patch Uploader) [04:56:48] (03PS1) 10Springle: rotate s7 master, demote db1039, promote db1033 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108303 [04:57:34] (03CR) 10Springle: [C: 032] rotate s7 master, demote db1039, promote db1033 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108303 (owner: 10Springle) [04:57:40] (03Merged) 10jenkins-bot: rotate s7 master, demote db1039, promote db1033 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108303 (owner: 10Springle) [05:01:32] !log springle synchronized wmf-config/db-eqiad.php 'promote db1033 s7 master' [05:01:38] Logged the message, Master [05:05:50] PROBLEM - MySQL Replication Heartbeat on db1028 is CRITICAL: CRIT replication delay 307 seconds [05:06:10] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:06:10] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 332 seconds [05:06:20] PROBLEM - MySQL Replication Heartbeat on db1041 is CRITICAL: CRIT replication delay 334 seconds [05:16:09] (03PS1) 10Springle: s7 master switch to db1033 [operations/dns] - 10https://gerrit.wikimedia.org/r/108304 [05:16:29] (03CR) 10Springle: [C: 032] s7 master switch to db1033 [operations/dns] - 10https://gerrit.wikimedia.org/r/108304 (owner: 10Springle) [05:17:00] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:19:00] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [05:19:10] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay -0 seconds [05:19:20] RECOVERY - MySQL Replication Heartbeat on db1041 is OK: OK replication delay -0 seconds [05:19:50] RECOVERY - MySQL Replication Heartbeat on db1028 is OK: OK replication delay -0 seconds [05:21:00] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay -1 seconds [05:24:39] (03PS1) 10Springle: Update coredb topology for mha. [operations/puppet] - 10https://gerrit.wikimedia.org/r/108305 [05:26:03] (03CR) 10Springle: [C: 032] Update coredb topology for mha. [operations/puppet] - 10https://gerrit.wikimedia.org/r/108305 (owner: 10Springle) [06:20:44] (03PS1) 10Springle: db63 ganglia aggregator for remaining pmtpa db boxes [operations/puppet] - 10https://gerrit.wikimedia.org/r/108308 [06:21:54] (03CR) 10Springle: [C: 032] db63 ganglia aggregator for remaining pmtpa db boxes [operations/puppet] - 10https://gerrit.wikimedia.org/r/108308 (owner: 10Springle) [06:23:56] (03PS1) 10Springle: pmtpa s7 master update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108309 [06:24:05] (03CR) 10jenkins-bot: [V: 04-1] pmtpa s7 master update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108309 (owner: 10Springle) [06:26:28] (03PS2) 10Springle: pmtpa s7 master update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108309 [06:26:55] (03CR) 10Springle: [C: 032] pmtpa s7 master update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108309 (owner: 10Springle) [06:27:02] (03Merged) 10jenkins-bot: pmtpa s7 master update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108309 (owner: 10Springle) [06:27:57] !log springle synchronized wmf-config/db-pmtpa.php 's7 master update' [06:28:04] Logged the message, Master [06:52:03] (03PS1) 10Springle: Disable dbstats slow query until better solution can be found. [operations/puppet] - 10https://gerrit.wikimedia.org/r/108310 [06:53:24] (03CR) 10Springle: [C: 032] Disable dbstats slow query until better solution can be found. [operations/puppet] - 10https://gerrit.wikimedia.org/r/108310 (owner: 10Springle) [07:04:21] !log xtrabackup clone db1007 to db1039 [07:04:29] Logged the message, Master [07:09:45] (03PS1) 10Springle: depool db1022 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108311 [07:10:06] (03CR) 10Springle: [C: 032] depool db1022 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108311 (owner: 10Springle) [07:10:12] (03Merged) 10jenkins-bot: depool db1022 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108311 (owner: 10Springle) [07:11:03] !log springle synchronized wmf-config/db-eqiad.php 'depool db1022' [07:11:10] Logged the message, Master [07:11:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [07:53:03] (03PS1) 10Matanya: coredb_mysql: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/108313 [08:24:12] (03PS1) 10Matanya: Torrus: move from manutius to netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108314 [08:25:29] (03PS2) 10Matanya: Torrus: move from manutius to netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108314 [08:57:00] matanya, you rock [08:57:12] what ori ? [08:57:26] you've been on a spree, it's cool [08:59:03] Thanks ori ! :) [09:02:57] (03CR) 10Ori.livneh: [C: 04-1] "Nice; small quibbles inline but good overall" (037 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [09:08:28] (03CR) 10Ori.livneh: [C: 04-1] "I generally like leaving user / group / mode unspecified for config files. The defaults (root / root / 0644) are sensible and the result i" (037 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 (owner: 10Matanya) [09:10:42] (03PS6) 10Matanya: gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 [09:11:27] (03CR) 10Ori.livneh: etherpad: convert into a module (035 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107567 (owner: 10Matanya) [09:16:11] (03CR) 10Ori.livneh: [C: 04-1] "Sorry :P" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [09:18:30] (03PS7) 10Matanya: gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 [09:24:17] matanya: https://gerrit.wikimedia.org/r/#/c/107555/7/modules/gitblit/manifests/init.pp omits the group {}.. [09:25:05] where ori ? [09:25:24] do you mean it sould be user => gitblit [09:25:29] matanya: well, where is the gitblit group resource? [09:25:31] group => gitblit [09:29:31] but there needs to be an actual group resource [09:30:01] (03PS11) 10Matanya: svn: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 [09:30:30] ori: can you take a look at the server a sec and see what it is? [09:31:04] matanya: i don't have the manifest open any more. which file specifically? [09:31:20] modules/gitblit/manifests/init.pp [09:31:34] no, i mean, which file should i check on the server? [09:31:42] i presume you meant i should check whom the owner/group of a file is [09:32:48] /var/lib/gitblit/data/gitblit.properties [09:33:05] and /etc/init.d/gitblit [09:34:09] -r--r--r-- 1 gitblit gitblit 49883 Sep 19 17:44 /var/lib/gitblit/data/gitblit.properties [09:34:15] -r-xr-xr-- 1 gitblit gitblit 1162 Jul 23 22:44 /etc/init.d/gitblit [09:34:22] ok, so gitblit it is [09:34:23] thanks [09:49:15] (03PS8) 10Matanya: gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 [09:53:46] (03PS5) 10Matanya: etherpad: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107567 [10:11:26] matanya: should we do gitblit? what do you say? [10:12:13] ori: wouldn't input from chad be appropriate? [10:12:28] he is the main maintainer/user, no? [10:12:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [10:12:52] yeah, but you're not changing functionality [10:13:17] you're just organizing it better, in accordance with principles that apply at the repo level [10:13:39] if this is your view, go ahead :) [10:18:17] (03PS9) 10Ori.livneh: gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [10:19:45] (03CR) 10Ori.livneh: [V: 032] gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [10:20:10] my money's on four fix-up commits [10:20:30] (03CR) 10Ori.livneh: [C: 032] gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [10:21:03] now lets see git.wikimeida.org isn't broken [10:21:09] *wikimedia [10:21:30] i need to merge and run puppet [10:21:42] puppet commits >10 lines tend to conceal about 9 bugs [10:22:25] yeah, needed 9 CR's to make this one happen :/ [10:22:35] matanya: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class gitblit::instance at /etc/puppet/manifests/role/gitblit.pp:12 on node antimony.wikimedia.org [10:22:40] that's #1 :) [10:23:10] * ori waits for fix-up [10:23:32] just a sec [10:26:29] * ori seals the exits [10:28:17] matanya: I still appreciate your work, just have been preoccupied with migration stuff lately. Planning to review a couple of your patches now... [10:28:26] ori, you haven't tested https://gerrit.wikimedia.org/r/#/c/107590/2 yet, right? I'll do that one now if not. [10:28:51] andrewbogott: nope, i haven't! [10:29:20] (03PS1) 10Matanya: gitblit: fix for https://gerrit.wikimedia.org/r/#/c/107555/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/108318 [10:29:39] ori: ^ [10:29:59] (03CR) 10Ori.livneh: [C: 032 V: 032] gitblit: fix for https://gerrit.wikimedia.org/r/#/c/107555/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/108318 (owner: 10Matanya) [10:30:09] thanks andrewbogott , it seems i created more work for ops, than help them [10:30:50] matanya: Absolutely everything you've done was on my to-do list! It's just a very long list :) [10:31:16] So, definitely reducing my workload, not increasing. [10:31:26] good to know that :) [10:31:46] ori: can we unseal the exit? [10:32:07] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Role::Gitblit] is already defined; cannot redefine at /etc/puppet/manifests/role/gitblit.pp:12 on node antimony.wikimedia.org [10:32:11] that's #2 :) [10:32:21] stupid scope [10:32:28] fix on the way [10:32:36] * ori snickers. [10:33:39] this happens to me all the time, by the way [10:33:40] ori: to make sure: it should be class {'::gitblit: [10:33:56] correct? [10:34:01] yes, except with a space between { and ' :) [10:34:12] yeah, yeah :P [10:34:12] I think ori is cheating -- by controlling the number of bugs he reports at a time he can influence the number of fixups. [10:34:33] i go as far as puppetd -tv goes [10:34:40] no more, no less [10:35:26] (03PS1) 10Matanya: gitblit: scope fix for https://gerrit.wikimedia.org/r/#/c/107555/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/108319 [10:36:20] ori , andrewbogott aren't you both suppose to do something nice on sunday? isn't it a day off or something? [10:36:33] (03PS2) 10Ori.livneh: gitblit: scope fix for https://gerrit.wikimedia.org/r/#/c/107555/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/108319 (owner: 10Matanya) [10:36:42] (03CR) 10Ori.livneh: [C: 032 V: 032] role::gitblit: fix scope resolution error [operations/puppet] - 10https://gerrit.wikimedia.org/r/108319 (owner: 10Matanya) [10:37:04] waiting for #3 [10:38:28] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Role::Gitblit] is already defined; cannot redefine at /etc/puppet/manifests/role/gitblit.pp:12 on node antimony.wikimedia.org [10:38:42] now i'm puzzled [10:40:07] ori: any clue? [10:40:19] it is only on manifests/role/gitblit.pp [10:40:27] it's the system role, i think [10:40:34] system::role { 'role::gitblit': description => 'Gitblit, a git viewer' } [10:40:43] so? [10:40:47] that is valid [10:40:58] or should be [10:41:26] yeah, you right [10:41:59] I think you need ::role::gitblit [10:42:19] when you include it. [10:42:35] that is the change i did [10:42:42] i hate scoping [10:42:44] um, ok, sorry, I'm behind. [10:43:20] i'm still not sure [10:43:50] what do you think ori ? [10:44:00] ..still not sure :) but looking. [10:44:56] oh, it's just me being dumb. i forgot to merge your commit on puppetmaster. [10:45:22] oh, ok. not push then. [10:45:25] the fact that the error was different is just a consequence of the order not being rigidly defined [10:45:58] err: /Stage[main]/Gitblit/User[gitblit]/home: change from /var/lib/gitblit to /nonexistent failed: Could not set home on user[gitblit]: Execution of '/usr/sbin/usermod -d /nonexistent gitblit' returned 8: usermod: user gitblit is currently logged in [10:46:01] not an error per se [10:46:06] but probably best to keep that /var/lib/gitblit [10:46:51] matanya: want to fix or should i? [10:47:10] meed just to declare a new home? [10:47:33] meed? [10:47:58] *nedd [10:48:01] **need [10:48:03] arrg [10:48:57] yeah, i'd set the home => '/nonexistent', line on the gitblit user resource to its current value on antimony (/var/lib/gitblit) [10:49:06] want to submit or should i? [10:49:15] (03PS1) 10Matanya: gitblit: correcting user home dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/108320 [10:49:18] ^ [10:51:01] (03PS2) 10Ori.livneh: gitblit: correcting user home dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/108320 (owner: 10Matanya) [10:51:21] (03CR) 10Ori.livneh: [C: 032 V: 032] gitblit: make gitblit user home dir /var/lib/gitblit [operations/puppet] - 10https://gerrit.wikimedia.org/r/108320 (owner: 10Matanya) [10:51:22] i like your commit messages [10:51:29] i should learn that [10:51:58] my look like i grabed them from http://whatthecommit.com/ [10:52:06] *mine [10:52:34] it's a joy to have a repo with a nice, descriptive history [10:52:48] agreed [10:53:01] If nothing else, code review is really good for forcing descriptive commit messages. [10:58:35] matanya: so, first the bad news [10:59:00] * matanya is listening [10:59:01] the gitblit init script doesn't have a 'status', so there's a bit of log churn as puppet keeps calling /etc/init.d/gitblit start on each run [10:59:37] but it's not actually doing anything; it correctly detects that it's already running. so it's really just log churn. something to fix, but not stop-the-world urgent. [11:00:15] the good news is that everything else worked [11:00:38] ori: it is a matter of has_status => false [11:00:58] which is quite easy fix. as foe the rest: Yay :) [11:00:59] !log graceful'd apache on antimony to verify Ibd528f8f2 [11:01:08] Logged the message, Master [11:02:01] i note for those following along that has_status would be patch #4 [11:02:02] yup, hard refresh, and it is still there :) [11:02:54] matanya: nice work! and thanks again for all that work. [11:05:41] (03PS1) 10Matanya: gitblit: service has no status [operations/puppet] - 10https://gerrit.wikimedia.org/r/108321 [11:05:49] Thanks a lot ori, here is the fix ^ [11:06:02] tomorrow! i'm done for the night, sorry [11:06:27] but feel free to poke me tomorrow sometime [11:06:44] sure. have a great night and thanks again :) [11:08:11] same, bye [11:21:39] (03CR) 10Andrew Bogott: [C: 031] "Looks good, and I've confirmed on labs that this is a no-op." [operations/puppet] - 10https://gerrit.wikimedia.org/r/107590 (owner: 10Matanya) [11:21:44] (03PS3) 10Andrew Bogott: url-downloader: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107590 (owner: 10Matanya) [11:23:18] (03CR) 10Andrew Bogott: [C: 032] url-downloader: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107590 (owner: 10Matanya) [11:34:47] (03PS2) 10Andrew Bogott: apt: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107358 (owner: 10Matanya) [11:34:55] (03CR) 10jenkins-bot: [V: 04-1] apt: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107358 (owner: 10Matanya) [11:44:18] andrewbogott: you want me to rebase it? [11:44:45] Oh, I missed that it failed :) Sure, go ahead and rebase if you're not in the middle of something. [11:51:18] andrewbogott: weird, it refuses to push: (no changes made) [11:51:36] Just change a word in the commit message :) [11:52:53] (03PS3) 10Matanya: apt: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107358 [11:54:55] ok, here andrewbogott ^ [11:55:06] thanks, running a quick test now... [12:02:20] (03PS3) 10Andrew Bogott: generic: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [12:02:23] (03PS8) 10Andrew Bogott: ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [12:03:17] (03CR) 10Andrew Bogott: [C: 032] apt: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107358 (owner: 10Matanya) [12:18:41] (03CR) 10Andrew Bogott: [C: 032] generic: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [12:39:54] (03CR) 10Andrew Bogott: [C: 032] ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [12:44:42] ah, a classic loophole. https://office.wikimedia.org/w/index.php?title=Special:UserLogin try to clock on the office IT link :) [12:44:49] *click [12:47:30] (03PS1) 10Andrew Bogott: Remove single-quotes around a $variable. [operations/puppet] - 10https://gerrit.wikimedia.org/r/108324 [12:49:16] (03CR) 10Andrew Bogott: [C: 032] "(obv.)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108324 (owner: 10Andrew Bogott) [12:55:37] Do you have to be logged in to get support about your login? [12:58:25] matanya, I'm heading to dinner. FYI, tomorrow is a US holiday so IRC will be a bit quiet. [12:58:51] thanks a lot andrewbogott [12:59:03] good evening, or what ever it is [13:13:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [15:42:57] Hi guys, someone knows why I can login correctly on wikitech.wikimedia.org but not anymore on gerrit? [15:44:02] do you have a user name there? [15:44:17] yes, sure I did some code rewiew [15:44:27] but from today i can't connect anymore [15:45:16] any error message? [15:45:43] what is your username? [15:45:53] a simple Invalid username or password. [15:46:00] guido.iaquinti [15:47:01] ilmerovingio: try to reach tim @ wikimedia-labs [15:47:16] #wikimedia-labs [15:47:40] i must leave. sorry couldn't help [15:47:49] don't worry, many thanks anyway [16:14:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [17:49:59] (03PS2) 10Matanya: RT: allow login via LDAP [operations/puppet] - 10https://gerrit.wikimedia.org/r/80577 (owner: 10Faidon Liambotis) [17:50:51] diff? [17:51:43] (03CR) 10Matanya: [C: 031] "patchset 1 was rebased." [operations/puppet] - 10https://gerrit.wikimedia.org/r/80577 (owner: 10Faidon Liambotis) [17:52:38] no diff, just merge please paravoid :) [17:52:54] it's more complicated than that [17:53:08] it is always ... [17:53:24] Heh. [17:53:55] we need to audit the existing accounts [17:54:19] i defently can't do that [17:54:20] and make LDAP a first class service, not just some random service running on virt [17:54:24] I know [17:54:41] I wasn't asking, I'm just explaining why it's more complicated :) [17:54:47] I've noted this on the changeset itself [17:54:57] ah, good to know [17:55:21] how's the puppet 3 work? need any merges there? [17:55:56] 4 patches [17:55:58] looking now [17:56:08] https://gerrit.wikimedia.org/r/#/c/108313/ [17:56:13] https://gerrit.wikimedia.org/r/#/c/107828/ [17:56:18] https://gerrit.wikimedia.org/r/#/c/107823/ [17:56:23] https://gerrit.wikimedia.org/r/#/c/107819/ [17:56:26] at your ease [17:56:32] (03CR) 10Faidon Liambotis: [C: 04-1] "As noted above, this was the not-so-great solution. Can you do the better one? :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 (owner: 10Matanya) [17:57:09] ok :) [17:57:15] * matanya lazed a bit [18:20:28] (03PS3) 10Matanya: udp2log: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 [18:21:06] (03CR) 10jenkins-bot: [V: 04-1] udp2log: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 (owner: 10Matanya) [18:40:47] (03PS4) 10Matanya: udp2log: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 [18:41:26] (03CR) 10jenkins-bot: [V: 04-1] udp2log: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 (owner: 10Matanya) [18:43:21] (03PS5) 10Matanya: udp2log: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 [18:43:58] (03CR) 10jenkins-bot: [V: 04-1] udp2log: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 (owner: 10Matanya) [18:47:19] (03PS6) 10Matanya: udp2log: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107828 [18:48:40] !log cr1-sdtpa/cr2-pmtpa: rolling back config to pre-Tampa outage [18:48:47] Logged the message, Master [18:49:03] Gloria: around? [18:50:50] Gloria: when you get back, please confirm that bugzilla works for you; we had disabled one of our transits because of an issue reported by you and it's hard for me to confirm it's working for everyone [18:50:56] paravoid: i think i'm drunk [18:55:31] (03PS1) 10Faidon Liambotis: Revert wmgRC2UDPAddress to ekrem [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108339 [18:55:52] (03CR) 10Faidon Liambotis: [C: 032] Revert wmgRC2UDPAddress to ekrem [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108339 (owner: 10Faidon Liambotis) [18:56:53] !log faidon updated /a/common to {{Gerrit|I48a026eb8}}: Revert wmgRC2UDPAddress to ekrem [18:57:00] Logged the message, Master [18:58:02] paravoid: Was the BZ issue specific to Gloria? [18:58:12] paravoid: Definitely working for me now. Thanks. :-) [18:58:22] If it hadn't been working, you've hard about it. ;-) [18:58:27] heard * [18:58:31] Words are hard. [18:58:45] scfc_de: no, but it was specific to some networks, one of which was Gloria's [18:59:04] !log faidon synchronized wmf-config/InitialiseSettings.php 'switch wmgRC2UDPAddress back to ekrem' [18:59:11] Logged the message, Master [18:59:13] paravoid: Ah, okay. [18:59:20] one of our transits was misbehaving, possibly affected by the fiber cuts themselves [18:59:27] (03PS5) 10Matanya: Cleanup the base class [operations/puppet] - 10https://gerrit.wikimedia.org/r/33066 (owner: 10Faidon Liambotis) [18:59:50] and we use multiple transits, each of which is well-connected to thousands of networks [19:01:01] (03CR) 10Matanya: [C: 031] "patchset 4 was rebased." [operations/puppet] - 10https://gerrit.wikimedia.org/r/33066 (owner: 10Faidon Liambotis) [19:05:11] (03PS2) 10Brian Wolff: https://bugzilla.wikimedia.org/show_bug.cgi?id=56408 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/108302 (owner: 10Gerrit Patch Uploader) [19:08:02] (03PS3) 10Matanya: Reexec procps on changes [operations/puppet] - 10https://gerrit.wikimedia.org/r/94949 (owner: 10Mark Bergsma) [19:15:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [19:51:25] (03PS2) 10Matanya: gitblit: service has no status [operations/puppet] - 10https://gerrit.wikimedia.org/r/108321 [19:51:37] oops [19:52:02] (03CR) 10Matanya: "Ori, please abandon this change. it was done already." [operations/puppet] - 10https://gerrit.wikimedia.org/r/96746 (owner: 10Ori.livneh) [19:54:19] (03CR) 10Matanya: [C: 031] retab mail.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/104806 (owner: 10Hashar) [19:59:06] (03CR) 10JanZerebecki: [C: 04-1] "This might not work as intended, as puppet still has no way to know if gitblit is running or not. start-stop-daemon has the --status argum" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108321 (owner: 10Matanya) [19:59:55] (03CR) 10JanZerebecki: [C: 04-1] gitblit: service has no status [operations/puppet] - 10https://gerrit.wikimedia.org/r/108321 (owner: 10Matanya) [20:26:23] (03PS1) 10Tim Landscheidt: Set up redirects for toolserver.org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/108465 [20:31:41] (03PS1) 10Tim Landscheidt: Script to test Toolserver redirects [operations/software] - 10https://gerrit.wikimedia.org/r/108467 [20:53:57] (03CR) 10Matanya: "Thank you, I will try to think of a better way to achieve this." [operations/puppet] - 10https://gerrit.wikimedia.org/r/108321 (owner: 10Matanya) [22:07:41] PROBLEM - MySQL Slave Delay on db69 is CRITICAL: CRIT replication delay 313 seconds [22:07:50] PROBLEM - MySQL Replication Heartbeat on db69 is CRITICAL: CRIT replication delay 319 seconds [22:16:41] PROBLEM - MySQL Slave Delay on db69 is CRITICAL: CRIT replication delay 327 seconds [22:16:50] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Fri 17 Jan 2014 06:59:52 PM UTC [22:16:50] PROBLEM - MySQL Replication Heartbeat on db69 is CRITICAL: CRIT replication delay 334 seconds [22:26:40] RECOVERY - MySQL Slave Delay on db69 is OK: OK replication delay 18 seconds [22:26:50] RECOVERY - MySQL Replication Heartbeat on db69 is OK: OK replication delay -1 seconds