[00:01:34] (03CR) 10Ori.livneh: "cleaned up" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134282 (owner: 10Hoo man) [00:03:35] (03CR) 10GWicke: "Is it clear what is causing the underlying issue?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135875 (owner: 10Aaron Schulz) [00:16:54] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:16:33 UTC [00:18:54] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:17:55 UTC [00:23:54] PROBLEM - Puppet freshness on lvs1001 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:23:03 UTC [00:25:39] (03CR) 10Ori.livneh: [C: 031] scap: ensure=>absent /usr/local/bin/sync-common-file [operations/puppet] - 10https://gerrit.wikimedia.org/r/135924 (owner: 10BryanDavis) [00:25:56] (03CR) 10Ori.livneh: [C: 031] scap: /usr/local/bin/sync-common-file is unused [operations/puppet] - 10https://gerrit.wikimedia.org/r/135925 (owner: 10BryanDavis) [00:29:54] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:29:03 UTC [00:33:54] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:32:59 UTC [00:40:37] https://gerrit.wikimedia.org/r/#/admin/projects/phabricator/phabricator appears to be broken [00:40:44] rather. the git repo is broken [00:40:54] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:40:36 UTC [00:41:26] $ git clone https://gerrit.wikimedia.org/r/phabricator/phabricator [00:41:26] Cloning into 'phabricator'... [00:41:26] Checking connectivity... done. [00:41:26] warning: remote HEAD refers to nonexistent ref, unable to checkout. [00:41:36] <^d> Nobody's pushed anything to master yet. [00:41:48] <^d> HEAD's pointing at refs/heads/master [00:43:01] so the repo's empty [00:43:05] <^d> Yep. [00:43:09] nice. [00:47:54] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:46:52 UTC [01:03:44] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Thu May 29 01:03:42 UTC 2014 [02:00:45] (03CR) 10BBlack: [C: 032 V: 032] fix daemonization stdio stuff [operations/debs/pybal] - 10https://gerrit.wikimedia.org/r/134651 (owner: 10BBlack) [02:36:48] !log LocalisationUpdate completed (1.24wmf5) at 2014-05-29 02:35:45+00:00 [02:36:54] Logged the message, Master [02:43:02] (03CR) 10Jeremyb: [C: 04-1] "John, do you want to fix "newikimedia.org"? (no new domains are being registered for this bug)" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133991 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [02:45:25] (03CR) 10Jeremyb: "renew +1" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133981 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [03:06:13] (03CR) 10Jeremyb: "Reedy, what do you think about DB name?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133982 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [03:06:42] (03CR) 10Jeremyb: "renew +1" [operations/dns] - 10https://gerrit.wikimedia.org/r/133980 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [03:11:18] !log LocalisationUpdate completed (1.24wmf6) at 2014-05-29 03:10:15+00:00 [03:11:23] Logged the message, Master [03:17:54] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:16:33 UTC [03:18:14] (03PS1) 10Springle: repool db1009 after raid tests [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/135991 [03:18:47] (03CR) 10Springle: [C: 032] repool db1009 after raid tests [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/135991 (owner: 10Springle) [03:18:56] (03Merged) 10jenkins-bot: repool db1009 after raid tests [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/135991 (owner: 10Springle) [03:19:31] !log springle Synchronized wmf-config/db-eqiad.php: repool db1009 in s2 (duration: 00m 08s) [03:19:54] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:17:55 UTC [03:24:54] PROBLEM - Puppet freshness on lvs1001 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:23:03 UTC [03:30:54] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:29:03 UTC [03:32:25] ori: yt [03:32:33] springle: hey [03:32:43] what did i break? [03:33:40] for some reason sync-file isn't syncing db-eqiad.php. wondering if the mediawiki::sync mearge earlier means i have to do something differently? [03:34:05] (wild guess looking at commit log and seeing 'sync') [03:35:02] i'll take a look [03:35:16] it deliberately doesn't work if the file has syntax errors [03:35:28] and it also has been hanging after successfully syncing at the !log part [03:36:03] it said this to me: http://paste.debian.net/102338/ [03:36:27] that looks right [03:36:32] which looks fine. only no traffic on db1009, and a random mw node has old db-eqiad.php [03:37:49] ok, first things first, to unblock you [03:37:55] sync-file has been recently ported to python [03:37:59] here's the old shell script: https://dpaste.de/Fwzr/raw [03:38:05] you can just run that for the time being [03:38:08] and i'll keep looking [03:38:26] bd808|BUFFER: fyi ^^ [03:38:33] ok thanks [03:39:41] heh [03:39:59] that's already broken by gerrit 135924 [03:40:25] oh not merged [03:40:37] well, can't find /usr/local/bin/sync-common-file [03:40:46] ori: i'll wait :) it's no hurry for db1009 [03:40:51] d'oh [03:40:58] though i might panic if we suddenly have to depool something [03:41:54] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:40:36 UTC [03:43:49] springle: try now [03:44:10] !log springle Synchronized wmf-config/db-eqiad.php: repool db1009 in s2 (duration: 00m 08s) [03:44:30] nope [03:44:33] same [03:45:52] !log disabled puppet on tin and copied sync-common-file from mediawiki/tools/scap@8f2a8356c38 into /usr/local/bin to debug sync issue [03:45:56] Logged the message, Master [03:48:06] wow, wtf [03:48:11] oh wait [03:48:20] try now -> which method [03:48:24] old or new [03:48:28] old [03:48:35] sorry, i should have been explicit [03:48:43] np trying [03:48:54] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:46:52 UTC [03:49:01] yep old works [03:49:32] ok, the reason i had to disable puppet is that sync-common-file has an ensure => absent on its head at the moment, since it was just purged from the cluster today [03:49:32] ori: thanks. i'll leave you in peace to debug :) [03:49:39] ok [03:49:41] since it's already gone from all hosts, i'll submit a commit to remove it from puppet [03:49:47] and i'll leave just tin's copy in place [03:49:50] !log springle synchronized wmf-config/db-eqiad.php 'repool db1009 in s2, take #2' [03:49:51] for now, until i figure it out [03:49:56] Logged the message, Master [03:51:57] !log db1009 mariadb 5.5.37 live trial with low load [03:52:02] Logged the message, Master [03:55:21] !log ori Synchronized README: Debugging sync-file (duration: 00m 06s) [03:55:26] Logged the message, Master [03:56:33] that worked just fine [03:56:36] that's so add [03:56:38] *odd [03:57:59] :) [03:58:08] if i had to guess, i'd say ssh agent forwarding is getting screwed up somehow [03:58:35] and each host is quietly rebuffing your attempt to log in [03:59:05] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu May 29 03:57:59 UTC 2014 (duration 57m 58s) [03:59:09] Logged the message, Master [04:00:50] i can definitely still manually ssh from tin to mw nodes [04:01:00] i guess the old sync-file proves that too [04:13:07] !log re-enabled puppet on tin [04:13:13] Logged the message, Master [04:15:05] https://gerrit.wikimedia.org/r/#/c/135994/ is a revert to the old status quo, if i get a review on that i'll sync it and leave it for bryan to debug [04:18:05] my money is on something odd with the --include --exclude stuff in new sync_common [04:20:42] !log updated scap to 9ba9014: Partially revert "Convert sync-dir and sync-file to python" [04:20:47] Logged the message, Master [04:22:45] springle: i just reverted it for now. could you sync /a/common/README to confirm? [04:26:59] !log springle synchronized README 'test sync-file' [04:27:03] Logged the message, Master [04:27:23] ori: seems ok [04:28:37] you might be right [04:29:23] the generated command-line is presumably rsync yadda yadda --include=db-eqiad.php --exclude=* [04:32:10] i normally run sync-file wmf-config/db-eqiad.php 'blah' [04:32:20] with cwd /a/common [04:32:41] i didn't think to try it directly from wmf-config dir [04:33:14] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:35:10] two points from the man page: [04:35:12] 1) "If a pattern excludes a particular parent directory, it can render a deeper include pattern ineffectual because rsync did not descend through that excluded section of the hierarchy. This is particularly important when using a trailing ’*’ rule." [04:35:31] 2) "if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname." [04:36:12] i don't know if it canonicalizes the path [04:36:20] but maybe it excluded . [04:37:10] anyways, things are consistent everywhere (no local hacks), and you're able to sync [04:37:13] so i'm calling it a night [04:37:17] * ori waves [04:37:36] ori: night! [04:41:04] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.006 second response time [06:18:54] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:16:33 UTC [06:20:54] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:17:55 UTC [06:25:54] PROBLEM - Puppet freshness on lvs1001 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:23:03 UTC [06:27:33] _joe_: 24h for you ^ :) [06:31:54] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:29:03 UTC [06:42:54] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:40:36 UTC [06:49:54] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Last successful Puppet run was Wed 28 May 2014 21:46:52 UTC [07:08:45] akosiaris: morning, wanted to touch base regarding sanger. where does it stand ? [07:09:49] matanya: meaning ? [07:10:37] I spoke to joel last night, asking him if he opened a ticket in rt regarding a replacemnt for sanger in eqiad [07:10:57] he said you are the person to conrtact regarding such a question, and you own the task [07:11:15] <_joe_> what the hell is happening to lvs servers? [07:11:24] <_joe_> akosiaris: did you took a look? [07:12:00] I know about #6163 but nothing moved on ever since [07:12:20] ah ok. So the LDAP part of sanger, yes I do own that task. We will be migrating the LDAP part to OpenLDAP and to a new machine in eqiad. I am developing the OpenLDAP module [07:12:27] _joe_: looking right now [07:15:06] duplicate definition ? [07:15:08] meh... [07:15:12] thanks akosiaris , please update the ticket in your spare time [07:15:27] akosiaris: i guess it is the admin yaml stuff [07:17:31] _joe_: the culprit seems to be https://gerrit.wikimedia.org/r/#/c/135928/ [07:17:48] bad guess :/ [07:19:55] <_joe_> akosiaris: will you fix that, or should we revert the change for now? [07:20:34] the puppet alert is not on those host changed in this patch [07:21:06] _joe_: trying to fix [07:21:44] matanya: sure it is. look at @@ -1643,6 +1646,12 @@ node /lvs100[1-6]\.wikimedia\.org/ { [07:22:39] stupid gerrit diff [07:22:49] why hide this? :/ [07:23:04] (03CR) 10Hashar: "Excellent! Thank you very much :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134739 (owner: 10Dzahn) [07:29:25] _joe_: here is a friend of yours. Look at class interface::bonding-tools [07:29:45] this is not going to break as long as it is included, but imagine a variable there ;-) [07:30:31] <_joe_> akosiaris: we're full of these [07:30:31] <_joe_> I thought we will change them once we need to [07:30:48] <_joe_> btw, merging naggen2 [07:30:56] <_joe_> I hope I don't screw neon up [07:30:57] yeah I know, I couldn't help myself though [07:31:34] (03PS9) 10Giuseppe Lavagetto: icinga: replace naggen [operations/puppet] - 10https://gerrit.wikimedia.org/r/135746 [07:35:37] (03CR) 10Giuseppe Lavagetto: [C: 032] icinga: replace naggen [operations/puppet] - 10https://gerrit.wikimedia.org/r/135746 (owner: 10Giuseppe Lavagetto) [07:36:58] <_joe_> hey ho, let's go! [07:38:58] (03PS1) 10Alexandros Kosiaris: Split interface-rps script File resources to class [operations/puppet] - 10https://gerrit.wikimedia.org/r/136000 [07:42:06] (03CR) 10Alexandros Kosiaris: [C: 032] Split interface-rps script File resources to class [operations/puppet] - 10https://gerrit.wikimedia.org/r/136000 (owner: 10Alexandros Kosiaris) [07:43:04] god I am an idiot [07:43:46] <_joe_> me too [07:43:47] <_joe_> I just broke something [07:44:47] (03PS1) 10Alexandros Kosiaris: Fix a typo introduced in 1a0605b [operations/puppet] - 10https://gerrit.wikimedia.org/r/136001 [07:45:40] <_joe_> akosiaris: let me merge a change quickli please [07:46:30] sure [07:46:39] (03PS1) 10Giuseppe Lavagetto: naggen2: fix template [operations/puppet] - 10https://gerrit.wikimedia.org/r/136002 [07:47:13] argh this is ugly :P [07:47:20] (03CR) 10Giuseppe Lavagetto: [C: 032] naggen2: fix template [operations/puppet] - 10https://gerrit.wikimedia.org/r/136002 (owner: 10Giuseppe Lavagetto) [07:47:50] <_joe_> akosiaris: yeah I was able to fuckup the simplest part of the commit [07:47:50] <_joe_> :/ [07:48:06] (03CR) 10Giuseppe Lavagetto: [V: 032] naggen2: fix template [operations/puppet] - 10https://gerrit.wikimedia.org/r/136002 (owner: 10Giuseppe Lavagetto) [07:49:24] btw. would it not work to do something like <% myvar = scope.lookupvar('::puppet::config') %> and then just do the evaluations ? like dsn <%= myvar['dbuser']:myvar['db'] %> and so on ? [07:49:24] <_joe_> akosiaris: that does not seem to be the only problem, crap. [07:49:40] (03CR) 10Alexandros Kosiaris: [C: 032] Fix a typo introduced in 1a0605b [operations/puppet] - 10https://gerrit.wikimedia.org/r/136001 (owner: 10Alexandros Kosiaris) [07:50:04] <_joe_> akosiaris: puppet is not running on the puppetmasters ATM [07:50:20] ok [07:50:27] <_joe_> akosiaris: the problem is, for some reason ::puppet::config is not available in that context [07:50:39] hmmm weird... [07:50:44] RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Thu May 29 07:50:43 UTC 2014 [07:50:48] <_joe_> now I have to figure out why [07:50:50] good to know [07:50:51] <_joe_> but go on with your changes :) [07:51:22] ok so I merged and this fixed the lvs puppet issue [07:51:26] <_joe_> I think that is the name clash :) [07:51:41] <_joe_> I know what is the best way to solve this [07:51:46] ahhhh, now it makes sense [07:52:08] ok I got it, yeah you were absolutely right yesterday :) [07:52:08] <_joe_> I told you that was a problem :) [07:52:58] <_joe_> so, now I have to understand how to manage this. [07:53:25] <_joe_> the easiest way would be to restore my stash that was reading from puppet.conf [07:53:34] <_joe_> in naggen2 [07:53:34] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Thu May 29 07:53:30 UTC 2014 [07:53:51] <_joe_> yeah ok, on it [07:55:05] <_joe_> ~ 20 mins and it will be ok [07:59:14] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Thu May 29 07:59:09 UTC 2014 [08:11:05] (03PS1) 10Giuseppe Lavagetto: naggen2: read config from puppet.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/136003 [08:11:34] RECOVERY - Puppet freshness on lvs1006 is OK: puppet ran at Thu May 29 08:11:32 UTC 2014 [08:12:04] <_joe_> [08:12:04] <_joe_> mmmh gerrit-wm where art thou? [08:12:54] (03PS2) 10Giuseppe Lavagetto: naggen2: read config from puppet.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/136003 [08:14:34] (03CR) 10Giuseppe Lavagetto: [C: 032] naggen2: read config from puppet.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/136003 (owner: 10Giuseppe Lavagetto) [08:14:50] grrrit-wm: _joe_ lookst for thee [08:15:12] But thou art here, in foro. [08:16:17] <_joe_> marktraceur: my ancient english knowledge is very limited [08:16:54] RECOVERY - Puppet freshness on lvs1005 is OK: puppet ran at Thu May 29 08:16:49 UTC 2014 [08:17:03] _joe_: "in foro" is actually overly complicated Latin, but the rest should be understandable. "in foro" means "in the forum" [08:17:15] ....roughly [08:17:40] <_joe_> I do understand in foro [08:17:43] Because "forum" in modern English could translate to too many things for it to be an accurate translation [08:17:44] RECOVERY - Puppet freshness on lvs1004 is OK: puppet ran at Thu May 29 08:17:40 UTC 2014 [08:17:50] <_joe_> I'm italian and I studied latin :P [08:17:59] Well then I will shut up. :) [08:18:00] <_joe_> I just can't answere you properly [08:18:48] <_joe_> (in foro means 'in the forum' in italian as well, even if we'd use an article) [08:18:54] _joe_: Something about pining after grrrit-wm and also gesturing meaningfully at a papier-mache' moon that your classmates created. My knowledge of Shakespeare is limited. [08:22:30] <_joe_> :) [08:24:19] (03PS1) 10Giuseppe Lavagetto: naggen2: fix ensure_resource [operations/puppet] - 10https://gerrit.wikimedia.org/r/136004 [08:26:49] (03CR) 10Giuseppe Lavagetto: [C: 032] naggen2: fix ensure_resource [operations/puppet] - 10https://gerrit.wikimedia.org/r/136004 (owner: 10Giuseppe Lavagetto) [08:28:06] <_joe_> I should do things with more calm [08:28:10] <_joe_> my bad [08:28:56] <_joe_> one last horrible commit and I'm done - I hope :( [08:29:28] <_joe_> If we did not have that NS clash, it would have been good the second time though [08:30:11] (03PS1) 10Giuseppe Lavagetto: naggen2: remove references to the config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/136005 [08:31:35] (03CR) 10Giuseppe Lavagetto: [V: 032] naggen2: remove references to the config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/136005 (owner: 10Giuseppe Lavagetto) [08:31:43] (03CR) 10Giuseppe Lavagetto: [C: 032] naggen2: remove references to the config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/136005 (owner: 10Giuseppe Lavagetto) [08:49:58] (03PS1) 10Giuseppe Lavagetto: naggen2: remove ensure_resource [operations/puppet] - 10https://gerrit.wikimedia.org/r/136006 [08:52:48] (03CR) 10Giuseppe Lavagetto: [C: 032] "This is hopefully the last one :(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136006 (owner: 10Giuseppe Lavagetto) [09:13:25] (03PS1) 10Filippo Giunchedi: install python-imaging on appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/136007 [09:15:45] (03PS1) 10Giuseppe Lavagetto: naggen2: improving log format and usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/136008 [09:17:00] (03CR) 10Giuseppe Lavagetto: [C: 031] install python-imaging on appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/136007 (owner: 10Filippo Giunchedi) [09:17:21] (03CR) 10Filippo Giunchedi: "I'm not an expert in the mediawiki module but it seems to make sense, please double check the RT too!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136007 (owner: 10Filippo Giunchedi) [09:17:36] (03CR) 10Giuseppe Lavagetto: [C: 032] naggen2: improving log format and usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/136008 (owner: 10Giuseppe Lavagetto) [09:18:19] matanya: nice (re: 24h) [09:18:27] :) [09:19:12] I was looking for someone familiar with mediawiki role for https://gerrit.wikimedia.org/r/#/c/136007/ + related RT [09:19:37] ah I see _joe_ +1'd too, thanks! [09:19:49] <_joe_> godog: yes I mean, it does no harm [09:20:01] <_joe_> that's why I gave +1 [09:20:26] <_joe_> also, you're installing python packages, you can't be wrong :P [09:21:21] haha true [10:42:36] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Thu 29 May 2014 07:41:48 UTC [10:47:07] <_joe_> uhm, admins.pp I guess [10:47:43] <_joe_> no, this is mine :( [10:50:53] (03CR) 10Nikerabbit: naggen2: remove ensure_resource (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/136006 (owner: 10Giuseppe Lavagetto) [10:52:17] <_joe_> Nikerabbit: there is one more serious problem there. I had a reason to use ensure_resource after all :( [10:57:36] PROBLEM - Puppet freshness on virt0 is CRITICAL: Last successful Puppet run was Thu 29 May 2014 07:57:07 UTC [10:58:05] (03PS1) 10Giuseppe Lavagetto: naggen: avoid clash with saltmaster for packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/136011 [11:05:14] (03CR) 10Giuseppe Lavagetto: [C: 032] "This fixes virt* puppet catalogs, and does not break prod puppet masters." [operations/puppet] - 10https://gerrit.wikimedia.org/r/136011 (owner: 10Giuseppe Lavagetto) [11:05:46] _joe_: I'm just learning by reading commits, giving drive-by comments while doing it [11:06:15] <_joe_> Nikerabbit: my commits from this morning are a shame :/ [11:07:26] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Thu May 29 11:07:24 UTC 2014 [11:07:43] _joe_: we all write bad code. the thing is to recognize it afterwards and avoid it in the future [11:08:28] <_joe_> Nikerabbit: no, very simply I was in a hurry to fix things and rushed a couple of commits [11:08:28] <_joe_> which is stupid and pointless [11:09:27] s/write bad code/do stupid things/ [11:09:28] <_joe_> well, now virt* are ok, so I can go to lunch [11:09:39] my lunch is ready too [11:27:36] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Thu May 29 11:27:29 UTC 2014 [11:35:39] _joe_: can you do a bigupload? [12:06:14] what is a bigupload? [12:49:34] paravoid: files over 100mb to commons. [12:49:43] I think it needs bugzilla [12:51:55] (03PS2) 10QChris: Allow to set up hive's auxpath globally [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/135539 [12:55:03] (03PS1) 10QChris: Use HCatalog as default auxpath for Hive [operations/puppet] - 10https://gerrit.wikimedia.org/r/136014 [12:56:39] (03CR) 10jenkins-bot: [V: 04-1] Use HCatalog as default auxpath for Hive [operations/puppet] - 10https://gerrit.wikimedia.org/r/136014 (owner: 10QChris) [12:57:02] (03CR) 10QChris: "> Hm, ok, but let's parameterize it then. Make a parameter [...]" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/135539 (owner: 10QChris) [12:58:43] <_joe_> matanya: I don't think I know how to do that sorry [12:58:58] thanks [13:00:20] (03CR) 10QChris: [C: 04-1] "Jenkins fails because the corresponding commit" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136014 (owner: 10QChris) [13:04:11] _joe_: future use: https://commons.wikimedia.org/wiki/Help:Server-side_upload [13:04:39] <_joe_> matanya: thanks, will read. I'm a total n00b [13:05:05] not really. but meh [13:05:15] found a work around [13:06:25] I think I managed to get some server side upload done a few months ago [13:07:04] video sucks on commons, in so many ways [13:07:40] I wish i could upload files from labs. will save so much time [13:10:10] I'm pretty sure you can, we even have a tool for that. :) http://tools.wmflabs.org/ia-upload/commons/init [13:13:58] Nemo_bis: it gives me internal eroor [13:14:02] *error [13:18:08] _joe_: the naggen files are not filebucketed [13:18:14] we have backup => false iirc [13:18:24] they are checksummed, but this can be disabled as well [13:18:29] with checksum => mtime [13:20:16] (03CR) 10Faidon Liambotis: [C: 032] Switch LVS to "performance" cpufreq governor [operations/puppet] - 10https://gerrit.wikimedia.org/r/135929 (owner: 10BBlack) [13:20:18] matanya: you may need to enable cookies [13:20:26] I have cookies [13:20:29] but that's for archive.org files for [13:20:33] + only [13:23:25] andre__: When you get a second, just a bit more detail on https://bugzilla.wikimedia.org/show_bug.cgi?id=65861 will help me track it down. [13:23:55] Coren, I'm not even sure if I understand the question correctly :-/ [13:24:07] (03CR) 10Faidon Liambotis: [C: 031] "Looks good, but add your name to Authors at the top :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135931 (owner: 10BBlack) [13:24:08] Coren, isn't that info in the header files in comment 0? [13:24:25] (03CR) 10Faidon Liambotis: [C: 032] enable RSS for LVS servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/135932 (owner: 10BBlack) [13:25:49] Coren: Received: from jenkins-bot by fab2.eqiad.wmflabs with local (Exim 4.76) ? [13:26:24] (03CR) 10Ottomata: Use HCatalog as default auxpath for Hive (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/136014 (owner: 10QChris) [13:28:36] (03CR) 10Ottomata: Allow to set up hive's auxpath globally (032 comments) [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/135539 (owner: 10QChris) [14:09:05] (03PS1) 10Giuseppe Lavagetto: rcs: install servers with trusty [operations/puppet] - 10https://gerrit.wikimedia.org/r/136022 [14:29:04] (03PS1) 10Alexandros Kosiaris: mysql-predump optimizations [operations/puppet] - 10https://gerrit.wikimedia.org/r/136026 [14:32:26] (03PS2) 10Alexandros Kosiaris: mysql-predump optimizations [operations/puppet] - 10https://gerrit.wikimedia.org/r/136026 [14:34:25] <_joe_> akosiaris: I was about to submit a comment about the missing -x :P [14:35:06] <_joe_> also, I don't understand why -x and not -e [14:35:28] cause I am an idiot [14:35:36] obviously -e and not -x [14:36:01] <_joe_> I was looking at the bash manual to see if I got something terribly wrong :P [14:36:38] and I even wrote a comment about the behaviour. I really do not know why I put -x there [14:36:42] thanks for catching that [14:37:18] (03PS3) 10Alexandros Kosiaris: mysql-predump optimizations [operations/puppet] - 10https://gerrit.wikimedia.org/r/136026 [14:37:28] (03PS2) 10Dzahn: admin yaml for tridge (backups) [operations/puppet] - 10https://gerrit.wikimedia.org/r/135957 [14:39:01] (03CR) 10Dzahn: [C: 032] admin yaml for tridge (backups) [operations/puppet] - 10https://gerrit.wikimedia.org/r/135957 (owner: 10Dzahn) [14:50:24] James_F|Away: SWAT in 10 minutes [14:59:08] (03PS4) 10Alexandros Kosiaris: mysql-predump optimizations [operations/puppet] - 10https://gerrit.wikimedia.org/r/136026 [14:59:38] anomie: Here. [15:00:13] * anomie begins SWAT [15:00:13] Oh, wow, we now have a bot? [15:00:17] Is this new? [15:00:35] James_FL A week old IIRC [15:00:37] I know mwalker|away was talking about this… [15:00:47] JohnLewis: Ah, clearly I just haven't been paying attention. :-) [15:01:03] James_F: Clearly :) [15:01:12] * anomie still doesn't like how the bot uses /notice instead of actually pinging in the channel [15:01:27] Indeed. [15:01:43] Especially since a /notice doesn't actually *ping* me [15:02:06] Helpful. [15:02:12] But it's in prettier colour! [15:02:19] (At least on irssi.) [15:02:46] * bd808 suggests pull requests to https://github.com/mattofak/jouncebot [15:02:52] Ping is configurable. (At least on irssi.) [15:03:09] a /notice is made for that? [15:03:28] (03PS1) 10Alexandros Kosiaris: ganeti evaluation shared IP [operations/dns] - 10https://gerrit.wikimedia.org/r/136055 [15:03:46] (03CR) 10Alexandros Kosiaris: [C: 032] mysql-predump optimizations [operations/puppet] - 10https://gerrit.wikimedia.org/r/136026 (owner: 10Alexandros Kosiaris) [15:03:47] The change would be at https://github.com/mattofak/jouncebot/blob/master/jouncebot.py#L135 [15:04:12] s/notice/privmsg/ [15:05:42] sounds like you're fixing the bot because the clients have an issue [15:06:13] mutante: Made for what? The problem here is that the bot is using /notice when that's not what /notice is for, surely? [15:06:17] according to the irc spec, the only time bots have to use notice is if they're responding to a privmsg [15:06:27] James_F: for sending notices to people? [15:06:40] mutante: /notice is normally used (IME, I could be wrong) for notices about the channel – not a ping to an individual to do a task. [15:06:48] !log anomie synchronized php-1.24wmf6/extensions/VisualEditor/modules/ve-mw/ 'SWAT: VisualEditor URL decoding and image alignment fixes. [[gerrit:135922]] [[gerrit:135946]]' [15:06:49] mutante: No, that's /query ;-) [15:06:57] James_F: ^ Test please. Don't forget you have two different changes included there. [15:06:58] Logged the message, Master [15:06:59] /query is a private message [15:07:05] anomie: Yup. Thanks. [15:07:07] mutante: Indeed. [15:07:22] do you want the bot to not talk in public? then i got you wrong [15:07:41] (03CR) 10Alexandros Kosiaris: [C: 04-2] "duh. I had this in my mind but instead of checking out and submitting my patch in this changeId I created a new changeId:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135517 (owner: 10Springle) [15:08:25] mutante: If all you want is to inform individuals, you should use /query. If you want to get people to do something, and for others to be aware, /privmsg in channel. If you want everyone to be vaguely aware of something related to the channel, /notice [15:08:47] mutante: Or at least that's the guidance/policy we used to have. Other policies might make sense, though. [15:09:41] James_F: ok, fair, so which of the 3 is the goal of jouncebot? [15:10:12] mutante: I think the goal of the bot is #2 not #3. It's a specific reminder to specific people (hey, go deploy), plus awareness to third parties (e.g. you, me, etc.) [15:10:28] mutante: But instead we could do /notice and /query combined, maybe? [15:10:34] * James_F is easy. [15:10:58] anomie: All tested, everything is great. Thank you! [15:11:02] James_F: no, you convinced me [15:11:05] * anomie is done with SWAT [15:14:26] (03PS1) 10Reedy: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136056 [15:14:28] (03PS1) 10Reedy: testwiki to 1.24wmf7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136057 [15:14:30] (03PS1) 10Reedy: Wikipedias to 1.24wmf6 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136058 [15:14:32] (03PS1) 10Reedy: group0 to 1.24wmf7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136059 [15:14:38] time for reedy spam [15:15:07] (03CR) 10Reedy: [C: 032] Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136056 (owner: 10Reedy) [15:15:15] (03Merged) 10jenkins-bot: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136056 (owner: 10Reedy) [15:15:27] (03PS1) 10Rush: admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 [15:15:29] (03CR) 10Reedy: [C: 032] testwiki to 1.24wmf7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136057 (owner: 10Reedy) [15:16:26] mutante: Maybe, OTOH, the logging bots could use /notice. [15:16:29] * James_F ponders. [15:16:52] (03Merged) 10jenkins-bot: testwiki to 1.24wmf7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136057 (owner: 10Reedy) [15:17:06] (03PS1) 10Christopher Johnson (WMDE): Icinga: new command "check_dispatch" for Wikidata [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 [15:17:39] !log reedy Started scap: testwiki to 1.24wmf7 and build l10n cache [15:17:44] Logged the message, Master [15:17:45] (03CR) 10jenkins-bot: [V: 04-1] admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 (owner: 10Rush) [15:29:10] (03PS2) 10Christopher Johnson (WMDE): Icinga: new command "check_dispatch" for Wikidata [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 [15:33:41] (03PS2) 10Rush: admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 [15:34:41] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "I'm strongly opposed to install a perl module via puppi. I'd rather package/backport libperl-json-path." [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [15:35:12] (03CR) 10jenkins-bot: [V: 04-1] admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 (owner: 10Rush) [15:37:21] (03PS3) 10Rush: admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 [15:39:14] (03CR) 10jenkins-bot: [V: 04-1] admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 (owner: 10Rush) [15:39:17] (03CR) 10Alexandros Kosiaris: [C: 032] ganeti evaluation shared IP [operations/dns] - 10https://gerrit.wikimedia.org/r/136055 (owner: 10Alexandros Kosiaris) [15:39:26] (03PS1) 10Alexandros Kosiaris: thallium,mercury ganeti eval machines setup [operations/puppet] - 10https://gerrit.wikimedia.org/r/136104 [15:40:11] (03PS4) 10Rush: admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 [15:42:03] !log reedy Finished scap: testwiki to 1.24wmf7 and build l10n cache (duration: 24m 24s) [15:42:03] (03CR) 10Alexandros Kosiaris: [C: 04-1] "I would say this is missing a partitioning scheme. Should be as easy as adding a line in" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136022 (owner: 10Giuseppe Lavagetto) [15:42:07] Logged the message, Master [15:42:55] (03PS5) 10Rush: admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 [15:42:58] <_joe_> akosiaris: oh I know that [15:42:58] <_joe_> :) [15:43:39] ah ok then. Sorry :-/ [15:43:58] bits broken ? [15:44:04] no js/css for me [15:45:26] PROBLEM - Apache HTTP on mw1152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:26] PROBLEM - Apache HTTP on mw1149 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:26] PROBLEM - Apache HTTP on mw1202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:28] akosiaris _joe_ ? [15:45:39] i predicted this [15:45:56] PROBLEM - Apache HTTP on mw1189 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:03] on it [15:46:06] PROBLEM - Apache HTTP on mw1190 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:09] thanks [15:46:26] PROBLEM - Apache HTTP on mw1191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:36] PROBLEM - Apache HTTP on mw1196 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:06] <_joe_> wtf [15:47:06] RECOVERY - Apache HTTP on mw1190 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 9.331 second response time [15:47:26] RECOVERY - Apache HTTP on mw1191 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 9.242 second response time [15:47:26] RECOVERY - Apache HTTP on mw1196 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 6.197 second response time [15:47:26] PROBLEM - Apache HTTP on mw1203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:36] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:43] (03CR) 10Dzahn: [C: 031] "+1, but as the comment said this would be temp. and should become real analytics users later" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 (owner: 10Rush) [15:47:53] Bits odwn? [15:47:56] Down, even. [15:48:01] yes Marybelle [15:48:06] PROBLEM - Apache HTTP on mw1150 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:26] PROBLEM - Apache HTTP on mw1207 is CRITICAL: Connection timed out [15:48:26] RECOVERY - Apache HTTP on mw1202 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 8.512 second response time [15:48:32] <_joe_> segfaults [15:49:17] PROBLEM - Apache HTTP on mw1208 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:33] oh fuck [15:49:46] RECOVERY - Apache HTTP on mw1189 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 3.987 second response time [15:49:58] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: (no message) [15:50:03] Logged the message, Master [15:50:05] I deployed too much [15:50:16] RECOVERY - Apache HTTP on mw1208 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 9.266 second response time [15:50:17] RECOVERY - Apache HTTP on mw1203 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.067 second response time [15:50:26] RECOVERY - Apache HTTP on mw1207 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 4.463 second response time [15:50:26] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.058 second response time [15:50:36] (03CR) 10Hashar: "IIRC Chase said yesterday that this is manageable via the recently added admin module." [operations/puppet] - 10https://gerrit.wikimedia.org/r/76678 (owner: 10Tim Starling) [15:50:49] (03CR) 10JanZerebecki: "Why do you think we need to find out what percent of our readers are still vulnerable to BEAST client-side?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [15:51:17] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data exceeded the critical threshold [500.0] [15:51:26] RECOVERY - Apache HTTP on mw1149 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 8.449 second response time [15:51:48] bits.wikimedia.org 500s, causing unstyled documents everywhere [15:51:57] liangent: known, thanks [15:52:26] (03CR) 10Ottomata: [C: 032] admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 (owner: 10Rush) [15:53:22] <_joe_> Reedy: what has happened exactly? [15:53:33] what's going on? [15:53:36] I deployed the new version to the wikipedias accidentally [15:53:40] usual post-deply bits downtime? [15:53:48] It's not deploy time [15:54:10] apc? [15:54:50] (03CR) 10Rush: [C: 032 V: 032] admin yaml analytics1009.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/136060 (owner: 10Rush) [15:55:06] RECOVERY - Apache HTTP on mw1150 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 7.262 second response time [15:56:26] RECOVERY - Apache HTTP on mw1152 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 3.794 second response time [15:58:35] (03PS2) 10Dzahn: admin yaml for palladium [operations/puppet] - 10https://gerrit.wikimedia.org/r/135958 [16:00:19] (03CR) 10Rush: [C: 031] admin yaml for palladium [operations/puppet] - 10https://gerrit.wikimedia.org/r/135958 (owner: 10Dzahn) [16:01:03] (03CR) 10Dzahn: [C: 032] admin yaml for palladium [operations/puppet] - 10https://gerrit.wikimedia.org/r/135958 (owner: 10Dzahn) [16:01:37] (03CR) 10Hoo man: "re check" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135725 (owner: 10Hoo man) [16:03:17] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [16:03:27] (03CR) 10Rush: "so yaml parser seems to ignore dupes and just assume the last read is authoritative, I wasn't too worried as a sort will fix this and reor" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135725 (owner: 10Hoo man) [16:03:35] (03CR) 10Hoo man: "recheck" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135725 (owner: 10Hoo man) [16:05:10] (03PS2) 10Dzahn: admin yaml for iodine [operations/puppet] - 10https://gerrit.wikimedia.org/r/135959 [16:06:38] (03CR) 10Hoo man: "Note: The jenkins failures were unrelated." [operations/puppet] - 10https://gerrit.wikimedia.org/r/135725 (owner: 10Hoo man) [16:07:45] (03CR) 10Dzahn: [C: 032] admin yaml for iodine [operations/puppet] - 10https://gerrit.wikimedia.org/r/135959 (owner: 10Dzahn) [16:11:11] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Just for the record, same opinion here. If all this is to be imported just for JSON::Path I will gladly package the dependency instead" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [16:15:26] so, what's going on? [16:15:57] (03PS1) 10Dzahn: admin yaml for potassium [operations/puppet] - 10https://gerrit.wikimedia.org/r/136115 [16:18:00] (03PS3) 10Ottomata: [WIP] Add CDH5 support, drop CDH4 support [operations/puppet/cdh4] (cdh5) - 10https://gerrit.wikimedia.org/r/135494 [16:18:37] we're still operating with three bits app servers [16:18:52] mw1151 hasn't been repooled [16:19:34] that's quite fragile obviously [16:19:50]