[00:18:29] (03Restored) 10Ori.livneh: HHVM: warm up the JIT by making web requests in Upstart post-start [puppet] - 10https://gerrit.wikimedia.org/r/150992 (owner: 10Ori.livneh) [00:19:03] ^ TimStarling [00:19:13] I abandoned it because I thought I was getting a little ahead of myself [00:54:06] (03CR) 10Ori.livneh: "Another approach would be to have a PHP file that exercises hot code paths, and to set it as the StartupDocument runtime option. See PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 1 failures [01:28:51] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [02:06:53] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3612 MB (3% inode=99%): [02:16:20] !log LocalisationUpdate completed (1.24wmf20) at 2014-09-22 02:16:20+00:00 [02:16:35] Logged the message, Master [02:29:09] !log LocalisationUpdate completed (1.24wmf21) at 2014-09-22 02:29:09+00:00 [02:29:16] Logged the message, Master [02:29:32] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Epic puppet fail [02:41:29] !log LocalisationUpdate completed (1.24wmf22) at 2014-09-22 02:41:29+00:00 [02:41:36] Logged the message, Master [02:48:42] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:00:34] RECOVERY - Disk space on virt0 is OK: DISK OK [03:36:40] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 22 03:36:40 UTC 2014 (duration 36m 39s) [03:36:47] Logged the message, Master [05:38:32] (03PS1) 10Yurik: Enable Graph ext on mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161908 [06:28:35] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Epic puppet fail [06:29:37] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:14] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:24] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:25] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:44] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:44] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:54] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:06] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:06] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:14] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:16] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:56] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 3 failures [06:39:08] (03PS1) 10Tim Starling: Fix profiling error CommonSettings.php-skin-include1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161911 [06:41:18] (03CR) 10Ori.livneh: [C: 031] Fix profiling error CommonSettings.php-skin-include1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161911 (owner: 10Tim Starling) [06:45:24] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:45:54] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:45:55] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:46:14] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:46:28] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:46:54] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:46:54] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:47:02] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:47:24] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:48:54] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:14] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:07:15] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:11:25] (03PS2) 10Ori.livneh: HHVM: warm up the JIT by making web requests in Upstart post-start [puppet] - 10https://gerrit.wikimedia.org/r/150992 [07:19:34] (03CR) 10Qgil: "I guess we are abandoning this patch, and the other ones related with Bugzilla and still open?" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/107410 (owner: 10Aklapper) [07:22:24] <_joe_> ori: never noticed that patch; it's something I thought we should do in fact [07:22:47] <_joe_> not sure if we'd need to clean up the bytecode repo as well [07:25:16] that would defeat the purpose of a persistent repo [07:25:33] anything that requires cleaning up the repo to fix is a serious bug and should be fixed upstream promptly, imo [07:25:49] <_joe_> I tend to agree [07:26:00] don't merge the patch above tho, i want to tweak it a little [07:26:01] <_joe_> btw we're using a quasi-persistent storage at the moment [07:26:16] <_joe_> meaning that the repo code is in-memory using tmpfs [07:30:00] <_joe_> ori: I was intending to tweak that as well :) [07:30:33] <_joe_> I have a couple of things more to figure out as well [07:30:44] <_joe_> like, does hhvm resolve symlinks? [07:34:16] <_joe_> (we have uncountable links to the same files in different docroots) [07:34:32] it does [07:34:43] <_joe_> I was almost sure [07:34:50] let me know what you'd like me to fix and i'll amend the patch [07:35:18] <_joe_> I think the way you call curl could be improved [07:36:08] <_joe_> also, I think there is a way to pass curl more than one url at a time [07:36:56] (03CR) 10Filippo Giunchedi: [C: 031] wmflib: add to_milliseconds() / to_seconds() [puppet] - 10https://gerrit.wikimedia.org/r/159692 (owner: 10Ori.livneh) [07:37:00] greetings [07:37:00] <_joe_> but I'd have to check the manual [07:37:10] <_joe_> ciao :) [07:37:36] ciao _joe_ [07:38:48] ori: I'll try to merge https://gerrit.wikimedia.org/r/#/c/161332/ and possibly https://gerrit.wikimedia.org/r/#/c/133274/ today, worst case tomorrow [07:39:05] <_joe_> \o/ [07:39:19] <_joe_> (grafana makes me happy) [07:39:52] hehe [07:40:37] <_joe_> we need to get rid of ganglia; we won't be able to do that if we don't have a decent frontend for graphite [07:40:39] godog: morning! awesome, thanks! [07:40:45] <_joe_> and a better graphite install as well [07:44:25] ori: np! [07:44:52] <_joe_> it is unbearably hot here [07:45:34] heh, after looking again at the whisper code if we want to keep it SSDs might be worth it, the end result is basically ~1 seek for each metric received which isn't very nice [07:48:38] "if we want to keep it" -- what else would we do? switch to megacarbon, opentsdb, or influxdb? [07:50:52] <_joe_> opentsdb is sexy [07:51:26] <_joe_> influxdb last I checked was utterly immature and full of bugs [07:51:32] <_joe_> but it was like march [07:51:38] <_joe_> before joining the foundation [07:52:01] <_joe_> wow it's almost 6 months I'm here [07:58:44] (03PS1) 10Giuseppe Lavagetto: Add public alias for config-master. [dns] - 10https://gerrit.wikimedia.org/r/161913 [07:58:51] ori: not much available right now indeed, I wanted to try backing a graphite-api with opentsdb but haven't found much time to experiment [07:59:28] <_joe_> godog: If I understand correctly, we'd need to write the "glue" by ourselves, right? [08:01:26] _joe_: yep, it could work 'out of the box' but what's needed I think is having the data pre-aggregated too otherwise there's no chance it'll work [08:04:02] _joe_: happy half year birthday! [08:06:48] (03PS1) 10Giuseppe Lavagetto: config-master: serve through misc-web-lb [puppet] - 10https://gerrit.wikimedia.org/r/161915 [08:09:28] (03PS2) 10Giuseppe Lavagetto: Remove the last references to pybal on fenari [puppet] - 10https://gerrit.wikimedia.org/r/160467 [08:14:17] (03PS3) 10Ori.livneh: HHVM: warm up the JIT by making web requests in Upstart post-start [puppet] - 10https://gerrit.wikimedia.org/r/150992 [08:56:59] _joe_: interested in hhvm feedback ? [09:34:17] <_joe_> matanya: of course I am [09:35:00] _joe_: users report on slower networks css is not loading on time, but only on hhvm, disabling hhvm css loads ok [09:36:56] <_joe_> matanya: reason can be they're not cached (still) [09:37:02] in the user words: works as advertised, hhvm is faster, but maybe due to not loading css :) [09:37:03] <_joe_> but I'd need URLs [09:37:11] give a amsec [09:37:27] * me a sec [09:37:31] <_joe_> also, file a bug :) [09:37:39] !log Jenkins: deleting old mediawiki extensions jobs (rm -fR /var/lib/jenkins/jobs/*testextensions-master). They are no more triggered and superseded by the *-testextension jobs. [09:37:44] Logged the message, Master [09:38:30] will do, thanks _joe_ [09:43:00] uselessd ftw [09:55:43] !log restarting jenkins [09:55:48] Logged the message, Master [09:58:26] _joe_: fwiw https://he.wikipedia.org/wiki/%D7%99%D7%95%D7%95%D7%9F_%D7%94%D7%90%D7%A8%D7%9B%D7%90%D7%99%D7%AA is an example [10:00:29] <_joe_> matanya: I see that perfectly [10:00:39] the bug ? [10:00:46] or the page? [10:00:48] <_joe_> no this url [10:00:51] <_joe_> this page [10:01:05] me too, but i'm on a very fast network [10:01:17] he said it happens only on slow networks [10:01:44] <_joe_> Maybe he has some cookies so that the urls are called with debug=true? [10:02:28] i doubt it, but i can ask [10:02:43] <_joe_> btw, full page rendering in an incognito window so without hhvm took me a fat 2 s more [10:03:09] <_joe_> (yes I cleared the cache before doing the measurement) [10:03:47] (03CR) 10Aklapper: "I'd do that once BZ is entirely gone" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/107410 (owner: 10Aklapper) [10:04:11] yes, hhvm is way faster for me too [10:04:13] !log Jenkins back and fully operational [10:04:19] Logged the message, Master [10:08:48] _joe_: breadcrumbs :D https://lists.wikimedia.org/pipermail/wikimedia-l/2014-September/074713.html [10:09:31] would be nice to have a list of pages which currently give 504 on save and see how well hhvm performs there [10:09:56] <_joe_> 504? [10:10:06] <_joe_> I'd have thought 503 [10:10:46] iirc that page used to give me 504 [10:13:16] I wish there was a way to enable beta features globally, going to 700~ sites to enable hhvm is annoying [10:13:58] <_joe_> matanya: it is [10:15:29] matanya: poke platform to have user preference added to CentralLogin or something similar [10:16:04] hashar: i'm still working on SUL, can't poke them before i finish my part :) [10:16:31] :D [10:17:09] but yeah, the interface looks different on every site i visit due to old css/js i once istalled/gadgets/beta features [10:17:37] very confusing, but don't have the time to clean up, wish is was global [10:17:54] * it [10:19:38] <_joe_> matanya: I kinda agree; I have to say that multi-wiki always-logged-in editors are probably a small subset even of the editors community [10:19:55] <_joe_> so probably not that important :) [10:20:02] i know i'm a unicorn :D [10:20:10] <_joe_> and I also see some implementation issues [10:20:36] <_joe_> given not every mediawiki install we have is at the same version, has the same config, etc etc [10:21:05] <_joe_> matanya: most WMF staff is constantly looking at a lot of wikipedias, so you're not alone :) [10:21:07] yeah, i know that, just general start of the week ranting [10:21:22] <_joe_> eheh I'm as annoyed as you [10:21:55] btw, _joe_ that is a good point, most WMF staff is constantly looking at a lot of wikipedias <-- what about other projects? [10:22:11] (03PS1) 10Nemo bis: [Planet Wikimedia] Add Maria Sefidari to English [puppet] - 10https://gerrit.wikimedia.org/r/161921 [10:22:38] yay, git is broken on this machine :/ [10:22:41] <_joe_> I'm using https://meta.wikimedia.org/wiki/User:GLavagetto_(WMF)/global.js to trick the system [10:22:52] i'll clone a new clone [10:23:06] <_joe_> matanya: other projects as well [10:23:47] not doable, but thanks :) https://he.wikipedia.org/wiki/%D7%9E%D7%A9%D7%AA%D7%9E%D7%A9:Matanya/common.js [10:24:19] RTL will break big time on sites like en,commons or meta [10:24:35] <_joe_> yep [10:25:09] code [10:25:14] i'll go fix some scoping instead of ranting, more productive [10:25:17] heading lunch [10:25:27] <_joe_> :)) [10:29:23] * YuviPanda|away waves at _joe_ [10:32:47] * _joe_ hides [10:33:19] <_joe_> YuviPanda: just joking... I'll take a look [10:33:26] _joe_: :) [10:34:06] (03CR) 10Giuseppe Lavagetto: [C: 032] Add public alias for config-master. [dns] - 10https://gerrit.wikimedia.org/r/161913 (owner: 10Giuseppe Lavagetto) [10:35:48] (03CR) 10Giuseppe Lavagetto: [C: 032] config-master: serve through misc-web-lb [puppet] - 10https://gerrit.wikimedia.org/r/161915 (owner: 10Giuseppe Lavagetto) [10:35:59] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [10:36:31] <_joe_> mmmh a 404 spike as well [10:36:53] <_joe_> someone doing some very lame crawling maybe [10:42:18] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 1 failures [10:42:51] <_joe_> deh [10:42:51] (03PS1) 10Giuseppe Lavagetto: misc-varnish: add palladium [puppet] - 10https://gerrit.wikimedia.org/r/161922 [10:42:53] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 1 failures [10:43:09] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 1 failures [10:43:26] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] misc-varnish: add palladium [puppet] - 10https://gerrit.wikimedia.org/r/161922 (owner: 10Giuseppe Lavagetto) [10:45:18] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [10:48:18] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [10:51:13] _joe_: me again, templates/varnish/upload-frontend.inc.vcl.erb doesn't seem to be called from anywhere, but i'm sure it is. can you please enlight me ? [10:52:19] <_joe_> mmh I need to look at it [10:52:40] <_joe_> but I'd say from the varnish class via the role/cache.pp file [10:52:45] <_joe_> but lemme check [10:54:47] <_joe_> varnish::instance { 'upload-frontend': in role/cache.pp [10:55:39] but where is the template being called from ? [10:56:05] vcl => ? [10:56:45] (03PS9) 10Yuvipanda: nagios_common: Refactor custom command definitions [puppet] - 10https://gerrit.wikimedia.org/r/161478 [10:57:46] <_joe_> yes [10:57:56] <_joe_> YuviPanda: give me anouther 10 minutes [10:58:10] _joe_: sure! I just added the shinken bits, testing now [11:00:29] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [11:00:55] (03PS3) 10Giuseppe Lavagetto: Remove the last references to pybal on fenari [puppet] - 10https://gerrit.wikimedia.org/r/160467 [11:01:09] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [11:01:33] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Remove the last references to pybal on fenari [puppet] - 10https://gerrit.wikimedia.org/r/160467 (owner: 10Giuseppe Lavagetto) [11:02:35] (03PS10) 10Yuvipanda: nagios_common: Refactor custom command definitions [puppet] - 10https://gerrit.wikimedia.org/r/161478 [11:11:23] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [11:11:38] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [11:12:18] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [11:12:19] <_joe_> ^^ that was the dumbest part of me [11:12:48] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [11:13:03] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "LGTM in general; just a couple of comments;" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/161478 (owner: 10Yuvipanda) [11:13:56] _joe_: bah, I've git set to strip all trailing spaces before commit, so I presumed that would pick it up. apparently it doesn't pick it up if you do a git add on a new file [11:14:57] <_joe_> YuviPanda: eheh that was a lame correction honestly [11:15:07] <_joe_> the other two remarks are more interesting [11:15:19] I find them useufl anyway :) [11:15:24] <_joe_> YuviPanda: I'm the one usually spreading whitespaces around, just because I'm lazy [11:16:45] _joe_: I'm wary of collecting things around, since collecting is a bit bogus on labs. It'll probably work here, but I prefer this method (more explicit and undistributed) [11:16:59] <_joe_> ok that's ok [11:17:06] <_joe_> I was doubtful anyway [11:17:08] <_joe_> :) [11:17:24] <_joe_> oh, move check_ganglia there as well then :) [11:17:42] <_joe_> (and maybe while we're at it, let's kill check_ganglia ASAP) [11:17:47] (03PS11) 10Yuvipanda: nagios_common: Refactor custom command definitions [puppet] - 10https://gerrit.wikimedia.org/r/161478 [11:18:02] _joe_: I'm going to move them all (we have a number of custom checks) in followup patches [11:18:44] <_joe_> ok! thanks! [11:19:02] _joe_: wondering if I should do one patch per check, or just move 'em all in one go [11:19:21] hmm, doing it in a couple of patches (not one per patch, but not all in one go) might make sense [11:20:01] <_joe_> one per patch is advisable [11:20:06] hmm, ok [11:20:11] _joe_: want to merge the current ones? [11:20:12] <_joe_> I've got no issues merging [11:20:15] <_joe_> yep [11:20:18] whee [11:20:21] <_joe_> give me 5 mins [11:20:23] sure [11:20:27] <_joe_> I'm patching hhvm atm [11:21:57] cool [11:22:30] _joe_: actually, hold off a moment, it seems to have messed up shinken [11:22:48] <_joe_> ok [11:23:04] <_joe_> it will take me more than 5 minutes it seems [11:23:15] heh [11:27:41] (03PS1) 10Matanya: udp2log: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/161926 [11:27:44] (03PS12) 10Yuvipanda: nagios_common: Refactor custom command definitions [puppet] - 10https://gerrit.wikimedia.org/r/161478 [11:27:46] (03PS5) 10Yuvipanda: nagios_common: Extract user_definitions from icinga [puppet] - 10https://gerrit.wikimedia.org/r/161446 [11:30:46] _joe_: ok, fixed [11:40:14] (03PS1) 10Gerrit Patch Uploader: Set wgUploadNavigationUrl for eowiki to localized version of UploadWizard [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161928 (https://bugzilla.wikimedia.org/69055) [11:40:19] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161928 (https://bugzilla.wikimedia.org/69055) (owner: 10Gerrit Patch Uploader) [11:43:44] <_joe_> YuviPanda: at this point, let's wait after lunch [11:43:51] _joe_: sure [11:55:50] (03PS2) 10Steinsplitter: Set wgUploadNavigationUrl for eowiki to localized version of UploadWizard [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161928 (https://bugzilla.wikimedia.org/69055) (owner: 10Gerrit Patch Uploader) [12:10:02] !log shutdown of db1050 to install trusty [12:10:07] Logged the message, Master [12:12:49] PROBLEM - Host db1050 is DOWN: PING CRITICAL - Packet loss = 100% [12:18:08] RECOVERY - Host db1050 is UP: PING OK - Packet loss = 0%, RTA = 1.50 ms [12:39:13] (03CR) 10Alexandros Kosiaris: [C: 032] Fix typo in mathoid LVS group description [puppet] - 10https://gerrit.wikimedia.org/r/161609 (owner: 10Catrope) [12:42:17] (03PS13) 10Yuvipanda: nagios_common: Refactor custom command definitions [puppet] - 10https://gerrit.wikimedia.org/r/161478 [12:42:19] (03PS1) 10Yuvipanda: nagios_common: Move check_dsh_groups into module [puppet] - 10https://gerrit.wikimedia.org/r/161934 [13:11:26] (03PS1) 10Giuseppe Lavagetto: backport of https://github.com/facebook/hhvm/pull/3811/ [debs/hhvm] - 10https://gerrit.wikimedia.org/r/161936 [13:11:44] <_joe_> Is there some proficient c++ coder around? [13:11:52] <_joe_> if so, please take a look at ^^ [13:12:20] <_joe_> I think I backported tim's PR correctly, but given I'm basically an illiterate, I'd use a review [13:12:46] <_joe_> s/proficient/not completely ignorant as myself/ [13:16:19] * YuviPanda looked, but then remembered he hadn't written any real C++ ever, so refrains [13:19:06] ( pasting from the labs channel ) where can I find the apparmor logs in my labs instance ? [13:19:37] tonythomas: https://wiki.ubuntu.com/DebuggingApparmor [13:19:59] _joe_: I'll take a look [13:21:00] <_joe_> Coren: so, I'm working on the 3.3.0 branch and tim worked on HEAD, which was heavily changed [13:22:41] * Coren loves the comment in the original code -- Nobody will ever do *that* is _always_ false. [13:23:13] YuviPanda: ok. found the log in /var/log/kern.log. Thanks [13:23:18] cool [13:23:39] Coren: I've written such comments on Android which are true! [13:24:08] mostly to catch a checked exception which says 'I do not know any encoding called utf-8', which can never happen (Android guarantees that) [13:24:26] also for a 'package not found' exception, when the name I'm passing it is the name of the currently running package [13:27:12] (03CR) 10coren: [C: 031] "This should be functionally equivalent to Tim's patch on github, provided the JIT classes have no significant semantic differences between" [debs/hhvm] - 10https://gerrit.wikimedia.org/r/161936 (owner: 10Giuseppe Lavagetto) [13:27:35] (03PS2) 10Yuvipanda: nagios_common: Move check_dsh_groups into module [puppet] - 10https://gerrit.wikimedia.org/r/161934 [13:27:57] _joe_: Insofar as I can tell, your patch does the right thing to match Tim's; but we presume their class that actually constructs the code is semantically the same. [13:28:29] And I have no way to tell how stable that api has been between the versions. [13:29:29] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Epic puppet fail [13:29:39] _joe_: I've to go for a bit in about 30min, think you will have time to merge the nagios_common stuff before that? [13:30:00] <_joe_> YuviPanda: I'll take a look [13:30:04] cool [13:30:14] <_joe_> Coren: it's 5 days of difference, really [13:30:14] I also presume that unlikelyIfThenElse is equivalent to unlikelyCond (which seems reasonable) [13:30:19] <_joe_> yep [13:30:34] <_joe_> they changed the name, but AFAICT not the signature of the function [13:31:06] Yeah; that's what I figured too. The name just sucked. :-) [13:33:06] <_joe_> actually, I was wrong [13:33:30] <_joe_> https://github.com/facebook/hhvm/blob/master/hphp/runtime/vm/jit/code-gen-x64.cpp#L195 vs https://github.com/facebook/hhvm/blob/HHVM-3.3/hphp/runtime/vm/jit/code-gen-x64.cpp#L159 [13:34:22] <_joe_> so, I need to add it [13:34:25] <_joe_> meh [13:35:16] You mean because of the return value? [13:35:34] <_joe_> yep [13:36:11] Tim's not /using/ the rv in his patch. [13:37:00] Oh! But there's an extra Vreg. [13:37:13] <_joe_> yep :) [13:38:16] <_joe_> re-working it [13:38:20] Bleh; so the functions are only /mostly/ equivalent. [13:39:04] * Coren tries to figure out why the hell the function would return one of its arguments. [13:39:06] <_joe_> I got confused because they converted themselves some unlikelyIfThenElse to unlikelyCond [13:39:24] <_joe_> because it modifies it internally AFAICS [13:39:47] Sure, but it's passed by ref. [13:44:31] <_joe_> anyway, correcting that [13:47:48] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [13:51:36] (03PS1) 10Yuvipanda: nagios_common: Move wikidata check to a check_command define [puppet] - 10https://gerrit.wikimedia.org/r/161939 [13:52:28] (03PS2) 10Yuvipanda: nagios_common: Move wikidata check to a check_command define [puppet] - 10https://gerrit.wikimedia.org/r/161939 [13:53:37] <_joe_> YuviPanda: I'll get to you eventually [13:53:51] _joe_: :) I'm just moving check commands one by one. [13:57:05] (03PS2) 10Giuseppe Lavagetto: backport of https://github.com/facebook/hhvm/pull/3811/ [debs/hhvm] - 10https://gerrit.wikimedia.org/r/161936 [13:57:16] <_joe_> Coren: ^^ [13:57:24] (03PS3) 10Yuvipanda: nagios_common: Move wikidata check into module [puppet] - 10https://gerrit.wikimedia.org/r/161939 [13:57:26] (03PS1) 10Yuvipanda: nagios_common: Move check_cert into module [puppet] - 10https://gerrit.wikimedia.org/r/161942 [13:58:34] Ah, you went with bringing in the templated function in rather than rework to use unlikelyIfThenElse? [13:58:54] <_joe_> Coren: yep :) [13:59:01] <_joe_> can't harm us I guess [14:02:48] (03CR) 10coren: [C: 031] "With the same caveat that this relies on the JIT semantics not having changed too much." [debs/hhvm] - 10https://gerrit.wikimedia.org/r/161936 (owner: 10Giuseppe Lavagetto) [14:03:48] <_joe_> Coren: I guess if that's wrong I' [14:03:54] <_joe_> ll notice runnign tests [14:04:03] Presumably. :-) [14:04:36] (03PS3) 10Giuseppe Lavagetto: backport of https://github.com/facebook/hhvm/pull/3811/ [debs/hhvm] - 10https://gerrit.wikimedia.org/r/161936 [14:04:54] <_joe_> I also added the test as a quick one, so that we'll run it when building the package [14:05:11] Better than the one in the original patch too. :-) [14:05:13] (03PS1) 10Yuvipanda: nagios_common: Move check_solr into module [puppet] - 10https://gerrit.wikimedia.org/r/161943 [14:08:44] * YuviPanda goes afk for a bit, will be back soon [14:10:41] mark__: Can you help me out with a networking issue? I need access between labcontrol2001 and virt1000 on ports 4444 and 8989. [14:10:56] Those ports are defined in iptables, but I don't see them actually used anyplace... [14:12:50] !log Jenkins deleted job mediawiki-core-lint , replaced by mediawiki-core-phplint [14:12:56] Logged the message, Master [14:13:02] (03CR) 10Giuseppe Lavagetto: [C: 032] nagios_common: Refactor custom command definitions [puppet] - 10https://gerrit.wikimedia.org/r/161478 (owner: 10Yuvipanda) [14:13:45] _joe_: yay. it is dependent on another patch tho [14:13:57] https://gerrit.wikimedia.org/r/#/c/161446/5 [14:14:16] <_joe_> ugh YuviPanda [14:14:22] <_joe_> I hate dependent patches :P [14:14:37] _joe_: :D I quite like them! all the patches are in a series of dependent patches :) [14:14:50] I stopped using git review because it sucks with dependent patches [14:15:22] _joe_: we should just merge the patch it depends on, run on neon and see if something breaks [14:15:28] (03PS6) 10Giuseppe Lavagetto: nagios_common: Extract user_definitions from icinga [puppet] - 10https://gerrit.wikimedia.org/r/161446 (owner: 10Yuvipanda) [14:16:57] (03CR) 10Giuseppe Lavagetto: [C: 032] nagios_common: Extract user_definitions from icinga [puppet] - 10https://gerrit.wikimedia.org/r/161446 (owner: 10Yuvipanda) [14:17:22] (03PS14) 10Yuvipanda: nagios_common: Refactor custom command definitions [puppet] - 10https://gerrit.wikimedia.org/r/161478 [14:17:27] I might have killed jenkins [14:17:37] _joe_: ^ needs another +2 (because of the rebase) [14:17:41] <_joe_> hashar: about time! [14:17:49] I'll rebase all the patches that depend on it locally once that gets merged [14:17:53] <_joe_> YuviPanda: yeah lemme verify the preceding one first [14:17:59] _joe_: ah cool [14:18:04] should be a no-op [14:18:04] _joe_: I have more or less a way to replace jenkins entirely :-D [14:18:13] _joe_: zuul -> gearman -> ansible :-D [14:18:28] ansible? [14:18:33] <_joe_> not sure that's a good idea [14:18:46] mmm, so we'll have salt, puppet and ansible? :D [14:19:10] (03PS6) 10Alexandros Kosiaris: module/role for url-downloader [puppet] - 10https://gerrit.wikimedia.org/r/159738 [14:19:10] <_joe_> YuviPanda: not sure we'll have salt forever [14:19:14] the Jenkins jobs are defined by a yaml schema in JJB [14:19:17] yeah, I saw the email [14:19:27] and the DSL is very very close to Ansible tasks definition [14:19:29] I don't know if we use it for anything other than as a dsh replacement. [14:19:43] so it is not too much work to have JJB describe ansible tasks instead of Jenkins jobs [14:19:45] <_joe_> YuviPanda: I'm pretty agreeing with ori [14:20:00] <_joe_> and - facter is lame but still in some ways better [14:20:26] _joe_: yeah. salt, at my first glance, seemed to be in a bit of an identity crisis as to what it is [14:20:38] (03PS7) 10Alexandros Kosiaris: module/role for url-downloader [puppet] - 10https://gerrit.wikimedia.org/r/159738 [14:22:17] (03CR) 10Alexandros Kosiaris: module/role for url-downloader (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/159738 (owner: 10Alexandros Kosiaris) [14:25:11] <_joe_> btw yuvi I like the direction your changes are going to [14:25:15] <_joe_> more modularization [14:25:16] _joe_: :D [14:25:32] <_joe_> else, I'd have seen -2s :) [14:25:55] _joe_: yeah! I think having manifests/misc/icinga.pp not exist would be a nice end goal of the first set of refactors [14:26:00] <_joe_> ok so, please rebase the other changes [14:26:09] YuviPanda: ah yuvi! were can I fill bug for the new monitoring system you are setting up ? :D [14:26:12] _joe_: yeah! [14:26:20] YuviPanda: I have a feature request [14:26:23] hashar: ah, good question. nowhere yet, but put it in the general labs component [14:26:32] <_joe_> YuviPanda: not having a 2k lines long commangs.cfg file [14:26:39] YuviPanda: will [14:26:43] _joe_: yeah, that as well :D [14:26:56] <_joe_> hashar: opsens use that wonderful bug tracking system called "/dev/null" [14:26:59] _joe_: yeah, doing [14:27:09] <_joe_> YuviPanda: we have a lot to teach you I see [14:27:22] (03PS3) 10Yuvipanda: nagios_common: Move check_dsh_groups into module [puppet] - 10https://gerrit.wikimedia.org/r/161934 [14:27:24] (03PS2) 10Yuvipanda: nagios_common: Move check_solr into module [puppet] - 10https://gerrit.wikimedia.org/r/161943 [14:27:26] (03PS2) 10Yuvipanda: nagios_common: Move check_cert into module [puppet] - 10https://gerrit.wikimedia.org/r/161942 [14:27:28] (03PS4) 10Yuvipanda: nagios_common: Move wikidata check into module [puppet] - 10https://gerrit.wikimedia.org/r/161939 [14:27:28] _joe_: ho I feel them in RT for prod and Wikimedia > labs infrastructure for labs [14:27:32] _joe_: rebased :) [14:27:35] _joe_: works well nowadays! [14:27:36] _joe_: haha :D [14:27:55] (03PS1) 10Alexandros Kosiaris: Remove tridge [puppet] - 10https://gerrit.wikimedia.org/r/161948 [14:28:53] _joe_: did you force a run on neon? [14:29:23] (03CR) 10Giuseppe Lavagetto: [C: 032] nagios_common: Move check_dsh_groups into module [puppet] - 10https://gerrit.wikimedia.org/r/161934 (owner: 10Yuvipanda) [14:29:38] <_joe_> YuviPanda: with each change ;) [14:29:45] _joe_: w00t [14:29:47] <_joe_> well, the next few ones are easy [14:29:49] things seem unbroken so far [14:29:52] <_joe_> so, just merging [14:29:53] yeah [14:29:54] <_joe_> they are [14:29:56] coool :) [14:30:36] (03CR) 10Giuseppe Lavagetto: [C: 032] nagios_common: Move wikidata check into module [puppet] - 10https://gerrit.wikimedia.org/r/161939 (owner: 10Yuvipanda) [14:31:05] (03CR) 10Giuseppe Lavagetto: [C: 032] nagios_common: Move check_cert into module [puppet] - 10https://gerrit.wikimedia.org/r/161942 (owner: 10Yuvipanda) [14:31:21] (03CR) 10Giuseppe Lavagetto: [C: 032] nagios_common: Move check_solr into module [puppet] - 10https://gerrit.wikimedia.org/r/161943 (owner: 10Yuvipanda) [14:35:31] <_joe_> YuviPanda: all seems good, but please give a look at the checks [14:36:34] _joe_: yeah, looking [14:39:25] _joe_: I looked at check_graphite, seems ok [14:39:47] (03PS4) 10Giuseppe Lavagetto: backport of https://github.com/facebook/hhvm/pull/3811/ [debs/hhvm] - 10https://gerrit.wikimedia.org/r/161936 [14:40:00] * YuviPanda checks wikidata [14:40:17] _joe_: others (at least the ones I've permission to submit) seem ok [14:42:05] _joe_: thanks for merging/CRing! I'll move the other checks later today / tomorrow [14:45:52] (03CR) 10Ori.livneh: [C: 031] "Does the test pass? +2 if so." [debs/hhvm] - 10https://gerrit.wikimedia.org/r/161936 (owner: 10Giuseppe Lavagetto) [14:50:48] FlorianSW, Glaisher: Ping for SWAT in 10 minutes [14:50:56] <_joe_> ori: the test doesn't work when moved to quick, so I mobved it back to slow [14:51:04] <_joe_> gonna run the slow ones now [14:51:15] anomie: present [14:52:58] FlorianSW: Are we still wanting to do your change in light of https://lists.wikimedia.org/pipermail/wikitech-l/2014-September/078731.html ? [14:54:00] anomie: i was thinking about this, too. But that's no matter. If the extension itself doesn't have a REL1_24, it will not be displayed. And the conservation of the thread goes to the direction to don't rebranch the REL1_24 :) [14:54:09] anomie: so i would say: yes :) [14:54:16] ok [14:55:25] * YuviPanda goes afk for a bit [15:00:17] FlorianSW: I'll do yours first [15:00:21] (03PS2) 10Anomie: Add REL1_24 as branch in ExtensionDistributor [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161666 (owner: 10Florianschmidtwelzow) [15:00:28] (03CR) 10Anomie: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161666 (owner: 10Florianschmidtwelzow) [15:00:29] anomie: ok :) thx :P [15:00:33] (03Merged) 10jenkins-bot: Add REL1_24 as branch in ExtensionDistributor [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161666 (owner: 10Florianschmidtwelzow) [15:01:11] !log anomie Synchronized wmf-config/CommonSettings.php: SWAT: Add REL1_24 as branch in ExtensionDistributor [[gerrit:161666]] (duration: 00m 10s) [15:01:12] FlorianSW: ^ test please [15:01:17] Logged the message, Master [15:01:29] I'll do mine next. [15:01:32] * FlorianSW is testing [15:01:51] anomie: great, it's working :) Thanks! [15:01:55] (03PS2) 10Anomie: 'securepoll-create-poll' for sysop on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161653 [15:02:02] (03CR) 10Anomie: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161653 (owner: 10Anomie) [15:02:06] (03Merged) 10jenkins-bot: 'securepoll-create-poll' for sysop on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161653 (owner: 10Anomie) [15:02:25] !log anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add securepoll-create-poll right to sysop on testwiki [[gerrit:161653]] (duration: 00m 09s) [15:02:27] anomie: ^ test please [15:02:31] Logged the message, Master [15:02:47] anomie: Seems to work [15:03:11] Glaisher: If you want your SWAT patches deployed, please respond. [15:03:25] * anomie doesn't even see yurik online [15:05:25] anomie: graph extension is still enabled on some wikis, and i know a demo page on mw.org, maybe do this without yurik? :/ [15:05:34] Reedy: Regarding Gerrit 161684 (removing some namespace aliases so the interwiki prefixes can work), is there a maintenance script to run? Or just let the wiki's users null edit pages? [15:05:57] FlorianSW: If you want to be responsible for testing it and such, that's fine with me. [15:06:17] anomie: yes, just see, if it works, if there is an error, i can't do anything :) [15:06:31] (03PS2) 10Anomie: Enable Graph ext on mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161908 (owner: 10Yurik) [15:06:37] (03CR) 10Anomie: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161908 (owner: 10Yurik) [15:06:42] (03Merged) 10jenkins-bot: Enable Graph ext on mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161908 (owner: 10Yurik) [15:07:00] !log anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Graph extension on mediawiki.org [[gerrit:161908]] (duration: 00m 09s) [15:07:01] FlorianSW: ^ test please [15:07:09] * FlorianSW testing [15:08:49] anomie: working, nothing obviously failing [15:09:27] anomie: I like how you keep the patten even for your own patches :) [15:09:30] (03PS1) 10Alexandros Kosiaris: url_downloader: assign to chromium [puppet] - 10https://gerrit.wikimedia.org/r/161951 [15:09:48] greg-g: Me too (: [15:10:18] <_joe_> anomie: at first I thought you were speaking to yourself :P [15:10:59] (03PS1) 10Yuvipanda: nagios_common: Move check_ssl_cert into module [puppet] - 10https://gerrit.wikimedia.org/r/161952 [15:11:34] Glaisher: Ping me when you're ready to test your SWAT patches, if it's before 15:50 UTC. [15:11:52] (03PS2) 10Giuseppe Lavagetto: puppet: introduce hiera for production [puppet] - 10https://gerrit.wikimedia.org/r/160924 [15:15:45] (03PS1) 10Andrew Bogott: Remove a bunch of iptables classes that haven't been used for ages. [puppet] - 10https://gerrit.wikimedia.org/r/161954 [15:15:47] (03PS1) 10Andrew Bogott: Set up firewall rules for ldap servers. [puppet] - 10https://gerrit.wikimedia.org/r/161955 [15:16:26] Can I get a review/advice from someone who has experience with ferm? https://gerrit.wikimedia.org/r/#/c/161955/1 [15:17:48] (03PS1) 10Alexandros Kosiaris: Change url-downloader to point to new IP [dns] - 10https://gerrit.wikimedia.org/r/161956 [15:18:25] YuviPanda: what about our really fun naming bikeshed/! [15:18:27] we merged already!? [15:18:34] ottomata: apparently :) [15:18:39] ottomata: we can still change he name in the future [15:18:53] ottomata: but I'm inclined to just move the git merge stuff out of monitoring module into nagios_common [15:19:20] either way, but I don't like the nagios_common name! [15:19:23] so I still want to bikeshed! [15:19:34] ottomata: :D reply on the thread! :D [15:19:44] no one has responded to my ideas there! [15:19:45] yet [15:20:05] ok, i'm really busy with some analytics reviews and meetings and kafkatee troubleshooting today, so i probably won't be able to do think about it much today [15:20:20] sooo, ok ok, keep working, this isn't being used in prod yet, right? [15:20:27] let's make sure we get that stuff worked out before we put it into prod [15:20:28] ottomata: it is :) [15:20:30] !!! [15:20:40] ottomata: were merged into neon earlier [15:20:48] but I see no reason we can't rename later if we need to [15:20:51] yeah guess not [15:20:57] ok... [15:21:15] hey _joe_, i think it is fine [15:21:29] but there was still outstanding conversation on that change...you sure it was ready to be merged? [15:22:44] ottomata: almost all were addressed, no? [15:22:53] except the naming bikeshed, which I don't think should block merges [15:23:34] ottomata: I had to remove support for templating (content => ) because of a puppet bug, so I'll have to find some way of factoring that in when I port the checks that use templates for config [15:27:55] * YuviPanda steps afk [15:29:12] <_joe_> ottomata: I know you don't like it, and I've been the usual bully, sorry :) [15:29:33] <_joe_> I figured we could continue discussing naming for now as it's not going to be complicated to change [15:29:43] <_joe_> while we don't block yuvi [15:29:59] <_joe_> whose changes are in all other respects very good [15:30:35] haha, you just wait til faidon comes back and makes your modules go through weeks of review to get them perfect...:) [15:33:53] <_joe_> I don't think choosing a name is part of "getting them perfect" [15:33:59] <_joe_> but, that's just me [15:39:32] Jeff_Green: ^ around ? [15:43:04] tonythomas: yup [15:43:28] (03PS1) 10Manybubbles: Lower throttle on two cirrus jobs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161960 [15:44:23] ah. great. I found out we had little bug in our exim regex - fixed in here https://gerrit.wikimedia.org/r/#/c/161679/ -- and in our labs instance -- exim is not able to do the curl request due to some reason ( some firewall blocking or so ? ) [15:45:05] Jeff_Green: I think its defineetly due to some permission issue out there [15:45:05] ok [15:45:17] did the regex fix get merged? [15:46:17] Jeff_Green: not yet. [15:46:39] I found symbol '+' turning up in the VERP addresses [15:46:55] is it in every field, or just the last field? [15:47:05] only in the last field [15:47:15] odd [15:47:19] I have pasted a sample one in the commit message [15:47:26] true that. [15:47:34] ya, i saw but I was wondering why that field was different [15:47:52] true. Let me recheck our hashing algorithms [15:48:28] (03PS1) 10Nikerabbit: Update CX config for changes in master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161961 [15:49:22] (03CR) 10Andrew Bogott: "My only real concern with this is that invoking ferm on a previously un-ferm'd system will spontaneously close a bunch of other ports that" [puppet] - 10https://gerrit.wikimedia.org/r/161955 (owner: 10Andrew Bogott) [15:49:27] hm… akosiaris, ferm review? https://gerrit.wikimedia.org/r/#/c/161955/ [15:52:24] !log reboot ms-be2001 into PXE to test a re-install [15:52:28] Logged the message, Master [15:53:58] (03CR) 10Jsahleen: [C: 031] "lgtm" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161961 (owner: 10Nikerabbit) [15:55:46] (03CR) 10KartikMistry: [C: 031] Update CX config for changes in master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161961 (owner: 10Nikerabbit) [15:56:49] (03CR) 10Nikerabbit: [C: 032] "+1+1 is +2, right?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161961 (owner: 10Nikerabbit) [15:56:54] (03Merged) 10jenkins-bot: Update CX config for changes in master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/161961 (owner: 10Nikerabbit) [15:58:50] Jeff_Green: any idea where the '+' came in between ? [15:59:50] no [16:01:06] (03CR) 10Alexandros Kosiaris: [C: 04-1] Set up firewall rules for ldap servers. (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/161955 (owner: 10Andrew Bogott) [16:01:14] andrewbogott: done ^ [16:02:58] Jeff_Green: we have a hash_hmac which should never(?) return a '+' which is base_64 encoded too [16:02:59] (03CR) 10Jgreen: [C: 04-1] Corrected the exim regex expression (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/161679 (owner: 1001tonythomas) [16:04:01] tonythomas: i wondered the same thing--also aren't you using hmac for each field? [16:04:38] tonythomas: note, assuming the + turns out to be sane your regex needs a minor fix [16:05:59] yup. we have the hmac snipped, and later base64 encoded [16:06:10] Jeff_Green: I am trying out the regex fix [16:06:14] will update in a while [16:06:21] k. [16:06:23] base64 encoded can contain + [16:06:30] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There is one unmerged change in mediawiki_config (dir /srv/mediawiki-staging/). [16:06:31] oh! yay ! [16:06:41] it's chars, numbers, + and / [16:06:43] I almost doubted the php hmacs [16:07:10] oh / too [16:07:11] so we should include the possibility of / too ? [16:07:13] https://en.wikipedia.org/wiki/Base64#Implementations_and_history [16:07:17] yeah ;) [16:07:57] ok. so something like \N^wiki-\w+-\w+-\w+-[+/\w]+$\N [16:07:58] ? [16:08:16] yeah [16:08:39] ok. let me test in the labs instance [16:09:50] ok. works as expected. now pushing in [16:12:30] (03PS2) 1001tonythomas: Corrected the exim regex expression [puppet] - 10https://gerrit.wikimedia.org/r/161679 [16:13:17] (03PS2) 10Andrew Bogott: Set up firewall rules for ldap servers. [puppet] - 10https://gerrit.wikimedia.org/r/161955 [16:13:46] Jeff_Green, hoo : done. now to our second issue. Just copy pasting something from labs earlier. I am able to get this command https://dpaste.de/cUK6 executed while the exim delivery is happening ( asking this shell to execute when the exim recieves a bounce, in the pipe transport ), but when I remove the echo " "> and make something like this [16:13:46] https://dpaste.de/yQSu -- exim shows a child process killed error message. [16:14:08] (03CR) 10jenkins-bot: [V: 04-1] Set up firewall rules for ldap servers. [puppet] - 10https://gerrit.wikimedia.org/r/161955 (owner: 10Andrew Bogott) [16:15:31] people in #exim told me that this can cause due to apparmor configs too [16:15:55] I tried killing all the configs and service of apparmor - but this problem seems to exist [16:16:15] what's exactly the problem? [16:16:20] tonythomas: curl is writing the output to stdout? [16:16:23] curl exiting with a non-zero exit code? [16:16:35] (03PS3) 10Andrew Bogott: Set up firewall rules for ldap servers. [puppet] - 10https://gerrit.wikimedia.org/r/161955 [16:16:47]