[00:00:04] RoanKattouw, ^d, marktraceur, MaxSem: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141216T0000). Please do the needful. [00:02:36] if nobody's interested, I guess I can [00:03:05] * MaxSem pokes aude, gi11es, marktraceur [00:04:01] ? [00:04:28] we will have something but it will take a bit longer, so might have to do it ourselves in a bit [00:05:08] eh, looked at the wrong row:P [00:05:18] heh [00:05:31] in such case, ping superm401, ebernhardson, yurikR [00:05:34] * aude is keeping jenkins busy :/ [00:07:05] MaxSem, sup? [00:07:30] going to deploy your stuff [00:07:34] kul :) [00:07:52] (03CR) 10MaxSem: [C: 032] Don't collapse sections on mobile WD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179513 (owner: 10MaxSem) [00:08:07] is Kul even working for us? [00:08:36] kul is always with us [00:09:47] eh, zuul queue is fubar [00:10:17] meh https://performance.wikimedia.org/xenon/theme/style.css [00:10:35] @import url('//fonts.googleapis.com/css?family=Open+Sans'); [00:10:49] third party inclusion -> bad [00:11:09] (03Merged) 10jenkins-bot: Don't collapse sections on mobile WD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179513 (owner: 10MaxSem) [00:12:00] !log maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/179513 (duration: 00m 06s) [00:12:06] Logged the message, Master [00:13:28] (03PS1) 10BryanDavis: Convert from wfErrorLog to MWLogger logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180082 [00:14:41] (03PS1) 10Hoo man: Don't load fonts from google on performance.wikimedia.org/xenon/ [puppet] - 10https://gerrit.wikimedia.org/r/180083 [00:14:57] ori: ^ [00:21:16] okayyy, ebernhardson and superm401 - since you're not around, I'm not deploying your changes [00:21:30] MaxSem, I'm here. [00:21:35] aha:) [00:21:44] MY SCARE TACTICS WORKS [00:22:06] Sorry, I forgot ACKing was required for SWAT. [00:22:10] I saw your earlier post. [00:22:31] MaxSem: it only pings me if it starts with ebernhardson :P [00:22:49] yup, I need someone to verify your changes work and don't break stuff [00:23:50] (03CR) 10MaxSem: [C: 032] Enable EventLogging for Flow [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179981 (owner: 10Mattflaschen) [00:24:24] MaxSem: hmm, probably fine but i think i have to wait ~5min for resource loader [00:25:20] ebernhardson, what's fine? [00:26:08] MaxSem: the patch you merged to wmf12 for me, https://gerrit.wikimedia.org/r/180043 [00:26:29] oh i suppose i can debug=1 it [00:26:33] well, I haven't deployed it yet because you weren't responding;) [00:26:34] but not quite the same :) [00:26:36] oh :P [00:28:23] (03Merged) 10jenkins-bot: Enable EventLogging for Flow [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179981 (owner: 10Mattflaschen) [00:28:51] pfft zuul [00:29:52] !log maxsem Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/179981 (duration: 00m 07s) [00:29:56] Logged the message, Master [00:29:57] superm401, ^ [00:30:14] Thanks, MaxSem. Should be fine, but I'll do a quick sanity check. [00:31:49] MaxSem,yeah, we're good. [00:32:43] MaxSem, there is another patch for zeroportal, but that one we should just bring it to master (or i could do it right after swat) [00:33:02] yep, too late to add stuff to swat [00:34:52] yurikR, my checkout of ZeroBanner is broken - could you make submodule updates? [00:35:29] MaxSem, on tin? [00:35:51] MaxSem, actually it still hasn't been merged by jenkins ( [00:35:52] in git [00:35:59] oh shit [00:36:07] just merged [00:36:09] sup? [00:36:44] aha, a bunch of wikidata jobs is gone [00:36:50] works again:) [00:37:55] !log maxsem Synchronized php-1.25wmf12/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/180043/ (duration: 00m 08s) [00:37:58] Logged the message, Master [00:38:01] ebernhardson, ^ [00:38:49] MaxSem, ideally i would love for https://gerrit.wikimedia.org/r/#/c/180029/ to also be deployed [00:38:57] but i could do it myself (it will be merge to master) [00:39:24] eh, looks too large for a swat [00:40:38] (03CR) 10Yurik: [C: 031] Update outbound X-CS behavior in light of unified [puppet] - 10https://gerrit.wikimedia.org/r/179571 (owner: 10Dr0ptp4kt) [00:40:50] bblack, when you have a sec ^ [00:41:43] jouncebot, next [00:41:44] In 15 hour(s) and 18 minute(s): Morning SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141216T1600) [00:42:15] greg-g, MaxSem, aude, i would like to go right after swat for zeroportal patch depl [00:44:23] MaxSem: looks good, works as expected. thanks [00:48:06] grmbl [00:48:19] yurikR, can you deploy ZB too then? [00:48:28] up to you ) [00:48:42] MaxSem, are you done? [00:49:05] need to do a scap [00:49:16] !log maxsem Started scap: (no message) [00:49:22] Logged the message, Master [00:53:07] bleh, that will be forever [00:53:36] (03PS1) 10Yuvipanda: Create column objects as well [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180098 [00:53:51] oh noes, scap? [00:54:07] what we would like to deploy needs a scap [00:59:00] (03PS2) 10Yuvipanda: Create column objects as well [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180098 [00:59:39] aude, when are you deploying? i added myself to the depl schedule [01:00:13] (and i don't need scap) [01:01:23] yurikR: soon as there is an opportunity [01:01:31] (03PS3) 10Yuvipanda: Create column objects as well [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180098 [01:01:33] aude, can you go right after me? [01:01:39] sure [01:01:42] i have two exts to push [01:01:59] ok [01:10:55] * yurikR is waiting for scap... oh max max, why oh why... [01:15:29] because sikrit [01:15:52] !log maxsem Finished scap: (no message) (duration: 26m 35s) [01:15:59] yurikR, aude ^^ [01:16:04] Logged the message, Master [01:16:05] :) [01:16:25] (03CR) 10Yuvipanda: [C: 032 V: 032] Create column objects as well [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180098 (owner: 10Yuvipanda) [01:16:45] eyi!!! [01:16:48] here i go [01:17:11] (03PS1) 10Yuvipanda: Read whitelist/greylist from files and compare to db [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180104 [01:20:21] !log yurik Synchronized php-1.25wmf11/extensions/ZeroBanner: (no message) (duration: 00m 06s) [01:20:30] Logged the message, Master [01:21:19] (03PS2) 10Yuvipanda: Read whitelist/greylist from files and compare to db [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180104 [01:30:05] !log yurik Synchronized php-1.25wmf12/extensions/ZeroPortal: (no message) (duration: 00m 07s) [01:30:11] Logged the message, Master [01:34:49] !log yurik Synchronized php-1.25wmf11/extensions/ZeroBanner: (no message) (duration: 00m 09s) [01:34:56] Logged the message, Master [01:35:28] (03PS3) 10Yuvipanda: Read whitelist/greylist from files and compare to db [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180104 [01:36:09] !log yurik Synchronized php-1.25wmf12/extensions/ZeroBanner: (no message) (duration: 00m 05s) [01:36:13] Logged the message, Master [01:36:24] aude, done [01:36:42] ok [01:37:23] (03PS4) 10Yuvipanda: Read whitelist/greylist from files and compare to db [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180104 [01:42:14] (03PS2) 10Ori.livneh: Don't load fonts from google on performance.wikimedia.org/xenon/ [puppet] - 10https://gerrit.wikimedia.org/r/180083 (owner: 10Hoo man) [01:42:21] hoo: thanks [01:42:40] (03CR) 10Ori.livneh: [C: 032 V: 032] Don't load fonts from google on performance.wikimedia.org/xenon/ [puppet] - 10https://gerrit.wikimedia.org/r/180083 (owner: 10Hoo man) [01:43:18] going to deploy now (and scap) [01:47:33] (03PS5) 10Yuvipanda: Read whitelist/greylist from files and compare to db [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180104 [01:47:43] !log aude Started scap: Update test.wikidata [01:47:51] Logged the message, Master [01:49:39] (03CR) 10Yuvipanda: [C: 032 V: 032] Read whitelist/greylist from files and compare to db [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180104 (owner: 10Yuvipanda) [01:53:03] (03PS1) 10Ori.livneh: xenon: Tweak the text of the footer [puppet] - 10https://gerrit.wikimedia.org/r/180109 [01:53:24] (03CR) 10Ori.livneh: [C: 032 V: 032] xenon: Tweak the text of the footer [puppet] - 10https://gerrit.wikimedia.org/r/180109 (owner: 10Ori.livneh) [01:55:56] (03PS1) 10Yuvipanda: Add feature to ignore databases with a particular regex [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180110 [01:57:34] (03PS2) 10Ori.livneh: hhvm: fix include_path [puppet] - 10https://gerrit.wikimedia.org/r/179974 (owner: 10Hashar) [01:58:29] (03CR) 10Yuvipanda: [C: 032 V: 032] Add feature to ignore databases with a particular regex [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180110 (owner: 10Yuvipanda) [02:05:11] (03PS1) 10Yuvipanda: Move commandline arguments to config file [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180112 [02:06:21] !log aude Finished scap: Update test.wikidata (duration: 18m 38s) [02:06:31] Logged the message, Master [02:12:16] !log l10nupdate Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s) [02:12:21] !log LocalisationUpdate completed (1.25wmf11) at 2014-12-16 02:12:20+00:00 [02:12:24] Logged the message, Master [02:12:29] Logged the message, Master [02:12:48] (03PS2) 10Yuvipanda: Move commandline arguments to config file [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180112 [02:14:37] (03PS3) 10Yuvipanda: Move commandline arguments to config file [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180112 [02:16:48] (03PS4) 10Yuvipanda: Move commandline arguments to config file [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180112 [02:19:02] (03CR) 10Yuvipanda: [C: 032 V: 032] Move commandline arguments to config file [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180112 (owner: 10Yuvipanda) [02:23:53] !log l10nupdate Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 02s) [02:23:58] !log LocalisationUpdate completed (1.25wmf12) at 2014-12-16 02:23:58+00:00 [02:24:00] Logged the message, Master [02:24:06] Logged the message, Master [02:31:44] (03CR) 10Ori.livneh: [C: 032] hhvm: fix include_path [puppet] - 10https://gerrit.wikimedia.org/r/179974 (owner: 10Hashar) [02:32:55] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [02:33:35] this alert always goes off when the spike has subsided [02:37:45] !log upgrade db1055 trusty [02:37:51] Logged the message, Master [02:42:11] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [02:54:22] (03PS1) 10Springle: upgrade db1055 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/180117 [02:55:24] (03CR) 10Springle: [C: 032] upgrade db1055 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/180117 (owner: 10Springle) [03:46:48] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 16 03:46:47 UTC 2014 (duration 46m 46s) [03:46:53] Logged the message, Master [04:38:51] (03PS3) 1020after4: Move the top level variables in phabricator role in a class [puppet] - 10https://gerrit.wikimedia.org/r/179930 (owner: 10Alexandros Kosiaris) [04:40:14] (03CR) 1020after4: [C: 031] Move the top level variables in phabricator role in a class [puppet] - 10https://gerrit.wikimedia.org/r/179930 (owner: 10Alexandros Kosiaris) [04:40:22] hi [04:40:34] Just got this: Request: POST http://phabricator.wikimedia.org/auth/login/mediawiki:mediawiki/, from 10.64.0.172 via cp1044 cp1044 ([10.64.0.172]:80), Varnish XID 1597571183 [04:40:35] Forwarded for: 198.73.209.1, 10.64.0.172 [04:40:37] Error: 503, Service Unavailable at Tue, 16 Dec 2014 04:32:53 GMT [04:41:21] (03CR) 1020after4: [C: 031] phabricator: strip Ubuntu 12.04 (precise) support [puppet] - 10https://gerrit.wikimedia.org/r/179882 (owner: 10Faidon Liambotis) [04:48:13] anyone around? [04:48:42] abartov: hi [04:49:22] is there somewhere else I need to report this? [04:49:35] how did you land on that page? [04:49:38] Or is the message inviting me to report to an Ops person itself outdated, because it's all captured automagically these days? [04:50:11] 1. I was looking at a Phab item, and clicked Subscribe. [04:50:41] if you try again, does the problem recur? [04:50:50] 2. I was told my session is something-something, and to re-login. [04:50:50] 3. I clicked the "via mediawiki" button [04:50:50] 4. got this message. [04:51:12] yes, when just refreshing the page. Will now try to retrace the flow from the ticket. [04:51:58] yup, happened again. [04:51:58] the message inviting you to report the error is not outdated -- you should report it [04:52:04] Request: POST http://phabricator.wikimedia.org/auth/login/mediawiki:mediawiki/, from 10.64.0.172 via cp1044 cp1044 ([10.64.0.172]:80), Varnish XID 1597576274 [04:52:12] Forwarded for: 198.73.209.1, 10.64.0.172 [04:52:12] Error: 503, Service Unavailable at Tue, 16 Dec 2014 04:51:39 GMT [04:52:31] twentyafterfour: around? [04:52:42] ori: yes [04:52:56] does abartov's problem ring a bell? [04:53:26] reading... [04:55:50] ori: hmm... familiar yes, but not something I am currently able to reproduce or debug effectively.. [04:56:23] abartov: I just tried the same and it's working for me. Can you log in ok via ldap username/password? [04:58:46] (03CR) 10Gage: [C: 032] logstash: port udp2log rules to monolog input [puppet] - 10https://gerrit.wikimedia.org/r/179758 (owner: 10BryanDavis) [04:59:17] twentyafterfour: seeing this in iridium:/var/log/apache2/phabricator_error.log : https://dpaste.de/yC2O/raw [05:00:33] ori: looking into it [05:00:41] cool, thanks [05:00:43] I think we might have inherited an upstream bug somewhere [05:00:54] thanks for the logs, btw. That's helpful [05:01:01] np [05:02:43] twentyafterfour: er, what username is that? My Wikimedia user? [05:08:02] abartov: it would be your wikitech username (if you've got an account there) [05:08:17] ah, right. [05:08:19] let me try [05:09:34] well, it lets me log in with ldap, and invites me to create an account. I don't actually want to go ahead, though, because I don't want two accounts, and do want to use my main WM identity. [05:12:22] abartov: right [05:13:19] abartov: can you try creating a 'private window' or 'incognito window' and see if mediawiki oauth login will work with a clean slate? ( without any of your usual cookies / session state ) [05:17:40] twentyafterfour: sure, hang on [05:19:11] twentyafterfour: same fail! [05:19:29] using a brand-new Firefox private session. [05:21:25] abartov: and your phabricator username is abartov? [05:22:34] (03PS1) 10Springle: repool db1055 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180119 [05:23:01] abartov: I don't see that username in phab. so what's your wiki username / phab username? [05:23:19] (03CR) 10Springle: [C: 032] repool db1055 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180119 (owner: 10Springle) [05:23:24] (03Merged) 10jenkins-bot: repool db1055 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180119 (owner: 10Springle) [05:24:19] yes [05:24:33] !log springle Synchronized wmf-config/db-eqiad.php: repool db1055, warm up (duration: 00m 07s) [05:24:33] that is, the wikitech/ldap username [05:24:56] my SUL usernames are "Ijon" (volunteer account) and "Asaf (WMF)" (staff account) [05:25:21] twentyafterfour: sorry, keep forgetting to flag your name. [05:27:06] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [05:27:26] abartov: so which is your phab account name that you can't access? [05:27:38] the SUL [05:28:11] twentyafterfour: I doubt it has anything to do with the specific account, though. [05:28:36] abartov: It must be something, my account is working fine. I'm just trying to find logs for you [05:28:47] which phabricator username is it? I can't find you [05:28:52] twentyafterfour: given a private session failed just by my clicking on the "via mediawiki" button, without grabbing credentials from a cookie and without getting to the OAuth dialogue... [05:29:17] abartov: that's odd it works when I do the exact same thing [05:29:52] twentyafterfour: the fact it works for you does not necessarily mean it's to do with my account. I'm guessing it could also be a cluster issue, with us on different ones? [05:30:31] abartov: that could be. still I'm trying to understand what's happening and I'm not getting anywhere [05:30:39] abartov: it doesn't seem to be an issue with phabricator at all [05:30:41] twentyafterfour: there's an actual error message here. Isn't that enough to find it in the logs? [05:30:58] abartov: I don't have any kind of access [05:31:09] all I can touch is phabricator [05:31:33] twentyafterfour: the precise failing scenario is: [05:32:00] twentyafterfour: 1. clicking the "via mediawiki" button sends the browser to https://phabricator.wikimedia.org/auth/start/?next=%2FT37489 [05:32:28] twentyafterfour: 2. which redirects to https://phabricator.wikimedia.org/auth/login/mediawiki:mediawiki/, which yields this error 503. [05:33:05] twentyafterfour: presumably, whatever is being contacted to issue the oauth request is failing. [05:33:18] twentyafterfour: so who does have access? [05:45:16] abartov: members of ops team. I'm still trying to tell if the problem is strictly with phab or something else, you never told me your phab username (not your SUL name but the name you used in phabricator) [05:46:36] twentyafterfour: I don't remember picking one. I did log in with my SUL credentials before. But if it isn't 'abartov', it should be 'Ijon'. [05:47:06] abartov: ah ha! I do see Ijon [05:51:21] abartov, ori: I can't find anything related to phabricator or that account which would cause this to happen... ori: any way to tell if those log messages correspond with abartov's login attempts? I can't tell for sure ... [05:54:36] twentyafterfour: thanks for looking into it! [05:56:18] ori: I was able to reproduce the errors you saw in the logs on iridium but I don't think they are related to the problem with abartov's login ... what I don't understand is why it's getting a 503 error from varnish...usually if phab is directly throwing an exception it will report that to the client rather than throwing up the varnish error page. [06:05:03] PROBLEM - Mediawiki Apple Dictionary Bridge on terbium is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string Wikimedia_Foundation not found on https://search.wikimedia.org:443https://search.wikimedia.org/?lang=ensite=wikipediasearch=Wikimedia_Foundationlimit=1 - 3389 bytes in 0.039 second response time [06:29:26] (03PS1) 10Springle: depool db1073 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180120 [06:29:58] (03CR) 10Springle: [C: 032] depool db1073 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180120 (owner: 10Springle) [06:30:03] (03Merged) 10jenkins-bot: depool db1073 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180120 (owner: 10Springle) [06:31:40] !log springle Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 00m 06s) [06:33:16] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:15] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:17] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:47] !log sync-common on mw1043 after sync-file fail [06:34:51] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 2 failures [06:35:40] useless morebots [06:36:11] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:33] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:37:23] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:38:49] !log <+logmsgbot> !log springle Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 00m 06s) [06:38:55] Logged the message, Master [06:38:59] !log 06:32 < springle> !log sync-common on mw1043 after sync-file fail [06:39:01] Logged the message, Master [06:43:36] (03PS1) 10KartikMistry: Beta: Fixed comment [puppet] - 10https://gerrit.wikimedia.org/r/180121 [06:48:28] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:48:31] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:05:01] !log upgrade db1073 trusty [07:05:06] Logged the message, Master [07:06:04] <_joe_> morning [07:06:07] <_joe_> hi sean [07:06:17] hey _joe_ [07:06:27] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:06:44] oh _joe_ is here.. must be time for my afternoon coffee :) [07:06:58] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:07:02] <_joe_> eheh [07:07:02] _joe_ o'clock [07:07:07] <_joe_> I just had 2 [07:07:11] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:07:21] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:07:26] <_joe_> one large moka cup and one espresso at a bar [07:07:57] <_joe_> I got enough caffeine to wake up an elephant, probably [07:08:07] espresso at bars is quite a popular italian thing, right [07:08:22] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:09:07] <_joe_> springle: our "bars" usually have little or no sitting space, you can get espresso, cappuccino, alcoholics and some quick food [07:09:59] <_joe_> that's one of the reasons it's hard for me to go work from a cafe' [07:10:40] ah gotcha [07:11:01] yeah, mostly coffee -> cafe, alcohol -> bar, here [07:11:05] at least in my area [07:11:26] <_joe_> yeah we don't really have that distinction [07:11:35] and cafe -> gunmen this week [07:12:06] <_joe_> in the north, where the winter is colder, people have "caffe' corretto", which is espresso with grappa or another liquor poured in [07:12:12] <_joe_> yeah I've seen :/ [07:13:03] <_joe_> (I won't note how some christian madman has shot and killed 6 people in the USA yesterday, but Italian newspapers only care about OMFG THE MUSLIM TERROR) [07:13:28] yeah similar here, though there are voices of reason around [07:14:03] <_joe_> but well, I can imagine your case has more relevance locally [07:14:19] <_joe_> also, the terrifying part is that all it takes is one madman with a gun [07:14:34] i think it surprised people, mostly. we're so far from all that stuff [07:14:43] <_joe_> yes [07:14:52] <_joe_> thanks dick cheney [07:15:09] yet our bombs are participating [07:15:21] <_joe_> in 2001, maybe .1% of the middle east saw Europe and Australia as enemies [07:15:26] <_joe_> now, maybe 50% [07:15:38] <_joe_> and for a good reason too. [07:15:48] <_joe_> nah [07:15:54] au government has just pulled more funding from foreign aid to put into defence spending :| [07:15:58] <_joe_> I vote to call that yuvipanda-wm [07:16:06] <_joe_> springle: sigh [07:23:00] (03PS2) 10Giuseppe Lavagetto: mediawiki: enhancements to hhvm_cleanup_cache [puppet] - 10https://gerrit.wikimedia.org/r/179102 [07:23:10] (03PS8) 10Andrew Bogott: Support bootstrap-vz for building labs debian images [puppet] - 10https://gerrit.wikimedia.org/r/179765 [07:24:28] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki: enhancements to hhvm_cleanup_cache [puppet] - 10https://gerrit.wikimedia.org/r/179102 (owner: 10Giuseppe Lavagetto) [07:27:10] <_joe_> did I say I hate git submodules? [07:27:23] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [07:28:30] _joe_: it bears repeating [07:28:52] <_joe_> now to fix 3 lines I need two commits [07:28:57] <_joe_> and 2 reviews [07:28:59] <_joe_> MEH [07:29:51] (03PS1) 10Giuseppe Lavagetto: cdh: fix templates causing warnings [puppet/cdh] - 10https://gerrit.wikimedia.org/r/180124 [07:41:32] (03PS1) 10KartikMistry: WIP: Add cxserver role for production [puppet] - 10https://gerrit.wikimedia.org/r/180125 [07:42:51] (03PS2) 10Giuseppe Lavagetto: Add some experimental settings to one server in each pool [puppet] - 10https://gerrit.wikimedia.org/r/178478 [07:45:06] (03PS2) 10KartikMistry: WIP: Add cxserver role for production [puppet] - 10https://gerrit.wikimedia.org/r/180125 [07:47:05] (03CR) 10Giuseppe Lavagetto: [C: 032] Add some experimental settings to one server in each pool [puppet] - 10https://gerrit.wikimedia.org/r/178478 (owner: 10Giuseppe Lavagetto) [07:48:59] (03Abandoned) 10Giuseppe Lavagetto: hhvm: define more jit configurations [puppet] - 10https://gerrit.wikimedia.org/r/175412 (owner: 10Giuseppe Lavagetto) [07:55:28] (03PS1) 10Giuseppe Lavagetto: deployment: make the keyholder key path configurable [puppet] - 10https://gerrit.wikimedia.org/r/180126 [07:57:08] (03CR) 10Giuseppe Lavagetto: "Please see how to do this next time here:" [puppet] - 10https://gerrit.wikimedia.org/r/179875 (owner: 10Hashar) [08:02:23] * yuvipanda-wm notifies _joe_ of all email to root@ [08:02:34] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.003 second response time [08:03:08] <_joe_> lol [08:03:36] Yeah I bet that happens :) [08:03:42] Err [08:03:48] I bet that's useful? [08:03:55] <_joe_> sure [08:03:59] * yuvipanda-wm just woke up, should probably head back to sleep [08:04:29] <_joe_> yuvipanda-wm: me too [08:05:06] <_joe_> yuvipanda-wm: garage punk usually helps waking me up [08:05:33] Aah [08:05:39] It is really hard to wake me up [08:05:51] A skill acquired from years of sleeping in college [08:06:40] (We had mandatory attendance) [08:07:10] Also what is a garage punk? [08:07:31] I presumed it was some kid playing loud things from a garage [08:07:38] <_joe_> it's a particular current of punk rock [08:07:40] But could also be a music genre [08:07:42] Aaah [08:07:46] <_joe_> the most raw and rock'n rolly [08:07:47] Heh [08:07:56] I see [08:09:05] <_joe_> https://en.wikipedia.org/wiki/Garage_punk [08:09:12] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.020 second response time [08:09:49] greetings [08:09:53] <_joe_> ciao godog [08:10:02] ciao _joe_ [08:10:09] I'll play the song now anyway [08:10:25] <_joe_> ehehe [08:10:30] <_joe_> which version? [08:10:54] https://www.youtube.com/watch?v=DpkDdLZGg30 this [08:11:23] <_joe_> I love the Willy DeVille mariachi version :P [08:11:39] the right hand guitar played left handed always looks funny [08:12:13] I'll give it a try [08:12:26] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [08:12:33] <_joe_> mh [08:13:54] (03CR) 10Giuseppe Lavagetto: [C: 032] deployment: make the keyholder key path configurable [puppet] - 10https://gerrit.wikimedia.org/r/180126 (owner: 10Giuseppe Lavagetto) [08:16:45] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 2 failures [08:17:41] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [08:17:52] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 1 failures [08:17:59] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Puppet has 2 failures [08:18:20] <_joe_> wha? [08:18:31] <_joe_> esams link troubles? [08:19:13] <_joe_> Could not evaluate: Connection timed out on the hosts [08:19:33] <_joe_> yeah we had a link hiccup clearly [08:23:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [08:27:01] (03PS1) 10Giuseppe Lavagetto: hhvm: fix yaml quotation [puppet] - 10https://gerrit.wikimedia.org/r/180129 [08:29:32] (03CR) 10Giuseppe Lavagetto: [C: 032] hhvm: fix yaml quotation [puppet] - 10https://gerrit.wikimedia.org/r/180129 (owner: 10Giuseppe Lavagetto) [08:29:57] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:30:16] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:30:22] (03PS1) 10KartikMistry: WIP: Add apertium role for production [puppet] - 10https://gerrit.wikimedia.org/r/180130 [08:32:17] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:33:57] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:42:37] !log upgrading trusty/codfw to diamond 3.5-3 [08:42:43] Logged the message, Master [09:03:26] godog: hmm, do you know how hard / easy it is to set up a per-project reprepo (or similar debian package local hosting solution that allows us separate trusty and precise packages)? [09:03:38] tools has custom packages, a few of them, and we also have a mix of distros now [09:08:38] is that "labsdebrepo" ? [09:09:45] yuvipanda-wm: manifests/misc/labsdebrepo.pp:class misc::labsdebrepo { ? [09:10:04] mutante: yes, but that doesn’t really support trusty/precise, AFAIK [09:10:09] of course I could be totally wrong as well :) [09:10:24] and you don’t need anything extra other than muddling with the packages as well [09:10:37] hmm, does it need to support a specific distro? [09:10:51] it just sets up sources.list , dpkg-scanpackages etc [09:11:12] i dont see anything specific to a distro name [09:16:39] yuvipanda-wm: good question, you might be able to use the reprepro module as-is and get it setup on the respective project puppetmaster perhaps with some puppet glue, what's the use case? [09:16:52] the trusty/precise part would be apt::pin [09:16:55] ? [09:17:24] godog: toollabs has a few custom packages, and some have different versions for precise/trusty [09:19:07] mutante: not sure what you mean? [09:19:18] yuvipanda-wm: I see, where are the packages stored now? [09:19:41] godog: with labsdebrepo, which is just on NFS with an exec for dpkg-scasnpackages [09:20:28] ah ok, well yeah since the instance storage is ephemeral anyways [09:21:10] godog: yeah [09:22:56] you want to use different packages per release, right [09:23:01] that would be APT pinning [09:23:08] and we have a puppet define for it [09:24:08] oh ok, so there's a single directory with dpkg-scanpackages and not per-distro directories? [09:24:58] yea, /data/project/repo [09:26:47] I see, yeah makes sense if packages don't diverge much I think [09:27:49] godog: yeah, one directory for everything [09:28:00] godog: oh wait, I could just have different directories and branchi n puppet [09:28:53] and change the sources.list accordingly, indeed, are there many different packages? [09:29:34] !log upgrade diamond in trusty/esams [09:29:41] Logged the message, Master [09:29:56] not yet :) but we have at least one where we ‘fixed’ it by just not installing it on trusty [09:30:52] haha wat [09:31:02] nobody’s complained yet... [09:31:53] !log upgrade diamond in trusty/eqiad [09:31:55] Logged the message, Master [09:32:12] paravoid: did you dig up anything interesting looking at the virt servers yesterday? [09:52:10] andrewbogott: have one more small lint fix for virt1000, just adding {} [09:54:00] PROBLEM - DPKG on db1027 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [09:54:49] ^ iU diamond [09:55:53] and also python-diamong [09:55:55] diamond [09:56:03] PROBLEM - DPKG on db1049 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [09:56:35] godog: ^ looks related to diamond upgrade [09:56:54] RECOVERY - DPKG on db1027 is OK: All packages OK [09:57:01] ah :) nvm [09:59:17] RECOVERY - DPKG on db1049 is OK: All packages OK [10:00:37] mutante: hehe salt and puppet racing [10:00:46] (now imagine that as a gif) [10:00:47] yep :) [10:00:52] haha [10:02:19] puppet forge has a puppet module to "manage a salt installation" [10:25:05] (03PS9) 10Andrew Bogott: Support bootstrap-vz for building labs debian images [puppet] - 10https://gerrit.wikimedia.org/r/179765 [10:34:41] (03PS12) 10Adrian Lang: [WIP] contint: Apply contint::qunit_localhost to labs slaves [puppet] - 10https://gerrit.wikimedia.org/r/168631 (https://bugzilla.wikimedia.org/72063) (owner: 10Krinkle) [10:42:25] (03CR) 10Daniel Kinzler: "This should be enabled, so we start accurately tracking the content model of every page (and every revision). However, changing a page's c" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170129 (https://bugzilla.wikimedia.org/49193) (owner: 10Spage) [10:44:47] (03PS1) 10Gilles: Disable thumbnail chaining [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180138 [10:52:15] (03CR) 10Hashar: "Thanks Alexandros. I address all issues in next patchset. Will then rebase it on top of https://gerrit.wikimedia.org/r/179930 which drops" (037 comments) [puppet] - 10https://gerrit.wikimedia.org/r/178810 (owner: 10Hashar) [10:52:32] (03PS7) 10Hashar: Basic rspec setup [puppet] - 10https://gerrit.wikimedia.org/r/178810 [11:25:35] (03CR) 10Alexandros Kosiaris: [C: 032] "Thanks for the reviews guys!" [puppet] - 10https://gerrit.wikimedia.org/r/179930 (owner: 10Alexandros Kosiaris) [11:30:11] (03CR) 10Dzahn: "could the hardcoded values be moved out of the script and into puppet role? like for example these:" [puppet] - 10https://gerrit.wikimedia.org/r/179765 (owner: 10Andrew Bogott) [11:32:28] (03CR) 10Dzahn: [C: 032] openstack-database-server: enclose variables [puppet] - 10https://gerrit.wikimedia.org/r/179452 (owner: 10Dzahn) [11:33:00] (03CR) 10Alexandros Kosiaris: [C: 032] Beta: Fixed comment [puppet] - 10https://gerrit.wikimedia.org/r/180121 (owner: 10KartikMistry) [11:55:39] (03PS8) 10Hashar: Basic rspec setup [puppet] - 10https://gerrit.wikimedia.org/r/178810 [11:56:58] (03CR) 10Hashar: "Rebased. I have droped /spec/spec_helper.rb since some new spec have been added to modules/wmflibs and that creates required fixtures pat" [puppet] - 10https://gerrit.wikimedia.org/r/178810 (owner: 10Hashar) [11:58:03] (03PS3) 10KartikMistry: WIP: Add cxserver role for production [puppet] - 10https://gerrit.wikimedia.org/r/180125 [12:23:19] (03PS1) 10Yuvipanda: Update ignore regex [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180151 [12:25:58] (03CR) 10Yuvipanda: [C: 032 V: 032] Update ignore regex [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/180151 (owner: 10Yuvipanda) [12:27:23] (03PS4) 10KartikMistry: Add cxserver role for production [puppet] - 10https://gerrit.wikimedia.org/r/180125 [12:31:42] (03PS5) 10KartikMistry: Add cxserver role for production [puppet] - 10https://gerrit.wikimedia.org/r/180125 [12:41:35] (03PS2) 10KartikMistry: Add apertium role for production [puppet] - 10https://gerrit.wikimedia.org/r/180130 [12:48:20] (03PS2) 10KartikMistry: Beta: Fix spacing and indentation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178772 [13:03:52] (03CR) 10Andrew Bogott: "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/179452 (owner: 10Dzahn) [13:04:56] (03PS1) 10Giuseppe Lavagetto: mediawiki: furl support for passing headers [puppet] - 10https://gerrit.wikimedia.org/r/180155 [13:07:38] (03PS1) 10Hashar: Refactor rake entry point for specs [puppet] - 10https://gerrit.wikimedia.org/r/180162 [13:14:51] PROBLEM - mailman list info on sodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:06] PROBLEM - puppet last run on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:07] PROBLEM - HTTP on sodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:29] PROBLEM - salt-minion processes on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:51] PROBLEM - HTTPS on sodium is CRITICAL: SSL_CERT CRITICAL: Error: verify depth is 6 [13:17:00] PROBLEM - mailman archives on sodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:13] PROBLEM - dhclient process on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:19:52] * _joe_ lunch [13:20:42]