[00:04:02] (03CR) 10Springle: [C: 04-1] Labs: puppetize labstore1005's mysql setup (036 comments) [puppet] - 10https://gerrit.wikimedia.org/r/200170 (https://phabricator.wikimedia.org/T88234) (owner: 10coren) [00:12:11] (03CR) 10Yuvipanda: Labs: puppetize labstore1005's mysql setup (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/200170 (https://phabricator.wikimedia.org/T88234) (owner: 10coren) [00:13:04] (03CR) 10coren: "Comments + questions inline" (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/200170 (https://phabricator.wikimedia.org/T88234) (owner: 10coren) [00:36:07] (03CR) 10Springle: Labs: puppetize labstore1005's mysql setup (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/200170 (https://phabricator.wikimedia.org/T88234) (owner: 10coren) [01:11:04] (03PS1) 10Springle: depool db1044 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200503 [01:12:36] (03CR) 10Springle: [C: 032] depool db1044 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200503 (owner: 10Springle) [01:14:16] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [01:14:27] !log springle Synchronized wmf-config/db-eqiad.php: depool db1044 (duration: 00m 09s) [01:14:36] Logged the message, Master [02:16:26] PROBLEM - puppet last run on mw2144 is CRITICAL: CRITICAL: Puppet has 1 failures [02:24:51] !log l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 05m 06s) [02:25:01] Logged the message, Master [02:28:21] !log LocalisationUpdate completed (1.25wmf22) at 2015-03-30 02:27:18+00:00 [02:28:25] Logged the message, Master [02:32:56] RECOVERY - puppet last run on mw2144 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [02:40:18] !log l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 03m 52s) [02:40:26] Logged the message, Master [02:43:06] !log LocalisationUpdate completed (1.25wmf23) at 2015-03-30 02:42:03+00:00 [02:43:10] Logged the message, Master [02:46:18] PROBLEM - puppet last run on labstore2001 is CRITICAL: CRITICAL: puppet fail [03:02:48] RECOVERY - puppet last run on labstore2001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:53:00] !log upgrade db1044 trusty [04:53:31] morebots: ? [04:55:36] morebots, help [04:55:53] it's trying to log [04:56:00] but failing. wikitech thing? [04:56:59] how do you know it's trying? [04:57:24] 2015-03-30 04:53:00,847 DEBUG: 'production-logbot' got '!log upgrade db1044 trusty'; Attempting to log. [04:57:41] is that from its log? [04:57:58] right, from tools-login [04:58:19] oh, I guess since you are ops you can just become whoever you like? [04:58:30] idk how to make it tell me more though [04:59:29] legoktm, ori: ^ [05:00:15] hi [05:00:21] I don't really know how that works :/ [05:00:29] !log restarted production logbot [05:00:35] * springle hopes... [05:02:27] PROBLEM - puppetmaster https on virt1000 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:04:07] morebots needs to die, etc [05:06:23] it dies perfectly well :) [05:07:01] about all it does atm. idk enough about it to debug right now [05:08:15] although, since i'm not able to login to wikitech to edit the SAL either... i guess it really is a wikitech issue [05:09:13] I can take a look [05:09:34] YuviPanda: awesome thanks [05:10:37] RECOVERY - puppetmaster https on virt1000 is OK: HTTP OK: Status line output matched 400 - 335 bytes in 6.892 second response time [05:11:31] aaaah [05:11:34] wikitech login is fucked [05:11:50] I thought we already established that :/ [05:11:58] I hadn’t read backscroll [05:11:59] sorry [05:12:05] !log restarted keystone on virt1000 [05:12:15] springle: if you’re still on tools-login can you kick morebots again? [05:12:44] hmm [05:12:51] wikitech login might be still fucked [05:13:53] YuviPanda: i am still logged on there, but yes, wiktech login still afu for me [05:13:59] yeah... [05:14:55] !log restarted apache on silver [05:15:05] springle: try again? wikitech just worked for me after ^ [05:15:07] Logged the message, Master [05:15:25] aha! [05:15:30] Krenair: springle ^ it is baaack [05:15:35] :) [05:15:39] restart apache [05:15:40] ok [05:15:44] Will remember that for next time :) [05:15:51] Krenair: usually it is just to restart keystone [05:15:59] Krenair: but apache is my next culprit... [05:16:00] right, I can't do that [05:16:02] Krenair: and then ldap. [05:16:02] yeah [05:16:07] only apache, I think [05:16:11] Krenair: oh, right [05:16:16] Krenair: you can on silver [05:16:54] I have emailed andrew [05:17:14] !log restarted production logbot [05:17:18] Logged the message, Master [05:17:23] \o/ [05:17:30] !log upgrade db1044 trusty [05:17:33] Logged the message, Master [05:18:11] At least I thought I probably could restart apache somewhere. [05:18:16] I haven't ever tried it. [05:18:24] yeah, silver’s apache... [05:18:31] I have no idea *why* that worked, mind you [05:18:46] it’s sunday night and I’m going to sleep soon... [05:19:04] YuviPanda: thanks for that [05:20:00] (03PS1) 10Springle: repool db1044 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200509 [05:20:27] (03CR) 10Springle: [C: 032] repool db1044 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200509 (owner: 10Springle) [05:20:32] (03Merged) 10jenkins-bot: repool db1044 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200509 (owner: 10Springle) [05:21:20] !log springle Synchronized wmf-config/db-eqiad.php: repool db1044, warm up (duration: 00m 07s) [05:21:26] Logged the message, Master [05:21:37] springle: congrats on the worldcup victory, if you’re into that :) [05:21:59] :D [05:22:25] YuviPanda: i keep the games on in the background [05:22:35] ah, I know what it was I could do [05:22:56] springle: :) I do too, but didn’t do for the finals. [05:23:08] too depressing? :P [05:23:20] so I'm having all connections to our sites fail... so far every other site is working [05:23:35] springle: I was moving continents :) [05:23:46] jamesofur: hi. where are you connecting from? [05:23:55] Comcast, my apartment [05:23:59] so Comcast San Francisco [05:24:08] sucks for me too on comcast [05:24:13] just notices that a minute ago [05:24:19] yeah, that's about the time [05:24:25] jamesofur: aspiecat how about GMail? [05:24:34] I’m on comcast sf too and it sucks for me as well, and gmail is dead too... [05:24:37] * YuviPanda digs [05:24:41] gmail works fine [05:24:43] gmail/google no workie [05:25:02] almost nothing works, except, like IRC :) [05:25:10] yeah [05:25:12] for me too [05:25:21] aspiecat: jamesofur can you do ‘ping en.wikipedia.org’? [05:25:28] doing [05:25:47] just timing out [05:25:53] same [05:25:54] PING en.wikipedia.org (198.35.26.96): 56 data bytes [05:26:06] so it found the dns [05:26:07] No issue for me, UK, TalkTalk [05:26:16] working from down under [05:26:16] DNS fail [05:26:25] I'm on google DNS [05:26:40] jamesofur: yeah, for me too. [05:26:41] what is google DNS IP? [05:26:48] * aspiecat can't google that atm ;) [05:26:48] That IP is text-lb.ulsfo [05:26:49] this sounds like something fucked up on comcast SF…. [05:26:50] 8.8.8.8 and 8.8.8.4 [05:27:01] plus some IPv6 ones I can give you [05:27:04] COMCAAAAAAAAST [05:27:34] (the one jamesofur gave - 198.35.26.96) [05:27:46] You're ruining my Sunday evening productivity burst, Comcast [05:27:54] yep comcast just died on me [05:28:00] AT&T is working fine though [05:28:38] phab works... [05:28:53] traceroute text-lb.ulsfo, text-lb.eqiad perhaps? [05:29:12] it's all back for me now it looks like [05:29:23] yup ping working again too (it was still going) [05:29:25] Me too. [05:29:28] yup [05:29:49] what's going on? [05:29:52] though the ping is alternating between around 10 and just over 200 which is od [05:29:53] odd [05:30:02] paravoid: comcast SF decided to hate us [05:30:13] though appears to be mostly back now.... [05:32:08] was it wikimedia-specific? [05:32:30] paravoid: comcast SF basically died for a bit (google, parts of facebook, some parts of us (to ulsfo maybe? phabricator worked), etc) [05:32:31] paravoid: nope [05:32:37] * YuviPanda is on comcast sf now [05:32:50] ok :) [05:33:07] (03PS1) 10Springle: depool db1035 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200510 [05:33:10] paravoid: were you already awake? :) [05:33:26] for some reason for me it was wikimedia specific, but it may have been because i used google DNS (and DNS was failing for some others, my DNS worked but not the connection) [05:33:38] (03CR) 10Springle: [C: 032] depool db1035 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200510 (owner: 10Springle) [05:33:42] (03Merged) 10jenkins-bot: depool db1035 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200510 (owner: 10Springle) [05:33:50] jamesofur: hangouts was the first tihng I noticed going down [05:33:53] you got a valid IP, jamesofur [05:34:02] right, but someone else said they didn't [05:34:10] which makes me wonder if that was part of it [05:34:15] yeah, aspiecat didn't [05:34:24] well it could’ve been comcast DNS going too? [05:34:32] yeah [05:34:44] !log springle Synchronized wmf-config/db-eqiad.php: depool db1035 (duration: 00m 07s) [05:34:46] but at the same time as some of their routing (I didn't hit the DNS crash but I did hit the routing) [05:34:49] * aspiecat was using comcast dns before [05:34:50] Logged the message, Master [05:35:07] they had a DNS crap shoot a couple days ago which is why I'm on google atm [05:35:19] (I'm not usually because I don't want to be on it at the office) [05:53:25] !log upgrade db1035 trusty [05:53:28] Logged the message, Master [06:07:32] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Mar 30 06:06:26 UTC 2015 (duration 6m 25s) [06:07:37] Logged the message, Master [06:19:31] (03Abandoned) 10Giuseppe Lavagetto: proxies: order the list by IP distance [tools/scap] - 10https://gerrit.wikimedia.org/r/200137 (owner: 10Giuseppe Lavagetto) [06:19:52] (03PS3) 10Giuseppe Lavagetto: mediawiki: install fonts metric-compatible with Calibri and Cambria [puppet] - 10https://gerrit.wikimedia.org/r/196173 (https://phabricator.wikimedia.org/T84842) [06:20:07] PROBLEM - Host db1035 is DOWN: PING CRITICAL - Packet loss = 100% [06:20:39] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki: install fonts metric-compatible with Calibri and Cambria [puppet] - 10https://gerrit.wikimedia.org/r/196173 (https://phabricator.wikimedia.org/T84842) (owner: 10Giuseppe Lavagetto) [06:21:01] <_joe_> is jenkins dead again? [06:21:18] (03CR) 10Giuseppe Lavagetto: [V: 032] mediawiki: install fonts metric-compatible with Calibri and Cambria [puppet] - 10https://gerrit.wikimedia.org/r/196173 (https://phabricator.wikimedia.org/T84842) (owner: 10Giuseppe Lavagetto) [06:25:14] !log db1035 restart failed, root fs errors [06:25:19] Logged the message, Master [06:25:21] sadface.gif [06:27:29] ACKNOWLEDGEMENT - Host db1035 is DOWN: PING CRITICAL - Packet loss = 100% Sean Pringle wont restart, root fs errors [06:30:08] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: puppet fail [06:30:17] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:57] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:18] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:19] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 3 failures [06:33:36] <_joe_> springle: ouch [06:33:58] PROBLEM - puppet last run on mw2097 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:17] PROBLEM - puppet last run on mw2123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:47] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:47] PROBLEM - puppet last run on mw2206 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:57] PROBLEM - puppet last run on mw2096 is CRITICAL: CRITICAL: Puppet has 2 failures [06:35:58] PROBLEM - puppet last run on mw2045 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:58] PROBLEM - puppet last run on mw2022 is CRITICAL: CRITICAL: Puppet has 2 failures [06:46:08] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:46:17] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:46:27] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:46:38] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:47:18] RECOVERY - puppet last run on mw2097 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:47:27] RECOVERY - puppet last run on mw2206 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:47:27] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:37] RECOVERY - puppet last run on mw2123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:37] RECOVERY - puppet last run on mw2096 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:37] RECOVERY - puppet last run on mw2045 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:47:37] RECOVERY - puppet last run on mw2022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:33:37] 6operations, 10Wikimedia-SVG-rendering, 7Upstream: Filter effect Gaussian blur filter not rendered correctly for small to medium thumbnail sizes - https://phabricator.wikimedia.org/T44090#1161511 (10Aklapper) The upstream fix is in librsvg version 2.40.9 (which first needs to get packaged and shipped by the... [08:40:10] 6operations, 10Wikimedia-SVG-rendering, 7Upstream: Filter effect Gaussian blur filter not rendered correctly for small to medium thumbnail sizes - https://phabricator.wikimedia.org/T44090#1161517 (10Krenair) >>! In T44090#1161511, @Aklapper wrote: > @Krenair: Whoever created that board, "Patch merged" etc is... [08:41:06] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 9 below the confidence bounds [08:59:53] (03PS3) 10Nemo bis: Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) [08:59:57] (03PS2) 10Filippo Giunchedi: scap: improve deploy2graphite [puppet] - 10https://gerrit.wikimedia.org/r/199857 (https://phabricator.wikimedia.org/T1387) [09:00:03] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] scap: improve deploy2graphite [puppet] - 10https://gerrit.wikimedia.org/r/199857 (https://phabricator.wikimedia.org/T1387) (owner: 10Filippo Giunchedi) [09:04:03] godog: are you around ? [09:06:04] I'll just raise my general concern here: All labs admins are now based in US, so the support is effctivily only in US times. This is not so bad, as it is only labs, but it means when we have issues, some people need to wait up to a day [09:06:24] (03PS1) 10Giuseppe Lavagetto: 3.6.1: fix compilation errors [debs/hhvm] - 10https://gerrit.wikimedia.org/r/200513 [09:07:59] I would like to hear some input on this, if anyone sees it as a problem. i.e andrewbogott_afk akosiaris paravoid Coren and any other that would like to speak :) [09:08:09] matanya: I guess depends on the issue, but I see your point, did you see this already? [09:08:24] yes, several times [09:08:41] namely the recent outages [09:09:10] and the daily lighthttpd issues for various tools [09:09:39] (03CR) 10Florianschmidtwelzow: [C: 031] "technically looks good" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [09:10:14] I can send a mail to ops if you think it justifies. [09:10:18] 7Puppet, 6operations, 5Patch-For-Review, 7Swift: puppet failure "invalid byte sequence in utf-8" while copying swift ring builder files - https://phabricator.wikimedia.org/T93614#1161567 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi [09:13:04] matanya: heh, yeah the list would be better than here [09:13:20] thanks [09:18:01] matanya: what do you suggest instead? [09:18:44] paravoid: or spread the admins, or delegate simple stuff to a tier 2 users [09:19:08] (03PS4) 10Nemo bis: Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) [09:19:27] spread them how? force them to move? :) [09:19:53] no :) make a few EU based Labs admins as well [09:20:15] or asian, for the sake of diversity [09:20:51] volunteer admins you mean? [09:20:58] if you may [09:21:12] i wouldn't care if it was WMF ops either [09:21:22] (03CR) 10Nemo bis: "1.25wmf24 has not been branched yet, so PS3 still gives time for further changes to be deployed at the same time if needed. (But none is k" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [09:21:53] throughing a name in the air: apergos [09:22:31] great timing for an example: https://tools.wmflabs.org/xtools/adminstats/?project=he.wikipedia.org [09:22:58] lighthttpd is down. [09:27:54] 6operations, 10Wikimedia-Git-or-Gerrit, 7Monitoring: Improve monitoring of https://git.wikimedia.org/ - https://phabricator.wikimedia.org/T94320#1161603 (10faidon) p:5Triage>3Lowest git.wm.org is known to be broken, see T73974. Monitoring wouldn't help us all that much... [09:31:59] 10Ops-Access-Requests, 6operations: Requesting access to analytics-privatedata-users for Lydia Pintscher - https://phabricator.wikimedia.org/T94390#1161609 (10Lydia_Pintscher) 3NEW [09:34:09] (03CR) 1020after4: "@Legoktm: what is the official source of truth for deployed extensions, and why do we (presumably) have more than one?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 (owner: 10Chad) [09:37:54] (03CR) 10Nemo bis: "The make-wmf-branch script, i.e. its default.conf file, is presumably what must necessarily be up to date or the code wouldn't reach the s" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 (owner: 10Chad) [09:42:31] 7Blocked-on-Operations, 6operations, 10Continuous-Integration, 6Release-Engineering, 6Scrum-of-Scrums: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#1161649 (10fgiunchedi) agreed if that's a regression we should automatically check whethe... [09:59:24] 6operations, 6Phabricator: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#1161692 (10fgiunchedi) today a new access request came in at https://phabricator.wikimedia.org/T94390 however I don't see the related blocking task i... [09:59:32] 6operations, 6Phabricator: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#1161693 (10fgiunchedi) 5Resolved>3Open [10:00:19] 6operations, 6Phabricator: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#991959 (10fgiunchedi) FTR I came here from https://wikitech.wikimedia.org/wiki/Ops_Clinic_Duty#Access_requests where the whole process is outlined on... [10:04:03] 6operations, 6Phabricator: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#1161698 (10mmodell) @fgiunchedi: I'm not sure why the menu entry for "ops access request" isn't configured. The code to support it has been merged fo... [10:07:30] (03CR) 1020after4: "@Nemo bis: good point, though that is (in my opinion) not the ideal place to keep the canonical list." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 (owner: 10Chad) [10:09:35] 10Ops-Access-Requests, 6operations: Requesting access to analytics-privatedata-users for Lydia Pintscher - https://phabricator.wikimedia.org/T94390#1161702 (10fgiunchedi) p:5Triage>3Normal hi, we'd need approvals for access from your manager and analytics, there's also three days grace time for this before... [10:09:56] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [10:10:46] (03CR) 10Hashar: [C: 04-1] "Thank to have taken in consideration my earlier remarks. Supporting DIST=jessie-wikimedia looks very user friendly." (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/194471 (owner: 10Alexandros Kosiaris) [10:18:02] 6operations, 10RESTBase, 7Monitoring, 5Patch-For-Review: Detailed cassandra monitoring: metrics and dashboards done, need to set up alerts - https://phabricator.wikimedia.org/T78514#1161717 (10fgiunchedi) [10:18:03] 6operations, 10RESTBase, 10RESTBase-Cassandra, 5Patch-For-Review: Cassandra/CQL query interface monitoring - https://phabricator.wikimedia.org/T93886#1161714 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi thanks @dzahn ! I'm tentatively closing this as resolved, we can revisit the deeper cql query chec... [10:24:52] (03PS1) 10Faidon Liambotis: Allocate IPv6 for dns-rec-lb.eqiad (AAAA/PTR) [dns] - 10https://gerrit.wikimedia.org/r/200519 [10:24:54] (03PS1) 10Faidon Liambotis: Allocate IPv4/IPv6 for dns-rec-lb.esams [dns] - 10https://gerrit.wikimedia.org/r/200520 [10:25:01] (03PS1) 10Faidon Liambotis: lvs: fix IP for codfw's dns_rec6 [puppet] - 10https://gerrit.wikimedia.org/r/200521 [10:25:03] (03PS1) 10Faidon Liambotis: lvs: fix IP for eqiad's dns_rec6 [puppet] - 10https://gerrit.wikimedia.org/r/200522 [10:25:05] (03PS1) 10Faidon Liambotis: lvs: add dns_rec/dns_rec6 for esams [puppet] - 10https://gerrit.wikimedia.org/r/200523 [10:30:31] (03CR) 10Faidon Liambotis: [C: 032] lvs: fix IP for codfw's dns_rec6 [puppet] - 10https://gerrit.wikimedia.org/r/200521 (owner: 10Faidon Liambotis) [10:31:46] (03PS1) 10Filippo Giunchedi: icinga: check mailman shunt queue too [puppet] - 10https://gerrit.wikimedia.org/r/200524 (https://phabricator.wikimedia.org/T93783) [10:33:28] 6operations, 10MediaWiki-API: mw1135 has errors, depooled - https://phabricator.wikimedia.org/T93626#1161787 (10Joe) 5Open>3Resolved [10:34:15] 6operations, 10ops-codfw, 3wikis-in-codfw: PXE doesn't work on mc2017-18 - https://phabricator.wikimedia.org/T90586#1161795 (10Joe) a:5Joe>3RobH [10:34:42] 6operations, 10ops-codfw, 3wikis-in-codfw: PXE doesn't work on mc2017-18 - https://phabricator.wikimedia.org/T90586#1062590 (10Joe) Reassigning to @RobH given I still can't install those two systems [10:43:03] (03PS2) 10Faidon Liambotis: lvs: add dns_rec/dns_rec6 for esams [puppet] - 10https://gerrit.wikimedia.org/r/200523 [10:43:05] (03PS2) 10Faidon Liambotis: lvs: fix IP for eqiad's dns_rec6 [puppet] - 10https://gerrit.wikimedia.org/r/200522 [10:43:14] (03PS5) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [10:44:32] (03CR) 10Faidon Liambotis: [C: 032] lvs: fix IP for eqiad's dns_rec6 [puppet] - 10https://gerrit.wikimedia.org/r/200522 (owner: 10Faidon Liambotis) [10:46:39] (03PS19) 10Alexandros Kosiaris: Package builder module [puppet] - 10https://gerrit.wikimedia.org/r/194471 [10:46:41] (03PS1) 10Alexandros Kosiaris: role::package::builder uses package_builder module [puppet] - 10https://gerrit.wikimedia.org/r/200525 [10:48:36] (03CR) 10Giuseppe Lavagetto: "This patch in its current format fixes a few bugs, but changes we're going to create should be inspected properly." [puppet] - 10https://gerrit.wikimedia.org/r/197533 (owner: 10Chad) [10:50:49] (03PS2) 10Faidon Liambotis: Allocate IPv4/IPv6 for dns-rec-lb.esams [dns] - 10https://gerrit.wikimedia.org/r/200520 [10:53:47] (03CR) 10Faidon Liambotis: [C: 032] Allocate IPv6 for dns-rec-lb.eqiad (AAAA/PTR) [dns] - 10https://gerrit.wikimedia.org/r/200519 (owner: 10Faidon Liambotis) [10:56:51] 6operations: flamegraph (xenon) is using most of fluorine's memory - https://phabricator.wikimedia.org/T94396#1161879 (10fgiunchedi) 3NEW [10:57:07] PROBLEM - puppet last run on rdb2003 is CRITICAL: CRITICAL: puppet fail [11:01:57] (03PS3) 10Faidon Liambotis: Allocate IPv4/IPv6 for dns-rec-lb.esams [dns] - 10https://gerrit.wikimedia.org/r/200520 [11:02:29] (03PS3) 10Faidon Liambotis: lvs: add dns_rec/dns_rec6 for esams [puppet] - 10https://gerrit.wikimedia.org/r/200523 [11:02:31] (03PS1) 10Faidon Liambotis: lvs: configure dns_rec for esams [puppet] - 10https://gerrit.wikimedia.org/r/200527 [11:03:31] (03CR) 10Alexandros Kosiaris: [C: 032] Fix two people's real names in the admin data [puppet] - 10https://gerrit.wikimedia.org/r/200495 (owner: 10Alex Monk) [11:03:58] (03PS6) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [11:08:35] <_joe_> mmm FFS puppet [11:10:08] (03PS1) 10Faidon Liambotis: Add role dnsrecursor to maerlant [puppet] - 10https://gerrit.wikimedia.org/r/200528 [11:10:10] (03PS1) 10Faidon Liambotis: Make maerlant an (esams) NTP server [puppet] - 10https://gerrit.wikimedia.org/r/200529 [11:10:17] (03PS1) 10Faidon Liambotis: Switch recursor1 to esams' new LVS dns-rec-lb IP [dns] - 10https://gerrit.wikimedia.org/r/200530 [11:13:58] RECOVERY - puppet last run on rdb2003 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [11:16:34] (03PS7) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [11:18:32] (03PS3) 10Glaisher: Add 'autopatrol' protection level to lvwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/196779 (https://phabricator.wikimedia.org/T92645) [11:19:10] (03CR) 10Glaisher: "issue resolved. good to go" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/196779 (https://phabricator.wikimedia.org/T92645) (owner: 10Glaisher) [11:22:29] (03CR) 10Alexandros Kosiaris: [C: 04-1] "So, you are killing the id element in _new and use the ip_oct all around. I am fine with that, just making sure. Also minor pedantic comme" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) (owner: 10Giuseppe Lavagetto) [11:23:36] 6operations, 6MediaWiki-Core-Team, 5Patch-For-Review: Store unsampled API and XFF logs - https://phabricator.wikimedia.org/T88393#1161994 (10fgiunchedi) also note that fluorine has another 300+GB free in the vg ``` root@fluorine:/a/mw-log/archive# vgs VG #PV #LV #SN Attr VSize VFree vg0 2 1... [11:23:40] <_joe_> akosiaris: I was already reverting that tbh [11:23:47] (03PS8) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [11:23:55] <_joe_> akosiaris: ^^ :) [11:24:47] <_joe_> (I realized "id" is more appropriate that "ip_oct" [11:24:58] cassandra module update in PS8 ? [11:25:18] yeah it is. But even more appropriate is to only use one and not both [11:26:14] 6operations, 6Labs, 7Monitoring, 5Patch-For-Review: Setup alarms for labstore* to check for network saturation - https://phabricator.wikimedia.org/T92629#1162002 (10fgiunchedi) a:3coren [11:26:44] <_joe_> oh shit, I hate submodules [11:27:28] (03PS1) 10KartikMistry: Enable ContentTranslation in bg, fr, mk, sl and sv [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200531 [11:27:37] (03CR) 10Alexandros Kosiaris: Ganeti module/role introduced (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/198794 (https://phabricator.wikimedia.org/T87258) (owner: 10Alexandros Kosiaris) [11:28:01] <_joe_> akosiaris: btw, I'm gonna take a good look at the ganeti module later today [11:30:43] (03CR) 10Alexandros Kosiaris: Ganeti module/role introduced (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/198794 (https://phabricator.wikimedia.org/T87258) (owner: 10Alexandros Kosiaris) [11:30:55] _joe_: ok, thanks [11:31:11] <_joe_> and btw, big surprise - the lists of ganglia clusters were not consistent in ganglia and ganglia_new [11:31:18] <_joe_> fixing it :/ [11:36:43] (03PS2) 10KartikMistry: Enable ContentTranslation in bg, fr, mk, sh and sl [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200531 [11:36:50] I had them syced a couple of months ago [11:36:53] synced [11:36:58] not surprised tbh [11:37:04] speaking of _new, godog, any plans for swift_new? :) [11:38:31] (03PS1) 10KartikMistry: CX: Enable ContentTranslation for bg, fr, mk, sh, sl pairs [puppet] - 10https://gerrit.wikimedia.org/r/200533 [11:38:42] (03PS2) 10Faidon Liambotis: Switch recursor1 to esams' new LVS dns-rec-lb IP [dns] - 10https://gerrit.wikimedia.org/r/200530 [11:40:05] (03PS3) 10Faidon Liambotis: Switch recursor1 to esams' new LVS dns-rec-lb IP [dns] - 10https://gerrit.wikimedia.org/r/200530 [11:40:16] paravoid: not in the short term, no [11:40:29] :( [11:40:43] it's one of the very very few last manifests/ [11:41:00] I don't think we should ever do _new again [11:41:11] both of ganglia & swift were failures in that regard [11:43:53] (03CR) 10Faidon Liambotis: [C: 032] Allocate IPv4/IPv6 for dns-rec-lb.esams [dns] - 10https://gerrit.wikimedia.org/r/200520 (owner: 10Faidon Liambotis) [11:45:45] I agree, that's usually due to time constraints [11:49:13] (03PS4) 10Faidon Liambotis: lvs: add dns_rec/dns_rec6 for esams [puppet] - 10https://gerrit.wikimedia.org/r/200523 [11:49:15] (03PS2) 10Faidon Liambotis: lvs: configure dns_rec for esams [puppet] - 10https://gerrit.wikimedia.org/r/200527 [11:49:17] (03PS2) 10Faidon Liambotis: Add role dnsrecursor to maerlant [puppet] - 10https://gerrit.wikimedia.org/r/200528 [11:50:15] (03CR) 10Faidon Liambotis: [C: 032] lvs: add dns_rec/dns_rec6 for esams [puppet] - 10https://gerrit.wikimedia.org/r/200523 (owner: 10Faidon Liambotis) [11:50:20] (03CR) 10Faidon Liambotis: [C: 032] lvs: configure dns_rec for esams [puppet] - 10https://gerrit.wikimedia.org/r/200527 (owner: 10Faidon Liambotis) [11:55:42] (03PS1) 10Faidon Liambotis: lvs: split the config for high-traffic2 & esams/ulsfo [puppet] - 10https://gerrit.wikimedia.org/r/200535 [11:55:47] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: puppet fail [11:56:04] (03CR) 10Faidon Liambotis: [C: 032] lvs: split the config for high-traffic2 & esams/ulsfo [puppet] - 10https://gerrit.wikimedia.org/r/200535 (owner: 10Faidon Liambotis) [11:58:39] (03PS9) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [11:59:07] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:59:52] (03PS3) 10Faidon Liambotis: Add role dnsrecursor to maerlant [puppet] - 10https://gerrit.wikimedia.org/r/200528 [12:00:08] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Add role dnsrecursor to maerlant [puppet] - 10https://gerrit.wikimedia.org/r/200528 (owner: 10Faidon Liambotis) [12:00:46] (03CR) 10KartikMistry: [C: 04-1] "Not to merge before, https://gerrit.wikimedia.org/r/200531" [puppet] - 10https://gerrit.wikimedia.org/r/200533 (owner: 10KartikMistry) [12:01:24] (03CR) 10Giuseppe Lavagetto: ganglia: DRY, use hiera (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) (owner: 10Giuseppe Lavagetto) [12:01:26] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet has 1 failures [12:02:19] (03PS2) 10Faidon Liambotis: Make maerlant an (esams) NTP server [puppet] - 10https://gerrit.wikimedia.org/r/200529 [12:02:25] (03CR) 10Faidon Liambotis: [C: 032] Make maerlant an (esams) NTP server [puppet] - 10https://gerrit.wikimedia.org/r/200529 (owner: 10Faidon Liambotis) [12:04:39] (03PS3) 10Giuseppe Lavagetto: ganglia: remove unused configs from ganglia::collector::config [puppet] - 10https://gerrit.wikimedia.org/r/198720 (https://phabricator.wikimedia.org/T93776) [12:10:20] (03PS1) 10Faidon Liambotis: realm/autoinstall: switch esams' recursor to dns-rec-lb [puppet] - 10https://gerrit.wikimedia.org/r/200537 [12:11:03] (03CR) 10Alexandros Kosiaris: "We had a recent discussion about the configuration file for parsoid in the puppet repo (T92636). I am thinking we should follow this appro" [puppet] - 10https://gerrit.wikimedia.org/r/200356 (owner: 10Mobrovac) [12:11:16] (03CR) 10Faidon Liambotis: [C: 032] realm/autoinstall: switch esams' recursor to dns-rec-lb [puppet] - 10https://gerrit.wikimedia.org/r/200537 (owner: 10Faidon Liambotis) [12:18:53] (03PS1) 10Faidon Liambotis: ntp.esams -> maerlant [dns] - 10https://gerrit.wikimedia.org/r/200538 [12:19:02] (03CR) 10jenkins-bot: [V: 04-1] ntp.esams -> maerlant [dns] - 10https://gerrit.wikimedia.org/r/200538 (owner: 10Faidon Liambotis) [12:19:46] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:21:10] (03PS2) 10Faidon Liambotis: ntp.esams -> maerlant [dns] - 10https://gerrit.wikimedia.org/r/200538 [12:22:40] 10Ops-Access-Requests, 6operations: Requesting access to analytics-privatedata-users for Lydia Pintscher - https://phabricator.wikimedia.org/T94390#1162073 (10Lydia_Pintscher) @Abraham: Please ok :) [12:23:45] (03PS1) 10KartikMistry: Beta: Add missing mk-sh pair [puppet] - 10https://gerrit.wikimedia.org/r/200540 [12:24:34] 10Ops-Access-Requests, 6operations: Requesting access to analytics-privatedata-users for Lydia Pintscher - https://phabricator.wikimedia.org/T94390#1162077 (10Lydia_Pintscher) Acknowledgement of Wikimedia Server Access Responsibilities signed. [12:30:01] (03CR) 10KartikMistry: "This can be merged." [puppet] - 10https://gerrit.wikimedia.org/r/200540 (owner: 10KartikMistry) [12:31:45] (03CR) 10Alexandros Kosiaris: [C: 032] Beta: Add missing mk-sh pair [puppet] - 10https://gerrit.wikimedia.org/r/200540 (owner: 10KartikMistry) [12:32:13] (03CR) 10Mobrovac: "This means we would need to put zotero's svc address and port directly in the config file located in the deploy repo, which I don't it's o" [puppet] - 10https://gerrit.wikimedia.org/r/200356 (owner: 10Mobrovac) [12:35:00] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [12:40:02] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:41:22] (03PS1) 10Faidon Liambotis: Add forward DNS for schleifenbauer (esams) [dns] - 10https://gerrit.wikimedia.org/r/200550 [12:41:45] (03CR) 10Faidon Liambotis: [C: 032] Switch recursor1 to esams' new LVS dns-rec-lb IP [dns] - 10https://gerrit.wikimedia.org/r/200530 (owner: 10Faidon Liambotis) [12:41:58] (03CR) 10Faidon Liambotis: [C: 032] ntp.esams -> maerlant [dns] - 10https://gerrit.wikimedia.org/r/200538 (owner: 10Faidon Liambotis) [12:42:18] (03CR) 10Faidon Liambotis: [C: 032] Add forward DNS for schleifenbauer (esams) [dns] - 10https://gerrit.wikimedia.org/r/200550 (owner: 10Faidon Liambotis) [12:50:44] 7Blocked-on-Operations, 6operations, 10Continuous-Integration, 6Scrum-of-Scrums: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1162172 (10akosiaris) A preliminary update on this. php-luasandbox > 2.0 r... [12:56:39] 6operations: boron passive checks aren't being collected - https://phabricator.wikimedia.org/T89983#1162185 (10Jgreen) Frack has an internal repo on boron, using reprepro. I guess we build a package and host it there. [13:12:31] (03PS2) 10coren: Labs: puppetize labstore1005's mysql setup [puppet] - 10https://gerrit.wikimedia.org/r/200170 (https://phabricator.wikimedia.org/T88234) [13:12:36] (03PS10) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [13:18:48] (03PS11) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [13:23:31] <_joe_> I keep finding small inconsistencies :) [13:25:26] (03PS1) 10BBlack: give maerlant some NTP upstreams [puppet] - 10https://gerrit.wikimedia.org/r/200563 [13:27:20] (03CR) 10BBlack: [C: 032] give maerlant some NTP upstreams [puppet] - 10https://gerrit.wikimedia.org/r/200563 (owner: 10BBlack) [13:33:39] Jeff_Green: around ? were going to get https://phabricator.wikimedia.org/T92877 finished off today :) [13:38:26] meh. looks like he's not yet around [13:41:43] 10Ops-Access-Requests, 6operations: Requesting access to analytics-privatedata-users for Lydia Pintscher - https://phabricator.wikimedia.org/T94390#1162339 (10Halfak) Hi. I've been helping Lydia through this process. I've confirmed with Manprit Brar that Lydia has signed an NDA. Lydia will need to be adde... [13:43:06] 10Ops-Access-Requests, 6operations: Requesting access to analytics-privatedata-users for Lydia Pintscher - https://phabricator.wikimedia.org/T94390#1162347 (10Halfak) [13:57:57] (03PS5) 10Alexandros Kosiaris: Ganeti module/role introduced [puppet] - 10https://gerrit.wikimedia.org/r/198794 (https://phabricator.wikimedia.org/T87258) [14:03:09] \o/ [14:04:17] !log reload apache on iodine [14:04:25] Logged the message, Master [14:18:27] (03CR) 10coren: "Comments inline." (0311 comments) [puppet] - 10https://gerrit.wikimedia.org/r/199267 (https://phabricator.wikimedia.org/T85606) (owner: 10coren) [14:18:47] (03PS3) 10coren: WIP: Proper labs_storage class [puppet] - 10https://gerrit.wikimedia.org/r/199267 (https://phabricator.wikimedia.org/T85606) [14:18:48] 7Blocked-on-Operations, 6operations, 10Continuous-Integration, 3Continuous-Integration-Isolation, and 2 others: Create a Debian package for Zuul - https://phabricator.wikimedia.org/T48552#489927 (10hashar) [14:19:40] Jeff_Green: thoughts on https://phabricator.wikimedia.org/T66795 ? [14:19:48] (03PS1) 10Alexandros Kosiaris: Ganeti eqiad cluster DNS and Service records [dns] - 10https://gerrit.wikimedia.org/r/200573 [14:24:25] (03PS12) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) [14:24:41] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 (https://phabricator.wikimedia.org/T93776) (owner: 10Giuseppe Lavagetto) [14:31:04] (03PS4) 10Giuseppe Lavagetto: ganglia: remove unused configs from ganglia::collector::config [puppet] - 10https://gerrit.wikimedia.org/r/198720 (https://phabricator.wikimedia.org/T93776) [14:33:05] (03CR) 10Giuseppe Lavagetto: [C: 032] ganglia: remove unused configs from ganglia::collector::config [puppet] - 10https://gerrit.wikimedia.org/r/198720 (https://phabricator.wikimedia.org/T93776) (owner: 10Giuseppe Lavagetto) [14:39:36] 6operations, 7Graphite, 7HHVM, 7Icinga: hhvm - Icinga - UNKNOWN queue size / busy threads - https://phabricator.wikimedia.org/T92967#1162621 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi recovered, see related [14:41:22] 6operations, 10ops-codfw, 3codfw-appserver-setup, 3wikis-in-codfw: mw2050 has probably a faulty disk - https://phabricator.wikimedia.org/T93858#1162625 (10Papaul) @Joe can you please tell me what drive has problem. Thanks. [14:42:26] jouncebot, next [14:42:27] In 0 hour(s) and 17 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150330T1500) [14:42:49] helpful [14:45:33] 6operations, 10Beta-Cluster, 6Labs: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#1162629 (10fgiunchedi) p:5High>3Normal the immediate issue of /var filling up with core dumps seems fixed, hence priority normal, however the path used for cores doesn't seem to exist (... [14:46:04] jouncebot, next [14:46:04] In 0 hour(s) and 13 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150330T1500) [14:46:21] 6operations, 10Beta-Cluster, 6Labs: HHVM core dumps in labs - https://phabricator.wikimedia.org/T1259#1162639 (10fgiunchedi) [14:47:47] * aude probably won't be ready for swat [14:47:56] jenkins broken for wikibase [14:49:35] I will override jenkins if it is blocking deployments [14:50:00] :) [14:50:10] Assuming the change is actually OK [14:50:11] 7Blocked-on-Operations, 6operations, 10Continuous-Integration, 6Scrum-of-Scrums: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1162647 (10Anomie) The code itself doesn't require HHVM at all to build th... [14:50:11] but we can't even prepare our things and it's not that urgent [14:50:17] Fun. :-) [14:50:27] can wait until later if i am still awake or tomorrow [14:50:31] ok, let's leave it for now then [14:51:13] manybubbles, marktraceur, ^d, thcipriani, Krenair_ (since you're active) - Who wants to SWAT this morning? [14:51:34] James_F, Nemo_bis, thedj, bd808, tonythomas: Ping for SWAT in about 9 minutes. [14:51:39] I was planning to [14:51:43] Krenair_: Ok! [14:51:50] I am here. Jeff_Green : you back ? [14:51:53] Pong anyway. [14:52:40] 6operations, 10ops-codfw: ms-be2002.codfw.wmnet: slot=4 dev=sde failed - https://phabricator.wikimedia.org/T94014#1162653 (10fgiunchedi) p:5Triage>3Normal [14:52:52] Aw, I could actually have done it. Tomorrow! [14:53:09] * anomie will try to remember marktraceur already claimed SWAT for tomorrow morning. [14:53:23] * James_F grins at deployers fighting over the privilege of being dumped with doing SWAT. ;-) [14:53:40] Today involves no extension changes [14:53:48] <^d> I feel like crap, I'm going back to bed. [14:53:56] * James_F hugs ^demon|sick [14:53:56] if our jenkins issues are resolved, i might grab a slot before the parsoid deploy later [14:54:00] and no double patches for separate branches [14:54:00] But from a distance. [14:54:02] instead of waiting until 1am [14:54:46] (03PS1) 10Alexandros Kosiaris: Ganeti partman configuration [puppet] - 10https://gerrit.wikimedia.org/r/200580 (https://phabricator.wikimedia.org/T94042) [14:55:55] bd808, thedj, Nemo_bis: you there? [14:56:31] tonythomas: yep here [14:56:53] Jeff_Green: great! [14:56:53] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Ganeti partman configuration [puppet] - 10https://gerrit.wikimedia.org/r/200580 (https://phabricator.wikimedia.org/T94042) (owner: 10Alexandros Kosiaris) [14:57:02] James_F: "Hugs [...] from a distance". How does /that/ work in practice? :-) [14:57:16] James_F: I've been lazy lately, I figured I'd try to be useful for once [14:57:26] Especially since I'll be out for 5 possible SWAT days. [14:57:37] Coren: You've seen air kisses, right? Air hugs are do-able. [14:57:46] marktraceur: Why break the habit of a lifetime? [14:58:17] James_F, I'm guessing the -1 on https://gerrit.wikimedia.org/r/#/c/196984/ is no longer relevant [14:58:55] Krenair_: Ha. Yeah. Removing. [14:59:06] (03CR) 10Jforrester: [C: 031] "That's now. :-)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/196984 (https://phabricator.wikimedia.org/T93386) (owner: 10Jforrester) [15:00:05] manybubbles, anomie, ^d, thcipriani, James_F: Respected human, time to deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150330T1500). Please do the needful. [15:01:04] (03CR) 10Alex Monk: [C: 032] Enable VisualEditor by default on "phase 5" Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/196984 (https://phabricator.wikimedia.org/T93386) (owner: 10Jforrester) [15:01:12] (03Merged) 10jenkins-bot: Enable VisualEditor by default on "phase 5" Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/196984 (https://phabricator.wikimedia.org/T93386) (owner: 10Jforrester) [15:02:05] !log krenair Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/196984/ - VE phase 5 (duration: 00m 07s) [15:02:08] Krenair_: o/ [15:02:08] Logged the message, Master [15:02:09] James_F, ^ [15:02:28] Krenair_: yes [15:02:36] My patch should be a prod no-op. Just a safety measure [15:02:44] 6operations, 10MediaWiki-Vagrant, 6WMF-Legal: RDoc puppet documentation should state license - https://phabricator.wikimedia.org/T93998#1162680 (10fgiunchedi) p:5Triage>3Low priority to low, also we're using `--mode rdoc` which bears this warning ``` WARNING: RDoc support is only available under Ruby 1.8... [15:02:47] Krenair_: Working. [15:03:05] (03PS2) 10Alex Monk: monolog: MWLoggerMonologSamplingHandler -> Monolog\Handler\SamplingHandler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200286 (owner: 10BryanDavis) [15:03:14] (03CR) 10Alex Monk: [C: 032] monolog: MWLoggerMonologSamplingHandler -> Monolog\Handler\SamplingHandler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200286 (owner: 10BryanDavis) [15:05:25] 7Blocked-on-Operations, 6operations, 10Continuous-Integration, 6Scrum-of-Scrums: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1162691 (10akosiaris) >>! In T88798#1162647, @Anomie wrote: > The code its... [15:05:30] 6operations, 10RESTBase, 10RESTBase-Cassandra: graphs for Cassandra metrics - https://phabricator.wikimedia.org/T93884#1162693 (10fgiunchedi) p:5Triage>3Normal I'm assuming you have graphite access now, in any case check out also https://gdash.wikimedia.org/ and related puppet module if that suits [15:05:53] (03Merged) 10jenkins-bot: monolog: MWLoggerMonologSamplingHandler -> Monolog\Handler\SamplingHandler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200286 (owner: 10BryanDavis) [15:06:29] !log krenair Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/200286/2 - should be a no-op (duration: 00m 08s) [15:06:31] bd808, [15:06:32] Logged the message, Master [15:07:25] seems ok [15:07:42] cool [15:08:10] ahh, now, Nemo_bis [15:09:14] Is the idea to prevent this from taking any effect until wmf24 goes to group0 (wednesday)? [15:09:24] and then effectively follow the train? [15:09:48] <_joe_> uhm [15:10:04] <_joe_> should we merge https://gerrit.wikimedia.org/r/#/c/200158/ maybe? [15:10:07] Krenair_: wmf23. [15:10:13] It won't work on wmf23 [15:10:27] Krenair_: It's meant to go live to group0 immediately, group1 tomorrow and group2 on Wednesday. [15:10:30] Krenair_: yes [15:10:32] <_joe_> bd808: or, we can merge it tomorrow - what do you think? [15:10:59] (03PS3) 10coren: Conntrack collector for diamond [puppet] - 10https://gerrit.wikimedia.org/r/192335 (https://phabricator.wikimedia.org/T90437) [15:11:00] Nemo_bis, James_F: So we're clear, this does not apply to wmf23, right? '1.25wmf23' is not > '1.25wmf23' :) [15:11:19] this will wait for '1.25wmf24' to become wgversion [15:11:42] Hmm. The patch is now > 1.24wmf23? Oh well. [15:12:04] Certainly makes me feel better about it [15:12:11] Krenair_: Just confirmed with Krenair_. [15:12:12] Krenair_: Greg-g, even. [15:12:21] It's meant to be > 1.24wmf22 given the required logic. [15:12:24] * James_F sighs at hacks. [15:12:26] Need me to update? [15:12:39] Ah. I thought you wanted some more days. :) [15:12:44] I don't *need* it to be updated [15:12:58] Greg-g's joining IRC now. [15:12:59] But I do need everyone to be in agreement about how we're going to do this [15:13:39] Greg's comment asked "not everything at once", which > 1.24wmf22 also does [15:13:44] If it's deployed now [15:14:18] * greg-g waves g'morning [15:14:47] hey [15:15:53] Krenair_: so yeah, the plan was for group0 today, group1 tomorrow and group2 on wednesday [15:16:12] godog: Nikerabbit has a jobrunner config change that AaronS has +1'd that would be nice to get deployed soon -- https://gerrit.wikimedia.org/r/#/c/197919 [15:16:24] Nemo_bis, let's make this a >= '1.25wmf23' [15:16:31] bd808: sure, looking [15:16:39] which should have that effect [15:17:06] Nemo_bis, are you going to amend it? [15:17:09] ok [15:17:19] Krenair_: thanks for helping [15:17:35] somone time to merge a few wgcopyurl witelistings? [15:18:06] (03PS5) 10Nemo bis: Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) [15:18:23] Steinsplitter: probably, add them to the wiki page? [15:18:50] (03CR) 10Nemo bis: "Turns out we don't require the additional time, but only to avoid deploying everything at once. Will go now." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [15:18:54] (03PS4) 10Filippo Giunchedi: Add dedicated runner for MessageIndexRebuildJob [puppet] - 10https://gerrit.wikimedia.org/r/197919 (https://phabricator.wikimedia.org/T90704) (owner: 10Nikerabbit) [15:19:03] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Add dedicated runner for MessageIndexRebuildJob [puppet] - 10https://gerrit.wikimedia.org/r/197919 (https://phabricator.wikimedia.org/T90704) (owner: 10Nikerabbit) [15:19:06] Nemo_bis, OK [15:19:07] greg-g: yes, this two: https://phabricator.wikimedia.org/maniphest/?statuses=open,stalled&assigned=PHID-USER-nuf2sujf7qrx4v5ixbs3#R [15:19:17] greg-g: we have our bouncehandler too in the queue :) [15:19:18] (03PS6) 10Alex Monk: Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [15:19:21] (03PS5) 10Giuseppe Lavagetto: ganglia: autogenerate datasources from the list of clusters [puppet] - 10https://gerrit.wikimedia.org/r/198721 (https://phabricator.wikimedia.org/T93776) [15:19:30] (03CR) 10Alex Monk: [C: 032] Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [15:19:33] (03PS2) 10Steinsplitter: Adding *.loc.gov to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/199854 (https://phabricator.wikimedia.org/T94017) [15:20:00] bd808 Nikerabbit merged, should be propagating [15:20:17] (03PS2) 10Steinsplitter: Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) [15:20:28] thanks! Now we can schedule the config change to start using that new queue [15:21:27] (03PS1) 10Faidon Liambotis: nescio.esams -> nescio, add IPv6 [dns] - 10https://gerrit.wikimedia.org/r/200582 [15:21:32] (03PS1) 10Faidon Liambotis: nescio.esams -> nescio [puppet] - 10https://gerrit.wikimedia.org/r/200583 [15:21:34] (03PS1) 10Faidon Liambotis: autoinstall: switch nescio to jessie [puppet] - 10https://gerrit.wikimedia.org/r/200584 [15:21:36] (03PS1) 10Faidon Liambotis: Deprecate the old in-subnet esams DNS IP (91.198.174.6) [puppet] - 10https://gerrit.wikimedia.org/r/200585 [15:21:42] akosiaris: need your zotero-fu [15:21:44] bblack: ^ [15:21:50] have aminute? [15:22:04] (03Abandoned) 10coren: Labs: Traffic shaping [WIP] [puppet] - 10https://gerrit.wikimedia.org/r/139139 (owner: 10coren) [15:22:56] mobrovac: give me 10-15 mins, working on something else right now [15:23:16] akosiaris: cool, thnx, ping me when you're "free" [15:24:04] bd808 Nikerabbit looks like a puppet run and restart of jobrunner is working (to my untrained eye at least) [15:24:07] (03CR) 10BBlack: [C: 031] nescio.esams -> nescio, add IPv6 [dns] - 10https://gerrit.wikimedia.org/r/200582 (owner: 10Faidon Liambotis) [15:24:15] nice [15:24:36] (03CR) 10Faidon Liambotis: [C: 032] nescio.esams -> nescio, add IPv6 [dns] - 10https://gerrit.wikimedia.org/r/200582 (owner: 10Faidon Liambotis) [15:25:22] (03PS2) 10Nuria: Add counter for absolute number of lines on log [debs/logster] - 10https://gerrit.wikimedia.org/r/200182 (https://phabricator.wikimedia.org/T94193) [15:25:24] (03CR) 10BBlack: [C: 031] nescio.esams -> nescio [puppet] - 10https://gerrit.wikimedia.org/r/200583 (owner: 10Faidon Liambotis) [15:25:26] godog: hey, around? :) [15:25:30] (03PS1) 10Alexandros Kosiaris: Assign role::ganeti to ganeti boxes [puppet] - 10https://gerrit.wikimedia.org/r/200587 [15:25:36] (03CR) 10BBlack: [C: 031] autoinstall: switch nescio to jessie [puppet] - 10https://gerrit.wikimedia.org/r/200584 (owner: 10Faidon Liambotis) [15:25:42] (03CR) 10BBlack: [C: 031] Deprecate the old in-subnet esams DNS IP (91.198.174.6) [puppet] - 10https://gerrit.wikimedia.org/r/200585 (owner: 10Faidon Liambotis) [15:25:59] (03CR) 10Faidon Liambotis: [C: 032] nescio.esams -> nescio [puppet] - 10https://gerrit.wikimedia.org/r/200583 (owner: 10Faidon Liambotis) [15:26:03] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: switch nescio to jessie [puppet] - 10https://gerrit.wikimedia.org/r/200584 (owner: 10Faidon Liambotis) [15:26:08] (03CR) 10Faidon Liambotis: [C: 032] Deprecate the old in-subnet esams DNS IP (91.198.174.6) [puppet] - 10https://gerrit.wikimedia.org/r/200585 (owner: 10Faidon Liambotis) [15:26:35] (03CR) 10coren: "Yes, the move is intentional. That mechanism hasn't been actually used for some time, but when it returns it needs to follow the dumps fi" [puppet] - 10https://gerrit.wikimedia.org/r/199639 (owner: 10coren) [15:27:32] 6operations, 6CA-team: secure.wikimedia.org entries still showing up in Google search results - https://phabricator.wikimedia.org/T93531#1162750 (10fgiunchedi) p:5Triage>3Low [15:28:23] 6operations, 3Interdatacenter-IPsec: IPsec: add firewall rules - https://phabricator.wikimedia.org/T85823#1162753 (10BBlack) [15:30:05] oh my... a fresh new d-i file worked out of the box on the very first installation. I can not believe it... [15:30:25] must be my lucky day, gonna buy a lottery ticket [15:30:28] gosh mediawiki-phpunit-zend is so slow [15:30:54] (03Merged) 10jenkins-bot: Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [15:30:55] !log reimaging nescio [15:30:58] Logged the message, Master [15:31:40] !log krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198691/ (duration: 00m 08s) [15:31:41] Nemo_bis, still there? sorry about that wait [15:31:43] Logged the message, Master [15:31:54] 1 down, 1 to go [15:32:17] (03PS12) 10Nuria: Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) [15:33:11] PROBLEM - Recursive DNS on 91.198.174.6 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:33:13] Krenair_: slow tests indeed :) I'll verify manually when the train goes [15:33:13] (03CR) 10jenkins-bot: [V: 04-1] Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) (owner: 10Nuria) [15:33:25] 6operations, 3Interdatacenter-IPsec: Fix ipv6 autoconf issues - https://phabricator.wikimedia.org/T94417#1162762 (10BBlack) 3NEW a:3BBlack [15:33:30] !log krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/198691/ (duration: 00m 06s) [15:33:33] Logged the message, Master [15:33:36] Nemo_bis, that is now synched [15:33:56] 6operations, 3Interdatacenter-IPsec: Fix ipv6 autoconf issues - https://phabricator.wikimedia.org/T94417#1162771 (10BBlack) [15:33:57] 6operations, 3Interdatacenter-IPsec: IPsec: roll-out plan - https://phabricator.wikimedia.org/T92604#1162770 (10BBlack) [15:34:03] it currently applies to the group0 wikis (test, test2, testwikidata, mediawiki, zero) [15:34:13] and will follow the train the rest of the way [15:34:37] tonythomas, are we going to be able to do this today? [15:34:42] thedj, you there? [15:34:52] yup [15:35:10] Krenair_: we are ready! Jeff_Green isn't it ? [15:35:19] or is there something blocking in ? [15:35:24] (03PS1) 10BBlack: add_ip6_mapped: remove temporary code from last year [puppet] - 10https://gerrit.wikimedia.org/r/200590 [15:35:26] (03PS1) 10BBlack: add_ip6_mapped: no-op refactor to make +token commit simpler [puppet] - 10https://gerrit.wikimedia.org/r/200591 [15:35:28] (03PS1) 10BBlack: add_ip6_mapped: use tokens on trusty/jessie+ [puppet] - 10https://gerrit.wikimedia.org/r/200592 [15:35:57] wooo [15:35:58] tonythomas, I don't think Jeff is here, is he? That's who is currently blocking deployment of this extension to wikipedias, right? [15:36:03] mobile got merged [15:36:15] tonythomas: sure [15:36:17] (03PS2) 10BBlack: add_ip6_mapped: use tokens on trusty/jessie+ [puppet] - 10https://gerrit.wikimedia.org/r/200592 (https://phabricator.wikimedia.org/T94417) [15:36:20] oh! [15:36:21] hi [15:36:23] yay ! we're here ! [15:36:33] 6operations, 10ops-esams: cp3011 hardware fault - https://phabricator.wikimedia.org/T92306#1162777 (10fgiunchedi) p:5Triage>3Normal [15:36:36] I just started merging thedj's patch [15:36:48] okey. wont be an issue. We will wait ! [15:36:50] (03CR) 10BBlack: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/200592 (https://phabricator.wikimedia.org/T94417) (owner: 10BBlack) [15:36:52] bouncehandler will be right after [15:37:02] (03CR) 10jenkins-bot: [V: 04-1] add_ip6_mapped: use tokens on trusty/jessie+ [puppet] - 10https://gerrit.wikimedia.org/r/200592 (https://phabricator.wikimedia.org/T94417) (owner: 10BBlack) [15:37:32] PROBLEM - NTP peers on nescio is CRITICAL: NTP CRITICAL: No response from NTP server [15:37:43] stupid vim [15:37:51] PROBLEM - Disk space on nescio is CRITICAL: Connection refused by host [15:37:51] PROBLEM - DPKG on nescio is CRITICAL: Connection refused by host [15:38:00] PROBLEM - salt-minion processes on nescio is CRITICAL: Connection refused by host [15:38:01] PROBLEM - dhclient process on nescio is CRITICAL: Connection refused by host [15:38:01] PROBLEM - configured eth on nescio is CRITICAL: Connection refused by host [15:38:01] PROBLEM - puppet last run on nescio is CRITICAL: Connection refused by host [15:38:21] PROBLEM - RAID on nescio is CRITICAL: Connection refused by host [15:38:31] (03PS3) 10BBlack: add_ip6_mapped: use tokens on trusty/jessie+ [puppet] - 10https://gerrit.wikimedia.org/r/200592 (https://phabricator.wikimedia.org/T94417) [15:39:12] PROBLEM - Host 91.198.174.6 is DOWN: PING CRITICAL - Packet loss = 100% [15:40:00] am guessing that's nescio? [15:40:17] 6operations, 10ops-codfw: ms-be2002.codfw.wmnet: slot=4 dev=sde failed - https://phabricator.wikimedia.org/T94014#1162787 (10Papaul) wiWl be receiving the replacement drive on site tomorrow morning. [15:40:20] sort of [15:40:40] bu yes, see !log and ignore [15:41:56] (03PS1) 10Nuria: Change metric prefix after logster changes [puppet] - 10https://gerrit.wikimedia.org/r/200593 [15:43:03] 6operations, 6Labs: bond0 connection on labstore1001 is unpuppetized - https://phabricator.wikimedia.org/T92622#1162796 (10fgiunchedi) p:5Triage>3Normal what was the problem with the switch btw? has it been fixed? [15:45:13] 6operations, 3Interdatacenter-IPsec: Fix ipv6 autoconf issues - https://phabricator.wikimedia.org/T94417#1162813 (10BBlack) https://gerrit.wikimedia.org/r/#/c/200592/ ^ (guess I need to remember to use correct task-ref syntax in PS1 every time) [15:46:23] 6operations, 10ops-codfw: setup and deploy mw2135 through mw2215 - https://phabricator.wikimedia.org/T86806#1162820 (10fgiunchedi) p:5Triage>3High [15:47:12] YuviPanda: Hey, I know you use graphite to watch metrics too; have you played with grafana at all yet? [15:47:28] YuviPanda: http://grafana.wikimedia.org/#/dashboard/db/labs-monitoring [15:48:00] 6operations, 10ops-codfw: setup and deploy mw2135 through mw2215 - https://phabricator.wikimedia.org/T86806#1162837 (10Papaul) @Fgiunchedi it is complete on my side [15:49:47] (03CR) 10Ottomata: [C: 032 V: 032] Add counter for absolute number of lines on log [debs/logster] - 10https://gerrit.wikimedia.org/r/200182 (https://phabricator.wikimedia.org/T94193) (owner: 10Nuria) [15:50:38] 6operations, 10Wikimedia-Git-or-Gerrit, 7Monitoring: Improve monitoring of https://git.wikimedia.org/ - https://phabricator.wikimedia.org/T94320#1162852 (10Dzahn) Even suggested a patch in the past that would let Icinga automatically restart gitblit when monitoring detects it as down but it was rejected for... [15:50:45] thedj [15:50:47] !log krenair Synchronized php-1.25wmf23/resources/src/mediawiki.action/mediawiki.action.edit.preview.js: https://gerrit.wikimedia.org/r/#/c/200044/ (duration: 00m 06s) [15:50:51] Logged the message, Master [15:50:53] please test [15:51:20] (03PS3) 10Alex Monk: Install bouncehandler extension everywhere ( including wikipedias ) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198220 (https://phabricator.wikimedia.org/T92877) (owner: 1001tonythomas) [15:51:28] (03CR) 10Alex Monk: [C: 032] Install bouncehandler extension everywhere ( including wikipedias ) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198220 (https://phabricator.wikimedia.org/T92877) (owner: 1001tonythomas) [15:51:33] (03Merged) 10jenkins-bot: Install bouncehandler extension everywhere ( including wikipedias ) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198220 (https://phabricator.wikimedia.org/T92877) (owner: 1001tonythomas) [15:52:15] wow ~! [15:52:28] Congrats :) [15:52:34] 6operations, 10ops-codfw: setup and deploy mw2135 through mw2215 - https://phabricator.wikimedia.org/T86806#1162869 (10fgiunchedi) @joe @robh is this complete? [15:52:49] !log krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198220/ - BounceHandler to Wikipedias (duration: 00m 07s) [15:52:52] Logged the message, Master [15:53:01] tonythomas, please test etc. [15:53:13] Nemo_bis: thanks ! Jeff_Green ready with the fake mail box ? [15:53:18] sec [15:54:17] (03PS1) 10Ottomata: Bump debian/changelog to version 0.0.11 [debs/logster] (debian) - 10https://gerrit.wikimedia.org/r/200594 [15:55:58] tonythomas: should be ready [15:56:17] Jeff_Green: what was the e-mail id anyway ? [15:56:30] (03PS4) 10coren: Labs: Split out labstores substance into roles [puppet] - 10https://gerrit.wikimedia.org/r/199639 [15:57:00] tonythomas: test123@trouser.org [15:57:12] i have no idea what the password was, I will reset [15:57:20] Jeff_Green: okey. making the account now [15:57:47] (03CR) 10coren: [C: 032] " Coren: +1 on the roles split. Do babysit :)" [puppet] - 10https://gerrit.wikimedia.org/r/199639 (owner: 10coren) [15:57:58] (03PS4) 10BBlack: add_ip6_mapped: use tokens on trusty/jessie+ [puppet] - 10https://gerrit.wikimedia.org/r/200592 (https://phabricator.wikimedia.org/T94417) [15:58:00] (03PS2) 10BBlack: add_ip6_mapped: remove temporary code from last year [puppet] - 10https://gerrit.wikimedia.org/r/200590 [15:58:02] (03PS2) 10BBlack: add_ip6_mapped: no-op refactor to make +token commit simpler [puppet] - 10https://gerrit.wikimedia.org/r/200591 [15:58:11] (03CR) 10BBlack: [C: 032 V: 032] add_ip6_mapped: remove temporary code from last year [puppet] - 10https://gerrit.wikimedia.org/r/200590 (owner: 10BBlack) [16:01:24] 6operations, 6Labs: bond0 connection on labstore1001 is unpuppetized - https://phabricator.wikimedia.org/T92622#1162915 (10coren) Not as far as I know. Last news I heard @faidon had a fair idea of what the issue might be, but fixing it would require downtime and the matter was set aside. @yuvipanda: the bond... [16:01:56] mobrovac: so, I suppose you wanted me for https://phabricator.wikimedia.org/T94169, right? Or is it something else ? [16:01:56] (03PS1) 10Andrew Bogott: Make labs resolv.conf play nice with resolvconf updates [puppet] - 10https://gerrit.wikimedia.org/r/200595 [16:02:23] akosiaris: yep [16:02:30] Steinsplitter, in future, can you make sure your patches make it into the swat list before the window opens? [16:02:45] (03CR) 10jenkins-bot: [V: 04-1] Make labs resolv.conf play nice with resolvconf updates [puppet] - 10https://gerrit.wikimedia.org/r/200595 (owner: 10Andrew Bogott) [16:02:47] (03CR) 10Dzahn: [C: 032] icinga: check mailman shunt queue too [puppet] - 10https://gerrit.wikimedia.org/r/200524 (https://phabricator.wikimedia.org/T93783) (owner: 10Filippo Giunchedi) [16:02:49] these ones are in the queue, they're fine [16:02:56] but I prefer to know about them in advance [16:02:58] (03PS5) 10BBlack: add_ip6_mapped: use tokens on trusty/jessie+ [puppet] - 10https://gerrit.wikimedia.org/r/200592 (https://phabricator.wikimedia.org/T94417) [16:03:00] (03PS3) 10BBlack: add_ip6_mapped: no-op refactor to make +token commit simpler [puppet] - 10https://gerrit.wikimedia.org/r/200591 [16:03:03] 6operations, 10ops-codfw, 3codfw-appserver-setup, 3wikis-in-codfw: mw2050 has probably a faulty disk - https://phabricator.wikimedia.org/T93858#1162919 (10Papaul) @Rob this system is out of warranty last date was 2014-06-14. i have a SATA 500 Gb disk here that we took out from graphite2001 can we use that... [16:03:05] Krenair_: i try [16:03:08] thanks [16:03:19] (03PS5) 10coren: Labs: Monitor network staturation on labstores [puppet] - 10https://gerrit.wikimedia.org/r/199297 (https://phabricator.wikimedia.org/T92629) [16:03:29] Jeff_Green: user verptestuser created in en.wiki [16:03:30] ori: yt? i'm going to try to push this for nuria: [16:03:31] https://phabricator.wikimedia.org/T90363 [16:03:37] she says there is box available already for this? [16:03:47] Steinsplitter, I think that if I merge one of them, the other will not be mergable without a rebase? [16:04:06] i think so [16:04:24] Jeff_Green: email confirmed. Sending first mail now [16:04:31] great [16:05:17] 6operations, 10Analytics-Cluster, 3Interdatacenter-IPsec: Secure inter-datacenter web request log (Kafka) traffic - https://phabricator.wikimedia.org/T92602#1162921 (10Gage) p:5Triage>3Normal [16:05:24] Jeff_Green: see that in the mailbox ? [16:05:41] yep [16:05:41] (03PS6) 10Giuseppe Lavagetto: ganglia: autogenerate datasources from the list of clusters [puppet] - 10https://gerrit.wikimedia.org/r/198721 (https://phabricator.wikimedia.org/T93776) [16:05:52] Jeff_Green: okey. safe to delete the account now, then [16:06:01] done [16:06:13] :) sending first to-be-bounce [16:06:46] sent. can you see that in the bouncehandler db ? [16:07:25] (03CR) 10coren: [C: 032] Labs: Monitor network staturation on labstores [puppet] - 10https://gerrit.wikimedia.org/r/199297 (https://phabricator.wikimedia.org/T92629) (owner: 10coren) [16:08:25] tonythomas: looking [16:08:31] okey ! [16:09:16] oh, hmm. which db should I be looking in? [16:09:30] there's nothing in wikishared [16:09:43] extension2 or something like that - and 'bouncerecords' ? [16:10:11] (03CR) 10Ottomata: [C: 032 V: 032] Bump debian/changelog to version 0.0.11 [debs/logster] (debian) - 10https://gerrit.wikimedia.org/r/200594 (owner: 10Ottomata) [16:10:51] tonythomas: hrm. no luck so far [16:16:37] 6operations, 6Labs, 7Monitoring, 5Patch-For-Review: Setup alarms for labstore* to check for network saturation - https://phabricator.wikimedia.org/T92629#1162974 (10coren) 5Open>3Resolved Now properly monitored with thresholds at levels suggested by Faidon. [16:16:46] 6operations, 6Labs, 7Monitoring: Setup alarms for labstore* to check for network saturation - https://phabricator.wikimedia.org/T92629#1162976 (10coren) [16:17:13] anything, tonythomas / Jeff_Green ? [16:17:38] Krenair nothing ~bad~ but we're trying to figure out which cluster/db to query for this wiki [16:17:40] akosiaris: so I merged https://gerrit.wikimedia.org/r/#/c/200157/ , deployed zotero-translators from tin and restarted both zotero and citoid, but no luck [16:17:50] (03PS13) 10Nuria: Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) [16:17:50] ah. I am stuck :\ "As an anti-spam measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes " [16:18:07] akosiaris: even though I can see the change deployed on sca100? [16:18:21] (03PS3) 10Alex Monk: Adding *.loc.gov to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/199854 (https://phabricator.wikimedia.org/T94017) (owner: 10Steinsplitter) [16:18:33] Krenair_: is there a way to override that one ( or should I change IP ? ) [16:18:44] (03CR) 10jenkins-bot: [V: 04-1] Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) (owner: 10Nuria) [16:19:03] not sure [16:19:13] what was the action you tried tonythomas? [16:19:19] Special:EmailUser [16:20:00] mobrovac: it does work though on Beta [16:20:13] tonythomas, and what wiki? [16:20:27] and user id? [16:20:37] akosiaris: oh zotero, we <3 U [16:20:44] english wiki - user id is 01tonythomas [16:23:25] Krenair_: are you able to unblock/do something for that one ? [16:23:32] I'm not sure :/ [16:24:07] (03PS14) 10Nuria: Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) [16:24:18] greg-g: can i have a deploy slot later, after parsoid? [16:24:42] we had issues with jenkins, so wasn't ready for swat and don't want to be awake at 1am :) [16:24:56] I can't find it tonythomas [16:24:59] (03CR) 10Legoktm: "extension-list is used for discovering which extensions have i18n messages, and not all extensions do (ActiveAbstract, WikimediaMaintenanc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 (owner: 10Chad) [16:25:04] (03CR) 10jenkins-bot: [V: 04-1] Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) (owner: 10Nuria) [16:25:38] Krenair_: with the userid = 01tonythomas ? ( the '01' is there ) [16:25:55] can you PM me your IP? [16:27:19] Krenair_: done ! [16:27:44] we are checking at exim logs btw - looks like the bounce did reach exim alright - and from there - we are figuring out [16:28:12] I can't find any limiter memc entry for that either [16:29:46] hmm. Thats strange [16:31:05] changed ip, now works ! [16:31:25] strange [16:31:47] 6operations, 7Monitoring: Job queue stats are broken - https://phabricator.wikimedia.org/T87594#1163028 (10fgiunchedi) likely related to T85641, i.e. mw stats scaffolding is being changed [16:32:04] 6operations, 10ops-eqiad: Increase asw-d-eqiad uplink capacity - https://phabricator.wikimedia.org/T92914#1163030 (10Cmjohnson) Faidon, I can't pick the same pattern on cr1 available ports 5/2/1 5/3/1 5/3/3 4/0/3 4/1/3 4/2/3 cr2 5/2/1 5/2/3 4/0/3 4/1/3 4/2/1 4/2/2 4/2/3 4/3/2 [16:32:40] tonythomas, do you connect via ipv6 by any change? :/ [16:33:11] Krenair_: I dont know about that though. But right now I see again an Action Throttled error again [16:33:38] is there some issue with the job queue ? [16:35:28] I don't think so [16:36:43] nothing from bouncehandler showing on mwscript showJobs.php enwiki --group [16:37:11] (03PS15) 10Nuria: Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) [16:37:15] no - it would be in test2.wikipedia.org', [16:37:15] emails are rate limited. [16:37:18] 6operations, 3wikis-in-codfw: deploy wtp2001-2020 - https://phabricator.wikimedia.org/T90271#1163064 (10RobH) [16:38:02] we are posting to the API of test2.wikipedia - can someone check its job queue log ? [16:38:22] Glaisher: any chance that I can send 5 emails in a row ? [16:38:43] add an exception to the throttle rule? [16:38:51] or get noratelimits right [16:39:07] Glaisher: I can do that ? or an admin must ? [16:39:19] sysops have noratelimit [16:40:02] hmm. I should find someone with the right ! [16:40:21] or you could get sysop assigned to your account [16:40:38] Glaisher: that would be great. I should request somewhere, right ? [16:40:50] let me try to see whether I can find one [16:41:01] Glaisher: okey ! [16:41:30] 6operations, 3Interdatacenter-IPsec: IPsec: roll-out plan - https://phabricator.wikimedia.org/T92604#1163078 (10Gage) p:5Triage>3Normal a:3Gage [16:41:42] 6operations, 3Interdatacenter-IPsec, 7Monitoring: Monitor IPsec status - https://phabricator.wikimedia.org/T92603#1163081 (10Gage) p:5Triage>3Normal a:3Gage [16:42:06] tonythomas: o/ how'd the deployment go? [16:42:58] 7Puppet, 6Multimedia, 6Release-Engineering, 6Scrum-of-Scrums, and 3 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1163083 (10Gilles) I'll target Jessie. The following packages have their version on Jessie match what the latest Sentry expects: python-beautifulsoup... [16:43:02] legoktm: we have the bounce reaching back to exim as per the logs - we need someone to check the test2.wikipedia.org', jobqueue logs [16:43:35] krenair@tin:/srv/mediawiki-staging$ mwscript showJobs.php test2wiki --group [16:43:35] krenair@tin:/srv/mediawiki-staging$ [16:44:01] (03PS16) 10Ottomata: Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) (owner: 10Nuria) [16:44:10] (03CR) 10Ottomata: [C: 032 V: 032] Adding logster to count requests to wikimetrics UI [puppet] - 10https://gerrit.wikimedia.org/r/197411 (https://phabricator.wikimedia.org/T94193) (owner: 10Nuria) [16:44:17] 6operations, 10Continuous-Integration, 6Labs: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1163086 (10fgiunchedi) p:5High>3Low looks like everything was working as expected on the alerting side, not sure there's any other action? [16:44:46] Krenair tonythomas I don't see any results from that jobqueue query [16:45:16] I don't see any BounceHandlerJob's that have been executed on test2.wp [16:45:30] hmm. That looks strange - let me send an email again [16:46:12] ah. again I get the AntiSpam rule :\ [16:47:44] tonythomas, still getting it? [16:47:57] Krenair_: actually - I am sending from https://en.wikipedia.org/wiki/Special:EmailUser ! [16:48:08] I just got sysop in test2.wikipedia.org [16:48:22] ok [16:48:44] (03PS1) 10Giuseppe Lavagetto: ganglia::web::view: order resources in template [puppet] - 10https://gerrit.wikimedia.org/r/200603 [16:49:15] test2.wikipedia.org is just the machine receiving the post requests - so - is it possible to get a sysop in enwiki ? ( or is there someone ready to spare few minutes ? ) [16:49:40] If I had a sysadmin flag I would just give you a global group with noratelimit :p [16:49:43] i was gonna volunteer and then realised i'm no longer a sysop. [16:49:53] <_joe_> akosiaris: https://gerrit.wikimedia.org/r/200603 what do you think? [16:49:55] legoktm: is still a sysop at enwiki, I think? [16:50:02] I am! [16:50:04] what's up? [16:50:08] \o/ [16:50:34] legoktm: great! can you https://en.wikipedia.org/wiki/Special:EmailUser and email the user Verptestuser ? [16:50:35] tonythomas: do you need some rights? [16:50:39] ok sure [16:50:46] legoktm: yeah. noratelimit ! [16:50:51] ( if possible :D ) [16:50:54] sent [16:50:58] 7Puppet, 6Multimedia, 6Release-Engineering, 6Scrum-of-Scrums, and 3 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1163144 (10Gilles) Oh and python-psycopg2 also makes the list [16:50:59] _joe_: that I've been wanting to fix that for a long time... [16:51:09] Jeff_Green: something in the db/exim/jobs ? [16:51:17] not sure though ordered_json and pretty_generate create the same looking output [16:51:33] aude: sure, add to calendar,what's it for? /me is in a budget meeting of doom [16:51:48] tonythomas: I see your last send in mail logs [16:52:07] and in the db too [16:52:09] yay! [16:52:12] Jeff_Green: wahayttt!! [16:52:14] wow ! [16:52:14] it worked ? [16:52:31] tonythomas: i see the previous one in the db too [16:52:45] phew. almost gave me a heart-ache [16:52:46] just kick me . . . i think i was reading the datestamp wrong [16:52:50] sorry [16:52:55] :DDD [16:52:57] yay! [16:53:01] hahah! thats alright - I almost got some rights in enwiki [16:53:07] oeky - so - legoktm : 3 more ? [16:53:08] ha [16:53:30] (03PS1) 10Papaul: add mgmt asset tag for db20(43-51) [dns] - 10https://gerrit.wikimedia.org/r/200605 [16:53:46] ( we have the limit set as 5 ) [16:54:04] congratulations guys !!!! so good to see that finally working. [16:54:11] (03PS2) 10Giuseppe Lavagetto: ganglia::web::view: order resources in template [puppet] - 10https://gerrit.wikimedia.org/r/200603 [16:54:16] tonythomas: 3 more sent :) [16:54:32] thedj: thanks ! ( not yet ) we should have the user unsubscribed in the next one [16:54:42] Jeff_Green: thats showing up in the db ? [16:55:06] tonythomas: yeah [16:55:14] wonderful - legoktm ! one more ? [16:55:37] (03CR) 10Alex Monk: [C: 032] Adding *.loc.gov to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/199854 (https://phabricator.wikimedia.org/T94017) (owner: 10Steinsplitter) [16:55:59] tonythomas: now it says "This user has not specified a valid email address." [16:56:02] (03CR) 10Alexandros Kosiaris: [C: 031] "Not sure they will get the same output, but it should be readable JSON in both cases. Plus no more ruby 1.8 hash problems :-)" [puppet] - 10https://gerrit.wikimedia.org/r/200603 (owner: 10Giuseppe Lavagetto) [16:56:05] tonythomas: so they've been unconfirmed? :D [16:56:07] yyyyyyay! [16:56:11] (03PS1) 10RobH: subra/suhail need to be raided [puppet] - 10https://gerrit.wikimedia.org/r/200608 (https://phabricator.wikimedia.org/T93261) [16:56:13] legoktm: yup !!! [16:56:17] Jeff_Green: legoktm ! finally ! [16:56:21] how cool ! finally [16:56:21] :D [16:56:24] congrats tonythomas [16:56:26] tonythomas: congrats! [16:56:42] Krenair_: thanks ! after one year, we can close that phab task ! [16:56:47] (03CR) 10RobH: [C: 032] subra/suhail need to be raided [puppet] - 10https://gerrit.wikimedia.org/r/200608 (https://phabricator.wikimedia.org/T93261) (owner: 10RobH) [16:56:52] (03CR) 10Alex Monk: [V: 032] Adding *.loc.gov to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/199854 (https://phabricator.wikimedia.org/T94017) (owner: 10Steinsplitter) [16:57:03] 6operations, 6Phabricator: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#1163171 (10chasemp) >>! In T87467#1161698, @mmodell wrote: > @fgiunchedi: I'm not sure why the menu entry for "ops access request" isn't configured.... [16:57:46] !log krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/199854/ (duration: 00m 08s) [16:57:50] Logged the message, Master [16:58:04] Steinsplitter, ^ [16:58:12] thanks Krenair_ :-) [16:58:13] as predicted, the other one no longer merges [16:59:00] I have to go, back in maybe 15-30ish minutes [16:59:46] !log aaron Synchronized php-1.25wmf23/includes/User.php: I3b733a0221462350f3a24d54ffe814357f379512 (duration: 00m 06s) [16:59:49] Logged the message, Master [17:00:45] 6operations, 10Wikidata-Query-Service: deploy haedus and capella with os for orientdb testing - https://phabricator.wikimedia.org/T84902#1163187 (10fgiunchedi) looks like this is completed, anything else left @joe ? [17:00:52] 6operations, 3codfw-appserver-setup, 3wikis-in-codfw: Set up the mediawiki application layer in codfw - https://phabricator.wikimedia.org/T86894#1163190 (10Dzahn) [17:00:55] 6operations, 5Patch-For-Review: Setup poolcounter servers for codfw - https://phabricator.wikimedia.org/T93261#1163188 (10Dzahn) 5Resolved>3Open unfortunately we have to install them one more time because we didn't get RAID setup, just LVM without RAID. [17:01:17] tonythomas: so what is you next 1 year long project ? :) [17:01:29] <_joe_> akosiaris: sigh http://puppet-compiler.wmflabs.org/657/change/200603/html/uranium.wikimedia.org.html [17:01:58] thedj: its going to be Extension:NewsLetter, it seems ( wont take 1 year, I promise ) - this one took loads of time testing [17:02:11] Would you like to close https://phabricator.wikimedia.org/T92877 tonythomas? [17:02:37] Krenair_: I am just closing in. Jeff_Green says we have 37k entries in the bounce_records already [17:02:37] well, something like this, you wanna make sure it works. good job sticking to it. [17:02:54] 37k ? wow [17:03:13] thedj: yeah! Thanks! [17:04:00] !log faidon and alex are working on carbon, puppet is disabled [17:04:03] Logged the message, Master [17:04:15] Krenair_: closed [17:04:28] puppet agent's message + nagios check should be enough imho [17:04:45] (03PS3) 10Steinsplitter: Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) [17:04:51] (03CR) 10jenkins-bot: [V: 04-1] Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) (owner: 10Steinsplitter) [17:05:25] Krenair_: rebase fails :S [17:05:43] tonythomas: you should email wikitech-l :) [17:05:48] yeah, needs to be done manually Steinsplitter [17:05:59] legoktm: of course. doing that one now ! [17:06:02] tonythomas: also you should close https://phabricator.wikimedia.org/T48640 :) [17:06:11] Krenair_: i can upload a new patch and abadone this if you like [17:06:17] Steinsplitter: want me to rebase ? [17:06:19] that's not necessary [17:06:25] 6operations, 10Wikimedia-General-or-Unknown: Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail - https://phabricator.wikimedia.org/T58414#1163219 (1001tonythomas) [17:06:35] legoktm: closed :) [17:06:44] * thedj reminds Krenair_ that he was leaving for a bit. [17:07:46] tonythomas: and this one too! https://phabricator.wikimedia.org/T71019 :D [17:07:48] thedj, I'm actually in a meeting [17:08:19] legoktm: ah. more are there - including the arch review [17:08:37] tonythomas: yeah, I think that can be resolved now [17:09:22] true. [17:12:56] (03CR) 10Giuseppe Lavagetto: "Not sure if this is a good idea given the resulting files have no pretty-formatting." [puppet] - 10https://gerrit.wikimedia.org/r/200603 (owner: 10Giuseppe Lavagetto) [17:13:17] (03CR) 10Dzahn: [C: 032] add mgmt asset tag for db20(43-51) [dns] - 10https://gerrit.wikimedia.org/r/200605 (owner: 10Papaul) [17:13:47] (03PS4) 10Steinsplitter: Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) [17:13:50] (03CR) 10jenkins-bot: [V: 04-1] Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) (owner: 10Steinsplitter) [17:15:01] Steinsplitter: you didn't have master up to date. [17:15:59] akosiaris: any news re zotero? [17:16:25] mobrovac: translations are timing out [17:16:46] not sure why yet. I got to relocate, will continue debugging later on [17:16:54] thedj: thanks, do it tomorrow - no time now [17:16:59] ok thnx akosiaris [17:17:02] 7Puppet, 6Multimedia, 6Release-Engineering, 6Scrum-of-Scrums, and 3 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1163294 (10Gilles) My attempt at a venv referencing these local packages when available is sitting under /srv/deployment/sentry/sentry on the sentry-pac... [17:18:36] (03PS5) 10TheDJ: Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) (owner: 10Steinsplitter) [17:18:46] Steinsplitter: there :) [17:19:20] very courteous, thx thedj :-) [17:20:07] !log subra - rebooting for reinstall [17:20:11] Logged the message, Master [17:20:11] tonythomas: did bouncehandler get mentioned in the Tech News ? [17:20:38] thedj: nope I think. I am just drafting a mail to wikitech-l btw [17:21:14] tonythomas: also, i was wondering. If a user logs in, does he need to go to his prefs to realize that his address has bounced ? [17:21:51] thedj: we are waiting for a threshold of 5 bounces from a user to consider him a failing recipient [17:22:29] yeah and then we mark it as unverified right ? but do we tell the user ? Was thinking ideal case for an Echo notification.... [17:22:35] but - once the user is unsubscribed, yeah - he will have to see that red box in Special:Preferences [17:23:00] thedj: that would be a good addition I think - but echo would send email ? and that too would bounce ? [17:23:24] doesn't matter, as long as it give feedback to the user when he's logged in. [17:24:27] thedj: I will report that one as a task in phab then [17:24:34] tonythomas: i'll do it :) [17:24:38] thedj: thanks :) [17:25:00] mailed to wikitech-l & time for me to quit lab. ciao [17:28:36] (03CR) 10Alex Monk: [C: 032] Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) (owner: 10Steinsplitter) [17:28:42] (03Merged) 10jenkins-bot: Adding socrates.leidenuniv.nl to wgCopyUploadsDomains for GWT upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200318 (https://phabricator.wikimedia.org/T93757) (owner: 10Steinsplitter) [17:29:11] PROBLEM - Host suhail is DOWN: PING CRITICAL - Packet loss = 100% [17:29:15] !log krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/200318/ (duration: 00m 06s) [17:29:20] Logged the message, Master [17:30:06] (03PS1) 10Faidon Liambotis: install-server: use /etc/dhcp consistently [puppet] - 10https://gerrit.wikimedia.org/r/200615 [17:30:41] (03CR) 10Faidon Liambotis: [C: 032] install-server: use /etc/dhcp consistently [puppet] - 10https://gerrit.wikimedia.org/r/200615 (owner: 10Faidon Liambotis) [17:30:50] 6operations, 10Wikimedia-SVG-rendering, 7Upstream: Filter effect Gaussian blur filter not rendered correctly for small to medium thumbnail sizes - https://phabricator.wikimedia.org/T44090#1163361 (10TheDJ) I made it, and yes indeed Patch Merged was meant to indicate patch merged upstream. I figured we would... [17:31:00] RECOVERY - Host suhail is UP: PING OK - Packet loss = 0%, RTA = 42.94 ms [17:32:03] mutante: can you restart nutcracker on mw1147? [17:32:15] * aspiecat is surprised that's still erroring out [17:32:55] (03CR) 10coren: [C: 032] "Yuvi's previous +1 should be good enough." [puppet] - 10https://gerrit.wikimedia.org/r/192335 (https://phabricator.wikimedia.org/T90437) (owner: 10coren) [17:34:41] PROBLEM - dhclient process on suhail is CRITICAL: Connection refused by host [17:34:41] PROBLEM - configured eth on suhail is CRITICAL: Connection refused by host [17:34:51] PROBLEM - poolcounter on suhail is CRITICAL: Connection refused by host [17:35:01] PROBLEM - DPKG on suhail is CRITICAL: Connection refused by host [17:35:02] PROBLEM - puppet last run on suhail is CRITICAL: Connection refused by host [17:35:11] PROBLEM - salt-minion processes on suhail is CRITICAL: Connection refused by host [17:35:31] PROBLEM - Disk space on suhail is CRITICAL: Connection refused by host [17:35:41] PROBLEM - RAID on suhail is CRITICAL: Connection refused by host [17:35:52] PROBLEM - Poolcounter connection on suhail is CRITICAL: Connection refused [17:37:28] 6operations, 10RESTBase, 10hardware-requests: Expand RESTBase cluster capacity - https://phabricator.wikimedia.org/T93790#1163395 (10GWicke) >>! In T93790#1148439, @faidon wrote: > What "does not have a lot of margin on IO bandwidth and storage capacity" mean exactly? Which resource is near exhaustion (IOPS,... [17:41:37] (03PS1) 10coren: Labs: collect conntrack stats for labsnet1001 [puppet] - 10https://gerrit.wikimedia.org/r/200619 (https://phabricator.wikimedia.org/T90437) [17:41:57] (03PS2) 10Gage: ipsec-global: add /bin to path [puppet] - 10https://gerrit.wikimedia.org/r/200276 [17:43:01] (03CR) 10Gage: [C: 032] ipsec-global: add /bin to path [puppet] - 10https://gerrit.wikimedia.org/r/200276 (owner: 10Gage) [17:49:30] 6operations, 10ops-codfw: rack and initial configuration of wtp2001-2020 - https://phabricator.wikimedia.org/T86807#1163441 (10RobH) wtp2001-2004 os installed, pending key signing, rest pending install (post ops meeting) [17:50:12] PROBLEM - Host suhail is DOWN: PING CRITICAL - Packet loss = 100% [17:53:43] !log suhail, rebooting to fix BIOS settings, reinstall [17:53:46] Logged the message, Master [17:54:40] RECOVERY - Host suhail is UP: PING OK - Packet loss = 0%, RTA = 43.91 ms [17:56:43] !log subra, wmf-reimage [17:56:46] Logged the message, Master [17:57:01] PROBLEM - Host 2620:0:862:1:a6ba:dbff:fe30:d0df is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:1:a6ba:dbff:fe30:d0df [18:02:35] (03PS1) 10Thcipriani: Assign swift roles via ENC [puppet] - 10https://gerrit.wikimedia.org/r/200625 (https://phabricator.wikimedia.org/T91553) [18:04:45] PROBLEM - Host suhail is DOWN: PING CRITICAL - Packet loss = 100% [18:05:56] RECOVERY - Host suhail is UP: PING OK - Packet loss = 0%, RTA = 44.24 ms [18:10:56] (03PS1) 10Faidon Liambotis: Override nameservers for lvs3002/4 [puppet] - 10https://gerrit.wikimedia.org/r/200629 [18:12:13] _joe_: re the wikdata rdf dump, this is the ticket https://phabricator.wikimedia.org/T93658 but the manual commands are more com [18:12:26] (03CR) 10Faidon Liambotis: [C: 032] Override nameservers for lvs3002/4 [puppet] - 10https://gerrit.wikimedia.org/r/200629 (owner: 10Faidon Liambotis) [18:12:43] 6operations, 5Patch-For-Review: deploy francium for html/zim dumps - https://phabricator.wikimedia.org/T93113#1163581 (10GWicke) [18:12:45] complicated because I let talk myself into accepting a blocked task for that [18:12:52] 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1163585 (10GWicke) [18:12:53] 6operations, 10Datasets-General-or-Unknown, 6Services, 10hardware-requests: Hardware for HTML / zim dumps - https://phabricator.wikimedia.org/T91853#1097489 (10GWicke) [18:13:45] as such there is not yet a script to shard and assemble the shards. and if we would not use that the dump is too slow [18:14:09] 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1155079 (10GWicke) Could we get plain shell access in the meantime? [18:23:17] 6operations, 10RESTBase, 10hardware-requests: Expand RESTBase cluster capacity - https://phabricator.wikimedia.org/T93790#1163710 (10akosiaris) > GCs don't scale to arbitrary heap sizes / garbage production rates without increasing pause times. This means that the load per Cassandra instance needs to be limi... [18:23:56] (03PS2) 10Greg Grossmeier: Remove Chris's access [puppet] - 10https://gerrit.wikimedia.org/r/200334 (https://phabricator.wikimedia.org/T94032) [18:25:54] godog: I just added you to that ticket since you're on duty this week :) ^ [18:26:49] 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1163739 (10RobH) Plain shell to do what and access what directories? Nothing for the service is puppetized at this time, correct? (Just not sure what access to grant when nothing is detailed or puppetized.) [18:27:03] Hostname: text-lb.esams.wikimedia.org [18:27:04] Mon, 30 Mar 2015 17:23:34 GMT 3600 seconds behind! [18:27:25] for HTTP "Date:" header [18:27:46] <_joe_> uhm, no. [18:27:58] <_joe_> oh sorry, 17:23 [18:28:02] 6operations, 7Monitoring: Finalize and deploy ganglia_new puppet module - https://phabricator.wikimedia.org/T83538#1163746 (10faidon) [18:28:06] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1163748 (10GWicke) [18:28:21] 6operations, 7Monitoring, 5Patch-For-Review: remove ganglia(old), replace with ganglia_new - https://phabricator.wikimedia.org/T93776#1163753 (10faidon) [18:28:47] _joe_: hmm, i think its actually a bug in this tool i was using. [18:29:08] 6operations, 7Monitoring, 5Patch-For-Review: remove ganglia(old), replace with ganglia_new - https://phabricator.wikimedia.org/T93776#1145909 (10faidon) [18:29:09] 6operations, 7Monitoring: Finalize and deploy ganglia_new puppet module - https://phabricator.wikimedia.org/T83538#915376 (10faidon) [18:29:11] since curl shows me the right hour. (or it's just a subset of the servers) [18:29:32] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1155079 (10GWicke) @robh, plain shell to test full dumps before proceeding to automate the whole setup. We can work with @arielglenn to install nodejs and possibly nginx via puppet. [18:29:55] <_joe_> thedj: what request are you performing? [18:30:15] i was just trying to find an alternative to ssllabs, https://sslanalyzer.comodoca.com/?url=en.wikipedia.org [18:30:25] and this reported that error [18:30:45] <_joe_> thedj: uhm I'll check all the frontends in esams just to be sure [18:30:55] <_joe_> thanks for reporting it [18:31:13] you never know with this summer time stuff in europe after last weekend :) [18:34:22] (03CR) 10Eevans: [C: 031] "Having access restricted to just the set of nodes in the RESTBase/Cassandra cluster would be ideal, but this patch would be an improvement" [puppet] - 10https://gerrit.wikimedia.org/r/200093 (owner: 10Dzahn) [18:37:15] <_joe_> thedj: the Http Date header is correct everywhere AFAICT [18:37:50] _joe_: good. my will complain with the comodoca [18:38:24] _joe_: ok prepared the actual command to manually create a dump: https://phabricator.wikimedia.org/T93658#1163823 [18:39:40] <_joe_> jzerebecki: thanks! [18:40:27] tonythomas: btw, awesome job :) [18:40:58] 6operations, 10RESTBase, 10RESTBase-Cassandra, 6Security, 5Patch-For-Review: iptables firewall to limit access to Cassandra services - https://phabricator.wikimedia.org/T92680#1163852 (10Eevans) That last change, the one that added `base::firewall` has broken the test environment. RESTBase is unable to... [18:41:05] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Install nodejs and nginx on francium - https://phabricator.wikimedia.org/T94457#1163853 (10GWicke) 3NEW a:3ArielGlenn [18:41:36] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1163862 (10GWicke) a:5GWicke>3ArielGlenn [18:42:19] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1163868 (10GWicke) [18:43:16] 6operations, 10MediaWiki-extensions-Sentry, 6Multimedia, 10hardware-requests, 3Multimedia-Sprint-2015-03-25: Procure hardware for Sentry - https://phabricator.wikimedia.org/T93138#1163872 (10Tgr) [18:45:40] 6operations, 10MediaWiki-extensions-Sentry, 6Multimedia, 10hardware-requests, 3Multimedia-Sprint-2015-03-25: Procure hardware for Sentry - https://phabricator.wikimedia.org/T93138#1163883 (10Tgr) This is a real request now :) Sorry for the initial confusion! [18:57:16] 6operations, 10Wikidata-Query-Service: deploy haedus and capella with os for orientdb testing - https://phabricator.wikimedia.org/T84902#1164013 (10JanZerebecki) As orientdb testing is over... @Manybubbles are these still needed? [18:57:42] 6operations, 10Wikidata-Query-Service: deploy haedus and capella with os for orientdb testing - https://phabricator.wikimedia.org/T84902#1164019 (10Manybubbles) 5Open>3declined a:3Manybubbles [19:05:51] 6operations, 6Services: reinstall OCG servers - https://phabricator.wikimedia.org/T84723#1164054 (10mobrovac) [19:09:27] 6operations, 10RESTBase, 10hardware-requests: Expand RESTBase cluster capacity - https://phabricator.wikimedia.org/T93790#1164062 (10mobrovac) >>! In T93790#1163710, @akosiaris wrote: > Heh, sounds like a hack to bypass a software problem. More precisely, to bypass a //Java design feature//(TM) :P > It is... [19:09:55] manybubbles: nothing needed to give those servers back/decomission? [19:10:06] jzerebecki: nah - they can go home [19:11:59] manybubbles: how does anyone know they are not needed anymore? [19:12:31] jzerebecki: I have no idea, actually. _joe_ - what should we have done to give those nodes back? [19:12:45] !log subra/suhail: re-add to puppet, initial runs [19:12:49] Logged the message, Master [19:17:31] 10Ops-Access-Requests, 6operations: Requesting access to stat1/EventLogging data for Pcoombe - https://phabricator.wikimedia.org/T94466#1164105 (10Pcoombe) 3NEW [19:20:45] 10Ops-Access-Requests, 6operations: Requesting access to stat1/EventLogging data for Pcoombe - https://phabricator.wikimedia.org/T94466#1164122 (10MeganHernandez_WMF) As Peter's supervisor, I approve this request. Thank you! [19:24:15] (03CR) 10Tim Landscheidt: [C: 04-1] Make labs resolv.conf play nice with resolvconf updates (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/200595 (owner: 10Andrew Bogott) [19:24:19] (03PS1) 10Tim Landscheidt: WIP: Snapshot [puppet] - 10https://gerrit.wikimedia.org/r/200648 (https://phabricator.wikimedia.org/T93691) [19:28:11] godog: Quick trivial review? https://gerrit.wikimedia.org/r/#/c/200619/ [19:30:28] (03PS1) 10Rush: remove RT queue translations for phab [puppet] - 10https://gerrit.wikimedia.org/r/200651 [19:32:23] PROBLEM - NTP on suhail is CRITICAL: NTP CRITICAL: Offset unknown [19:32:24] 6operations: Upload fixed version of Dec 2011 Editor Survey codebook - https://phabricator.wikimedia.org/T94455#1164179 (10Dzahn) [19:33:11] 6operations: Upload fixed version of Dec 2011 Editor Survey codebook - https://phabricator.wikimedia.org/T94455#1163816 (10Dzahn) uploaded to dataset1001 and replaced the files as requested [19:34:02] RECOVERY - NTP on suhail is OK: NTP OK: Offset -0.0006300210953 secs [19:34:21] 6operations: Upload fixed version of Dec 2011 Editor Survey codebook - https://phabricator.wikimedia.org/T94455#1164193 (10Dzahn) 5Open>3Resolved the .csv with the original data is untouched. resolving [19:34:47] (03CR) 10Rush: [C: 032] "as discussed in the ops meeting today" [puppet] - 10https://gerrit.wikimedia.org/r/200651 (owner: 10Rush) [19:38:14] 6operations, 5Patch-For-Review: Setup poolcounter servers for codfw - https://phabricator.wikimedia.org/T93261#1164211 (10Dzahn) reinstalled. both are up again, new puppet certs etc. and have a software RAID now. /dev/md0 [19:38:26] 6operations, 3codfw-appserver-setup, 3wikis-in-codfw: Set up the mediawiki application layer in codfw - https://phabricator.wikimedia.org/T86894#1164221 (10Dzahn) [19:38:27] 6operations, 5Patch-For-Review: Setup poolcounter servers for codfw - https://phabricator.wikimedia.org/T93261#1164220 (10Dzahn) 5Open>3Resolved [19:41:47] (03CR) 10Mobrovac: "@Ottomata , you mention the argument of (MW|labs)-vagrant. I have a patch coming up (https://gerrit.wikimedia.org/r/#/c/195106) that adds " [puppet] - 10https://gerrit.wikimedia.org/r/196335 (https://phabricator.wikimedia.org/T92560) (owner: 10Eevans) [19:54:55] (03CR) 10Tim Landscheidt: [C: 04-1] "Not working yet." [puppet] - 10https://gerrit.wikimedia.org/r/200648 (https://phabricator.wikimedia.org/T93691) (owner: 10Tim Landscheidt) [19:57:42] (03CR) 10GWicke: [C: 031] Citoid: switch from localsettings.js to config.yaml [puppet] - 10https://gerrit.wikimedia.org/r/200356 (owner: 10Mobrovac) [19:59:58] 6operations, 10Wikidata-Query-Service: deploy haedus and capella with os for orientdb testing - https://phabricator.wikimedia.org/T84902#1164328 (10Dzahn) Should have a decom/reclaim ticket to follow-up with https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Reclaim_or_Decommission [20:00:04] gwicke, cscott, arlolra, subbu: Respected human, time to deploy Services – Parsoid / OCG / Citoid / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150330T2000). Please do the needful. [20:03:32] 6operations: reclaim / decom haedus and capella - https://phabricator.wikimedia.org/T94474#1164357 (10Dzahn) 3NEW [20:04:02] 6operations, 10Wikidata-Query-Service: deploy haedus and capella with os for orientdb testing - https://phabricator.wikimedia.org/T84902#933577 (10Dzahn) [20:04:04] 6operations: reclaim / decom haedus and capella - https://phabricator.wikimedia.org/T94474#1164369 (10Dzahn) [20:04:21] 6operations, 10Wikidata-Query-Service: deploy haedus and capella with os for orientdb testing - https://phabricator.wikimedia.org/T84902#933577 (10Dzahn) --> https://phabricator.wikimedia.org/T94474 [20:05:31] mutante: thx :) [20:09:28] !log deployed parsoid sha 29a5dafb [20:09:31] (03CR) 10Yuvipanda: [C: 031] Labs: collect conntrack stats for labsnet1001 [puppet] - 10https://gerrit.wikimedia.org/r/200619 (https://phabricator.wikimedia.org/T90437) (owner: 10coren) [20:09:32] Logged the message, Master [20:10:26] (03CR) 10coren: [C: 032] Labs: collect conntrack stats for labsnet1001 [puppet] - 10https://gerrit.wikimedia.org/r/200619 (https://phabricator.wikimedia.org/T90437) (owner: 10coren) [20:14:52] (03PS1) 10coren: Labs: *correctly* collect conntrack stats [puppet] - 10https://gerrit.wikimedia.org/r/200722 (https://phabricator.wikimedia.org/T90437) [20:16:05] (03CR) 10Yuvipanda: "*poke*" [puppet] - 10https://gerrit.wikimedia.org/r/197533 (owner: 10Chad) [20:16:42] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: puppet fail [20:18:44] (03PS1) 10Aude: Remove randomness param from change dispatcher script cronjob [puppet] - 10https://gerrit.wikimedia.org/r/200724 [20:19:49] (03PS1) 10Dzahn: dumps: add classes for zim dumps, add imagemagick [puppet] - 10https://gerrit.wikimedia.org/r/200725 (https://phabricator.wikimedia.org/T94457) [20:20:39] (03CR) 10jenkins-bot: [V: 04-1] dumps: add classes for zim dumps, add imagemagick [puppet] - 10https://gerrit.wikimedia.org/r/200725 (https://phabricator.wikimedia.org/T94457) (owner: 10Dzahn) [20:21:56] (03CR) 10coren: [C: 032] "Trivial fix." [puppet] - 10https://gerrit.wikimedia.org/r/200722 (https://phabricator.wikimedia.org/T90437) (owner: 10coren) [20:22:05] godog: around and have a moment? [20:22:09] or is it really late in europe? [20:22:13] (03PS2) 10Dzahn: dumps: add classes for zim dumps, add imagemagick [puppet] - 10https://gerrit.wikimedia.org/r/200725 (https://phabricator.wikimedia.org/T94457) [20:22:14] I think it’s really late in europe [20:22:21] (03CR) 10Ottomata: "Ha, well, that kinda defeats the purpose, doesn't it? If I wanted to develop and test a change to the cassandra puppet module that is use" [puppet] - 10https://gerrit.wikimedia.org/r/196335 (https://phabricator.wikimedia.org/T92560) (owner: 10Eevans) [20:22:40] * aude doesn't think it's really late :P [20:22:50] but to eath their own... [20:23:22] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:23:31] (03CR) 10Ottomata: "If you like, find me in IRC and we can try to resolve this more real time. :)" [puppet] - 10https://gerrit.wikimedia.org/r/196335 (https://phabricator.wikimedia.org/T92560) (owner: 10Eevans) [20:23:36] each* [20:24:39] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1164462 (10Dzahn) >For ZIM dumps, we'll additionally need other tools like imagemagick how to get a list of the other tools bes... [20:28:37] (03PS2) 10Dzahn: cassandra: firewall hole for port 9042 [puppet] - 10https://gerrit.wikimedia.org/r/200093 [20:31:13] (03CR) 10Dzahn: [C: 032] "what eevans said, it's an improvement over current and it's for test but should be improved by using server list from hiera" [puppet] - 10https://gerrit.wikimedia.org/r/200093 (owner: 10Dzahn) [20:33:16] 6operations, 6CA-team, 6Commons, 6MediaWiki-API-Team, 10SUL-Finalization: db1068 (s4/commonswiki slave) is missing data about at least 6 users - https://phabricator.wikimedia.org/T91920#1164532 (10Legoktm) [20:33:48] (03CR) 10Dzahn: "@eevans: on xenon: tcp dpt:9042" [puppet] - 10https://gerrit.wikimedia.org/r/200093 (owner: 10Dzahn) [20:37:56] YuviPanda: for you most guys are like +8 or more in EU [20:38:14] chasemp: yeah, going from having them be -3 to +8 is disconcerting [20:38:19] I HAVE LOST HALF MY TIME WHAT IS HAPPENING [20:38:23] err [20:38:24] TEAM [20:38:29] TIME also is appropriate maybe [20:38:40] you traveled back in time and these are teh consequnces [20:38:53] oh well [20:38:58] mobrovac: you gots nodejs foo, yes? [20:39:46] (03PS1) 10Arlolra: Fix 5xx retry for parsoid backend [puppet] - 10https://gerrit.wikimedia.org/r/200732 [20:39:49] (03PS1) 10RobH: adding wtp systems to site.pp [puppet] - 10https://gerrit.wikimedia.org/r/200734 (https://phabricator.wikimedia.org/T90271) [20:40:11] YuviPanda, now get used to getting up at a consistent time and something as horrible as sleeping during night time! :P [20:41:17] ottomata: euh, kinda [20:41:20] (03PS1) 10EBernhardson: Enable Flow on zhwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200735 (https://phabricator.wikimedia.org/T94387) [20:41:22] ottomata: need help? [20:41:50] mobrovac: um, yes, but i just made it work, but i have no idea why this works [20:41:58] haha [20:42:00] so [20:42:04] http://www.codeshare.io/aITwc [20:42:28] what i just did to make this work, is move the require('kafka-node') above the socketio requires [20:42:32] and then it magically works [20:42:39] if i do kafka = require... below that [20:42:44] nothing ever happens on change [20:42:47] (03PS1) 10EBernhardson: Enable autoconfirmed editing of flow posts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200736 (https://phabricator.wikimedia.org/T94278) [20:42:53] MaxSem: https://github.com/jnordberg/irccloudapp [20:43:44] ottomata: there's no kafka line in the upper ex [20:44:05] ottomata: maybe you don't have the kafka-node module installed ? [20:44:43] who are the right people to add here? https://gerrit.wikimedia.org/r/200732 [20:44:47] naw it works [20:44:49] i can produce manually [20:44:59] and yes mobrovac that is the weird part [20:45:03] the first example works, with no require [20:45:06] the second example does not work [20:45:29] IF require('kafka-node') happens AFTER require(socket.io-client) [20:47:12] ottomata: i'm betting that's because either kafka and/or socket.io mess with the global stream class somehow [20:47:30] hm. [20:47:46] ok, welp, i guess this works if I just do that require first. will keep poking at it, thanks mobrovac [20:47:51] lemme know if you want to talk about submodule :) [20:48:44] ottomata: that's gonna have to wait till tomorrow, you've caught me on the way out ;) [20:48:50] ok cool [20:50:42] PROBLEM - puppet last run on multatuli is CRITICAL: CRITICAL: puppet fail [20:51:04] ping urandom gwicke ^^ if you want to continue the discussion with ottomata re puppet/cassandra [20:51:25] haha [20:52:08] :P [20:54:53] (03PS2) 10Andrew Bogott: Make labs resolv.conf play nice with resolvconf [puppet] - 10https://gerrit.wikimedia.org/r/200595 (https://phabricator.wikimedia.org/T93691) [20:58:16] 6operations, 5Patch-For-Review, 3wikis-in-codfw: deploy wtp2001-2020 - https://phabricator.wikimedia.org/T90271#1164705 (10RobH) [21:00:05] aude: Dear anthropoid, the time has come. Please deploy Wikidata (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150330T2100).