[00:00:05] woah [00:00:06] numpy [00:00:38] http://www.numpy.org/ ? [00:04:00] bblack: yeah, just wasn’t expecting coal to use something that heavy [00:10:39] Coren_away: still around oh alright :) [00:12:21] Coren_away: I didn’t subscribe labs-announce to labs-l properly because I had to allow posts from labs-announce to labs-l :) You’re listed as only admin of labs-l... [00:21:51] 6operations, 10Parsoid, 6Services: Lets consider upgrading our Node.js installs to io.js (once decent Debian packages are ready) - https://phabricator.wikimedia.org/T91855#1214408 (10tstarling) Will Node.js continue to be supported for non-WMF installations? [00:26:50] (03PS1) 10Papaul: added asset tag mgmt dns for db2052-db2070 [dns] - 10https://gerrit.wikimedia.org/r/204682 [00:35:21] PROBLEM - Host mw2031 is DOWN: PING CRITICAL - Packet loss = 100% [00:35:41] RECOVERY - Host mw2031 is UPING WARNING - Packet loss = 54%, RTA = 43.68 ms [00:40:40] (03PS1) 10GWicke: Add sqlite3 and pixz utilities on html dumps host [puppet] - 10https://gerrit.wikimedia.org/r/204684 [00:42:34] 6operations, 10Parsoid, 6Services: Lets consider upgrading our Node.js installs to io.js (once decent Debian packages are ready) - https://phabricator.wikimedia.org/T91855#1214476 (10GWicke) @tstarling: Current versions, yes. [00:58:37] (03CR) 10Springle: "Fair enough. Note that I said 'untested' though, not undocumented. :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204443 (owner: 10Aaron Schulz) [01:02:53] (03PS1) 10Yuvipanda: dynamicproxy: Add ferm rules for http / https [puppet] - 10https://gerrit.wikimedia.org/r/204685 (https://phabricator.wikimedia.org/T96335) [01:13:02] (03PS1) 10Yuvipanda: tools: Explicitly open port for proxylistener [puppet] - 10https://gerrit.wikimedia.org/r/204688 (https://phabricator.wikimedia.org/T96335) [01:13:31] (03PS2) 10Yuvipanda: dynamicproxy: Add ferm rules for http / https [puppet] - 10https://gerrit.wikimedia.org/r/204685 (https://phabricator.wikimedia.org/T96335) [01:13:43] (03CR) 10Yuvipanda: [C: 032 V: 032] dynamicproxy: Add ferm rules for http / https [puppet] - 10https://gerrit.wikimedia.org/r/204685 (https://phabricator.wikimedia.org/T96335) (owner: 10Yuvipanda) [01:13:53] (03PS2) 10Yuvipanda: tools: Explicitly open port for proxylistener [puppet] - 10https://gerrit.wikimedia.org/r/204688 (https://phabricator.wikimedia.org/T96335) [01:18:08] (03CR) 10Yuvipanda: [C: 032] tools: Explicitly open port for proxylistener [puppet] - 10https://gerrit.wikimedia.org/r/204688 (https://phabricator.wikimedia.org/T96335) (owner: 10Yuvipanda) [01:24:14] (03PS1) 10Ori.livneh: PopUps: disabled by default; requires BetaFeatures if set as beta feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204689 [01:24:35] ok [01:24:45] so, popups will be disabled everywhere for a couple of minutes while i deploy this change [01:27:36] don’t you mean hovercards? [01:27:38] (03CR) 10Ori.livneh: [C: 032] PopUps: disabled by default; requires BetaFeatures if set as beta feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204689 (owner: 10Ori.livneh) [01:27:53] YuviPanda: hang on while I collec^H^H^H gather my thoughts [01:28:04] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1214628 (10GWicke) To test a full run before we set up a cron job, please perform a manual test run: ``` git clone https://github.com/gwicke/htmldumper.git cd htmldumper npm install nod... [01:28:15] https://wikitech.wikimedia.org/wiki/We_suck_at_naming should be more inclusive [01:28:39] https://wikitech.wikimedia.org/wiki/We_suck_at_naming_and_you_probably_do,_too [01:29:12] I just created that page too [01:29:20] heh [01:29:33] come on, jenkins [01:29:40] :D [01:30:29] (03CR) 10Ori.livneh: [V: 032] PopUps: disabled by default; requires BetaFeatures if set as beta feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204689 (owner: 10Ori.livneh) [01:31:10] !log marked as "Oversight" accounts as not to be renamed (utr_status=11) [01:31:14] Logged the message, Master [01:31:30] Thanks Lego [01:31:32] YuviPanda: greg ok'd a deploy, but I'd feel better if I knew I asked someone in ops to be a backup in case shit hits the fan. Can you be that guy? [01:31:41] (I don't think shit will hit the fan.) [01:31:44] sure. [01:31:47] thanks [01:31:51] * shit proceeds to hit fan [01:32:02] BUT THERE IS NO FAN [01:32:10] !log ori Synchronized wmf-config: I7fde63453: PopUps: disabled by default; requires BetaFeatures if set as beta feature (duration: 00m 11s) [01:32:14] Logged the message, Master [01:32:26] OK, step 1 went well [01:33:50] !log ori Synchronized php-1.26wmf2/extensions/Popups: Update Popups for Ie4cc455f: Act as a beta feature if so configured (duration: 00m 12s) [01:33:54] Logged the message, Master [01:34:27] !log ori Synchronized php-1.26wmf1/extensions/Popups: Update Popups for Ie4cc455f: Act as a beta feature if so configured (duration: 00m 12s) [01:34:31] Logged the message, Master [01:34:48] step 2 also went well [01:37:07] !log marked as "Steward" accounts as not to be renamed (utr_status=11) [01:37:12] Logged the message, Master [01:39:21] (03PS1) 10Ori.livneh: Popups: enable as beta feature by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204692 [01:39:56] (03CR) 10Ori.livneh: [C: 032] Popups: enable as beta feature by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204692 (owner: 10Ori.livneh) [01:40:51] (03CR) 10Ori.livneh: [V: 032] Popups: enable as beta feature by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204692 (owner: 10Ori.livneh) [01:41:15] ori: \o/ yay. also poke if you have a few mins after? I’ve a few ‘using statsd for useful metrics’ questions. [01:41:16] (if) [01:41:36] !log ori Synchronized wmf-config/InitialiseSettings.php: I95f8c010e: Popups: enable as beta feature by default (duration: 00m 12s) [01:41:39] sure, there are a few more steps tho :) [01:41:42] Logged the message, Master [01:41:44] !log paused forceRenameUsers around wm2008wiki [01:41:48] Logged the message, Master [01:42:12] ori: :) sure [01:42:16] * YuviPanda isn’t going anywhere [01:42:43] step 3 also went well [01:43:43] legoktm: kudos for !logging verbosely [01:43:46] :) [01:47:01] (03PS1) 10Yuvipanda: tools: Enable firewall on webproxies [puppet] - 10https://gerrit.wikimedia.org/r/204693 (https://phabricator.wikimedia.org/T96335) [01:47:22] (03PS4) 10Ori.livneh: Enable Hovercards by default on Catalan and Greek Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/197038 (https://phabricator.wikimedia.org/T88164) (owner: 10Werdna) [01:47:30] legoktm: could you do me a favor and force merge https://en.wikipedia.org/w/index.php?title=Special%3ACentralAuth&target=CheckUser and https://en.wikipedia.org/w/index.php?title=Special%3ACentralAuth&target=Checkuser ? to then be locked [01:47:32] (03CR) 10Yuvipanda: [C: 032] tools: Enable firewall on webproxies [puppet] - 10https://gerrit.wikimedia.org/r/204693 (https://phabricator.wikimedia.org/T96335) (owner: 10Yuvipanda) [01:47:40] none of those accounts are actually being used as a role account [01:47:50] (03CR) 10Yuvipanda: [V: 032] tools: Enable firewall on webproxies [puppet] - 10https://gerrit.wikimedia.org/r/204693 (https://phabricator.wikimedia.org/T96335) (owner: 10Yuvipanda) [01:48:05] (03CR) 10Ori.livneh: [C: 032 V: 032] Enable Hovercards by default on Catalan and Greek Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/197038 (https://phabricator.wikimedia.org/T88164) (owner: 10Werdna) [01:49:31] !log ori Synchronized wmf-config/InitialiseSettings.php: I6fa034f4a: Enable Hovercards by default on Catalan and Greek Wikipedias (T88164) (duration: 00m 12s) [01:49:38] Logged the message, Master [01:56:18] (03CR) 10Yuvipanda: "I'll note that we no longer use the ircecho package - it was one file + an init script, and I've moved those to the ops/puppet repo a whil" [debs/ircecho] - 10https://gerrit.wikimedia.org/r/204054 (owner: 10Muehlenhoff) [02:01:53] (03PS1) 10Yuvipanda: dynamicproxy: Explicitly open port for dynamicproxy-api [puppet] - 10https://gerrit.wikimedia.org/r/204694 [02:06:22] Eloquence: http://i.imgur.com/FiG1F91.png [02:07:28] ta [02:08:22] hi Eloquence; are you going to be leaving us?? [02:16:37] 6operations, 7Graphite: Counters now only provide rates (multiplied by 1000?) - https://phabricator.wikimedia.org/T95703#1214639 (10yuvipanda) Should we do a trial run on labs? :) [02:17:05] I'm off. legoktm, YuviPanda -- thanks again for your help w/this. [02:17:18] * YuviPanda is very good at sitting still and doing nothing [02:20:15] (03PS1) 10Yuvipanda: labmon: Enable extended statsite counters for labs [puppet] - 10https://gerrit.wikimedia.org/r/204695 (https://phabricator.wikimedia.org/T95703) [02:20:17] gwicke: ori ^ [02:20:31] hmm, I wonder if I should just wait for godog too [02:20:54] probably. [02:21:03] * CultivatingPatie nce [02:21:56] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Explicitly open port for dynamicproxy-api [puppet] - 10https://gerrit.wikimedia.org/r/204694 (owner: 10Yuvipanda) [02:25:22] !log l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 39s) [02:25:30] Logged the message, Master [02:27:12] (03PS1) 10Dzahn: lots of indentation fixes [puppet] - 10https://gerrit.wikimedia.org/r/204696 (https://phabricator.wikimedia.org/T93645) [02:29:45] !log LocalisationUpdate completed (1.26wmf1) at 2015-04-17 02:28:41+00:00 [02:29:49] Logged the message, Master [02:35:06] (03PS1) 10Yuvipanda: dynamicproxy: Include firewall for base proxy [puppet] - 10https://gerrit.wikimedia.org/r/204697 (https://phabricator.wikimedia.org/T96335) [02:35:08] (03PS1) 10Yuvipanda: dynamicproxy: Do not bind redis only on localhost [puppet] - 10https://gerrit.wikimedia.org/r/204698 (https://phabricator.wikimedia.org/T96335) [02:38:51] PROBLEM - puppet last run on mw2141 is CRITICAL Puppet has 1 failures [02:39:56] (03CR) 10GWicke: [C: 031] labmon: Enable extended statsite counters for labs [puppet] - 10https://gerrit.wikimedia.org/r/204695 (https://phabricator.wikimedia.org/T95703) (owner: 10Yuvipanda) [02:43:51] !log l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 10s) [02:43:59] Logged the message, Master [02:46:47] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Include firewall for base proxy [puppet] - 10https://gerrit.wikimedia.org/r/204697 (https://phabricator.wikimedia.org/T96335) (owner: 10Yuvipanda) [02:47:41] !log LocalisationUpdate completed (1.26wmf2) at 2015-04-17 02:46:38+00:00 [02:47:45] Logged the message, Master [02:55:12] RECOVERY - puppet last run on mw2141 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:08] (03PS1) 10Yuvipanda: dynamicproxy: Don't open up proxy only to internal users [puppet] - 10https://gerrit.wikimedia.org/r/204699 [02:57:00] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Do not bind redis only on localhost [puppet] - 10https://gerrit.wikimedia.org/r/204698 (https://phabricator.wikimedia.org/T96335) (owner: 10Yuvipanda) [02:57:31] (03PS2) 10Yuvipanda: dynamicproxy: Don't open up proxy api only to internal users [puppet] - 10https://gerrit.wikimedia.org/r/204699 [02:57:39] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Don't open up proxy api only to internal users [puppet] - 10https://gerrit.wikimedia.org/r/204699 (owner: 10Yuvipanda) [02:57:45] (03CR) 10Yuvipanda: [V: 032] dynamicproxy: Don't open up proxy api only to internal users [puppet] - 10https://gerrit.wikimedia.org/r/204699 (owner: 10Yuvipanda) [02:58:43] 6operations, 7HTTPS, 5HTTPS-by-default: Force all Wikimedia cluster traffic to be over SSL for all users (logged-in and anon) - https://phabricator.wikimedia.org/T49832#1214664 (10Tony_Tan_98) I see that the HTTPS infrastructure is now able to serve HTTPS-by-default (https://www.mediawiki.org/w/index.php?tit... [03:02:48] (03PS1) 10Yuvipanda: tools: Install redis-tools on trusty bastions [puppet] - 10https://gerrit.wikimedia.org/r/204700 [03:03:16] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Install redis-tools on trusty bastions [puppet] - 10https://gerrit.wikimedia.org/r/204700 (owner: 10Yuvipanda) [03:05:23] 6operations, 7HTTPS, 5HTTPS-by-default: Force all Wikimedia cluster traffic to be over SSL for all users (logged-in and anon) - https://phabricator.wikimedia.org/T49832#1214665 (10BBlack) With the completion of the scaling work, whether and when we flip the switch to force HTTPS for more (or all) wikis is mo... [03:12:44] (03PS1) 10Yuvipanda: tools: Allow redis access between proxies [puppet] - 10https://gerrit.wikimedia.org/r/204701 (https://phabricator.wikimedia.org/T96335) [03:13:56] (03CR) 10Yuvipanda: [C: 032] tools: Allow redis access between proxies [puppet] - 10https://gerrit.wikimedia.org/r/204701 (https://phabricator.wikimedia.org/T96335) (owner: 10Yuvipanda) [03:18:18] jamesofur: uh, we can't do merges yet [03:23:09] legoktm: errr, sorry I just want to attach the unattached accounts (all unused/not role accounts) to the corresponding global account [03:23:54] I don't need to actually merge the two accounts [03:24:05] I'm happy to just lock both global accounts :) [03:24:15] jamesofur: ah, just merge the unattached ones to the global? [03:24:18] yup [03:24:22] for both of those [03:25:40] !log attached Checkuser@enwiki to Checkuser@global [03:25:49] Logged the message, Master [03:26:27] !log attached CheckUser@dewiki,enwiki,metawiki to CheckUser@global [03:26:29] jamesofur: ^ [03:26:32] Logged the message, Master [03:26:38] thank ye [03:28:47] and both are locked [03:33:04] (03PS1) 10Yuvipanda: tools: Register only with 'active' proxy [puppet] - 10https://gerrit.wikimedia.org/r/204702 (https://phabricator.wikimedia.org/T96334) [03:33:56] !log restarting forceRenameUsers.php (SUL finalization) on the rest of the small wikis, starting with wm2008wiki [03:34:00] Logged the message, Master [03:34:07] (03CR) 10Yuvipanda: [C: 032] tools: Register only with 'active' proxy [puppet] - 10https://gerrit.wikimedia.org/r/204702 (https://phabricator.wikimedia.org/T96334) (owner: 10Yuvipanda) [04:04:50] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected [04:32:35] (03CR) 10Aaron Schulz: [C: 032] Use "groupLoadsBySection" for enwiki for consistency, just like s4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204443 (owner: 10Aaron Schulz) [04:32:43] (03Merged) 10jenkins-bot: Use "groupLoadsBySection" for enwiki for consistency, just like s4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204443 (owner: 10Aaron Schulz) [04:33:16] !log aaron Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 12s) [04:33:23] Logged the message, Master [04:35:16] (03PS1) 10Aaron Schulz: Set "recentchanges" group for s2-s7 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204704 [04:35:49] (03CR) 10Aaron Schulz: [C: 032] Set "recentchanges" group for s2-s7 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204704 (owner: 10Aaron Schulz) [04:35:54] (03Merged) 10jenkins-bot: Set "recentchanges" group for s2-s7 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204704 (owner: 10Aaron Schulz) [04:36:22] !log aaron Synchronized wmf-config/db-eqiad.php: Set "recentchanges" group for s2-s7 (duration: 00m 11s) [04:36:26] Logged the message, Master [04:49:11] PROBLEM - puppet last run on mw2090 is CRITICAL puppet fail [05:08:51] RECOVERY - puppet last run on mw2090 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [05:32:18] (03CR) 10Mxn: "These images have all been protected over at Commons." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [05:38:06] (03PS1) 10Legoktm: jenkins job validation, do not submit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204708 [05:38:42] (03Abandoned) 10Legoktm: jenkins job validation, do not submit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204708 (owner: 10Legoktm) [05:44:59] (03PS1) 10Legoktm: data_admin: Use yaml.safe_load() [puppet] - 10https://gerrit.wikimedia.org/r/204709 [05:48:13] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 17 05:47:10 UTC 2015 (duration 47m 9s) [05:48:18] Logged the message, Master [06:29:21] PROBLEM - puppet last run on labcontrol2001 is CRITICAL puppet fail [06:30:01] PROBLEM - puppet last run on mw1002 is CRITICAL puppet fail [06:30:01] PROBLEM - puppet last run on cp1061 is CRITICAL Puppet has 2 failures [06:31:11] PROBLEM - puppet last run on cp4014 is CRITICAL Puppet has 1 failures [06:31:53] PROBLEM - puppet last run on cp4003 is CRITICAL Puppet has 1 failures [06:32:11] PROBLEM - puppet last run on cp1056 is CRITICAL Puppet has 1 failures [06:32:12] PROBLEM - puppet last run on virt1006 is CRITICAL Puppet has 1 failures [06:32:51] PROBLEM - puppet last run on cp4004 is CRITICAL Puppet has 1 failures [06:32:52] PROBLEM - puppet last run on cp3014 is CRITICAL Puppet has 1 failures [06:35:41] PROBLEM - puppet last run on mw1099 is CRITICAL Puppet has 1 failures [06:35:50] PROBLEM - puppet last run on mw1092 is CRITICAL Puppet has 1 failures [06:36:21] PROBLEM - puppet last run on mw2206 is CRITICAL Puppet has 1 failures [06:36:30] PROBLEM - puppet last run on mw2045 is CRITICAL Puppet has 2 failures [06:36:30] PROBLEM - puppet last run on mw2066 is CRITICAL Puppet has 1 failures [06:45:30] RECOVERY - puppet last run on virt1006 is OK Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:45:40] RECOVERY - puppet last run on mw1099 is OK Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:45:41] RECOVERY - puppet last run on mw1092 is OK Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:46:11] RECOVERY - puppet last run on cp3014 is OK Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on mw2206 is OK Puppet is currently enabled, last run 1 second ago with 0 failures [06:46:30] RECOVERY - puppet last run on mw2045 is OK Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on mw2066 is OK Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on cp1061 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:46:50] RECOVERY - puppet last run on cp4003 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:01] RECOVERY - puppet last run on cp1056 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:31] RECOVERY - puppet last run on labcontrol2001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:41] RECOVERY - puppet last run on cp4014 is OK Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:47:42] RECOVERY - puppet last run on cp4004 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:10] RECOVERY - puppet last run on mw1002 is OK Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:20:11] (03CR) 10Muehlenhoff: [C: 031] "I don't see the strict need for dropping it (the original patch by Daniel had ripemd160, not ripemd), but with the preference in sha2 it's" [puppet] - 10https://gerrit.wikimedia.org/r/185329 (owner: 10Dzahn) [07:28:01] PROBLEM - very high load average likely xfs on ms-be1009 is CRITICAL - load average: 216.34, 133.80, 65.82 [07:38:34] (03CR) 10Alex Monk: [C: 04-1] "Looks like there's some details to clean up on the task." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/203783 (https://phabricator.wikimedia.org/T94842) (owner: 10Dereckson) [07:44:18] (03PS5) 10Alex Monk: Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [08:04:55] (03CR) 10Alex Monk: [C: 04-1] "Doesn't merge... Do we still want to do this?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/131914 (https://bugzilla.wikimedia.org/48618) (owner: 10Withoutaname) [08:06:45] (03CR) 10Muehlenhoff: [C: 032] "The additional change for precise doesn't change anything unless the client explicitly specifies a preference for aes256-ctr (and if it do" [puppet] - 10https://gerrit.wikimedia.org/r/185325 (owner: 10Dzahn) [08:09:19] !log reboot ms-be1009, xfs woes [08:09:26] Logged the message, Master [08:12:18] (03CR) 10Muehlenhoff: "I also checked Lucid (which is still in use on sodium) and it also supports all the ciphers in the pre-jessie/precise case." [puppet] - 10https://gerrit.wikimedia.org/r/185325 (owner: 10Dzahn) [08:17:05] _joe_: quick question re https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/service/manifests/node.pp;98ffd5c7f7f716402e79ed4290fae8db9762eadb$142 [08:17:25] Class['packages::nodejs']] != Package['nodejs'] ? [08:17:33] i.e. what's the diff [08:18:17] (03CR) 10Muehlenhoff: [C: 04-1] "Sorry, I hadn't thought of lucid before: It doesn't support sha2, so sshd would fail to start." [puppet] - 10https://gerrit.wikimedia.org/r/185329 (owner: 10Dzahn) [08:20:32] RECOVERY - very high load average likely xfs on ms-be1009 is OK - load average: 7.72, 2.40, 0.84 [08:21:31] <_joe_> mobrovac: look at the code of function require_package [08:21:40] k :) [08:21:47] <_joe_> or well [08:22:02] <_joe_> if you don't want to get your feet wet with puppet internals [08:22:10] <_joe_> (which are, of course, gross) [08:22:13] (03CR) 10Muehlenhoff: [C: 04-1] "Similar to my earlier comment wrt MACs we also need to cater for lucid:" [puppet] - 10https://gerrit.wikimedia.org/r/185321 (owner: 10Dzahn) [08:22:23] (03CR) 10Alexandros Kosiaris: [C: 032] package_builder: Reflect rename of /etc/apt/preferences.d/wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/204673 (owner: 10Tim Landscheidt) [08:22:42] <_joe_> what we do is declare a class packages::$package_name.gsub('-','_') [08:22:51] <_joe_> where the package is included [08:23:16] there's also host_scope.function_create_resources(['package', { package_name => { :ensure => :present } }]) [08:23:20] <_joe_> since classes are floating, you're not guaranteed that the class would be included before the resource needing the package [08:23:24] so it seems to me both are created [08:23:43] <_joe_> yeah we're abusing puppet internals, and puppet is not smart enough to find that dependency [08:24:06] too smart for puppet [08:24:09] godog: extended counter metrics might be the solution to my problem :) I added a patch to enable that just for beta [08:24:09] muaaawh [08:24:12] <_joe_> I've debugged this and once I thought I had a fix, which meant re-writing like 10% of the parser [08:24:23] Going to go sleep soon tho and my laptop is in the office... [08:24:39] <_joe_> mobrovac: so we go with requiring the class :P [08:24:59] _joe_: rewriting the parser would have been more fun though :D [08:25:46] <_joe_> mobrovac: people at puppetlabs thought the same, they're rewriting puppet in clojure [08:25:57] <_joe_> (and tons of jruby of course) [08:26:02] really? hehe [08:26:19] jruby? mh [08:26:54] rubinius is better imho [08:28:18] <_joe_> well I think of joining the nice and clean syntax of ruby with the performance, stability and maintainability of the JVM [08:28:30] <_joe_> it's like you get the best of both worlds [08:29:12] <_joe_> a bit like the Alfa Romeo Arna. A car built with an Alfa - Nissan joint venture. Where Alfa put the technology and Nissan the design [08:29:16] The nice clean syntax of java and the performance and stability of ruby? [08:29:55] <_joe_> YuviPanda: I find ruby horrible and dumbed down step child of perl [08:30:13] oh now _joe_ [08:30:20] <_joe_> :D [08:30:24] <_joe_> gotcha [08:30:25] Heh I've never written it but I have been planning to spend a few days on it [08:30:41] Anything that gets hated and loved with a passion seems worth exploring [08:30:46] <_joe_> mobrovac: you're coding in JAVASCRIPT, even erlang would look fine to you right now :P [08:30:53] perl is super-nice, but when people start abusing it and using it for *all tasks* it becomes crazy [08:31:01] _joe_: touché [08:31:25] * YuviPanda continues standing barefoot in kitchen eating ice cream [08:31:27] YuviPanda: it is worth [08:31:56] <_joe_> YuviPanda: so write a nodejs app with a mongodb backend to do the cluster orchestrator for toollabs [08:32:08] <_joe_> < YuviPanda> Anything that gets hated and loved with a passion seems worth exploring [08:32:14] (03CR) 10Alexandros Kosiaris: [C: 04-2] "This will not work. Gerrit can not listen on a privileged port (<1024). Only root processes can listen on privileged ports. After which th" [puppet] - 10https://gerrit.wikimedia.org/r/172313 (https://bugzilla.wikimedia.org/35611) (owner: 10Dereckson) [08:32:34] at $JOB-1 we had a whole inventory/monitoring/billing/etc/etc system written exclusively in perl [08:32:35] <_joe_> I don't agree, it strongly depends on who hates it and who loves it :P [08:33:02] <_joe_> mobrovac: oh gee, I once had to extend sql-ledger [08:33:12] (and that was a hosting company, mind you) [08:33:36] perl + TT templating FTW [08:34:02] (03CR) 10Filippo Giunchedi: [C: 031] "remember this means renaming non-extended counters to not lose previous data, happy to assist with that" [puppet] - 10https://gerrit.wikimedia.org/r/204695 (https://phabricator.wikimedia.org/T95703) (owner: 10Yuvipanda) [08:34:28] YuviPanda: hehe still up? saw the patch I think we're fine trying it labs, need to check the numbers in production if disk space fits [08:34:45] godog: yeah I am but have no laptop (is in office) [08:34:51] Went to karaoke and got late [08:35:15] godog: wanna merge and shepherd it through? I'll owe you a drink of choice if you do :) [08:35:15] pics or didn't happen [08:35:32] _joe_: hehe idk if that is worse than gridengine :p [08:36:05] godog: heh, no barbie girl tho. Just bohemian rhapsody and dome other risque songs [08:36:13] haha [08:36:20] YuviPanda: what about renaming existing counters tho? [08:36:39] You did offer to help with that ;) [08:37:18] it would be possible to whitelist *all* domains on commons? upload_by_url is rescricted (only trusted users)? [08:37:24] * Steinsplitter pokes YuviPanda [08:37:43] Wondering if there is some sort of security risk [08:38:10] From labs to prod? [08:38:22] from www --> commons [08:38:28] Oh [08:38:47] Well I dont know much about that :( poke someone in the multimedia team maybe? [08:39:00] _joe_: in your expert opinion, should https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/wmflib/lib/puppet/parser/functions/array_concat.rb;95c8b418f1d69cb917359d384a7c245984e55541$22 be turned into "elsif !args.nil?" ? In mine, it should [08:39:03] YuviPanda: fair enough, I did alright [08:39:12] ok yuvi [08:39:14] thx [08:39:18] s/args/arg [08:40:10] godog: :) [08:42:56] Anyway. Bedtime. Night [08:43:29] bye bye YuviPanda [08:49:44] (03CR) 10Hashar: [C: 031] "Gotta rebase :( But all fine to me, that is mostly tabs -> space replacements and arrows alignments." [puppet] - 10https://gerrit.wikimedia.org/r/204696 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [08:55:32] pff [08:55:36] stupid cluster wide dependencies [08:55:44] python-requests 2.0.0 got pushed to Precise :/ [08:58:53] <_joe_> hashar: what? [08:59:11] (03PS1) 10Alexandros Kosiaris: Issue new certificate for virt-star [puppet] - 10https://gerrit.wikimedia.org/r/204718 (https://phabricator.wikimedia.org/T96291) [08:59:12] python-requests: [08:59:12] Installed: 0.8.2-1 [08:59:12] Candidate: 2.0.0-1~precise+1 [08:59:14] on Precise [08:59:28] <_joe_> is that on apt.wm.org? [08:59:41] yes [08:59:45] luckily I have no impact [09:00:01] since barely any jobs are still running on Precise and Zuul doesn't depend on python-requests [09:00:59] (03PS2) 10Alexandros Kosiaris: Issue new certificate for virt-star [puppet] - 10https://gerrit.wikimedia.org/r/204718 (https://phabricator.wikimedia.org/T96291) [09:01:08] _joe_: forget me, noop for CI anyway :) [09:01:29] the jobs depending on python-requests are all running on Trusty which has requests v 2.2.1 [09:02:37] !log apt-get upgrade on gallium and lanthanum [09:02:43] Logged the message, Master [09:05:31] PROBLEM - DPKG on gallium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [09:05:39] ^ me [09:06:08] (03PS1) 10Mobrovac: service::node: Allow extra packages to be installed [puppet] - 10https://gerrit.wikimedia.org/r/204721 [09:06:24] akosiaris: _joe_: ^^ [09:07:11] RECOVERY - DPKG on gallium is OK: All packages OK [09:08:46] (03PS2) 10Filippo Giunchedi: labmon: Enable extended statsite counters for labs [puppet] - 10https://gerrit.wikimedia.org/r/204695 (https://phabricator.wikimedia.org/T95703) (owner: 10Yuvipanda) [09:08:54] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] labmon: Enable extended statsite counters for labs [puppet] - 10https://gerrit.wikimedia.org/r/204695 (https://phabricator.wikimedia.org/T95703) (owner: 10Yuvipanda) [09:09:11] godog: yay thanks [09:09:32] (03PS5) 10Hashar: zuul: switch install to a Debian package [puppet] - 10https://gerrit.wikimedia.org/r/202714 (https://phabricator.wikimedia.org/T48552) [09:10:10] (03PS1) 10KartikMistry: CX: Enable Content Translation in given wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204722 (https://phabricator.wikimedia.org/T95848) [09:11:14] YuviPanda: that didn't even work :( I'll fix [09:13:30] zuul: postinst called with unknown argument `triggered' [09:13:31] bah [09:26:09] (03PS1) 10KartikMistry: CX: Enable Content Translation in given wikis [puppet] - 10https://gerrit.wikimedia.org/r/204725 (https://phabricator.wikimedia.org/T95848) [09:30:48] (03PS1) 10Filippo Giunchedi: statsite: allow instance settings to be fetched from hiera [puppet] - 10https://gerrit.wikimedia.org/r/204727 [09:31:11] (03CR) 10KartikMistry: [C: 04-1] "Planned for 21/04 deployment. Do not merge before that :)" [puppet] - 10https://gerrit.wikimedia.org/r/204725 (https://phabricator.wikimedia.org/T95848) (owner: 10KartikMistry) [09:33:40] (03CR) 10Hashar: "I gave the Zuul server upgrade a try this morning on a labs instance and there is some manual steps required when switching." [puppet] - 10https://gerrit.wikimedia.org/r/202714 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [09:42:29] (03PS1) 10Aude: Add subscriptionLookupMode setting for wikidata + testwikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204728 [09:48:52] (03CR) 10Mark Bergsma: [C: 031] "Looks a lot more sensible." [debs/pybal] - 10https://gerrit.wikimedia.org/r/204035 (owner: 10Giuseppe Lavagetto) [09:56:27] (03CR) 10Alexandros Kosiaris: [C: 032] Reimage copper as jessie with role::package::builder [puppet] - 10https://gerrit.wikimedia.org/r/203043 (owner: 10Alexandros Kosiaris) [09:56:37] (03PS2) 10Alexandros Kosiaris: Reimage copper as jessie with role::package::builder [puppet] - 10https://gerrit.wikimedia.org/r/203043 [09:56:48] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Reimage copper as jessie with role::package::builder [puppet] - 10https://gerrit.wikimedia.org/r/203043 (owner: 10Alexandros Kosiaris) [10:06:04] (03CR) 10Giuseppe Lavagetto: [C: 031] statsite: allow instance settings to be fetched from hiera [puppet] - 10https://gerrit.wikimedia.org/r/204727 (owner: 10Filippo Giunchedi) [10:08:09] (03PS2) 10Filippo Giunchedi: statsite: allow instance settings to be fetched from hiera [puppet] - 10https://gerrit.wikimedia.org/r/204727 [10:08:18] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] statsite: allow instance settings to be fetched from hiera [puppet] - 10https://gerrit.wikimedia.org/r/204727 (owner: 10Filippo Giunchedi) [10:12:20] PROBLEM - configured eth on copper is CRITICAL: Connection refused by host [10:12:31] PROBLEM - dhclient process on copper is CRITICAL: Connection refused by host [10:12:40] PROBLEM - RAID on copper is CRITICAL: Connection refused by host [10:12:41] PROBLEM - Disk space on copper is CRITICAL: Connection refused by host [10:12:51] PROBLEM - DPKG on copper is CRITICAL: Connection refused by host [10:13:10] PROBLEM - salt-minion processes on copper is CRITICAL: Connection refused by host [10:13:11] PROBLEM - puppet last run on copper is CRITICAL: Connection refused by host [10:22:49] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "I'm not sure this is the correct approach. I'd prefer to have a module (or packages declared in roles) as it's cleaner." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/204721 (owner: 10Mobrovac) [10:23:37] (03PS1) 10Alexandros Kosiaris: squid3: use extra_modules in Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/204737 [10:40:58] (03CR) 10Alexandros Kosiaris: [C: 032] squid3: use extra_modules in Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/204737 (owner: 10Alexandros Kosiaris) [10:44:30] PROBLEM - DPKG on copper is CRITICAL: Connection refused by host [10:44:41] PROBLEM - Disk space on copper is CRITICAL: Connection refused by host [10:45:01] PROBLEM - RAID on copper is CRITICAL: Connection refused by host [10:45:23] 6operations, 7Graphite, 5Patch-For-Review: Counters now only provide rates (multiplied by 1000?) - https://phabricator.wikimedia.org/T95703#1214989 (10fgiunchedi) extended counters have been enabled on labmon1001, production will be harder because for each counter that means a 7x increase current counters u... [10:45:30] PROBLEM - configured eth on copper is CRITICAL: Connection refused by host [10:45:40] PROBLEM - dhclient process on copper is CRITICAL: Connection refused by host [10:45:51] PROBLEM - puppet last run on copper is CRITICAL: Connection refused by host [10:46:02] PROBLEM - salt-minion processes on copper is CRITICAL: Connection refused by host [10:46:37] (03PS1) 10Alexandros Kosiaris: base: bash.bashrc ERB variable qualifiers [puppet] - 10https://gerrit.wikimedia.org/r/204740 [10:46:39] (03PS1) 10Alexandros Kosiaris: base: be pedantic about template filenames [puppet] - 10https://gerrit.wikimedia.org/r/204741 [10:47:05] 6operations, 7Graphite, 5Patch-For-Review: Counters now only provide rates (multiplied by 1000?) - https://phabricator.wikimedia.org/T95703#1214993 (10fgiunchedi) >>! In T95703#1214989, @fgiunchedi wrote: > so let's say 45k counters give or take, that's another ~300G which we don't have ATM. that's not stri... [11:03:52] (03PS1) 10Faidon Liambotis: Revert "Depool esams, network backhaul issues" [dns] - 10https://gerrit.wikimedia.org/r/204743 [11:04:00] (03PS2) 10Faidon Liambotis: Revert "Depool esams, network backhaul issues" [dns] - 10https://gerrit.wikimedia.org/r/204743 [11:05:14] (03PS1) 10Alexandros Kosiaris: WIP: Create a shim module for citoid around service::node [puppet] - 10https://gerrit.wikimedia.org/r/204744 [11:06:11] (03CR) 10Faidon Liambotis: [C: 032] Revert "Depool esams, network backhaul issues" [dns] - 10https://gerrit.wikimedia.org/r/204743 (owner: 10Faidon Liambotis) [11:10:35] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "This is overall my idea, so LGTM. A small comment on the class." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/204744 (owner: 10Alexandros Kosiaris) [11:11:01] RECOVERY - Disk space on copper is OK: DISK OK [11:11:20] RECOVERY - RAID on copper is OK Active: 4, Working: 4, Failed: 0, Spare: 0 [11:11:41] RECOVERY - configured eth on copper is OK - interfaces up [11:11:51] RECOVERY - dhclient process on copper is OK: PROCS OK: 0 processes with command name dhclient [11:12:20] RECOVERY - salt-minion processes on copper is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [11:12:21] RECOVERY - DPKG on copper is OK: All packages OK [11:12:41] PROBLEM - NTP on copper is CRITICAL: NTP CRITICAL: Offset unknown [11:15:21] RECOVERY - puppet last run on copper is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [11:15:22] (03CR) 10Mobrovac: "On the one hand, it seems wasteful to some extent to have a module and put two lines inside of it. On the other, it sure does solve the pr" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/204744 (owner: 10Alexandros Kosiaris) [11:21:01] RECOVERY - NTP on copper is OK: NTP OK: Offset -0.01069152355 secs [11:22:08] (03PS1) 10ArielGlenn: disable rate limits on html dumps backend, proxy has them [puppet] - 10https://gerrit.wikimedia.org/r/204745 [11:31:30] 6operations: Encrypted password storage - https://phabricator.wikimedia.org/T96130#1215023 (10MoritzMuehlenhoff) > Off topic: can you link your issued WMF wiki account to this phab one? https://www.mediawiki.org/wiki/Phabricator/Help#Creating_your_account > > We should be able to sort out the nda membership th... [11:31:59] (03CR) 10Alexandros Kosiaris: "After talking with mobrovac and _joe_ on IRC, I posted an alternative approach here https://gerrit.wikimedia.org/r/204744" [puppet] - 10https://gerrit.wikimedia.org/r/204721 (owner: 10Mobrovac) [11:34:39] (03Abandoned) 10Muehlenhoff: Add a systemd unit file (Bug: T95055) [debs/ircecho] - 10https://gerrit.wikimedia.org/r/204054 (owner: 10Muehlenhoff) [11:35:03] (03Abandoned) 10Muehlenhoff: Simplify package build, also the stepping stone for adding a systemd unit file (Bug: T95055) [debs/ircecho] - 10https://gerrit.wikimedia.org/r/204045 (owner: 10Muehlenhoff) [11:36:27] (03CR) 10ArielGlenn: [C: 032] disable rate limits on html dumps backend, proxy has them [puppet] - 10https://gerrit.wikimedia.org/r/204745 (owner: 10ArielGlenn) [11:38:21] PROBLEM - puppet last run on copper is CRITICAL puppet fail [11:43:04] 6operations, 10MediaWiki-General-or-Unknown: img_metadata queries for Djvu files regularly saturate s4 slaves - https://phabricator.wikimedia.org/T96360#1215048 (10faidon) 3NEW [11:54:52] RECOVERY - puppet last run on copper is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [11:55:10] (03PS1) 10Tim Landscheidt: Labs: Create /etc/motd unconditionally [puppet] - 10https://gerrit.wikimedia.org/r/204748 (https://phabricator.wikimedia.org/T85307) [11:56:26] (03CR) 10Faidon Liambotis: [C: 04-2] "Eww, no. Let's fix the PAM config instead, cf. T85910." [puppet] - 10https://gerrit.wikimedia.org/r/204748 (https://phabricator.wikimedia.org/T85307) (owner: 10Tim Landscheidt) [12:01:29] (03Abandoned) 10Mobrovac: service::node: Allow extra packages to be installed [puppet] - 10https://gerrit.wikimedia.org/r/204721 (owner: 10Mobrovac) [12:10:33] (03CR) 10Alexandros Kosiaris: [C: 032] zuul: switch install to a Debian package [puppet] - 10https://gerrit.wikimedia.org/r/202714 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [12:10:45] akosiaris: grazie mille [12:10:49] or something like that [12:11:14] hashar: de rien [12:13:36] 7Blocked-on-Operations, 6operations, 5Patch-For-Review: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1215090 (10ArielGlenn) I tweaked the puppet manifest for nginx on francium a little, no big deal. GWicke: I am ready to put some dumps from the labs ins... [12:14:02] lets switch zuul to the debian package *evil* [12:14:19] !log Switching Zuul scheduler on gallium.wikimedia.org to the Debian package version [12:14:23] Logged the message, Master [12:15:26] (03PS1) 10Alexandros Kosiaris: package_builder: Use mirrors.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/204750 [12:16:06] (03CR) 10Tim Landscheidt: "This isn't meant as a /replacement/ for fixing T85910, but as a temporary workaround until that is fixed (I forgot to make the commit mess" [puppet] - 10https://gerrit.wikimedia.org/r/204748 (https://phabricator.wikimedia.org/T85307) (owner: 10Tim Landscheidt) [12:22:27] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] package_builder: Use mirrors.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/204750 (owner: 10Alexandros Kosiaris) [12:22:58] (03CR) 10Alexandros Kosiaris: [C: 032] base: bash.bashrc ERB variable qualifiers [puppet] - 10https://gerrit.wikimedia.org/r/204740 (owner: 10Alexandros Kosiaris) [12:23:31] (03CR) 10Alexandros Kosiaris: [C: 032] base: be pedantic about template filenames [puppet] - 10https://gerrit.wikimedia.org/r/204741 (owner: 10Alexandros Kosiaris) [12:25:31] PROBLEM - zuul_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-server [12:26:22] ah damn [12:26:29] forgot to update the icinga monitor [12:26:50] PROBLEM - zuul_merger_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-merger [12:27:24] !log Zuul should be back up now [12:27:25] (03Abandoned) 10Tim Landscheidt: Fix motd on Trusty instances [puppet] - 10https://gerrit.wikimedia.org/r/181789 (https://phabricator.wikimedia.org/T85307) (owner: 10Tim Landscheidt) [12:27:32] Logged the message, Master [12:28:01] PROBLEM - puppet last run on copper is CRITICAL puppet fail [12:29:41] RECOVERY - puppet last run on copper is OK Puppet is currently enabled, last run 13 seconds ago with 0 failures [12:31:56] (03PS1) 10Hashar: zuul: update monitoring regex check [puppet] - 10https://gerrit.wikimedia.org/r/204753 [12:32:23] (03CR) 10Hashar: "I have ran both checks manually on gallium:" [puppet] - 10https://gerrit.wikimedia.org/r/204753 (owner: 10Hashar) [12:32:39] https://gerrit.wikimedia.org/r/204753 would fix the icinga errors for zuul on gallium [12:33:24] ? /usr/share/python/zuul/bin/python ??? [12:33:33] that is bad path naming ... [12:33:36] yeah [12:33:37] :( [12:33:49] since we have a bunch of python modules missing [12:34:04] I came up with a deb package that provides the missing deps in a venv [12:34:14] which is under /usr/share/python/zuul [12:34:29] and to have python to lookup the modules there, it has to be a binary :( [12:34:47] if we had a symlink or used /usr/bin/python that would not give us the modules in the venv :/ [12:34:50] we tried! [12:35:09] yeah, now that I read the commit message better it makes sense [12:35:33] there is a reason I am not in love with venvs... [12:35:44] (03CR) 10Alexandros Kosiaris: [C: 032] zuul: update monitoring regex check [puppet] - 10https://gerrit.wikimedia.org/r/204753 (owner: 10Hashar) [12:36:04] that nicely solve the issue of having different soft depending on different libs [12:36:26] but it does not solve the security updating problem [12:36:37] actually it makes it worse... many times worse [12:37:07] but if you are going to be updating that venv often, I am ok with it [12:37:35] aim is to get rid of it eventually [12:37:42] and only use plain packages [12:37:48] I need to get rid of Precise though [12:39:52] and now that copper is online I can return on https://gerrit.wikimedia.org/r/#/c/203073/6/manifests/role/ci.pp,cm and figure out what is going on there [12:40:08] ah yeah that one is a mess :/ [12:40:41] (03CR) 10Alexandros Kosiaris: [C: 032] Kill the old unused package-builder manifests [puppet] - 10https://gerrit.wikimedia.org/r/203040 (owner: 10Alexandros Kosiaris) [12:46:37] akosiaris: I will not miss that old one. Was probably the most evil one we had floating around [12:46:57] lol [12:47:06] have you seen the note at the top of the file? [12:47:22] it has been used to interview candidates apparently [12:47:34] see my commit message [12:47:39] It is not used anywhere and we pretty sure no longer use it for [12:47:39] screening [12:47:58] evil screening! [12:48:11] RECOVERY - zuul_merger_service_running on gallium is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [12:48:11] btw I am trying to apply role::ci::labs on a jessie VM host [12:48:31] RECOVERY - zuul_service_running on gallium is OK: PROCS OK: 2 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-server [12:48:34] no, role::ci::slave::labs [12:48:49] that one brings in all the mediawiki::packages and related class [12:48:55] so it takes a couple hours or so [12:49:04] I have one setup alredy if you want [12:49:23] integration-slave-jessie-1001.eqiad.wmflabs [12:49:29] what I did to play test the patch is something like: [12:49:52] dpkg --purge pbuilder cowbuilder ; rm -fR /var/cache/pbuilder /mnt/pbuilder [12:50:06] then apply the patch on integration-puppetmaster.eqiad.wmflabs [12:50:12] and run puppet agent -tv to see what happens [12:50:32] if you do the purge / rm and run puppet [12:50:40] you should see cowbuilder / pbuidler being installed [12:50:43] then the cow images created [12:50:45] hmm, thanks that is actually a good idea [12:50:48] and the mount occuring AFTER [12:51:05] btw, I just got the class included on a jessie host [12:51:09] it is not exactly applying ... [12:51:20] it reports tons of errors [12:51:28] then on the next puppet run, it fails because it cant symlink /var/cache/pbuilder because it has some content [12:51:36] yeah bunch of errors are related to missing fonts packages [12:51:41] and ruby [12:51:47] and other obsolete softwares like ruby 1.8 [12:51:48] and ffmpeg [12:51:54] yup [12:51:54] and and and ... [12:51:56] ok [12:51:57] Jessie went with libav [12:52:02] good that it is known [12:52:26] I have filled a task about it and faidon did a first pass on them. We have a bit more work to do to be Jessie compliant [12:52:46] anytime I see a puppet error either I fix it in ops/puppet or report it [13:08:50] 7Blocked-on-Operations, 6operations, 10Continuous-Integration, 5Continuous-Integration-Isolation, and 2 others: Create a Debian package for Zuul - https://phabricator.wikimedia.org/T48552#1215158 (10hashar) 5Open>3Resolved We now have Zuul packages for our Precise and Trusty distributions thanks to @fg... [13:10:21] (03PS8) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [13:10:23] (03PS3) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [13:10:25] (03PS8) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [13:10:27] (03PS1) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [13:10:30] (03PS1) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [13:14:18] (03CR) 10Aude: "I'm not sure *dropbox is a good idea, since that is quite broad, or at least prefer not to be the person that approves this." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204485 (owner: 10Steinsplitter) [13:21:13] (03Abandoned) 10Steinsplitter: Whitelisting *.dropbox.com for GWToolset upload. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204485 (owner: 10Steinsplitter) [13:22:50] PROBLEM - puppet last run on ms-be2015 is CRITICAL Puppet has 1 failures [13:25:24] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215214 (10JohnLewis) a:5JohnLewis>3None [13:25:39] (03PS2) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [13:25:41] (03PS9) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [13:25:43] (03PS2) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [13:25:45] (03PS4) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [13:25:47] (03PS9) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [13:25:49] (03PS1) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [13:31:51] 6operations, 7HTTPS, 5Patch-For-Review: Replace SHA1 certificates with SHA256 - https://phabricator.wikimedia.org/T73156#1215262 (10Aklapper) So we're only missing T92709 here to close this? Or is T91504 also a dependency? [13:32:54] aude: what if we whitelist the domain for two hours or so to allow uploads. that schouldn't be a big deal? [13:33:28] maybe [13:34:12] * aude doesn't think i should be the one to approve, but that might be acceptable [13:34:51] * Steinsplitter pokes hashar [13:34:54] :) [13:35:04] what's the purpose of the whitelist for upload limitation in the first place? [13:35:05] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215272 (10Dzahn) a:3Dzahn @vshchepakina It works for me now! Check out http://store.wikimedia.org/ it is not a redirect anymore but i can see the sh... [13:35:15] for glam ? [13:35:16] Steinsplitter: aude: hi [13:35:22] bblack: yes, for the GlamWikiToolset [13:35:22] We can temporary approve, sure [13:35:33] I mean, I have my guesses, but I'm curious what the explicit reason is for the restriction in this case [13:35:50] there is no way we are going to whitelist dropbox.com for sure [13:36:04] bblack: I think it's mostly so that people don't screw to much with the extension and to restrict what we pull into the cluster [13:36:14] it's restricted permission wise for gwtoolset [13:36:14] see that sounds very wishy-washy :) [13:36:19] the glam tool is used for MASSIVE bulk imports of data from libraries museum etc [13:36:32] but think this is another protection [13:36:39] and the whitelist is validated by a team of volunteers which are in contacts with the libraries / museum [13:36:55] ---> https://commons.wikimedia.org/wiki/Commons:Upload_tools/wgCopyUploadsDomains [13:36:58] "so people don't screw around too much" is very vague. Something like "the way this is built, someone could load horrible remote code into our system from these domains" would be a better reason [13:37:02] aude: Do you know how much data we want to suck in [13:37:04] ? [13:37:04] as I understand it, the idea of the whitelist is to sign-off the fact the third party agree about commons terms and licenses [13:37:11] hoo: i think ~500 [13:37:12] that is driven by NL chapter iirc [13:37:14] 500 files [13:37:16] aude: GiB? [13:37:16] * aude could easily do it with pywikibot [13:37:25] but since it's ready for gwtoolset then that's good [13:37:28] also, the current last entry in that list: '*.adlibhosting.com', // Amsterdam Museum, York Museums Trust, etc [13:37:29] and what it's intended for [13:37:33] hoo: don't know [13:37:38] is that not unlike dropbox in that it's a whole hosting service? [13:37:44] what kind of files do you want added ? [13:37:47] aude: I could just pull it onto a personal server of mine or so and then they can upload [13:37:52] or we could do server side upload [13:37:53] hoo: yeah, thought of that [13:37:53] or whatever [13:38:03] or copy them to something wmde has [13:38:05] what we have done so far is to use the import maintenance script [13:38:15] I did that a lot, yes [13:38:17] it's easy [13:38:47] yeah [13:39:00] we barely have doc though beside https://wikitech.wikimedia.org/wiki/Uploading_large_files [13:39:20] RECOVERY - puppet last run on ms-be2015 is OK Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:39:38] ~1mb per file [13:39:39] oh! it was just the domainname sounded suspiciously like a random hosting provider to me. in fact adlibhosting is not such a thing :) [13:39:46] you would need a .txt file associated with each media file that get the wikitext to be put on the File: as a comment [13:40:13] * aude checked the adlibhosting thing :) [13:40:16] one day maybe we will have an /incoming/ ftp server that would let folks manually validate / approve the uploads :) [13:40:19] was also suspicious [13:40:35] wondering why we have this kind of whitelist... is it fore security reason? [13:40:45] Steinsplitter: assume so [13:41:02] * aude doesn't even have permission for gwtoolset on commons [13:41:06] even though i am admin [13:41:10] so don't know a lot about it [13:42:22] I also don't really know about it [13:42:27] but I approved domains in the past [13:42:29] it's no big deal [13:42:32] usually [13:42:33] pywikibot is something like https://github.com/filbertkm/toolserver/blob/master/aaa.py [13:42:42] the script is ancient but probably easy to adapt [13:42:58] really not difficult :) [13:43:16] aude: Yeah... we have various options [13:43:20] can be done with non-bot account if it's throttled enough [13:43:24] and supervised [13:43:25] do you know who of WMDE wnated to do this? [13:43:27] (03CR) 10Andrew Bogott: [C: 032] Issue new certificate for virt-star [puppet] - 10https://gerrit.wikimedia.org/r/204718 (https://phabricator.wikimedia.org/T96291) (owner: 10Alexandros Kosiaris) [13:43:29] Are they on IRC? [13:43:34] hoo: i do and can talk to them on monday [13:43:44] Ok, but not today? [13:43:45] not sure they are still around today [13:43:47] Nico WMDE [13:43:50] yeah [13:44:00] * aude was going to chat with them on monday [13:44:06] Still online according to IRC, let's see [13:44:09] ok [13:44:14] hoo: if you are familliar with maintenance/ImportImages.php please give a shot at https://wikitech.wikimedia.org/wiki/Uploading_large_files :) any bit or reference link would help [13:45:00] hashar: I think using --help is just the best to do [13:45:22] Could add documentation on how to use the proxy to load stuff onto tin/terbium, though, I guess [13:45:56] (03PS3) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [13:45:58] (03PS10) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [13:46:00] (03PS3) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [13:46:02] (03PS5) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [13:46:04] (03PS2) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [13:46:06] (03PS10) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [13:46:08] (03PS1) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [13:46:10] Steinsplitter: another option is to get these files somewhere onto toollabs [13:46:10] (03PS1) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [13:46:18] or labs in general [13:46:28] 6operations, 10ops-fundraising: DNS for betelgeuse and beryllium - https://phabricator.wikimedia.org/T96365#1215305 (10Jgreen) 3NEW a:3Jgreen [13:46:29] afaik, subdomains can be allotted for things there [13:46:51] cold be a general purpose project for these things [13:46:57] could* [13:48:59] aude: does not work, because of the webproxy [13:49:05] Steinsplitter: ah [13:50:19] hoo: yeah probably :) [13:50:40] hoo: probably want to use terbium for long running jobs [13:51:07] Yeah, usually terbium [13:54:12] Talked to Nico and we'll probably end up whitelisting my hooserv.net and use that [13:54:35] easy enough [13:54:55] if wmde doesn't have such thing [13:55:21] Probably not fast enough with handing out access etc [13:55:29] * aude could easily do that for wikimediadc, if i was still actively involved [13:55:46] I don't think I have shell on any wmde stuff [13:55:50] yeah :/ [13:56:54] (03PS1) 10Andrew Bogott: Add labvirt hosts to tls_allowed_dn_list [puppet] - 10https://gerrit.wikimedia.org/r/204763 [13:57:59] (03CR) 10Andrew Bogott: [C: 032] Add labvirt hosts to tls_allowed_dn_list [puppet] - 10https://gerrit.wikimedia.org/r/204763 (owner: 10Andrew Bogott) [13:59:05] (03PS4) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [13:59:07] (03PS11) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [13:59:09] (03PS4) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [13:59:11] (03PS6) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [13:59:13] (03PS3) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [13:59:15] (03PS2) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [13:59:17] (03PS2) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [13:59:19] (03PS11) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [14:02:28] (03CR) 10Faidon Liambotis: "No, I'm not comfortable with adding $::realm branching into a completely unrelated module because Labs has a broken PAM config. Let's fix " [puppet] - 10https://gerrit.wikimedia.org/r/204748 (https://phabricator.wikimedia.org/T85307) (owner: 10Tim Landscheidt) [14:05:36] akosiaris: mind dealing with https://phabricator.wikimedia.org/T83062 ? [14:06:02] (since copper is now being used that can be closed I guess :) ) [14:06:47] thanks JohnFLewis :) [14:06:50] (done) [14:07:09] paravoid: welcome [14:15:29] 6operations, 10OTRS, 6Security, 7HTTPS: SSL-config of the OTRS is outdated - https://phabricator.wikimedia.org/T91504#1215382 (10JanZerebecki) [14:15:31] 6operations, 7HTTPS, 5Patch-For-Review: Replace SHA1 certificates with SHA256 - https://phabricator.wikimedia.org/T73156#1215381 (10JanZerebecki) [14:17:04] (03PS1) 10Hoo man: Whitelist hooserv.net for GWT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204766 [14:17:15] aude: hashar ^ [14:17:31] 6operations, 7HTTPS, 5Patch-For-Review: Replace SHA1 certificates with SHA256 - https://phabricator.wikimedia.org/T73156#1215404 (10JanZerebecki) Yes, only the part of that ticket related to the SHA1 in the cert is also a dependency. [14:17:33] (03PS1) 10Andrew Bogott: Update tls_allowed_dn_list again [puppet] - 10https://gerrit.wikimedia.org/r/204767 [14:18:53] (03CR) 10Andrew Bogott: [C: 032] Update tls_allowed_dn_list again [puppet] - 10https://gerrit.wikimedia.org/r/204767 (owner: 10Andrew Bogott) [14:21:26] (03Abandoned) 10Giuseppe Lavagetto: cache: mock change to show use of hiera [puppet] - 10https://gerrit.wikimedia.org/r/204504 (owner: 10Giuseppe Lavagetto) [14:21:41] hoo: ok with me [14:21:53] * aude not doing deploys on friday though [14:22:15] I think we have a semi-official "deploy that whenever you want" from Greg for these change [14:22:21] :) [14:23:59] Can you +1? [14:24:20] * aude can do that much [14:24:36] (03CR) 10Aude: [C: 031] Whitelist hooserv.net for GWT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204766 (owner: 10Hoo man) [14:25:10] (03CR) 10Hoo man: [C: 032] Whitelist hooserv.net for GWT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204766 (owner: 10Hoo man) [14:25:17] (03Merged) 10jenkins-bot: Whitelist hooserv.net for GWT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204766 (owner: 10Hoo man) [14:26:26] !log hoo Synchronized wmf-config/InitialiseSettings.php: wgCopyUploadsDomains: add hooserv.net for commons (duration: 00m 11s) [14:26:40] I just love scap's flying pig :D [14:26:48] :-D [14:27:03] :) [14:27:21] hoo: aude .... [14:27:36] what is wrong in importing from terbium directly? [14:27:54] hashar: They have meta data prepared for GWT [14:28:02] GWT is more than just an uploader [14:28:06] hoo can just copy the files [14:28:07] ahh [14:28:14] and then some other person can use gwt [14:28:20] I already have: hoo: aude .... [14:28:23] whoops [14:28:23] so I guess it is a corner case :) [14:28:29] http://hooserv.net/tmp/wmde-gwt/ [14:28:31] go go upload! [14:28:34] heh [14:28:51] the amazing hoo! [14:29:13] :D [14:29:53] Now off to make some pancakes... it's Friday \o/ [14:30:01] PROBLEM - Apache HTTP on mw1096 is CRITICAL - Socket timeout after 10 seconds [14:30:15] (03PS1) 10Tim Landscheidt: dynamicproxy: Open firewall for proxymanager [puppet] - 10https://gerrit.wikimedia.org/r/204770 (https://phabricator.wikimedia.org/T88216) [14:31:01] PROBLEM - HHVM rendering on mw1096 is CRITICAL - Socket timeout after 10 seconds [14:37:40] PROBLEM - HHVM busy threads on mw1096 is CRITICAL 33.33% of data above the critical threshold [86.4] [14:40:25] 6operations: Upgrade codfw,ulsfo,esams LVS to jessie - https://phabricator.wikimedia.org/T96375#1215460 (10BBlack) 3NEW [14:40:58] 6operations: Upgrade codfw,ulsfo,esams LVS to jessie - https://phabricator.wikimedia.org/T96375#1215468 (10BBlack) [14:45:19] 7Blocked-on-Operations, 6operations, 5Patch-For-Review: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1215476 (10GWicke) @ArielGlenn, as described in https://phabricator.wikimedia.org/T94093#1214628 the dump-all-wikis script currently expects one working... [14:46:31] PROBLEM - puppet last run on db2009 is CRITICAL puppet fail [14:47:41] PROBLEM - HHVM queue size on mw1096 is CRITICAL 33.33% of data above the critical threshold [80.0] [14:48:21] (03PS1) 10Tim Landscheidt: ocg: Fix ferm rules [puppet] - 10https://gerrit.wikimedia.org/r/204772 [14:48:34] (03PS5) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [14:48:36] (03PS12) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [14:48:38] (03PS5) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [14:48:40] (03PS7) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [14:48:41] 6operations, 7HTTPS, 5HTTPS-by-default: Force all Wikimedia cluster traffic to be over SSL for all users (logged-in and anon) - https://phabricator.wikimedia.org/T49832#1215477 (10Tony_Tan_98) I see. Just for your reference, the English Wikipedia community had a discussion (https://en.wikipedia.org/wiki/Wiki... [14:48:42] (03PS4) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [14:48:44] (03PS3) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [14:48:46] (03PS3) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [14:48:48] (03PS12) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [14:48:50] (03PS1) 10BBlack: normalize on $cluster_tier as numeric [puppet] - 10https://gerrit.wikimedia.org/r/204773 [14:50:52] still no jouncebot [14:53:02] (03PS1) 10Jgreen: correct spelling berrylium-->beryllium [dns] - 10https://gerrit.wikimedia.org/r/204775 [14:55:40] (03PS1) 10Alexandros Kosiaris: Update package_builder/README.md [puppet] - 10https://gerrit.wikimedia.org/r/204776 [14:58:02] (03CR) 10Jgreen: [C: 032 V: 031] correct spelling berrylium-->beryllium [dns] - 10https://gerrit.wikimedia.org/r/204775 (owner: 10Jgreen) [15:04:50] RECOVERY - puppet last run on db2009 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:45] 6operations: deploy db2043-2066 - https://phabricator.wikimedia.org/T89365#1215513 (10Papaul) [15:11:46] 6operations, 10ops-codfw: rack/wire/initial setup of db2043-db2070 - https://phabricator.wikimedia.org/T89368#1215510 (10Papaul) 5stalled>3Resolved a:3Papaul mgmt settings complete , test complete db2052 10.193.2.140 ge-6/0/0 D6 db2053 10.193.2.141 ge-6/0/1 D6 db2054 10.193.2.142 ge-6/0/2 D6 db2055 10.... [15:11:50] (03PS1) 10Jgreen: add primary ethernet IP for betelgeuse [dns] - 10https://gerrit.wikimedia.org/r/204779 [15:12:08] (03PS2) 10Alexandros Kosiaris: Update package_builder/README.md [puppet] - 10https://gerrit.wikimedia.org/r/204776 [15:16:57] (03CR) 10Jgreen: [C: 032 V: 031] add primary ethernet IP for betelgeuse [dns] - 10https://gerrit.wikimedia.org/r/204779 (owner: 10Jgreen) [15:19:29] !log DNS updates, for a couple of fundraising hosts [15:19:36] Logged the message, Master [15:20:59] (03PS6) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [15:21:01] (03PS6) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [15:21:03] (03PS5) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [15:21:05] (03PS2) 10BBlack: normalize on $cluster_tier as strings [puppet] - 10https://gerrit.wikimedia.org/r/204773 [15:21:07] (03PS4) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [15:21:09] (03PS4) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [15:21:11] (03PS13) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [15:21:35] (03CR) 10Glaisher: [C: 031] "Thanks for working on this." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [15:22:58] 6operations, 10ops-fundraising: DNS for betelgeuse and beryllium - https://phabricator.wikimedia.org/T96365#1215538 (10Jgreen) 5Open>3Resolved corrected beryllium spelling, added DNS for primary IP for betelgeuse [15:23:55] 6operations, 10ops-fundraising: DNS for betelgeuse and beryllium - https://phabricator.wikimedia.org/T96365#1215549 (10Jgreen) [15:26:50] (03Abandoned) 10Tim Landscheidt: Labs: Create /etc/motd unconditionally [puppet] - 10https://gerrit.wikimedia.org/r/204748 (https://phabricator.wikimedia.org/T85307) (owner: 10Tim Landscheidt) [15:28:42] 6operations, 10ops-fundraising: DNS for betelgeuse and beryllium - https://phabricator.wikimedia.org/T96365#1215565 (10Jgreen) [15:38:09] (03CR) 10KartikMistry: "To be deploy on: 21st April, Tuesday." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204722 (https://phabricator.wikimedia.org/T95848) (owner: 10KartikMistry) [15:41:40] (03PS1) 10Ottomata: Use GlobalSign cert in cert chain for archiva [puppet] - 10https://gerrit.wikimedia.org/r/204787 [15:42:28] paravoid: still around? [15:42:30] yes [15:42:44] https://gerrit.wikimedia.org/r/#/c/203544/ could you look at that when you have time please? [15:43:05] ottomata: https://gerrit.wikimedia.org/r/#/c/197341/ [15:43:17] bblack: what do you think we should do about that? ( https://gerrit.wikimedia.org/r/#/c/197341/ ) [15:45:24] (03CR) 10RobH: Use GlobalSign cert in cert chain for archiva (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/204787 (owner: 10Ottomata) [15:46:09] paravoid: crap, I think I meant to merge that up when I merged the others, but must have put off a final review for "the next morning" and then forgot about it, like a month ago [15:46:27] heh [15:46:33] it's scarier than the others [15:46:45] it has the capability of messing all of our certs :) [15:46:48] at this point I'd definitely have to re-review the impact to remember it all [15:49:19] JohnFLewis: looks okay but it's premature I guess? [15:49:35] we don't have servers there to serve those endpoints, so what's the point [15:49:51] paravoid: true but it's there for when its needed :) [15:49:55] jgage: hiyaaa [15:50:16] 10Ops-Access-Requests, 6operations: Give joal access to eventlog1001.eqiad.wmnet - https://phabricator.wikimedia.org/T95905#1215650 (10Ottomata) This will be the eventlogging-admins group. [15:53:41] (03PS2) 10Ottomata: Use GlobalSign cert in cert chain for archiva by default [puppet] - 10https://gerrit.wikimedia.org/r/204787 [15:54:41] (03CR) 10RobH: [C: 031] "Looks like it will work to me!" [puppet] - 10https://gerrit.wikimedia.org/r/204787 (owner: 10Ottomata) [15:55:46] (03PS3) 10Ottomata: Use GlobalSign cert in cert chain for archiva by default [puppet] - 10https://gerrit.wikimedia.org/r/204787 [15:57:21] (03CR) 10Ottomata: [C: 032] Use GlobalSign cert in cert chain for archiva by default [puppet] - 10https://gerrit.wikimedia.org/r/204787 (owner: 10Ottomata) [15:57:54] (03PS7) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [15:57:56] (03PS13) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [15:57:58] (03PS7) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [15:58:00] (03PS8) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [15:58:02] (03PS6) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [15:58:04] (03PS3) 10BBlack: normalize on $cluster_tier as numeric [puppet] - 10https://gerrit.wikimedia.org/r/204773 [15:58:06] (03PS5) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [15:58:08] (03PS5) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [15:58:10] (03PS14) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [15:59:40] papaul: I'm updating the tasks for this [15:59:43] are ganglia graphs in UTC? [15:59:44] since we made them before we had a template [16:00:25] 6operations: deploy db2043-2066 - https://phabricator.wikimedia.org/T89365#1215691 (10RobH) 5Open>3Resolved all but db2052-db2070 are done. I'll have those install in another task. [16:00:34] tgr: I'd imagine so [16:01:20] robh:ok [16:02:34] 6operations: install/setup/deploy db2052-db2070 - https://phabricator.wikimedia.org/T96383#1215693 (10RobH) 3NEW a:3RobH [16:02:45] 6operations: install/setup/deploy db2052-db2070 - https://phabricator.wikimedia.org/T96383#1215703 (10RobH) [16:02:47] 6operations, 10ops-codfw: rack/wire/initial setup of db2043-db2070 - https://phabricator.wikimedia.org/T89368#1034772 (10RobH) [16:03:00] papaul: Ok, so https://phabricator.wikimedia.org/T96383 is tracking the install of the new codfw database systems in row D [16:03:07] since all the other ones were done on older tasks [16:03:27] 6operations: install/setup/deploy db2052-db2070 - https://phabricator.wikimedia.org/T96383#1215693 (10RobH) [16:03:44] papaul: now, normally i'm right here doing the install & the network setup [16:03:51] but we're going to setup this as if I'm not [16:03:58] so you're going to create all the sub-tasks as needed [16:04:04] (including a network task ;) [16:04:25] for others: papaul and I are walking through the process to get a system installed. [16:05:18] 6operations: install/setup/deploy db2052-db2070 - https://phabricator.wikimedia.org/T96383#1215709 (10RobH) [16:05:33] So, there is a list of checkbox items that every system install has to have done [16:05:44] robh:ok [16:05:47] we've accomplished the onsite steps of drac setup, plus I setup the dns for it a long time ago [16:05:51] so you will work on - network switch setup (port description & vlan) [create sub-task in network project, link here as a blocker.] [16:06:00] so pull your port info off the one task [16:06:10] and create a sub-task off of https://phabricator.wikimedia.org/T96383 [16:06:24] and have it in the operations & network projects [16:06:43] you'll want to note that these are databases, so they will be in the vlan-private1-d-codfw vlan [16:07:02] for production (non labs, non analytics), each row has a public vlan and private vlan [16:07:26] so your task should list the required vlan, as well as each port and the server attached [16:07:43] now, in the past, when im doing the install, i just pull this info out of the one task [16:07:45] and simply do it [16:07:59] but, we want to get you ramped up, and unless you are in the switch like that, you need to make tasks =] [16:08:39] the nice part is the first 4 steps on an install (dns, drac & bios, network, install-module updates) are all needed to move forward, but not in any specific order [16:08:56] the rest are order-dependent [16:09:32] We also need to check and see what the other databases in codfw are running. I expect they are trusty, but a quick look is easy to confirm. [16:09:46] normally, i'll have this info in the task when i hand it off to you [16:14:20] tgr: as a general rule everything for ops is in UTC (the timezone on the hosts, all logs, all stats graphs, etc) [16:14:36] thanks [16:15:04] except the stupid inspect mode in ganglia [16:15:05] (03PS1) 10Ottomata: Use chained cert for archiva if ca_name is set [puppet] - 10https://gerrit.wikimedia.org/r/204792 [16:15:06] so confusing :) [16:15:09] I suspected but an outage correlates so nicely with the deploy train in PDT, wanted to be sure [16:15:10] 6operations: install/setup/deploy db2052-db2070 - https://phabricator.wikimedia.org/T96383#1215731 (10RobH) [16:15:27] papaul: So once you make the network task, I'll give you the last part, the install server module update [16:16:18] paravoid: http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Video+scalers+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report - is this inspect mode? [16:16:33] (03PS2) 10Ottomata: Use chained cert for archiva if ca_name is set [puppet] - 10https://gerrit.wikimedia.org/r/204792 [16:16:50] inspect mode is when you click the Inspect blue button [16:17:03] tgr: no, inspect is the javascript graph with mouseovers etc. [16:17:06] and what bblack said [16:17:12] also, wth is wrong with videoscalers? :) [16:17:36] (03CR) 10Ottomata: [C: 032] Use chained cert for archiva if ca_name is set [puppet] - 10https://gerrit.wikimedia.org/r/204792 (owner: 10Ottomata) [16:17:50] (the blue inspect buttons are here: http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Video%20scalers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1429287427&g=cpu_report&z=large ) [16:17:51] robh: can you create the network task this one so i can see how you do it [16:17:57] sure [16:18:30] I don't have that problem though, because I put my laptop in UTC :) [16:18:32] paravoid: not sure yet, all transcode jobs are failing since wednesday [16:18:44] that sounds serious [16:18:46] might not even get to the scalers [16:19:05] https://phabricator.wikimedia.org/T96236 is the report [16:20:19] so yeah if deploy train correlates, that's a good target to look at [16:20:48] 6operations, 7network: setup network switch ports / vlans for db2053-2070 - https://phabricator.wikimedia.org/T96385#1215764 (10RobH) 3NEW a:3RobH [16:20:56] no, the train is wed 11h PDT and this happened at wed 11h UTC [16:21:08] papaul: ^ there we go, just a listing of each port and the host attached [16:21:08] oh! [16:21:12] plus teh vlan request [16:21:32] robh:ok [16:21:33] so then i update the install task with the link to that in the description (but its already linked in the subtask/blockers) [16:21:51] 6operations: install/setup/deploy db2052-db2070 - https://phabricator.wikimedia.org/T96383#1215776 (10RobH) [16:22:06] papaul: Ok, so I can work on that (it'll take af ew minutes) [16:22:12] but before i do, we'll get you started on the next step [16:22:24] You'll have to pull the primary network MAC address for each of the new hosts [16:22:31] ok [16:22:58] which is on the platform pagte [16:22:58] https://wikitech.wikimedia.org/wiki/HP_DL3N0#Polling_for_MAC_Address [16:23:16] papaul: You'll also need to have a git copy of operations/puppet [16:23:19] like you do operations/dns [16:23:25] i do [16:23:53] we'll be adding these to modules/install-server/files/dhcpd/linux-host-entries.ttyS1-115200 [16:24:06] ok [16:24:14] So, you'll see in there the entries up to db2052 [16:24:23] you'll be duplicating and editing them for db2053-2070 [16:24:58] so go ahead and pull the mac info and start updating the file. if you have questions, ask =] (I'll be doing the network task now.) [16:25:12] If I wasn't here to do this, you could either bug the folks who know how directly [16:25:19] or just make the ops clinic person help you track someone down ;D [16:25:28] in this case, jgage would simply ask me to do it [16:25:30] heh [16:25:47] robh: what the ilo cli to show the MAC i know the commandonn Dell but not on HP [16:25:57] https://wikitech.wikimedia.org/wiki/HP_DL3N0#Polling_for_MAC_Address [16:26:16] thanks [16:26:18] one of the first things added to any platform doc ;D [16:26:48] now, you can use a for loop, or a tiny python script to poll all the data, and we're totally going to go into using them (at least the for loop) [16:26:57] but its not that many, so just do a manual poll for today [16:27:18] when i had to do this for every single mw system there [16:27:24] you can be damn certain no one did it manually [16:27:33] ok [16:27:41] (03CR) 10Arlolra: "See also T96239" [puppet] - 10https://gerrit.wikimedia.org/r/200732 (owner: 10Arlolra) [16:29:36] (03CR) 10RobH: [C: 032] added asset tag mgmt dns for db2052-db2070 [dns] - 10https://gerrit.wikimedia.org/r/204682 (owner: 10Papaul) [16:31:21] 6operations, 7network: setup network switch ports / vlans for db2053-2070 - https://phabricator.wikimedia.org/T96385#1215790 (10RobH) You may note I detailed placing in the private row d vlan. If these were in different rows, I would have split them up into the various groups, or detailed each one's location... [16:32:23] (03PS8) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [16:32:25] (03PS14) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [16:32:27] (03PS8) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [16:32:29] (03PS9) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [16:32:31] (03PS7) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [16:32:33] (03PS4) 10BBlack: normalize on $cluster_tier as string words [puppet] - 10https://gerrit.wikimedia.org/r/204773 [16:32:35] (03PS6) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [16:32:37] (03PS6) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [16:32:39] (03PS15) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [16:33:08] <^d> bblack: i <3 you [16:33:50] <^d> death to $realm conditionals [16:34:00] I'm trying! there's a lot of it [16:37:54] Robh: did we installed db2043 -db2051? [16:38:19] bblack, can you review https://gerrit.wikimedia.org/r/#/c/200732/ when you get a chance? [16:38:55] !log making User:Maintenance script a 'bot' on all wikis [16:38:59] Logged the message, Master [16:39:48] hrmm [16:39:52] papaul: i thought we did, but perhaps not [16:39:54] let me check [16:40:01] papaul: we may edit the task to include them [16:40:16] seems only installed up to db2042 [16:40:26] yes [16:40:42] because i received 28 db servers [16:40:54] 6operations: deploy db2043-2066 - https://phabricator.wikimedia.org/T89365#1215822 (10RobH) 5Resolved>3Open incorrect on my part, it seems they are installed up to db2042 [16:41:00] and 9 in c6 and19 in d6 [16:41:25] papaul: i thought the row c stuff had been done, it seems not [16:41:34] robh: no [16:41:43] ok, let me clean up ticekts [16:41:47] o [16:41:48] you'll need to pull mac info for those as well [16:41:48] ok [16:41:56] ok doing that [16:42:05] cool, i'll update shortly, mid switch config now [16:42:13] ok [16:42:46] subbu: remind me what's going on there? the backend fails randomly, so retry5xx is going to make it eventually succeed for the same URL quickly? [16:43:35] i'll ask arlo to come in here .. i haven't investigated it myself .. he has. [16:44:18] arlolra, subbu: remind me what's going on there? the backend fails randomly, so retry5xx is going to make it eventually succeed for the same URL quickly? [16:44:23] papaul: hrmm [16:44:29] it seems row d needs more networking work that i expected [16:44:38] as in, stuff beyond me to tie it into the rest of things [16:45:10] robh:ok [16:45:52] robh: so you can not finish the network switch setup step? [16:45:58] 6operations, 7network: setup network switch ports / vlans for db2053-2070 - https://phabricator.wikimedia.org/T96385#1215835 (10RobH) a:5RobH>3mark It appears that I cannot commit these changes. When I attempt to do so (I've added in the vlan as well), I get the following: warning: VLAN default is not en... [16:46:12] papaul: nope, but you can still get all the mac info and stuff for install server updates [16:46:14] 6operations, 5Interdatacenter-IPsec: Update 3.19 kernel to 3.19.3 - https://phabricator.wikimedia.org/T96146#1215840 (10MoritzMuehlenhoff) [16:46:20] that way when the network is fixed, they'll be ready for reboot into installer [16:46:26] robh:ok [16:46:33] thats why we do the subtasks [16:46:34] =] [16:46:57] I updated the sub-task, but considering it is already Friday evening for Mark, I'd think this won't get done until Monday [16:47:37] 10Ops-Access-Requests, 6operations: Give joal access to eventlog1001.eqiad.wmnet - https://phabricator.wikimedia.org/T95905#1215869 (10Ottomata) @gage you are on clinic duty this week, you got this one? [16:49:12] bblack, subbu: (i don't know about the "randomly" in that sentence) if the service returns a 5xx, varnish retries the req 3 times and , failing that, then (the crucial part) returns the 5xx error from the service itself. but without the retry503 setting being 4 (greater than the 3 above), it returns varnish's 503 error instead ... at least that's what i gathered [16:50:01] 6operations, 5Interdatacenter-IPsec: Update 3.19 kernel to 3.19.3 - https://phabricator.wikimedia.org/T96146#1215896 (10MoritzMuehlenhoff) 5Open>3Resolved 3.19.3 was already built in Debian experimental and included the IPSEC patch. Since the 3.19.4 update doesn't fix any further security issues and most o... [16:50:16] what does the "3" come from? I wasn't aware of this behavior. I would've assumed retry5xx just changes how many times it tries before it still serves a 5xx to the user [16:50:17] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215903 (10vshchepakina) @Dzahn yay!!!! Thank you so much everyone for making it happen!!!!!! I really appreciate all of your help!!!!! [16:50:42] sorry retry503 [16:51:16] i'm going to be deploying CSS fixes for the Popups extension in a moment, on the request of the team (and with greg's approval) [16:51:49] bblack: look at the definition of 5xx [16:51:52] looking at the relevant VCL code now... [16:51:54] <^d> Someone got a second for a trivial puppet change for gerrit? [16:51:57] ori, you're a popups dev now? o_0 [16:52:38] !log ori Synchronized php-1.26wmf1/extensions/Popups: Iebaefdcf5: Update Popups for cherry-picks (duration: 00m 11s) [16:52:43] Logged the message, Master [16:53:04] MaxSem: I don't think they have anyone with deployment rights [16:53:33] wow, that's rather obtuse how those retry params work :) [16:53:35] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215930 (10Krenair) What's left to do here? Does @Dzahn's comment need to be answered first? [16:53:55] !log ori Synchronized php-1.26wmf2/extensions/Popups: I654c5cf8b: Update Popups for cherry-picks (duration: 00m 12s) [16:53:59] Logged the message, Master [16:55:02] (03Abandoned) 10Chad: Better support checking out MediaWiki & extension masters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 (owner: 10Chad) [16:55:51] and yeah, like I said last week, it seems all the current clusters differ on these parameters. the current settings are probably not-quite-right in many places! :) [16:56:04] but yes, I think we can push this one forward for parsoid, ok [16:56:11] bblack: thanks [16:56:44] can you rebase it? it will have conflict, the code moved to a new file, basically. [16:56:59] sure, one sec [16:57:01] modules/role/manifests/cache/parsoid.pp [16:57:06] is the new place [16:57:38] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215958 (10jeremyb) @vshchepakina, both URLs work now. neither is a redirect to the other. Dzahn wanted to know about what to do with the old URL. can we... [16:58:23] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215963 (10Dzahn) @jeremyb or we can just leave it as it is and call it resolved? hmm not sure [17:00:02] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215964 (10vshchepakina) @Dzahn @Krenair @jeremyb sorry about that! Yes, could the old URL be redirected to the new http://store.wikimedia.org/ please? [17:00:55] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215967 (10Dzahn) actually shopify said not to change the CNAME for the old shop URL. to make it a redirect we would have to. i'm inclined to not do them... [17:03:58] (03PS9) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [17:04:00] (03PS9) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [17:04:02] (03PS8) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [17:04:04] (03PS5) 10BBlack: normalize on $cluster_tier as string words [puppet] - 10https://gerrit.wikimedia.org/r/204773 [17:04:06] (03PS7) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [17:04:08] (03PS7) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [17:04:10] (03PS16) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [17:04:14] !log ori Synchronized php-1.26wmf1/extensions/Popups: Ie92d15985: Update Popups for cherry-picks (duration: 00m 13s) [17:04:14] (03PS1) 10Dzahn: shop/store: switch old URL to cluster for redirect [dns] - 10https://gerrit.wikimedia.org/r/204810 (https://phabricator.wikimedia.org/T92438) [17:04:19] Logged the message, Master [17:04:39] !log ori Synchronized php-1.26wmf2/extensions/Popups: I48fbafe4d: Update Popups for cherry-picks (duration: 00m 11s) [17:04:42] Logged the message, Master [17:05:05] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1215978 (10Dzahn) >>! In T92438#1215964, @vshchepakina wrote: > @Dzahn @Krenair @jeremyb sorry about that! Yes, could the old URL be redirected to the ne... [17:05:44] (03CR) 10John F. Lewis: [C: 031] shop/store: switch old URL to cluster for redirect [dns] - 10https://gerrit.wikimedia.org/r/204810 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [17:06:29] (03PS2) 10Arlolra: Fix 5xx retry for parsoid backend [puppet] - 10https://gerrit.wikimedia.org/r/200732 [17:06:42] bblack: ^ [17:07:18] (03CR) 10Dzahn: ""too big to fail"?:) thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/204696 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [17:07:34] mutante: haha! :) [17:08:32] arlolra: deployed, takes up to ~25m to take effect on the cache nodes [17:08:41] :) thanks again [17:10:00] !log running forceRenameUsers.php (SUL finalization) on all medium wikis [17:10:05] Logged the message, Master [17:11:17] grrrit-wm confused me for a sec there heh. It dissappeared just during my C:@ [17:11:21] err C:2 [17:12:21] (03CR) 10Dzahn: [C: 031] "now ok" [puppet] - 10https://gerrit.wikimedia.org/r/204559 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [17:13:07] jgage: ^^ nice deploy for you there :) [17:13:43] heh:) [17:14:44] . thanks. [17:15:14] discussing with mutante :) [17:19:14] (03PS10) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [17:19:16] (03PS15) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [17:19:18] (03PS10) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [17:19:20] (03PS10) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [17:19:22] (03PS9) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [17:19:24] (03PS6) 10BBlack: normalize on $cluster_tier as string words [puppet] - 10https://gerrit.wikimedia.org/r/204773 [17:19:26] (03PS8) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [17:19:28] (03PS8) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [17:19:30] (03PS17) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [17:20:16] i hear this documentation may be out of date. anybody know details? https://wikitech.wikimedia.org/wiki/Application_servers#Deploying_config [17:20:28] also, how urgent is this shop/store stuff? [17:21:02] (03CR) 10BBlack: [C: 031] "These 9 patches (204514 -> 204760) validated on the compiler with 10 canary hosts (t1+t2 text/bits/mobile/upload, misc, parsoid) to have n" [puppet] - 10https://gerrit.wikimedia.org/r/204514 (owner: 10BBlack) [17:21:51] PROBLEM - puppet last run on cp3033 is CRITICAL puppet fail [17:22:05] (03CR) 10BBlack: [C: 031] "These 9 patches (204514 -> 204760) validated on the compiler with 10 canary hosts (t1+t2 text/bits/mobile/upload, misc, parsoid) to have n" [puppet] - 10https://gerrit.wikimedia.org/r/204760 (owner: 10BBlack) [17:22:22] jgage: shopify seem to say it won't break keeping it as a CNAME to not urgent I guess [17:25:12] thanks. i've reviewed the change and it does look ok, but friday deploys are against policy. [17:27:29] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Found the reason this was failing" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/203073 (https://phabricator.wikimedia.org/T95545) (owner: 10Hashar) [17:27:40] (03PS1) 10Gergő Tisza: Fix invalid JSON in job runner config [puppet] - 10https://gerrit.wikimedia.org/r/204815 (https://phabricator.wikimedia.org/T96236) [17:30:35] (03PS2) 10Gergő Tisza: Fix invalid JSON in job runner config [puppet] - 10https://gerrit.wikimedia.org/r/204815 (https://phabricator.wikimedia.org/T96236) [17:30:44] (03PS3) 10Gergő Tisza: Fix invalid JSON in job runner config [puppet] - 10https://gerrit.wikimedia.org/r/204815 (https://phabricator.wikimedia.org/T96236) [17:31:44] ^^ could someone review this? it breaks video uploads on Commons [17:33:45] looking [17:34:00] (03CR) 10Ori.livneh: [C: 032] Fix invalid JSON in job runner config [puppet] - 10https://gerrit.wikimedia.org/r/204815 (https://phabricator.wikimedia.org/T96236) (owner: 10Gergő Tisza) [17:34:19] tgr: thanks [17:34:24] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216089 (10GWicke) [17:35:18] thanks ori, do I need to do something to apply this (run puppet agent on tmh1001/1002?) or just wait? [17:35:30] i'll force a puppet run on those hosts [17:35:37] otherwise it's 20 mins [17:35:41] thanks [17:35:55] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:37:01] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216101 (10GWicke) [17:37:55] 6operations, 5Interdatacenter-IPsec: Update 3.19 kernel to 3.19.3 - https://phabricator.wikimedia.org/T96146#1216107 (10Gage) Berkelium and Curium are now upgraded to Debian's 3.19.3 kernels containing the IPsec patch. Next, I will test the aes256gcm and ESN behavior. [17:38:07] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:38:11] RECOVERY - puppet last run on cp3033 is OK Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:38:39] tgr: done; can you verify the fix somehow? [17:40:34] ori: yes, but it will take a while [17:40:42] nod [17:41:05] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216124 (10GWicke) [17:41:39] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:42:20] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:43:28] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216135 (10GWicke) [17:44:01] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:44:25] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:45:20] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:47:21] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216155 (10GWicke) [17:49:43] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216163 (10GWicke) [17:52:04] (03PS1) 10Alexandros Kosiaris: Improve package_builder tests [puppet] - 10https://gerrit.wikimedia.org/r/204821 [17:52:06] hey, just saw [17:52:15] sigh, such a stupid bug [17:52:17] thanks tgr [18:02:43] (03PS1) 10Papaul: added primary network MAC address for db2043-db2070 [puppet] - 10https://gerrit.wikimedia.org/r/204829 [18:14:04] (03Abandoned) 10Dzahn: add loadbalancer service records for codfw [dns] - 10https://gerrit.wikimedia.org/r/196069 (https://phabricator.wikimedia.org/T92377) (owner: 10Dzahn) [18:14:22] (03Abandoned) 10Dzahn: config-geo: add codfw with text-addrs v4 and v6 [dns] - 10https://gerrit.wikimedia.org/r/196076 (https://phabricator.wikimedia.org/T92377) (owner: 10Dzahn) [18:25:54] 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216329 (10GWicke) [18:28:12] 6operations, 7HTTPS, 5Patch-For-Review: Replace SHA1 certificates with SHA256 - https://phabricator.wikimedia.org/T73156#1216337 (10Dzahn) T91504 is also a dependency. Signature algorithm SHA1withRSA WEAK https://www.ssllabs.com/ssltest/analyze.html?d=ticket.wikimedia.org (deleted cached version to re-ch... [18:28:30] PROBLEM - puppet last run on cp3010 is CRITICAL puppet fail [18:29:05] (03PS1) 10Gage: Give joal sudo on eventlog1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/204838 (https://bugzilla.wikimedia.org/95905) [18:29:24] whoa, bugzilla url [18:29:34] i guess i should have said Task:Tfoo instead of Bug:Tfoo eh [18:30:01] jgage: no Bug:T is correct [18:30:15] but the T was missing [18:30:29] oops, thanks [18:30:42] T1234 will create a link but that's it [18:30:47] Bug1234 will be BZ [18:30:54] (03PS2) 10Gage: Give joal sudo on eventlog1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/204838 (https://phabricator.wikimedia.org/T95905) [18:30:58] Bug:T1234 will link to phab AND let the gerrit bot do it's stuff [18:32:13] (03PS3) 10Dzahn: Give joal sudo on eventlog1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/204838 (https://phabricator.wikimedia.org/T95905) (owner: 10Gage) [18:33:39] oh i need to grant him access to hafnium as well [18:34:01] jgage: there is eventlogging-admins and eventlogging-roots [18:34:04] does he really need roots? [18:34:29] ottomata suggested the latter, but i suppose you are correct [18:34:32] 232 description: Login access for EventLogging investigation [18:34:44] well i just read the description of the group, that is -admins [18:34:48] and then i see the request where he says [18:34:58] " I'll basically check the logs and the databases in case of errors." [18:35:00] according to the ticket, he is asking for sudo, he is going to be doing the same things that the other eventlogging roots do [18:35:06] restart eventlogging for deploiys, etc. [18:35:25] joal|night: , right? [18:35:38] hmm, well the request seems to match the admins group [18:35:52] "for investigation" and "check the logs" [18:36:01] ottomata will know better [18:36:21] i didnt see stuff about restarting stuff [18:36:31] nuria: can you clarify which level of access joal is requesting for eventlogging? [18:36:45] ottomata: sudo on box [18:36:58] ottomata: that's it [18:37:23] ottomata: EL requires to have sudo on the box to which is deployed to tail logs even [18:38:05] in theory, shouldn't eventlogging-admins be able to restart eventlogging? that would be my expectation based on the group name. [18:38:06] the members of both of those groups are the identical people ... [18:40:13] "sudo" is abmbigious. it can be "permissions that admins have to restart services" and it can be "sudo ALL ALL, so it's the same as "Full root" [18:40:55] i would also expect that -admins have permissions to do the service restart [18:41:28] (03PS1) 10Aaron Schulz: Lowered "max lag" to 10 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204843 [18:41:37] but it seems they dont (so far) and -admins is really just login, so more like -users [18:41:47] maybe we should add those privileges to them [18:45:10] RECOVERY - puppet last run on cp3010 is OK Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:47:36] !log legoktm Synchronized php-1.26wmf2/extensions/CentralAuth/: forceRenameUsers: Replace _ in database name with - https://gerrit.wikimedia.org/r/204827 (duration: 00m 14s) [18:47:41] Logged the message, Master [18:48:22] !log legoktm Synchronized php-1.26wmf1/extensions/CentralAuth/: forceRenameUsers: Replace _ in database name with - https://gerrit.wikimedia.org/r/204827 (duration: 00m 13s) [18:48:25] Logged the message, Master [18:49:52] ori: confirmed [18:49:59] (03PS4) 10Gage: Give joal sudo on eventlog1001.eqiad.wmnet + grant access to hafnium [puppet] - 10https://gerrit.wikimedia.org/r/204838 (https://phabricator.wikimedia.org/T95905) [18:50:14] jgage, mutante, if you want to work on that, i think just sync up with nuria about what is needed [18:50:30] now to find a way to re-run all failed transcode jobs [18:50:46] but remember, analytics gets basically 0 support on eventlogging from ops [18:50:52] which is one of the reasons for -roots [18:50:54] tgr: are they marked abandonded? [18:51:00] in showJobs [18:51:39] I think so [18:51:50] ottomata: ok, i'm creating a separate ticket to clarify the meaning & powers of those groups [18:52:28] for now i'm giving joal access to both groups so that he's equal with other folks [18:54:41] tgr: $iter = JobQueueGroup::singleton()->get( 'JOBTYPE' )->getAllAbandonedJobs(); foreach ( $iter as $job ) { JobQueueGroup::singleton()->push( $job ); } [18:54:53] eval.php ;) [18:55:23] blazecat: thx, will try [19:00:20] (03PS2) 10Dzahn: Add sqlite3 and pixz utilities on html dumps host [puppet] - 10https://gerrit.wikimedia.org/r/204684 (owner: 10GWicke) [19:01:41] thank you very much jgage ;) [19:01:51] (03PS3) 10Dzahn: Add sqlite3 and pixz utilities on html dumps host [puppet] - 10https://gerrit.wikimedia.org/r/204684 (owner: 10GWicke) [19:01:58] my pleasure joal :) [19:02:02] (03PS3) 10Alexandros Kosiaris: Update package_builder/README.md [puppet] - 10https://gerrit.wikimedia.org/r/204776 [19:02:22] (03CR) 10Gage: [C: 032] Give joal sudo on eventlog1001.eqiad.wmnet + grant access to hafnium [puppet] - 10https://gerrit.wikimedia.org/r/204838 (https://phabricator.wikimedia.org/T95905) (owner: 10Gage) [19:09:23] 10Ops-Access-Requests, 6operations: Give joal access to eventlog1001.eqiad.wmnet - https://phabricator.wikimedia.org/T95905#1216550 (10Gage) 5Open>3Resolved a:3Gage Added joal to eventlogging-roots + eventlogging-admins to match nuria, millimetric, mforns. This gives him sudo on eventlog1001 as well as a... [19:10:39] <^d> !log running refreshLinks for ptwiki in screen on terbium for T91401. If it causes problems just kill it and ping me later. [19:10:44] Logged the message, Master [19:11:23] !log aaron Synchronized php-1.26wmf2/includes/User.php: 2f1e93058f6247c81835a01b13e7473d5c5d060e (duration: 00m 12s) [19:11:27] Logged the message, Master [19:12:06] robh: https://phabricator.wikimedia.org/T70387#1081796 maybe something you want to look at? [19:12:45] 10Ops-Access-Requests, 6operations: Requesting access to hafnium for mforns - https://phabricator.wikimedia.org/T96164#1216559 (10Gage) Hi Marcel, As a member of eventlogging-admins, you already have access to this host. Please try to ssh to mforns@hafnium.wikimedia.org (not .eqiad.wmnet) and let us know if y... [19:13:12] JohnFLewis: i buy certs [19:13:16] thats a self signed implementation issue [19:13:24] and betacluster, so i'm not really involved. [19:13:36] I think... [19:14:18] robh: the self-signed thing is said because ops haven't done anything regarding a real cert (or so) I believe. [19:14:38] hrmm, im still not sure if this would be me directly, it seems like anyone can do it (plus im cooking lunch!) [19:14:46] jgage: ^ hey clinic duty dude ;D [19:15:02] (03CR) 10Dzahn: [C: 032] "just changed the grouping of packages a bit and added comments" [puppet] - 10https://gerrit.wikimedia.org/r/204684 (owner: 10GWicke) [19:15:04] robh: I just see ssl and go 'rob' to be honest, you defer as you want :) [19:15:05] why does an opsen need to populate it though? [19:15:14] why dont they puppetize a fake key for this specifically? [19:15:21] yo [19:15:30] greg-g / bd808 ^^ [19:15:58] 6operations, 10Beta-Cluster: Beta Cluster no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#1216564 (10RobH) Shouldn't the option of : a) Have an Opsen populate /etc/ssl/private/star.wmflabs.org.key on all of the frontend boxes for beta [0]. This private key must match the public key in... [19:16:00] i've updated my thoughts [19:16:09] anytime someone says 'have opsen manually' i automatically wonder why [19:16:17] if its a crash fix, thats one thing [19:16:22] but this is to populate a required file [19:16:26] and it would be rquired on reinstalls as well [19:16:30] hence, it should be puppetized. [19:16:46] jgage: i may have been pre-emptive in my attempt to make this your problem. [19:17:03] heh ok [19:17:15] love that suggestion that ops are unwilling to help [19:17:36] (03CR) 10Dzahn: "gwicke:" [puppet] - 10https://gerrit.wikimedia.org/r/204684 (owner: 10GWicke) [19:17:37] i didnt know it existed until just now [19:17:54] 10Ops-Access-Requests, 6operations: Requesting access to hafnium for mforns - https://phabricator.wikimedia.org/T96164#1216573 (10mforns) Hi Gage, Indeed, I have access to hafnium.wikimedia.org. Thanks for you help! [19:17:57] 6operations, 10Deployment-Systems, 6Services: Automate compiling service dependencies using production Jessie libraries - https://phabricator.wikimedia.org/T94611#1168065 (10mobrovac) A PR promoting the usage of a local Docker container to build the proper dependencies and set up the deploy repo can be found... [19:18:43] 6operations, 10Beta-Cluster: Beta Cluster no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#1216577 (10RobH) I also think suggesting that ops isn't willing to help is both counter-productive, and incorrect. I was pinged on this today by someone not even on the task, and this doesn't have... [19:19:05] if no one asks the clinic person, there is a lot of shit going on [19:19:15] and they didnt put a blocked by ops tag, so how the heck would we know? [19:19:32] it wouldnt show up in clinic triage, except as unassigned ops tasks, which is a huge list [19:19:39] unassigned and blocked by ops list is far, far smaller. [19:19:41] robh: they assume you all read operations tickets every day and can pick out new ones? :p [19:20:24] yea, i get that folks get frustrated, but there are a ton of tasks now flowing through the operations project [19:20:24] 6operations, 10Parsoid, 6Services, 7service-runner: Decide whether to install heapdump by default, or continue to install npm & install on demand - https://phabricator.wikimedia.org/T95431#1216578 (10mobrovac) The [service-runner PR #30](https://github.com/wikimedia/service-runner/pull/30) goes in the dire... [19:20:32] i dont bother to try and drink from that firehose anymore [19:20:56] i triage based on the blocking, patches, and hardware projects more than simple operations. we have an unassigned operations list for the clinic person to try to get to [19:21:11] but they get to that after access requests, patch for review tasks, and blocked on ops tasks [19:21:18] 6operations, 10Beta-Cluster: Beta Cluster no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#1216582 (10Dzahn) duplicate of T50501 ? [19:21:34] robh: jgage JohnFLewis: duplicate of https://phabricator.wikimedia.org/T50501 ? [19:22:18] 6operations, 10Beta-Cluster, 10Wikimedia-Labs-Infrastructure: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#526960 (10Dzahn) Is T70387 a duplicate of this? [19:22:24] mutante: ones an RT and ones a BZ. Can't be dupes, similar/same perhaps [19:22:52] (merge basically but I'm not sure as one is about getting a cert while the other is about SSL not working?) [19:23:00] well RT and BZ tickets have been dupes in the past in some cases [19:23:05] <^d> jgage: You able to look at a 1-liner puppet thing for gerrit? [19:23:28] JohnFLewis: i dont like merge because it's not a real merge [19:23:37] JohnFLewis: as in "content does not get merged in" [19:23:48] it is just closing one of them [19:23:52] mutante: indeed, it seems like they are the same [19:24:08] RT would actually merge, that was nicer imho [19:24:10] and neither one have a blocked by ops =P [19:24:29] thats like being mad at the waiter for not knowing what you want before you order. [19:24:35] ;] [19:24:38] ^d sure. url? [19:24:40] JohnFLewis: well that's very related though, SSL is not working because there are no certs [19:24:45] or just tag me as reviewer [19:24:47] <^d> jgage: https://gerrit.wikimedia.org/r/#/c/204211/ thx! [19:24:50] k [19:25:02] JohnFLewis: the difference could be "self-signed" vs. "buying actual certs" though [19:25:26] mutante: the former shouldn't even exist though :) [19:25:34] (03PS2) 10Gage: Use Diffusion to support r1234 links in Gerrit [puppet] - 10https://gerrit.wikimedia.org/r/204211 (owner: 10Chad) [19:26:00] (03CR) 10Dzahn: [C: 031] Use Diffusion to support r1234 links in Gerrit [puppet] - 10https://gerrit.wikimedia.org/r/204211 (owner: 10Chad) [19:26:05] * robh burned his pizza lunch ;_; [19:26:10] i didnt hear the timer. [19:26:39] (03CR) 10Gage: [C: 032] Use Diffusion to support r1234 links in Gerrit [puppet] - 10https://gerrit.wikimedia.org/r/204211 (owner: 10Chad) [19:27:12] hah, i wanted to try a random SVN link in diffusion and i had this in my browser history [19:27:15] https://phabricator.wikimedia.org/rSVN115794 [19:27:20] "This is the way the world ends: Not with a bang but a whimper." [19:29:48] hmm i want to give someone access to tin for deployments. in manifests/role/deployment.pp, class role::deployment::server says $deployment_group = 'wikidev', but in modules/admin/data/data.yaml that group is empty and clearly the group i need is 'deployment'. what am i missing? [19:31:09] jgage: add him to "deployment" and you will be fine [19:31:23] 6operations, 10Continuous-Integration, 5Continuous-Integration-Isolation, 5Patch-For-Review, 7Upstream: Create a Debian package for NodePool on Debian Jessie - https://phabricator.wikimedia.org/T89142#1216616 (10hashar) During our Apr 17th checkin, we pointed out a new .deb package targeting Jessie shoul... [19:31:29] wikidev will just happen [19:31:34] jgage: deployment is loaded via hiera or something now [19:31:39] yeah, but where is the reference to that group that i'm missing [19:31:53] they need to have that gid for deployment permissions [19:32:08] historic reasons [19:32:33] jgage: https://github.com/wikimedia/operations-puppet/blob/production/hieradata/hosts/tin.yaml#L8 ? [19:32:41] i guess it's because deployment is in admin::groups in hieradata/hosts/tin.yaml [19:33:04] 5 wikidev: [19:33:04] 6 gid: 500 [19:33:04] 7 description: container group for primary user groups. [19:33:10] jgage: ^ [19:33:22] mutante: we're talking about something different :p [19:33:36] everybody with shell is also a "wikidev" [19:33:37] ok [19:33:45] oho [19:34:08] check your own groups. gid=500(wikidev) groups=500(wikidev),700(ops) [19:34:14] YuviPanda: how long does it take for wikibugs to rejoin [19:34:47] No idea :) ask the maintainers? legoktm and valhallasw [19:35:07] Negative24: it rejoins whenever a bug is supposed to be send to this channel [19:35:27] ah that explains quite a bit [19:35:41] Negative24: it will *always* join -dev, -labs, and -feed upon connection to freenode though [19:36:00] it keeps getting kicked for flooding [19:37:03] (03PS1) 10Gage: Add mforns as deployer [puppet] - 10https://gerrit.wikimedia.org/r/204861 (https://phabricator.wikimedia.org/T96163) [19:37:07] patches welcome :) both myself and valhallasw are super busy and don't have much time to fix it [19:39:30] !log restarted forceRenameUsers.php (SUL finalization) on bgwiki (and then other medium wikis) [19:39:36] Logged the message, Master [19:40:40] eh, 503 from gerrit. come on [19:40:49] ok, just very temp [19:40:50] I'm still wondering where jouncebot is [19:41:20] YuviPanda, can't you deal with jouncebot? [19:42:50] 7Blocked-on-Operations, 6operations, 5Patch-For-Review: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1216684 (10Dzahn) sqlite3 and pixz have been added by https://gerrit.wikimedia.org/r/#/c/204684/ [19:42:56] 6operations, 5Continuous-Integration-Isolation: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216685 (10hashar) 3NEW [19:43:09] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1216691 (10chasemp) [19:44:09] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1216702 (10chasemp) p:5Triage>3Normal [19:44:26] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1216676 (10chasemp) @greg can you stamp this with your undying love and approval? [19:44:36] !log legoktm Synchronized php-1.26wmf2/extensions/CentralAuth/: LocalRenameUserJob: Don't validate the 'from' username if 'force' is true - https://gerrit.wikimedia.org/r/204846 (duration: 00m 12s) [19:44:42] Logged the message, Master [19:45:16] !log legoktm Synchronized php-1.26wmf1/extensions/CentralAuth/: LocalRenameUserJob: Don't validate the 'from' username if 'force' is true - https://gerrit.wikimedia.org/r/204846 (duration: 00m 12s) [19:45:19] Logged the message, Master [19:46:58] 6operations, 5Continuous-Integration-Isolation: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216708 (10Dzahn) a:3Dzahn [19:49:14] jgage: as mutante said, wikidev is beyond abused as a group as is basically a shared PUG [19:51:19] whuzza pug? [19:51:34] placeholder user group? [19:52:29] Krenair: I can but I don’t think I should. The only bot I should be responsible for is grrrit-wm on account of writing it... [19:52:34] Krenair: I can give you access though [19:52:45] <^d> jgage: Something like that. It used to be called mortals. [19:52:50] jgage: primary user group [19:52:51] YuviPanda, that would simplify things [19:52:53] <^d> Once upon a time, there were the roots. [19:52:53] thanks [19:52:56] Krenair: yup. let me do that now [19:53:00] <^d> There were only roots [19:53:09] <^d> And then, there were those who would not be root [19:53:12] mortals is a great name [19:53:15] <^d> So they were termed mortals [19:53:21] my iimpressionis wikidev was like devs who needed to tinker in prod at some point [19:53:27] Krenair: you’re alex monk on wikitech right? [19:53:27] mortals is the old name of "deployment", not of wikidev [19:53:28] but then it was reappropriate for file sharing [19:53:31] YuviPanda, yes [19:53:33] and it just got weird [19:53:46] Right, at one point the people who ran the servers were "the developers" [19:53:51] all shell users have been in "wikidev" [19:53:53] And some people still think that [19:54:08] there also used to be no difference between "what type of shell" [19:54:10] Krenair: added a bunch of people to it including yourself. [19:54:23] shell meant deploying meant being a wikidev [19:54:42] so long story short is wikidev is now a terrible name :D [19:54:48] but changing it is fraught with peril [19:55:10] <^d> Not a terrible name even, but yes changing it will Break Shit [19:55:22] YuviPanda, yay [19:55:24] jouncebot, next [19:55:24] In 67 hour(s) and 4 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150420T1500) [19:55:29] there are a lot of ppl who are not "devs" who are in wikidev [19:55:30] Krenair: :) thank you :) [19:55:36] heh .. at deviantart I deployed code on my first week, committed code my first day, and didn't get any sort of shell access for several years. Here it took almost three months for my first accepted commit. [19:55:39] Krenair: can you actually remove my name from that list? :) [19:55:40] <^d> chasemp: Fair enough :) [19:56:01] and shortly after that I had widespread shell access [19:56:14] YuviPanda, I can remove your access to the tool I guess [19:56:17] can't you do that? [19:56:24] Krenair: not sure, haven’t tried... [19:56:29] Krenair: but it might be more legit if you do? :) [19:56:30] twentyafterfour: yeah we need to automatize more :) [19:56:44] I mean, I’ve root so I do have access everywhere… but it’ll take my name off that list [19:57:10] my main use for a shell has always been to look at logs and we have https://logstash.wikimedia.org/ now so that is a good step forward [19:57:17] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Requesting access to tin.eqiad.wmnet for mforns - https://phabricator.wikimedia.org/T96163#1216740 (10Gage) 5Open>3Resolved a:3Gage Done! [19:57:22] YuviPanda, what are those service groups on bastion? [19:57:37] Krenair: on the bastion project? no idea... [19:57:41] they shouldn’t exist, I think. [19:58:29] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Requesting access to tin.eqiad.wmnet for mforns - https://phabricator.wikimedia.org/T96163#1216746 (10mforns) o/ Thanks! [19:59:18] 10Ops-Access-Requests, 6operations: Requesting access to hafnium for mforns - https://phabricator.wikimedia.org/T96164#1216747 (10Gage) 5Open>3Resolved a:3Gage [19:59:31] YuviPanda, I think I removed you [19:59:39] although it still shows on the page strangley [20:00:40] 6operations, 5Continuous-Integration-Isolation: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216754 (10hashar) There is a puppet change to have ferm rules use hiera instead of networks.pp: //contint: move zuul_merger_hosts to hi... [20:10:01] <^d> YuviPanda: I'm kinda stuck on the dsh thing. [20:11:49] 6operations, 5Continuous-Integration-Isolation: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216809 (10Dzahn) a:5Dzahn>3None [20:15:39] 6operations, 7Graphite: scale graphite deployment (tracking) - https://phabricator.wikimedia.org/T85451#1216834 (10GWicke) @fgiunchedi, what is the plan for increasing SSD space? If we plan to continue using a single graphite node per DC, can we order larger SSDs and replace the current ones to have some headr... [20:23:03] eh, physikerwelt works for NIST? [20:25:00] 6operations, 5Continuous-Integration-Isolation, 5Patch-For-Review: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216857 (10hashar) Patch is stalled pending discussion about the behavior of puppet @resolve(()) if it can't DNS res... [20:25:59] ^d: I primarily disagree with _joe_ but haven’t had time to overlap so booooo [20:26:18] <^d> Eh we'll figure it out monday then [20:27:03] ^d: let me amend it, moment [20:28:15] 6operations, 5Continuous-Integration-Isolation, 5Patch-For-Review: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216867 (10hashar) a:3hashar [20:28:45] 6operations, 5Continuous-Integration-Isolation, 5Patch-For-Review: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216685 (10hashar) p:5Normal>3High [20:28:50] bah, grrrit-wm is dead. [20:29:05] ^d: see https://gerrit.wikimedia.org/r/#/c/204331/ now [20:31:27] <^d> YuviPanda: Ahh, join() was the magic I wanted :D [20:31:42] ^d: :) [20:31:52] ^d: so as it stands *now* the patch is ok, can you test it somewhere? [20:32:12] <^d> I can, uno momento [20:35:05] 6operations, 10ops-eqiad: mw1031 has a bad uplink - https://phabricator.wikimedia.org/T95896#1216899 (10Gage) p:5Triage>3Normal [20:35:13] 6operations, 10Continuous-Integration, 5Continuous-Integration-Isolation: Provide Debian package python-pymysql for jessie-wikimedia - https://phabricator.wikimedia.org/T96131#1216901 (10hashar) a:3Andrew Following up Apr 17th checkin, @Andrew kindly accepted to look at the package :) As to why upstream i... [20:35:32] 6operations, 10Continuous-Integration, 5Continuous-Integration-Isolation: Provide Debian package python-pymysql for jessie-wikimedia - https://phabricator.wikimedia.org/T96131#1216904 (10hashar) p:5Normal>3Low The task is not blocking the implementation of the service since I have manually installed the... [20:36:22] 6operations, 6Services: Migrate SCA cluster to Jessie - https://phabricator.wikimedia.org/T96017#1216907 (10Gage) p:5Triage>3High [20:38:04] JohnFLewis, thanks. I hate it when people get in the way of cleaning up these sorts of things. [20:38:11] any volunteers to do a swat deployment for Apache config? [20:38:22] <^d> YuviPanda: SUCCESS! [20:38:41] JohnFLewis, now get someone from ops to approve it :p [20:38:42] mutante: apache falls out of SWAT :) [20:38:48] <^d> YuviPanda: https://phabricator.wikimedia.org/P533 [20:38:52] Krenair: mutante linked me to it so ;) [20:38:59] aha [20:39:06] Yeah I don't think deployers can deal with apache config mutante? [20:39:13] I thought that was an ops thing [20:39:25] i didn't specify a group:) [20:41:28] bblack: Wanna look at ^^? [20:41:32] Erm. [20:41:39] https://gerrit.wikimedia.org/r/#/c/204932/ <-- that [20:42:41] well then it's not really a "swat deployment".. [20:43:14] 6operations, 10incident-20150410-flowdataloss, 7database: Better backup coverage for X1 database cluster - https://phabricator.wikimedia.org/T95835#1216955 (10Gage) p:5Triage>3High [20:45:37] sorry for using the wrong terminology /me moves on [20:52:01] 6operations, 5Continuous-Integration-Isolation: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1216990 (10hashar) Following Friday 2015-04-10 checkin, @chasemp talked to @mark about the labs VLANs. During today checkin, that is stalled pending further discussion... [21:02:39] oh for goodness' sake gerrit... [21:02:49] -label:Code-Review<0 [21:03:00] 500 Internal server error [21:03:14] even without the - [21:03:23] it's fine with label:Code-Review<=-1 [21:04:01] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 2 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/). [21:10:47] (03CR) 10BBlack: [C: 031] "+1 with nits, which I don't really care whether you fix or not, I guess they're more matters of opinion." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/204932 (https://phabricator.wikimedia.org/T96045) (owner: 10coren) [21:11:13] (03PS5) 10Chad: Clean up dsh config, move to hiera [puppet] - 10https://gerrit.wikimedia.org/r/204331 [21:11:17] <^d> YuviPanda: ^ [21:11:45] (03CR) 10Alex Monk: [C: 04-1] "Does not merge, tests fail, and what is that ID in the commit message?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175007 (owner: 10Ori.livneh) [21:12:03] (03CR) 10jenkins-bot: [V: 04-1] Clean up dsh config, move to hiera [puppet] - 10https://gerrit.wikimedia.org/r/204331 (owner: 10Chad) [21:12:24] (03CR) 10Dzahn: "the ferm docs unfortunately have this to say about @resolve:" [puppet] - 10https://gerrit.wikimedia.org/r/201882 (https://phabricator.wikimedia.org/T87519) (owner: 10Dzahn) [21:12:39] (03CR) 10Yuvipanda: Clean up dsh config, move to hiera (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/204331 (owner: 10Chad) [21:13:00] (03CR) 10coren: Labs: set_strip_cache for labstores (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/204932 (https://phabricator.wikimedia.org/T96045) (owner: 10coren) [21:13:16] Coren: bblack paravoid also was messing around with sysfs earlier, I think he had a module... [21:13:42] YuviPanda: In this particular case, the glob is needed because we can't know the names in advance. [21:13:55] (03PS2) 10coren: Labs: set_strip_cache for labstores [puppet] - 10https://gerrit.wikimedia.org/r/204932 (https://phabricator.wikimedia.org/T96045) [21:14:10] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Open firewall for proxymanager [puppet] - 10https://gerrit.wikimedia.org/r/204770 (https://phabricator.wikimedia.org/T88216) (owner: 10Tim Landscheidt) [21:14:17] (03CR) 10Chad: Clean up dsh config, move to hiera (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/204331 (owner: 10Chad) [21:14:27] <^d> YuviPanda: Crazy direction, or no? [21:15:09] ^d: the role? what do you think it’ll be named? [21:15:25] <^d> heck if I know [21:15:32] <^d> maybe I'm overthinking all of this. [21:15:45] (03CR) 10coren: [C: 032] Labs: set_strip_cache for labstores [puppet] - 10https://gerrit.wikimedia.org/r/204932 (https://phabricator.wikimedia.org/T96045) (owner: 10coren) [21:16:38] Oh bah. YuviPanda, you the one +2 Tim's change? [21:16:50] (03PS4) 10BBlack: codfw: add datacenter to dns config [dns] - 10https://gerrit.wikimedia.org/r/203544 (owner: 10John F. Lewis) [21:16:53] Ok if merge? [21:16:54] Coren: heh yeah :) [21:16:57] Coren: merged already. [21:16:59] that script needs locking [21:17:05] I mean puppet-merge [21:17:08] also rebase before merging! :D [21:17:11] Coren: yup [21:17:16] Coren: I merged... [21:17:20] both mine and yours [21:17:39] Ah, we did the puppe-merge at the same time too. :_) [21:17:47] Which script needs locking? [21:17:55] bblack: bah order, who needs it :) [21:19:12] Coren: puppet-merge :) [21:19:31] (03CR) 10BBlack: [C: 031] "I've validated the layout here and reviewed all the related address spaces (see updates in T83110), and it's all good. My amend was just " [dns] - 10https://gerrit.wikimedia.org/r/203544 (owner: 10John F. Lewis) [21:20:06] bblack: oh tier grouping, kay. Thanks for looking at it :) [21:22:07] JohnFLewis: I think I'll go ahead and push that now, or else I'll forget monday. the risk is pretty much zero anyways. [21:22:23] bblack: yeah. awesome if you do :) [21:23:21] (03CR) 10BBlack: [C: 032] codfw: add datacenter to dns config [dns] - 10https://gerrit.wikimedia.org/r/203544 (owner: 10John F. Lewis) [21:23:42] JohnFLewis: bblack: nice!! [21:24:04] JohnFLewis: you beat me to that and i abandoned the other patches [21:24:22] bblack: awesome [21:24:52] adding a new DC isnt happening every day [21:25:44] if I have my way, it could become a lot more common heh [21:25:44] bblack: on the good side, I just host'd text-lb.codfw.wikimedia.org and it responded happily :) [21:25:50] I want datacenters everywhere :) [21:27:03] bblack: http://www.datacentermap.com/indonesia/denpasar/nusantara-data-center-bali.html [21:27:56] heh [21:28:15] only mutante would want to go to denpasar on expense account rather than hong kong or tokyo [21:28:28] lol, right [21:28:40] so frugal! [21:28:44] well my sister's moving there in a couple months, so we could make her the unofficial DC tech [21:28:46] Hard Rock Cafe hotel suite ?:p [21:28:54] bblack: ask Mark if a new quarterly goal can be 'open a DC up somewhere in the world for fun' [21:28:57] :p [21:29:16] we're already pushing for it in planning, not just for fun [21:29:32] the fun is complementary [21:29:33] in an ideal world, just to play catch-up we should have 1-2 sites in asia, a second one in europe, and one in south america [21:29:39] bblack: new caching one, think I saw it listed :0 [21:29:41] *:) [21:29:48] but but .. it's connected to Equinix Singapore [21:29:52] our asia times are the worst [21:30:02] worse than south america for sure [21:30:02] and East Asia actually could help ?:p [21:30:07] it's embarrassing bad [21:30:47] yeah getting even 1 site in asia is top priority [21:31:04] bblack: singapore? :p [21:31:19] there's a lot involved in a location decision [21:31:29] yeah true [21:31:33] but yes, I think on the balance of things, Singapore is pretty high in the runnings [21:31:35] chasemp: one of the biggest changes of me moving to SF is how much faster gerrit is... [21:31:37] singapore and hong kong are probably by far the easiest and also epically expensive [21:31:48] Oh, I guess Hong Kong is in China [21:31:50] like the nearest airport must have a cool name [21:31:54] so, nevermind about HK :) [21:32:02] YuviPanda: I don't doubt it, I've run some numbers using ripe atlas now and doing the monitoring service demos [21:32:13] do it in India so you can have the Indian government send police to airport to pick up any employees... [21:32:16] right, HK has legal/politics issues. Tokyo would be better on network/legal/policy than SG, but even more expensive [21:32:18] (true story, happened about 3 years ago) [21:32:25] SG strikes a nice balance on a lot of factors [21:32:40] singapore has cheap internet too comparatively [21:32:47] I have a friend who works there [21:33:12] I have several friends who live there too, so yeah I'm all for having another reason to drop by :) [21:33:22] me too :) [21:33:26] bblack: I feel like this is decided :D [21:33:38] ok, deal, let's just all move [21:33:46] caching site only [21:34:00] dont need anyone to move ;D [21:34:06] shhhhhh [21:34:12] fuck no [21:34:19] im not letting you all leave me as the only PST pager! [21:34:23] neverrrrr [21:34:56] Apartments are even more exensive in singapore than in SF. No one wants to actually /move/ there, just visit frequently on a per diem. [21:34:59] no one can leave the continental US timezones unless you find someone else to take your place! [21:35:02] bblack: you need to get CyprusOne in Singapore please near Singapore Changi Airport :p [21:35:09] the name would be awesome, cosin :p [21:35:15] heh [21:35:20] JohnFLewis: we no longer use closest [21:35:23] bblack: so the dude I know runs the tech at http://www.zalora.sg/ [21:35:27] but now the largest/most recognizable [21:35:29] robh: oh? [21:35:29] all haskell (or soon to be) [21:35:37] I think there's only one anyways [21:35:38] that has to be worth a trip to looky-loo [21:35:38] yep, hence the new dallas eq caching site [21:35:40] robh: that is the largest one :) [21:35:41] isnt eqdal as i named it [21:35:49] mark and faidon overrode the naming scheme [21:35:57] ahh, thats cool then, just sayin [21:36:03] (the override makes sense) [21:36:07] JohnFLewis: CyprusOne, CyrusOne, confusing :p [21:36:16] since eqdal and codfw are 20 miles apart [21:36:21] SPF|Cloud: hsuh [21:36:23] *hush [21:36:24] easier to call them both dfw [21:38:22] we could actually make a word out of that side name too, if we were careful about provider choice [21:38:26] *site [21:38:35] infrastructure page updated to reflect new reality [21:38:37] I see basin, resin, and rosin in a words file [21:39:18] lots of possibilities if you treat sin$ as short for sing [21:39:36] musin (musing), etc [21:39:44] posin heh [21:39:45] just put an e at the end of cosin and you'll appeal to a lot of mathematicians :p [21:40:29] If I know one thing about robh, it's that he despises naming convention conversations but is strikingly always at the center of them :) [21:40:54] nah this one makes sense [21:41:04] i just wanted to point out that the page on wikitech was wrong [21:41:18] didnt want them to come up with an awesome name and tell them no ;D [21:41:21] yurik: https://gerrit.wikimedia.org/r/#/c/204760/ ? [21:41:26] wait, there's an eqdfw now? [21:41:31] there's not [21:41:33] bblack, looking [21:41:47] oh robh, did i get that right? we are renaming codfw to codal ?:) [21:41:55] * mutante prepares patches :p [21:41:57] you read it backwards [21:42:01] completely backwards [21:42:06] we arent renaming anything [21:42:09] heh, ok [21:42:24] the new peering site will be called eqdfw [21:42:24] bblack, don't we have another realm, other than labs & prod? [21:42:32] probably! [21:42:45] that's part of what hierization addresses, yes [21:42:45] so you want to enable geo lookup there as well? [21:42:51] no idea! [21:42:59] https://gerrit.wikimedia.org/r/#/c/204760/8/modules/role/manifests/cache/mobile.pp,cm [21:43:05] yes [21:43:26] in general, $::realm in puppet outside of prod/labs values is a mess, I'm not sure it's right anywhere [21:43:40] bblack, than sure, go ahead, will +1 [21:43:47] I think the other realm-like things are for beta/staging [21:44:07] mostly I just wanted to check whether killing ESI was going to break something you still had going there [21:44:36] (03CR) 10Yurik: [C: 031] "Seems ok, except that geo stuff will now be enabled for all realms, not just labs & prod." [puppet] - 10https://gerrit.wikimedia.org/r/204760 (owner: 10BBlack) [21:44:40] MaxSem, ^ [21:46:40] it's at the end of 8 other pending refactor patches, none of it's merging until at least monday [21:51:18] 6operations, 10Continuous-Integration, 5Continuous-Integration-Isolation, 5Patch-For-Review, 7Upstream: Create a Debian package for NodePool on Debian Jessie - https://phabricator.wikimedia.org/T89142#1217217 (10Andrew) [21:51:21] 6operations, 10Continuous-Integration, 5Continuous-Integration-Isolation: Provide Debian package python-pymysql for jessie-wikimedia - https://phabricator.wikimedia.org/T96131#1217215 (10Andrew) 5Open>3Resolved done! [21:58:07] 6operations, 10Wikimedia-Labs-wikitech-interface: Can not log into wikitech.wikimedia.org - https://phabricator.wikimedia.org/T96240#1217252 (10Andrew) [21:59:26] 6operations, 10RESTBase, 10hardware-requests: Expand RESTBase cluster capacity - https://phabricator.wikimedia.org/T93790#1217256 (10GWicke) Update: [Current growth rates](http://grafana.wikimedia.org/#/dashboard/db/cassandra-restbase-eqiad?panelId=12&fullscreen) are back around 20G/day per node, primarily b... [22:00:40] !log legoktm Synchronized php-1.26wmf2/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: LocalRenameUserJob: In force mode, bypass all Title/User validation - https://gerrit.wikimedia.org/r/204945 (duration: 00m 14s) [22:00:46] Logged the message, Master [22:01:29] !log legoktm Synchronized php-1.26wmf1/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: LocalRenameUserJob: In force mode, bypass all Title/User validation - https://gerrit.wikimedia.org/r/204945 (duration: 00m 11s) [22:01:33] Logged the message, Master [22:10:31] (03PS5) 10Dzahn: sshd: set Message Authentication Code ciphers [puppet] - 10https://gerrit.wikimedia.org/r/185329 [22:11:03] (03CR) 10Dzahn: "done. <% elsif scope.function_os_version(['ubuntu == precise'])%> .." [puppet] - 10https://gerrit.wikimedia.org/r/185329 (owner: 10Dzahn) [22:13:06] greg-g: I'm planning to push out a minor messaging patch, for fundraising stuff on donatewiki. [22:15:02] (03PS2) 10Dzahn: Note which LDAP groups are allowed in HTTP login prompts mentioning labs [puppet] - 10https://gerrit.wikimedia.org/r/204291 (owner: 10Alex Monk) [22:17:12] (03CR) 10Dzahn: [C: 032] "sure, can be helpful. these questions have been asked quite a bit before." [puppet] - 10https://gerrit.wikimedia.org/r/204291 (owner: 10Alex Monk) [22:18:36] awight: messageing patch? as in, just a string change? [22:19:31] greg-g: yep, i18n strings [22:19:33] (03CR) 10Legoktm: "Thank you :D" [puppet] - 10https://gerrit.wikimedia.org/r/204291 (owner: 10Alex Monk) [22:19:46] greg-g: well, a submodule update in which I cherry-picked i18n changes. [22:19:47] awight: kk [22:20:45] legoktm, you're welcome [22:20:59] legoktm, I'm sure we have all sorts of HTTP auth around the place which I didn't pick up on however [22:26:11] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1217336 (10Dzahn) They are waiting in code review now and need Apache deployment. Robh said to put it on the Deployment Calendar and we should probably n... [22:26:21] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1217337 (10Dzahn) They are waiting in code review now and need Apache deployment. Robh said to put it on the Deployment Calendar and we should probably n... [22:39:22] !log fixed bad SULF renames on be_x_oldwiki, cbk_zamwiki, fiu_vrowiki, pa_uswikimedia, roa_rupwiki, roa_rupwiktionary, zh_min_nanwikibooks, zh_min_nanwikiquote, zh_min_nanwikisource [22:39:28] Logged the message, Master [22:52:19] (03CR) 10Dzahn: "you are also removing dbproxy1004. is that on purpose? if so, you may wanna mention that in the commit message" [puppet] - 10https://gerrit.wikimedia.org/r/204829 (owner: 10Papaul) [22:54:51] !log awight Synchronized php-1.26wmf1/extensions/DonationInterface/: T94246: change legal text for recurring donation forms (duration: 00m 14s) [22:54:56] Logged the message, Master [22:55:11] !log awight Synchronized php-1.26wmf2/extensions/DonationInterface/: T94246: change legal text for recurring donation forms (duration: 00m 14s) [22:55:15] Logged the message, Master [22:57:21] 7Blocked-on-Operations, 6operations, 5Patch-For-Review: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1217377 (10GWicke) Marko and Eric reviewed the code at https://github.com/wikimedia/htmldumper/pull/3/files. It is also pushed to https://gerrit.wikimedi... [23:05:40] greg-g: I think I'm done with the deployment. But I suspect that my new change is being overridden by l10n cache or something :( [23:05:58] fwiw, https://donate.wikimedia.org/w/index.php?title=Special%3AAllMessages&prefix=Donate+interface-monthly-cancel [23:06:03] awight: if you change i18n messages, you have to scap. [23:07:11] but legoktm thanks! [23:07:14] s/but/ [23:08:01] !log awight Started scap: T94246: Change legal text for recurring donations [23:08:07] Logged the message, Master [23:14:43] (03PS1) 10GWicke: htmldumper 0.1.0 with dependencies [dumps/html/deploy] - 10https://gerrit.wikimedia.org/r/204964 (https://phabricator.wikimedia.org/T94457) [23:18:51] 6operations, 7Graphite, 5Patch-For-Review: Counters now only provide rates (multiplied by 1000?) - https://phabricator.wikimedia.org/T95703#1217405 (10GWicke) Also see T85451 about scaling graphite storage. [23:25:17] Krenair: dumb question: how do I change the policy on a phab ticket? :/ [23:26:03] JohnFLewis, Edit Task [23:26:27] JohnFLewis: you can't really, you can only change via the security drop down [23:26:44] the priv to tinker directly with permissions is limited to #security [23:27:12] https://www.mediawiki.org/wiki/Phabricator/Security [23:27:14] that's why then. chasemp I changed it from security confidential to none but it remained private :/ [23:28:00] what's it? [23:28:49] https://www.mediawiki.org/wiki/Phabricator/Security#How_to_Lower_the_Security_of_a_Task [23:29:53] context is https://phabricator.wikimedia.org/T90351 [23:30:50] ah Krenair fixed it thanks [23:31:14] chasemp: so in order to actually be able to do it, I need to be in 'security'? [23:31:43] I think there's some fundraising group somewhere able to do it [23:31:52] 7Blocked-on-Operations, 6operations, 5Patch-For-Review: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1217433 (10Dzahn) And i guess there's a reason why we should work on our internal tools rather than github. [23:31:55] 6operations, 10Parsoid, 6Services: Lets consider upgrading our Node.js installs to io.js (once decent Debian packages are ready) - https://phabricator.wikimedia.org/T91855#1217434 (10GWicke) [23:34:12] they can technically for their own use cases but shouldn't in the general case [23:34:37] 6operations, 10Parsoid, 6Services: Let's consider upgrading our Node.js installs to io.js (once decent Debian packages are ready) - https://phabricator.wikimedia.org/T91855#1217436 (10Ricordisamoa) [23:35:01] tldr; the decision was made that only ppl who can access security oriented / sensitive content can make something public that is hidden [23:36:21] chasemp: though doesn't that also apply for NDA though? as it is sensitive content [23:39:12] so who the policy is set to is separate from who can manipulate the policy [23:39:20] and who can manipulate policy is a global right [23:39:26] it's a lack of granularity in permissions atm [23:40:13] indeed but I don't get why the NDA group can't exactly be given the permission but if its agreed otherwise, I'll leave it as such. [23:40:13] it's a long and sordid tale the bulk of which should be on https://www.mediawiki.org/wiki/Phabricator/Security [23:44:18] (03CR) 10Eevans: [C: 031] "LGTM" [dumps/html/deploy] - 10https://gerrit.wikimedia.org/r/204964 (https://phabricator.wikimedia.org/T94457) (owner: 10GWicke) [23:47:02] !log awight Finished scap: T94246: Change legal text for recurring donations (duration: 39m 00s) [23:47:08] Logged the message, Master [23:47:17] 10ops-fundraising, 10Fundraising-Backlog: Need Civi access for Donor Services agent - https://phabricator.wikimedia.org/T95011#1217495 (10CCogdill_WMF) Hey @Jgreen, was Kristie included in the list of people who you send the cert update to earlier this week? If you could give us an ETA on when she can get set... [23:59:45] 7Blocked-on-Operations, 6operations, 5Patch-For-Review: Install nodejs, nginx and other dependencies on francium - https://phabricator.wikimedia.org/T94457#1217535 (10GWicke) 'work on', yeah ;)