[00:00:49] (03PS2) 10Mobrovac: Change-Prop: Encode page title before calling links module [puppet] - 10https://gerrit.wikimedia.org/r/308586 (owner: 10Ppchelko) [00:10:04] (03CR) 10Mobrovac: [C: 031] "Cherry-picked in beta." [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) (owner: 10Mobrovac) [00:12:23] RECOVERY - puppet last run on db2069 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [00:37:34] (03PS2) 10Aaron Schulz: Avoid pointless ChronologyProtector duplicate key notices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308662 [00:38:57] (03PS3) 10Aaron Schulz: Avoid pointless ChronologyProtector duplicate key notices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308662 [01:05:06] (03PS4) 10BryanDavis: Add a "now" command [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/308086 [01:07:08] jouncebot: now [01:07:08] No deployments scheduled for the next 11 hour(s) and 52 minute(s) [01:07:56] (03CR) 10BryanDavis: [V: 032] "With PS4:" [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/308086 (owner: 10BryanDavis) [01:09:51] (03PS5) 10BryanDavis: Add a "now" command [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/308086 [02:17:59] PROBLEM - puppet last run on mw2201 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:21:22] (03PS24) 10Alex Monk: Add python version of maintain-replicas script [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) [02:21:37] (03CR) 10Alex Monk: "> Have you done any specific testing? Can you describe them/paste the output?" [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) (owner: 10Alex Monk) [02:22:17] (03CR) 10jenkins-bot: [V: 04-1] Add python version of maintain-replicas script [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) (owner: 10Alex Monk) [02:23:57] (03PS25) 10Alex Monk: Add python version of maintain-replicas script [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) [02:38:49] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 16m 46s) [02:38:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:42:53] RECOVERY - puppet last run on mw2201 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [02:45:17] !log l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 6 02:45:17 UTC 2016 (duration 6m 28s) [02:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:52:01] (03PS1) 10KartikMistry: WIP: Apertium: Update new packages for Jessie migration [puppet] - 10https://gerrit.wikimedia.org/r/308679 (https://phabricator.wikimedia.org/T144588) [06:16:54] (03PS1) 10Elukey: Remove low maxrequestworkers setting for ImageScalers [puppet] - 10https://gerrit.wikimedia.org/r/308684 [06:21:22] ah snap I just seen the issue with icinga-wm, I thought it was up and running after my restart of ircecho [06:21:38] volans: when you are online let me know if I broke it or if it was another thing [06:21:51] really sorry, I tried to stop it when rhodium went nuts [06:25:23] (03CR) 10Elukey: "Removing the explicit setting brings MaxRequestWorkers to 1000, like the appservers . Not sure if image scalers need special settings but " [puppet] - 10https://gerrit.wikimedia.org/r/308684 (owner: 10Elukey) [06:38:38] _joe_ about --^ - I am not sure if '100' would be better as first step, or the 1000 calculation that mpm_config.pp does are good for this use case. If you have time let's discuss it later on :) [06:44:02] (brb!) [06:44:59] <_joe_> 100 is surely better [06:45:11] <_joe_> and yes, scalers have special requirements [06:48:42] (03PS5) 10Giuseppe Lavagetto: scap::source: use puppet to manage directory creation [puppet] - 10https://gerrit.wikimedia.org/r/306429 [06:48:45] (03PS2) 10Elukey: Remove low maxrequestworkers setting for ImageScalers [puppet] - 10https://gerrit.wikimedia.org/r/308684 [06:50:01] (03CR) 10Giuseppe Lavagetto: [C: 031] Remove low maxrequestworkers setting for ImageScalers [puppet] - 10https://gerrit.wikimedia.org/r/308684 (owner: 10Elukey) [06:50:10] (03CR) 10jenkins-bot: [V: 04-1] scap::source: use puppet to manage directory creation [puppet] - 10https://gerrit.wikimedia.org/r/306429 (owner: 10Giuseppe Lavagetto) [06:52:46] (03PS6) 10Giuseppe Lavagetto: scap::source: use puppet to manage directory creation [puppet] - 10https://gerrit.wikimedia.org/r/306429 [06:53:57] (03CR) 10jenkins-bot: [V: 04-1] scap::source: use puppet to manage directory creation [puppet] - 10https://gerrit.wikimedia.org/r/306429 (owner: 10Giuseppe Lavagetto) [06:54:34] !log installing chromium security update on osmium [06:54:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:04:19] (03PS7) 10Giuseppe Lavagetto: scap::source: use puppet to manage directory creation [puppet] - 10https://gerrit.wikimedia.org/r/306429 [07:06:51] elukey: I'm not but here I am :) [07:07:35] (03PS1) 10Jcrespo: nagios: Add marostegui to the dba contact group [puppet] - 10https://gerrit.wikimedia.org/r/308686 [07:14:39] (03CR) 10Elukey: [C: 032] Remove low maxrequestworkers setting for ImageScalers [puppet] - 10https://gerrit.wikimedia.org/r/308684 (owner: 10Elukey) [07:21:00] !log reimaging mw2170 to jessie [07:21:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:22:36] !log Increasing MaxRequestWorkers on Eqiad Imagescalers - mw129[3-8] - from 30 to 100 (one at the time checking metrics) [07:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:28:21] The apache scoreboard on mw1293 looks nice, metrics are fine. Proceeding [07:31:45] (03PS8) 10Giuseppe Lavagetto: scap::source: use puppet to manage directory creation [puppet] - 10https://gerrit.wikimedia.org/r/306429 [07:31:54] <_joe_> elukey: cool [07:34:40] <_joe_> sigh [07:34:48] (03PS9) 10Giuseppe Lavagetto: scap::source: use puppet to manage directory creation [puppet] - 10https://gerrit.wikimedia.org/r/306429 [07:36:33] good morning [07:38:48] <_joe_> right now, I want to burn puppet down in flames [07:38:59] (03PS1) 10Alexandros Kosiaris: Specify master branch to push in private repo [puppet] - 10https://gerrit.wikimedia.org/r/308687 [07:39:01] (03PS1) 10Alexandros Kosiaris: On git masters, make /srv/private gitpuppet owned [puppet] - 10https://gerrit.wikimedia.org/r/308688 [07:39:03] (03PS1) 10Alexandros Kosiaris: puppetmaster: absent the post-merge hook on backends [puppet] - 10https://gerrit.wikimedia.org/r/308689 [07:40:25] !log shutting down mw2140-mw2214 for hardware maintenance (T142726) [07:40:26] T142726: Multiple servers in codfw fail to respond to IPMI commands during reimaging - https://phabricator.wikimedia.org/T142726 [07:40:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:41:30] !log correction: shutting down mw2140-mw2147 and mw2200-mw2214 for hardware maintenance (T142726) [07:41:31] T142726: Multiple servers in codfw fail to respond to IPMI commands during reimaging - https://phabricator.wikimedia.org/T142726 [07:41:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:45:15] (03CR) 10Giuseppe Lavagetto: [C: 032] Specify master branch to push in private repo [puppet] - 10https://gerrit.wikimedia.org/r/308687 (owner: 10Alexandros Kosiaris) [07:45:29] (03CR) 10Giuseppe Lavagetto: "I honestly thought to have fixed this, thanks :)" [puppet] - 10https://gerrit.wikimedia.org/r/308687 (owner: 10Alexandros Kosiaris) [07:45:56] (03CR) 10Giuseppe Lavagetto: [C: 031] On git masters, make /srv/private gitpuppet owned [puppet] - 10https://gerrit.wikimedia.org/r/308688 (owner: 10Alexandros Kosiaris) [07:46:14] (03CR) 10Giuseppe Lavagetto: [C: 031] puppetmaster: absent the post-merge hook on backends [puppet] - 10https://gerrit.wikimedia.org/r/308689 (owner: 10Alexandros Kosiaris) [07:46:28] _joe_: thanks! [07:46:33] <_joe_> akosiaris: yw [07:46:44] 06Operations, 10Traffic, 07HTTPS: wmflabs.org should enforce HTTPS - https://phabricator.wikimedia.org/T144790#2610285 (10abian) [07:46:58] <_joe_> akosiaris: I am stuck in fixing something else, but after that I can start with puppetdb in codfw [07:47:10] _joe_: no worries, take your time [07:49:01] (03CR) 10Alexandros Kosiaris: [C: 04-1] postgres/osm: Make accessible from production and labs networks (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308594 (owner: 10Muehlenhoff) [07:49:42] PROBLEM - MegaRAID on db1020 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) [07:51:42] (03CR) 10Alexandros Kosiaris: [C: 032] "This restriction is actually temporary (as in at some point we should be able to have modules/role/manifests/role.pp files). But it's nice" [puppet] - 10https://gerrit.wikimedia.org/r/308666 (https://phabricator.wikimedia.org/T144774) (owner: 10Legoktm) [07:51:48] (03PS3) 10Alexandros Kosiaris: Add test to ensure there are no *.pp files in modules/role/manifests/ [puppet] - 10https://gerrit.wikimedia.org/r/308666 (https://phabricator.wikimedia.org/T144774) (owner: 10Legoktm) [07:51:50] (03CR) 10Alexandros Kosiaris: [V: 032] Add test to ensure there are no *.pp files in modules/role/manifests/ [puppet] - 10https://gerrit.wikimedia.org/r/308666 (https://phabricator.wikimedia.org/T144774) (owner: 10Legoktm) [07:52:58] <_joe_> akosiaris: uhm I'm not sure I agree [07:53:39] that it's temporary ? [07:53:43] or in general ? [07:55:03] (03PS2) 10Alexandros Kosiaris: Rename ferm service for postgres/puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/308592 (owner: 10Muehlenhoff) [07:55:07] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Rename ferm service for postgres/puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/308592 (owner: 10Muehlenhoff) [07:56:24] _joe_ all imagescalers updated, everything seems good afaics [08:02:16] <_joe_> elukey: great job [08:04:47] ^marostegui check the raid alert [08:05:01] checking [08:05:02] volans, I suppose your script it is not on yet, right? [08:06:18] jynus: the one that automatically do the task? not yet :( pending review [08:06:28] but you can use: sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli [08:06:42] to have a nice view :) ( marostegui too ) [08:07:29] it is ok, I prefer to show him the "old days of doing things" [08:07:51] in this case there is a broken disk, another one with some media error and a bit hot [08:08:54] volans: ah, the good and lovely megacli syntax :) [08:09:25] (03PS2) 10Alexandros Kosiaris: Specify master branch to push in private repo [puppet] - 10https://gerrit.wikimedia.org/r/308687 [08:09:27] (03CR) 10Alexandros Kosiaris: [V: 032] Specify master branch to push in private repo [puppet] - 10https://gerrit.wikimedia.org/r/308687 (owner: 10Alexandros Kosiaris) [08:09:43] (03PS2) 10Alexandros Kosiaris: On git masters, make /srv/private gitpuppet owned [puppet] - 10https://gerrit.wikimedia.org/r/308688 [08:09:43] marostegui: yeah in this case, hence the script that show you only failing stuff grouped properly, try it, I'd love some feedback too [08:09:47] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] On git masters, make /srv/private gitpuppet owned [puppet] - 10https://gerrit.wikimedia.org/r/308688 (owner: 10Alexandros Kosiaris) [08:09:55] (03PS2) 10Alexandros Kosiaris: puppetmaster: absent the post-merge hook on backends [puppet] - 10https://gerrit.wikimedia.org/r/308689 [08:09:59] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] puppetmaster: absent the post-merge hook on backends [puppet] - 10https://gerrit.wikimedia.org/r/308689 (owner: 10Alexandros Kosiaris) [08:10:18] marostegui: some slight improvement under review in https://gerrit.wikimedia.org/r/#/c/303992/ [08:10:22] may someone land a change to Nodepool config for me please? Got acked by Andrew https://gerrit.wikimedia.org/r/#/c/308583/ [08:10:48] that is to stop injecting metadata properties to the images being booted on labs. They are no more needed and prevent booting instances :D [08:11:20] (03CR) 10Alexandros Kosiaris: [C: 032] apertium-ita: Initial Debian packaging [debs/contenttranslation/apertium-ita] - 10https://gerrit.wikimedia.org/r/308153 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:11:57] (03CR) 10Alexandros Kosiaris: [C: 032] apertium-srd: Initial Debian packaging [debs/contenttranslation/apertium-srd] - 10https://gerrit.wikimedia.org/r/308154 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:13:51] volans, jynus in this case, that we need which disk failed and everything. What is the procedure? Do we need to contact the DC guys to get it swapped? [08:14:03] yep [08:14:20] Can someone educate me on how to do it so I can take care of it? [08:14:21] sadly we cannot (yet) change disks from the internets [08:14:28] well, I was about to [08:15:14] as an example T135157 [08:15:15] T135157: db1023 Degraded RAID - https://phabricator.wikimedia.org/T135157 [08:15:18] No, maybe I thought they also get the alert and can swap them themselves after giving us a heads up [08:15:30] that is what volans is working on [08:15:53] yeah, that's T142085, under review [08:15:54] T142085: Automate creation of Phab task for failed disks - https://phabricator.wikimedia.org/T142085 [08:16:06] for obvious reasons DBs have a higher disk failure rate than others [08:16:58] the important part is the right project #ops-$DATACENTER so that the right guy is notified [08:17:14] a more usual report would be for example: https://phabricator.wikimedia.org/T116499 [08:17:19] <_joe_> jynus: don't get fancy, just because you actually /use/ those disks :P [08:17:51] you mean unlike mediawiki, where they only use it once after boot?, _joe_ [08:19:13] <_joe_> jynus: yes [08:19:21] <_joe_> or most other things :P [08:19:31] actually [08:19:43] newer databases will not use disk [08:19:54] except for the transaction log [08:19:58] <_joe_> well on writes :) [08:20:10] volans: just read T142085, and it makes sense to me. mdadm though, I have had really awful experiences when replacing a disk (kernel hung, most of the time, 2.6.31-5 if I remember correctly, a while ago I know), no reason not to alert and send an email to the DC guys though [08:20:11] T142085: Automate creation of Phab task for failed disks - https://phabricator.wikimedia.org/T142085 [08:20:39] !log T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-ita_0.9.0~r72553-1+wmf1 [08:20:39] !log T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-srd_0.9.0~r72792-1+wmf1 [08:20:40] T144174: Package apertium-srd, apertium-ita and apertium-srd-ita - https://phabricator.wikimedia.org/T144174 [08:20:41] T144174: Package apertium-srd, apertium-ita and apertium-srd-ita - https://phabricator.wikimedia.org/T144174 [08:20:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:20:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:21:09] marostegui: that's way in the task there will be also the message "Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware." [08:21:38] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-srd-ita] - 10https://gerrit.wikimedia.org/r/308179 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:21:59] replacing a disk in a slave in the passive datacenter is different than the master of s1 in the active datacenter (when jynus was on vacation of course :-P ) [08:22:15] why? [08:22:51] different in the sense of that you want to be around and probably pick the time when to do it [08:22:59] no, I don't [08:23:08] even replacing a disk on a db master has no issues [08:23:18] theoretically [08:23:20] impact, yes [08:23:28] but it is safe for us [08:24:56] chris never waits for me, and he shouldn't [08:25:13] a different thing is md raids that may be problematic? [08:25:23] (I do not know) [08:27:13] sorry cannot stay longer now but I personally would like to be around and "ready" on a master or critical slave because although quite rare issues can happen swapping disks [08:27:28] and happened to me in the past [08:27:45] personal view of course ;) [08:28:26] sorry gotta go now, happy to continue at another time :) [08:30:08] 06Operations, 10ops-eqiad: db1060 degraded array - https://phabricator.wikimedia.org/T144793#2610348 (10Marostegui) [08:32:04] 06Operations, 10ops-eqiad: db1060 degraded array - https://phabricator.wikimedia.org/T144793#2610360 (10jcrespo) [08:32:29] got missed: may I get a nodepool config change to be merged please ? https://gerrit.wikimedia.org/r/#/c/308583/ oked by andrew [08:32:42] that prevents me from refreshing the base image we use to spawn instances [08:33:09] hashar: ? debian-glue UNSTABLE in 1m 45s ? [08:33:20] https://gerrit.wikimedia.org/r/#/c/308179/ what is unstable ? [08:33:32] I 'll have a look at 308583 [08:33:39] gotta dig in the console log [08:33:45] never seen UNSTABLE before ... [08:33:50] https://integration.wikimedia.org/ci/job/debian-glue/581/ [08:33:53] looks like TAP is failing [08:34:09] due to piuparts [08:34:10] 44 - After purging files have been modified: /var/log/apt/eipp.log.xz not owned [08:34:17] (03PS2) 10Alexandros Kosiaris: nodepool: stop injecting show:true to images [puppet] - 10https://gerrit.wikimedia.org/r/308583 (https://phabricator.wikimedia.org/T91782) (owner: 10Hashar) [08:34:22] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] nodepool: stop injecting show:true to images [puppet] - 10https://gerrit.wikimedia.org/r/308583 (https://phabricator.wikimedia.org/T91782) (owner: 10Hashar) [08:34:37] thx :] [08:34:41] will run puppet on labnodepool [08:34:48] for UNSTABLE that is kind of a special status [08:34:51] I thought piuparts was not voting [08:35:04] and the piuparts step emits a TAP Result [08:35:17] TAP == ? [08:35:19] that is processed by the Jenkins Tap plugin which apparently flag the build UNSTABLE if something fails [08:35:32] tap is a test output protocol from perl [08:35:34] 06Operations, 10ops-eqiad: db1020 degraded array - https://phabricator.wikimedia.org/T144793#2610365 (10Marostegui) [08:35:41] similar to Junit in java world [08:35:55] http://testanything.org/ ah ok [08:36:06] After purging files have been modified: /var/log/apt/eipp.log.xz not owned [08:36:08] that is strange [08:36:52] "piuparts should ignore /var/log/apt/eipp.log.xz in purge test" https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830527 [08:36:53] bah [08:37:00] lol [08:37:18] so that is apt leaving a file behind somehow [08:38:36] > I've commited the fix to git [08:38:44] and obviously the author failed to point to the git commit :D [08:38:56] ah holger [08:39:06] but should be fixed in piuparts 0.72 [08:39:27] so I suppose we should upgrade [08:39:40] looks like a race from my 10000ft view [08:39:43] ah [08:40:07] well the apertium-srd-ita change https://gerrit.wikimedia.org/r/#/c/308179/ is marked as using "unstable" as a target [08:40:22] ah [08:40:25] ok that probably explains it [08:40:30] https://packages.debian.org/search?keywords=piuparts [08:40:40] claims that 0.72 landed in testing and unstable [08:40:47] so maybe the cow image is out of date :( [08:41:16] (03PS2) 10KartikMistry: apertium-srd-ita: Initial Debian packaging [debs/contenttranslation/apertium-srd-ita] - 10https://gerrit.wikimedia.org/r/308179 (https://phabricator.wikimedia.org/T144174) [08:42:09] debian glue has support to auto update the cow images, but I am pretty sure we got that disabled [08:42:29] on the basis that the cow images are provided via puppet which set up daily cron to update them [08:42:31] they anyway get autoupdated by cron [08:42:42] yeah [08:43:40] export SKIP_COWBUILDER_UPDATE="true [08:43:47] that is in the Jenkins job [08:43:53] to prevent debian glue from updating them :D [08:43:56] hashar: so, I 've changed the debian/changelog to say jessie-wikimedia [08:44:01] and now jenkins +2s [08:44:06] so we are good to go [08:44:07] magic! [08:44:11] but I must admit I am puzzled [08:44:35] puzzled by how the build succeeded in the first place, puzzled as to why it was marked unstable [08:44:42] ah [08:44:49] so the .deb packaging works all well [08:44:54] then piuparts is run against it [08:45:01] especially the former... it should not successfully build [08:45:02] piuparts fail and report bunch of issues in a TAP format [08:45:16] the Jenkins TAP plugin read the TAP output and whenever something fails flag the build with UNSTABLE [08:45:57] the UNSTABLE build defintely has a .deb attached https://integration.wikimedia.org/ci/job/debian-glue/581/ [08:45:58] ah... the build was against sid [08:46:02] so I guess the package build just fine against unstable [08:46:03] which is why it succeeded! [08:46:07] yeah ok [08:46:37] and one can control which distro is used "simply" by pointing debian/changelog to one of the distro [08:46:49] yeah I had forgotten that [08:46:55] and I was the one implementing it :P [08:47:03] yeah [08:47:06] and it is totally awesome [08:47:21] dch , git-review [08:47:21] * akosiaris grumbles something about omega 3 oils and memory [08:47:22] done :] [08:47:51] anyway, all's well that ends well. Thanks! Help appreciated [08:47:51] the thing that is really missing is to write some doc about it [08:47:57] add the job to a few more .deb [08:48:12] and give some kind of short pres for the usual packagers [08:48:32] (03CR) 10Alexandros Kosiaris: [C: 032] apertium-srd-ita: Initial Debian packaging [debs/contenttranslation/apertium-srd-ita] - 10https://gerrit.wikimedia.org/r/308179 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:48:51] (03PS1) 10Yuvipanda: base: Set default locale to be nicer than C [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/308692 (https://phabricator.wikimedia.org/T144794) [08:51:20] hashar: yes, we should probably do a recap at how packages are being built these days. Most usual packagers are up to date for most stuff but this one is probably helpful [08:51:26] cron has: 34 7 * * * /usr/sbin/cowbuilder --update --basepath "/mnt/pbuilder/base-sid-amd64.cow" >/dev/null 2>&1 [08:51:43] am I incorrect in thinking that any errors/output is hapilly entirely discarded? [08:51:50] yes [08:51:55] er, no [08:51:59] you are not incorrent that is [08:52:10] all errors/output is discarded indeed [08:52:27] will log in the unstable one and check the piuparts version [08:52:54] but the reason is that we try to avoid cron spam and unless we actually have somewhere to store it, we 'd rather ditch that output [08:53:50] we could store it locally of course.. but it is not in an exactly friendly output format. Although that is probably easily fixed [08:54:11] ah got it [08:54:12] ['eatmydata', 'tar', '-C', '/tmp/tmpnfDRXu', '-zxf', '/var/cache/pbuilder/base-unstable-amd64.tgz'] [08:54:19] piuparts uses a .tgz [08:54:23] !log T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-srd-ita_0.9.0~r72554-1+wmf1 [08:54:24] T144174: Package apertium-srd, apertium-ita and apertium-srd-ita - https://phabricator.wikimedia.org/T144174 [08:54:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:54:28] instead of being run in an up-to-date cow image [08:54:57] (got it about cron spam being discarded. kind of make sense) [08:55:21] piupparts tries to access the image on it's own ? [08:55:31] why ? [08:55:45] https://integration.wikimedia.org/ci/job/debian-glue/581/consoleFull probably has the answers [08:55:52] udo distribution=unstable piuparts_wrapper /mnt/jenkins-workspace/workspace/debian-glue/apertium-srd-ita_0.9.0~r72554-1+wmf1+0~20160906082147.581+unstable~1.gbp350e56_all.deb [08:55:57] ah, it's not running inside the pbuilder job, it's running outside it [08:56:05] RECOVERY - DPKG on ms-be1022 is OK: All packages OK [08:56:09] that is a wrapper around piuparts provided by debian-glue [08:56:25] but it invokes /usr/sbin/piuparts --basetgz=/var/cache/pbuilder/base-unstable-amd64.tgz [08:56:46] RECOVERY - puppet last run on ms-be1022 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:56:48] hmm does piuparts even support COW images ? [08:56:55] your bait? [08:57:00] lol [08:57:11] pretty sure it does not :] [08:57:38] another way would be to have the wrapper and the .deb to be bindmounted [08:57:44] and run the command inside the cow image [08:58:08] bait on 0.001% that piuparts support cow images [09:00:06] no it seems it does not [09:05:38] 06Operations, 10ops-eqiad, 10media-storage: diagnose failed disks on ms-be1027 - https://phabricator.wikimedia.org/T140374#2610427 (10fgiunchedi) a:03Cmjohnson @Cmjohnson thoughts on this? we'd need to involve HP to debug further and/or replace parts ? [09:07:11] 06Operations, 10ops-eqiad, 10media-storage: diagnose failed(?) sda on ms-be1022 - https://phabricator.wikimedia.org/T140597#2610430 (10fgiunchedi) a:05fgiunchedi>03Cmjohnson the error is still there at boot for sdb, I tried stressing the disk by writing / reading files to the raid array but no further er... [09:12:27] !log shutting down mw2153-mw2162 for hardware maintenance (T142726) [09:12:28] T142726: Multiple servers in codfw fail to respond to IPMI commands during reimaging - https://phabricator.wikimedia.org/T142726 [09:12:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:13:04] 06Operations, 10ops-eqiad: Broken disk on copper - https://phabricator.wikimedia.org/T144261#2593788 (10fgiunchedi) also note that copper has SSDs ready for it to be installed in {T130759} we should do that instead [09:22:12] (03CR) 10Gilles: [C: 032] "No idea how it disappeared..." [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/308591 (owner: 10Filippo Giunchedi) [09:23:15] !log Stopping mysql on db1064 for maintenance - T144723 [09:23:16] T144723: Reimage & upgrade db1064 - https://phabricator.wikimedia.org/T144723 [09:23:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:23:29] 06Operations: Multiple servers in codfw fail to respond to IPMI commands during reimaging - https://phabricator.wikimedia.org/T142726#2610443 (10MoritzMuehlenhoff) Here's the second batch of machines to fix. They are depooled from the cluster and powered down. Please fix the IPMI setting and power them up when d... [09:32:38] (03PS1) 10Alexandros Kosiaris: puppetmaster: Add apache::mod::lbmethod_byrequests [puppet] - 10https://gerrit.wikimedia.org/r/308697 [09:32:58] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM! thanks for tracking this down" [puppet] - 10https://gerrit.wikimedia.org/r/308565 (owner: 10Gehel) [09:35:41] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "PC happy at https://puppet-compiler.wmflabs.org/3970/" [puppet] - 10https://gerrit.wikimedia.org/r/308697 (owner: 10Alexandros Kosiaris) [09:35:46] (03PS2) 10Alexandros Kosiaris: puppetmaster: Add apache::mod::lbmethod_byrequests [puppet] - 10https://gerrit.wikimedia.org/r/308697 [09:35:48] (03CR) 10Alexandros Kosiaris: [V: 032] puppetmaster: Add apache::mod::lbmethod_byrequests [puppet] - 10https://gerrit.wikimedia.org/r/308697 (owner: 10Alexandros Kosiaris) [09:41:38] (03PS2) 10Muehlenhoff: postgres/osm: Restrict to labs networks [puppet] - 10https://gerrit.wikimedia.org/r/308594 [09:44:31] 06Operations: graphite-web cronspam - https://phabricator.wikimedia.org/T144797#2610461 (10elukey) [09:45:49] !log rysnc running from db1064 to dbstore1001 (T144723) [09:45:50] T144723: Reimage & upgrade db1064 - https://phabricator.wikimedia.org/T144723 [09:45:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:48:54] (03PS1) 10Filippo Giunchedi: thumbor: don't double log timestamps [puppet] - 10https://gerrit.wikimedia.org/r/308698 (https://phabricator.wikimedia.org/T139606) [09:49:32] (03CR) 10Alexandros Kosiaris: [C: 032] postgres/osm: Restrict to labs networks [puppet] - 10https://gerrit.wikimedia.org/r/308594 (owner: 10Muehlenhoff) [09:49:36] (03PS3) 10Alexandros Kosiaris: postgres/osm: Restrict to labs networks [puppet] - 10https://gerrit.wikimedia.org/r/308594 (owner: 10Muehlenhoff) [09:49:38] (03CR) 10Alexandros Kosiaris: [V: 032] postgres/osm: Restrict to labs networks [puppet] - 10https://gerrit.wikimedia.org/r/308594 (owner: 10Muehlenhoff) [09:53:01] ACKNOWLEDGEMENT - MegaRAID on db1020 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Jcrespo https://phabricator.wikimedia.org/T144793 [10:02:55] 06Operations: Remove mw2061-mw2074 - https://phabricator.wikimedia.org/T144745#2608981 (10mark) Yes, the batch we bought in May was supposed to replace these. However, we have budget in this new FY to expand capacity. [10:06:34] 06Operations, 10Traffic, 07HTTPS: wmflabs.org should enforce HTTPS - https://phabricator.wikimedia.org/T144790#2610533 (10Aklapper) @abian: What does this task cover which is not already covered by T102367 ? Or what is its relationship to T102367? [10:12:50] (03CR) 10Gilles: [C: 031] thumbor: don't double log timestamps [puppet] - 10https://gerrit.wikimedia.org/r/308698 (https://phabricator.wikimedia.org/T139606) (owner: 10Filippo Giunchedi) [10:15:40] PROBLEM - Host mw2207 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:40] PROBLEM - Host mw2204 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:40] PROBLEM - Host mw2202 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:40] PROBLEM - Host mw2210 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:40] PROBLEM - Host mw2212 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:41] PROBLEM - Host mw2213 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:41] PROBLEM - Host mw2214 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:41] PROBLEM - Host mw2209 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:42] PROBLEM - Host mw2206 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:42] PROBLEM - Host mw2200 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:43] PROBLEM - Host mw2211 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:43] PROBLEM - Host mw2203 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:56] 06Operations, 10Ops-Access-Requests, 10LDAP-Access-Requests, 06Release-Engineering-Team, and 3 others: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270#2458739 (10MoritzMuehlenhoff) I don't think access to the security IRC channel is a go... [10:17:04] mmmm these are probably the mw servers turned off [10:17:53] yes confirmed [10:17:55] oops, I meant to set 2 days of downtime instead of 2 hours, fixed [10:18:03] thanks! [10:19:58] 06Operations, 10LDAP-Access-Requests, 06Release-Engineering-Team, 06Security-Team, and 2 others: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270#2610549 (10MoritzMuehlenhoff) [10:21:48] 06Operations: unaccepted salt keys - https://phabricator.wikimedia.org/T143846#2610552 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff [10:22:26] <_joe_> moritzm: ahah ok [10:35:11] PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1272 [10:35:38] 06Operations, 06Labs, 13Patch-For-Review: grafana-labs.wikimedia.org doesn't reflect grafana-labs-admin.wikimedia.org - https://phabricator.wikimedia.org/T143556#2610559 (10fgiunchedi) a:05fgiunchedi>03None unassigning as I don't think I'll have time to work on the last bit (add `Anonymous` viewer rights... [10:37:57] (03PS5) 10Filippo Giunchedi: Add ability to dual-serve a portion of Swift rewrite.py traffic to Thumbor [puppet] - 10https://gerrit.wikimedia.org/r/298431 (https://phabricator.wikimedia.org/T140072) (owner: 10Gilles) [10:39:16] (03CR) 10Filippo Giunchedi: [C: 032] thumbor: don't double log timestamps [puppet] - 10https://gerrit.wikimedia.org/r/308698 (https://phabricator.wikimedia.org/T139606) (owner: 10Filippo Giunchedi) [10:39:21] (03PS2) 10Filippo Giunchedi: thumbor: don't double log timestamps [puppet] - 10https://gerrit.wikimedia.org/r/308698 (https://phabricator.wikimedia.org/T139606) [10:40:11] RECOVERY - check_mysql on db1008 is OK: Uptime: 11550438 Threads: 2 Questions: 330493348 Slow queries: 92283 Opens: 24156 Flush tables: 2 Open tables: 540 Queries per second avg: 28.613 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 433 [10:40:28] 06Operations: unaccepted salt keys - https://phabricator.wikimedia.org/T143846#2581021 (10jcrespo) These are servers to be decomission, but I cannot access all of them to shut them down: ``` db2001.codfw.wmnet db2002.codfw.wmnet db2003.codfw.wmnet db2004.codfw.wmnet db2005.codfw.wmnet db2008.codfw.wmnet db2009.c... [10:40:38] (03PS1) 10Giuseppe Lavagetto: wmflib: add map_resources function [puppet] - 10https://gerrit.wikimedia.org/r/308705 [10:41:01] <_joe_> akosiaris: your opinion would be greatly appreciated on ^^ [10:41:19] 06Operations, 10Traffic, 07HTTPS: wmflabs.org should enforce HTTPS - https://phabricator.wikimedia.org/T144790#2610579 (10abian) >>! In T144790#2610533, @Aklapper wrote: > @abian: What does this task cover which is not already covered by T102367 ? > Or what is its relationship to T102367? I think T102367 c... [10:44:58] 06Operations, 10Traffic, 13Patch-For-Review: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502#2610589 (10ema) >>! In T131502#2608658, @ema wrote: > We suspect that the bug(s) encountered while upgrading ulsfo might have been caused by running a mix of Varnish 3 and Varnish 4 th... [10:45:20] 06Operations: unaccepted salt keys - https://phabricator.wikimedia.org/T143846#2610590 (10MoritzMuehlenhoff) The keys for mw* were all caused by broken IPMI settings on the servers: The wmf-reimage script removes the Salt key in the initial step, but then fails to initiate the PXE reboot due failing IPMI command... [10:45:57] (03CR) 10jenkins-bot: [V: 04-1] wmflib: add map_resources function [puppet] - 10https://gerrit.wikimedia.org/r/308705 (owner: 10Giuseppe Lavagetto) [10:47:41] <_joe_> what didn't you like, you stupid jenkins [10:48:04] <_joe_> argh, rubocop [10:50:27] !log shutting down db2001-2009 [10:50:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:50:39] (03PS2) 10Giuseppe Lavagetto: wmflib: add map_resources function [puppet] - 10https://gerrit.wikimedia.org/r/308705 [10:53:48] 06Operations: unaccepted salt keys - https://phabricator.wikimedia.org/T143846#2610599 (10MoritzMuehlenhoff) The salt key for mw1291 was removed; mw129[12] were image scalers, which were repurposed as Thumbor servers. [10:54:54] 06Operations: unaccepted salt keys - https://phabricator.wikimedia.org/T143846#2610602 (10jcrespo) ``` $ salt-key -l pre Unaccepted Keys: aluminium.wikimedia.org cp3022.esams.wmnet ``` [11:06:52] (03PS2) 10Giuseppe Lavagetto: puppetdb: use on codfw puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/308569 [11:12:45] (03CR) 10Giuseppe Lavagetto: [C: 032] Change-Prop: Encode page title before calling links module [puppet] - 10https://gerrit.wikimedia.org/r/308586 (owner: 10Ppchelko) [11:12:49] (03CR) 10Mobrovac: [C: 031] "Cherry-picked in beta" [puppet] - 10https://gerrit.wikimedia.org/r/308586 (owner: 10Ppchelko) [11:12:53] (03PS3) 10Giuseppe Lavagetto: Change-Prop: Encode page title before calling links module [puppet] - 10https://gerrit.wikimedia.org/r/308586 (owner: 10Ppchelko) [11:13:09] (03CR) 10Giuseppe Lavagetto: [V: 032] Change-Prop: Encode page title before calling links module [puppet] - 10https://gerrit.wikimedia.org/r/308586 (owner: 10Ppchelko) [11:17:45] (03PS1) 10Giuseppe Lavagetto: puppetmaster::puppetdb::client: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/308707 [11:18:02] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] puppetmaster::puppetdb::client: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/308707 (owner: 10Giuseppe Lavagetto) [11:19:39] !log disabled puppet on db1064 - going to be reimaged - T144723 [11:19:40] T144723: Reimage & upgrade db1064 - https://phabricator.wikimedia.org/T144723 [11:19:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [11:23:16] 06Operations: unaccepted salt keys - https://phabricator.wikimedia.org/T143846#2581021 (10akosiaris) ``` > $ salt-key -l pre > Unaccepted Keys: > aluminium.wikimedia.org > ``` Checked aluminium and accepted the key. Seems like I had forgotten it back then. [11:24:05] (03CR) 10Giuseppe Lavagetto: [C: 032] "does the right thing: https://puppet-compiler.wmflabs.org/3972/" [puppet] - 10https://gerrit.wikimedia.org/r/308569 (owner: 10Giuseppe Lavagetto) [11:24:12] (03PS3) 10Giuseppe Lavagetto: puppetdb: use on codfw puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/308569 [11:24:17] (03CR) 10Giuseppe Lavagetto: [V: 032] puppetdb: use on codfw puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/308569 (owner: 10Giuseppe Lavagetto) [11:27:07] 06Operations: unaccepted salt keys - https://phabricator.wikimedia.org/T143846#2610645 (10MoritzMuehlenhoff) 05Open>03Resolved Also accepted the key cp3022, which was recently reinstalled. All done now. We should have an Icinga check for this, I'll open a Phab task. [11:31:03] 06Operations, 10Icinga: Icinga check for unaccepted Salt keys - https://phabricator.wikimedia.org/T144801#2610651 (10MoritzMuehlenhoff) [11:31:10] 06Operations, 10Icinga: Icinga check for unaccepted Salt keys - https://phabricator.wikimedia.org/T144801#2610665 (10MoritzMuehlenhoff) p:05Triage>03Normal [11:32:49] !log change-prop deploying e14892b [11:32:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [11:35:34] PROBLEM - puppet last run on mw2231 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:43:20] <_joe_> that ^^ is expected [11:44:32] (03PS3) 10Giuseppe Lavagetto: ssh::client: allow using puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/304982 (https://phabricator.wikimedia.org/T142846) [11:52:13] (03PS1) 10Muehlenhoff: role::postgres: Restrict to labs networks [puppet] - 10https://gerrit.wikimedia.org/r/308711 [11:52:24] (03PS2) 10Muehlenhoff: role::postgres: Restrict to labs networks [puppet] - 10https://gerrit.wikimedia.org/r/308711 [11:52:57] (03CR) 10Alexandros Kosiaris: [C: 032] "NOOP in https://puppet-compiler.wmflabs.org/3973/ so merging" [puppet] - 10https://gerrit.wikimedia.org/r/304982 (https://phabricator.wikimedia.org/T142846) (owner: 10Giuseppe Lavagetto) [11:59:32] (03PS1) 10Elukey: Reduce cronspam to root@ from zookeeper zkCleanup script [puppet] - 10https://gerrit.wikimedia.org/r/308715 (https://phabricator.wikimedia.org/T132324) [12:03:52] (03CR) 10Alexandros Kosiaris: [C: 04-2] "Unless I misunderstood something, this function is a wrapper around create_resources with the extra functionality of replacing @@title@@ w" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308705 (owner: 10Giuseppe Lavagetto) [12:05:01] 06Operations: APT and Puppet failing on ms-be1022 - https://phabricator.wikimedia.org/T144776#2610711 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi thanks @Volans ! fixed by `apt -f install` and likely caused by / running out of disk space earlier in the day [12:10:45] PROBLEM - puppet last run on scb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:12:01] g'day! ping paravoid [12:16:16] paravoid, PM ~ [12:19:30] 06Operations: ganglia-monitor and puppet failing on bast3001 - https://phabricator.wikimedia.org/T144778#2610768 (10fgiunchedi) @volans indeed, it doesn't seem to be able to start back up again by itself after a reboot. Also manually starting it via `/etc/init.d/ganglia-monitor start` seemed to work, but also re... [12:20:22] (03PS1) 10Alexandros Kosiaris: puppetmaster: Fix /etc/puppetdb.conf [puppet] - 10https://gerrit.wikimedia.org/r/308716 [12:22:29] (03CR) 10Alexandros Kosiaris: "pcc oked in https://puppet-compiler.wmflabs.org/3975/puppetmaster2001.codfw.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/308716 (owner: 10Alexandros Kosiaris) [12:22:31] (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: Fix /etc/puppetdb.conf [puppet] - 10https://gerrit.wikimedia.org/r/308716 (owner: 10Alexandros Kosiaris) [12:28:15] (03PS1) 10Alexandros Kosiaris: puppetdb.conf: Remove whitespace typo [puppet] - 10https://gerrit.wikimedia.org/r/308717 [12:28:30] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] puppetdb.conf: Remove whitespace typo [puppet] - 10https://gerrit.wikimedia.org/r/308717 (owner: 10Alexandros Kosiaris) [12:33:05] (03PS1) 10Alexandros Kosiaris: puppetdb: One more whitespace typo fix [puppet] - 10https://gerrit.wikimedia.org/r/308718 [12:33:11] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] puppetdb: One more whitespace typo fix [puppet] - 10https://gerrit.wikimedia.org/r/308718 (owner: 10Alexandros Kosiaris) [12:33:31] (03CR) 10Giuseppe Lavagetto: "Actually if you have a list of titles of resources (take redis instances as an example, but we have plenty in our codebase) where you need" [puppet] - 10https://gerrit.wikimedia.org/r/308705 (owner: 10Giuseppe Lavagetto) [12:34:42] (03PS2) 10BBlack: ssl_ciphersuite: remove DHE+chapoly [puppet] - 10https://gerrit.wikimedia.org/r/308433 [12:36:01] by any chance, have we somehow phased out the package libapache2-mod-php5 from puppet? [12:38:21] (03CR) 10BBlack: [C: 032] ssl_ciphersuite: remove DHE+chapoly [puppet] - 10https://gerrit.wikimedia.org/r/308433 (owner: 10BBlack) [12:38:27] 06Operations, 06Labs, 10wikitech.wikimedia.org: Can't login wikitech - https://phabricator.wikimedia.org/T144805#2610832 (10Shizhao) [12:39:55] 06Operations: ganglia-monitor and puppet failing on bast3001 - https://phabricator.wikimedia.org/T144778#2610844 (10MoritzMuehlenhoff) The current setup is a mix of sysvinit/systemd; the aggregators are provided via a service unit, while the ganglia-monitor.service is provided via the sysv init script. This shou... [12:41:13] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 602 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5280834 keys - replication_delay is 602 [12:41:18] hashar: in which context, on the app servers? [12:41:26] ah yeah sorry [12:41:35] the manifest I am using for Nodepool no more included it [12:41:39] includes [12:42:03] though it has: include mediawiki::packages::php5 [12:42:06] A while ago I cleaned up the depedencies so that on jessie only the bare minimum of PHP is installed, let me find the commit [12:43:04] hashar: 35f6983b1a60 in ops/puppet [12:43:31] ah yeah [12:43:42] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5243472 keys - replication_delay is 0 [12:44:01] that is the trace I was looking for :] [12:44:10] but that one is known to work [12:44:18] and I explicltly include mediawiki::packages::php5 [12:44:24] RECOVERY - puppet last run on scb2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:51:37] zeljkof: i will not be able to handle the SWAT if there is any change [12:52:00] hashar: I am just checking, I think there is one [12:52:17] and I am at car service, tethering from phone :| [12:52:51] wrong, there is none, problem solved :) [12:53:14] RECOVERY - puppet last run on mw2231 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:54:48] (03PS1) 10Elukey: Add a proxy to the Yarn UI on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/308721 (https://phabricator.wikimedia.org/T116192) [12:56:52] 06Operations, 10Traffic, 07Browser-Support-Internet-Explorer, 07HTTPS: Internet Explorer 6 can not reach https://*.wikipedia.org - https://phabricator.wikimedia.org/T143539#2610885 (10MoritzMuehlenhoff) 05Open>03declined Closing the ticket. IE6/XP is unsupported for quite a while now and we're not plan... [12:57:09] (03PS2) 10Elukey: Add a proxy to the Yarn UI on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/308721 (https://phabricator.wikimedia.org/T116192) [12:57:55] 06Operations: Upgrade mc1* cluster to Linux 4.4 - https://phabricator.wikimedia.org/T143695#2610892 (10MoritzMuehlenhoff) p:05Triage>03High [12:58:11] 06Operations: ganglia-monitor and puppet failing on bast3001 - https://phabricator.wikimedia.org/T144778#2610893 (10MoritzMuehlenhoff) p:05Triage>03Normal [12:58:45] 06Operations, 06Operations-Software-Development, 07HHVM, 13Patch-For-Review: Upgrade all mw* servers to debian jessie - https://phabricator.wikimedia.org/T143536#2610895 (10MoritzMuehlenhoff) p:05Triage>03High [12:59:00] 06Operations: Enhance account handling (meta bug) - https://phabricator.wikimedia.org/T142815#2610897 (10MoritzMuehlenhoff) p:05Triage>03Normal [13:00:04] hashar, Dereckson, addshore, and aude: Dear anthropoid, the time has come. Please deploy European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160906T1300). [13:00:36] Hello. [13:06:31] 06Operations, 06Commons, 06Multimedia: Deploy a PHP and HHVM patch (Exif values retrieved incorrectly if they appear before IFD) - https://phabricator.wikimedia.org/T140419#2610916 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff Sure, we can merge that patch the next time we upgrade to a HHVM security release.... [13:07:32] (03PS4) 10BBlack: ciphersuites: drop DHE SHA256 option from mid [puppet] - 10https://gerrit.wikimedia.org/r/308434 [13:07:34] (03PS1) 10BBlack: ciphersuites: remove DHE+AES256 from strong list [puppet] - 10https://gerrit.wikimedia.org/r/308722 [13:07:57] (03CR) 10BBlack: [C: 032 V: 032] ciphersuites: remove DHE+AES256 from strong list [puppet] - 10https://gerrit.wikimedia.org/r/308722 (owner: 10BBlack) [13:08:08] (03CR) 10BBlack: [C: 032 V: 032] ciphersuites: drop DHE SHA256 option from mid [puppet] - 10https://gerrit.wikimedia.org/r/308434 (owner: 10BBlack) [13:08:36] (03CR) 10Filippo Giunchedi: [C: 031] service::node: Compile the file holding puppet-controlled vars [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) (owner: 10Mobrovac) [13:09:22] 06Operations, 10ops-eqiad: db1020 degraded array - https://phabricator.wikimedia.org/T144793#2610922 (10MoritzMuehlenhoff) a:03Cmjohnson [13:09:42] zeljkof: i will not be able to handle the SWAT if there is any change [13:09:44] err [13:09:50] Dereckson: I guess there is no swat today [13:10:02] hashar: nothing to do, problem solved :) [13:10:40] oh [13:10:42] (03PS3) 10Elukey: Add a proxy to the Yarn UI on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/308721 (https://phabricator.wikimedia.org/T116192) [13:11:03] interestingly, on the CI Jessie slaves puppet does not know about libapache2-mod-php5 anymore [13:11:48] (03CR) 10Filippo Giunchedi: [C: 032] Add ability to dual-serve a portion of Swift rewrite.py traffic to Thumbor [puppet] - 10https://gerrit.wikimedia.org/r/298431 (https://phabricator.wikimedia.org/T140072) (owner: 10Gilles) [13:11:55] (03PS6) 10Filippo Giunchedi: Add ability to dual-serve a portion of Swift rewrite.py traffic to Thumbor [puppet] - 10https://gerrit.wikimedia.org/r/298431 (https://phabricator.wikimedia.org/T140072) (owner: 10Gilles) [13:13:04] (03PS1) 10BBlack: remove rest.wikimedia.org from DNS [dns] - 10https://gerrit.wikimedia.org/r/308723 (https://phabricator.wikimedia.org/T133001) [13:14:57] (03PS1) 10BBlack: text VCL: remove support for rest.wm.o [puppet] - 10https://gerrit.wikimedia.org/r/308724 (https://phabricator.wikimedia.org/T133001) [13:15:25] (03CR) 10BBlack: [C: 032] remove rest.wikimedia.org from DNS [dns] - 10https://gerrit.wikimedia.org/r/308723 (https://phabricator.wikimedia.org/T133001) (owner: 10BBlack) [13:24:34] (03PS1) 10Alexandros Kosiaris: puppetdb: Cast settings::storeconfigs_backend [puppet] - 10https://gerrit.wikimedia.org/r/308727 [13:25:48] (03CR) 10jenkins-bot: [V: 04-1] puppetdb: Cast settings::storeconfigs_backend [puppet] - 10https://gerrit.wikimedia.org/r/308727 (owner: 10Alexandros Kosiaris) [13:27:03] PROBLEM - puppet last run on scb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:27:11] (03CR) 10Elukey: [C: 04-1] "-1 to me since I gave you the wrong command. Hosts need to be set inactive rather than only depooled, because otherwise we'll miss to upda" (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/308520 (https://phabricator.wikimedia.org/T143536) (owner: 10Volans) [13:27:42] PROBLEM - puppet last run on mw2231 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:27:47] (03PS2) 10Alexandros Kosiaris: puppetdb: Cast settings::storeconfigs_backend [puppet] - 10https://gerrit.wikimedia.org/r/308727 [13:28:02] (03PS1) 10Muehlenhoff: Drop old code only relevant for precise/magnesium [puppet] - 10https://gerrit.wikimedia.org/r/308728 [13:29:04] 06Operations, 06Performance-Team, 10Traffic, 10Wikimedia-Stream, and 2 others: HTTPS-only for stream.wikimedia.org - https://phabricator.wikimedia.org/T140128#2610970 (10BBlack) Have we sent any announcement about this to the community? We might have already, just not tracked in here. [13:29:51] <_joe_> akosiaris: so casting to string works? [13:29:57] <_joe_> I thought you said it didn't [13:30:12] <_joe_> that's why I tried with inline_template() [13:30:12] _joe_: ETOOMANYPEBKACs [13:30:17] <_joe_> oh [13:30:24] forgive me for I have sinned [13:30:28] twice in a few minutes [13:30:31] (03CR) 10Rush: [C: 031] "Thanks Alex, seems reasonable for the use case to me." [puppet] - 10https://gerrit.wikimedia.org/r/304190 (owner: 10Alexandros Kosiaris) [13:31:32] (03CR) 10Alexandros Kosiaris: [C: 032] puppetdb: Cast settings::storeconfigs_backend [puppet] - 10https://gerrit.wikimedia.org/r/308727 (owner: 10Alexandros Kosiaris) [13:31:36] (03PS3) 10Alexandros Kosiaris: puppetdb: Cast settings::storeconfigs_backend [puppet] - 10https://gerrit.wikimedia.org/r/308727 [13:31:38] (03CR) 10Alexandros Kosiaris: [V: 032] puppetdb: Cast settings::storeconfigs_backend [puppet] - 10https://gerrit.wikimedia.org/r/308727 (owner: 10Alexandros Kosiaris) [13:32:16] 06Operations, 06Performance-Team, 10Traffic, 10Wikimedia-Stream, and 2 others: HTTPS-only for stream.wikimedia.org - https://phabricator.wikimedia.org/T140128#2610981 (10BBlack) Nevermind, found it via google: https://lists.wikimedia.org/pipermail/wikitech-l/2016-June/085928.html [13:32:23] RECOVERY - puppet last run on scb2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:45] (03CR) 10Rush: "If I understand, we want to put an arcanist token on the nodepool instances. In this model it is allocated via the private repo on pallad" [puppet] - 10https://gerrit.wikimedia.org/r/298097 (owner: 1020after4) [13:36:38] (03PS1) 10Mobrovac: Change-Prop: Increase tranclusion concurrency [puppet] - 10https://gerrit.wikimedia.org/r/308732 [13:36:56] (03PS1) 10Filippo Giunchedi: thumbor: better firejail comment about required /sbin [puppet] - 10https://gerrit.wikimedia.org/r/308733 [13:37:22] (03PS2) 10Filippo Giunchedi: thumbor: better firejail comment about required /sbin [puppet] - 10https://gerrit.wikimedia.org/r/308733 [13:37:42] (03CR) 10Muehlenhoff: [C: 031] thumbor: better firejail comment about required /sbin [puppet] - 10https://gerrit.wikimedia.org/r/308733 (owner: 10Filippo Giunchedi) [13:37:44] (03PS6) 10Rush: Add modules-load.d/kmod configuration for br_netfilter for Linux >= 3.18 [puppet] - 10https://gerrit.wikimedia.org/r/306633 (https://phabricator.wikimedia.org/T142388) (owner: 10Muehlenhoff) [13:39:17] (03PS3) 10BBlack: Remove geoiplookup DNS entries [dns] - 10https://gerrit.wikimedia.org/r/305422 (https://phabricator.wikimedia.org/T100902) [13:39:20] (03CR) 10Filippo Giunchedi: [C: 032] thumbor: better firejail comment about required /sbin [puppet] - 10https://gerrit.wikimedia.org/r/308733 (owner: 10Filippo Giunchedi) [13:39:21] 06Operations, 10Ops-Access-Requests, 06Research-and-Data, 10Research-collaborations, 10Research-management: Request access to data for WDQS research - https://phabricator.wikimedia.org/T142780#2611007 (10AlexKrauseTUD) 05Resolved>03Open [13:39:39] (03CR) 10Rush: [C: 031] Add modules-load.d/kmod configuration for br_netfilter for Linux >= 3.18 [puppet] - 10https://gerrit.wikimedia.org/r/306633 (https://phabricator.wikimedia.org/T142388) (owner: 10Muehlenhoff) [13:42:22] (03PS1) 10Alexandros Kosiaris: puppetdb: Fix typo in ssh known hosts generator [puppet] - 10https://gerrit.wikimedia.org/r/308734 [13:42:47] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] puppetdb: Fix typo in ssh known hosts generator [puppet] - 10https://gerrit.wikimedia.org/r/308734 (owner: 10Alexandros Kosiaris) [13:43:19] (03PS2) 10Alexandros Kosiaris: Change-Prop: Increase tranclusion concurrency [puppet] - 10https://gerrit.wikimedia.org/r/308732 (owner: 10Mobrovac) [13:43:23] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Change-Prop: Increase tranclusion concurrency [puppet] - 10https://gerrit.wikimedia.org/r/308732 (owner: 10Mobrovac) [13:43:47] thnx akosiaris! [13:44:00] <_joe_> akosiaris: thanks for fixing that error too [13:44:04] (03PS2) 10Rush: labstore: ignore autoloader layout lint error [puppet] - 10https://gerrit.wikimedia.org/r/308320 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [13:44:08] <_joe_> I must have trusted the readme :) [13:45:02] the readme is wrong obviously [13:48:00] (03PS1) 10Giuseppe Lavagetto: ssh::client: fix function call in template [puppet] - 10https://gerrit.wikimedia.org/r/308736 [13:49:08] (03PS2) 10Giuseppe Lavagetto: ssh::client: fix function call in template [puppet] - 10https://gerrit.wikimedia.org/r/308736 [13:50:14] (03CR) 10Rush: [C: 032] labstore: ignore autoloader layout lint error [puppet] - 10https://gerrit.wikimedia.org/r/308320 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [13:52:31] (03CR) 10Giuseppe Lavagetto: [C: 032] ssh::client: fix function call in template [puppet] - 10https://gerrit.wikimedia.org/r/308736 (owner: 10Giuseppe Lavagetto) [13:52:56] (03PS3) 10Giuseppe Lavagetto: ssh::client: fix function call in template [puppet] - 10https://gerrit.wikimedia.org/r/308736 [13:52:58] (03CR) 10Giuseppe Lavagetto: [V: 032] ssh::client: fix function call in template [puppet] - 10https://gerrit.wikimedia.org/r/308736 (owner: 10Giuseppe Lavagetto) [13:53:46] (03PS1) 10Filippo Giunchedi: swift: add thumbor configuration [puppet] - 10https://gerrit.wikimedia.org/r/308737 [13:54:07] 06Operations, 06Labs, 10wikitech.wikimedia.org: Can't login wikitech - https://phabricator.wikimedia.org/T144805#2610832 (10Krenair) It's two-factor authentication. You must've enabled it on your account. [13:56:28] _joe_, mutante, can I get feedback about whether or not the 'ocg' labs project is still good for something? cscott says he thinks it's in use but deferred to you [13:58:44] 06Operations, 10Ops-Access-Requests: hdfs / hive access for 'filippo' - https://phabricator.wikimedia.org/T144811#2611049 (10fgiunchedi) [13:58:51] (03PS2) 10Filippo Giunchedi: swift: add thumbor configuration [puppet] - 10https://gerrit.wikimedia.org/r/308737 [14:00:02] hehe, godog do you have to go through the same process as others? technically this is not an escalation of privilege for you [14:00:07] if you lke you can make a patch and I'll +1 it [14:00:44] <_joe_> andrewbogott: the answer to that is yes, it's still needed, given I have no idea what is the fate of ocg going to be [14:00:57] ok! Thanks [14:00:57] <_joe_> andrewbogott: until ocg is in prod, that's needed [14:01:13] <_joe_> no one is using it right now, but the day we do... [14:01:18] <_joe_> it better be there [14:01:36] (03PS1) 10Filippo Giunchedi: admin: add filippo to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/308738 (https://phabricator.wikimedia.org/T144811) [14:01:42] that's fine, I've marked it as in use [14:02:05] ottomata: yeah see above, I did the task/patch to keep track of it and be able to use my user [14:02:38] ottomata: unless there's another way, e.g. 'ops' already has access? [14:03:53] (03CR) 10Ottomata: "Nice! To bad we can't dynamically proxy to the currently active namenode, instead of just analytics1001 :)" [puppet] - 10https://gerrit.wikimedia.org/r/308721 (https://phabricator.wikimedia.org/T116192) (owner: 10Elukey) [14:03:59] godog: naw, we'd have to do some ACL crap to grant access to more than one group [14:05:31] ottomata: does it mean "ok +1" or "Nice but -1?" I didn't get it :D [14:06:03] I know that we will not able to switch automagically to the active namenode but it shouldn't be a big problem for this use case no? [14:06:55] (03CR) 10Ottomata: [C: 032] role::statistics: Limit to production networks [puppet] - 10https://gerrit.wikimedia.org/r/308576 (owner: 10Muehlenhoff) [14:06:58] (03CR) 10Ottomata: [C: 031] role::statistics: Limit to production networks [puppet] - 10https://gerrit.wikimedia.org/r/308576 (owner: 10Muehlenhoff) [14:11:26] (03CR) 10Filippo Giunchedi: [C: 031] admin: add filippo to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/308738 (https://phabricator.wikimedia.org/T144811) (owner: 10Filippo Giunchedi) [14:11:26] ottomata: ack, thanks! [14:11:31] (03CR) 10Filippo Giunchedi: [C: 032] admin: add filippo to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/308738 (https://phabricator.wikimedia.org/T144811) (owner: 10Filippo Giunchedi) [14:12:30] 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: hdfs / hive access for 'filippo' - https://phabricator.wikimedia.org/T144811#2611111 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi access granted [14:12:41] elukey: + 1 means ok nice do it! [14:12:49] ahhh okok :D [14:12:51] thanks :) [14:12:51] was just noting a nice to have [14:12:56] that would not be easy to get :) [14:13:14] (03PS4) 10Elukey: Add a proxy to the Yarn UI on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/308721 (https://phabricator.wikimedia.org/T116192) [14:13:57] (03PS2) 10Alexandros Kosiaris: Move labtest hiera in regex.yaml [puppet] - 10https://gerrit.wikimedia.org/r/304190 [14:14:23] (03CR) 10Filippo Giunchedi: [C: 032] "PCC https://puppet-compiler.wmflabs.org/3982/" [puppet] - 10https://gerrit.wikimedia.org/r/308737 (owner: 10Filippo Giunchedi) [14:14:29] (03CR) 10Ottomata: [C: 031] "+1, but concur with Moritz" [puppet] - 10https://gerrit.wikimedia.org/r/308544 (https://phabricator.wikimedia.org/T144726) (owner: 10Elukey) [14:14:31] (03PS3) 10Filippo Giunchedi: swift: add thumbor configuration [puppet] - 10https://gerrit.wikimedia.org/r/308737 [14:14:33] (03PS1) 10Giuseppe Lavagetto: ssh::client: further fixes to known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/308740 [14:17:28] <_joe_> https://cdn.meme.am/instances/56608824.jpg [14:17:46] <_joe_> best. meme. ever. [14:17:52] hahah nice [14:18:20] I'd need this on my desk while waiting https://i.redd.it/gu5vxicmi16x.gif [14:18:23] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] ssh::client: further fixes to known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/308740 (owner: 10Giuseppe Lavagetto) [14:18:59] (03CR) 10Ottomata: [C: 031] "Does the submodule update need to be here too?" [puppet] - 10https://gerrit.wikimedia.org/r/308715 (https://phabricator.wikimedia.org/T132324) (owner: 10Elukey) [14:19:43] ottomata: already merged yesterday --^ [14:19:43] (03CR) 10Ottomata: [C: 031] "analytics-cisco can be deleted" [puppet] - 10https://gerrit.wikimedia.org/r/306501 (owner: 10Dzahn) [14:19:58] (03PS3) 10Andrew Bogott: openstack: Import openstack-wikistatus nova plugin [puppet] - 10https://gerrit.wikimedia.org/r/308169 (https://phabricator.wikimedia.org/T144317) (owner: 10Alex Monk) [14:21:06] (03PS3) 10Alexandros Kosiaris: Move labtest hiera in regex.yaml [puppet] - 10https://gerrit.wikimedia.org/r/304190 [14:21:10] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Move labtest hiera in regex.yaml [puppet] - 10https://gerrit.wikimedia.org/r/304190 (owner: 10Alexandros Kosiaris) [14:23:51] (03CR) 10Andrew Bogott: [C: 032] openstack: Import openstack-wikistatus nova plugin [puppet] - 10https://gerrit.wikimedia.org/r/308169 (https://phabricator.wikimedia.org/T144317) (owner: 10Alex Monk) [14:23:56] (03PS6) 10Elukey: Add a proxy to the Yarn UI on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/308721 (https://phabricator.wikimedia.org/T116192) [14:24:06] (03CR) 10Elukey: [C: 032 V: 032] Add a proxy to the Yarn UI on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/308721 (https://phabricator.wikimedia.org/T116192) (owner: 10Elukey) [14:24:30] (after two +2 from jenkins I decided to +2 myself :) [14:24:55] :) [14:25:47] (03PS4) 10Filippo Giunchedi: swift: add thumbor configuration [puppet] - 10https://gerrit.wikimedia.org/r/308737 [14:26:37] (03PS4) 10Andrew Bogott: openstack: Import openstack-wikistatus nova plugin [puppet] - 10https://gerrit.wikimedia.org/r/308169 (https://phabricator.wikimedia.org/T144317) (owner: 10Alex Monk) [14:26:51] PROBLEM - puppet last run on labvirt1014 is CRITICAL: CRITICAL: Puppet last ran 14 days ago [14:28:56] (03PS5) 10Andrew Bogott: openstack: Import openstack-wikistatus nova plugin [puppet] - 10https://gerrit.wikimedia.org/r/308169 (https://phabricator.wikimedia.org/T144317) (owner: 10Alex Monk) [14:29:45] RECOVERY - puppet last run on labvirt1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:29:52] (03CR) 1020after4: "Any way I go about putting the token there it's going to be readable by the code running in the unit tests. I can't think of any way to av" [puppet] - 10https://gerrit.wikimedia.org/r/298097 (owner: 1020after4) [14:30:15] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/salt-common] [14:36:21] (03PS1) 10Elukey: Fix Yarn apache configuration with the missing ldap password [puppet] - 10https://gerrit.wikimedia.org/r/308742 [14:37:41] (03CR) 10Elukey: [C: 032] Fix Yarn apache configuration with the missing ldap password [puppet] - 10https://gerrit.wikimedia.org/r/308742 (owner: 10Elukey) [14:40:27] (03CR) 10Rush: "Sure, I'm not really advocating for or against. But it doesn't make sense to pretend it's private by putting it in the private repo if an" [puppet] - 10https://gerrit.wikimedia.org/r/298097 (owner: 1020after4) [14:40:43] (03PS1) 10Alexandros Kosiaris: postgres: Ensure postgresl-contrib installed [puppet] - 10https://gerrit.wikimedia.org/r/308743 [14:41:20] 06Operations, 06Performance-Team, 10Thumbor, 13Patch-For-Review: add thumbor to production infrastructure - https://phabricator.wikimedia.org/T139606#2611201 (10fgiunchedi) in terms of gradual deployment, I sampled thumbnail requests over an hour for non-commons wikis, top 30 being: ```lines=10 56349 wi... [14:41:33] exit [14:48:34] (03PS2) 10Alexandros Kosiaris: postgres: Ensure postgresl-contrib installed [puppet] - 10https://gerrit.wikimedia.org/r/308743 [14:48:44] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "pcc happy at https://puppet-compiler.wmflabs.org/3983/" [puppet] - 10https://gerrit.wikimedia.org/r/308743 (owner: 10Alexandros Kosiaris) [14:50:16] RECOVERY - puppet last run on mw2231 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:53:17] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:53:39] (03PS1) 10Filippo Giunchedi: swift: enable shadow thumb requests for small wikis [puppet] - 10https://gerrit.wikimedia.org/r/308746 (https://phabricator.wikimedia.org/T139606) [14:57:56] 06Operations: Multiple servers in codfw fail to respond to IPMI commands during reimaging - https://phabricator.wikimedia.org/T142726#2611215 (10Papaul) I checked the settings on mw2170, I disable and enable again the IPMI over LAN settings, you can try again. [15:00:17] (03PS1) 10Giuseppe Lavagetto: puppetdb: allow large POSTs from the puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/308747 [15:01:33] 06Operations, 10Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2611221 (10Milimetric) That was after I updated the password, @Nuria, we'll see if others still have problems. [15:02:49] (03CR) 10Giuseppe Lavagetto: [C: 032] puppetdb: allow large POSTs from the puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/308747 (owner: 10Giuseppe Lavagetto) [15:03:56] <_joe_> meh [15:03:58] (03PS2) 10Giuseppe Lavagetto: puppetdb: allow large POSTs from the puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/308747 [15:04:28] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] puppetdb: allow large POSTs from the puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/308747 (owner: 10Giuseppe Lavagetto) [15:05:15] (03PS2) 10Filippo Giunchedi: swift: enable shadow thumb requests for small wikis [puppet] - 10https://gerrit.wikimedia.org/r/308746 (https://phabricator.wikimedia.org/T139606) [15:05:17] (03PS1) 10Filippo Giunchedi: thumbor: fix swift account name [puppet] - 10https://gerrit.wikimedia.org/r/308748 [15:06:28] (03CR) 10Volans: "@elukey, replies inline" (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/308520 (https://phabricator.wikimedia.org/T143536) (owner: 10Volans) [15:14:55] (03PS1) 10Giuseppe Lavagetto: puppetdb: fix variable scoping [puppet] - 10https://gerrit.wikimedia.org/r/308750 [15:15:31] 06Operations, 06Performance-Team, 10Traffic, 10Wikimedia-Stream, and 2 others: HTTPS-only for stream.wikimedia.org - https://phabricator.wikimedia.org/T140128#2611233 (10BBlack) Looking at the past couple days of access logs from ori's nginx patch above, it looks like the current split is still 88% insecur... [15:18:14] (03CR) 10Giuseppe Lavagetto: [C: 032] puppetdb: fix variable scoping [puppet] - 10https://gerrit.wikimedia.org/r/308750 (owner: 10Giuseppe Lavagetto) [15:20:45] (03CR) 10Elukey: Automation: automatically reimage host (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/308520 (https://phabricator.wikimedia.org/T143536) (owner: 10Volans) [15:21:38] (03PS1) 10Alex Monk: Follow-up I65c58485: Fix style [puppet] - 10https://gerrit.wikimedia.org/r/308751 [15:25:27] (03CR) 10Andrew Bogott: [C: 032] "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/308751 (owner: 10Alex Monk) [15:26:23] (03PS2) 10Andrew Bogott: Follow-up I65c58485: Fix style [puppet] - 10https://gerrit.wikimedia.org/r/308751 (owner: 10Alex Monk) [15:28:44] /go lindseyanne_ [15:28:49] sigh, no [15:30:39] PROBLEM - puppet last run on mw2231 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:32:00] (03PS2) 10Filippo Giunchedi: thumbor: fix swift account name [puppet] - 10https://gerrit.wikimedia.org/r/308748 [15:33:46] (03CR) 10Filippo Giunchedi: [C: 032] thumbor: fix swift account name [puppet] - 10https://gerrit.wikimedia.org/r/308748 (owner: 10Filippo Giunchedi) [15:34:08] (03Draft1) 10Paladox: Add support for searching gerrit using bug:T1 [puppet] - 10https://gerrit.wikimedia.org/r/308753 [15:34:52] RECOVERY - Host mw2200 is UP: PING OK - Packet loss = 0%, RTA = 36.90 ms [15:36:01] RECOVERY - Host mw2201 is UP: PING OK - Packet loss = 0%, RTA = 47.51 ms [15:37:11] RECOVERY - Host mw2202 is UP: PING OK - Packet loss = 0%, RTA = 37.03 ms [15:37:21] RECOVERY - Host mw2203 is UP: PING OK - Packet loss = 0%, RTA = 37.35 ms [15:42:21] RECOVERY - Host mw2204 is UP: PING OK - Packet loss = 0%, RTA = 40.12 ms [15:43:09] RECOVERY - Host mw2206 is UP: PING OK - Packet loss = 0%, RTA = 37.83 ms [15:43:58] (03PS2) 10Elukey: Reduce cronspam to root@ from zookeeper zkCleanup script [puppet] - 10https://gerrit.wikimedia.org/r/308715 (https://phabricator.wikimedia.org/T132324) [15:44:10] RECOVERY - Host mw2207 is UP: PING OK - Packet loss = 0%, RTA = 36.10 ms [15:44:40] RECOVERY - Host mw2205 is UP: PING OK - Packet loss = 0%, RTA = 36.40 ms [15:45:06] (03CR) 10Elukey: [C: 032] Reduce cronspam to root@ from zookeeper zkCleanup script [puppet] - 10https://gerrit.wikimedia.org/r/308715 (https://phabricator.wikimedia.org/T132324) (owner: 10Elukey) [15:50:11] RECOVERY - Host mw2208 is UP: PING OK - Packet loss = 0%, RTA = 36.94 ms [15:50:20] RECOVERY - Host mw2209 is UP: PING WARNING - Packet loss = 58%, RTA = 228.04 ms [15:51:34] RECOVERY - Host mw2210 is UP: PING OK - Packet loss = 0%, RTA = 46.22 ms [15:52:20] RECOVERY - Host mw2211 is UP: PING OK - Packet loss = 0%, RTA = 36.92 ms [15:55:27] 06Operations, 06Commons, 06Multimedia: Deploy a PHP and HHVM patch (Exif values retrieved incorrectly if they appear before IFD) - https://phabricator.wikimedia.org/T140419#2611294 (10matmarex) Thank you for the reply! > We're currently no longer using PHP on production app servers, is that relevant for PH... [15:55:50] (03PS2) 10Paladox: Add support for searching gerrit using bug:T1 [puppet] - 10https://gerrit.wikimedia.org/r/308753 (https://phabricator.wikimedia.org/T85002) [15:56:03] (03PS3) 10Paladox: Add support for searching gerrit using bug:T1 [puppet] - 10https://gerrit.wikimedia.org/r/308753 (https://phabricator.wikimedia.org/T85002) [15:56:27] (03CR) 10Paladox: "Tested here http://gerrit-test.wmflabs.org/gerrit/#/q/bug:T67 and http://gerrit-test.wmflabs.org/gerrit/#/q/bug:67" [puppet] - 10https://gerrit.wikimedia.org/r/308753 (https://phabricator.wikimedia.org/T85002) (owner: 10Paladox) [15:58:17] (03PS3) 10Andrew Bogott: Follow-up I65c58485: Fix style [puppet] - 10https://gerrit.wikimedia.org/r/308751 (owner: 10Alex Monk) [16:00:04] godog, moritzm, and _joe_: Dear anthropoid, the time has come. Please deploy Puppet SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160906T1600). [16:00:04] mobrovac and Dereckson: A patch you scheduled for Puppet SWAT(Max 8 patches) is about to be deployed. Please be available during the process. [16:01:50] * mobrovac is here [16:02:07] taking a look mobrovac [16:02:12] RECOVERY - Host mw2213 is UP: PING OK - Packet loss = 0%, RTA = 36.18 ms [16:02:55] (03PS2) 10Filippo Giunchedi: Change-Prop: Rerender mobile-sections when wikidata item is changed [puppet] - 10https://gerrit.wikimedia.org/r/308230 (https://phabricator.wikimedia.org/T144632) (owner: 10Ppchelko) [16:03:00] RECOVERY - Host mw2212 is UP: PING OK - Packet loss = 0%, RTA = 37.49 ms [16:03:30] RECOVERY - Host mw2214 is UP: PING OK - Packet loss = 0%, RTA = 37.08 ms [16:05:08] (03PS2) 10Dzahn: Drop old code only relevant for precise/magnesium [puppet] - 10https://gerrit.wikimedia.org/r/308728 (owner: 10Muehlenhoff) [16:05:25] (03PS3) 10Muehlenhoff: RT: Drop old code only relevant for precise/magnesium [puppet] - 10https://gerrit.wikimedia.org/r/308728 [16:05:32] (03CR) 10Dzahn: [C: 032] RT: Drop old code only relevant for precise/magnesium [puppet] - 10https://gerrit.wikimedia.org/r/308728 (owner: 10Muehlenhoff) [16:06:32] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Change-Prop: Rerender mobile-sections when wikidata item is changed [puppet] - 10https://gerrit.wikimedia.org/r/308230 (https://phabricator.wikimedia.org/T144632) (owner: 10Ppchelko) [16:06:34] (03CR) 10Dzahn: "wasnt technically related to precise/jessie but i moved it behind misc-web varnish during that migration, that's why it didnt need https s" [puppet] - 10https://gerrit.wikimedia.org/r/308728 (owner: 10Muehlenhoff) [16:07:33] godog: i can run puppet and apply it [16:07:59] mobrovac: yup thanks, it is merged [16:08:35] thnx godog [16:08:40] * mobrovac running puppet [16:09:34] (03PS4) 10Dzahn: RT: Drop old code only relevant for precise/magnesium [puppet] - 10https://gerrit.wikimedia.org/r/308728 (owner: 10Muehlenhoff) [16:10:39] Dereckson: what's a wikitech url we can test your change with? [16:10:47] the change being https://gerrit.wikimedia.org/r/#/c/308671 [16:10:54] * Dereckson checks. [16:11:24] !log change-prop restarting to pick up https://gerrit.wikimedia.org/r/308230 [16:11:25] https://wikitech.wikimedia.org/w/images/math/6/1/9/619a7845480ba7a8a749dc56a6de7c60.png [16:11:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:13:46] (03PS3) 10Filippo Giunchedi: Wikitech: Serve /srv/math-images as /w/images/math [puppet] - 10https://gerrit.wikimedia.org/r/308671 (https://phabricator.wikimedia.org/T126338) (owner: 10Dereckson) [16:14:17] (03CR) 10Dzahn: "hashar, actually it's enforced since we had 0 of them left in the repo. then some creep back in again when people override the vote" [puppet] - 10https://gerrit.wikimedia.org/r/308316 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:15:23] (03CR) 10Filippo Giunchedi: [C: 032] Wikitech: Serve /srv/math-images as /w/images/math [puppet] - 10https://gerrit.wikimedia.org/r/308671 (https://phabricator.wikimedia.org/T126338) (owner: 10Dereckson) [16:16:24] (03PS4) 10Paladox: manifests/role/dns: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308316 (https://phabricator.wikimedia.org/T93645) [16:16:41] (03CR) 10Dzahn: [C: 032] "http://puppet-compiler.wmflabs.org/3988/" [puppet] - 10https://gerrit.wikimedia.org/r/308316 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:17:10] (03PS5) 10Paladox: manifests/role/dns: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308316 (https://phabricator.wikimedia.org/T93645) [16:17:15] mutante ^^ i reabsed it [16:17:31] paladox: i am already doing it, no need to hit rebase again [16:17:38] oh sorry [16:18:19] (03PS4) 10Andrew Bogott: Follow-up I65c58485: Fix style [puppet] - 10https://gerrit.wikimedia.org/r/308751 (owner: 10Alex Monk) [16:18:37] Puppet indicates a failure: https://integration.wikimedia.org/ci/job/operations-puppet-doc/26157/console [16:18:54] Dereckson that is [16:18:56] a known [16:18:58] problem [16:19:03] and likly fixed by [16:19:27] https://gerrit.wikimedia.org/r/#/c/307654/ [16:19:38] Dereckson https://phabricator.wikimedia.org/T143233 [16:20:54] Dereckson: patch works but I keep forgetting apache needs Alias most-specific-first, mind swapping it? [16:21:07] I've manually swapped it on silver to test [16:21:48] okay I'm preparing a follow-up patch [16:22:03] RECOVERY - puppet last run on mw2231 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:22:15] kk thanks! [16:23:44] (03PS8) 10Paladox: role/cxserver: Fix role::cxserver not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308498 (https://phabricator.wikimedia.org/T93645) [16:25:26] godog: mine's good too, thnx! [16:25:46] mobrovac: np! [16:28:04] (03PS1) 10Dereckson: Wikitech: fix Apache Alias order [puppet] - 10https://gerrit.wikimedia.org/r/308761 [16:28:37] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Wikitech: fix Apache Alias order [puppet] - 10https://gerrit.wikimedia.org/r/308761 (owner: 10Dereckson) [16:30:43] PROBLEM - cassandra-b CQL 10.192.32.138:9042 on restbase2004 is CRITICAL: Connection refused [16:30:52] (03PS6) 10Dzahn: manifests/role/dns: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308316 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:30:55] godog: thanks for the deployment [16:31:30] is u [16:31:30] always unreliable ..every time [16:31:30] eee [16:33:13] Dereckson: np, looks like the test url works [16:35:09] godog: I tested another equation, Math extension succesfully generated the file, Apache served it fine too [16:36:08] Do we need to do something special at Varnish level? [16:37:25] 06Operations, 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Enable math extension on wikitech - https://phabricator.wikimedia.org/T126338#2611496 (10Dereckson) 05Open>03Resolved @Halfak All works fine. You can add back your equations on https://wikitech.wikimedia.o... [16:37:41] halfak: ^ [16:40:05] Dereckson: Could you do some queries on meta's user_properties table for me? [16:41:37] (03PS4) 10Dzahn: ganglia: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308339 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:41:50] (03PS1) 10Urbanecm: Enable RC patrol for fiwiki and some permissions changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308764 (https://phabricator.wikimedia.org/T144817) [16:42:15] Glaisher: yes, do you have a ticket somewhere? [16:42:16] (03CR) 10Dzahn: [C: 032] ganglia: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308339 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:42:34] ostriches: are you around? :) [16:42:35] thanks ^^ mutante :) [16:42:38] Dereckson: It's for debugging https://phabricator.wikimedia.org/T144780 [16:42:46] addshore: I am, what's up? [16:42:51] Some queries on translation notifications rows [16:43:31] I have 5 patches that I would like to add to the branch for the RevisionSlider that has just been made! is that okay with you? Just checking youll grab them with the train! (they are in master) [16:44:01] (03PS3) 10Dzahn: deployment: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308336 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:44:35] (03CR) 10Dzahn: [C: 032] deployment: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308336 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:45:05] we missed the branch by about 30 mins because of a failing browser test ;) [16:45:26] addshore: Merge them and I'll make sure to pull the core submodule update. [16:45:43] merge them in the branch? :) Will do / doing now! [16:45:50] Thanks! [16:46:09] 06Operations, 10Traffic: Strong cipher preference ordering for cache terminators - https://phabricator.wikimedia.org/T144626#2611554 (10BBlack) Link to mozilla NSS bug on doing AES-NI pref hacks on the client side (no real info there yet): https://bugzilla.mozilla.org/show_bug.cgi?id=1279584 [16:46:34] (03CR) 10Jcrespo: [C: 04-2] "Please see https://gerrit.wikimedia.org/r/301076" [puppet] - 10https://gerrit.wikimedia.org/r/308315 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:46:39] (03PS3) 10Dzahn: nagios_common: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308344 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:46:41] addshore: Weeks where I do the train the branch cut tends to happen about 9am :) [16:47:06] (03Abandoned) 10Paladox: manifests/role: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308315 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:47:06] 9am pst? ;) [16:47:14] (03CR) 10Paladox: [C: 031] Delete coredb_mysql module and dependent roles and modules [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [16:47:47] (03CR) 10Dzahn: "cool, i had just added you here to confirm that :)" [puppet] - 10https://gerrit.wikimedia.org/r/308315 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:48:19] (03PS2) 10BBlack: text VCL: remove support for rest.wm.o [puppet] - 10https://gerrit.wikimedia.org/r/308724 (https://phabricator.wikimedia.org/T133001) [16:48:56] (03CR) 10Dzahn: [C: 032] nagios_common: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308344 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:48:59] (03CR) 10Jcrespo: "This is still blocked on analytics to stop using or migrating mysql_wmf::mylvmbackup to the mariadb module." [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [16:49:18] (03CR) 10BBlack: [C: 032 V: 032] "DNS change went out a few hours ago, TTL was ~10 mins." [puppet] - 10https://gerrit.wikimedia.org/r/308724 (https://phabricator.wikimedia.org/T133001) (owner: 10BBlack) [16:49:44] (03PS3) 10BBlack: text VCL: remove support for rest.wm.o [puppet] - 10https://gerrit.wikimedia.org/r/308724 (https://phabricator.wikimedia.org/T133001) [16:49:45] 06Operations, 10hardware-requests: CODFW: (2) hardware access request for PUPPET - https://phabricator.wikimedia.org/T142219#2611576 (10RobH) 05Open>03Resolved These systems now have been received and have racking tasks (linked in the dependency chain of tasks.) [16:49:47] (03CR) 10BBlack: [V: 032] text VCL: remove support for rest.wm.o [puppet] - 10https://gerrit.wikimedia.org/r/308724 (https://phabricator.wikimedia.org/T133001) (owner: 10BBlack) [16:50:20] (03PS1) 10Andrew Bogott: Add egg-info files for wikistatus plugin. [puppet] - 10https://gerrit.wikimedia.org/r/308770 [16:50:26] (03PS3) 10BBlack: deployment caches: enforce secure POST (etc) [puppet] - 10https://gerrit.wikimedia.org/r/307762 [16:50:45] (03CR) 10BBlack: [C: 032 V: 032] deployment caches: enforce secure POST (etc) [puppet] - 10https://gerrit.wikimedia.org/r/307762 (owner: 10BBlack) [16:52:07] ostriches: all merged! many thanks :) [16:53:32] 06Operations, 10Citoid, 10ContentTranslation-CXserver, 10RESTBase, and 4 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2611591 (10BBlack) `rest.wikimedia.org` is gone. We've still got `citoid.wikimedia.org` and `cxserver.w... [16:54:15] addshore: Already pulled to tin, they'll be in the scap [16:54:25] ostriches: great! :) [16:54:25] (03CR) 10Jcrespo: "Sorry there is so much old code; my first year here could be simplified as "making sure we could finally drop this unmaintained code". We " [puppet] - 10https://gerrit.wikimedia.org/r/308315 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [16:55:31] !log demon@tin Started scap: 1.28.0-wmf.18 initial scap for l10n build (testwiki) [16:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:58:22] 06Operations, 10Cassandra, 06Services: restbase2004-b.codfw.wmnet CQL service failure - https://phabricator.wikimedia.org/T144826#2611611 (10Eevans) [17:00:04] yurik, gwicke, cscott, arlolra, subbu, halfak, and Amir1: Respected human, time to deploy Services – Graphoid / Parsoid / OCG / Citoid / ORES (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160906T1700). Please do the needful. [17:00:14] (03CR) 10Yuvipanda: google_api_proxy: Add role for Google-api-proxy Labs project (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308664 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [17:00:25] bd808 ^ bikeshed [17:00:52] * bd808 looks for a paintbrush [17:00:54] (03CR) 10Paladox: "@Jcrespo do you know how we can do that please? Is there a easy way since I carn't find a replacement mariadb module that can replace that" [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [17:01:04] bd808 almost. I care about it at least a little bit since I spent time cleaning stuff outta there, and also people tended to ping me / us everytime there was a problem with a role with the word 'labs' in it [17:01:10] yuvipanda: so andrewbogott suggested the role::labs::* location. [17:01:23] but I get your pain [17:01:52] it has to be in some sub-dir or T119042 melts the world [17:01:53] T119042: Import vs autoload: the puppet parser is a bad joke that stopped being funny years ago. - https://phabricator.wikimedia.org/T119042 [17:02:21] !log T144826: Restaring Cassandra on restbase2004-a.codfw.wmnet [17:02:22] T144826: restbase2004-b.codfw.wmnet CQL service failure - https://phabricator.wikimedia.org/T144826 [17:02:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:03:07] yuvipanda: so... I could invent a new kind of generic sub-module like 'projects' or something I guess [17:03:32] bd808 I think in that case I don't particularly care now and we should drop the bikeshed :) [17:03:41] bd808 if you've tested this, I'll merge [17:04:00] mutante: https://etherpad.wikimedia.org/p/TechOps-goals-FQ2-FY1617 [17:04:38] yuvipanda: for certain values of "test", yes. That project doesn't have a self-hosted master so I haven't really applied it anywhere yet [17:05:03] bd808 ok, in that case I'll merge, and then you can test and I can merge followups. [17:05:03] 06Operations, 10Cassandra, 06Services: restbase2004-b.codfw.wmnet CQL service failure - https://phabricator.wikimedia.org/T144826#2611658 (10Eevans) [17:05:05] does that sound good? [17:05:16] RECOVERY - check_disk on barium is OK: DISK OK - free space: / 16710 MB (31% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803150 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643340 MB (35% inode=99%) [17:05:22] yuvipanda: :) yes, please [17:05:40] (03PS4) 10Yuvipanda: google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308664 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [17:05:46] (03CR) 10Yuvipanda: [C: 032 V: 032] google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308664 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [17:07:27] (03Abandoned) 1020after4: Contint: add pyflakes/flake8 for pep8 linting [puppet] - 10https://gerrit.wikimedia.org/r/306851 (owner: 1020after4) [17:07:36] (03PS1) 10Chad: checkoutMW: I don't think we need to update branch pointers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308772 [17:08:09] (03PS2) 10Chad: checkoutMW: I don't think we need to update branch pointers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308772 [17:08:28] (03CR) 1020after4: "The reasoning for this is that I'm trying to get `arc lint` working... will have to setting for tox." [puppet] - 10https://gerrit.wikimedia.org/r/306851 (owner: 1020after4) [17:09:26] jynus: Are you still available to help with monitoring of the PageAssessments roll-out today? [17:09:27] bd808 let me know if you need followups [17:09:53] yuvipanda: thanks! I'll put the role on my existing host and see what melts. :) [17:09:55] ACKNOWLEDGEMENT - cassandra-b CQL 10.192.32.138:9042 on restbase2004 is CRITICAL: Connection refused eevans https://phabricator.wikimedia.org/T144826 [17:10:27] !log T144826: Starting online scrub [17:10:28] T144826: restbase2004-b.codfw.wmnet CQL service failure - https://phabricator.wikimedia.org/T144826 [17:10:30] bd808 yeah ok [17:10:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:11:28] (03PS1) 10Mobrovac: Change Prop: Send the if-unmodified-since header on transclusion updates [puppet] - 10https://gerrit.wikimedia.org/r/308773 [17:11:54] (03PS1) 10Chad: group0 to 1.28.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308774 [17:13:33] !log T144826: Lifting compaction throttle [17:13:34] T144826: restbase2004-b.codfw.wmnet CQL service failure - https://phabricator.wikimedia.org/T144826 [17:13:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:16:03] (03CR) 10Yuvipanda: [C: 031] "lgtm, can you bump debian version too?" [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/307463 (https://phabricator.wikimedia.org/T143282) (owner: 10BryanDavis) [17:17:29] 06Operations, 10Cassandra, 06Services: restbase2004-b.codfw.wmnet data corruption - https://phabricator.wikimedia.org/T144826#2611694 (10Eevans) p:05Triage>03High [17:19:26] (03CR) 10Jcrespo: "Paladox: note that mariadb module is in a submodule, maybe you missed that?" [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [17:19:42] (03PS2) 10Urbanecm: Enable RC patrol for fiwiki and some related changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308764 (https://phabricator.wikimedia.org/T144817) [17:20:24] !log T144826: Ephemerally increasing compactor thread count from 10 to 20 [17:20:25] T144826: restbase2004-b.codfw.wmnet data corruption - https://phabricator.wikimedia.org/T144826 [17:20:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:21:49] (03CR) 10Paladox: "@Jcrespo I guess if your ok with moving the class, we could just move over https://github.com/wikimedia/operations-puppet/blob/f4420ead5ea" [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [17:22:10] (03CR) 10Paladox: "Also thanks for the link to mariadb, I eventually found it." [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [17:26:34] !log starting Parsoid deploy [17:26:51] !log starting Parsoid deploy [17:26:55] (03PS1) 10Elukey: Move yarn.w.o traffic to stat1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/308776 (https://phabricator.wikimedia.org/T116192) [17:26:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:28:25] (03PS1) 10Hashar: zuul: refactor to use hiera [puppet] - 10https://gerrit.wikimedia.org/r/308778 [17:29:43] (03CR) 10jenkins-bot: [V: 04-1] zuul: refactor to use hiera [puppet] - 10https://gerrit.wikimedia.org/r/308778 (owner: 10Hashar) [17:32:06] (03PS1) 10BryanDavis: external_proxy: Respect XFF headers [puppet] - 10https://gerrit.wikimedia.org/r/308780 (https://phabricator.wikimedia.org/T144290) [17:32:30] yuvipanda: ^ one little thing I left out. Works well other than that [17:33:29] (03CR) 10Yuvipanda: external_proxy: Respect XFF headers (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308780 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [17:34:22] yuvipanda: well... good question. I wonder if it's in puppet somewhere [17:34:40] (03PS2) 10Andrew Bogott: Add egg-info files for wikistatus plugin. [puppet] - 10https://gerrit.wikimedia.org/r/308770 [17:35:31] (03PS3) 10Andrew Bogott: Add egg-info files for wikistatus plugin. [puppet] - 10https://gerrit.wikimedia.org/r/308770 [17:37:48] 06Operations, 06MediaWiki-Stakeholders-Group, 10Traffic, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2611800 (10BBlack) Bump - the geoiplookup JSON service (the hostname and the /geoiplookup) path will go away sometime this week, preferably t... [17:38:10] !log updated Parsoid to version 7863e6ad (T142617) [17:38:10] (03CR) 10BryanDavis: [C: 04-1] external_proxy: Respect XFF headers (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308780 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [17:38:11] T142617: Parsoid doesn't url-decode link before sending it to mediawiki-title - https://phabricator.wikimedia.org/T142617 [17:38:12] (03CR) 10Andrew Bogott: [C: 032] Add egg-info files for wikistatus plugin. [puppet] - 10https://gerrit.wikimedia.org/r/308770 (owner: 10Andrew Bogott) [17:38:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:47:24] !log demon@tin Finished scap: 1.28.0-wmf.18 initial scap for l10n build (testwiki) (duration: 51m 52s) [17:47:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:48:45] kaldari, on a meeting [17:49:00] but I am around [17:55:38] subbu: arlolra: will try to repool manually [17:55:49] ok .. but, is that a scap issue? [17:56:27] (03PS1) 10Paladox: Copy over mysql_wmf::mylvmbackup to mariadb [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/308785 [17:56:51] (03CR) 10Paladox: "@Jcrespo done here https://gerrit.wikimedia.org/r/308785" [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [17:56:58] 06Operations, 10Cassandra, 06Services: restbase2004-b.codfw.wmnet data corruption - https://phabricator.wikimedia.org/T144826#2611912 (10Eevans) [17:57:34] are there any logs about why it failed to repool / notifications that it wasn't? [17:58:27] subbu: arlolra: wtp1009.eqiad.wmnet: pooled changed no => yes [17:58:33] no, it's not a scap issue [17:58:40] (03CR) 10Paladox: "Slight changes to it, I replaced MySQL with MariaDB" [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/308785 (owner: 10Paladox) [17:59:40] what went wrong then? [18:00:04] anomie, ostriches, thcipriani, hashar, and twentyafterfour: Dear anthropoid, the time has come. Please deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160906T1800). [18:00:04] James_F: A patch you scheduled for Morning SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [18:00:10] * James_F waves. [18:00:18] I can do it [18:00:24] (03CR) 10Jcrespo: "Thank you very much, I will give it a look ASAP." [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [18:00:25] Cool. [18:00:36] (03PS5) 10Dzahn: puppetmaster: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308347 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [18:00:39] (03PS2) 10BryanDavis: external_proxy: Respect XFF headers [puppet] - 10https://gerrit.wikimedia.org/r/308780 (https://phabricator.wikimedia.org/T144290) [18:00:42] (03PS3) 10Chad: Enable VisualEditor by default for logged-out users on Indic-script Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304118 (https://phabricator.wikimedia.org/T142586) (owner: 10Jforrester) [18:01:04] (03CR) 10Ottomata: [C: 031] Move yarn.w.o traffic to stat1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/308776 (https://phabricator.wikimedia.org/T116192) (owner: 10Elukey) [18:01:31] (03CR) 10Paladox: "Your welcome, I hope it looks ok, it was a copy and paste with slight changes such as using references to mariadb instead of MySQL." [puppet] - 10https://gerrit.wikimedia.org/r/301076 (owner: 10Jcrespo) [18:01:33] (03CR) 10Dzahn: [C: 032] "no-op http://puppet-compiler.wmflabs.org/3989/" [puppet] - 10https://gerrit.wikimedia.org/r/308347 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [18:02:29] (03CR) 10Chad: [C: 032] Enable VisualEditor by default for logged-out users on Indic-script Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304118 (https://phabricator.wikimedia.org/T142586) (owner: 10Jforrester) [18:02:52] (03CR) 10Paladox: "One we are happy with this we can submit a follow up that moves analytics_cluster to use this class." [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/308785 (owner: 10Paladox) [18:02:55] (03Merged) 10jenkins-bot: Enable VisualEditor by default for logged-out users on Indic-script Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304118 (https://phabricator.wikimedia.org/T142586) (owner: 10Jforrester) [18:03:14] (03PS2) 10RobH: new star.wmfusercontent.org certificates [puppet] - 10https://gerrit.wikimedia.org/r/307534 (https://phabricator.wikimedia.org/T140649) [18:03:16] (03PS4) 10Dzahn: docker: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308337 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [18:05:04] (03CR) 10Dzahn: [C: 032] docker: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308337 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [18:05:12] ^^ thanks [18:05:50] !log demon@tin Synchronized wmf-config/InitialiseSettings.php: Enable VisualEditor by default for logged-out users on Indic-script wikipædias (duration: 00m 50s) [18:05:54] 10Blocked-on-Operations, 06Operations, 10Parsoid, 10Salt: Disabling agent forwarding breaks dsh based restarts for Parsoid (required for deployments) - https://phabricator.wikimedia.org/T102039#2611972 (10ssastry) 05Open>03declined Parsoid has moved to scap3, so this is no longer an issue. [18:05:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:06:00] James_F: You're live :D [18:06:03] (03CR) 10Dzahn: [C: 032] phabricator: Add text/x-python to files.viewable-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/307456 (owner: 10Paladox) [18:06:11] (03PS7) 10Dzahn: phabricator: Add text/x-python to files.viewable-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/307456 (owner: 10Paladox) [18:06:22] mutante ^^ thanks :) [18:07:10] ostriches: ostriches Thanks! [18:07:38] Looks good to me. [18:08:06] (03CR) 1020after4: [C: 031] "yeah seems sane to me" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308772 (owner: 10Chad) [18:08:50] James_F: np. Easy swat today :) [18:09:33] paladox: you can try it now [18:09:45] Ok thanks [18:12:20] mutante it works [18:12:22] thankyou [18:12:23] :) [18:12:57] (03CR) 1020after4: [C: 031] Fix setting phabricator logo [puppet] - 10https://gerrit.wikimedia.org/r/308308 (owner: 10Paladox) [18:13:23] great, ok, then i will also do that one, with the +1 now :) [18:13:34] thanks [18:13:54] (03PS4) 10Dzahn: Fix setting phabricator logo [puppet] - 10https://gerrit.wikimedia.org/r/308308 (owner: 10Paladox) [18:14:02] (03CR) 10Dzahn: [C: 032] Fix setting phabricator logo [puppet] - 10https://gerrit.wikimedia.org/r/308308 (owner: 10Paladox) [18:17:23] (03CR) 10Dzahn: [V: 032] "already verified" [puppet] - 10https://gerrit.wikimedia.org/r/308308 (owner: 10Paladox) [18:18:41] logo change applied [18:19:16] thanks [18:19:44] yw, bbl [18:19:49] ok [18:19:51] 06Operations, 10Cassandra, 06Services: restbase2004-b.codfw.wmnet data corruption - https://phabricator.wikimedia.org/T144826#2612010 (10Eevans) @fgiunchedi reports (on IRC), that he checked the raid status and all disks are reported as being healthy. [18:20:27] (03CR) 10Thcipriani: [C: 031] "handled in updateWikiVersions" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308772 (owner: 10Chad) [18:21:25] (03PS3) 10Chad: checkoutMW: I don't think we need to update branch pointers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308772 [18:21:56] Is something, like, apocalyptic happening to the Commons API or uploads? [18:22:54] (03CR) 10Chad: [C: 032] checkoutMW: I don't think we need to update branch pointers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308772 (owner: 10Chad) [18:23:04] Never mind [18:23:11] !log running updates on cache_misc systems to update wmfusercontent certificate [18:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:23:22] (03Merged) 10jenkins-bot: checkoutMW: I don't think we need to update branch pointers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308772 (owner: 10Chad) [18:24:48] !log demon@tin Synchronized multiversion/checkoutMediaWiki: rm branch pointer junk (duration: 00m 45s) [18:24:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:25:12] (03CR) 10RobH: [C: 032] new star.wmfusercontent.org certificates [puppet] - 10https://gerrit.wikimedia.org/r/307534 (https://phabricator.wikimedia.org/T140649) (owner: 10RobH) [18:25:21] (03PS3) 10RobH: new star.wmfusercontent.org certificates [puppet] - 10https://gerrit.wikimedia.org/r/307534 (https://phabricator.wikimedia.org/T140649) [18:30:50] mutante: do you understand the dmesg output in https://phabricator.wikimedia.org/T144826? hpsa is HP Smart Array afaik, but that host should have a software array [18:31:00] gwicke: I was going to make a hiera change that affects services-deploy.services.eqiad.wmflabs… but it looks like puppet is broken there already? [18:31:37] any idea what's happening there? [18:31:38] mutante: do the HPs require to use the raid controller? [18:31:54] gwicke: What I want to do is add "trebuchet::deployment-server": services-deploy to the hiera config so I can deprecate the global variable that that uses currently [18:32:40] ostriches: Any objections to me quickly updating Wikidata for wmf18? [18:32:42] /cc mobrovac ^^ [18:33:07] jynus: During testing today, I discovered an obscure edge case bug with PageAssessments, so I'm going to fix that and then reschedule the roll out for another time, possibly next week. So you're off the hook for now :) [18:33:07] 06Operations, 10Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2612049 (10Tbayer) Thanks - I should clarify that the bug was about the user "piwik" instead of "piwik-wmf-admin", and the password for the former still doesn't work. I am fine with using the admi... [18:33:15] mutante: to clarify, is that raid controller still in the picture somehow, even if we have software raid? [18:33:55] andrewbogott: not sure whose services-deploy is really [18:33:57] lemme look around [18:34:44] mobrovac: it was weird, I ran puppet (but didn't enable it) and it did a bunch of things [18:34:48] like the cron was disabled, maybe? [18:35:34] i'm getting my pubkey denied there [18:35:36] wth? [18:35:44] kaldari, no problem; I can monitor things at any time; just send a ping this way [18:36:11] moritzm: are you still around? ^^^ [18:36:20] andrewbogott: can you check if services-deploy is associated with any proxy? [18:36:56] jynus: will do. Thanks! [18:37:24] mobrovac: not as easily as you can, I'm not projectadmin there [18:37:42] k [18:38:40] (03PS1) 10Chad: getMWVersion: Unused, dubiously useful [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308789 [18:39:06] (03PS1) 10Chad: Remove updateGroup1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308791 [18:39:35] hoo: Go ahead [18:41:06] andrewbogott: euh, could you reset my settings on horizon/wikitech and remove 2fa? i lost my phone so can't log in now there any more ... [18:41:54] mobrovac: can you touch a file 'reset2fa' in your homedir on the labs bastion for ID confirmation? [18:42:01] anyone know if wmf.18 has been cut yet? [18:42:12] kaldari: It has [18:42:17] rats [18:42:37] curse my metal body [18:42:38] :) [18:42:53] I'm being super lenient today though, if you merge to your wmf.18 branch on an extension or core soon enough I'll pull them at noon :) [18:43:04] andrewbogott: done on bastion-01 [18:44:54] what's your wikitech username? [18:45:08] andrewbogott: Mobrovac [18:46:26] mobrovac: ok, reset [18:46:30] thnx [18:47:13] mobrovac: since puppet is broken there anyway I'm going to go ahead and make that hiera change. It should have been a no-op (if that instance was doing anything and had working puppet) [18:47:33] kk andrewbogott, works for me, thnx [18:48:35] RECOVERY - HTTPS-wmfusercontent on phab.wmfusercontent.org is OK: SSL OK - Certificate *.wmfusercontent.org valid until 2017-08-31 15:56:02 +0000 (expires in 358 days) [18:49:04] !log restbase deploy 4c239f2fa to staging [18:49:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:51:34] andrewbogott: hm, horizon let me log in (after i've re-set-up 2fa on wikitech), but when i click to change the project, it asks me for the 2fa code again and says i'm not authorised [18:51:43] not sure for what exactly [18:51:47] 'click to change the project'? [18:51:56] 06Operations, 10Mail, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown, 05Security: DMARC: Users cannot send emails via a wiki's [[Special:EmailUser]] - https://phabricator.wikimedia.org/T66795#685159 (10Colin) Why are these email bug reports not being dealt with? One suggestion, that huge group... [18:51:59] You need to select 'make active project' in the pull-down on the right [18:52:15] ah right [18:52:18] ok, it worked now [18:53:39] andrewbogott: gwicke: services-deploy.services.eqiad.wmnet does not seem to be used for anything [18:53:45] i'm proposing to get rid of it [18:53:54] works for me :) [18:54:12] (03CR) 10Chad: "Since this will trigger a gerrit restart I'd like to bundle it with something that actually restarts it for a good reason :)" [puppet] - 10https://gerrit.wikimedia.org/r/256663 (https://phabricator.wikimedia.org/T75997) (owner: 10Thiemo Mättig (WMDE)) [18:54:25] !log restbase deploy 4c239f2fa canary on restbase1007 [18:54:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:55:07] hoo: 5 mins until group1, you get your stuff merved? [18:55:11] kaldari: Likewise ^? [18:55:45] ostriches: I +2ed it [18:55:52] waiting on jenkins [18:55:55] ostriches: it's merging. almost done... [18:57:02] (03PS4) 10Volans: Automation: automatically reimage host [puppet] - 10https://gerrit.wikimedia.org/r/308520 (https://phabricator.wikimedia.org/T143536) [18:57:24] Thanks guys [18:58:43] !log restbase deploy 4c239f2fa [18:58:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:00:04] ostriches: Dear anthropoid, the time has come. Please deploy MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160906T1900). [19:00:30] ostriches: meh, still waiting on Jenkins :P [19:01:21] PROBLEM - Restbase LVS codfw on restbase.svc.codfw.wmnet is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:01:27] jouncebot: Hold yer horses [19:02:01] PROBLEM - restbase endpoints health on restbase2003 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:02:02] ostriches: yay, merged now! [19:02:06] PROBLEM - restbase endpoints health on restbase2006 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:02:06] PROBLEM - restbase endpoints health on restbase2002 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:02:08] ostriches: Wikidata wmf18 has been merged [19:02:37] gate and submit is all clear: https://integration.wikimedia.org/zuul/ [19:02:38] PROBLEM - restbase endpoints health on restbase1015 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:03:06] PROBLEM - restbase endpoints health on restbase2001 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:03:06] PROBLEM - restbase endpoints health on restbase2008 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:03:06] PROBLEM - restbase endpoints health on restbase2005 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:03:06] PROBLEM - restbase endpoints health on restbase2004 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:03:06] PROBLEM - restbase endpoints health on restbase2009 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:03:08] PROBLEM - restbase endpoints health on restbase2007 is CRITICAL: /page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage had an unexpected value for header content-type: application/json: charset=utf-8: profile=https://www.mediawiki.org/wiki/Specs/Summary/1.0.0 [19:03:46] Ok I see PageAssessments in. [19:03:50] RESTBase alerts should be fine [19:04:05] should stop as soon as the rolling restart finishes [19:04:55] ostriches: that one's mine [19:05:04] Yep [19:05:10] RECOVERY - restbase endpoints health on restbase1015 is OK: All endpoints are healthy [19:05:22] hoo: I'm not seeing wmf.18's submodule bump [19:05:36] RECOVERY - restbase endpoints health on restbase2001 is OK: All endpoints are healthy [19:05:36] !log cache misc updates for wmfusercontent complete [19:05:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:05:56] ostriches: weird [19:06:20] RECOVERY - Restbase LVS codfw on restbase.svc.codfw.wmnet is OK: All endpoints are healthy [19:06:34] (03PS2) 10BryanDavis: webservice: Warn when using lighttpd-precise [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/307463 (https://phabricator.wikimedia.org/T143282) [19:06:47] (03PS2) 10Hashar: zuul: refactor to use hiera [puppet] - 10https://gerrit.wikimedia.org/r/308778 [19:07:02] RECOVERY - restbase endpoints health on restbase2003 is OK: All endpoints are healthy [19:07:07] RECOVERY - restbase endpoints health on restbase2002 is OK: All endpoints are healthy [19:08:06] RECOVERY - restbase endpoints health on restbase2004 is OK: All endpoints are healthy [19:08:06] RECOVERY - restbase endpoints health on restbase2005 is OK: All endpoints are healthy [19:08:30] (03CR) 10BryanDavis: "> lgtm, can you bump debian version too?" [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/307463 (https://phabricator.wikimedia.org/T143282) (owner: 10BryanDavis) [19:08:50] (03CR) 10jenkins-bot: [V: 04-1] zuul: refactor to use hiera [puppet] - 10https://gerrit.wikimedia.org/r/308778 (owner: 10Hashar) [19:09:32] RECOVERY - restbase endpoints health on restbase2006 is OK: All endpoints are healthy [19:10:36] RECOVERY - restbase endpoints health on restbase2009 is OK: All endpoints are healthy [19:10:36] RECOVERY - restbase endpoints health on restbase2008 is OK: All endpoints are healthy [19:10:38] RECOVERY - restbase endpoints health on restbase2007 is OK: All endpoints are healthy [19:11:36] hoo: I'm going to go ahead [19:11:45] We'll clean it up afterwords or during swat worst case [19:12:11] (03CR) 10Chad: [C: 032] group0 to 1.28.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308774 (owner: 10Chad) [19:12:23] (03CR) 10Hashar: "Puppet compiler is not happy https://puppet-compiler.wmflabs.org/3991/" [puppet] - 10https://gerrit.wikimedia.org/r/308778 (owner: 10Hashar) [19:12:27] ostriches: Ok [19:12:41] We can scap it later on, if needed [19:13:05] (03PS2) 10Chad: group0 to 1.28.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308774 [19:15:16] (03PS1) 10Ppchelko: Change-Prop: Increase concurrency for transclusions [puppet] - 10https://gerrit.wikimedia.org/r/308803 [19:15:38] !log demon@tin Started scap: group0 to wmf.18 [19:15:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:24:39] (03PS19) 1020after4: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [19:25:20] (03PS20) 1020after4: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [19:26:30] (03CR) 1020after4: [C: 032] Remove updateGroup1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308791 (owner: 10Chad) [19:26:47] (03CR) 1020after4: [V: 032] Remove updateGroup1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308791 (owner: 10Chad) [19:27:09] (03PS2) 1020after4: Remove updateGroup1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308791 (owner: 10Chad) [19:27:32] (03CR) 1020after4: [V: 032] Remove updateGroup1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308791 (owner: 10Chad) [19:29:19] (03PS1) 10Hashar: zuul: stop logging paramiko [puppet] - 10https://gerrit.wikimedia.org/r/308805 (https://phabricator.wikimedia.org/T137525) [19:36:12] (03CR) 10BryanDavis: [C: 032] base: Set default locale to be nicer than C [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/308692 (https://phabricator.wikimedia.org/T144794) (owner: 10Yuvipanda) [19:36:30] (03Merged) 10jenkins-bot: base: Set default locale to be nicer than C [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/308692 (https://phabricator.wikimedia.org/T144794) (owner: 10Yuvipanda) [19:43:11] !log demon@tin Finished scap: group0 to wmf.18 (duration: 27m 33s) [19:43:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:46:13] (03PS1) 10Chad: Tweak GlobalTitleFail logging, I'm going to start fixing this [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308808 [19:48:05] !log demon@tin Synchronized multiversion/: rm useless file (duration: 01m 05s) [19:48:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:53:35] (03PS1) 10Andrew Bogott: wikistatus: Rename some oslo. imports to oslo_ [puppet] - 10https://gerrit.wikimedia.org/r/308809 [19:55:00] (03CR) 10Andrew Bogott: [C: 032] wikistatus: Rename some oslo. imports to oslo_ [puppet] - 10https://gerrit.wikimedia.org/r/308809 (owner: 10Andrew Bogott) [19:55:58] 06Operations, 06Performance-Team, 10scap, 07Epic: During deployment old servers may populate new cache URIs - https://phabricator.wikimedia.org/T47877#2612381 (10Krinkle) [19:57:05] (03PS1) 10BBlack: ciphersuites: drop mid-level SHA-2 HMAC suites [puppet] - 10https://gerrit.wikimedia.org/r/308810 [20:05:00] (03PS1) 10Andrew Bogott: mediawiki_singlenode: Move $::labs_mediawiki_hostname to a param [puppet] - 10https://gerrit.wikimedia.org/r/308812 [20:09:40] (03CR) 10Alex Monk: [C: 031] "maybe add Bug: T101447 to the commit message" [puppet] - 10https://gerrit.wikimedia.org/r/308812 (owner: 10Andrew Bogott) [20:09:46] ostriches: What's the state? Is Wikidata wmf18 out? [20:12:37] Ah not yet [20:12:40] Lemme look [20:12:45] (well the original one is) [20:13:43] (03PS1) 10Andrew Bogott: role::labs::lvm::mnt: Allow mount_point to be set from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308820 (https://phabricator.wikimedia.org/T101447) [20:13:46] wmf.18 is currently running 4ae994a6adbc37d35aacf800700e09321bd93347 [20:14:55] (03PS2) 10Andrew Bogott: mediawiki_singlenode: Move $::labs_mediawiki_hostname to a param [puppet] - 10https://gerrit.wikimedia.org/r/308812 (https://phabricator.wikimedia.org/T101447) [20:15:34] (03CR) 10BBlack: [C: 032] ciphersuites: drop mid-level SHA-2 HMAC suites [puppet] - 10https://gerrit.wikimedia.org/r/308810 (owner: 10BBlack) [20:15:39] (03CR) 10Alex Monk: [C: 031] "we should still kill this with fire though" [puppet] - 10https://gerrit.wikimedia.org/r/308820 (https://phabricator.wikimedia.org/T101447) (owner: 10Andrew Bogott) [20:16:41] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 10.192.48.44 on port 6479 [20:17:57] ostriches: https://gerrit.wikimedia.org/r/308821 [20:18:40] (03PS1) 10Andrew Bogott: toollabs: Allow $::node_dedicated_tool to be set from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308823 (https://phabricator.wikimedia.org/T101447) [20:19:05] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5245202 keys - replication_delay is 0 [20:19:21] hoo: Well, obvious reason is obvious in retrospect.... [20:19:23] "Wikidata": "wmf/1.28.0-wmf.15", [20:19:34] oh, well [20:19:36] yeah [20:19:41] that needs to be manually bumped [20:19:47] but I didn't come to that in time today [20:19:57] Actually I thought aud.e would be deploying today [20:20:26] (03CR) 10Alex Monk: [C: 031] toollabs: Allow $::node_dedicated_tool to be set from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308823 (https://phabricator.wikimedia.org/T101447) (owner: 10Andrew Bogott) [20:20:28] So since that doesn't track master like the others, it means the submodules of core won't update automatically either. [20:20:35] Since we have custom gitmodules entries [20:21:14] ostriches: True [20:21:21] (03CR) 10Aaron Schulz: [C: 031] Tweak GlobalTitleFail logging, I'm going to start fixing this [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308808 (owner: 10Chad) [20:21:35] (03CR) 10Aaron Schulz: "Make sure it's not still spammy though with the sample rate" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308808 (owner: 10Chad) [20:22:08] (03CR) 10Chad: "It's pretty damn spammy." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308808 (owner: 10Chad) [20:22:09] ostriches: https://gerrit.wikimedia.org/r/308825 [20:22:50] hoo: Since the Wikidata repo is just the meta-repository that pulls things together for wmf deployment, is there a reason it can't track master? [20:23:23] ostriches: Yes, because master is updated with Wikibase master daiyl [20:23:49] (03PS1) 10Andrew Bogott: ldap::role::client::labs: Allow restricted_from and _to from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308826 (https://phabricator.wikimedia.org/T101447) [20:24:53] hoo: Hmm, if we swapped it for a reference that you just update when you want to update master, we should probably do something like CentralNotice does :) [20:25:10] A ref-name that doesn't mention a version # [20:25:32] (03CR) 10Alex Monk: [C: 031] ldap::role::client::labs: Allow restricted_from and _to from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308826 (https://phabricator.wikimedia.org/T101447) (owner: 10Andrew Bogott) [20:25:33] ostriches: Nah, we don't want to update all wmf versions at once usually [20:25:41] we should just use the normal deploy schedule [20:25:45] but *rage* [20:25:47] Or that :) [20:25:56] last one https://gerrit.wikimedia.org/r/308827 [20:26:00] for next week [20:27:14] !log demon@tin Started scap: scap 4 wikidata <3 [20:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:27:21] hoo: ^ [20:27:23] :D Thanks [20:27:51] yw [20:28:43] 06Operations, 10Traffic: OpenSSL 1.1 deployment for cache clusters - https://phabricator.wikimedia.org/T144523#2612589 (10BBlack) On the "potential post-1.1.0 issues" front so far, nothing too serious, but: * This double-free probably impacts our currently-under-testing nginx-1.11+openssl-1.1 build: https://g... [20:41:25] (03PS1) 10Rush: tools: exec-manage [puppet] - 10https://gerrit.wikimedia.org/r/308851 [20:41:59] (03PS2) 10Rush: tools: exec-manage option parsing and list option [puppet] - 10https://gerrit.wikimedia.org/r/308851 [20:42:22] (03CR) 10Andrew Bogott: [C: 032] mediawiki_singlenode: Move $::labs_mediawiki_hostname to a param [puppet] - 10https://gerrit.wikimedia.org/r/308812 (https://phabricator.wikimedia.org/T101447) (owner: 10Andrew Bogott) [20:42:27] (03PS3) 10Andrew Bogott: mediawiki_singlenode: Move $::labs_mediawiki_hostname to a param [puppet] - 10https://gerrit.wikimedia.org/r/308812 (https://phabricator.wikimedia.org/T101447) [20:53:08] !log demon@tin Finished scap: scap 4 wikidata <3 (duration: 25m 53s) [20:53:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:54:47] (03CR) 10Madhuvishy: [C: 031] "Cool!" [puppet] - 10https://gerrit.wikimedia.org/r/308851 (owner: 10Rush) [20:56:46] !log T133805: Disabling Puppet for GC experiment on restbase1013.eqiad.wmnet [20:56:47] T133805: Isolated testing of GC settings for aggressive Cassandra chunk_length_kb values - https://phabricator.wikimedia.org/T133805 [20:56:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:59:15] !log T133805: Restarting Cassandra to apply G1 region size of 16M on restbase1013-a.eqiad.wmnet [20:59:16] T133805: Isolated testing of GC settings for aggressive Cassandra chunk_length_kb values - https://phabricator.wikimedia.org/T133805 [20:59:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:59:44] (03PS2) 10Chad: WIP: Remove "p" symlink from WMF config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306471 [21:00:04] (03PS3) 10Chad: Remove "p" symlink from WMF config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306471 [21:01:55] (03CR) 10Aaron Schulz: [C: 031] Remove "p" symlink from WMF config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306471 (owner: 10Chad) [21:05:22] (03PS1) 10Alex Monk: openstack: Import nova_ldap designate plugin [puppet] - 10https://gerrit.wikimedia.org/r/308875 (https://phabricator.wikimedia.org/T144317) [21:05:24] (03PS1) 10Alex Monk: Follow-up I026e4f57: Fix style [puppet] - 10https://gerrit.wikimedia.org/r/308876 [21:06:33] (03CR) 10jenkins-bot: [V: 04-1] openstack: Import nova_ldap designate plugin [puppet] - 10https://gerrit.wikimedia.org/r/308875 (https://phabricator.wikimedia.org/T144317) (owner: 10Alex Monk) [21:09:03] (03CR) 10BBlack: [C: 031] Move yarn.w.o traffic to stat1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/308776 (https://phabricator.wikimedia.org/T116192) (owner: 10Elukey) [21:12:57] (03PS2) 10Alex Monk: openstack: Import nova_ldap designate plugin [puppet] - 10https://gerrit.wikimedia.org/r/308875 (https://phabricator.wikimedia.org/T144317) [21:12:59] (03PS2) 10Alex Monk: Follow-up I026e4f57: Fix style [puppet] - 10https://gerrit.wikimedia.org/r/308876 [21:16:07] 06Operations, 10Cassandra, 06Services: restbase2004-b.codfw.wmnet data corruption - https://phabricator.wikimedia.org/T144826#2612731 (10Eevans) Current status: An online scrub has been running since 17:10UTC. For purposes of this scrub, compaction throughput has been set to 0 (disabled), and the number of... [21:16:41] (03PS2) 10Andrew Bogott: role::labs::lvm::mnt: Allow mount_point to be set from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308820 (https://phabricator.wikimedia.org/T101447) [21:17:56] (03CR) 10Andrew Bogott: [C: 032] role::labs::lvm::mnt: Allow mount_point to be set from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308820 (https://phabricator.wikimedia.org/T101447) (owner: 10Andrew Bogott) [21:30:04] MaxSem: Respected human, time to deploy Kartographer window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160906T2130). Please do the needful. [21:34:33] yurik, James_F, edsanders: https://gerrit.wikimedia.org/r/#/c/308797/ pulled on mw1099 [21:34:55] (03PS2) 10Andrew Bogott: toollabs: Allow $::node_dedicated_tool to be set from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308823 (https://phabricator.wikimedia.org/T101447) [21:35:27] MaxSem: Do you not want it in wmf.18 too? [21:36:12] MaxSem: (Are you deploying with the train or rushing into wmf.17?) [21:36:18] oops, I thought we managed to do it before the cutoff. cherrypicking [21:36:28] (03CR) 10Ppchelko: [C: 031] Change Prop: Send the if-unmodified-since header on transclusion updates [puppet] - 10https://gerrit.wikimedia.org/r/308773 (owner: 10Mobrovac) [21:36:37] James_F, i think its already in 18 [21:36:37] i might be wrong, need to double check [21:36:44] nope [21:36:55] yurik: It isn't. [21:37:00] (03CR) 10Andrew Bogott: [C: 032] toollabs: Allow $::node_dedicated_tool to be set from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308823 (https://phabricator.wikimedia.org/T101447) (owner: 10Andrew Bogott) [21:37:02] Hence why I asked. :-) [21:37:17] Cherry-picked to wmf.18 as https://gerrit.wikimedia.org/r/#/c/308880/ [21:37:35] MaxSem, ^ [21:37:59] ah, never mind, you are on top of it... need faster internet without lag :) [21:38:17] yurik: Move to SF! ;-) [21:38:44] James_F, actually internet in ru was much faster than NY ... and SF, especially in the office :-P [21:39:36] yurik: And in .uk, but oh well. [21:39:52] MaxSem, maplink seems to work on 1099 -- https://en.wikivoyage.org/wiki/User:Yurik/Sandbox?veaction=edit [21:40:11] (03PS3) 10Rush: tools: exec-manage option parsing and list option [puppet] - 10https://gerrit.wikimedia.org/r/308851 [21:40:59] (03CR) 10Rush: [C: 032 V: 032] tools: exec-manage option parsing and list option [puppet] - 10https://gerrit.wikimedia.org/r/308851 (owner: 10Rush) [21:47:32] !log maxsem@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/308300/ (duration: 00m 49s) [21:47:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:47:53] (03PS1) 10Hashar: wmflib: os_version now fail when lsb vars are missing [puppet] - 10https://gerrit.wikimedia.org/r/308882 [21:48:21] (03PS2) 10Jdlrobson: Do not show Wikidata descriptions on meta or mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307968 [21:48:23] (03CR) 10Andrew Bogott: [C: 032] ldap::role::client::labs: Allow restricted_from and _to from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308826 (https://phabricator.wikimedia.org/T101447) (owner: 10Andrew Bogott) [21:48:26] (03PS2) 10Andrew Bogott: ldap::role::client::labs: Allow restricted_from and _to from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308826 (https://phabricator.wikimedia.org/T101447) [21:48:42] (03PS2) 10Jdlrobson: Enable on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) [21:48:53] (03CR) 10jenkins-bot: [V: 04-1] Do not show Wikidata descriptions on meta or mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307968 (owner: 10Jdlrobson) [21:48:55] !log maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/308300/ (duration: 00m 48s) [21:48:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:48:59] (03CR) 10Hashar: "I keep hitting that one when using rspec-puppet. The fail() hints at setting lsb* facts." [puppet] - 10https://gerrit.wikimedia.org/r/308882 (owner: 10Hashar) [21:49:08] yurik, ^^^ [21:49:35] (03CR) 10jenkins-bot: [V: 04-1] Enable on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) (owner: 10Jdlrobson) [21:49:45] MaxSem, works, thx, did you +2 18? [21:49:59] yup [21:50:52] (03PS3) 10Jdlrobson: Do not show Wikidata descriptions on meta or mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307968 [21:52:12] (03PS1) 10Yuvipanda: tools: Provision /srv volume for prometheus [puppet] - 10https://gerrit.wikimedia.org/r/308883 [21:52:27] (03CR) 10jenkins-bot: [V: 04-1] wmflib: os_version now fail when lsb vars are missing [puppet] - 10https://gerrit.wikimedia.org/r/308882 (owner: 10Hashar) [21:52:31] (03PS2) 10Yuvipanda: tools: Provision /srv volume for prometheus [puppet] - 10https://gerrit.wikimedia.org/r/308883 [21:52:38] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Provision /srv volume for prometheus [puppet] - 10https://gerrit.wikimedia.org/r/308883 (owner: 10Yuvipanda) [21:53:24] (03PS3) 10Andrew Bogott: ldap::role::client::labs: Allow restricted_from and _to from hiera [puppet] - 10https://gerrit.wikimedia.org/r/308826 (https://phabricator.wikimedia.org/T101447) [21:53:27] (03PS2) 10Hashar: wmflib: os_version now fail when lsb vars are missing [puppet] - 10https://gerrit.wikimedia.org/r/308882 [21:53:47] (03Draft2) 10Paladox: Fix broken refs/meta/config diffusion links in access section [puppet] - 10https://gerrit.wikimedia.org/r/308885 (https://phabricator.wikimedia.org/T137354) [21:53:51] (03Draft1) 10Paladox: Fix broken refs/meta/config diffusion links in access section [puppet] - 10https://gerrit.wikimedia.org/r/308885 (https://phabricator.wikimedia.org/T137354) [21:57:59] (03PS3) 10Paladox: Fix broken refs/meta/config diffusion links in access section [puppet] - 10https://gerrit.wikimedia.org/r/308885 (https://phabricator.wikimedia.org/T137354) [21:58:05] (03PS4) 10Paladox: Fix broken refs/meta/config diffusion links in access section [puppet] - 10https://gerrit.wikimedia.org/r/308885 (https://phabricator.wikimedia.org/T137354) [21:58:49] (03PS1) 10Rush: labstore: nfs-exports/nfs-exports-daemons settle on nfs-exportd [puppet] - 10https://gerrit.wikimedia.org/r/308886 [22:02:13] yurik, now https://gerrit.wikimedia.org/r/#/c/308298/ ? [22:02:31] MaxSem, sounds good [22:02:43] MaxSem, 17 only, right? [22:03:09] aha [22:04:55] (03CR) 10Madhuvishy: [C: 031] labstore: nfs-exports/nfs-exports-daemons settle on nfs-exportd [puppet] - 10https://gerrit.wikimedia.org/r/308886 (owner: 10Rush) [22:05:50] yurik, pulled on mw1099 [22:06:32] MaxSem, i don't think we have any wikis with maplink only [22:06:43] testing general funcitonaliyt... [22:06:47] 06Operations, 06Labs, 10wikitech.wikimedia.org: Can't login wikitech - https://phabricator.wikimedia.org/T144805#2612937 (10Peachey88) @Shizhao When was the last time you logged into wikitech successfully? There was a mistake at one stage that resulted in a number of people accidently having 2FA activated,... [22:06:56] general isn't broken... [22:07:33] MaxSem, seems to be ok [22:07:46] ok, syncing [22:08:35] (03PS1) 10Hashar: (WIP) Puppet compile an host via rspec (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/308889 [22:09:00] !log maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/308887/ (duration: 00m 48s) [22:09:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:10:05] (03CR) 10jenkins-bot: [V: 04-1] (WIP) Puppet compile an host via rspec (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/308889 (owner: 10Hashar) [22:10:55] (03PS1) 10MaxSem: Deploy Kartographer everywhere public [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308890 (https://phabricator.wikimedia.org/T144062) [22:11:01] yurik, ^ [22:11:19] (03PS2) 10MaxSem: Deploy Kartographer everywhere public [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308890 (https://phabricator.wikimedia.org/T144062) [22:11:21] (03CR) 10Yurik: [C: 031] Deploy Kartographer everywhere public [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308890 (https://phabricator.wikimedia.org/T144062) (owner: 10MaxSem) [22:13:03] (03PS3) 10MaxSem: Deploy Kartographer everywhere public [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308890 (https://phabricator.wikimedia.org/T144062) [22:13:19] (03CR) 10MaxSem: [C: 032] Deploy Kartographer everywhere public [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308890 (https://phabricator.wikimedia.org/T144062) (owner: 10MaxSem) [22:13:52] (03Merged) 10jenkins-bot: Deploy Kartographer everywhere public [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308890 (https://phabricator.wikimedia.org/T144062) (owner: 10MaxSem) [22:14:24] (03PS3) 10Jdlrobson: Enable descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) [22:14:29] yurik, pulled on mw1099 [22:14:35] * yurik looks [22:15:12] (03CR) 10jenkins-bot: [V: 04-1] Enable descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) (owner: 10Jdlrobson) [22:15:56] MaxSem, weird, is not showing up in VE [22:16:07] trying it on enwiki [22:16:31] edsanders, could you take a look on wm1099 pls? [22:16:35] tried, same shit [22:16:38] :eek: [22:17:16] MaxSem, in the worst case, VE won't break it, but will continue showing it as wiki markup. Is this a blocker? [22:17:49] i wonder if restbase or parsoid needs to be restarted? [22:18:10] то понос то золотуха [22:18:46] is Julien done for the day? [22:18:57] MaxSem, can it be that parsoid continues to use other settings despite what you have on wm1099? [22:19:20] parsoid cares not of MW settings [22:19:27] i think its not in julien's domain... more like gwicke ? [22:19:29] parsoid caches mw settings for extensions etc [22:20:09] gwicke, we are deploying a new version, with enwiki support for a new wiki tag -- and VE is not showing it correctly [22:20:34] it works fine on enwv [22:20:48] is it because we only have it enabled on mw1099 ? [22:20:50] yeah, it might need a kick to pick up the new extension tag [22:21:09] MaxSem, i think we should go ahead and sync, and then kick gwicke :) [22:21:19] and it'll only pick that up if the API server it contacts has the extension enabled [22:21:44] the parsoid folks should be able to help yoyu [22:21:57] yep, that's what i thought - in other words we cannot really test deployments via single server 1099 [22:22:12] MaxSem, i say go ahead and finish the sync [22:22:19] what do yo uthink ? [22:23:00] VisualEditor failed to load: Error: Module ext.kartographer.style has failed dependencies [22:23:36] MaxSem, is that still on 1099? [22:23:51] yep [22:24:37] MaxSem, i don't see it - how did you get that msg? [22:24:46] hmm ah: we his a debug server, but VE can't [22:24:54] s/his/hit/ [22:25:02] s/VE/Parsoid/ [22:25:03] hehe [22:25:15] let's sync and see how it works [22:26:40] !log maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/308890/3 (duration: 00m 47s) [22:26:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:26:48] oh noes [22:28:13] MaxSem, i guess now is the time to ping parsoid ppl :) [22:28:21] gwicke, Pchelolo ? [22:28:21] nope [22:28:36] sup? [22:28:43] everything seems good to me [22:28:47] yurik: #mediawiki-parsoid [22:29:16] MaxSem, are you seeing any problems? [22:29:40] GET https://en.wikipedia.org/api/rest_v1/page/graph/png/User%3AYurik%2Fsandbox/0/815af543571e17c18666bb53912c282f6aa20e22.png 400 () [22:30:26] first of all, need to wait for modules to update [22:30:43] "TypeError: this.options.quickList is null" everywhere on wmf18 wikis :/ [22:30:48] MaxSem, this was unrelated [22:30:53] my graph is broken on that page [22:31:22] bleh href="/wiki/Special:Map///" [22:31:28] fucking PHP [22:31:43] (03PS1) 10EBernhardson: Cirrus: Support document versioning [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308896 (https://phabricator.wikimedia.org/T144039) [22:32:31] MaxSem, nah, this is fine :) i don't have lat/long/zoom for the maplink, that's why it shows that way [22:32:31] also, we shouldn'tt allow empty maplinks without parameters specified [22:33:15] why not? if it has geojson, it should position itself correctly [22:34:27] (03CR) 10EBernhardson: "This makes an assumption that the new version of the plugin will be 2.3.4.1. The previous version was 2.3.4, i made a guess we will attach" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308896 (https://phabricator.wikimedia.org/T144039) (owner: 10EBernhardson) [22:35:53] well, I'm talking about when it doesn't [22:50:48] (03PS4) 10Jdlrobson: Enable descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) [22:52:09] !log restarting Parsoid [22:52:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:53:15] * hoo rages about code on github [22:53:31] Also not being able to push to repos directly to make PRs is painful [22:54:19] yurik: where? [22:55:07] edsanders, never mind, its all good, your VE link patch is deployed [22:56:40] yurik: is the a phab task for [22:58:12] !log Parsoid restarted to pick up new wiki config after deploy (T144062) [22:58:13] T144062: Epic: Enable on Wikipedias - https://phabricator.wikimedia.org/T144062 [22:58:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:58:48] thanks arlolra ! [22:59:13] cc MaxSem [22:59:50] np, you probably need to ask RB to reparse that page you were testing [23:00:04] RoanKattouw, ostriches, MaxSem, and Dereckson: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160906T2300). Please do the needful. [23:00:04] tgr and Jdlrobson: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [23:00:07] Hello. [23:00:13] I can SWAT this evening. [23:00:15] mobrovac knows the incantation [23:00:26] I'll also have later a small patch to add (config change for skwiki) [23:00:26] o/ [23:00:28] ostriches: greg-g: Who can merge in https://github.com/wikimedia/jquery.uls/pull/238 ? [23:00:37] UNB bugs in code on github make me sad [23:01:00] Just cherry-pick it and commit to gerrit? [23:01:16] \o [23:01:21] Oh, unless it's developed there? [23:01:23] I dunno [23:01:29] ostriches: It's developed there [23:02:11] ask in #mediawiki-i18n? [23:04:23] tgr: Good hint, thanks [23:05:38] I mean worst case I or another admin could merge [23:05:54] (03PS6) 10Dereckson: Do not use $wgExtensionFunctions to set globals [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307893 (https://phabricator.wikimedia.org/T143055) (owner: 10Gergő Tisza) [23:06:24] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307893 (https://phabricator.wikimedia.org/T143055) (owner: 10Gergő Tisza) [23:06:30] (03CR) 10Gergő Tisza: "Looks like the content got lost in PS4?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) (owner: 10Jdlrobson) [23:06:50] (03Merged) 10jenkins-bot: Do not use $wgExtensionFunctions to set globals [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307893 (https://phabricator.wikimedia.org/T143055) (owner: 10Gergő Tisza) [23:06:57] tgr looks lke it! [23:07:30] tgr: live on mw1099 [23:07:34] (03PS5) 10Jdlrobson: Enable descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) [23:07:57] (03CR) 10Bmansurov: [C: 04-1] "to get Jon's attention" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) (owner: 10Jdlrobson) [23:08:17] Dereckson, i need to sync kartotherian when you are done [23:09:20] Dereckson: it's an Echo-related patch, not sure it can be tested on mw1099 [23:11:09] Dereckson: at the very least nothing seems broken, so IMO it should be fine to deploy and test afterwards [23:12:52] yurik: okay I'll ping you [23:13:01] tgr: I confirm, nothing in logs worrying [23:13:08] (03PS1) 10Dereckson: Expand array to allow future changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308900 [23:13:08] let's send that in prod [23:13:10] (03PS1) 10Dereckson: Restore Redaktor namespace on sk.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308901 (https://phabricator.wikimedia.org/T143472) [23:14:31] (03CR) 10Bmansurov: Enable descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) (owner: 10Jdlrobson) [23:14:48] !log dereckson@tin Synchronized wmf-config/CommonSettings.php: Do not use $wgExtensionFunctions to set globals (T143055) (duration: 00m 47s) [23:14:49] T143055: Echo notificationCategories usergroups parameter disappears - https://phabricator.wikimedia.org/T143055 [23:14:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:15:04] jdlrobson: https://gerrit.wikimedia.org/r/#/c/307969/ needs a manual rebase [23:15:45] (03PS4) 10Dereckson: Do not show Wikidata descriptions on meta or mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307968 (owner: 10Jdlrobson) [23:15:50] Dereckson: really? why? [23:15:55] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307968 (owner: 10Jdlrobson) [23:16:03] (03PS6) 10Jdlrobson: Enable descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) [23:16:12] ^ Dereckson rebased fine in gerrit [23:16:21] (03Merged) 10jenkins-bot: Do not show Wikidata descriptions on meta or mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307968 (owner: 10Jdlrobson) [23:16:25] oh cool, the button was broken on my side [23:16:46] (and I infered button disabled = merge conflict, please rebase manually yourself) [23:17:12] Dereckson: thanks, verified [23:17:26] you're welcome, thanks for testing [23:17:34] jdlrobson: meta/mw part live on mw1099 [23:18:16] Dereckson: looks good! [23:18:34] (03PS7) 10Jdlrobson: Enable Wikidata descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) [23:18:43] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) (owner: 10Jdlrobson) [23:19:10] (03Merged) 10jenkins-bot: Enable Wikidata descriptions on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307969 (https://phabricator.wikimedia.org/T143345) (owner: 10Jdlrobson) [23:19:50] jdlrobson: part 2 live too [23:20:08] er [23:20:14] we lost meta [23:20:17] $ cat dblists/nowikidatadescriptiontaglines.dblist [23:20:19] mediawikiwiki [23:20:32] works! good stuff Dereckson :) [23:20:46] (03PS3) 10Dzahn: lvs: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308341 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:21:07] ah no, we don't have lost meta [23:21:14] metawiki[dereckson@tin mediawiki-staging (master=)]$ [23:21:21] It's \n story. [23:23:30] !log dereckson@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: Update wikis where Wikidata descriptions is shown or not ([[Gerrit:307968]] and [[Gerrit:307969]], T143345) (duration: 00m 46s) [23:23:31] T143345: Deploy Wikidata descriptions to mobile web stable channel Wikipedias 2nd half - https://phabricator.wikimedia.org/T143345 [23:23:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:23:37] (03PS2) 10Dereckson: Expand array to allow future changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308900 [23:24:02] (03CR) 10Dereckson: [C: 032] "SWAT (no-op)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308900 (owner: 10Dereckson) [23:24:14] (03CR) 10Dzahn: [C: 032] lvs: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308341 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:24:15] thanks Dereckson [23:24:29] (03Merged) 10jenkins-bot: Expand array to allow future changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308900 (owner: 10Dereckson) [23:24:44] jdlrobson: you're welcome [23:25:03] 308900 live on mw1099 [23:25:14] (03PS4) 10Paladox: Add support for searching gerrit using bug:T1 [puppet] - 10https://gerrit.wikimedia.org/r/308753 (https://phabricator.wikimedia.org/T85002) [23:25:18] (03PS5) 10Paladox: Fix broken refs/meta/config diffusion links in access section [puppet] - 10https://gerrit.wikimedia.org/r/308885 (https://phabricator.wikimedia.org/T137354) [23:26:11] thansk ^^ [23:26:14] thanks [23:27:19] tested, works fine [23:27:30] (03PS2) 10Dereckson: Restore Redaktor namespace on sk.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308901 (https://phabricator.wikimedia.org/T143472) [23:29:02] (03CR) 10Dzahn: "i tried init.pp before and it did not work, that's basically why i havent moved all of manifests/role/ already" [puppet] - 10https://gerrit.wikimedia.org/r/308311 (owner: 10Hashar) [23:29:05] 06Operations: Multiple servers in codfw fail to respond to IPMI commands during reimaging - https://phabricator.wikimedia.org/T142726#2613245 (10Papaul) @MoritzMuehlenhoff second batch complete. [23:29:49] (03PS3) 10Dereckson: Restore Redaktor namespace on sk.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308901 (https://phabricator.wikimedia.org/T143472) [23:30:02] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308901 (https://phabricator.wikimedia.org/T143472) (owner: 10Dereckson) [23:30:32] (03Merged) 10jenkins-bot: Restore Redaktor namespace on sk.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308901 (https://phabricator.wikimedia.org/T143472) (owner: 10Dereckson) [23:30:33] (03CR) 10Paladox: ircyall: move role to module/role (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308311 (owner: 10Hashar) [23:30:53] 308901 live on mw1099 [23:31:35] strange [23:31:52] works but it doesn't take sk default localised namespace as alias [23:32:58] And I'm not sure if that's expected or not. [23:33:41] So to be safe, I'm going to prepare a fix to add the default sk localisation as an alias. [23:36:34] (03PS3) 10Dzahn: statsd_proxy: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308356 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:39:06] (03CR) 10Dzahn: [C: 032] statsd_proxy: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308356 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:39:16] ^^ thanks :) [23:41:25] 06Operations, 10MediaWiki-ResourceLoader, 06Performance-Team, 10Traffic: Expires header for load.php should be relative to request time instead of cache time - https://phabricator.wikimedia.org/T105657#2613260 (10Krinkle) I synced with @BBlack on IRC just now and resolved the confusion @Catrope and I were... [23:41:26] (03PS1) 10Dereckson: Set User namespace aliases on sk.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308902 (https://phabricator.wikimedia.org/T143472) [23:41:45] 06Operations, 10MediaWiki-ResourceLoader, 06Performance-Team, 10Traffic: Expires header for load.php should be relative to request time instead of cache time - https://phabricator.wikimedia.org/T105657#1448575 (10Krinkle) a:05Krinkle>03None [23:41:46] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308902 (https://phabricator.wikimedia.org/T143472) (owner: 10Dereckson) [23:42:13] (03PS1) 10Alex Monk: Kill references to $::instancename [puppet] - 10https://gerrit.wikimedia.org/r/308903 (https://phabricator.wikimedia.org/T101447) [23:42:15] (03Merged) 10jenkins-bot: Set User namespace aliases on sk.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308902 (https://phabricator.wikimedia.org/T143472) (owner: 10Dereckson) [23:42:32] 308902 live on mw1099 [23:42:38] (03CR) 10Paladox: "@Giuseppe Lavagetto or @Alexandros Kosiaris I'm not sure what to do now?" [puppet] - 10https://gerrit.wikimedia.org/r/308498 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:43:39] Okay, this time, all works as expected. [23:45:28] RoanKattouw: you remember some months ago the day you asked me why I added to IS namespaces localised in core? I've now the root cause: that's because if the wiki customizes a namespace, it doesn't read default MediaWiki core localisation. [23:46:07] (03PS1) 10Legoktm: Update doc comment on mediawiki::packages::php5 [puppet] - 10https://gerrit.wikimedia.org/r/308904 [23:46:11] (03CR) 10Dzahn: [C: 031] role: move mediawiki::install to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308322 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [23:46:16] Hmm, wait, what [23:46:25] For custom namespaces (100 and above) I suppose that's right [23:46:44] But for built-in namespaces (15 and below) I thought it would read the default localizatoin [23:46:45] for NS_USER, I don't like the fact it ignores the core value :/ [23:46:51] yes I agree [23:47:02] !log dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix User namespace localisation for sk.wikipedia (T143472) (duration: 00m 47s) [23:47:03] T143472: Gender dependant namespaces "User" and "User talk" should be updated on Slovak Wikipedia - https://phabricator.wikimedia.org/T143472 [23:47:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:47:37] and surprise, if you say NS_USER = "Quux" in wgExtraNamespaces, you lost User: [23:48:57] (03PS3) 10Dzahn: lvs: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308342 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:49:58] yurik: SWAT is done [23:50:11] Dereckson, thx! [23:50:16] syncing... [23:50:48] (03CR) 10Dzahn: [C: 032] lvs: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308342 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:51:12] thanks ^^ [23:52:53] !log Ran namespaceDupes maintenance script on skwiki (0 pages to fix, 1 links to fix, fixed) (T143472) [23:52:54] T143472: Gender dependant namespaces "User" and "User talk" should be updated on Slovak Wikipedia - https://phabricator.wikimedia.org/T143472 [23:52:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:54:04] !log deployed kartotherian - adjusting geoshapes arg, and bumping deps - https://gerrit.wikimedia.org/r/#/c/308899/ [23:54:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:54:56] (03PS3) 10Dzahn: pybal: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308349 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:55:03] (03CR) 10Dzahn: [C: 032] pybal: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308349 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:58:12] (03CR) 10Dzahn: [C: 04-2] "per "@paladox: Please open a task to discuss this further" and "uploaded 5 months ago"" [puppet] - 10https://gerrit.wikimedia.org/r/281071 (https://phabricator.wikimedia.org/T131622) (owner: 10Paladox)