[05:25:06] PROBLEM - pdfrender on scb1001 is CRITICAL: connect to address 10.64.0.16 and port 5252: Connection refused [07:47:18] 10Operations, 10Traffic, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4216086 (10Krenair) a:03Krenair [08:24:58] 10Operations, 10Wikispeech, 10Wikispeech-WMSE: TTS server deployment strategy - https://phabricator.wikimedia.org/T193072#4216119 (10Theklan) 05Open>03stalled This is a blocker for T183780 [08:27:42] (03PS2) 10Jforrester: security: Remove dangerous unused 'botadmin' group at mlwik{tionary|isource} [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433136 (https://phabricator.wikimedia.org/T152296) (owner: 10MarcoAurelio) [08:27:46] (03CR) 10Jforrester: [C: 031] security: Remove dangerous unused 'botadmin' group at mlwik{tionary|isource} [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433136 (https://phabricator.wikimedia.org/T152296) (owner: 10MarcoAurelio) [08:30:56] RECOVERY - pdfrender on scb1001 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.003 second response time [08:31:04] !log restarted pdfrender on scb1001 [08:31:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:34:03] (03PS1) 10Volans: wmf-auto-reimage: validate certificate fingerprint [puppet] - 10https://gerrit.wikimedia.org/r/433928 [09:03:36] 10Operations, 10Availability (MediaWiki-MultiDC), 10Patch-For-Review, 10Performance-Team (Radar), 10User-Joe: mcrouter production architecture - https://phabricator.wikimedia.org/T192771#4216271 (10Joe) [09:04:08] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Please see https://gerrit.wikimedia.org/r/#/c/431736/ and followups; we should coordinate a bit on this." [puppet] - 10https://gerrit.wikimedia.org/r/433913 (https://phabricator.wikimedia.org/T194225) (owner: 10Aaron Schulz) [09:56:08] (03PS2) 10Volans: wmf-auto-reimage: validate certificate fingerprint [puppet] - 10https://gerrit.wikimedia.org/r/433928 [10:10:47] (03PS1) 10Ayounsi: UDP2LOG: Remove sysvinit config file [puppet] - 10https://gerrit.wikimedia.org/r/433946 [10:15:43] (03CR) 10Ayounsi: "https://puppet-compiler.wmflabs.org/compiler02/11248/" [puppet] - 10https://gerrit.wikimedia.org/r/433946 (owner: 10Ayounsi) [10:18:18] (03PS2) 10Ayounsi: UDP2LOG: Remove sysvinit config file [puppet] - 10https://gerrit.wikimedia.org/r/433946 [10:20:00] (03CR) 10Alexandros Kosiaris: [C: 031] UDP2LOG: Remove sysvinit config file [puppet] - 10https://gerrit.wikimedia.org/r/433946 (owner: 10Ayounsi) [10:20:29] (03CR) 10Ayounsi: [C: 032] "https://puppet-compiler.wmflabs.org/compiler02/11249/" [puppet] - 10https://gerrit.wikimedia.org/r/433946 (owner: 10Ayounsi) [10:27:54] (03CR) 10Vgutierrez: [C: 031] "nitpick comment inline, LGTM :D" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/433928 (owner: 10Volans) [10:29:42] (03CR) 10Volans: "Thanks for the review, addressed comments." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/433928 (owner: 10Volans) [10:34:55] (03PS3) 10Volans: wmf-auto-reimage: validate certificate fingerprint [puppet] - 10https://gerrit.wikimedia.org/r/433928 [12:24:25] (03PS4) 10Matěj Suchánek: Update Wikidata property blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/430045 [13:33:37] !log mholloway-shell@tin Started deploy [kartotherian/deploy@58d2b0a]: Update maps/kartotherian/package to 7520fa5 [13:33:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:04] !log mholloway-shell@tin Finished deploy [kartotherian/deploy@58d2b0a]: Update maps/kartotherian/package to 7520fa5 (duration: 02m 28s) [13:36:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:38:10] 10Operations, 10Traffic, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4216999 (10Krenair) Some of my work on this is being blocked by T195059 [13:39:52] (03PS1) 10Urbanecm: Revert "Temp rate limit for arwiki due to mass vandalism" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433987 (https://phabricator.wikimedia.org/T192668) [13:43:50] (03PS1) 10Urbanecm: Raise the rate limits for Commons to 200 edits/minute for all logged-in users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433988 (https://phabricator.wikimedia.org/T194864) [13:45:48] (03CR) 10Kaldari: [C: 031] Raise the rate limits for Commons to 200 edits/minute for all logged-in users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433988 (https://phabricator.wikimedia.org/T194864) (owner: 10Urbanecm) [13:46:31] (03CR) 10Reedy: "Commit summary doesn't match title of the task... The task says "Raise the rate limit for autopatrollers on Commons"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433988 (https://phabricator.wikimedia.org/T194864) (owner: 10Urbanecm) [13:49:26] 10Operations, 10Traffic, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217093 (10tstarling) I explained ChronologyProtector to @Joe and @BBlack just now. They seemed happy with the idea of not sending a useDC cookie for no... [13:50:50] 10Operations, 10Cloud-VPS: Cannot add or update records under DNS zones in Horizon - https://phabricator.wikimedia.org/T195059#4217099 (10Krenair) [14:09:32] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] "Thanks for the update! I manually confirmed all the additions in this patch. To the people with merge-rights here, please click that +2 bu" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/430045 (owner: 10Matěj Suchánek) [14:11:24] (03CR) 10Ladsgroup: "It needs deployment. Schedule for SWAT in Monday" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/430045 (owner: 10Matěj Suchánek) [14:19:36] (03PS2) 10Urbanecm: Raise the rate limits for Commons to 200 edits/minute for all logged-in users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433988 (https://phabricator.wikimedia.org/T194864) [14:21:28] (03PS1) 10Arturo Borrero Gonzalez: reprepro: fetch mono suite from upstream apt repo [puppet] - 10https://gerrit.wikimedia.org/r/433996 (https://phabricator.wikimedia.org/T194665) [14:22:12] (03CR) 10jerkins-bot: [V: 04-1] reprepro: fetch mono suite from upstream apt repo [puppet] - 10https://gerrit.wikimedia.org/r/433996 (https://phabricator.wikimedia.org/T194665) (owner: 10Arturo Borrero Gonzalez) [14:24:54] (03PS2) 10Arturo Borrero Gonzalez: reprepro: fetch mono suite from upstream apt repo [puppet] - 10https://gerrit.wikimedia.org/r/433996 (https://phabricator.wikimedia.org/T194665) [14:36:29] (03PS3) 10Urbanecm: Raise the rate limits for Commons to higher values than global 90 edits/minute [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433988 (https://phabricator.wikimedia.org/T194864) [14:41:23] 10Operations, 10Cloud-VPS: Cannot add or update records under DNS zones in Horizon - https://phabricator.wikimedia.org/T195059#4217221 (10Krenair) p:05High>03Triage [14:43:34] bblack, https://krenair.hopto.org is running on a labs machine [14:44:14] there's a central LE service on there with some scripts to pull down the public and private parts to a place nginx will read them from [14:44:31] the central LE service uses acme_tiny to request the cert from LE [14:44:37] (03CR) 10Urbanecm: "@Reedy: It doesn't, but it should match what the patch would do for real (and this is something it matches). Aslo, please note that the ta" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/433988 (https://phabricator.wikimedia.org/T194864) (owner: 10Urbanecm) [14:46:20] 10Operations, 10Traffic, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4217229 (10Krenair) https://krenair.hopto.org is running on a labs machine There's a central LE service on there which uses acme_tiny to request the cert from L... [14:53:25] (03CR) 10Chad: [V: 032 C: 032] Initial stable-2.15 fork for wikimedia [software/gerrit/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/432180 (owner: 10Chad) [15:00:40] no_justification :) [15:15:29] 10Operations, 10Traffic, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4217296 (10BBlack) Some after-thoughts on design issues and such (I haven't looked at any code!): * We should look hard for a good abstract ACME library that a... [15:19:47] 10Operations, 10Traffic, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4217300 (10BBlack) Also: * We should assume by default we want all certificates to be dual-issued as ECDSA+RSA variants and served to clients in both forms (I... [15:23:44] 10Operations, 10Traffic: Setup a new PKI software as an alternative to the puppet CA for managing services certificates - https://phabricator.wikimedia.org/T194031#4186323 (10BBlack) @Joe - So we're looking at doing something just for the LetsEncrypt (ACME) use-case over in T194962. The idea is this will mana... [15:25:40] 10Operations, 10Traffic, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4214163 (10BBlack) Also: naming bikshedding stuff: we should name/implement this as a generic ACME tool rather than LE-specific, and just make LE be the default... [15:29:17] 10Operations, 10Traffic, 10codfw-rollout: Enable VCL applayer datacenter-switch via confd - https://phabricator.wikimedia.org/T127485#4217331 (10BBlack) 05Open>03declined At this point, we're pushing off commit-free DC switching to post-ATS (sometime during the latter part of next FY, probably). [15:38:54] 10Operations, 10Traffic, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217378 (10BBlack) Right. Just to re-state for clarity, the sort of logic we should be implementing in VCL (in the cache layers) will look like this ps... [15:39:10] 10Operations, 10Traffic, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217380 (10BBlack) [15:45:11] 10Operations, 10Traffic, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217391 (10Joe) About ChronologyProtector: - If ChronologyProtector kicks in, it should send back a specific header to v... [15:57:15] 10Operations, 10Traffic, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217406 (10BBlack) Right, I forgot, that was discussed as an optimization (vs having ChronologyProtector just timeout -> f... [16:02:02] (03PS1) 10Alexandros Kosiaris: Stop gitignoring /dist [software/debmonitor] - 10https://gerrit.wikimedia.org/r/434019 [16:02:04] (03PS1) 10Alexandros Kosiaris: Add a blubber.yaml file [software/debmonitor] - 10https://gerrit.wikimedia.org/r/434020 [16:02:06] (03PS1) 10Alexandros Kosiaris: Add a basic requirements.txt file for the pipeline [software/debmonitor] - 10https://gerrit.wikimedia.org/r/434021 [16:20:42] 10Operations, 10Traffic, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217453 (10tstarling) Currently, ChronologyProtector times out after 10 seconds, this is configurable. Timeout causes "lag... [16:30:55] (03CR) 10Alexandros Kosiaris: [C: 031] ganeti: add interactive script to create VMs [puppet] - 10https://gerrit.wikimedia.org/r/433296 (owner: 10Dzahn) [16:52:00] (03PS1) 10Volans: facter: refactor the net_driver fact [puppet] - 10https://gerrit.wikimedia.org/r/434032 [16:53:00] (03CR) 10jerkins-bot: [V: 04-1] facter: refactor the net_driver fact [puppet] - 10https://gerrit.wikimedia.org/r/434032 (owner: 10Volans) [16:56:31] (03PS2) 10Volans: facter: refactor the net_driver fact [puppet] - 10https://gerrit.wikimedia.org/r/434032 [16:57:31] (03CR) 10jerkins-bot: [V: 04-1] facter: refactor the net_driver fact [puppet] - 10https://gerrit.wikimedia.org/r/434032 (owner: 10Volans) [16:59:18] (03PS3) 10Volans: facter: refactor the net_driver fact [puppet] - 10https://gerrit.wikimedia.org/r/434032 [17:13:51] 10Operations, 10Move-Files-To-Commons, 10TCB-Team, 10Wikimedia-Extension-setup, and 2 others: Deploying FileExporter and FileImporter - https://phabricator.wikimedia.org/T190716#4217597 (10Imarlier) @Joe @fgiunchedi Would probably be a good idea if one/both of you took a look at this, just to be sure that... [18:34:00] krenair@deployment-secureredirproto:~$ ls -lh /etc/certcentral/acct.key [18:34:00] -r-x------ 1 www-data www-data 0 May 19 18:28 /etc/certcentral/acct.key [18:34:00] krenair@deployment-secureredirproto:~$ sudo -u www-data openssl genrsa -out /etc/certcentral/acct.key 2048 [18:34:00] genrsa: Can't open "/etc/certcentral/acct.key" for writing, Permission denied [18:36:25] oops [18:36:31] just realised what I did [19:07:25] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] Stop gitignoring /dist [software/debmonitor] - 10https://gerrit.wikimedia.org/r/434019 (owner: 10Alexandros Kosiaris) [19:32:25] (03PS1) 10Gilles: Serve WebP variants for the hottest thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/434055 (https://phabricator.wikimedia.org/T27611) [19:33:01] (03CR) 10jerkins-bot: [V: 04-1] Serve WebP variants for the hottest thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/434055 (https://phabricator.wikimedia.org/T27611) (owner: 10Gilles) [19:34:15] (03PS2) 10Gilles: Serve WebP variants for the hottest thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/434055 (https://phabricator.wikimedia.org/T27611) [21:55:09] (03PS3) 10Aaron Schulz: mcrouter: add support for listening on the ssl port [puppet] - 10https://gerrit.wikimedia.org/r/431736 (https://phabricator.wikimedia.org/T192370) (owner: 10Giuseppe Lavagetto) [22:22:44] (03CR) 10Aaron Schulz: [C: 031] mcrouter: add support for listening on the ssl port [puppet] - 10https://gerrit.wikimedia.org/r/431736 (https://phabricator.wikimedia.org/T192370) (owner: 10Giuseppe Lavagetto) [22:57:39] (03PS5) 10Krinkle: mtail: Add xcachestatus to varnishrls [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978) [22:57:42] (03PS6) 10Krinkle: mtail: Add xcachestatus to varnishrls [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978) [22:58:04] (03PS7) 10Krinkle: mtail: Add xcachestatus to varnishrls [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978)