[00:15:28] 10Operations, 10ops-eqsin: rack/setup scs-eqsin.mgmt.eqsin.wmnet - https://phabricator.wikimedia.org/T181569#3855837 (10RobH) Ok, smarthands ticket (and directions were included with ticket) for the hookup of scs and application of other network gear hostname labels are on EQ smarthands ticket 1-122991824011.... [00:20:44] (03PS2) 10EddieGP: Move wikipedia.org to www.wikipedia.org vhost [puppet] - 10https://gerrit.wikimedia.org/r/398396 [00:39:38] (03CR) 10EddieGP: [C: 04-1] "> I don't think this unification is possible - a ton of configuration" [puppet] - 10https://gerrit.wikimedia.org/r/398396 (owner: 10EddieGP) [00:40:11] (03PS3) 10EddieGP: [WIP] Move wikipedia.org to www.wikipedia.org vhost [puppet] - 10https://gerrit.wikimedia.org/r/398396 [01:03:14] (03PS2) 10EddieGP: beta: Combine commons, deployments, meta and zero vhost [puppet] - 10https://gerrit.wikimedia.org/r/398399 [01:11:19] (03CR) 10EddieGP: "I found another vhost in beta_specific.conf (deployments) and one in wikimedia.conf (zero) that are identical to the two in remnant.conf (" [puppet] - 10https://gerrit.wikimedia.org/r/398399 (owner: 10EddieGP) [03:05:42] PROBLEM - pdfrender on scb1002 is CRITICAL: connect to address 10.64.16.21 and port 5252: Connection refused [06:44:11] !log start manual backup of db1029 onto db1056 [06:44:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:19:09] (03CR) 10Eranroz: [C: 04-1] "see https://phabricator.wikimedia.org/T172914" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/398823 (https://phabricator.wikimedia.org/T172914) (owner: 10Ladsgroup) [07:27:02] (03CR) 10Elukey: [C: 031] Parameterize kafka.ssl.cipher.suites (031 comment) [puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/399689 (https://phabricator.wikimedia.org/T177225) (owner: 10Ottomata) [07:32:32] (03CR) 10Elukey: [C: 031] "Let's also make sure that TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 works on cp1008, iirc when I tried it vk didn't like it a lot (but it ma" [puppet] - 10https://gerrit.wikimedia.org/r/399700 (https://phabricator.wikimedia.org/T167304) (owner: 10Ottomata) [08:25:48] (03PS1) 10Jcrespo: Revert "mariadb: Remove references to db1055 & db1056" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399782 [08:34:43] (03PS2) 10Jcrespo: mariadb: Repool db1055 & db1056 as x1 replicas [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399782 (https://phabricator.wikimedia.org/T183470) [08:38:02] (03PS3) 10Jcrespo: mariadb: Repool db1055 & db1056 as x1 replicas [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399782 (https://phabricator.wikimedia.org/T183470) [08:46:27] (03CR) 10Jayprakash12345: "@Chad, If you have Merge right and Production. You can merge like you do sometime before." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399556 (https://phabricator.wikimedia.org/T181774) (owner: 10Jayprakash12345) [08:58:40] (03PS1) 10Jcrespo: mariadb: Productionize db1055 and db1056 by enabling notifications [puppet] - 10https://gerrit.wikimedia.org/r/399786 (https://phabricator.wikimedia.org/T183469) [09:07:29] (03CR) 10Jcrespo: [C: 031] "This can go any time, probably will not be done until next year." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399782 (https://phabricator.wikimedia.org/T183470) (owner: 10Jcrespo) [09:07:45] (03CR) 10Jcrespo: [C: 032] mariadb: Productionize db1055 and db1056 by enabling notifications [puppet] - 10https://gerrit.wikimedia.org/r/399786 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [09:18:15] 10Operations, 10ops-eqiad, 10hardware-requests, 10monitoring, and 2 others: decom uranium - https://phabricator.wikimedia.org/T183209#3856289 (10Volans) p:05Triage>03Normal [09:20:21] 10Operations, 10MediaWiki-Vagrant, 10Patch-For-Review: Import kibana package from jessie into stretch - https://phabricator.wikimedia.org/T183071#3856290 (10Volans) p:05Triage>03Normal [09:40:49] (03PS1) 10Jcrespo: mariadb: Decommissioning proposal [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399792 (https://phabricator.wikimedia.org/T134476) [09:42:17] (03CR) 10jerkins-bot: [V: 04-1] mariadb: Decommissioning proposal [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399792 (https://phabricator.wikimedia.org/T134476) (owner: 10Jcrespo) [09:43:15] (03PS2) 10Jcrespo: mariadb: Decommissioning proposal [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399792 (https://phabricator.wikimedia.org/T134476) [09:43:54] (03Abandoned) 10Jcrespo: Upcoming mediawiki-core hardware expansion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/338996 (https://phabricator.wikimedia.org/T158580) (owner: 10Jcrespo) [09:48:19] (03PS3) 10Jcrespo: mariadb: Decommissioning proposal [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399792 (https://phabricator.wikimedia.org/T134476) [09:49:45] (03CR) 10Jcrespo: [C: 04-2] "This is a first draft, but it should be good enough. Still needs to be optimized for:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399792 (https://phabricator.wikimedia.org/T134476) (owner: 10Jcrespo) [11:40:23] (03PS1) 10Urbanecm: Update logo for chrwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399799 (https://phabricator.wikimedia.org/T180553) [11:41:31] 10Operations, 10Wikimedia-Mailing-lists: Emails send by subscribers don't arrive on the mailing list Moderators-nl - https://phabricator.wikimedia.org/T181906#3856481 (10Magere_Hein) I tried to send a message twice from a gmail account to this mailing list earlier (12:09 CET and 12:10 CET), but that failed mis... [12:22:46] PROBLEM - Varnish HTTP text-backend - port 3128 on cp4028 is CRITICAL: connect to address 10.128.0.128 and port 3128: Connection refused [12:23:46] RECOVERY - Varnish HTTP text-backend - port 3128 on cp4028 is OK: HTTP OK: HTTP/1.1 200 OK - 218 bytes in 0.157 second response time [12:40:13] Hello, why is arbcom_cswiki in both wikipedia.dblist and special.dblist in operations/mediawiki-config? [13:06:09] 10Operations, 10Analytics-Cluster, 10Analytics-Kanban, 10Traffic, 10User-Elukey: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3856606 (10BBlack) https://en.wikipedia.org/wiki/There_are_known_knowns * Known Knowns: * Restrict ciphersuite selection to a strong FS+AE... [13:07:07] Urbanecm: I guess it should not be in wikipedia.dblist? [13:07:18] not sure. maybe error or you may found a reason in the commit msg? [13:07:30] Sagan, thats what I think too, but it is there... [13:07:45] Urbanecm: I guess other arbcom wikis are not in that dblist? [13:08:36] They are [13:08:40] I can take a look if I find that commit [13:08:48] Urbanecm: at wikipedia.dblist? [13:09:54] Yes [13:10:05] BTW commit for arbcom_cswiki is https://gerrit.wikimedia.org/r/#/c/323843/ [13:10:12] Sagan, ^^^ [13:10:19] Urbanecm: hm, but then I don't see a problem with that behaviour, do you? [13:10:45] Urbanecm: you can just enable blame here: https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/dblists/wikipedia.dblist [13:10:46] I do not think that wikis should be in more than one family (Wikipedia, Wiktionary, Wikibooks etc.+Special) [13:11:19] hm, but that would be a thing affecting all arbcom wikis then. Guess that looks like we need a task [13:11:23] maybe there is a reason for that [13:12:20] Ok, so do you think I should create a task? [13:15:46] (03PS1) 10Urbanecm: Switch from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) [13:17:05] (03CR) 10jerkins-bot: [V: 04-1] Switch from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [13:17:14] !log repooling cp4032 - T183176 [13:17:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:17:28] T183176: cp4032 memory error - https://phabricator.wikimedia.org/T183176 [13:17:32] Morning all, hows everyone? [13:20:34] 10Operations, 10ops-ulsfo, 10Traffic: cp4032 memory error - https://phabricator.wikimedia.org/T183176#3845938 (10BBlack) 05Open>03Resolved [13:27:51] (03Abandoned) 10Urbanecm: Update logo for chrwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399799 (https://phabricator.wikimedia.org/T180553) (owner: 10Urbanecm) [13:29:29] (03PS1) 10Urbanecm: Update chrwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399806 (https://phabricator.wikimedia.org/T180553) [13:31:04] (03CR) 10jerkins-bot: [V: 04-1] Update chrwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399806 (https://phabricator.wikimedia.org/T180553) (owner: 10Urbanecm) [13:37:14] !log restbase depool restbase2008 for T179419 [13:37:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:37:29] T179419: Migrate mathoid storage from legacy to new strategy - https://phabricator.wikimedia.org/T179419 [13:41:20] 10Operations, 10Domains, 10Traffic: Purchase domains mediawi.ki and media.wiki to use as a url shortener - https://phabricator.wikimedia.org/T180657#3856695 (10BBlack) 05stalled>03declined I don't think there's anything to do here. We have a shortener domain that's operational at the #traffic level in `... [13:47:05] 10Operations, 10Analytics, 10Research, 10Traffic, and 4 others: Referrer policy for browsers which only support the old spec - https://phabricator.wikimedia.org/T180921#3773297 (10BBlack) Open Q's from my POV: 1) Flipping these Edges/Safaris to `origin` is going to deny us information on internal referrer... [13:50:09] 10Operations, 10ops-esams, 10Traffic: cp3048 crashed - https://phabricator.wikimedia.org/T180424#3856715 (10BBlack) 05Open>03Resolved a:03BBlack Clearing this ticket for now, it's been over a month. Will revive if it crashes again. [13:50:43] 10Operations, 10ops-esams, 10Traffic: cp3043 disk failure - https://phabricator.wikimedia.org/T179953#3856728 (10BBlack) Any update on disk replacement here? [14:02:47] !log mobrovac@tin Started deploy [mathoid/deploy@7f39b71]: Update Mathoid to v0.7.0 in CODFW only to prefill storage - T179419 T172767 [14:02:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:02:59] T179419: Migrate mathoid storage from legacy to new strategy - https://phabricator.wikimedia.org/T179419 [14:02:59] T172767: Prepare mathoid 0.7 release (tracking) - https://phabricator.wikimedia.org/T172767 [14:05:44] !log mobrovac@tin Finished deploy [mathoid/deploy@7f39b71]: Update Mathoid to v0.7.0 in CODFW only to prefill storage - T179419 T172767 (duration: 02m 57s) [14:05:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:13:35] !log mobrovac@tin Started deploy [mathoid/deploy@6c29c09]: Update Mathoid to v0.7.0 in CODFW only to prefill storage, take 2 - T179419 T172767 [14:13:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:13:47] T179419: Migrate mathoid storage from legacy to new strategy - https://phabricator.wikimedia.org/T179419 [14:13:47] T172767: Prepare mathoid 0.7 release (tracking) - https://phabricator.wikimedia.org/T172767 [14:17:58] !log mobrovac@tin Finished deploy [mathoid/deploy@6c29c09]: Update Mathoid to v0.7.0 in CODFW only to prefill storage, take 2 - T179419 T172767 (duration: 04m 23s) [14:18:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:33:54] 10Operations, 10Traffic: Unified certs bloat reduction? - https://phabricator.wikimedia.org/T183554#3856802 (10BBlack) p:05Triage>03Normal [14:42:19] (03PS1) 10Chad: Scap clean: simplify logic and actually delete thigns again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399814 [14:43:24] (03PS2) 10BBlack: switch esams to digicert-2017 unified cert [puppet] - 10https://gerrit.wikimedia.org/r/399683 [14:45:06] (03CR) 10BBlack: [C: 032] switch esams to digicert-2017 unified cert [puppet] - 10https://gerrit.wikimedia.org/r/399683 (owner: 10BBlack) [14:48:54] !log esams TLS cert switch from digicert-2016 to digicert-2017 [14:49:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:15] (03PS1) 10BBlack: eqsin: a few more basic puppetization fixups [puppet] - 10https://gerrit.wikimedia.org/r/399817 (https://phabricator.wikimedia.org/T156027) [15:00:50] (03PS2) 10BBlack: eqsin: a few more basic puppetization fixups [puppet] - 10https://gerrit.wikimedia.org/r/399817 (https://phabricator.wikimedia.org/T156027) [15:10:57] (03PS1) 10Volans: PuppetDB backend: add support for API v4 [software/cumin] - 10https://gerrit.wikimedia.org/r/399821 (https://phabricator.wikimedia.org/T182575) [15:13:12] (03CR) 10jerkins-bot: [V: 04-1] PuppetDB backend: add support for API v4 [software/cumin] - 10https://gerrit.wikimedia.org/r/399821 (https://phabricator.wikimedia.org/T182575) (owner: 10Volans) [15:14:46] (03CR) 10BBlack: [C: 032] eqsin: a few more basic puppetization fixups [puppet] - 10https://gerrit.wikimedia.org/r/399817 (https://phabricator.wikimedia.org/T156027) (owner: 10BBlack) [15:21:04] 10Operations, 10Mail, 10Surveys: Qualtrics email-LDAP issue - https://phabricator.wikimedia.org/T176666#3856934 (10Neil_P._Quinn_WMF) [15:21:06] (03PS1) 10Lucas Werkmeister (WMDE): Don’t check constraints on example properties [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399825 (https://phabricator.wikimedia.org/T183267) [15:27:40] (03PS1) 10Dzahn: DHCP: switch from jessie to stretch as default installer [puppet] - 10https://gerrit.wikimedia.org/r/399826 (https://phabricator.wikimedia.org/T182215) [15:37:21] 10Operations, 10Mail, 10Surveys: Qualtrics email-LDAP issue - https://phabricator.wikimedia.org/T176666#3856998 (10Neil_P._Quinn_WMF) >>! In T176666#3635538, @herron wrote: > FWIW I do see messages from the qualtrics mail system being relayed through the wikimedia.org MX and onward to google. Are there any... [15:40:12] (03PS2) 10Volans: PuppetDB backend: add support for API v4 [software/cumin] - 10https://gerrit.wikimedia.org/r/399821 (https://phabricator.wikimedia.org/T182575) [15:40:13] (03PS1) 10Volans: ClusterShell backend: fix execute() return code [software/cumin] - 10https://gerrit.wikimedia.org/r/399829 [15:42:16] (03PS2) 10Chad: Scap clean: simplify logic and actually delete thigns again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399814 [15:45:18] (03CR) 10Volans: "Tested in labs with the experimental version of the API v4 of our current version of PuppetDB (changing '/pdb/query/v4/' with '/v4/')." [software/cumin] - 10https://gerrit.wikimedia.org/r/399821 (https://phabricator.wikimedia.org/T182575) (owner: 10Volans) [15:48:14] (03PS3) 10Chad: Scap clean: simplify logic and actually delete thigns again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399814 [15:50:08] !log demon@tin Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 01m 47s) [15:50:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:50:31] (03PS4) 10Chad: Scap clean: simplify logic and actually delete thigns again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399814 [15:52:13] !log demon@tin Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 01m 18s) [15:52:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:43] (03CR) 10Chad: [C: 032] "Already live" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399814 (owner: 10Chad) [15:54:35] (03Merged) 10jenkins-bot: Scap clean: simplify logic and actually delete thigns again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399814 (owner: 10Chad) [15:56:48] !log demon@tin Synchronized scap/plugins/clean.py: no-op (duration: 00m 51s) [15:56:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:57:01] (03CR) 10jenkins-bot: Scap clean: simplify logic and actually delete thigns again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399814 (owner: 10Chad) [16:00:46] 10Operations, 10Mail, 10Surveys: Qualtrics cannot send email to wikimedia.org addresses - https://phabricator.wikimedia.org/T176666#3857086 (10Neil_P._Quinn_WMF) [16:00:55] 10Operations, 10Cloud-Services, 10Cloud-VPS: logrotate/disk space on silver for nutcracker log - https://phabricator.wikimedia.org/T120683#3857088 (10Andrew) 05Resolved>03declined [16:03:39] 10Operations, 10Contributors-Analysis, 10Mail, 10Surveys: Qualtrics cannot send email to wikimedia.org addresses - https://phabricator.wikimedia.org/T176666#3857105 (10Neil_P._Quinn_WMF) [16:05:40] !log demon@tin Synchronized README: forcing co-master sync (duration: 00m 51s) [16:05:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:16:13] (03Draft3) 10Jayprakash12345: Add new language Lingua Franca Nova - lfn.wikipedia.org [dns] - 10https://gerrit.wikimedia.org/r/399831 (https://phabricator.wikimedia.org/T183561) [16:18:31] (03PS1) 10ArielGlenn: fix up one more time the xml misc dumps dir for rsyncs [puppet] - 10https://gerrit.wikimedia.org/r/399835 [16:18:45] 10Operations, 10Puppet: Fix regex.yaml single-regex issue - https://phabricator.wikimedia.org/T183565#3857130 (10BBlack) [16:19:18] (03CR) 10ArielGlenn: [C: 032] fix up one more time the xml misc dumps dir for rsyncs [puppet] - 10https://gerrit.wikimedia.org/r/399835 (owner: 10ArielGlenn) [16:21:04] 10Operations, 10Puppet: Fix regex.yaml single-regex issue - https://phabricator.wikimedia.org/T183565#3857140 (10BBlack) p:05Triage>03Normal [16:21:32] (03PS2) 10BBlack: eqsin: deeper configuration details [puppet] - 10https://gerrit.wikimedia.org/r/392639 (https://phabricator.wikimedia.org/T156027) [16:21:34] (03PS1) 10BBlack: remove empty lvs_ulsfo from hieradata regexes [puppet] - 10https://gerrit.wikimedia.org/r/399836 (https://phabricator.wikimedia.org/T183565) [16:22:22] (03CR) 10jerkins-bot: [V: 04-1] eqsin: deeper configuration details [puppet] - 10https://gerrit.wikimedia.org/r/392639 (https://phabricator.wikimedia.org/T156027) (owner: 10BBlack) [16:24:55] (03CR) 10BBlack: [C: 032] remove empty lvs_ulsfo from hieradata regexes [puppet] - 10https://gerrit.wikimedia.org/r/399836 (https://phabricator.wikimedia.org/T183565) (owner: 10BBlack) [16:32:54] (03CR) 10BBlack: [C: 04-1] "I think this is pretty close, but needs some more review. It also does touch the config of many other services in subtle ways, so is a no" [puppet] - 10https://gerrit.wikimedia.org/r/392639 (https://phabricator.wikimedia.org/T156027) (owner: 10BBlack) [16:42:06] !log demon@tin Pruned MediaWiki: 1.31.0-wmf.11 [keeping static files] (duration: 01m 16s) [16:42:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:42:26] 10Operations, 10Mail: E-mail for people in different OIT LDAP object unit - https://phabricator.wikimedia.org/T159750#3077335 (10Neil_P._Quinn_WMF) @bbogaert, there are multiple uses for a limited-access OU (strategy contractors, other temporary contractors with different situations, Qualtrics email sending),... [16:43:23] 10Operations, 10Continuous-Integration-Infrastructure (shipyard): npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569#3857222 (10hashar) [16:46:34] (03CR) 10Gehel: ClusterShell backend: fix execute() return code (032 comments) [software/cumin] - 10https://gerrit.wikimedia.org/r/399829 (owner: 10Volans) [16:47:46] PROBLEM - puppet last run on labtestservices2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:52:20] 10Operations: alternatives to racktables ? - https://phabricator.wikimedia.org/T84001#3857287 (10Dzahn) I'll assume the existence of https://netbox.wikimedia.org/login/?next=/ means that evaluation has been done elswhere and this ticket from 2014 was declined. [16:53:04] 10Operations, 10Tracking: Hardware Automation Workflow - Overall Tracking - https://phabricator.wikimedia.org/T116063#3857291 (10Dzahn) [16:53:06] 10Operations: alternatives to racktables ? - https://phabricator.wikimedia.org/T84001#3857288 (10Dzahn) 05stalled>03Resolved a:03Dzahn well. or resolved.. however you want to see it. Just a bit sad that there were 0 comments over the years. [16:54:05] 10Operations: Migrate racktables to servermon - https://phabricator.wikimedia.org/T88424#3857300 (10Dzahn) Has this been declined in favor of netbox?? https://netbox.wikimedia.org/login/?next=/ [16:58:27] 10Puppet, 10MediaWiki-Vagrant, 10MediaWiki-extensions-SendGrid: Create a MW-Vagrant role for SendGrid extension - https://phabricator.wikimedia.org/T183571#3857388 (10D3r1ck01) [17:01:07] (03PS1) 10Thcipriani: l10nupdate: don't run over holidays [puppet] - 10https://gerrit.wikimedia.org/r/399839 [17:01:09] (03PS1) 10Thcipriani: l10nupdate: turn back on after holidays [puppet] - 10https://gerrit.wikimedia.org/r/399840 [17:03:35] (03CR) 10Chad: [C: 031] l10nupdate: don't run over holidays [puppet] - 10https://gerrit.wikimedia.org/r/399839 (owner: 10Thcipriani) [17:03:54] mutante: are you available today to look at ^ (ensuring no automated deployments run while everyone is gone) [17:07:14] 10Operations, 10Continuous-Integration-Infrastructure (shipyard): npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569#3857222 (10Paladox) Why are we using npm 1.4? I thought we upgraded to npm3? [17:12:46] RECOVERY - puppet last run on labtestservices2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:17:20] (03CR) 10Volans: [C: 031] "Compiler result is https://puppet-compiler.wmflabs.org/compiler02/9459/" [puppet] - 10https://gerrit.wikimedia.org/r/399839 (owner: 10Thcipriani) [17:18:12] (03CR) 10Volans: [C: 032] l10nupdate: don't run over holidays [puppet] - 10https://gerrit.wikimedia.org/r/399839 (owner: 10Thcipriani) [17:22:15] (03PS1) 10Volans: Revert "l10nupdate: don't run over holidays" [puppet] - 10https://gerrit.wikimedia.org/r/399848 [17:22:43] (03CR) 10Volans: "TO BE REVERTED once the holiday break ends" [puppet] - 10https://gerrit.wikimedia.org/r/399848 (owner: 10Volans) [17:30:50] volans: well done ) [17:37:48] 10Operations, 10ops-eqiad, 10hardware-requests, 10netops: unrack/decom pfw1-eqiad and pfw2-eqiad - https://phabricator.wikimedia.org/T183390#3857514 (10ayounsi) I looked at scs-c1-eqiad and pfw1 and pfw2 are not there. You might need to connect directly to the firewalls to do the wipe. [17:57:30] (03PS2) 10Milimetric: Add and link readme pages for analytics datasets [puppet] - 10https://gerrit.wikimedia.org/r/395917 (https://phabricator.wikimedia.org/T167033) [18:04:03] (03PS13) 10MarcoAurelio: Extension:Translate default permissions for Wikimedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/385953 (https://phabricator.wikimedia.org/T178793) [18:08:56] 10Puppet, 10MediaWiki-Vagrant, 10MediaWiki-extensions-SendGrid, 10Patch-For-Review: Create a MW-Vagrant role for SendGrid extension - https://phabricator.wikimedia.org/T183571#3857597 (10D3r1ck01) p:05Triage>03Normal [18:12:08] 10Operations, 10Cloud-VPS, 10Toolforge, 10cloud-services-team (Kanban): Toolforge's static webserver broken by Puppet changes and stale nginx packages - https://phabricator.wikimedia.org/T175885#3857602 (10Krinkle) [18:12:58] (03PS2) 10MarcoAurelio: Allow eswiki bureaucrats to add/remove 'accountcreator' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/395775 (https://phabricator.wikimedia.org/T182201) [18:13:05] (03PS2) 10MarcoAurelio: Add gwtoolset to GlobalGroupPermissions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/395806 [18:13:15] (03PS2) 10MarcoAurelio: [do not merge yet] wikimania2017: closing the wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/396581 (https://phabricator.wikimedia.org/T182493) [18:21:32] (03CR) 10MarcoAurelio: "Actually, one of the techblog links above redirect to https://blog.wikimedia.org/2010/06/15/usability-why-did-we-move-the-search-box/ redi" [puppet] - 10https://gerrit.wikimedia.org/r/394743 (https://phabricator.wikimedia.org/T181878) (owner: 10Framawiki) [18:28:30] !log mobrovac@tin Started deploy [mathoid/deploy@7c5f8e2]: Better handling of chem rules (codfw only) - T183557 [18:28:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:28:42] T183557: Mathoid v0.7.0 not accepting chem formula - https://phabricator.wikimedia.org/T183557 [18:31:15] !log mobrovac@tin Finished deploy [mathoid/deploy@7c5f8e2]: Better handling of chem rules (codfw only) - T183557 (duration: 02m 45s) [18:31:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:41:09] (03PS1) 10Catrope: labs: Fix language code on deploymentwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399867 [18:47:14] hi bawolff [18:47:20] hi [18:53:09] (03PS2) 10Urbanecm: Switch from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) [18:55:40] (03PS2) 10Urbanecm: Update chrwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399806 (https://phabricator.wikimedia.org/T180553) [18:59:05] 10Operations, 10ops-eqiad: Rack/cable/configure asw2-a/b/c-eqiad switch stack - https://phabricator.wikimedia.org/T183585#3857747 (10ayounsi) p:05Triage>03Normal [19:25:13] 10Operations, 10Mail: E-mail for people in different OIT LDAP object unit - https://phabricator.wikimedia.org/T159750#3857842 (10bbogaert) @MoritzMuehlenhoff , @Neil_P._Quinn_WMF , How about if we design a way where Office IT can create the groups that are needed in G-Suite without Ops having to make separate... [19:28:36] 10Operations, 10Mail: E-mail for people in different OIT LDAP object unit - https://phabricator.wikimedia.org/T159750#3857856 (10Neil_P._Quinn_WMF) @bbogaert, from my perspective (as someone who both works with Qualtrics and manages temporary contractors), that would work perfectly! [19:58:16] PROBLEM - BGP status on cr2-ulsfo is CRITICAL: BGP CRITICAL - AS6939/IPv4: Active, AS6939/IPv6: Active [20:10:22] (03PS1) 10Reedy: Add language-pack-eu to toollabs exec [puppet] - 10https://gerrit.wikimedia.org/r/399885 (https://phabricator.wikimedia.org/T183591) [20:33:29] (03PS1) 10Chad: Remove trailing slash, $prefix already does that [puppet] - 10https://gerrit.wikimedia.org/r/399892 [20:38:46] RECOVERY - BGP status on cr2-ulsfo is OK: BGP OK - up: 84, down: 0, shutdown: 2 [20:47:28] (03PS1) 10Andrew Bogott: puppet enc getter: only allow acccess to VMs and puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) [20:47:34] (03CR) 10jerkins-bot: [V: 04-1] puppet enc getter: only allow acccess to VMs and puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) (owner: 10Andrew Bogott) [20:50:54] (03PS2) 10Andrew Bogott: puppet enc getter: only allow acccess to VMs and puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) [20:51:30] (03CR) 10jerkins-bot: [V: 04-1] puppet enc getter: only allow acccess to VMs and puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) (owner: 10Andrew Bogott) [20:53:30] (03PS3) 10Andrew Bogott: puppet enc getter: only allow acccess to VMs and puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) [20:55:37] (03PS4) 10Phantom42: Fix linewrap issue on wikimedia error page [puppet] - 10https://gerrit.wikimedia.org/r/395552 (https://phabricator.wikimedia.org/T180656) [20:56:36] (03CR) 10Phantom42: Fix linewrap issue on wikimedia error page (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/395552 (https://phabricator.wikimedia.org/T180656) (owner: 10Phantom42) [20:57:39] (03PS1) 10Dzahn: smokeping: convert to profile [puppet] - 10https://gerrit.wikimedia.org/r/399902 [20:57:48] (03CR) 10jerkins-bot: [V: 04-1] smokeping: convert to profile [puppet] - 10https://gerrit.wikimedia.org/r/399902 (owner: 10Dzahn) [20:58:25] (03CR) 10Andrew Bogott: "The puppet compiler seems broken, so this is blocked on a proper test run there." [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) (owner: 10Andrew Bogott) [20:59:32] (03PS2) 10Dzahn: smokeping: convert to profile [puppet] - 10https://gerrit.wikimedia.org/r/399902 [21:26:14] (03CR) 10Andrew Bogott: [C: 032] Add language-pack-eu to toollabs exec [puppet] - 10https://gerrit.wikimedia.org/r/399885 (https://phabricator.wikimedia.org/T183591) (owner: 10Reedy) [21:31:26] (03CR) 10Dzahn: [C: 032] "no-op but fixes 3 style violations: http://puppet-compiler.wmflabs.org/9461/" [puppet] - 10https://gerrit.wikimedia.org/r/399902 (owner: 10Dzahn) [21:31:58] (03PS3) 10Dzahn: smokeping: convert to profile [puppet] - 10https://gerrit.wikimedia.org/r/399902 [21:32:26] andrewbogott: a compiler run just worked for me.. hmm [21:34:44] mutante: when I see that 'unknown variable realm' I assume that somehow the facts are out of date or missing for a given host... [21:34:44] https://puppet-compiler.wmflabs.org/compiler02/9460/labpuppetmaster1001.wikimedia.org/prod.labpuppetmaster1001.wikimedia.org.err [21:34:59] but maybe I misunderstand? [21:35:22] I got good runs yesterday so I figured that the state is 'broken for some nodes and not for others' [21:35:53] andrewbogott: no, you dont misunderstand. that's all true [21:35:58] i also got the "realm" error yesterday [21:36:08] but just 5 minutes ago on sometihng different i could compile just fine [21:36:50] my assumptions were "that's a general issue on everything" and then "it's fixed now" but those were wrong and the "broken for some nodes not for others" seems right [21:37:30] or it's just "sometimes it works" [21:37:50] puppet compiler farms out to different nodes, right? so maybe it's broken on one of them? [21:38:18] could be, yea [21:42:06] sigh [21:42:41] (03CR) 10Dzahn: "or do you think i should convert all of that to use http now?" [puppet] - 10https://gerrit.wikimedia.org/r/399826 (https://phabricator.wikimedia.org/T182215) (owner: 10Dzahn) [21:47:16] (03PS1) 10Smalyshev: [WIP] Add loading DCAT-AP data into dcatap namespace on WDQS [puppet] - 10https://gerrit.wikimedia.org/r/399954 (https://phabricator.wikimedia.org/T178978) [21:54:39] 10Operations, 10MediaWiki-Vagrant, 10Release-Engineering-Team, 10Epic, and 2 others: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#3858236 (10hashar) [22:28:47] 10Operations, 10Patch-For-Review: install_server: switch to stretch as default install image - https://phabricator.wikimedia.org/T182215#3858317 (10Dzahn) question to all: Do you think we should also change ALL of the hosts to use http to get the installer nowadays? ``` option pxelinux.pathprefix "htt... [23:03:17] (03PS1) 10Dzahn: librenms: convert role to profile, variables to params [puppet] - 10https://gerrit.wikimedia.org/r/399966 [23:03:45] (03CR) 10jerkins-bot: [V: 04-1] librenms: convert role to profile, variables to params [puppet] - 10https://gerrit.wikimedia.org/r/399966 (owner: 10Dzahn) [23:07:00] (03PS1) 10Dzahn: rancid: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/399968 [23:08:46] PROBLEM - IPv4 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 22 probes of 301 (alerts on 19) - https://atlas.ripe.net/measurements/1791210/#!map [23:10:27] (03PS2) 10Dzahn: librenms: convert role to profile, variables to params [puppet] - 10https://gerrit.wikimedia.org/r/399966 [23:12:06] 10Operations, 10Cloud-VPS, 10Traffic, 10netops, 10cloud-services-team (Kanban): Evaluate the possibility to add Juniper images to Openstack - https://phabricator.wikimedia.org/T180179#3749582 (10Krenair) >>! In T180179#3749635, @madhuvishy wrote: > Noting here that proprietary software is not usually ins... [23:13:47] RECOVERY - IPv4 ping to codfw on ripe-atlas-codfw is OK: OK - failed 1 probes of 301 (alerts on 19) - https://atlas.ripe.net/measurements/1791210/#!map [23:17:02] (03PS1) 10Dzahn: zotero: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/399969