[00:06:26] (03PS1) 10Dzahn: releases: add a MOTD warning based on active/passive server [puppet] - 10https://gerrit.wikimedia.org/r/368529 [00:08:25] (03PS2) 10Dzahn: releases: add a MOTD warning based on active/passive server [puppet] - 10https://gerrit.wikimedia.org/r/368529 [00:09:21] (03CR) 10jerkins-bot: [V: 04-1] releases: add a MOTD warning based on active/passive server [puppet] - 10https://gerrit.wikimedia.org/r/368529 (owner: 10Dzahn) [00:13:21] (03PS3) 10Dzahn: releases: add a MOTD warning based on active/passive server [puppet] - 10https://gerrit.wikimedia.org/r/368529 [00:19:07] (03PS4) 10Dzahn: releases: add a MOTD warning based on active/passive server [puppet] - 10https://gerrit.wikimedia.org/r/368529 [00:22:09] (03CR) 10Dzahn: [C: 032] releases: add a MOTD warning based on active/passive server [puppet] - 10https://gerrit.wikimedia.org/r/368529 (owner: 10Dzahn) [01:16:44] 10Operations: librenms - syslog stopped working after migration - https://phabricator.wikimedia.org/T172008#3483408 (10Dzahn) After following some red herrings like thinking it's about ferm/firewall and IPv6,... it turned out it wasn't and killing `php /usr/local/sbin/librenms-syslog ` and letting it start again... [01:18:10] 10Operations: librenms - syslog stopped working after migration - https://phabricator.wikimedia.org/T172008#3483409 (10Dzahn) 05Open>03Resolved https://librenms.wikimedia.org/device/device=145/tab=logs/section=syslog/ shows current logs again [01:39:10] (03PS11) 10Krinkle: varnish: Switch browsersec to use errorpage template [puppet] - 10https://gerrit.wikimedia.org/r/355338 (https://phabricator.wikimedia.org/T113114) [01:39:12] (03CR) 10Krinkle: "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/350966 (https://phabricator.wikimedia.org/T113114) (owner: 10Krinkle) [01:50:46] (03PS12) 10Krinkle: varnish: Switch browsersec to use errorpage template [puppet] - 10https://gerrit.wikimedia.org/r/355338 (https://phabricator.wikimedia.org/T113114) [01:55:58] ema: I've fixed it per your feedback ^ [03:26:59] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 838.10 seconds [04:24:18] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 173.05 seconds [04:39:04] 10Operations, 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Performance-Team, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3483476 (10Krinkle) [04:41:10] (03CR) 10Krinkle: [C: 031] "Added for next Tuesday" [puppet] - 10https://gerrit.wikimedia.org/r/364148 (https://phabricator.wikimedia.org/T103886) (owner: 10Giuseppe Lavagetto) [04:43:25] 10Operations, 10MediaWiki-Configuration, 10Performance-Team, 10Availability (Multiple-active-datacenters), and 5 others: Integrating MediaWiki (and other services) with dynamic configuration - https://phabricator.wikimedia.org/T149617#3483484 (10Krinkle) [05:51:19] PROBLEM - pdfrender on scb1003 is CRITICAL: connect to address 10.64.32.153 and port 5252: Connection refused [06:15:52] <_joe_> !log restarting pdfrender on scb1003 [06:16:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:16:29] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.002 second response time [08:27:53] 10Operations, 10Domains, 10Education-Program-Dashboard, 10Traffic: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332#3483524 (10Yury_Bulka) Maybe something like dash.wmflabs.org at least? [09:00:24] (03CR) 10MarcoAurelio: Initial configuration for hiwikiversity (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368165 (https://phabricator.wikimedia.org/T168765) (owner: 10Urbanecm) [09:05:06] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483534 (10MarcoAurelio) > Topic: विषय: Topic_talk: विषय_वार्ता: Hello. I see a problem with this. Flow namespace is also called "Topic" an... [11:08:18] (03CR) 10Urbanecm: [C: 04-1] "See the comment" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368165 (https://phabricator.wikimedia.org/T168765) (owner: 10Urbanecm) [12:25:38] PROBLEM - carbon-cache too many creates on graphite1001 is CRITICAL: CRITICAL: 1.67% of data above the critical threshold [1000.0] [12:35:46] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483624 (10Jayprakash12345) >>! In T168765#3483534, @MarcoAurelio wrote: >> Topic: विषय: Topic_talk: विषय_वार्ता: > > Hello. I see a proble... [12:59:09] (03Draft1) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [12:59:12] (03PS2) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [12:59:43] (03PS3) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [13:16:03] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483656 (10MarcoAurelio) @Jayprakash12345 Flow is a ~~horrible~~ extension which is being installed on this wiki as well. The namespace that... [13:22:31] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3376137 (10YmKavishwar) Subject=विषय Topic=मुद्दा [13:22:48] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483666 (10Liuxinyu970226) >>! In T168765#3483656, @MarcoAurelio wrote: > @Jayprakash12345 Flow is a ~~horrible~~ extension which is being i... [13:23:37] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483668 (10Jayprakash12345) >>! In T168765#3483663, @YmKavishwar wrote: > Subject=विषय > Topic=मुद्दा No [13:23:49] PROBLEM - carbon-cache too many creates on graphite1001 is CRITICAL: CRITICAL: 1.67% of data above the critical threshold [1000.0] [13:24:17] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483669 (10Urbanecm) Discuss this somewhere else and give us FINAL TRANSLATIONS we can use for configuration, please. Thank you! [13:25:14] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483670 (10MarcoAurelio) Yes I concur with @Urbanecm - another option is not to enable Flow there if you're not going to use it. [13:27:42] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3483671 (10Jayprakash12345) >>! In T168765#3483670, @MarcoAurelio wrote: > Yes I concur with @Urbanecm - another option is not to enable Flo... [13:28:58] RECOVERY - carbon-cache too many creates on graphite1001 is OK: OK: Less than 1.00% above the threshold [500.0] [13:36:25] (03PS1) 10Ottomata: Temporarily reactivate ironhold's account [puppet] - 10https://gerrit.wikimedia.org/r/368548 (https://phabricator.wikimedia.org/T171696) [13:37:43] (03CR) 10Ottomata: [C: 032] Temporarily reactivate ironhold's account [puppet] - 10https://gerrit.wikimedia.org/r/368548 (https://phabricator.wikimedia.org/T171696) (owner: 10Ottomata) [13:38:30] (03PS4) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [13:44:33] (03PS5) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [13:46:55] (03PS6) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [13:47:39] (03CR) 10Paladox: "The logo is from https://upload.wikimedia.org/wikipedia/commons/thumb/8/81/Wikimedia-logo.svg/752px-Wikimedia-logo.svg.png and this http:/" [puppet] - 10https://gerrit.wikimedia.org/r/368547 (owner: 10Paladox) [13:52:07] 10Operations, 10ops-eqiad, 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3483702 (10Ottomata) If we decide to keep broker ids (seeming less likely, I will test some stuff in labs this week), then I... [14:06:28] (03PS7) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [15:06:56] (03PS6) 10Umherirrender: Add ar_content_format and ar_content_model to labs views [puppet] - 10https://gerrit.wikimedia.org/r/363851 (https://phabricator.wikimedia.org/T89741) [16:23:08] PROBLEM - puppet last run on labvirt1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:12:59] PROBLEM - Check Varnish expiry mailbox lag on cp1074 is CRITICAL: CRITICAL: expiry mailbox lag is 2097581 [17:35:10] (03PS1) 10Reedy: extension.json usage for Scribunto [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368554 (https://phabricator.wikimedia.org/T139800) [18:02:47] (03CR) 10Paladox: [C: 04-1] "Will cause gerrit to fail to start up. Working on a fix upstream." [puppet] - 10https://gerrit.wikimedia.org/r/368547 (owner: 10Paladox) [18:10:33] (03PS8) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [18:12:58] RECOVERY - Check Varnish expiry mailbox lag on cp1074 is OK: OK: expiry mailbox lag is 0 [18:13:28] (03CR) 10Paladox: "We won't be able to use the existing url we are currently using as it will look really weird and not nice so instead i've done it with jus" [puppet] - 10https://gerrit.wikimedia.org/r/368547 (owner: 10Paladox) [18:13:40] (03CR) 10Paladox: "url = logo" [puppet] - 10https://gerrit.wikimedia.org/r/368547 (owner: 10Paladox) [18:33:23] (03PS9) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [18:35:40] (03PS10) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [18:36:37] (03CR) 10jerkins-bot: [V: 04-1] Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 (owner: 10Paladox) [18:37:12] (03PS11) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [18:38:10] (03CR) 10jerkins-bot: [V: 04-1] Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 (owner: 10Paladox) [18:51:21] (03PS12) 10Paladox: Gerrit: Add wmf branding to PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/368547 [18:55:53] (03CR) 10Paladox: [C: 031] "tested on 2.14 and found no errors or problems. It won't show on that version but does on 2.15." [puppet] - 10https://gerrit.wikimedia.org/r/368547 (owner: 10Paladox) [19:55:22] (03CR) 10Umherirrender: [C: 031] extension.json usage for Scribunto [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368554 (https://phabricator.wikimedia.org/T139800) (owner: 10Reedy) [20:00:43] 10Operations, 10Commons, 10MediaWiki-extensions-Scribunto, 10Patch-For-Review, 10Wikimedia-log-errors: Some Commons pages transcluding Template:Countries_of_Europe HTTP 500/503 when accessed from non-English languages specified in the template - https://phabricator.wikimedia.org/T171392#3484073 (10zhuyife... [22:33:19] ACKNOWLEDGEMENT - HP RAID on ms-be1017 is CRITICAL: CRITICAL: Slot 1: OK: 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, 2I:4:1, 2I:4:2 - Controller: OK - Cache: Permanently Disabled - Cable Error - Battery/Capacitor: Recharging nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T172051 [22:33:23] 10Operations, 10ops-eqiad: Degraded RAID on ms-be1017 - https://phabricator.wikimedia.org/T172051#3484109 (10ops-monitoring-bot) [22:52:26] PROBLEM - Debian mirror in sync with upstream on sodium is CRITICAL: /srv/mirrors/debian is over 14 hours old.