[08:37:23] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Add x-request-id to httpd (apache) logs - https://phabricator.wikimedia.org/T244545 (10Joe) [09:34:50] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Performance-Team (Radar): Ensure apcu incr/decr are atomic (Upgrade php-apcu) - https://phabricator.wikimedia.org/T236800 (10jijiki) 05Open→03Resolved done, please reopen if something is not working as expected [10:24:14] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Add x-request-id to httpd (apache) logs - https://phabricator.wikimedia.org/T244545 (10jijiki) p:05Triage→03Medium [10:42:06] 10serviceops, 10OTRS: OTRS CVE-2020-1768, CVE-2019-11358 - https://phabricator.wikimedia.org/T244713 (10akosiaris) [10:42:25] 10serviceops, 10OTRS: OTRS CVE-2020-1768, CVE-2019-11358 - https://phabricator.wikimedia.org/T244713 (10akosiaris) 05Open→03Resolved p:05Triage→03Medium [10:43:00] 10serviceops, 10OTRS, 10Security: OTRS CVE-2020-1768, CVE-2019-11358 - https://phabricator.wikimedia.org/T244713 (10akosiaris) Adding security team as an FYI [11:50:07] 10serviceops, 10Operations: Update php-xdebug to 2.9.2 in apt.wm.o component/php72 - https://phabricator.wikimedia.org/T244716 (10jijiki) [11:50:57] 10serviceops, 10Analytics, 10Operations, 10vm-requests: Create a ganeti VM in eqiad: an-tool1008 - https://phabricator.wikimedia.org/T244717 (10jijiki) [11:51:05] 10serviceops, 10Analytics, 10Operations, 10vm-requests: Create a ganeti VM in eqiad: an-tool1008 - https://phabricator.wikimedia.org/T244717 (10jijiki) p:05Triage→03Medium [11:51:48] 10serviceops, 10Analytics, 10Operations, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10jijiki) p:05Triage→03Medium [14:24:32] hi akosiaris! i know last week was crazy!...gentle bump on our discussions [14:25:17] I love how otto is opening with a gentle touch [14:33:00] hahah, yes, gentle touch, even better than gentle bump. [15:12:04] 10serviceops, 10observability, 10Wikimedia-Incident: Add alert for app servers in prod serving outdated MediaWiki branches - https://phabricator.wikimedia.org/T242023 (10mmodell) >>! In T242023#5842303, @hnowlan wrote: > Is it likely that there would be an update on the deploy server but not on the app serve... [15:27:18] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2334.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [15:43:23] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2334.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2334.codfw.wmnet'] ` [16:46:48] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10Release-Engineering-Team (Deployment services): Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10hnowlan) [16:48:25] 10serviceops, 10observability, 10Wikimedia-Incident: Add alert for app servers in prod serving outdated MediaWiki branches - https://phabricator.wikimedia.org/T242023 (10hnowlan) This has been rolled out to all `profile::mediawiki` hosts as `check_mw_wikiversion_difference` in nrpe and `Ensure local MW vers... [18:29:22] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10Release-Engineering-Team (Deployment services): Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10mmodell) it would be nice if the version information could be broken down in some way so that a small change to one file didn'... [18:56:44] 10serviceops, 10Operations, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10jijiki) On mwdebug1001 we have deployed a config similar to T240684#5826966, where we failover in case of a TKO to the gutter pool servers (mc-... [18:57:28] 10serviceops, 10Operations, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10jijiki) [19:33:26] 10serviceops, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Test-Coverage: Add pcov PHP extension to wikimedia apt so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Jdforrester-WMF) >>! In T243847#585... [19:51:35] 10serviceops, 10Analytics, 10Operations, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10Dzahn) Just to confirm. It should still have a public IP in wikimedia.org ? [20:03:22] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10Release-Engineering-Team (Deployment services): Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10Dzahn) The problem i see with that is how do you define what a "small" change is? It feels like when we call things "trivial"... [20:51:30] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) [21:28:37] 10serviceops, 10Core Platform Team, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10aaron) a:05aaron→03None [21:28:53] 10serviceops, 10Core Platform Team, 10Operations, 10Performance-Team (Radar), 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10aaron) [22:37:42] 10serviceops, 10Operations: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) [22:39:08] 10serviceops, 10Operations: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) [22:39:45] 10serviceops, 10Operations: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) @jijiki @Urbanecm I added 2 more canary appservers. now we have: ` mwdebug2001 (row A, ganeti VM) mwdebug2002 (row B, ganeti VM) mw2163 (C3, physical) mw2164 (C3, physical) mw2271 (D3, physical)... [22:40:27] 10serviceops, 10Operations: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) done now?