[00:06:56] 10Operations, 10MediaWiki-Authentication-and-authorization, 10Security-Team, 10Traffic, 10Security: Investigate usefulness of SameSite cookies for logged-in accounts - https://phabricator.wikimedia.org/T158604 (10Tgr) Also, if I read the spec correctly (and this more approachable [[http://www.sjoerdlangk... [02:57:09] (03PS2) 10Legoktm: Improve Python 3 compatibility [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561262 [02:57:56] (03CR) 10Legoktm: "> Patch Set 1: Code-Review-1" [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561262 (owner: 10Legoktm) [02:58:45] (03CR) 10jerkins-bot: [V: 04-1] Improve Python 3 compatibility [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561262 (owner: 10Legoktm) [02:59:40] (03CR) 10Legoktm: "> 18:57:18 E: /srv/pbuilder/aptcache/trusty-amd64: No such directory" [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561262 (owner: 10Legoktm) [05:39:34] (03PS1) 10Ammarpad: Disable MobileFrontend Mainpage special casing on frwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562025 (https://phabricator.wikimedia.org/T241888) [05:58:14] (03PS9) 10Ammarpad: Add minerva custom log for la.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/557439 (https://phabricator.wikimedia.org/T240728) [06:00:39] (03CR) 10Ammarpad: "> This is ready to SWAT. Thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/557439 (https://phabricator.wikimedia.org/T240728) (owner: 10Ammarpad) [07:45:14] 10Operations, 10Traffic: Add more detailed instructions to the "sec-advice" page - https://phabricator.wikimedia.org/T241309 (10Nemo_bis) I don't necessarily agree with splitting to tabs or other things, it's better to have specific tasks. [07:46:28] 10Operations, 10Traffic, 10HTTPS, 10Voice & Tone: sec-warning page uses the term "Wikipedia" incorrectly - https://phabricator.wikimedia.org/T241656 (10Nemo_bis) 05duplicate→03Open [07:47:35] 10Operations, 10Traffic, 10HTTPS, 10Voice & Tone: sec-warning page uses the term "Wikipedia" incorrectly - https://phabricator.wikimedia.org/T241656 (10Nemo_bis) I've clarified the title per T241309#5776114. The other tasks are orthogonal to this and it's just confusing to merge them. [07:55:10] 10Operations, 10Mail: Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail - https://phabricator.wikimedia.org/T58414 (10Nemo_bis) >>! In T58414#5772209, @chasemp wrote: > seems resolved from comments I don't see a comment confirming that WMF has been added to the bulk senders whi... [09:55:34] (03PS1) 10Ammarpad: Add ipblock-exempt and extendedconfirmed to bot group on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562040 (https://phabricator.wikimedia.org/T241904) [11:43:57] PROBLEM - Router interfaces on cr1-codfw is CRITICAL: CRITICAL: host 208.80.153.192, interfaces up: 133, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [11:43:57] PROBLEM - Router interfaces on cr4-ulsfo is CRITICAL: CRITICAL: host 198.35.26.193, interfaces up: 68, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [12:49:25] (03PS10) 10Urbanecm: Add minerva custom logo for la.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/557439 (https://phabricator.wikimedia.org/T240728) (owner: 10Ammarpad) [12:49:58] (03CR) 10Urbanecm: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562040 (https://phabricator.wikimedia.org/T241904) (owner: 10Ammarpad) [12:53:17] (03CR) 10Urbanecm: "small note" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561927 (https://phabricator.wikimedia.org/T235717) (owner: 10Gergő Tisza) [12:59:33] (03CR) 10Urbanecm: Rearrange of wmgEnableGeoData (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561658 (owner: 10Zoranzoki21) [13:03:00] (03CR) 10Zoranzoki21: Rearrange of wmgEnableGeoData (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561658 (owner: 10Zoranzoki21) [13:09:56] !log mwmaint1002: mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Coffeeandcrumbs /home/urbanecm/T241917 (T241917) [13:09:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:10:01] T241917: Server-side upload for Coffeeandcrumbs - https://phabricator.wikimedia.org/T241917 [13:38:02] (03PS2) 10Ammarpad: Add sandboxlink for eswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559576 (https://phabricator.wikimedia.org/T241163) [13:38:39] (03PS5) 10Ammarpad: Add new namespace and aliases for zh.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/558827 (https://phabricator.wikimedia.org/T241023) [14:26:14] (03PS1) 10Superzerocool: Add a new throttle rule and remove outdated rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) [14:26:16] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [14:54:48] (03PS2) 10Elukey: graphite::wmcs::archiver: move warnings to /dev/null [puppet] - 10https://gerrit.wikimedia.org/r/561892 [14:56:39] (03CR) 10Elukey: [C: 03+2] "Andrew: merging as it seems a very trivial change that can be reverted anytime :)" [puppet] - 10https://gerrit.wikimedia.org/r/561892 (owner: 10Elukey) [15:38:39] ACKNOWLEDGEMENT - Router interfaces on cr1-codfw is CRITICAL: CRITICAL: host 208.80.153.192, interfaces up: 133, down: 1, dormant: 0, excluded: 0, unused: 0: CDanis Zayo case TTN-0003800485 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [15:38:39] ACKNOWLEDGEMENT - Router interfaces on cr4-ulsfo is CRITICAL: CRITICAL: host 198.35.26.193, interfaces up: 68, down: 1, dormant: 0, excluded: 0, unused: 0: CDanis Zayo case TTN-0003800485 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:04:18] (03CR) 10Zoranzoki21: [C: 03+1] "Change looks good!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [16:31:44] 10Operations, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Dendelele) [16:53:31] 10Operations, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Krenair) Redirect target is a 404 [17:12:09] 10Operations, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Krenair) Also, registering a domain costs money - but @Dendelele is tied to a non-(WMF) CentralAuth account that doesn't seem to have a WMF user account disclaimer either. [19:49:09] 10Operations, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Elukey: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen - https://phabricator.wikimedia.org/T208934 (10jijiki) [19:49:49] 10Operations, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Elukey: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen - https://phabricator.wikimedia.org/T208934 (10jijiki) [19:49:53] 10Operations, 10MediaWiki-Cache, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Mcrouter periodically reports soft TKOs for mc1029 (was mc1035, mc1022) leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10jijiki) [19:52:14] 10Operations, 10ops-codfw, 10serviceops: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet - https://phabricator.wikimedia.org/T241796 (10jijiki) [19:52:43] 10Operations, 10ops-eqiad, 10serviceops: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet - https://phabricator.wikimedia.org/T241795 (10jijiki) [19:57:59] 10Operations, 10ops-codfw, 10serviceops: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet - https://phabricator.wikimedia.org/T241796 (10jijiki) [19:58:06] 10Operations, 10ops-eqiad, 10serviceops: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet - https://phabricator.wikimedia.org/T241795 (10jijiki) [19:58:08] 10Operations, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Elukey: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen - https://phabricator.wikimedia.org/T208934 (10jijiki) [20:18:37] 10Operations, 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [20:27:37] 10Operations, 10Puppet, 10Cloud-Services: Update redis puppet class to support stretch - https://phabricator.wikimedia.org/T166233 (10jijiki) 05Open→03Resolved a:03jijiki Given we have installed redis servers in stretch, I think we can mark this as resolved [20:37:26] 10Operations, 10Design-Research, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Aklapper) Hi @Dendelele and welcome to Wikimedia Phabricator. Where was this discussed in public before creating this task? [20:45:13] 10Operations, 10serviceops, 10HHVM, 10MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), and 2 others: Remove HHVM from production - https://phabricator.wikimedia.org/T229792 (10jijiki) 05Open→03Resolved [20:45:16] 10Operations, 10serviceops: SRE FY19-20 Q1 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10jijiki) [21:09:47] PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 33318360 and 5 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:11:35] RECOVERY - Postgres Replication Lag on maps1002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 0 and 69 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [22:22:59] RECOVERY - Router interfaces on cr1-codfw is OK: OK: host 208.80.153.192, interfaces up: 135, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [22:22:59] RECOVERY - Router interfaces on cr4-ulsfo is OK: OK: host 198.35.26.193, interfaces up: 70, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [23:43:05] PROBLEM - Host cp3065 is DOWN: PING CRITICAL - Packet loss = 100% [23:53:46] !log jiji@cumin1001 conftool action : set/pooled=no; selector: name=cp3065.esams.wmnet [23:53:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:56:00] !log powecycle cp3065.esams.wmnet T238305 [23:56:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:56:03] T238305: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305