[00:02:34] 10Operations, 10Patch-For-Review: Upgrade install servers to Buster - https://phabricator.wikimedia.org/T224576 (10Dzahn) @Muehlenhoff @ayounsi Tested squid on re-installed install1003.wikimedia.org: [planet1001:~] $ http_proxy="http://install1003.wikimedia.org:8080; export HTTPS_PROXY=http://install1003.... [00:08:02] (03PS4) 10Dzahn: switch webproxy CNAMEs to new install servers [dns] - 10https://gerrit.wikimedia.org/r/569680 (https://phabricator.wikimedia.org/T224576) [00:30:31] (03PS5) 10Dzahn: switch webproxy CNAMEs to new install servers [dns] - 10https://gerrit.wikimedia.org/r/569680 (https://phabricator.wikimedia.org/T224576) [00:32:02] (03PS6) 10Dzahn: switch webproxy CNAMEs to new install servers [dns] - 10https://gerrit.wikimedia.org/r/569680 (https://phabricator.wikimedia.org/T224576) [00:34:19] (03PS7) 10Dzahn: switch webproxy CNAMEs to new install servers [dns] - 10https://gerrit.wikimedia.org/r/569680 (https://phabricator.wikimedia.org/T224576) [00:36:55] 10Operations, 10cloud-services-team (Kanban): Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3 - https://phabricator.wikimedia.org/T241719 (10Krenair) [00:47:17] (03CR) 10Krinkle: jobrunner: add simple HTTP check (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/576301 (https://phabricator.wikimedia.org/T243096) (owner: 10Hnowlan) [00:47:59] (03CR) 10Krinkle: "See also these two puppet patches, which seem to be doing a similar thing. Merge/close as needed?" [puppet] - 10https://gerrit.wikimedia.org/r/576301 (https://phabricator.wikimedia.org/T243096) (owner: 10Hnowlan) [00:48:50] (03PS3) 10Krinkle: Remove references to obsolete rpc/RunJobs.php endpoint [puppet] - 10https://gerrit.wikimedia.org/r/575392 (https://phabricator.wikimedia.org/T243096) (owner: 10Aaron Schulz) [00:50:01] (03PS4) 10Dzahn: install_server: switch TFTP servers in DHCP to new install servers [puppet] - 10https://gerrit.wikimedia.org/r/569684 (https://phabricator.wikimedia.org/T224576) [00:57:30] (03Abandoned) 10Dzahn: switch install_server and failover in hieradata/common.yaml [puppet] - 10https://gerrit.wikimedia.org/r/569687 (https://phabricator.wikimedia.org/T224576) (owner: 10Dzahn) [01:02:40] (03PS1) 10Dzahn: hiera/aptrepo: rename install_server variables to aptrepo_server [puppet] - 10https://gerrit.wikimedia.org/r/577701 (https://phabricator.wikimedia.org/T224576) [01:05:57] (03CR) 10Dzahn: [C: 03+1] "https://puppet-compiler.wmflabs.org/compiler1001/21328/" [puppet] - 10https://gerrit.wikimedia.org/r/577701 (https://phabricator.wikimedia.org/T224576) (owner: 10Dzahn) [01:11:34] (03PS2) 10Dzahn: hiera/aptrepo: rename install_server variables to aptrepo_server [puppet] - 10https://gerrit.wikimedia.org/r/577701 (https://phabricator.wikimedia.org/T224576) [01:18:53] (03PS2) 10Dzahn: DHCP: remove old install servers and use new servers as next-server [puppet] - 10https://gerrit.wikimedia.org/r/569686 (https://phabricator.wikimedia.org/T224576) [01:29:16] (03PS1) 10Reedy: Prune passed throttles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577704 [01:30:44] (03PS2) 10Reedy: Prune passed throttles and date sort leftovers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577704 [01:31:37] God some of these throttle rules don't make sense [01:31:48] 35 people expected [01:32:02] So we set a value of 50 on 2 /22s? [01:32:14] (03CR) 10Reedy: [C: 03+2] Prune passed throttles and date sort leftovers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577704 (owner: 10Reedy) [01:33:24] (03Merged) 10jenkins-bot: Prune passed throttles and date sort leftovers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577704 (owner: 10Reedy) [01:35:34] !log reedy@deploy1001 Synchronized wmf-config/throttle.php: tidy up (duration: 00m 56s) [01:35:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:50:34] (03PS1) 10Dzahn: site: reorganize mediawiki appserver structure [puppet] - 10https://gerrit.wikimedia.org/r/577705 [01:50:47] off [01:51:21] (03CR) 10Dzahn: [C: 04-2] site: reorganize mediawiki appserver structure [puppet] - 10https://gerrit.wikimedia.org/r/577705 (owner: 10Dzahn) [02:09:53] 10Operations, 10Performance-Team, 10serviceops, 10Patch-For-Review: Test gutter pool failover in production and memcached 1.5.x - https://phabricator.wikimedia.org/T240684 (10aaron) >>! In T240684#5928018, @jijiki wrote: > 1 minute could be ok if we our case was only short unavailabilities. One of the rea... [04:10:54] PROBLEM - mediawiki originals uploads -hourly- for eqiad on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe1005:9112 job=statsd_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [04:13:00] PROBLEM - mediawiki originals uploads -hourly- for codfw on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe2005:9112 job=statsd_exporter site=codfw https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [04:18:57] (03PS1) 10Alex Monk: Fix incorrect name of safe_service_restart parameter [puppet] - 10https://gerrit.wikimedia.org/r/577725 (https://phabricator.wikimedia.org/T247151) [06:31:40] RECOVERY - mediawiki originals uploads -hourly- for codfw on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [06:31:54] RECOVERY - mediawiki originals uploads -hourly- for eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [07:07:47] (03PS1) 10RhinosF1: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 [07:13:00] (03PS2) 10RhinosF1: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 [07:13:37] (03PS3) 10RhinosF1: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) [07:16:41] Reedy: That is going to be so close to the event [07:18:46] (03PS5) 10RhinosF1: Remove expired throttle config from throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577346 [07:21:07] Urbanecm, RoanKattouw, Niharika: Can one of you agree to deploy 577728 and run the script as indicated to purge cache on Monday EU Mid-Day SWAT? (That's 11:00-12:00 UTC) [07:24:07] Can someone also type "recheck" on the patch to trigger Jenkins CI? [07:26:14] PROBLEM - Old JVM GC check - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is CRITICAL: 120 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1 [07:44:38] PROBLEM - Check systemd state on db2085 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:54:12] Amir1: It looks like you're on EU Mid-Day SWAT. Please can you see above and advise? [08:12:34] RECOVERY - Check systemd state on db2085 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:22:52] PROBLEM - Check systemd state on db2084 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:32:42] (03PS1) 10Fomafix: Update EmacsWiki in Interwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577740 (https://phabricator.wikimedia.org/T227053) [09:45:01] (03PS1) 10Elukey: profile::swap: use nodejs 10 by default [puppet] - 10https://gerrit.wikimedia.org/r/577742 (https://phabricator.wikimedia.org/T247055) [09:52:54] (03PS2) 10Elukey: profile::swap: use nodejs 10 by default [puppet] - 10https://gerrit.wikimedia.org/r/577742 (https://phabricator.wikimedia.org/T247055) [09:56:23] (03PS3) 10Elukey: profile::swap: use nodejs 10 by default [puppet] - 10https://gerrit.wikimedia.org/r/577742 (https://phabricator.wikimedia.org/T247055) [09:59:30] RECOVERY - Old JVM GC check - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is OK: (C)100 gt (W)80 gt 75.25 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1 [10:09:03] (03PS4) 10Elukey: profile::swap: use nodejs 10 by default [puppet] - 10https://gerrit.wikimedia.org/r/577742 (https://phabricator.wikimedia.org/T247055) [10:16:05] (03PS1) 10Elukey: icinga::monitor::elasticsearch::old_jvm_gc_checks: fix grafana URL [puppet] - 10https://gerrit.wikimedia.org/r/577746 [10:32:58] (03PS2) 10Elukey: icinga::monitor::elasticsearch::old_jvm_gc_checks: fix grafana URL [puppet] - 10https://gerrit.wikimedia.org/r/577746 [10:39:46] (03CR) 10Elukey: "Gehel: feel free to -2 this if it doesn't make sense, but I noticed that the icinga link for https://grafana.wikimedia.org/d/000000462/ela" [puppet] - 10https://gerrit.wikimedia.org/r/577746 (owner: 10Elukey) [10:44:09] (03PS5) 10Elukey: profile::swap: use nodejs 10 by default [puppet] - 10https://gerrit.wikimedia.org/r/577742 (https://phabricator.wikimedia.org/T247055) [10:51:01] Reedy: replied [10:54:22] (03CR) 10Zoranzoki21: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [10:56:02] (03CR) 10jerkins-bot: [V: 04-1] Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [10:56:59] (03PS6) 10Elukey: profile::swap: use nodejs 10 by default [puppet] - 10https://gerrit.wikimedia.org/r/577742 (https://phabricator.wikimedia.org/T247055) [11:04:10] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1003/21336/" [puppet] - 10https://gerrit.wikimedia.org/r/577742 (https://phabricator.wikimedia.org/T247055) (owner: 10Elukey) [11:11:24] (03PS4) 10Reedy: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [11:12:37] (03CR) 10jerkins-bot: [V: 04-1] Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [11:18:05] (03PS5) 10Reedy: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [11:19:10] (03CR) 10jerkins-bot: [V: 04-1] Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [11:22:29] (03PS6) 10Reedy: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [11:23:30] (03CR) 10jerkins-bot: [V: 04-1] Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [11:25:18] (03PS7) 10Reedy: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [11:36:10] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3052 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [11:45:20] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3052 is OK: HTTP OK: HTTP/1.0 200 OK - 22058 bytes in 3.281 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [12:28:06] (03CR) 10Reedy: [C: 03+2] Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [12:29:05] (03Merged) 10jenkins-bot: Add Throttle Rule for Senior Citizens Write Wikipedia Course - 2020-03-09 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577728 (https://phabricator.wikimedia.org/T247149) (owner: 10RhinosF1) [12:33:02] RECOVERY - Check systemd state on db2084 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:43:57] Reedy: have you sync’d file and cleared the cache [12:44:05] No [12:44:22] Reedy: can you or set a slot for it? [12:44:42] Yes [12:44:50] Otherwise someone will shout at me for merging a patch and not deploying it [12:47:43] Reedy: pls :) [12:48:06] !log reedy@deploy1001 Synchronized wmf-config/throttle.php: T247149 (duration: 01m 07s) [12:48:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:48:12] T247149: Throttle rule for 2020-03-09 - Senior Citizens Write Wikipedia course - https://phabricator.wikimedia.org/T247149 [13:16:04] Reedy: has cache been cleared as it’s less than 72 hrs before per task/instructions [13:16:17] Yes [13:16:25] Just saw task [13:16:30] Which is why I wrote a script, posted the comment with hte script and closed the task ;) [13:16:52] Thanks for deploying given it’s your day off and out of the windows [13:19:11] Reedy: don’t merge but can you force https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/577346/ to think there’s no conflict. By the time that’s deployed it’ll be safe [13:20:32] It needs rebasing [13:20:37] I removed most of those yesterday [13:21:18] Reedy: gerrit won’t auto rebase though [13:21:26] Indeed [13:21:29] jgit is crap [13:21:45] Can you as im mobile? [13:22:08] I imagine with the other patch landing, it'll need a manual rebase anyway [13:22:44] Okay, It will be safe when I’ve scheduled that for deploy to be exactly the same [13:24:09] Your patch is a no-op now [13:24:47] or is there one to remove.. [13:26:16] (03PS1) 10Elukey: profile::oozie::server: allow to set the sharedlibpath [puppet] - 10https://gerrit.wikimedia.org/r/577771 (https://phabricator.wikimedia.org/T244499) [13:28:48] (03PS6) 10Reedy: Remove expired throttle config from throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577346 (owner: 10RhinosF1) [13:36:15] Reedy: there’s 2 as the one you just merged will have expirrd [13:36:29] I’ll re-remove that in a sec [13:36:33] Not at 5PM UTC it won't [13:36:40] Your rule is to 5:01PM UTC [13:38:40] Reedy: swat isn’t 5pm UTC. It’s 6pm on Monday [13:38:47] Sure [13:38:50] But read your commit summary [13:39:06] I’ll fix that with it [13:40:35] (03PS2) 10Elukey: profile::oozie::server: allow to set the sharedlibpath [puppet] - 10https://gerrit.wikimedia.org/r/577771 (https://phabricator.wikimedia.org/T244499) [13:44:28] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1002/21338/" [puppet] - 10https://gerrit.wikimedia.org/r/577771 (https://phabricator.wikimedia.org/T244499) (owner: 10Elukey) [13:44:34] (03CR) 10Elukey: [C: 03+2] profile::oozie::server: allow to set the sharedlibpath [puppet] - 10https://gerrit.wikimedia.org/r/577771 (https://phabricator.wikimedia.org/T244499) (owner: 10Elukey) [13:47:16] (03PS7) 10RhinosF1: Remove expired throttle config from throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577346 [13:47:41] Reedy: kick jenkins pls “recheck” [13:48:10] We should just whitelist you [13:48:21] Reedy: or that [13:52:58] done [13:54:22] (03CR) 10RhinosF1: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577346 (owner: 10RhinosF1) [13:55:04] Reedy: Thanks, I’ve manually kicked it myself this time and from now it *should* like me [13:55:24] (03CR) 10jerkins-bot: [V: 04-1] Remove expired throttle config from throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577346 (owner: 10RhinosF1) [13:57:07] (03PS8) 10RhinosF1: Remove expired throttle config from throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577346 [13:57:38] I say CI should like me, it’s me :) [13:57:47] It never does [14:11:02] write valid php then ;) [14:12:16] I write valid code, It’s style checks that always fail. I need to learn code style. I’ve self taught and done a lot for myself, for fun so it’s never got into me [14:12:33] RhinosF1: fwiw it is possible to run phpcs locally [14:13:30] cdanis: i used the web editor for that but i’ll remember [14:13:44] ah ok [14:15:58] RhinosF1: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/577728/2/wmf-config/throttle.php was not valid PHP :) [14:16:09] Two syntax errors, never mind phpcs issues [14:17:46] Reedy: that’s called alseep me [14:18:12] ;) [14:20:02] * RhinosF1 had a tiring week. Doing PHP at 7am on a saturday was garunteed to be messy [15:53:02] 10Operations, 10ops-eqiad, 10serviceops: (Need by: TBD) rack/setup/install kubernetes10[07-14].eqiad.wmnet - https://phabricator.wikimedia.org/T241850 (10Jclark-ctr) a:05Jclark-ctr→03Christopher name rack_name position. Switchport kubernetes1007 A3 8 16 kubernetes1008 A5 17 20 kuberne... [15:53:41] 10Operations, 10ops-eqiad, 10serviceops: (Need by: TBD) rack/setup/install kubernetes10[07-14].eqiad.wmnet - https://phabricator.wikimedia.org/T241850 (10Jclark-ctr) [16:30:32] PROBLEM - IPMI Sensor Status on mw1373 is CRITICAL: Sensor Type(s) Temperature, Power_Supply Status: Critical [Status = Critical, PS Redundancy = Critical] https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Power_Supply_Failures [16:34:18] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [16:34:20] RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 242, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [17:32:28] PROBLEM - mediawiki originals uploads -hourly- for codfw on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe2005:9112 job=statsd_exporter site=codfw https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [17:32:30] PROBLEM - mediawiki originals uploads -hourly- for eqiad on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe1005:9112 job=statsd_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [17:41:03] 10Operations, 10ops-eqiad, 10DC-Ops: (Need by: ASAP) rack/setup/install stat1008 - https://phabricator.wikimedia.org/T246472 (10Jclark-ctr) [17:43:23] 10Operations, 10ops-eqiad, 10DC-Ops: (Need by: ASAP) rack/setup/install stat1008 - https://phabricator.wikimedia.org/T246472 (10Jclark-ctr) {F31670032} {F31670033} Installed GPU no problems server will only fit 1 more GPU. [17:45:07] 10Operations, 10ops-eqiad, 10DC-Ops: (Need by: ASAP) rack/setup/install stat1008 - https://phabricator.wikimedia.org/T246472 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson Racked and cabled handing over to Chris for configuration Rack a6 unit 17. switchport 22 [19:08:18] Is https://phabricator.wikimedia.org/T247172 acceptable for SWAT? Or do I have to sync with train? [19:09:38] It can't go with the train [19:09:42] It's not a MW core change [19:09:46] It's a site request [19:13:29] Ok [19:13:44] Good point [19:15:15] Complex changes going in train obviously would only apply to code not config [19:17:04] * RhinosF1 keeps getting timeouts trying to view the settings file to be confident about the change [19:22:09] view it how? [19:25:23] Reedy: gtiles in gerrit [19:25:30] It loaded eventually [19:25:37] initialisesettings is huge [19:26:30] That’s why. It loaded. I might have to make this change locally later rather than via the web. [19:33:39] (03PS1) 10RhinosF1: Create Define/Define Talk: namespace on scowiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577802 [19:40:51] (03PS2) 10RhinosF1: Create Define/Define Talk: namespace on scowiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577802 (https://phabricator.wikimedia.org/T247172) [19:41:25] (03CR) 10RhinosF1: [C: 04-1] "Will sort tabs out myself later, doing mobile before I forget what I looked at" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577802 (https://phabricator.wikimedia.org/T247172) (owner: 10RhinosF1) [19:42:35] (03CR) 10jerkins-bot: [V: 04-1] Create Define/Define Talk: namespace on scowiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/577802 (https://phabricator.wikimedia.org/T247172) (owner: 10RhinosF1) [20:11:22] PROBLEM - mediawiki originals uploads -hourly- for eqiad on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe1005:9112 job=statsd_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [20:11:22] PROBLEM - mediawiki originals uploads -hourly- for codfw on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe2005:9112 job=statsd_exporter site=codfw https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [20:18:32] Reedy: do we set the wikit:sco: interwiki or can stewards override that? [20:18:47] See 2nd point on the task [20:19:38] I don't think stewards can [20:19:57] Do we do local interwikis? [20:21:12] Reedy: i’m not entirely sure what’s handling it. The wikt:sco interwiki they want to go the new namespace for now but I’m not sure whether it can be set in Special:Interwiki or whether we have a config file doing it [20:21:38] Can't be done with Special:Interwiki [20:21:45] And we have a config file, but that's auto generated [20:22:00] And there's no way to have wiki specific values [20:22:30] Reedy: so is that declined then? [20:23:05] Oh, unless $wgLocalInterwikis ? [20:23:34] but that's to another wiki [20:23:39] Not a namespace ont he current wiki [20:24:11] Well this would be [20:24:31] I don't think MW supports that [20:25:09] They want wikt:sco globally to point to sco.wiki/Define: [20:26:23] wikit.sco: would go to sco.wikipedia.org/wiki/Define: [20:26:46] Yes, you don't need to keep stating it [20:26:51] I don't know if https://meta.wikimedia.org/wiki/Interwiki_map will support : [20:27:16] Although not sure that’s best as well as in theory you’d assume that wikt:sco:Talk would go the talk namespace of define but that definately doesn’t work [20:27:40] I'm not sure why they need/want a global interwiki for it [20:28:20] Because it’s supposed to act as a replacement for not having a wiktionary in scots is this namespace [20:28:35] As they aren’t big enough for new wikis [20:28:35] And? [20:28:49] It's not something we support [20:29:03] So they want the interwiki that would point there to go to that namespace [20:29:14] Anyway, it doesn’t work so I’ll decline that [20:29:32] Heh [20:29:33] >Editing administrators should remember that the prefix (first column) cannot contain spaces ( ), colons (:), ampersands (&), or equal signs (=). They should avoid likely conflict with languages (en:, fr:, etc). Wikimedia Foundation projects shouldn't be added here, since they are added automatically. Unlike the first column suggests, none of the letters in the prefixes are case sensitive. [20:31:14] * RhinosF1 has crossed it off the task [20:49:45] Reedy: if import/export doesn’t timeout then it should work. I might see if Urbanecm can try during the during the window. [20:50:23] You'd have to ask someone like jeblad how well it works [20:50:37] The local admins can do the import themselves though [20:50:50] So really shouldn't need other people to do it [20:50:50] Ok [20:51:41] I'm...honestly not sure this (the task itself) is a good idea. When there will be sco.wikt, will we move the namespace there? idk... [20:52:36] Will there ever be? [20:52:39] Any time soon? :) [20:52:54] Possibly and no [20:53:20] Urbanecm: that’d be a choice between sco.wiki and the new sco.wikt [20:53:30] Reedy: idk - up to the langcom IG [20:54:32] Urbanecm: someone asked during the RfC and MJL said: [20:54:48] Wiktionary isn't really a "project" as much as it is a set of projects. We have English Wiktionary and Simple English Wiktionary, yes. Would there ever be a Scots Wiktionary? It is highly unlikely. It'd be the third (and smallest) Anglic language Wiktionary and would require we split our time in half. [20:54:48] There's also a lot more already established oddities if you look for them. We have a news section despite not being a newspaper, and we have a portal system despite not being a directory. Also, English Wikisource currently serves as the repository for all texts in Scots despite the two being separate languages. Things aren't cut and dry. [20:55:59] well, creating the namespace would also require someone working at content there and someone at enwikt/simplewikt. The news argument is compelling, through. [20:56:10] once again, I'm really not sure on this one - since i was mentioned, just wanted to comment on that [20:58:00] It’s a nice idea and I fully get the rationale behind it. It’s probably better than creating a wiki to be barely used and cleaner than incubator. I imagine if it was there on wiki it might be more encouraging to work on and it’s one admin team etc.. doing everything so less work [20:58:29] One vandal hits two wikis means twice the work. They hit only one and there’s half the work. Half the areas to monitor etc.. [20:58:39] i see [21:10:25] Reedy, Urbanecm: see the task, looks like you’re not the only ones wondering [22:01:38] RECOVERY - mediawiki originals uploads -hourly- for eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [22:01:38] RECOVERY - mediawiki originals uploads -hourly- for codfw on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [22:11:42] PROBLEM - Old JVM GC check - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is CRITICAL: 143.5 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1 [22:39:20] PROBLEM - Old JVM GC check - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 109 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1 [23:00:24] PROBLEM - Old JVM GC check - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 100.3 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1 [23:08:36] (03PS1) 10BryanDavis: Remove legacy cluster images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/577818 [23:13:58] (03CR) 10BryanDavis: [C: 03+1] "One note inline about an unrelated change needed for this file that I realized because of this patch. :)" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/577279 (https://phabricator.wikimedia.org/T246689) (owner: 10Bstorm) [23:35:44] RECOVERY - Old JVM GC check - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is OK: (C)100 gt (W)80 gt 76.27 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1