[00:00:04] <jouncebot>	 RoanKattouw, Niharika, and Urbanecm: (Dis)respected human, time to deploy Evening backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20201113T0000). Please do the needful.
[00:00:04] <jouncebot>	 No GERRIT patches in the queue for this window AFAICS.
[01:30:42] <TimStarling>	 !log on mwmaint1002 running fixT260485.php unmerged fixup script from https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMaintenance/+/640348
[01:30:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[01:35:07] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for fkaelin - https://phabricator.wikimedia.org/T267816 (10fkaelin)
[01:35:23] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to researchers for fkaelin - https://phabricator.wikimedia.org/T267817 (10fkaelin)
[01:46:28] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for fkaelin - https://phabricator.wikimedia.org/T267816 (10fkaelin) @Ottomata and @leila, I think your approvals are needed for this review, thank you!
[01:46:41] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to researchers for fkaelin - https://phabricator.wikimedia.org/T267817 (10fkaelin) @Ottomata and @leila, I think your approvals are needed for this review, thank you!
[01:49:50] <icinga-wm>	 PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[02:03:19] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to researchers for fkaelin - https://phabricator.wikimedia.org/T267817 (10leila) (omg! one of your first phab tasks!:)  Approved.
[02:03:47] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for fkaelin - https://phabricator.wikimedia.org/T267816 (10leila) approved. thanks!
[06:29:02] <icinga-wm>	 PROBLEM - ores on ores2001 is CRITICAL: connect to address 10.192.0.12 and port 8081: Connection refused https://wikitech.wikimedia.org/wiki/Services/Monitoring/ores
[06:42:22] <icinga-wm>	 RECOVERY - ores on ores2001 is OK: HTTP OK: HTTP/1.0 200 OK - 6397 bytes in 0.080 second response time https://wikitech.wikimedia.org/wiki/Services/Monitoring/ores
[07:35:26] <icinga-wm>	 PROBLEM - puppet last run on an-tool1009 is CRITICAL: CRITICAL: Puppet last ran 2 days ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun
[07:39:04] <wikibugs>	 (03PS1) 10Elukey: profile::prometheus::analytics: change target class for hive [puppet] - 10https://gerrit.wikimedia.org/r/640914
[07:41:10] <icinga-wm>	 RECOVERY - puppet last run on an-tool1009 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun
[07:45:12] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] profile::prometheus::analytics: change target class for hive [puppet] - 10https://gerrit.wikimedia.org/r/640914 (owner: 10Elukey)
[07:49:44] <icinga-wm>	 RECOVERY - Check systemd state on deneb is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[07:51:18] <wikibugs>	 (03PS1) 10Elukey: profile::prometheus::analytics: remove hive_test target [puppet] - 10https://gerrit.wikimedia.org/r/640915
[07:52:54] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] profile::prometheus::analytics: remove hive_test target [puppet] - 10https://gerrit.wikimedia.org/r/640915 (owner: 10Elukey)
[07:54:34] <wikibugs>	 10Operations, 10MediaWiki-Parser: Varnish 503 errors on page with large number of flag icons. - https://phabricator.wikimedia.org/T267804 (10Joe) p:05High→03Low From the diuscussion on VP:T I would assume the editor just hit the maximum amount of templates to include in a single article. Lowering priority...
[08:00:04] <jouncebot>	 Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20201113T0800)
[08:00:46] <icinga-wm>	 PROBLEM - HTTPS on apt1001 is CRITICAL: SSL CRITICAL - OCSP staple validity for apt.wikimedia.org has 86356 seconds left https://wikitech.wikimedia.org/wiki/APT_repository
[08:05:39] <_joe_>	 uhh
[08:11:00] <_joe_>	 anyone with better knowledge of acmechief than me around?
[08:12:09] <vgutierrez>	 morning _joe_
[08:12:13] <vgutierrez>	 let me check that
[08:12:14] <_joe_>	 vgutierrez: hi
[08:12:29] <vgutierrez>	 after I reach for my yubikey... :)
[08:12:35] <_joe_>	 ahah do you have an highlight on "acmechief" that sends a bat-signal on your roof?
[08:12:43] <vgutierrez>	 indeed
[08:12:52] <_joe_>	 vgutierrez: I'm happy to learn more though
[08:12:53] <vgutierrez>	 yet another way of pinging me
[08:13:29] <_joe_>	 so I guess the ocsp stapling list is fetched from LE
[08:14:51] <_joe_>	 vgutierrez: for instance, where should I look for relevant logs?
[08:15:22] <vgutierrez>	 Nov 13 07:00:01 acmechief1001 systemd[1]: Reloaded acme-chief Service.
[08:15:22] <vgutierrez>	 Nov 13 08:00:01 acmechief1001 systemd[1]: Reloading acme-chief Service.
[08:15:22] <vgutierrez>	 Nov 13 08:00:01 acmechief1001 systemd[1]: Reloaded acme-chief Service.
[08:15:34] <vgutierrez>	 at some point acme-chief stops reloading properly
[08:15:35] <vgutierrez>	 :/
[08:15:42] <vgutierrez>	 that's journalctl -u acme-chief -f on acmechief1001
[08:15:48] <vgutierrez>	 !log restart acme-chief on acmechief1001
[08:16:02] <_joe_>	 oh it doesn't have a separate file?
[08:16:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:16:19] <vgutierrez>	 nope, in systemd we trust
[08:16:25] <_joe_>	 because I found acme-chief-certs-sync has it 
[08:16:39] <_joe_>	 and its' mostly logging that /nonexistent is a non existing directory
[08:16:41] <vgutierrez>	 that's the tiny script for syncing them across H/A nodes
[08:16:47] <vgutierrez>	 *HA
[08:16:52] <vgutierrez>	 so acmechief1001 and acmechief2001
[08:17:00] <_joe_>	 yeah I got that
[08:17:21] <vgutierrez>	 vgutierrez@acmechief1001:~$ sudo -i journalctl -u acme-chief --since "08:00" |grep -i apt
[08:17:21] <vgutierrez>	 Nov 13 08:15:57 acmechief1001 acme-chief-backend[2115]: Refreshing live OCSP response for certificate apt / ec-prime256v1
[08:17:21] <vgutierrez>	 Nov 13 08:15:57 acmechief1001 acme-chief-backend[2115]: live OCSP response refreshed successfully for apt / ec-prime256v1
[08:17:21] <vgutierrez>	 Nov 13 08:15:57 acmechief1001 acme-chief-backend[2115]: Refreshing live OCSP response for certificate apt / rsa-2048
[08:17:21] <vgutierrez>	 Nov 13 08:15:57 acmechief1001 acme-chief-backend[2115]: live OCSP response refreshed successfully for apt / rsa-2048
[08:17:36] <vgutierrez>	 fixed.. next puppet on aptXXXX nodes will refresh the OCSP response
[08:17:46] <_joe_>	 yes
[08:17:56] <_joe_>	 I am already running it
[08:18:00] <vgutierrez>	 thx <3
[08:19:32] <icinga-wm>	 RECOVERY - HTTPS on apt1001 is OK: SSL OK - OCSP staple validity for apt.wikimedia.org has 603629 seconds left:Certificate apt.wikimedia.org valid until 2020-12-16 07:00:14 +0000 (expires in 32 days) https://wikitech.wikimedia.org/wiki/APT_repository
[08:20:08] <wikibugs>	 (03CR) 10Matthias Mullie: Generation of json dumps for wikimedia commons (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/629121 (https://phabricator.wikimedia.org/T259067) (owner: 10Cparle)
[08:20:57] <wikibugs>	 10Operations, 10Traffic, 10Wikispore, 10HTTPS: Make Wikispore HTTPS-only - https://phabricator.wikimedia.org/T260701 (10Tgr) I think this is as easy as enabling `$wgForceHTTPS` and maybe setting a HSTS header.
[08:33:36] <icinga-wm>	 RECOVERY - HTTPS non-canonical-redirect-4 on ncredir3002 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 347185 seconds left:Certificate *.wikispecies.net valid until 2020-12-22 08:04:01 +0000 (expires in 38 days) https://wikitech.wikimedia.org/wiki/Ncredir
[08:47:17] <icinga-wm>	 ACKNOWLEDGEMENT - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. ayounsi https://phabricator.wikimedia.org/T253833 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[08:47:17] <icinga-wm>	 ACKNOWLEDGEMENT - Check systemd state on netbox2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. ayounsi https://phabricator.wikimedia.org/T253833 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[08:51:24] <XioNoX>	 _joe_: no more netbox alerts until monday morning
[08:51:40] <wikibugs>	 10Operations, 10Traffic, 10Wikispore, 10HTTPS: Make Wikispore HTTPS-only - https://phabricator.wikimedia.org/T260701 (10Zblace) I know tech people will not love this question, but access, sustainability and low-resource enthusiasts in me would love to ask:  What are the urgencies in pushing for HTTPS-only...
[08:54:39] <_joe_>	 XioNoX: <3
[08:55:39] <wikibugs>	 (03PS1) 10ArielGlenn: update pagerange so it can be used with revinfo input [dumps] - 10https://gerrit.wikimedia.org/r/640920
[08:55:57] <wikibugs>	 (03PS1) 10ArielGlenn: only generate revinfo files during production of page meta history files [dumps] - 10https://gerrit.wikimedia.org/r/640921 (https://phabricator.wikimedia.org/T263319)
[08:59:30] <wikibugs>	 (03PS2) 10ArielGlenn: option for text pass fixup script to write files to specified directory [dumps] - 10https://gerrit.wikimedia.org/r/640420
[09:51:03] <wikibugs>	 (03PS2) 10Hashar: gerrit: remove obsolete profile::gerrit::java_version [puppet] - 10https://gerrit.wikimedia.org/r/639272
[09:51:24] <wikibugs>	 (03CR) 10Hashar: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/639272 (owner: 10Hashar)
[10:00:11] <wikibugs>	 (03CR) 10Hashar: [V: 03+1] "No op for Gerrit https://puppet-compiler.wmflabs.org/compiler1003/617/ :)" [puppet] - 10https://gerrit.wikimedia.org/r/639272 (owner: 10Hashar)
[10:03:51] <wikibugs>	 10Operations, 10netops: an-worker1113 not in librenms and doesn't show up on juno's interface description - https://phabricator.wikimedia.org/T267827 (10elukey)
[10:08:00] <wikibugs>	 (03PS2) 10Hashar: gerrit: move java config from role to profile [puppet] - 10https://gerrit.wikimedia.org/r/639273
[10:17:11] <wikibugs>	 10Operations, 10ops-eqiad: an-worker1113 not in librenms and doesn't show up on juno's interface description - https://phabricator.wikimedia.org/T267827 (10ayounsi) ` an-worker1113:~$ sudo lldpcli [lldpcli] # show neighbors  SysName:      asw2-d-eqiad PortID:       ifname xe-4/0/18 PortDescr:    an-worker1114...
[10:23:31] <wikibugs>	 (03PS5) 10Giuseppe Lavagetto: Special docroot for thankyou.wikipedia.org [puppet] - 10https://gerrit.wikimedia.org/r/639771 (https://phabricator.wikimedia.org/T259312) (owner: 10Ejegg)
[10:23:33] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: jobrunner: add SERVERGROUP environment variable [puppet] - 10https://gerrit.wikimedia.org/r/640923 (https://phabricator.wikimedia.org/T266515)
[10:25:48] <wikibugs>	 10Operations, 10ops-eqiad: an-worker1113 not in librenms and doesn't show up on juno's interface description - https://phabricator.wikimedia.org/T267827 (10Peachey88)
[10:28:36] <wikibugs>	 10Operations, 10Beta-Cluster-Infrastructure, 10Traffic, 10User-Ryasmeen: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar)
[10:30:08] <icinga-wm>	 RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[10:57:54] <wikibugs>	 (03CR) 10Ejegg: "Thanks for the fixes @joe! Sorry I didn't get to those sooner." [puppet] - 10https://gerrit.wikimedia.org/r/639771 (https://phabricator.wikimedia.org/T259312) (owner: 10Ejegg)
[11:01:49] <wikibugs>	 (03CR) 10Awight: [C: 03+1] "Haven't tested, but it looks right!" [puppet] - 10https://gerrit.wikimedia.org/r/638565 (https://phabricator.wikimedia.org/T267028) (owner: 10Hashar)
[11:06:21] <_joe_>	 ejegg|away: I won't merge the change until monday, but I had to do another apache change now, so I told myself "why not"
[11:17:40] <wikibugs>	 (03PS4) 10Lucas Werkmeister (WMDE): Remove migration settings in Wikibase.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/631431 (https://phabricator.wikimedia.org/T264286) (owner: 10Tobias Andersson)
[11:17:42] <wikibugs>	 (03PS2) 10Lucas Werkmeister (WMDE): Remove migration settings in InitialiseSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/631496 (https://phabricator.wikimedia.org/T264286) (owner: 10Tobias Andersson)
[11:19:17] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] Remove migration settings in Wikibase.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/631431 (https://phabricator.wikimedia.org/T264286) (owner: 10Tobias Andersson)
[11:19:23] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] Remove migration settings in InitialiseSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/631496 (https://phabricator.wikimedia.org/T264286) (owner: 10Tobias Andersson)
[11:50:25] <wikibugs>	 10Operations, 10TechCom, 10serviceops, 10Performance Issue, and 3 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10daniel)
[11:58:30] <wikibugs>	 10Operations, 10netops: No Juniper alarms in SNMP for MX204 - https://phabricator.wikimedia.org/T241105 (10ayounsi) At least Alarms are spamming us in Rancid, so hard to miss :)
[11:59:43] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: profile::lvs::realserver: use poolcounter for guarding service restarts [puppet] - 10https://gerrit.wikimedia.org/r/640928 (https://phabricator.wikimedia.org/T266055)
[12:06:46] <wikibugs>	 10Operations, 10ops-eqiad: an-worker1113 not in librenms and doesn't show up on juno's interface description - https://phabricator.wikimedia.org/T267827 (10jijiki) p:05Triage→03Medium
[12:07:04] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to researchers for fkaelin - https://phabricator.wikimedia.org/T267817 (10jijiki) p:05Triage→03Medium
[12:07:15] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for fkaelin - https://phabricator.wikimedia.org/T267816 (10jijiki) p:05Triage→03Medium
[12:07:49] <wikibugs>	 10Operations, 10User-DannyS712: Access to security IRC channel for DannyS712 - https://phabricator.wikimedia.org/T267800 (10jijiki) p:05Triage→03Low
[12:10:11] <wikibugs>	 10Operations: wmf_auto_restart_{jenkins,rsync} failing on releases2002 - https://phabricator.wikimedia.org/T267795 (10jijiki) p:05Triage→03Medium
[12:13:19] <wikibugs>	 10Operations, 10LDAP-Access-Requests: LDAP access for Jan Jaquemot - https://phabricator.wikimedia.org/T267771 (10jijiki) p:05Triage→03Medium
[12:13:55] <wikibugs>	 10Operations, 10LDAP-Access-Requests: LDAP access for Till Mletzko - https://phabricator.wikimedia.org/T267744 (10jijiki) p:05Triage→03Medium
[12:36:05] <wikibugs>	 (03PS1) 10Matthias Mullie: Add Media Search survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/640934 (https://phabricator.wikimedia.org/T258419)
[13:14:19] <wikibugs>	 10Operations, 10User-DannyS712: Access to #mediawiki_security IRC channel for DannyS712 - https://phabricator.wikimedia.org/T267800 (10Urbanecm) See {T233235} and {T239093} for past such requests. See T239093#5804013 for info how to do this.
[13:14:22] <wikibugs>	 10Operations, 10User-DannyS712: Access to #mediawiki_security IRC channel for DannyS712 - https://phabricator.wikimedia.org/T267800 (10Urbanecm)
[15:14:59] <icinga-wm>	 PROBLEM - Disk space on centrallog1001 is CRITICAL: DISK CRITICAL - free space: / 1747 MB (3% inode=97%): /tmp 1747 MB (3% inode=97%): /var/tmp 1747 MB (3% inode=97%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=centrallog1001&var-datasource=eqiad+prometheus/ops
[16:38:44] <_joe_>	 cdanis: around?
[16:38:58] <cdanis>	 I would really rather not be
[16:39:00] <cdanis>	 what's up?
[16:39:27] <_joe_>	 the nel exporter is causing the disk on centrallog to fill up it seems
[16:39:33] <cdanis>	 looking
[16:39:45] <_joe_>	 prometheus-node-exporter[763]: outs_by_asn_created label
[16:39:53] <_joe_>	 lots of that
[16:40:03] <_joe_>	 I am ok with you just telling me where to look :)
[16:41:55] <cdanis>	 I've stopped the exporters and removed their files from /var/lib/prometheus/node.d
[16:42:22] <_joe_>	 I'll see what I can do with the log
[16:42:31] <cdanis>	 that's an incredible amount of logspam
[16:42:41] <_joe_>	 yeah :/
[16:42:44] <cdanis>	 from node-exporter
[16:42:56] <_joe_>	 and thanks, please go back to your day off
[16:43:56] <cdanis>	 I guess node-exporter doesn't like it when you have one metric name defined in multiple files?
[16:43:58] <cdanis>	 sigh
[16:44:04] <_joe_>	 yeah looks like it
[16:44:09] <_joe_>	 very stupid logs too
[16:44:38] <cdanis>	 it's a horrible error message, and I wrote C++ professionally for years :P
[16:44:45] <_joe_>	 ahah
[16:50:26] <_joe_>	 !log manually rotate user.log on centrallog1001 and moved it to /srv/user.log.manual-rotation
[16:50:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:54:14] <cdanis>	 _joe_: 'syslog' and 'messages' are both very large too
[16:54:19] <cdanis>	 :/ sorry
[16:54:25] <_joe_>	 cdanis: np
[16:54:37] <_joe_>	 user.log is the largest one
[17:01:26] <_joe_>	 the check should recover soon
[17:07:37] <icinga-wm>	 PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=atlas_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[17:08:41] <icinga-wm>	 PROBLEM - Host mr1-eqiad.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100%
[17:09:17] <icinga-wm>	 RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[17:14:23] <icinga-wm>	 RECOVERY - Host mr1-eqiad.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 2.26 ms
[17:18:55] <icinga-wm>	 RECOVERY - Disk space on centrallog1001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=centrallog1001&var-datasource=eqiad+prometheus/ops
[18:13:45] <icinga-wm>	 PROBLEM - Maps tiles generation on alert1001 is CRITICAL: CRITICAL: 100.00% of data under the critical threshold [5.0] https://wikitech.wikimedia.org/wiki/Maps/Runbook https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1
[18:26:37] <icinga-wm>	 PROBLEM - Ensure hosts are not performing a change on every puppet run on puppetdb1002 is CRITICAL: CRITICAL: the following (5) node(s) change every puppet run: ms-be2031.codfw.wmnet, puppetmaster1001.eqiad.wmnet, peek2001.codfw.wmnet, deploy1002.eqiad.wmnet, wdqs1009.eqiad.wmnet https://wikitech.wikimedia.org/wiki/Puppet%23check_puppet_run_changes
[20:57:03] <wikibugs>	 10Operations, 10MediaWiki-Parser: Varnish 503 errors on page with large number of flag icons. - https://phabricator.wikimedia.org/T267804 (10Aklapper) Hmm, going to the wikipage; selecting `Show Preview`, and then expanding `Parser profiling data` shows nothing close to limits.  (Same for looking at the `NewPP...
[21:28:28] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=enwiki epadmin # T105570
[21:28:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:28:36] <stashbot>	 T105570: Remove old user groups from Wikimedia wikis (afttest, afttest-hide, etc.) - https://phabricator.wikimedia.org/T105570
[21:32:30] <DannyS712>	 Urbanecm can you also run `mwscript maintenance/emptyUserGroup.php --wiki=aawiki communityapplica` ?
[21:32:35] <DannyS712>	 for the same task
[21:32:38] <Urbanecm>	 sure
[21:33:22] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=aawiki communityapplica # T105570
[21:33:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:33:40] <Urbanecm>	 I'm currently finding all the non-existing groups
[21:34:05] <DannyS712>	 oh, I just closed the task - I ran a search for all non-existing groups and those two were the only ones I found
[21:34:22] <Urbanecm>	 oh, good :)
[21:34:29] <Urbanecm>	 no problem, I'll just run it anyway if I find anything
[21:34:32] <Urbanecm>	 DannyS712: how did you do the search?
[21:34:45] <DannyS712>	 fake version of foreachwiki on toolforge
[21:35:08] <Urbanecm>	 mind sharing the code/link DannyS712 ?
[21:35:35] <DannyS712>	 while read wiki; do echo "SELECT GROUP_CONCAT(DISTINCT ug_group) AS 'groups' FROM user_groups WHERE ug_group NOT IN ('bot', 'sysop', 'interface-admin', 'bureaucrat', 'steward', 'accountcreator', 'import', 'transwiki', 'ipblock-exempt', 'oversight', 'confirmed', 'flow-bot', 'abusefilter', 'checkuser', 'autopatrolled', 'patroller',
[21:35:35] <DannyS712>	 'translationadmin', 'rollbacker')" | sql $wiki | sed "/^groups$/d" | sed "/^NULL$/d" | sed "/^ERROR 1044/d" | sed "s/^/$wiki\t/g"; done < <(curl https://noc.wikimedia.org/conf/dblists/all.dblist)
[21:35:54] <DannyS712>	 the ERROR 1044 part didn't work though
[21:36:14] <Urbanecm>	 because it's in std err
[21:36:30] <DannyS712>	 oh
[21:36:37] <Urbanecm>	 DannyS712: you'd have to redirect stderr to stdout for it to work properly
[21:36:47] <Urbanecm>	 sth like sql $wiki 2>&1 would work
[21:37:14] <DannyS712>	 okay, thanks
[21:37:22] <Urbanecm>	 np
[21:38:48] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=test2wiki upwizcampeditors # T105570
[21:38:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:38:56] <stashbot>	 T105570: Remove old user groups from Wikimedia wikis (afttest, afttest-hide, etc.) - https://phabricator.wikimedia.org/T105570
[21:39:29] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=testwiki flood # T105570
[21:39:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:40:17] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=bnwiki editor # T105570
[21:40:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:42:09] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=enwikinews reviewer # T105570
[21:42:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:42:37] <wikibugs>	 10Operations, 10Traffic, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Facenapalm) This issue is a thing in ruwiki as well. We have a user https://ru.wikipedia.org/wiki/Участник:Var_...
[22:04:36] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=testwiki editor # T105570
[22:04:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:04:43] <stashbot>	 T105570: Remove old user groups from Wikimedia wikis (afttest, afttest-hide, etc.) - https://phabricator.wikimedia.org/T105570
[22:06:01] <Urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=myvwiki autopatrolled # T105570
[22:06:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:55:59] <wikibugs>	 10Operations, 10MediaWiki-Parser: Varnish 503 errors on page with large number of flag icons. - https://phabricator.wikimedia.org/T267804 (10RoySmith) User:Davidwr reports on the VP:T thread that [[ https://en.wikipedia.org/w/index.php?title=List_of_shipwrecks_in_1837&action=edit&oldid=785040195 | this diff ]]...