[00:03:06] @JohnLewis, around? [00:04:13] JohnLewis, see DM. Will DM @NDKilla as well [01:02:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jt9nt [01:02:53] [02miraheze/puppet] 07paladox 03fe65d13 - puppetserver: Change hiera file to use more locations [01:02:55] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [01:02:56] [02puppet] 07paladox opened pull request 03#1660: puppetserver: Change hiera file to use more locations - 13https://git.io/Jt9nq [01:03:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-1/±0] 13https://git.io/Jt9nY [01:03:33] [02miraheze/puppet] 07paladox 0342a1457 - Rename modules/vmlib/lib/puppet/functions/expand_path.rb to modules/vmlib/lib/puppet/functions/vmlib/expand_path.rb [01:03:35] [02puppet] 07paladox synchronize pull request 03#1660: puppetserver: Change hiera file to use more locations - 13https://git.io/Jt9nq [01:03:40] [02puppet] 07paladox closed pull request 03#1659: puppetserver: Change hiera file to use more locations - 13https://git.io/Jt9Ob [01:03:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [01:03:43] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:27:44] Universal_Omega, I'm still seeing an average of 20-30 jobs backed up across multiple wikis. Mind running jobs on all wikis to clear the backlog? [01:36:10] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php on Jobrunner3 - very high backlog of outstanding jobs on alot of wikis. [01:36:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:58] thanks :) [01:39:05] dmehus: no problem... when there are 43 on meta, a few on test3, up to 70 on some wikis, and has been like that for an hour, this was definitely needed. [01:42:37] Universal_Omega, yeah, I had hunch it was needed on all wikis when I saw that many on Meta, `testwiki`, and a few other wikis [01:43:20] Not sure how long it'll take for the jobs to clear, though [02:15:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jt9WO [02:15:13] [02miraheze/puppet] 07paladox 0333ac802 - Update hiera.6.yaml [02:15:15] [02puppet] 07paladox synchronize pull request 03#1660: puppetserver: Change hiera file to use more locations - 13https://git.io/Jt9nq [02:43:58] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:44:19] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:23] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:24] RECOVERY - services3 Puppet on services3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:24] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:25] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:26] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:27] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:30] RECOVERY - services4 Puppet on services4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:44:31] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:44:32] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:37] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:42] RECOVERY - rdb3 Puppet on rdb3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:46] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:44:52] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:45:04] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:45:11] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:11] RECOVERY - gluster3 Puppet on gluster3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:13] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:45:13] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:18] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:45:25] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:25] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:27] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:27] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:29] RECOVERY - ldap2 Puppet on ldap2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:29] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:32] RECOVERY - dbbackup2 Puppet on dbbackup2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:40] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:46] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:49] RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:46:02] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:46:06] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [05:57:35] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.63, 3.05, 2.55 [05:59:31] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.27, 3.21, 2.67 [06:02:31] RECOVERY - db12 APT on db12 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [06:09:27] RECOVERY - db13 APT on db13 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [06:09:43] RECOVERY - cp12 APT on cp12 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:10:05] RECOVERY - mw10 APT on mw10 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:10:41] RECOVERY - test3 APT on test3 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:12:30] RECOVERY - cp11 APT on cp11 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:15:51] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [06:18:56] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:20:34] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 32 packages available for upgrade (0 critical updates). [06:21:46] RECOVERY - phab2 APT on phab2 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:23:48] RECOVERY - mem2 APT on mem2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:24:53] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.83, 3.22, 2.76 [06:24:55] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [06:25:49] RECOVERY - mail2 APT on mail2 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [06:26:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.65, 2.93, 2.70 [06:28:34] RECOVERY - mw8 APT on mw8 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:31:07] RECOVERY - cp3 APT on cp3 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [06:31:17] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [06:32:33] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:38:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.23, 3.17, 2.77 [06:39:51] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [06:40:03] RECOVERY - services4 APT on services4 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [06:40:08] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 97 packages available for upgrade (0 critical updates). [06:40:19] RECOVERY - cp10 APT on cp10 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:42:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.50, 3.14, 2.88 [06:43:58] RECOVERY - db11 APT on db11 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [06:46:24] RECOVERY - ns1 APT on ns1 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [06:49:59] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:50:29] RECOVERY - mon2 APT on mon2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:50:39] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [06:50:47] RECOVERY - services3 APT on services3 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [06:54:47] RECOVERY - mw11 APT on mw11 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:54:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.74, 3.02, 2.72 [06:56:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.15, 2.78, 2.68 [06:59:03] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [07:01:11] RECOVERY - rdb3 APT on rdb3 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [07:01:56] RECOVERY - ns2 APT on ns2 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [07:40:59] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.20, 3.26, 2.72 [07:46:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.74, 3.71, 3.12 [07:48:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.31, 3.23, 3.02 [08:00:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.87, 3.25, 2.94 [08:02:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.27, 2.86, 2.83 [08:26:51] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.74, 3.11, 2.66 [08:28:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.35, 2.80, 2.60 [08:31:52] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [08:33:46] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (1 critical updates). [08:34:47] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [08:35:10] PROBLEM - rdb3 APT on rdb3 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (1 critical updates). [08:35:16] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (1 critical updates). [08:36:02] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [08:36:06] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [08:36:29] PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (1 critical updates). [08:36:32] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (1 critical updates). [08:36:35] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [08:36:41] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [08:39:07] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (1 critical updates). [08:41:35] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 251.37 ms [08:42:08] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [08:44:21] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [08:44:36] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (1 critical updates). [08:44:48] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [08:48:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.10, 3.21, 2.59 [08:49:56] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (1 critical updates). [08:50:09] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 98 packages available for upgrade (1 critical updates). [08:50:36] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (1 critical updates). [08:50:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.98, 3.47, 2.77 [08:51:47] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:51:58] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [08:52:29] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [08:52:30] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:52:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 1.88, 2.82, 2.61 [08:53:01] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (1 critical updates). [08:54:10] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [08:55:01] PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (1 critical updates). [08:55:28] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [08:56:56] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (1 critical updates). [08:57:12] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki terribletvshowswiki --r "[[phab:T866|Requested]]" /home/reception/ttsdel.txt [08:57:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:27] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 23 packages available for upgrade (1 critical updates). [08:58:56] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (1 critical updates). [09:00:05] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [09:04:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.42, 3.37, 2.80 [09:06:50] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.98, 3.57, 2.95 [09:08:50] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.05, 3.37, 2.96 [09:55:44] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt9ww [09:55:46] [02miraheze/GlobalNewFiles] 07Universal-Omega 03ab3785e - Fix paths [09:56:35] miraheze/GlobalNewFiles - Universal-Omega the build passed. [14:29:55] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki kelevarwiki --overwrite [14:29:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:39] .op [15:29:39] Attempting to OP... [15:29:54] .deop [15:29:54] Attempting to OP... [15:37:08] RhinosF1, can you fix the extra space before "Status" next time you're opped? [15:37:23] .op [15:37:24] Attempting to OP... [15:37:32] ty [15:37:35] .deop [15:37:36] Attempting to OP... [16:26:31] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 1.82, 5.63, 3.37 [16:28:29] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.45, 3.91, 3.00 [16:30:26] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.45, 2.82, 2.71 [16:34:35] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.58, 3.59, 2.14 [16:36:38] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.49, 2.48, 1.90 [17:40:15] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki horribletvshowepisodeswiki --r "[[Phab:T6866|Requested]]" /home/reception/hhdel.txt [17:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:13:22] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jt9be [18:13:24] [02miraheze/ManageWiki] 07Universal-Omega 0398a8c34 - Add help message support and improve label to ManageWikiNamespacesAdditional [18:13:25] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [18:13:45] [02ManageWiki] 07Universal-Omega opened pull request 03#250: Add help message support and improve label to ManageWikiNamespacesAdditional - 13https://git.io/Jt9bU [18:14:52] miraheze/ManageWiki - Universal-Omega the build passed. [18:15:37] [02ManageWiki] 07Universal-Omega edited pull request 03#250: ManageWikiNamespacesAdditional: add help message support and improve label (T6878) - 13https://git.io/Jt9bU [18:17:23] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 16.19, 5.17, 2.14 [18:23:23] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.91, 3.55, 2.62 [18:25:22] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.43, 2.86, 2.47 [18:29:17] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki terribletvshowswiki [18:29:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:45:16] Hello, can i know how i can patrol edits globally? (for example LW or nonciclopediawiki) [18:48:46] paladox: any idea why this script doesn't work `sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki terribletvshowswiki --r "[[phab:T6866|Requested]]" /home/reception/ttsdel.txt`? [18:48:53] it says Deleted! but it's not actually working [18:49:03] nope :/ [18:49:45] maybe need to do -r [18:50:16] paladox: also that wiki keeps bringing tons of jobs: `"jobs": 642,` right now. Any ideas why about that? [18:50:18] i think --delete [18:50:30] oh ^^^ [18:50:31] i forget if that script needs it [18:50:48] i don't see the option here https://phabricator.wikimedia.org/source/mediawiki/browse/master/maintenance/deleteBatch.php$44 though [18:50:49] [ deleteBatch.php · mediawiki ] - phabricator.wikimedia.org [18:51:19] Universal_Omega hmm [18:51:52] ignore me, i found "problem" [18:52:05] 2021-02-21 18:51:57 deleteLinks File:German_Baywatch_Nights_Season_2_DVD.jpg pageId=20081 timestamp=20210221174411 namespace=6 title=German_Baywatch_Nights_Season_2_DVD.jpg requestId=7662ef9da17485516874076e (uuid=90cde73e0c554bf48f11cf14519cad5f,timestamp=1613929451) t=5 error=deleteLinks: Page #20081 exists [18:52:08] jobs failing [18:52:16] timing out? [18:52:18] Oh [18:52:28] Zppix don't think so [18:52:48] paladox: tried to run jobs and that seems to fail too [18:52:55] it runs them very quick but then there's still some left apparently [18:53:12] well it errors [18:53:44] https://github.com/wikimedia/mediawiki/blob/master/includes/jobqueue/jobs/DeleteLinksJob.php#L59 [18:53:45] [ mediawiki/DeleteLinksJob.php at master · wikimedia/mediawiki · GitHub ] - github.com [18:54:49] I wonder if adding an interval will help [18:56:12] Reception123. Maybe if the first files in script already exist? That seems to be the error. [18:57:01] oh [18:57:07] i see the issue [18:57:19] Reception123 why are there just File: namespaces without page titles? [18:57:44] i also see duplicates [18:57:51] paladox: it's because that's the only way I could copy them [18:57:55] from Special:UnusedFiles [18:58:55] it sounds like it'd be easier to nuke the NS [18:59:02] heh [18:59:35] are all the jbrs backlogged with jobs? if not try doing it on the one with least jobs [19:00:39] I tried on both [19:00:48] some reason its not deleting. [19:01:32] root@jobrunner3:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki terribletvshowswiki --r "[[phab:T6866|Requested]]" list [19:01:32] File:Card.png Deleted! [19:01:33] [a8660d48a028dbca5f0bc163] [no req] Wikimedia\Rdbms\DBTransactionError from line 1519 of /srv/mediawiki/w/includes/libs/rdbms/database/Database.php: Explicit transaction still active. A caller may have caught an error. Open transactions: LocalFile::lockingTransaction [19:01:33] Backtrace: [19:02:03] Universal_Omega i think you resolved this before ^ [19:02:09] paladox: sounds like a db issue [19:03:20] yeh [19:05:15] paladox: could there be two db entires or one incomplete entry for the file in the db preventing deletion? [19:05:33] links me to https://github.com/wikimedia/mediawiki/blob/87be54f903c46450ac726c597ba8e1f0bda2158e/includes/filerepo/file/LocalFile.php#L2407 [19:05:34] [ mediawiki/LocalFile.php at 87be54f903c46450ac726c597ba8e1f0bda2158e · wikimedia/mediawiki · GitHub ] - github.com [19:07:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jt9Au [19:07:09] [02miraheze/mw-config] 07paladox 03b72621f - Allow logging class LocalFile to make it through [19:07:10] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [19:07:12] [02mw-config] 07paladox opened pull request 03#3737: Allow logging class LocalFile to make it through - 13https://git.io/Jt9Az [19:08:11] miraheze/mw-config - paladox the build passed. [19:08:25] [02mw-config] 07paladox closed pull request 03#3737: Allow logging class LocalFile to make it through - 13https://git.io/Jt9Az [19:08:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt9Ag [19:08:28] could migration of files cause it paladox? maybe the permissions on the file are wrong [19:08:28] [02miraheze/mw-config] 07paladox 032e9e236 - Allow logging class LocalFile to make it through (#3737) [19:08:30] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [19:08:31] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [19:09:21] miraheze/mw-config - paladox the build passed. [19:11:39] paladox: this is different then what I did. Mine was my own script, I was able to fix it by throwing an exception that caused the transaction to rollback when it was stuck. [19:11:52] oh [19:12:16] i'm not sure how to fix this [19:13:50] is able to be manually deleted? [19:15:49] paladox: actually... this could be a similar issue to what happend on ATT actually, which I did fix. What is the file, and try manually deleting. If that fails I know why and can fix it, probably. [19:16:09] File:Card.png [19:16:10] Universal_Omega [19:16:19] though seems to be happening to all files in Reception123 list [19:17:13] paladox I know why. [19:17:18] oh? [19:19:15] paladox: I wouldn't run this unless sure it would work, which I can verify in a minute but, `DELETE FROM oldimage WHERE oi_archive_name='';` may work. [19:19:26] thanks! [19:20:21] No problem. [19:21:06] When you're finished, I have requested the maintenance bot to remove a bunch of useless user blogs on `crappygameswiki` as there a lot of them that needs cleaning up, and it would take forever for me to delete every last one.https://phabricator.miraheze.org/T6882 I [19:21:07] [ ⚓ T6882 Tons of unused user blogs that should get deleted from Crappy Games Wiki ] - phabricator.miraheze.org [19:34:44] Universal_Omega i guess you'll run the command, didn't want to run it just in case :) [19:34:56] `!log DELETE FROM oldimage WHERE oi_archive_name=''; on terribletvshowswiki [19:35:06] !log DELETE FROM oldimage WHERE oi_archive_name=''; on terribletvshowswiki [19:35:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:30] !log sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki terribletvshowswiki [19:35:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:41] !log sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki terribletvshowswiki --r "[[phab:T6866|Requested]]" /home/reception/ttsdel.txt [19:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:55] Reception123: works [19:35:59] paladox ^ [19:36:11] thanks! [19:36:15] Universal_Omega: great! [19:37:32] paladox: no problem. Reception123: yep. see https://terribletvshows.miraheze.org/wiki/Special:Log/delete now. working correctly. [19:37:34] [ Deletion log - Terrible TV Shows Wiki ] - terribletvshows.miraheze.org [20:42:06] !log changed a Phabricator email [20:42:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:28] PROBLEM - cloud3 Puppet on cloud3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [20:56:34] PROBLEM - services4 Puppet on services4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [20:56:35] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:56:37] PROBLEM - db13 Puppet on db13 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [20:56:42] PROBLEM - rdb3 Puppet on rdb3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [20:56:44] PROBLEM - mon2 Puppet on mon2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [20:56:53] PROBLEM - db12 Puppet on db12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [20:57:04] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [20:57:09] PROBLEM - jobrunner3 Puppet on jobrunner3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [20:57:09] PROBLEM - gluster3 Puppet on gluster3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [20:57:10] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [20:57:11] PROBLEM - db11 Puppet on db11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [20:57:18] PROBLEM - graylog2 Puppet on graylog2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [20:57:27] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [20:57:27] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [20:57:29] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [20:57:30] PROBLEM - dbbackup1 Puppet on dbbackup1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [20:57:32] PROBLEM - ldap2 Puppet on ldap2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [20:57:32] PROBLEM - mail2 Puppet on mail2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [20:57:34] PROBLEM - dbbackup2 Puppet on dbbackup2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [20:57:40] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [20:57:46] PROBLEM - jobrunner4 Puppet on jobrunner4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [20:57:49] PROBLEM - phab2 Puppet on phab2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [20:57:58] PROBLEM - cp12 Puppet on cp12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [20:58:03] PROBLEM - cp11 Puppet on cp11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 22 minutes ago with 0 failures [20:58:06] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [20:58:21] PROBLEM - mem2 Puppet on mem2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [20:58:23] PROBLEM - services3 Puppet on services3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [20:58:25] PROBLEM - cloud4 Puppet on cloud4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [20:58:25] PROBLEM - cloud5 Puppet on cloud5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 33 minutes ago with 0 failures [20:58:26] PROBLEM - cp10 Puppet on cp10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [20:58:31] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 33 minutes ago with 0 failures [20:58:35] PROBLEM - gluster4 Puppet on gluster4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [21:24:14] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 150.87 ms [21:24:20] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.51 ms [21:24:26] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 151.77 ms [21:47:04] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 116.49 ms [21:48:41] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 100.82 ms [21:48:53] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 102.65 ms [23:04:19] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 8.33, 7.33, 3.47 [23:08:20] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.65, 3.70, 2.85 [23:10:20] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.24, 2.57, 2.54