[00:02:44] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3853 MB (15% inode=93%); [00:18:13] PROBLEM - db7 APT on db7 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:22:47] PROBLEM - misc1 APT on misc1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:25:22] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:35:19] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [00:36:41] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:37:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [00:38:42] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 269.49 ms [00:39:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:41:58] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[fr.gyaanipedia.co.in_private] [00:45:56] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:22:16] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:35:34] PROBLEM - rdb2 APT on rdb2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:50:18] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ24z [01:50:20] [02miraheze/services] 07MirahezeSSLBot 03821bb98 - BOT: Updating services config for wikis [02:29:31] @System Administrators jobrunner1 seems to be having problems again; a page on Meta marked for translation more than two hours ago now still hasn't been updated by system user FuzzyBot. Looking at https://icinga.miraheze.org/dashboard#!/monitoring/list/services?service_problem=1&sort=service_severity, I do see 12 notices flagged as "critical," one of which is for jobrunner1; however, they seem to be related to needed application updates, [02:29:32] so that's probably not related. Is there a dashboard I can use to view the current backlog of tasks on jobrunner1? [02:29:33] [ Icinga Web 2 Login ] - icinga.miraheze.org [02:34:47] I do see this: > NOTIFICATION > 15:39:46 > [icingaadmin] CRITICAL - Plugin timed out after 30 seconds > > NOTIFICATION > 15:39:46 > [icingaadmin, irc, johnflewis, paladox, southparkfan] CRITICAL - Plugin timed out after 30 seconds That's about right, now that I think about it, as the last page successfully updated after being marked for translation was ~15:00. The most recent one at ~18:47 hasn't been updated. So, it could be that [02:34:47] the job runner for Extension:Translate has stopped, maybe? [02:39:45] Disregard the message from 7:34 PM...those notifications were from 24 July 2020. Still, there seems to be problems with jobrunner1. [02:42:01] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:57:42] @doug I'm forcing manual job runs for all wikis until i can get someone from SRE [02:57:42] Zppix: 2020-07-30 - 16:13:56CDT tell Zppix Status? [03:00:15] Thanks, @Zppix. 🙂 [03:00:59] !log '/usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs -- Jobrunner1 is not running jobs automatically [03:01:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:52] @Doug i managed to restart it so it should auto run jobs now, give it a little bit to catch up meta alone has over 800 jobs in queue [03:30:47] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:48:24] Oh fuck I'm here [03:48:41] > @Doug i managed to restart it so it should auto run jobs now, give it a little bit to catch up meta alone has over 800 jobs in queue @Zppix thanks...wow, yeah, sometimes that's best to restart the process. Yeah, I don't mark any more pages on Meta until it catches up. You can usually tell when the jobrunner is behind because things like the [[Special:Watchlist]] and [[Special:RecentChanges]] can't keep up. [03:48:58] welcome, @RhinosF1...sorry your sleep was so short 😦 [03:50:50] Zppix: what do you know? [04:46:00] !log restarted jobrunner service [04:46:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:29] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:09:39] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:18:02] !log sudo service redis-server restart on jobrunner1 [05:18:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:18:35] !log sudo service jobrunner restart on jobrunner1 [05:18:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:20:16] Reception123: did it stop again already? [05:20:29] Zppix: yep [05:20:36] wth [05:20:50] for now it seems to be running [05:20:54] Zppix: you can sleep, I'm awake now [05:21:24] Reception123: no jobs are running [05:21:27] i was going to bed anyway, i just saw the log msgs [05:21:42] RhinosF1: did you use foreach to run manually? [05:21:45] May have to resort to manual job runs until we find a fix [05:22:19] Reception123: I haven't yet as I want to see if it does them itself and try and get it on a crontab [05:22:33] RhinosF1: https://github.com/miraheze/puppet/commit/d98641b76515d3d8d41c4df4a92df34baf2819c8 could it be that you made a syntax error here? [05:22:34] [ +2 crons for GlobalUsage (#1461) · miraheze/puppet@d98641b · GitHub ] - github.com [05:22:39] that's what the logs seem to suggest (syntax error) [05:22:39] i would manually run loginwiki and metawiki regardless [05:22:53] I'll check [05:23:21] !log manually run loginwiki jobs on jobrunner1 [05:23:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:33] !log manually run metawikijobs on jobrunner1 [05:23:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:59] !log rhinos@jobrunner1:/var/log/mediawiki$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki='wright009wiki' --type='LocalGlobalUserPageCacheUpdateJob' --maxtime='60' --memory-limit='192M' --result=json [05:26:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:36] Reception123: i wonder if it hitting the mem limit and killing itself [05:26:55] Zppix: ask me in a minute [05:27:59] im off to bed its 12:30AM [05:28:35] okay, goodnight @Zppix [05:29:02] fixing jobrunner is UBN Reception123 RhinosF1 [05:29:18] Definitely possible [05:29:27] But if that's the case what can we do? [05:29:55] for now we probably need to do manual job runs to keep job queue flowing [05:30:19] otherwise jobrunner may never catch back up [05:30:40] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:30:45] Agree with that...hopefully we can eventually figure out what the cause was, so we can correct whatever keeps causing the traffic surges and task backlog [05:31:24] !log rhinos@jobrunner1:/var/log/mediawiki$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=wright009wiki [05:31:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:31:42] RhinosF1: just run foreach [05:31:48] in a screen or soemthing [05:32:09] Zppix: I'm doing it for a reason [05:32:23] I think it just hated one of the jobs [05:32:55] Ah, probably to try and diagnose which job is causing the problem, eh @RhinosF1? [05:33:14] Reception123, Zppix: let's give it 5 minutes and see if any now go through or we get more errors [05:33:29] @Doug: I know which, I have no clue why it hated it. [05:33:32] Ok [05:34:38] Reception123: if it's not moving within the next ~10 mins then we need to put a cron together as a plaster [05:34:41] > @Doug: I know which, I have no clue why it hated it. @RhinosF1 Well, that's positive actually that you've narrowed down which one. Was it the GlobalUsage one above? [05:35:01] @doug: LocalGlobalUserPageCacheUpdateJob [05:35:05] > @Reception123 if it's not moving within the next ~10 mins then we need to put a cron together as a plaster ^ +1 to this [05:35:18] RhinosF1: but the jobs run fine? [05:35:26] > @Doug: LocalGlobalUserPageCacheUpdateJob Ah, yeah, meant to say or that one. Thanks. 🙂 [05:35:34] Zppix: jobrunner was moaning about the json output [05:35:48] oh [05:35:48] .in 1 minute test 1 [05:35:48] RhinosF1: Okay, will remind at 2020-07-31 - 06:36:48BST [05:35:54] anyway night [05:35:58] .in 6mins please work [05:35:59] RhinosF1: Okay, will remind at 2020-07-31 - 06:41:58BST [05:36:48] RhinosF1: test 1 [05:37:36] Reception123: same error, another wiki [05:38:01] Reception123: check /var/log/redis [05:40:18] Ok [05:41:59] RhinosF1: please work [05:45:22] !log restart jobrunner (again) [05:45:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:22] Reception123: let's hope paladox can get it moving now we have a cause [05:48:08] Yeah [05:50:05] @RhinosF1 did you guys find out anything more other than just which job it was (LocalGlobalUserPageCacheUpdateJob)? [05:50:36] we know the error and it's affects every wiki the user has an account on it seems [05:51:08] ah, ok [05:52:55] okay, with the problem at least diagnosed, I'm going to say good night then as it's getting late here as well...hopefully it won't take you too much longer and you can get some more sleep [05:53:17] i wont get back to sleep [05:54:19] > i wont get back to sleep aww, that sucks...this thing got you too wide awake, I guess? maybe you can catch it up tomorrow night [05:55:24] yep [05:59:06] !log rebooted jobrunner1 [05:59:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:45] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:03:40] RECOVERY - mw4 APT on mw4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:12:13] RECOVERY - db7 APT on db7 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:16:08] !log forcing a manual run of jobs on all wikis to clear backlog as the job runner hasn't crashed yet [06:16:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:15] RECOVERY - cp3 APT on cp3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:25:09] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:26:46] RECOVERY - misc1 APT on misc1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:27:34] RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:29:17] RECOVERY - mon1 APT on mon1 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [06:31:29] RECOVERY - db11 APT on db11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:34:01] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:46:07] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:46:28] RECOVERY - mw7 APT on mw7 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:49:23] RECOVERY - mw6 APT on mw6 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:52:08] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JJ2Kj [06:52:10] [02miraheze/ssl] 07Reception123 0370f9140 - rm wiki.bydorn.de per T5990 [06:54:39] RECOVERY - mw5 APT on mw5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:00:41] RECOVERY - ldap1 APT on ldap1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:03:29] [02mw-config] 07Reception123 closed pull request 03#3194: Add "editor" user group and protection level to sesupportwiki T5989 - 13https://git.io/JJ2Zw [07:03:31] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ26M [07:03:32] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:03:32] [02miraheze/mw-config] 07Amanda-Catherine 03a9fa56d - Add "editor" user group and protection level to sesupportwiki T5989 (#3194) [07:04:05] !log sudo apt-get upgrade on services2 [07:04:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:05:30] RECOVERY - services2 APT on services2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:08:20] Reception123 Thank you! But there misssing "u" i think [07:08:21] https://cdn.discordapp.com/attachments/435711390544560128/738654262451765278/IMG_20200731_090729.jpg [07:08:41] Oh, I didn't notice the small typo [07:08:43] I'll fix it [07:09:16] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ2iu [07:09:17] [02miraheze/mw-config] 07Reception123 03a5d55bf - typo [07:13:40] https://cdn.discordapp.com/attachments/435711390544560128/738655604964458546/IMG_20200731_091324.jpg [07:15:56] Or it need few minutes to deploy? :) [07:17:07] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±4] 13https://git.io/JJ2Pg [07:17:09] [02miraheze/mw-config] 07Reception123 0319b3486 - Add TwitterTag ext. T5862 [07:17:10] [02mw-config] 07Reception123 created branch 03Reception123-patch-3 - 13https://git.io/vbvb3 [07:17:21] @MrJaroslavik it should've worked but to be sure wait until :22 [07:17:41] [02mw-config] 07Reception123 opened pull request 03#3195: Add TwitterTag ext. T5862 - 13https://git.io/JJ2PV [07:18:08] Delays are not problem for me if it will work :D [07:18:12] Thanks again. [07:18:25] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/JJ2Po [07:18:27] [02miraheze/mediawiki] 07Reception123 03b49f099 - Install TwitterTag ext. [07:23:23] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:08] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:29:14] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:30:16] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:30:37] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:30:43] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:31:21] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:32:16] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:32:34] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:33:03] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:21] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:42:40] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:43:08] !log jobrunner backlog is critical and being monitored [07:43:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:19] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:51:30] PROBLEM - gluster2 APT on gluster2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:07:37] PROBLEM - wiki.starship.digital - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 92, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 80, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname [08:07:38] 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wiki.starship.digital. IN A: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 U [08:07:38] P port 53 answered SERVFAIL [08:07:46] PROBLEM - vmcodex.net - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 92, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 80, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'A')[0]) [08:07:46] File "/usr/lib/python3/dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query vmcodex.net. IN A: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [08:14:19] RECOVERY - wiki.starship.digital - reverse DNS on sslhost is OK: rDNS OK - wiki.starship.digital reverse DNS resolves to cp6.miraheze.org [08:14:30] RECOVERY - vmcodex.net - reverse DNS on sslhost is OK: rDNS OK - vmcodex.net reverse DNS resolves to cp7.miraheze.org [08:31:29] hi JohnLewis [08:33:55] hi [08:43:25] hi [08:43:32] yeah relay back [08:50:21] [02puppet] 07RhinosF1 opened pull request 03#1463: check_reverse_dns: handle an exception - 13https://git.io/JJ29M [08:50:37] JohnLewis: that should solve the DNS messy alert [09:03:56] [02mw-config] 07Reception123 closed pull request 03#3195: Add TwitterTag ext. T5862 - 13https://git.io/JJ2PV [09:03:58] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JJ2Hw [09:03:59] [02miraheze/mw-config] 07Reception123 03c9a0127 - Add TwitterTag ext. T5862 (#3195) [09:04:01] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://git.io/vbvb3 [09:04:02] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-3 [09:05:45] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki' on mw* and jbr1 [09:05:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:14] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr1 [09:07:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:28] [02miraheze/mw-config] 07Reception123 deleted branch 03Drop-php-72 [09:07:30] [02mw-config] 07Reception123 deleted branch 03Drop-php-72 - 13https://git.io/vbvb3 [09:07:31] [02mw-config] 07Reception123 deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [09:07:33] [02miraheze/mw-config] 07Reception123 deleted branch 03paladox-patch-1 [09:07:34] [02miraheze/mw-config] 07Reception123 deleted branch 03paladox-patch-3 [09:07:36] [02mw-config] 07Reception123 deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [09:08:36] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:13:32] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.08, 5.89, 3.79 [09:13:33] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.61, 6.28, 3.90 [09:14:05] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03test-tag-hack [+0/-0/±1] 13https://git.io/JJ2Qm [09:14:06] [02miraheze/mw-config] 07RhinosF1 0331240b7 - Update LocalWiki.php [09:14:08] [02mw-config] 07RhinosF1 created branch 03test-tag-hack - 13https://git.io/vbvb3 [09:14:24] [02mw-config] 07RhinosF1 opened pull request 03#3196: Update LocalWiki.php - 13https://git.io/JJ2QY [09:15:32] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.48, 5.77, 4.00 [09:15:33] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.82, 6.10, 4.13 [09:15:40] miraheze/mw-config/test-tag-hack/31240b7 - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/713618895 [09:15:57] Reception123: ^ good sign [09:16:56] $log hack #3196 of mw-config onto test2 [09:17:30] Reception123: it works [09:19:03] :) [09:19:05] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03test-tag-hack [+0/-0/±1] 13https://git.io/JJ2QX [09:19:06] [02miraheze/mw-config] 07RhinosF1 033d42d3d - it worked! [09:19:08] [02mw-config] 07RhinosF1 synchronize pull request 03#3196: Update LocalWiki.php - 13https://git.io/JJ2QY [09:19:15] Reception123: i'll let travis pass then merge [09:19:28] RhinosF1: great! then you can close the task [09:19:35] and thanks for including the task as a comment, we need to do that more [09:19:38] it's quite helpful [09:19:58] Reception123: :) [09:20:06] now to wait for travis [09:20:44] which is on a go slow [09:21:27] $log? [09:21:49] JohnLewis: what about it [09:22:06] isn't it !log? [09:22:07] https://www.irccloud.com/pastebin/oo880dZT/ [09:22:07] [ Snippet | IRCCloud ] - www.irccloud.com [09:22:29] JohnLewis: oh ye, that's called jumping back and forth between two logbots [09:23:04] !log cherry-picked 3d42d3d onto test2 (will reset after merge) [09:23:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:15] Reception123: github checks seems to be this thing to break. It's being slow to report status [09:24:43] web says it passed so merging [09:25:12] [02mw-config] 07RhinosF1 closed pull request 03#3196: Update LocalWiki.php - 13https://git.io/JJ2QY [09:25:14] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ27T [09:25:15] [02miraheze/mw-config] 07RhinosF1 036b15d1b - LocalWiki: add meta tag for T5981 (#3196) [09:25:17] [02mw-config] 07RhinosF1 deleted branch 03test-tag-hack - 13https://git.io/vbvb3 [09:25:18] [02miraheze/mw-config] 07RhinosF1 deleted branch 03test-tag-hack [09:25:26] PROBLEM - wiki.ldmsys.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ldmsys.net could not be found [09:27:00] [02puppet] 07JohnFLewis reviewed pull request 03#1463 commit - 13https://git.io/JJ27n [09:27:14] !log reset git to remove previous hack [09:27:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:37] JohnLewis: do you know puppet on test2 says "Error: '/usr/bin/git pull --quiet' returned 1 instead of one of [0] [09:27:37] Error: /Stage[main]/Mediawiki/Git::Clone[MediaWiki config]/Exec[git_pull_MediaWiki config]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/git pull --quiet' returned 1 instead of one of [0] (corrective)" [09:28:05] that's me [09:28:06] fixing [09:29:05] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [09:30:58] fixed [09:33:04] [02puppet] 07RhinosF1 synchronize pull request 03#1463: check_reverse_dns: handle an exception - 13https://git.io/JJ29M [09:33:04] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:33:17] [02puppet] 07RhinosF1 reviewed pull request 03#1463 commit - 13https://git.io/JJ27w [09:33:23] JohnLewis: ^ [09:38:09] PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:47:28] PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Sun 16 Aug 2020 09:39:48 GMT +0000). [09:48:54] PROBLEM - vault.aics.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vault.aics.cf' expires in 15 day(s) (Sun 16 Aug 2020 09:45:14 GMT +0000). [09:54:25] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:55:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ25b [09:55:21] [02miraheze/ssl] 07MirahezeSSLBot 0388ba297 - Bot: Update SSL cert for wiki.cdntennis.ca [10:01:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ2dZ [10:01:10] [02miraheze/ssl] 07MirahezeSSLBot 035957f83 - Bot: Update SSL cert for vault.aics.cf [10:07:30] RECOVERY - wiki.cdntennis.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cdntennis.ca' will expire on Thu 29 Oct 2020 08:55:13 GMT +0000. [10:11:08] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [10:16:16] RECOVERY - vault.aics.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'vault.aics.cf' will expire on Thu 29 Oct 2020 09:01:01 GMT +0000. [10:32:28] !log deleted and dropped wizpedia101wiki for recreation [10:32:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:53] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki reallifevillainswiki --r "Requested - T5987" /home/reception/rlv4.txt [10:35:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:41:12] !log manually runJobs on metawiki and loginwiki [10:41:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:57] !log reception@mw6:~$ sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php --type='LocalGlobalUserPageCacheUpdateJob' on mw6 (backlog is still clogged) [10:48:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:40] Can anyone explain to me why a user needs CheckUser or Oversight permission, when on wiki is active only he? [11:38:00] don't ask me but that sounds quite absurd [11:38:03] They don't [11:40:52] It could be useful on wikis with many active users and frequent attacks/PII, but not in that case... [11:41:27] On WMF also are active projects without CUs/OSs [11:44:29] How should an request for these permissions be defined here? What requirements except the NDA? Only wiki owner request? Please no. [11:44:50] (Sorry for these shouts btw) [11:49:31] They'll be an RfC soon [11:52:28] Also @Reception123, i see this on Special:ProtectedPages but level is not on &action=protect [11:52:28] https://cdn.discordapp.com/attachments/435711390544560128/738725764026859570/IMG_20200731_135157.jpg [12:01:25] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03why-doesnt--run-faster-exist [+0/-0/±1] 13https://git.io/JJ2hP [12:01:27] [02miraheze/mw-config] 07RhinosF1 032e185ae - [TEMP] - Up JobRunRate We need to do something or this queue is never going to catch up. Would this work? [12:01:28] [02mw-config] 07RhinosF1 created branch 03why-doesnt--run-faster-exist - 13https://git.io/vbvb3 [12:01:30] [02mw-config] 07RhinosF1 opened pull request 03#3197: [TEMP] - Up JobRunRate - 13https://git.io/JJ2h1 [12:01:59] JohnLewis would ^ work [12:03:07] miraheze/mw-config/why-doesnt--run-faster-exist/2e185ae - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/713663412 [12:07:23] !log runJobs on meta+login to ensure backlog stays low [12:07:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:08:20] [02mw-config] 07MrJaroslavik opened pull request 03#3198: Add import option from Wikipedia to Meta-Wiki - 13https://git.io/JJ2jk [12:09:02] I tried create pull request, is it in correct format? [12:11:20] @MrJaroslavik: Looks good [12:11:42] Ok [12:14:02] RhinosF1 can you close this pull request (as decline)? I will create new pull request in hour. https://github.com/miraheze/landing/pull/37 [12:14:03] [ Add Czech translate by MrJaroslavik · Pull Request #37 · miraheze/landing · GitHub ] - github.com [12:14:23] ye [12:14:35] [02landing] 07RhinosF1 commented on pull request 03#37: Add Czech translate - 13https://git.io/JJ2jg [12:14:37] [02landing] 07RhinosF1 closed pull request 03#37: Add Czech translate - 13https://git.io/JfzLh [12:15:05] Thanks [12:15:14] @MrJaroslavik that's weird.. [12:15:37] What is weird? [12:16:22] the protection level thing [12:16:33] Yes, it is. [12:16:34] Reception123: check your mw6 screen [12:16:42] RhinosF1: ok, on it [12:17:07] RhinosF1: I mean it's still continuing [12:17:51] Reception123: good. my mw7 is moving a bit faster, I'm hoping yours will overtake for GUP eventually. [12:18:30] Reception123: who has access to do right on conductwiki + ggroup? [12:19:09] RhinosF1: see PM [12:21:51] PROBLEM - puppet2 APT on puppet2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:32:37] @Reception123: you'll have to use discord as irc cloud is being a pain [12:33:51] [02mw-config] 07JohnFLewis commented on pull request 03#3197: [TEMP] - Up JobRunRate - 13https://git.io/JJaeh [12:33:53] [02mw-config] 07JohnFLewis closed pull request 03#3197: [TEMP] - Up JobRunRate - 13https://git.io/JJ2h1 [12:35:04] [02mw-config] 07RhinosF1 commented on pull request 03#3197: [TEMP] - Up JobRunRate - 13https://git.io/JJavJ [12:36:45] [02mw-config] 07JohnFLewis commented on pull request 03#3197: [TEMP] - Up JobRunRate - 13https://git.io/JJavL [12:37:13] JohnLewis: they're not moving that fast though [12:38:35] Doing it on web requests won’t help more, command line is the fastest method [12:39:17] you have 6 servers you could run it on, plus two aren’t utilised for web so can have more demand put on them [12:39:54] JohnLewis: i'm going mobile now and due to an issue with irc cloud am discord only but if you make the screens (mw[6 (reception)|7 (me)] go faster or want to spawn it on more servers, please do. [12:41:58] Making things go faster isn’t possible, making more processes is possible though I don’t have access right now [12:42:44] Neither do I but @Reception123 might and if so then please spawn more and watch load. [12:43:01] Ideally more runners would be best in future [12:43:45] And not allow the GUP job to take them all up [12:43:59] @Reception123; I've no irc, use discord [12:44:25] Ok [12:59:13] [02landing] 07MrJaroslavik opened pull request 03#40: Add Czech Translation (2) - 13https://git.io/JJafy [13:00:59] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:14:05] PROBLEM - cp9 APT on cp9 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:22:05] @MrJaroslavik see review [13:24:10] [02landing] 07MrJaroslavik synchronize pull request 03#40: Add Czech Translation (2) - 13https://git.io/JJafy [13:24:40] In this format? [13:25:05] Or Čeština (Czech)? [13:26:35] [02landing] 07RhinosF1 reviewed pull request 03#40 commit - 13https://git.io/JJaTZ [13:26:36] [02landing] 07RhinosF1 reviewed pull request 03#40 commit - 13https://git.io/JJaTn [13:27:21] @MrJaroslavik ^ [13:27:33] Looking [13:32:01] [02landing] 07MrJaroslavik reviewed pull request 03#40 commit - 13https://git.io/JJaTd [13:32:44] @MrJaroslavik: yeah that line removal looks fine, you are right it shouldn't be there [13:33:22] Sorry, i don't know how comments system works :( (on github) [13:33:29]