[00:02:36] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3711 MB (19% inode=93%); [00:56:31] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.69, 1.97, 1.32 [00:58:32] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.49, 1.40, 1.19 [01:20:27] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:26] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Thu 20 Aug 2020 14:59:34 GMT +0000. [01:47:04] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.00, 1.85, 1.44 [01:49:10] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.08, 1.57, 1.38 [01:51:35] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.10, 20.16, 15.75 [01:53:30] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.30, 18.46, 15.69 [02:50:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfxn2 [02:50:22] [02miraheze/services] 07MirahezeSSLBot 03035f7ac - BOT: Updating services config for wikis [05:03:13] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.39, 18.58, 16.14 [05:05:10] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 13.08, 16.56, 15.70 [05:26:21] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.43, 2.16, 1.43 [05:28:23] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.65, 1.64, 1.33 [06:10:26] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfx4h [06:10:27] [02miraheze/services] 07MirahezeSSLBot 03d601489 - BOT: Updating services config for wikis [06:20:25] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfxBc [06:20:27] [02miraheze/services] 07MirahezeSSLBot 030adb324 - BOT: Updating services config for wikis [07:33:16] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.90, 21.37, 16.99 [07:35:10] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.15, 17.77, 16.18 [09:05:42] !log remove my own email from status subscribers and switch staff@ to point to tech@ [09:05:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:10:49] !log also update our email to tech@ [09:10:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:51:08] Good morning [11:11:50] Good morning [11:12:29] Good morning avengium [11:13:08] hi [11:13:18] Where can i request a new feature? i want segment of 60 and 90 days for "recent changes" [11:19:29] avengium: set RecentChangesMaxAge to 90 days in ManageWiki settings and then use the options box on Special:RecentChanges [11:23:37] yes. but i want to keep the max at 180 days. I want to request intermediate segments. Is something difficult to code? [11:23:52] maybe is something dependant on mediawiki, not miraheze [11:31:56] avengium: do you see the recent changes link days option? [11:32:17] Hmm 60+90 aren't there [11:32:29] avengium: I will add them later [11:32:46] .in 9hours add 60+90 to RC Link Days [11:32:46] RhinosF1: Okay, will remind at 2020-06-23 - 21:32:46BST [11:39:26] Thanks RhinosF1 =D (y) [11:39:39] Np [11:40:41] Hello test-mh! If you have any questions, feel free to ask and someone should answer soon. [11:42:02] test-mh: hi [11:42:08] RhinosF1 Hello. [11:42:23] test-mh: how can we help? [11:42:28] RhinosF1 I'm running an bot (mh-bot) [11:42:38] test-mh: for what purpose? [11:42:54] Reception123 I need to use Miraheze Toolforge. [11:43:05] Well done Sigyn [13:23:37] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.26, 21.01, 18.01 [13:25:33] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.05, 17.58, 17.10 [13:53:23] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.64, 20.38, 18.40 [13:55:23] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.57, 18.46, 17.95 [14:53:35] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.23, 2.48, 1.47 [14:59:39] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.11, 1.92, 1.58 [15:01:42] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.44, 1.45, 1.45 [15:03:20] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.05, 19.17, 17.87 [15:05:18] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.54, 17.39, 17.38 [15:12:35] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Full, 4274 files, 12.40MB, 2020-06-07 15:12:00 (2.3 weeks ago) [15:50:24] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfxby [15:50:25] [02miraheze/services] 07MirahezeSSLBot 038e726f5 - BOT: Updating services config for wikis [16:08:20] Hello FijoNombre! If you have any questions, feel free to ask and someone should answer soon. [16:08:49] Reception123 Miraheze Toolforge is available??? [16:13:37] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.63, 1.83, 1.28 [16:15:39] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.88, 1.54, 1.24 [16:31:47] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.32, 1.96, 1.52 [16:33:50] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.84, 1.63, 1.45 [17:16:03] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.18, 2.21, 1.65 [17:18:08] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.46, 1.62, 1.51 [17:32:09] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2113 MB (10% inode=93%); [17:49:26] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.24, 2.09, 1.53 [17:51:24] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.09, 1.67, 1.44 [18:24:00] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.10, 1.93, 1.44 [18:25:59] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.44, 1.44, 1.32 [18:49:46] paladox: I'm still mobile and I just got 3 hour old watchlist email - runJobs? [18:50:47] !log root@jobrunner1:/home/paladox# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php [18:50:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:02:10] paladox: we seem to be doing that ~once a week now. What's the cron config for it? [19:02:36] https://github.com/miraheze/puppet/blob/master/modules/mediawiki/templates/jobrunner.json.erb [19:02:37] [ puppet/jobrunner.json.erb at master · miraheze/puppet · GitHub ] - github.com [19:02:53] and https://gerrit.wikimedia.org/g/mediawiki/services/jobrunner/+/refs/heads/master [19:02:54] [ refs/heads/master - mediawiki/services/jobrunner - Gitiles ] - gerrit.wikimedia.org [19:03:37] paladox: how often is it running? [19:03:43] Automatically [19:03:47] it's not really a cron [19:03:50] it runs every second [19:03:54] it uses redis [19:04:02] What's going on then? [19:04:07] Something isn't right [19:04:14] GUP backlogs the queue [19:04:20] paladox: GUP? [19:04:26] Global User Page [19:04:39] Hmmm [19:04:49] paladox: we need to look at something [19:05:10] look at what? [19:05:18] the wmf have resources to handle this :( [19:05:40] also they have significantly less wikis too [19:06:32] paladox: is there anyway to isolate the GUP job so less resources are being consumed by it? [19:06:50] well there is a way to create seperate queues to not back log things [19:06:54] but JohnLewis objected to that [19:07:45] JohnLewis: comments? [20:13:54] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.71, 20.95, 17.80 [20:15:51] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.10, 19.38, 17.60 [20:32:47] RhinosF1: add 60+90 to RC Link Days [21:10:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfpLs [21:10:24] [02miraheze/services] 07MirahezeSSLBot 037f52fb0 - BOT: Updating services config for wikis [21:35:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfpts [21:35:16] [02miraheze/services] 07MirahezeSSLBot 03ce781df - BOT: Updating services config for wikis [21:38:47] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.67, 2.16, 1.48 [21:40:47] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.55, 1.57, 1.34 [21:53:17] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.12, 19.14, 15.76 [21:54:57] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.49, 1.77, 1.52 [21:55:14] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.43, 16.78, 15.32 [21:58:56] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 4.88, 2.78, 1.93 [22:03:01] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.98, 1.71, 1.68 [22:04:59] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.71, 1.40, 1.57 [22:13:05] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.45, 1.77, 1.70 [22:15:07] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.50, 1.30, 1.53 [22:48:22] my search bar does not seem to work on my wiki [22:48:30] every time i try and search a page, nothing comes up [22:50:52] Do you have either of the TitleKey or Wikibase extensions enabled on your wiki? [22:56:22] let me check [22:57:04] ah i found the problem, thank you so much! [22:57:29] wait [22:57:43] i just enabled TitleKey and it's still not showing up [22:57:44] odd [22:58:12] ahh, whoops, the problem I'm thinking of occurred if you had either of those enabled [22:58:59] everytime i search anything it says "no match was found" [23:02:02] So, the idea is to disable the TitleKey extension (unless you need it for something), and if you have wikibase added, go to Special:ManageWiki/settings#mw-section-wikibase and uncheck "Enable Entity Search UI" (I think) [23:04:28] ok [23:06:15] PROBLEM - bacula2 Bacula Databases db6 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:06:48] ah it worked [23:06:53] thanks man [23:08:16] PROBLEM - bacula2 Bacula Databases db6 on bacula2 is WARNING: WARNING: Full, 22675 files, 97.02GB, 2020-06-07 09:40:00 (2.4 weeks ago) [23:10:51] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [23:12:52] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 82779 files, 3.261GB, 2020-06-07 15:19:00 (2.3 weeks ago)