[00:03:50] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:51] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:34:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [03:56:41] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3035 [04:08:44] PROBLEM - mw1 Disk Space on mw1 is WARNING: DISK WARNING - free space: / 9604 MB (12% inode=98%); [04:49:21] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.81, 1.63, 0.90 [04:51:19] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.75, 2.07, 1.15 [04:53:20] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.41, 1.42, 1.02 [04:53:38] PROBLEM - bacula1 Bacula Lizardfs2 Lizardfs Chunkserver2 on bacula1 is UNKNOWN: NRPE: Unable to read output [04:53:42] PROBLEM - bacula1 Bacula Static Lizardfs2 on bacula1 is UNKNOWN: NRPE: Unable to read output [04:53:44] PROBLEM - bacula1 Bacula Static Lizardfs1 on bacula1 is UNKNOWN: NRPE: Unable to read output [04:53:58] PROBLEM - bacula1 Bacula Daemon on bacula1 is CRITICAL: PROCS CRITICAL: 1 process with UID = 110 (bacula) [04:54:18] PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output [04:54:24] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output [04:54:50] PROBLEM - bacula1 Bacula Lizardfs1 Lizardfs Chunkserver1 on bacula1 is UNKNOWN: NRPE: Unable to read output [05:00:54] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 69568 files, 4.111GB, 2018-09-22 14:40:00 (2.1 weeks ago) [05:01:10] RECOVERY - bacula1 Bacula Lizardfs1 Lizardfs Chunkserver1 on bacula1 is OK: OK: Full, 7 files, 148.0KB, 2018-10-07 04:45:00 (16.1 minutes ago) [05:01:54] RECOVERY - bacula1 Bacula Daemon on bacula1 is OK: PROCS OK: 2 processes with UID = 110 (bacula) [05:02:02] RECOVERY - bacula1 Bacula Lizardfs2 Lizardfs Chunkserver2 on bacula1 is OK: OK: Full, 5 files, 139.1KB, 2018-10-07 04:49:00 (13.0 minutes ago) [05:02:12] RECOVERY - bacula1 Bacula Misc4 Lizardfs Master on bacula1 is OK: OK: Full, 16 files, 149.4MB, 2018-10-07 04:43:00 (19.2 minutes ago) [05:02:18] RECOVERY - bacula1 Bacula Static Lizardfs1 on bacula1 is OK: OK: Full, 754423 files, 79.37GB, 2018-09-23 18:21:00 (1.9 weeks ago) [05:02:26] RECOVERY - bacula1 Bacula Static Lizardfs2 on bacula1 is OK: OK: Full, 866559 files, 85.22GB, 2018-09-24 01:30:00 (1.9 weeks ago) [05:02:42] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 1176363 files, 52.94GB, 2018-10-07 04:39:00 (23.7 minutes ago) [05:02:46] RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 1568 files, 1.760MB, 2018-09-30 00:08:00 (1.0 weeks ago) [09:16:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [13:03:15] It's s *syadmin* matter [13:03:25] See https://meta.miraheze.org/wiki/Staff [13:03:26] Title: [ Staff - Miraheze Meta ] - meta.miraheze.org [13:06:13] No problem, and sorry for the trouble [13:08:55] hi! [13:09:07] what was the url where you can download the wiki dumps? [13:11:00] Enfaru hi, we do store db backups in case of emergency. Though not sure how easy it would be to retrieve them. [13:11:18] Also if you did not reopen the wiki, then that would be why. [13:12:29] Enfaru it was in this query: "SELECT wiki_dbname from cw_wikis WHERE wiki_closed_timestamp <= NOW() - INTERVAL 120 DAY;" [13:12:41] editing a wiki does not reopen it [13:12:51] you have to go to ManageWiki and untick close [13:12:56] hmm [13:12:57] JohnLewis ^^ [13:19:13] Hmm, ticking closed and unticking it works for me on test1 [13:20:01] Yup we know. [13:34:46] PROBLEM - mw1 Disk Space on mw1 is CRITICAL: DISK CRITICAL - free space: / 8441 MB (10% inode=98%); [13:36:46] RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 21759 MB (28% inode=99%); [13:40:40] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 409 [13:42:40] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [14:43:18] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Full, 69568 files, 4.111GB, 2018-09-22 14:40:00 (2.1 weeks ago) [16:03:10] Hello [16:04:14] hi [16:05:25] hi [16:06:17] Did you guys receive the email I sent? [16:07:42] christ [16:07:49] i would love it if spam email didnt actually get forwarded through [16:07:54] 378 unread emails [16:08:07] PuppyKun: lol [16:08:15] AmandaCatherine: if you mean the one from yesterday its there [16:08:44] I didn’t send one today :) [16:09:11] oh. half of those emails were literally stacked emails about postfix smtp errors. yay i just delete the whole thing [16:09:13] blame john [16:09:30] Lol [16:14:34] [02mw-config] 07Amanda-Catherine opened pull request 03#2512: Namespaces for several wikis T3683 - 13https://git.io/fxsKz [16:15:57] Amanda-Catherine/mw-config/patch-1/10222c4 - Amanda The build failed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/87129659 [16:16:24] Grrrr [16:17:25] “PHP Parse error: syntax error, unexpected ''+centralwiki'' (T_CONSTANT_ENCAPSED_STRING), expecting ')' in ./LocalSettings.php on line 3043” [16:17:31] WTF does that mean [16:19:50] Reception123 paladox what the heck does that mean? ^^ [16:19:59] * AmandaCatherine is confused [16:20:34] [02mw-config] 07paladox reviewed pull request 03#2512 commit - 13https://git.io/fxsK1 [16:20:41] [02mw-config] 07paladox reviewed pull request 03#2512 commit - 13https://git.io/fxsKM [16:20:45] you're missing commas in that section [16:21:36] Oh, I just copy pasted from the Phab task [16:21:46] Guess that’s a bad idea [16:22:28] yeah, there's also at least one case of bad tabbing [16:23:10] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2512: Namespaces for several wikis T3683 - 13https://git.io/fxsKz [16:23:35] Wiki-1776: looks like you done goofed [16:25:24] Yay, it passed now [16:26:02] Amanda-Catherine/mw-config/patch-1/25512fb - Amanda The build was fixed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/87129915 [16:38:42] have no idea why custom CSS is not loading but I can't obviously log in on desktop so I'm fucked up lol [16:38:48] s/CSS/JS [16:38:48] revi meant to say: have no idea why custom JS is not loading but I can't obviously log in on desktop so I'm fucked up lol [16:41:25] and I guess you guys still have no idea wtf is wrong with the "Session Hijacking" stuff? [16:41:50] Nope, SPF|Cloud did look at it a few years ago though. [16:43:24] uh... ok [16:43:40] maybe time to have my own wiki, being unable to log in on my own wiki is so frustrating [16:44:24] did you clean your cookies? [16:44:42] no, I don't want to do it since it will log me out everywhere [17:16:12] paladox: yes, the issue with custom domains is so irritating [17:16:21] yep [17:16:25] paladox: I think it even warrants a high priority task [17:16:30] because it annoys me (and probably many others) so much [17:16:34] ok [17:16:46] Reception123 feel free to file one. [17:20:07] [02mw-config] 07paladox closed pull request 03#2512: Namespaces for several wikis T3683 - 13https://git.io/fxsKz [17:20:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxsPu [17:20:10] [02miraheze/mw-config] 07Amanda-Catherine 03eb7e3a0 - Namespaces for several wikis T3683 (#2512) * Namespaces for several wikis T3683 * Update LocalSettings.php [17:43:20] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.76, 1.48, 0.60 [17:47:34] connect to host 172.245.38.205 port 5666: Connection refused [17:48:31] !log rebooted bacula1 due to high load (no backups were in progress) [17:48:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:49:20] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.50, 0.17, 0.06 [17:49:38] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Full, 69568 files, 4.111GB, 2018-09-22 14:40:00 (2.2 weeks ago) [17:52:47] Anyone with Phab admin T3685 needs review [19:15:45] @NeoMahler oh, hi, sorry i forgot to reply to your question earlier, it's https://static.miraheze.org/dumps/ [19:15:46] Title: [ Index of /dumps/ ] - static.miraheze.org [19:18:42] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5639 [20:01:59] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [20:08:31] HTTP CRITICAL - Unable to open TCP socket [20:10:31] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 443 bytes in 0.033 second response time [20:33:57] aaahh yes, I kept forgetting the static. part [20:33:59] thanks! [20:42:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs7G [20:42:33] [02miraheze/puppet] 07paladox 037477243 - Add php-fpm status to nginx [20:42:35] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:43:42] [02puppet] 07paladox opened pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [20:45:18] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [20:45:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs7l [20:45:21] [02miraheze/puppet] 07paladox 03b70f405 - Update mediawiki.conf [20:46:31] !log upgrade phabricator on misc4 [20:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:47:48] hi paladox [20:47:52] hi [20:49:43] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [20:49:53] paladox: sorry, can you check this line and see if it is correctly written "UcroniasWiki" with "S"? https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L3666 [20:49:54] Title: [ mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub ] - github.com [20:50:28] ucroniawiki [20:50:30] shows ^^ [20:51:07] dosen't have a "s" [20:51:41] You can fix it? I'm very busy. [20:51:51] ok [20:52:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxs79 [20:52:24] [02miraheze/mw-config] 07paladox 031d67107 - ucroniawiki -> ucroniaswiki [20:53:00] thanks, thanks? [20:53:06] !* [20:54:25] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [20:54:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs7x [20:54:28] [02miraheze/puppet] 07paladox 03f2c44a8 - Update mediawiki.conf [20:54:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs7j [20:54:42] [02miraheze/puppet] 07paladox 0353bb482 - Update mediawiki.conf [20:54:44] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [20:55:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs5U [20:55:23] [02miraheze/puppet] 07paladox 0396afe02 - Update grafana.conf [20:55:24] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [20:56:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs5L [20:56:06] [02miraheze/puppet] 07paladox 03a18a82e - Update icinga2.conf [20:56:07] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [20:56:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs5q [20:56:36] [02miraheze/puppet] 07paladox 03a3817c4 - Update nginx.conf [20:56:37] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [20:57:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs5G [20:57:04] [02miraheze/puppet] 07paladox 035c2788e - Update nginx.conf [20:57:05] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [21:04:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs5K [21:04:42] [02miraheze/puppet] 07paladox 034ab6e38 - Update phab.miraheze.wiki.conf [21:04:44] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [21:05:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs56 [21:05:03] [02miraheze/puppet] 07paladox 03d7ac97b - Update phabricator.miraheze.org.conf [21:05:04] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [21:09:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxs5H [21:09:10] [02miraheze/puppet] 07paladox 03ffffd67 - Update www-7.2.conf [21:09:11] [02puppet] 07paladox synchronize pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [21:12:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+5/-1/±1] 13https://git.io/fxs5F [21:12:15] [02miraheze/puppet] 07paladox 032a9bcb3 - Add php-fpm prometheus exporter [21:12:30] [02puppet] 07paladox closed pull request 03#873: Add php-fpm status to nginx - 13https://git.io/fxs7c [21:12:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/fxs5A [21:12:33] [02miraheze/puppet] 07paladox 0313d638f - Add php-fpm status to nginx (#873) * Add php-fpm status to nginx * Update mediawiki.conf * Update mediawiki.conf * Update mediawiki.conf * Update grafana.conf * Update icinga2.conf * Update nginx.conf * Update nginx.conf * Update phab.miraheze.wiki.conf * Update phabricator.miraheze.org.conf * Update www-7.2.conf [21:12:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:12:36] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:13:43] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:13:59] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [21:14:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxs5j [21:14:43] [02miraheze/puppet] 07paladox 03a04953d - Update www-7.2.conf [21:16:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxsdI [21:16:25] [02miraheze/puppet] 07paladox 03f6b0a3a - Update prometheus-php-fpm.systemd [21:17:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxsds [21:17:29] [02miraheze/puppet] 07paladox 037fd69b3 - Include prometheus::php_fpm on misc1 as a trial run [21:20:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxsdn [21:20:14] [02miraheze/puppet] 07paladox 031f75ef7 - Update monitoring.pp [21:20:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxsdW [21:20:21] [02miraheze/puppet] 07paladox 0331c7610 - Update prometheus.yml.erb [21:22:52] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/prometheus-phpfpm-exporter] [21:24:07] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:24:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxsdz [21:24:10] [02miraheze/puppet] 07paladox 0323aef6e - Fix path to prometheus-phpfpm-exporter [21:24:11] [02puppet] 07paladox opened pull request 03#874: Fix path to prometheus-phpfpm-exporter - 13https://git.io/fxsdg [21:24:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxsdV [21:24:52] [02miraheze/puppet] 07paladox 03cb226ed - Fix path to phpfpm_exporter [21:24:55] [02puppet] 07paladox closed pull request 03#874: Fix path to prometheus-phpfpm-exporter - 13https://git.io/fxsdg [21:24:57] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:24:58] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:28:40] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [21:28:46] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [21:30:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxsd7 [21:30:21] [02miraheze/puppet] 07paladox 0396e1da3 - Update prometheus-php-fpm.systemd [21:32:46] PROBLEM - misc1 Puppet on misc1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [21:34:46] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:12:16] Oh, the bug preventing settings from being changed means I can't change the license from CC-BY-SA to CC-BY. [22:12:46] Though I'll try to do so by just noting it on the community noticeboard for now. [22:24:59] what bug? [22:31:28] I rember seeing one mentioned, though it might have been specific to changes to the logo not saving. [22:31:38] Though it's the same form, so I figured it'd be the same bug. [22:32:02] I'll try looking for it soon, but right now I'm trying to swim some before the pool closes for winter. [22:40:32] MDude: it’s possible but it would be a different bug [22:40:44] Cc JohnLewis ^^ [22:41:25] it sounds like thr bug resolved several days ago