[01:07:17] RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 4442156 files, 398.9GB, 2019-06-16 01:05:00 (2.3 minutes ago) [01:13:42] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 3.47, 2.04, 0.93 [01:19:39] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.74, 1.95, 1.28 [01:21:40] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.18, 2.03, 1.39 [01:27:38] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.44, 1.85, 1.55 [01:29:38] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.18, 1.63, 1.50 [01:43:44] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 3.70, 2.49, 1.84 [01:47:38] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.28, 1.86, 1.73 [01:49:38] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.23, 1.66, 1.68 [01:53:38] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.67, 2.18, 1.88 [01:55:38] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.22, 1.78, 1.77 [01:57:38] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.15, 1.57, 1.69 [02:10:33] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.05, 1.81, 1.72 [02:16:21] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.19, 1.97, 1.87 [02:18:17] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.48, 1.44, 1.69 [02:19:26] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 79592 files, 1.998GB, 2019-06-16 02:18:00 (1.4 minutes ago) [02:24:09] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.29, 1.93, 1.80 [02:26:05] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.35, 1.70, 1.73 [02:26:36] RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 4066 files, 8.300MB, 2019-06-16 02:25:00 (1.6 minutes ago) [02:28:01] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.41, 1.58, 1.68 [02:31:57] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:32:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:32:29] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.009 second response time [02:32:32] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 42% [02:32:38] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [02:32:50] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [02:33:41] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 95% [02:33:57] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [02:34:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [02:34:32] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [02:36:32] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 41% [02:36:59] aware of ^^ [02:37:33] PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 50% [02:38:32] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 39% [02:40:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [02:42:29] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 17% [02:43:38] RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 347 bytes in 0.013 second response time [02:43:40] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 24616 MB (6% inode=96%); [02:45:54] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [02:46:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [02:46:53] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4227 bytes in 0.030 second response time [02:46:56] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [02:47:09] !log restarted mysql (at 03:43 bst) [02:48:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 52% [02:49:18] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [02:50:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [02:51:14] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 50% [02:51:54] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17725 bytes in 0.080 second response time [02:51:57] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [02:52:04] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:52:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:52:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 51% [02:52:29] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41136 bytes in 0.119 second response time [02:52:34] !log <+paladox> !log restarted mysql (at 03:43 bst) [02:52:35] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5805 bytes in 0.028 second response time [02:52:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:52:38] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:52:45] !log due to out of space on db4 [02:52:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:52:50] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:52:53] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19074 bytes in 0.128 second response time [02:52:56] RECOVERY - db4 MySQL on db4 is OK: Uptime: 236 Threads: 63 Questions: 47793 Slow queries: 3267 Opens: 3804 Flush tables: 1 Open tables: 800 Queries per second avg: 202.512 [02:53:10] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 8% [02:53:41] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 1% [02:54:29] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [02:56:01] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjakw [02:56:03] [02miraheze/ssl] 07paladox 03601479b - Update public.revi.wiki.crt [02:56:56] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjakr [02:56:57] [02miraheze/ssl] 07paladox 03d295947 - Update udc-japan.xyz.crt [02:57:44] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjak6 [02:57:46] [02miraheze/ssl] 07paladox 0372024a8 - Update zw.fontainebleau-avon.fr.crt [03:01:27] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_32 [+0/-0/±2] 13https://git.io/fjakP [03:01:28] [02miraheze/mediawiki] 07paladox 037a4230e - Update Tweeki [03:01:30] [02miraheze/mediawiki] 07paladox 034d61420 - Merge branch 'REL1_32' of github.com:miraheze/mediawiki into REL1_32 [03:03:20] RECOVERY - udc-japan.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'udc-japan.xyz' will expire on Sat 14 Sep 2019 01:56:32 AM GMT +0000. [03:03:25] RECOVERY - public.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'public.revi.wiki' will expire on Sat 14 Sep 2019 01:55:27 AM GMT +0000. [03:04:31] RECOVERY - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'zw.fontainebleau-avon.fr' will expire on Sat 14 Sep 2019 01:57:24 AM GMT +0000. [03:04:39] [02miraheze/mediawiki] 07paladox pushed 034 commits to 03REL1_33 [+0/-0/±5] 13https://git.io/fjakX [03:04:41] [02miraheze/mediawiki] 07reedy 03f938437 - Setup RELEASE-NOTES for commits after 1.33.0-rc.0 Change-Id: I072a0b01fa7dec9fd86a68277fe70c18184de992 [03:04:42] [02miraheze/mediawiki] 07kghbln 03bc572f8 - Installer: Update link to PHP intl away from old PECL package PHP 5.5 was the last version supported by PECL intl package. Now the PHP intl extension is used instead. Bug: T225558 Change-Id: I68cb7a549c899e69da9a8cfea5a69b9acb41e8ae (cherry picked from commit 7f0f6af2902cb7cf1406df5b8ee8cd12a5a88f1f) [03:04:44] [02miraheze/mediawiki] 07paladox 03da6da00 - Update Tweeki [03:04:45] [02miraheze/mediawiki] 07paladox 031aaf513 - Merge branch 'REL1_33' of https://github.com/wikimedia/mediawiki into REL1_33 [03:04:46] [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [03:08:27] !log running lc on mw* and test1 [03:08:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:10:50] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:14:53] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:02:49] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.43, 2.41, 0.99 [04:03:20] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:04:13] PROBLEM - test1 Current Load on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:09:11] PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 0.61, 3.76, 2.72 [04:10:08] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.12, 1.46, 1.22 [04:10:58] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.20, 1.53, 1.41 [04:11:09] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.24, 2.61, 2.41 [05:47:21] [02miraheze/landing] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaLW [05:47:22] [02miraheze/landing] 07Reception123 03203234a - fix typo [05:57:07] !log running wikibackups (./wikibackups.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php) [05:57:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:43:40] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21490 MB (5% inode=96%); [06:47:40] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 22276 MB (6% inode=96%); [06:49:40] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 20054 MB (5% inode=96%); [06:51:40] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 22829 MB (6% inode=96%); [06:53:40] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 20480 MB (5% inode=96%); [07:11:40] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 23720 MB (6% inode=96%); [10:35:40] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21963 MB (5% inode=96%); [11:25:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjamx [11:25:11] [02miraheze/services] 07MirahezeSSLBot 03ab07b97 - BOT: Updating services config for wikis [13:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaOs [13:25:12] [02miraheze/services] 07MirahezeSSLBot 03de6c9c2 - BOT: Updating services config for wikis [14:07:30] Is there some way to clean cache on wiki? https://isv.miraheze.org/wiki/ is being redirected to wrong title, but I already fixed wiki name after Unicode conversion [14:07:31] [ Medžuviki, svobodna enciklopedija ] - isv.miraheze.org [16:50:25] [02mw-config] 07The-Voidwalker opened pull request 03#2685: update mwp blacklist - 13https://git.io/fjaGb [16:51:13] [02mw-config] 07Reception123 closed pull request 03#2685: update mwp blacklist - 13https://git.io/fjaGb [16:51:14] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaGN [16:51:16] [02miraheze/mw-config] 07The-Voidwalker 03c98d4ab - update mwp blacklist (#2685) [17:02:31] [02mw-config] 07The-Voidwalker opened pull request 03#2686: upd moderation install - 13https://git.io/fjaZI [17:05:16] [02mw-config] 07paladox closed pull request 03#2686: upd moderation install - 13https://git.io/fjaZI [17:05:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaZY [17:05:19] [02miraheze/mw-config] 07The-Voidwalker 0317c6912 - upd moderation install (#2686) * upd moderation install * add automoderated and moderator groups [19:05:40] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1419 [19:24:31] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [19:24:33] !log remove read from horridreceptionwikiswiki [19:24:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:24:40] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown) [19:25:08] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 326 failures. Last run 2 minutes ago with 326 failures. Failed resources (up to 3 shown): File[espiral.org_private],File[wiki.lbcomms.co.za],File[wiki.lbcomms.co.za_private],File[www.openonderwijs.org] [19:25:09] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 318 failures. Last run 2 minutes ago with 318 failures. Failed resources (up to 3 shown) [19:25:26] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 306 failures. Last run 2 minutes ago with 306 failures. Failed resources (up to 3 shown) [19:25:34] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 3 minutes ago with 24 failures. Failed resources (up to 3 shown) [19:25:54] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [19:26:24] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 306 failures. Last run 3 minutes ago with 306 failures. Failed resources (up to 3 shown) [19:26:32] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 299 failures. Last run 3 minutes ago with 299 failures. Failed resources (up to 3 shown) [19:26:32] !log running /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --removeperms=read '*' (mw1) [19:26:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:26:47] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 302 failures. Last run 4 minutes ago with 302 failures. Failed resources (up to 3 shown) [19:27:27] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 234 failures. Last run 3 minutes ago with 234 failures. Failed resources (up to 3 shown): File[allthetropes.org],File[allthetropes.org_private],File[aman.info.tm],File[aman.info.tm_private] [19:29:18] ^ paladox does that have anything to do with you? :P [19:29:38] Reception123 it's the daily GC (and OOM) [19:29:47] GC? [19:29:53] Garbage collection [19:29:56] oh [19:32:25] !log delete from mw_permissions where perm_group = 'modifyGroupPermission.php'; on db4 [19:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:32:41] !log /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --removeperms=read \"*\" (mw1) [19:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:33:33] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:33:54] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:34:02] !log drop delete from mw_permissions where perm_group = '"*"'; [19:34:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:34:24] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:34:29] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:34:30] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:34:40] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:34:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:35:08] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:35:10] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:35:21] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:35:26] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:35:28] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:36:18] !log delete from mw_permissions where perm_group = '\'*\''; on db4 [19:36:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:40:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaCe [19:40:13] [02miraheze/services] 07MirahezeSSLBot 03e778422 - BOT: Updating services config for wikis [19:43:23] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [19:51:48] !log UPDATE mw_permissions SET perm_permissions = REPLACE(perm_permissions, "read", "") WHERE perm_group = "*"; on db4 (1525 rows updated) [19:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:53:21] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:07:23] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 602 [20:08:42] !log upgrade phabricator on misc4 [20:08:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaCP [20:15:12] [02miraheze/services] 07MirahezeSSLBot 0347e41fd - BOT: Updating services config for wikis [20:17:23] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [21:03:27] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaWy [21:03:28] [02miraheze/ManageWiki] 07paladox 032683c24 - Fix i18n for "managewiki-header-settings" [21:07:42] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjaW9 [21:07:43] [02miraheze/mediawiki] 07paladox 03694cd42 - Update MW [21:08:21] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjaWH [21:08:22] [02miraheze/ManageWiki] 07JohnFLewis 03f0062a0 - it's a double array [21:34:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjalv [21:34:39] [02miraheze/puppet] 07paladox 03571d90b - Redirect /Stewards'_noticeboard to /wiki/Stewards'_noticeboard under mobile [22:27:47] paladox: i have now seen the most ugly version string, see the php version on special:Version [22:28:21] heh [22:28:24] "7.2.19-1+0~20190531112637.22+stretch~1.gbp75765b (fpm-fcgi)" [23:01:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fja8e [23:01:36] [02miraheze/puppet] 07paladox 03d9ef8c7 - Re enable ssl renewal system on mw1