[00:57:59] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fj2Lp
[00:58:00] <Not-7290>	 [02miraheze/mw-config] 07paladox 03fecbe74 - Include "--logs --include-files --uploads" in the DataDump dumps This doesn't include the actual file so no copyright concerns. It only includes the log.
[00:58:02] <Not-7290>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[00:58:03] <Not-7290>	 [02mw-config] 07paladox opened pull request 03#2683: Include "--logs --include-files --uploads" in the DataDump dumps - 13https://git.io/fj2Lh
[01:01:09] <Not-7290>	 [02mw-config] 07paladox synchronize pull request 03#2683: Include "--logs --include-files --uploads" in the DataDump dumps - 13https://git.io/fj2Lh
[01:01:11] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fj2te
[01:01:13] <Not-7290>	 [02miraheze/mw-config] 07paladox 0360938a6 - Update LocalExtensions.php
[01:01:21] <Not-7290>	 [02mw-config] 07paladox edited pull request 03#2683: Include "--logs --uploads" in the DataDump dumps - 13https://git.io/fj2Lh
[01:04:46] <paladox>	 !log running dumpBackup.php for allthetropes (on mw1)
[01:04:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:23:04] <icinga-miraheze>	 PROBLEM - test1 php-fpm on test1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.3'
[01:23:09] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures
[01:27:10] <icinga-miraheze>	 PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[04:02:41] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 10.25, 4.32, 1.85
[04:03:24] <icinga-miraheze>	 PROBLEM - misc4 Current Load on misc4 is CRITICAL: CRITICAL - load average: 5.76, 3.40, 1.51
[04:05:21] <icinga-miraheze>	 RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 1.38, 2.58, 1.43
[04:08:35] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.39, 1.76, 1.51
[04:10:34] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.34, 1.27, 1.35
[05:40:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj2mo
[05:40:13] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 034937ce1 - BOT: Updating services config for wikis
[05:50:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj2mS
[05:50:13] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03b1fdedf - BOT: Updating services config for wikis
[06:25:10] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj2Yf
[06:25:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03a75dcc7 - BOT: Updating services config for wikis
[08:02:37] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 11.80, 4.43, 1.75
[08:03:28] <icinga-miraheze>	 PROBLEM - misc4 Current Load on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:04:00] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.42, 1.88, 0.87
[08:05:16] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:05:29] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:07:28] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 hours ago with 0 failures
[08:08:04] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[08:09:16] <icinga-miraheze>	 PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:09:29] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures
[08:11:19] <icinga-miraheze>	 RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 29%
[08:12:20] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[08:12:47] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[08:13:40] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:14:04] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[08:14:20] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[08:14:47] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Diff, 4265 files, 86.67MB, 2019-05-22 21:00:00 (3.1 weeks ago)
[08:15:37] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 hours ago with 0 failures
[08:19:30] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:21:25] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'udc-japan.xyz' expires in 13 day(s) (Wed 26 Jun 2019 09:55:13 PM GMT +0000).
[08:22:41] <icinga-miraheze>	 PROBLEM - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:24:36] <icinga-miraheze>	 PROBLEM - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zw.fontainebleau-avon.fr' expires in 13 day(s) (Wed 26 Jun 2019 09:59:51 PM GMT +0000).
[08:25:30] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:27:25] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'udc-japan.xyz' expires in 13 day(s) (Wed 26 Jun 2019 09:55:13 PM GMT +0000).
[08:28:41] <icinga-miraheze>	 PROBLEM - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:29:35] <icinga-miraheze>	 PROBLEM - public.revi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:30:36] <icinga-miraheze>	 PROBLEM - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zw.fontainebleau-avon.fr' expires in 13 day(s) (Wed 26 Jun 2019 09:59:51 PM GMT +0000).
[08:30:46] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[08:31:29] <icinga-miraheze>	 PROBLEM - public.revi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'public.revi.wiki' expires in 10 day(s) (Sun 23 Jun 2019 06:42:19 PM GMT +0000).
[08:32:44] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[08:37:19] <icinga-miraheze>	 PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 0.30, 1.83, 3.66
[08:38:13] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.10, 0.81, 1.93
[08:39:16] <icinga-miraheze>	 RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.13, 1.27, 3.24
[08:42:09] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.06, 0.43, 1.52
[08:52:34] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.31, 0.37, 1.97
[08:56:34] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.09, 0.28, 1.57
[12:36:25] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5168
[14:28:11] <ZppixBot>	 Hello testuser6475! If you have any questions feel free to ask and someone should answer soon.
[15:07:04] <icinga-miraheze>	 RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 4 processes with command name 'php-fpm7.3'
[15:07:08] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[15:07:10] <icinga-miraheze>	 RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.009 second response time
[15:15:23] <Not-7290>	 [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj2lV
[15:15:24] <Not-7290>	 [02miraheze/DataDump] 07paladox 03dc3de5c - Remove use of APIListModules in extension.json The api we are using is not a query, so we doin't need APIListModules. Change-Id: I298b892502ebd34999190a66c8190df790e53b4d
[15:17:38] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fj2l1
[15:17:39] <Not-7290>	 [02miraheze/mediawiki] 07paladox 030355cd1 - Update DataDump
[15:19:20] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fj2lM
[15:19:21] <Not-7290>	 [02miraheze/mediawiki] 07paladox 0323bfa8e - Update DataDump
[15:21:53] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fj2lQ
[15:21:55] <Not-7290>	 [02miraheze/mediawiki] 07paladox 035a1381d - Remove duplicate Mask from .gitmodules
[15:23:04] <icinga-miraheze>	 PROBLEM - test1 php-fpm on test1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.3'
[15:23:08] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures
[15:23:10] <icinga-miraheze>	 PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[15:29:30] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[15:30:37] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[15:33:29] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[15:34:35] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[15:35:57] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[15:47:48] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 767
[16:03:38] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[16:46:17] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 7796
[17:48:10] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki sdiywiki --r="Deletion of pages in Category:CGS Serge kit image" --u="Reception123" /home/reception/sdiy-pages-delete.txt
[17:48:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:55:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj2Rs
[17:55:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 031ca5fc6 - BOT: Updating services config for wikis
[18:19:43] <Not-7290>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fj2RM
[18:19:45] <Not-7290>	 [02miraheze/ManageWiki] 07translatewiki 03aabeb73 - Localisation updates from https://translatewiki.net.
[18:26:04] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[18:26:20] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[18:27:40] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 26946 MB (7% inode=96%);
[18:37:55] <ZppixBot>	 Hello dt_! If you have any questions feel free to ask and someone should answer soon.
[18:38:04] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[18:38:20] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[19:17:30] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:19:25] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'udc-japan.xyz' expires in 13 day(s) (Wed 26 Jun 2019 09:55:13 PM GMT +0000).
[20:33:13] <MH-Discord>	 <Zppix> Paladox is upload by url disabled by default if so is it possible to enable per wiki (just curious)
[20:33:58] <paladox>	 @Zppix, it's disabled by default. Yes it's possible to enable it per wiki (through MW)
[20:34:19] <MH-Discord>	 <Zppix> Paladox i assume through mw means via mw-config repo?
[20:34:28] <paladox>	 nope, ManageWiki
[20:35:04] <MH-Discord>	 <Zppix> Paladox perhaps ManageWiki should be abbrevated as MW 😄 maybe MWiki or something xD
[20:35:19] <paladox>	 lol
[22:20:16] <paladox>	 !log running updateSearchIndexConfig.php for pointmanwiki
[22:20:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:21:41] <paladox>	 !log running forceSearchIndex.php for pointmanwiki
[22:21:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:41:23] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[22:49:52] <paladox>	 !log root@test1:/home/reception# rm -rf wikibackups
[22:49:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master