[00:48:18] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 40624 MB (11% inode=96%); [01:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jewa4 [01:35:09] [02miraheze/services] 07MirahezeSSLBot 0352685b9 - BOT: Updating services config for wikis [02:50:14] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40290 MB (10% inode=96%); [03:43:22] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [03:49:11] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Thu 23 Jan 2020 09:51:20 AM GMT +0000. [04:11:35] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:15:49] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Thu 23 Jan 2020 09:51:20 AM GMT +0000. [04:25:21] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:29:10] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Thu 23 Jan 2020 09:51:20 AM GMT +0000. [04:37:49] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:39:44] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Thu 23 Jan 2020 09:51:20 AM GMT +0000. [04:54:08] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:56:02] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Thu 23 Jan 2020 09:51:20 AM GMT +0000. [05:35:29] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [05:41:12] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Thu 23 Jan 2020 09:51:20 AM GMT +0000. [05:46:58] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [05:52:45] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Thu 23 Jan 2020 09:51:20 AM GMT +0000. [05:56:44] PROBLEM - wiki.hackdown.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [06:26:50] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2920 MB (12% inode=94%); [08:06:14] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 40480 MB (11% inode=96%); [09:14:15] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40261 MB (10% inode=96%); [12:41:58] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [15:08:17] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 40315 MB (11% inode=96%); [15:15:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewSw [15:15:11] [02miraheze/services] 07MirahezeSSLBot 03c11865b - BOT: Updating services config for wikis [15:20:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewSi [15:20:09] [02miraheze/services] 07MirahezeSSLBot 03894c76f - BOT: Updating services config for wikis [15:20:21] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40289 MB (10% inode=96%); [15:20:46] Reception123: ^ [15:20:55] Can you purge bin logs? [15:23:44] .in 30 minutes annoy someone to fix db4 [15:23:45] RhinosF1: Okay, I will set the reminder for: 2019-11-12 - 15:53:44GMT [15:53:44] RhinosF1: annoy someone to fix db4 [15:54:48] JohnLewis: db4 is running out of disk space - can you consider purging bin logs? [15:55:35] if it's disk space is warning, I'd prefer to leave it [15:56:29] because warning should be at 40G left, which isn't really a bad number tbf [15:56:40] JohnLewis: why? It'll soon become critical and the longer we leave it then the more of a chance we miss it running out and it falls over [15:56:46] It goes down very quickly [15:56:59] it shouldn't use 40G quickly [15:57:17] plus binlogs are useful and critical, removing them should only be in a critical situation [15:57:20] It used to go down pretty quick before we had db5 [15:57:38] but it doesn't now [15:57:47] what happened then and what happens now are two different things [15:58:02] and back then, removing binlogs took it to warning from critical [15:58:05] I'll keep an eye out [15:58:33] okay [16:00:49] .in 90 minutes check disk space on DBs [16:00:49] RhinosF1: Okay, I will set the reminder for: 2019-11-12 - 17:30:49GMT [16:16:48] I’m very certain that 40gb will last days (even maybe another week) [16:27:05] JohnLewis: can you take down the CoCC vote Notice [16:33:15] Yeah it’s gone now [16:33:27] as well as me /me goes to meeting [16:38:03] JohnLewis: thx and congrats [16:56:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewHX [16:56:54] [02miraheze/puppet] 07paladox 03aa176f2 - Update mfsmaster.cfg.erb [16:57:37] !log restart lizardfs-master on lizardfs6 [16:58:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:30:49] RhinosF1: check disk space on DBs [18:33:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [18:35:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:51:42] revi: you around? [19:14:48] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jew5Q [19:14:50] [02miraheze/mw-config] 07Pix1234 03ea14bf0 - + preloader [19:15:44] Zppix syntax error [19:15:51] where [19:15:56] miraheze/mw-config/master/ea14bf0 - zppix1 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/611016913 [19:17:00] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jew5A [19:17:02] [02miraheze/mw-config] 07Pix1234 0381d8ec7 - syntax [19:17:10] found it paladox [19:18:10] miraheze/mw-config/master/81d8ec7 - zppix1 The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/611017782 [19:18:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jew5p [19:18:15] [02miraheze/mw-config] 07paladox 03ec347d9 - fix [19:19:36] [02miraheze/mediawiki] 07Pix1234 pushed 032 commits to 03REL1_33 [+1/-0/±7] 13https://git.io/Jewdv [19:19:37] [02miraheze/mediawiki] 07Pix1234 033795ab3 - + preloader [19:19:39] [02miraheze/mediawiki] 07Pix1234 030fa35db - Merge branch 'REL1_33' of https://github.com/miraheze/mediawiki into REL1_33 [19:19:40] [ GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. ] - github.com [19:20:19] Zppix done [19:20:26] paladox: ? [19:20:31] I fixed the error [19:20:39] oh [19:20:42] you did it [19:20:47] paladox: lol [19:20:51] uh [19:20:52] actually [19:20:57] there was 2 syntax errors [19:21:08] paladox: did you get the other one? [19:21:12] yup [19:21:24] paladox: ok weird notificio didnt say anything [19:21:38] it won't if it found one of the errors first [19:21:55] paladox: no i meant about your commit [19:22:02] oh [19:22:50] [02miraheze/mediawiki] 07Pix1234 pushed 032 commits to 03REL1_34 [+1/-0/±7] 13https://git.io/Jewdt [19:22:51] [02miraheze/mediawiki] 07Pix1234 03f1d8035 - + preloader [19:22:52] [02miraheze/mediawiki] 07Pix1234 031954336 - Merge branch 'REL1_34' of https://github.com/miraheze/mediawiki into REL1_34 [19:22:53] [ GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. ] - github.com [19:29:20] !log rebuild LC on mw[123] and test1 [19:29:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:31:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [19:33:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:36:32] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 41232 MB (11% inode=96%); [19:43:30] [02miraheze/mw-config] 07Pix1234 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jewd6 [19:43:32] [02miraheze/mw-config] 07Pix1234 037fc13a4 - no need to set test1wiki as true for preloader anymore [19:43:33] [02miraheze/mw-config] 07Pix1234 033459418 - Merge branch 'master' of https://github.com/miraheze/mw-config [19:43:34] [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com [20:00:54] hi [20:01:29] hi [20:01:35] Hey [20:23:12] [02puppet] 07paladox synchronize pull request 03#1144: gluster: Add secure-access file - 13https://git.io/JewIC [20:23:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JewFz [20:23:15] [02miraheze/puppet] 07paladox 034e57909 - Update secure-access [20:24:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JewF2 [20:24:21] [02miraheze/puppet] 07paladox 03b79d17a - Update mount.pp [20:24:22] [02puppet] 07paladox synchronize pull request 03#1144: gluster: Add secure-access file - 13https://git.io/JewIC [20:27:48] [02puppet] 07paladox closed pull request 03#1144: gluster: Add secure-access file - 13https://git.io/JewIC [20:27:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JewFw [20:27:51] [02miraheze/puppet] 07paladox 03af0d225 - gluster: Add secure-access file (#1144) * gluster: Add secure-access file What this does is: * Sets ssl certificate, CA and private key instead of using a default file. * Sets ssl-cert-depth to 2. * Create secure-access * Update secure-access * Update secure-access * Update secure-access * Update mount.pp [20:27:53] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [20:27:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [20:28:16] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.22, 2.74, 1.88 [20:28:39] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 4.02, 3.17, 2.15 [20:32:39] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 2.72, 3.52, 2.55 [20:34:17] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.37, 3.73, 2.67 [20:34:39] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 2.15, 2.93, 2.45 [20:36:18] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.67, 3.26, 2.62 [20:42:21] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 7.52, 5.01, 3.43 [20:42:46] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 8.20, 5.01, 3.42 [20:48:14] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.67, 3.27, 3.21 [20:50:39] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 2.68, 3.49, 3.36 [20:52:39] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 2.15, 2.97, 3.18 [21:37:44] .at 15:10 UTC You have a to-do list [21:37:44] RhinosF1: Okay, I will set the reminder for: 2019-11-13 - 15:09:59UTC [21:38:17] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40274 MB (10% inode=96%); [22:05:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewNP [22:05:54] [02miraheze/puppet] 07paladox 038a3b15d - Enable gluster on mw* [22:07:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewNM [22:07:40] [02miraheze/puppet] 07paladox 03ceea59c - Update common.yaml [22:09:49] PROBLEM - www.itpedia.sk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itpedia.sk' expires in 15 day(s) (Thu 28 Nov 2019 10:07:19 PM GMT +0000). [22:10:10] PROBLEM - itpedia.sk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itpedia.sk' expires in 15 day(s) (Thu 28 Nov 2019 10:07:19 PM GMT +0000). [22:10:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewNH [22:10:35] [02miraheze/ssl] 07MirahezeSSLBot 030a170f5 - Bot: Update SSL cert for itpedia.sk [22:13:34] PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.scvo.org' expires in 15 day(s) (Thu 28 Nov 2019 10:11:09 PM GMT +0000). [22:13:40] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:13:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewN5 [22:13:49] [02miraheze/ssl] 07MirahezeSSLBot 03d60c1a9 - Bot: Update SSL cert for wiki.scvo.org [22:15:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JewNb [22:15:45] [02miraheze/puppet] 07paladox 0307689f5 - gluster: Adjust default options [22:15:46] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [22:15:48] [02puppet] 07paladox opened pull request 03#1145: gluster: Adjust default options - 13https://git.io/JewNA [22:15:59] [02puppet] 07paladox closed pull request 03#1145: gluster: Adjust default options - 13https://git.io/JewNA [22:16:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewNx [22:16:02] [02miraheze/puppet] 07paladox 034278ddc - gluster: Adjust default options (#1145) [22:18:00] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [22:18:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [22:23:31] PROBLEM - wiki.cloudytheology.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cloudytheology.com' expires in 15 day(s) (Thu 28 Nov 2019 10:19:41 PM GMT +0000). [22:23:37] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Mon 10 Feb 2020 09:13:41 PM GMT +0000. [22:23:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewAT [22:23:46] [02miraheze/ssl] 07MirahezeSSLBot 03d7bbba5 - Bot: Update SSL cert for wiki.cloudytheology.com [22:23:49] RECOVERY - www.itpedia.sk - LetsEncrypt on sslhost is OK: OK - Certificate 'itpedia.sk' will expire on Mon 10 Feb 2020 09:10:26 PM GMT +0000. [22:24:10] RECOVERY - itpedia.sk - LetsEncrypt on sslhost is OK: OK - Certificate 'itpedia.sk' will expire on Mon 10 Feb 2020 09:10:26 PM GMT +0000. [22:26:39] PROBLEM - wiki.xfbs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.xfbs.net' expires in 15 day(s) (Thu 28 Nov 2019 10:23:24 PM GMT +0000). [22:26:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewAL [22:26:53] [02miraheze/ssl] 07MirahezeSSLBot 03d6fb070 - Bot: Update SSL cert for wiki.xfbs.net [22:32:37] RECOVERY - wiki.xfbs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xfbs.net' will expire on Mon 10 Feb 2020 09:26:46 PM GMT +0000. [22:33:31] RECOVERY - wiki.cloudytheology.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cloudytheology.com' will expire on Mon 10 Feb 2020 09:23:39 PM GMT +0000. [22:39:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JewAW [22:39:22] [02miraheze/puppet] 07paladox 0326529a7 - gluster: Add systemd script to mount [22:39:24] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [22:39:25] [02puppet] 07paladox opened pull request 03#1146: gluster: Add systemd script to mount - 13https://git.io/JewAl [22:40:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+1/-0/±0] 13https://git.io/JewA4 [22:40:43] [02miraheze/puppet] 07paladox 035566f58 - Create gluster-mount.systemd.erb [22:40:44] [02puppet] 07paladox synchronize pull request 03#1146: gluster: Add systemd script to mount - 13https://git.io/JewAl [22:41:19] sorry for the join & rejoin spam, i was configuring some stuff [22:41:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JewAB [22:41:38] [02miraheze/puppet] 07paladox 034dc5d12 - Update gluster-mount.systemd.erb [22:41:40] [02puppet] 07paladox synchronize pull request 03#1146: gluster: Add systemd script to mount - 13https://git.io/JewAl [22:51:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewAX [22:51:32] [02miraheze/puppet] 07paladox 03b248498 - mediawiki: Ensure glusters mount is mounted [22:55:38] [02puppet] 07paladox closed pull request 03#1146: gluster: Add systemd script to mount - 13https://git.io/JewAl [22:55:40] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [22:55:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [23:10:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewxL [23:10:53] [02miraheze/puppet] 07paladox 0344a6c7f - mediawiki: Add static-temp back [23:14:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jewx3 [23:14:18] [02miraheze/mw-config] 07paladox 031e51302 - Move allthetropeswiki and metawiki to the new mount [23:17:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jewxn [23:17:34] [02miraheze/puppet] 07paladox 036f90578 - Update lizardfs6.yaml [23:22:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewxW [23:22:33] [02miraheze/mw-config] 07paladox 03ef87ebc - move testwiki over to new mount [23:24:09] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:54:38] !log install fio on lizardfs6 [23:54:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master