[00:10:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:10:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [00:12:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:12:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:15:03] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 17.17, 10.80, 6.64 [00:15:15] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 12.59, 7.55, 4.07 [00:16:12] wow that load average [00:19:50] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.94, 6.91, 4.08 [00:19:54] Voidwalker: could be worse [00:20:10] I had over 90 once, at Orain's mailserver [00:20:36] hah, that's kinda impressive [00:21:50] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.35, 5.76, 4.02 [00:23:15] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 2.48, 6.51, 5.42 [00:27:02] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.00, 7.46, 7.66 [00:33:02] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.13, 5.12, 6.52 [00:43:27] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.01, 1.61, 1.31 [00:45:26] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.20, 1.38, 1.26 [01:49:03] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 6.41, 11.88, 7.91 [01:53:02] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.54, 7.06, 6.81 [01:55:03] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 2.65, 5.64, 6.32 [02:03:47] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.16, 1.76, 1.55 [02:05:46] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.35, 1.59, 1.51 [03:31:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.85, 1.79, 1.44 [03:33:47] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.48, 1.90, 1.51 [03:35:45] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.43, 1.73, 1.50 [03:37:44] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.17, 1.47, 1.42 [03:49:34] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 1.17, 2.45, 1.97 [03:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQ2O [03:55:13] [02miraheze/services] 07MirahezeSSLBot 03f64140a - BOT: Updating services config for wikis [04:42:04] Hello prettypig852! If you have any questions feel free to ask and someone should answer soon. [04:43:08] Hi [04:43:43] Anybody here [04:43:56] Seek help from HongKong [05:15:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQaW [05:15:13] [02miraheze/services] 07MirahezeSSLBot 0337d1ae6 - BOT: Updating services config for wikis [05:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQa8 [05:20:14] [02miraheze/services] 07MirahezeSSLBot 03aa2a689 - BOT: Updating services config for wikis [06:15:03] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.18, 6.40, 5.39 [06:17:02] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.56, 6.04, 5.38 [06:26:41] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3047 MB (12% inode=94%); [07:37:07] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Fri 23 Aug 2019 07:34:56 AM GMT +0000). [07:37:17] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.10, 0.69, 2.00 [07:37:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQwm [07:37:22] [02miraheze/ssl] 07MirahezeSSLBot 03c6ab7e8 - Bot: Update SSL cert for wiki.counterculturelabs.org [07:41:17] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.89, 0.61, 1.64 [07:45:07] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Tue 05 Nov 2019 06:37:14 AM GMT +0000. [08:02:30] can i get some help from another wiki creator? [10:15:17] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.19, 1.50, 1.00 [10:17:17] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.50, 1.06, 0.89 [11:03:17] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.13, 1.30, 0.76 [11:05:17] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.56, 1.03, 0.73 [11:20:13] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.21, 1.66, 1.08 [11:22:12] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.84, 1.73, 1.17 [11:24:11] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.24, 1.57, 1.18 [11:35:23] PROBLEM - espiral.org - Gandi on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:19] PROBLEM - espiral.org - Gandi on sslhost is WARNING: WARNING - Certificate 'espiral.org' expires in 16 day(s) (Fri 23 Aug 2019 11:59:59 PM GMT +0000). [12:10:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQiG [12:10:13] [02miraheze/services] 07MirahezeSSLBot 03b7e1ce7 - BOT: Updating services config for wikis [12:46:16] PROBLEM - lizardfs3 Disk Space on lizardfs3 is WARNING: DISK WARNING - free space: / 16888 MB (10% inode=98%); [12:46:34] PROBLEM - lizardfs2 Disk Space on lizardfs2 is WARNING: DISK WARNING - free space: / 16886 MB (10% inode=98%); [13:33:57] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.97, 1.41, 1.14 [13:37:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.59, 1.23, 1.14 [14:03:13] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:15] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:20] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:34] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 2 minutes ago with 15 failures. Failed resources (up to 3 shown) [14:03:47] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:55] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:13] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 203 failures. Last run 2 minutes ago with 203 failures. Failed resources (up to 3 shown) [14:04:15] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:18] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [14:04:21] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:22] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 3 minutes ago with 16 failures. Failed resources (up to 3 shown) [14:04:23] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:28] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [14:04:35] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:37] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [14:04:48] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 3 minutes ago with 18 failures. Failed resources (up to 3 shown) [14:04:57] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:05:02] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 201 failures. Last run 3 minutes ago with 201 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [14:12:00] JohnLewis, paladox: is that normal? ^ [14:12:05] yup [14:12:10] it'll recover [14:12:11] PROBLEM - ahmsaqib.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:34] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:13:48] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:13:57] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:14:15] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:14:17] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:14:22] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:14:23] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [14:14:28] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [14:14:34] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:14:35] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:14:36] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:14:47] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:14:57] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:15:13] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:15:16] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:15:20] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:16:04] thats better [14:16:13] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:16:21] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:17:10] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:22:44] GC? [14:23:03] Voidwalker: see the IRC page [14:24:03] yeah, I see all the icinga alerts, just asking if it was garbage collection [14:24:21] Voidwalker: oh I think so - Paladox? [14:24:31] yup [14:24:36] Voidwalker: thought you meant like Group Contact or something [14:24:46] ah, :P [14:26:13] you can confirm that at https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=puppet1.miraheze.org&var-port=9100 [14:26:14] [ Grafana ] - grafana.miraheze.org [15:04:41] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [16:59:11] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQyf [16:59:13] [02miraheze/mw-config] 07Reception123 035713cf3 - allow users to generate image dumps [17:22:48] @Stewards (or at least meta admin) : PM [17:24:14] @RhinosF1 I'm available now so as a Meta admin you can PM 😃 [17:26:02] @Reception123 PM sent [17:26:42] replied [17:44:21] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 4.63, 2.61, 1.48 [17:45:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQya [17:45:49] [02miraheze/mw-config] 07paladox 03a39f62b - Compress crystalmaidenswiki [17:48:29] !log starting import for crystalmaidenswiki - T4577 [17:48:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:50:00] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 25436 MB (6% inode=96%); [18:03:02] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.11, 6.70, 5.11 [18:05:03] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.24, 6.76, 5.34 [18:09:34] Hello [18:12:04] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [18:12:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjQyQ [18:12:07] [02miraheze/mw-config] 07paladox 03e485d01 - Unset cargo db variable This is uneeded as it uses the wiki dbname rather then a shared db and duplicates things. [18:12:09] [02mw-config] 07paladox opened pull request 03#2715: Unset cargo db variable - 13https://git.io/fjQy7 [18:24:26] AmandaCath: hi [18:28:37] Hi Reception123 [18:31:21] AmandaCath: hello [18:32:22] Hi RhinosF1 [18:33:18] AmandaCath: How R u [18:34:38] Good [18:34:50] Getting ready for some incoming severe weather [18:38:11] AmandaCath: we'll stay safe, and let's hope the power keeps up so you can stay wiki-ing [18:40:40] Yeah hopefully... last year I lost power frequently during severe weather [18:41:22] But I'm now on hydro power from a nearby waterfall and dam, so it should be more reliable [18:41:51] The natural gas infrastructure seemed to be quite unreliable [18:42:54] AmandaCath: better for everything then [18:46:35] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±2] 13https://git.io/fjQSE [18:46:37] [02miraheze/mediawiki] 07paladox 03a6322d1 - Update Cargo and PF [18:49:29] [02mw-config] 07NDKilla commented on pull request 03#2715: Unset cargo db variable - 13https://git.io/fjQSg [18:49:39] my git-search-fu is bad [18:50:38] [02mw-config] 07paladox commented on pull request 03#2715: Unset cargo db variable - 13https://git.io/fjQSa [18:52:52] o/ submodules! [18:53:15] Did GitHub update their display/usage of submodules? https://git.io/fjQSE looks really different than I remember [18:53:17] [ Comparing 6dac9498b62a...a6322d1d50e4 · miraheze/mediawiki · GitHub ] - git.io [18:53:44] Also we're considering switching to github instead of phabricator as the authoritative host at work and submodules would probably be nice for things. xD [18:53:51] hi [18:54:16] Paladox: what does that mean to us? [18:54:40] PuppyKun that's been like that for a while [18:54:54] RhinosF1 im not sure what you mean? [18:55:09] PuppyKun you'll like gerrit then :P [18:55:13] better submodule integration [18:55:52] Paladox: switching to GitHub as the authoritative host at work? [18:56:01] Hispano76_: hello [18:56:37] RhinosF1 that's not us [18:56:50] that's PuppyKun work (he has a real job :)) [18:58:13] Paladox: okay!! Don't most adults here have 'real jobs' (although personally Miraheze is a fun, volunteer job (still real)) [18:59:34] Well i know some of the staff do :) [19:00:49] Yeah apparently github has free unlimited private repos now [19:01:07] and that's more or less the only thing we used phab for [19:02:12] so tomorrow is going to be migrate everything to github and then something like a2dissite phab && service apache2 reload [19:04:05] [02mw-config] 07paladox closed pull request 03#2715: Unset cargo db variable - 13https://git.io/fjQy7 [19:04:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQSP [19:04:08] [02miraheze/mw-config] 07paladox 03585235e - Unset cargo db variable (#2715) This is uneeded as it uses the wiki dbname rather then a shared db and duplicates things. [19:04:09] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [19:04:11] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [19:16:39] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [19:41:17] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.50, 0.83, 1.86 [19:45:17] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.14, 0.85, 1.62 [20:07:06] Is there any reason we can't have https://www.mediawiki.org/wiki/Manual:$wgUniversalEditButton on Miraheze wikis? [20:07:06] [ Manual:$wgUniversalEditButton - MediaWiki ] - www.mediawiki.org [20:09:09] Oh it's true by default anyway!! [21:35:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjQQY [21:35:15] [02miraheze/services] 07MirahezeSSLBot 03ff51193 - BOT: Updating services config for wikis [22:45:28] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.20, 7.31, 6.12 [22:49:22] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.95, 6.67, 6.14