[00:24:25] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.28, 2.00, 1.62 [00:26:23] [02MirahezeMagic] 07The-Voidwalker opened pull request 03#135: Patch 11 - 13https://git.io/JJ3lk [00:26:27] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.63, 1.56, 1.51 [00:26:35] [02MirahezeMagic] 07The-Voidwalker edited pull request 03#135: restore previous changes and fixes - 13https://git.io/JJ3lk [00:29:25] Someone else mentioned Extension:PortableInfoboxes in another channel...think it was @el komodos drago. Thanks @Void for knowing what the likely problem was and helping Dimitri. 🙂 [00:51:50] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:57:53] :) [01:22:56] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.33, 2.03, 1.60 [01:26:52] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.86, 1.66, 1.56 [01:30:49] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.18, 1.97, 1.78 [01:32:50] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.75, 1.52, 1.63 [01:37:24] Who's there? [01:40:04] im here [01:43:08] Zppix: https://privado.miraheze.org/wiki/Data:Rafaela_L%C3%B3pez_Aguado_de_Ray%C3%B3n can you see the photograph or is it blurry? [01:43:09] [ Data:Rafaela López Aguado de Rayón - Privado ] - privado.miraheze.org [01:46:46] i can see it [01:47:09] hispano76 [01:47:56] Ok, thanks Zppix [01:48:03] np [02:01:57] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.54, 2.24, 1.79 [02:05:53] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.85, 1.70, 1.69 [03:33:53] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.01, 2.11, 1.54 [03:35:51] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.15, 1.76, 1.48 [03:37:48] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.90, 1.44, 1.39 [03:48:23] hello! any guesses as to why https://spcodex.wiki/wiki/Special:Statistics always reports 0 "Active users", despite having users listed at https://spcodex.wiki/wiki/Special:ActiveUsers ? [03:48:25] [ Statistics - SPCodex ] - spcodex.wiki [03:48:26] [ Active users list - SPCodex ] - spcodex.wiki [03:50:41] the only thing that bothers me is that it's at Special:CreateAccount, so it appears as if you're joining an inactive community [03:51:35] I could just override MediaWiki:Createacct-benefit-head3 , hehe [04:01:24] musikanimal: i can try running initSiteStats [04:01:55] that'd be great! [04:02:50] musikanimal: whats the dbname? [04:02:55] solarawiki [04:03:35] musikanimal: okay it ran now what does it say [04:03:49] still says zero :( [04:03:55] Hmmm [04:03:58] Thats weird [04:04:08] Could it be cache? [04:04:41] probably. It's been at zero since day one (late March) [04:05:24] I cant think of what else to try [04:05:26] it may be a cron job or something that got stuck or was never turned on. Just another guess [04:06:36] ShowJobs.php shows 0 for the wiki musikanimal [04:08:00] Could it be your cache? [04:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3zh [04:35:13] [02miraheze/services] 07MirahezeSSLBot 0383e44ca - BOT: Updating services config for wikis [06:24:58] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:33:09] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:56] RECOVERY - services2 APT on services2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:02:22] Anyone around for review of template for RequestForAdoption? https://meta.miraheze.org/wiki/User:MrJaroslavik/Requests_for_adoption Feel free to test it - "Add request" button. [07:02:23] [ User:MrJaroslavik/Requests for adoption - Miraheze Meta ] - meta.miraheze.org [07:04:26] @Zppix Hello, is pinned message here still actual? [07:06:12] @MrJaroslavik: it isn't [07:08:53] Ok [07:45:02] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.76, 2.50, 1.61 [07:47:00] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.72, 1.87, 1.48 [07:48:59] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.58, 1.42, 1.36 [07:56:54] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.51, 3.58, 2.33 [08:00:50] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.52, 1.93, 1.92 [08:01:18] Noting RhinosF1's reply above, which I assume was something different to this, prompted me to look at the pinned message(s) for this channel. "IRC RELAY BOT NOTICE The irc bot may go down near the end of the month every month this is due to the limitation of using a free hosting provider, the bot should automatically reconnect on the first of the next month. Thank you for understanding, Zppix IRC Relay bot maintainer." @Zppix, is that [08:01:19] still the case, or do we need to update that in any way? [08:06:50] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.38, 1.10, 1.57 [08:22:02] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3Ku [08:22:03] [02miraheze/MirahezeMagic] 07translatewiki 03a83b499 - Localisation updates from https://translatewiki.net. [08:22:04] [ Main page - translatewiki.net ] - translatewiki.net [08:26:34] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.76, 2.43, 1.60 [08:27:41] It needa removing altogether [08:28:30] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.85, 1.80, 1.46 [08:30:30] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.81, 1.44, 1.36 [08:30:44] Reception123: can you check cp8 is sane? [09:00:25] [02dns] 07RhinosF1 opened pull request 03#162: Add tools1 and make it puppetmaster - 13https://git.io/JJ36S [09:00:44] [02dns] 07RhinosF1 edited pull request 03#162: [WIP/DNM] Add tools1 and make it puppetmaster - 13https://git.io/JJ36S [09:05:35] Seems fine now [09:06:02] ok [09:44:25] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 4407 files, 12.88MB, 2020-07-14 09:43:00 (1.4 minutes ago) [09:44:52] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [09:45:40] Reception123: ^ [09:45:45] fyi paladox [09:55:03] No access right now [10:01:42] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:01:53] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 37%, RTA = 151.72 ms [10:03:47] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.02, 0.04, 0.05 [10:03:56] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 101.32 ms [10:06:10] PROBLEM - wiki.nvda-nl.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nvda-nl.org' expires in 15 day(s) (Thu 30 Jul 2020 09:58:16 GMT +0000). [10:16:55] !log sudo umount /mnt/mediawiki-static and then mount on gluster1 [10:16:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:21:00] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:22:04] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 86%, RTA = 198.61 ms [10:22:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:22:59] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:23:00] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:23:31] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 11359 MB (88% inode=92%); [10:24:19] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 101.11 ms [10:24:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:24:59] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:05:01] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.41, 2.23, 1.48 [11:06:58] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.69, 1.70, 1.38 [12:11:17] PROBLEM - zh.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zh.gyaanipedia.co.in' expires in 15 day(s) (Thu 30 Jul 2020 12:02:30 GMT +0000). [12:16:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ39N [12:16:40] [02miraheze/ssl] 07MirahezeSSLBot 03e072405 - Bot: Update SSL cert for zh.gyaanipedia.co.in [12:25:21] RECOVERY - zh.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.gyaanipedia.co.in' will expire on Mon 12 Oct 2020 11:16:31 GMT +0000. [13:26:47] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.90, 1.71, 1.31 [13:28:44] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.67, 1.34, 1.22 [13:50:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3d0 [13:50:21] [02miraheze/services] 07MirahezeSSLBot 0380f6167 - BOT: Updating services config for wikis [14:00:43] Hello mapperific! If you have any questions, feel free to ask and someone should answer soon. [14:01:00] hi mapperific [14:01:06] hello! [14:01:39] How can we help? [14:02:17] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.75, 3.09, 2.02 [14:02:57] https://i.imgur.com/QK93Npn.png I imported the infobox template and its related templates from Wikipedia but the border doesnt seem to be showing up, basically the box has no outline [14:05:21] You're missing some css [14:05:33] * RhinosF1 really needs to look at the toolkits [14:05:41] Ah, how do I get that? Sorry, am new to this whole thing [14:06:16] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.50, 1.73, 1.70 [14:08:15] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 5.05, 2.77, 2.06 [14:09:55] I found it! Thanks, didnt know it was because i lacked the CSS from Wikipedia [14:18:05] Yep [14:22:57] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJ3bT [14:22:59] [02miraheze/puppet] 07JohnFLewis 0355eebe9 - convert foreachwikiindblist to new schema [14:23:50] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJ3bY [14:23:52] [02miraheze/puppet] 07JohnFLewis 03404fc6a - convert foreachwikiindblist to new schema [14:26:07] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.83, 1.41, 2.00 [14:31:17] [02miraheze/WikiDiscover] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJ3bD [14:31:18] [02miraheze/WikiDiscover] 07JohnFLewis 0392bcb6a - remove $wgWikiDiscover*List [14:33:05] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:33:14] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:33:23] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:33:58] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:34:37] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:34:54] PROBLEM - wiki.archlinuxfamily.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.archlinuxfamily.org' expires in 15 day(s) (Thu 30 Jul 2020 14:30:12 GMT +0000). [14:35:58] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.78, 1.11, 1.59 [14:39:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3NO [14:39:16] [02miraheze/ssl] 07MirahezeSSLBot 03ecd1eb9 - Bot: Update SSL cert for wiki.archlinuxfamily.org [14:42:04] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.73, 2.08, 1.86 [14:44:05] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.26, 1.78, 1.78 [14:55:09] RECOVERY - wiki.archlinuxfamily.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.archlinuxfamily.org' will expire on Mon 12 Oct 2020 13:39:07 GMT +0000. [14:55:59] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 5.25, 3.21, 2.29 [14:59:52] ello, am here again, apologies. https://i.imgur.com/pHTZWaT.png how would i get rid of this category thing? [14:59:53] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.85, 1.99, 2.00 [15:00:34] mapperific: find out what's including them categories and remove them. [15:01:55] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJ3Ah [15:01:57] [02miraheze/mediawiki] 07JohnFLewis 03c6dde40 - WD [15:03:55] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:04:16] PROBLEM - chouverse.ncps.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'chouverse.ncps.xyz' expires in 15 day(s) (Thu 30 Jul 2020 15:00:05 GMT +0000). [15:04:21] PROBLEM - dev.funshitposting.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dev.funshitposting.xyz' expires in 15 day(s) (Thu 30 Jul 2020 15:01:52 GMT +0000). [15:04:45] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3x2 [15:04:47] [02miraheze/puppet] 07JohnFLewis 03daaeaf8 - rm jq from mediawiki::packages [15:05:52] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.85, 1.30, 1.70 [15:05:54] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3xr [15:05:55] [02miraheze/puppet] 07JohnFLewis 034b84167 - remove DBListGenerator cron [15:07:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3xX [15:07:42] [02miraheze/ssl] 07MirahezeSSLBot 03f3d328d - Bot: Update SSL cert for chouverse.ncps.xyz [15:08:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3x1 [15:08:11] [02miraheze/ssl] 07MirahezeSSLBot 034542e94 - Bot: Update SSL cert for dev.funshitposting.xyz [15:08:47] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3xM [15:08:48] [02miraheze/puppet] 07JohnFLewis 03b34d304 - all.dblist -> databases.json [15:08:58] Is there a way I can keep the categories but remove it from showing on the right? [15:09:12] Not entirely sure [15:13:03] .in 6hours JohnLewis or RhinosF1 update foreachwikiindblist [15:13:03] RhinosF1: Okay, will remind at 2020-07-14 - 22:13:03BST [15:13:19] hm? [15:13:46] JohnLewis: so I don't forget to update any tech guides [15:14:22] well, I'm going to once I've sent an email [15:14:26] It's proceedure [15:14:49] Ok [15:17:45] RECOVERY - chouverse.ncps.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'chouverse.ncps.xyz' will expire on Mon 12 Oct 2020 14:07:33 GMT +0000. [15:17:59] RECOVERY - dev.funshitposting.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'dev.funshitposting.xyz' will expire on Mon 12 Oct 2020 14:08:02 GMT +0000. [15:22:48] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:23:14] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.64, 2.34, 1.99 [15:23:25] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:23:32] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:14] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.89, 1.83, 1.85 [15:26:20] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [15:27:00] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:27:14] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.28, 1.92, 1.87 [15:29:12] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.41, 1.69, 1.79 [15:33:10] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.42, 1.82, 1.80 [15:33:52] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±16] 13https://git.io/JJ3pP [15:33:53] [02miraheze/mediawiki] 07tstarling 03f6af5c0 - Introduce $wgForceHTTPS Add $wgForceHTTPS. When set to true: * It makes the HTTP to HTTPS redirect unconditional and suppresses the forceHTTPS cookie. * It makes session cookies be secure. * In the Action API, it triggers the existing deprecation warning and avoids more expensive user/session checks. * In login and signup, it suppresses the old hidden form fields [15:33:53] for protocol switching. * It hides the prefershttps user preference. Other changes: * Factor out the HTTPS redirect in MediaWiki::main() into maybeDoHttpsRedirect() and shouldDoHttpRedirect(). Improve documentation. * User::requiresHTTPS() reflects $wgForceHTTPS whereas the Session concept of "force HTTPS" does not. The documentation of User::requiresHTTPS() says that it includes configuration, and retaining this [15:33:53] definition was beneficial for some callers. Whereas Session::shouldForceHTTPS() was used fairly narrowly as the value of the forceHTTPS cookie, and injecting configuration into it is not so easy or beneficial, so I left it as it was, except for clarifying the documentation. * Deprecate the following hooks: BeforeHttpsRedirect, UserRequiresHTTPS, CanIPUseHTTPS. No known extension uses them, and they're not compatible [15:33:53] with the long-term goal of ending support for mixed-protocol wikis. BeforeHttpsRedirect was documented as unstable from its inception. CanIPUseHTTPS was a WMF config hack now superseded by GFOC's SNI sniffing. * For tests which failed with $wgForceHTTPS=true, I mostly split the tests, testing each configuration value separately. * Add ArrayUtils::cartesianProduct() as a helper for generating combinations of boolean [15:33:53] options in the session tests. Bug: T256095 Change-Id: Iefb5ba55af35350dfc7c050f9fb8f4e8a79751cb (cherry picked from commit c75eef91bf27046cab9f0857e84f093f9926298f) [15:35:11] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.60, 1.62, 1.72 [15:37:09] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.45, 1.57, 1.69 [15:41:33] PROBLEM - wiki.archlinuxfamily.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.archlinuxfamily.de' expires in 15 day(s) (Thu 30 Jul 2020 15:32:58 GMT +0000). [15:44:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ3hk [15:44:41] [02miraheze/ssl] 07MirahezeSSLBot 0327f242c - Bot: Update SSL cert for wiki.archlinuxfamily.de [15:55:16] RECOVERY - wiki.archlinuxfamily.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.archlinuxfamily.de' will expire on Mon 12 Oct 2020 14:44:32 GMT +0000. [16:01:16] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.65, 1.71, 1.59 [16:03:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJ3je [16:03:15] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.56, 1.66, 1.59 [16:03:15] [02miraheze/puppet] 07paladox 037bd3d60 - Switch admin to write-user for ldap [16:03:17] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://git.io/vbiAS [16:03:18] [02puppet] 07paladox opened pull request 03#1442: Switch admin to write-user for ldap - 13https://git.io/JJ3jv [16:03:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJ3jf [16:03:43] [02miraheze/puppet] 07paladox 03797f98e - Update ldap.toml.erb [16:03:45] [02puppet] 07paladox synchronize pull request 03#1442: Switch admin to write-user for ldap - 13https://git.io/JJ3jv [16:04:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJ3jT [16:04:10] [02miraheze/puppet] 07paladox 0311a4ea4 - Update resources.ini.erb [16:04:12] [02puppet] 07paladox synchronize pull request 03#1442: Switch admin to write-user for ldap - 13https://git.io/JJ3jv [16:04:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJ3jk [16:04:36] [02miraheze/puppet] 07paladox 0361f65bf - Update config.ini.php.erb [16:04:37] [02puppet] 07paladox synchronize pull request 03#1442: Switch admin to write-user for ldap - 13https://git.io/JJ3jv [16:07:28] [02puppet] 07paladox closed pull request 03#1442: Switch admin to write-user for ldap - 13https://git.io/JJ3jv [16:07:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JJ3jm [16:07:31] [02miraheze/puppet] 07paladox 035b84ac2 - Switch admin to write-user for ldap (#1442) * Switch admin to write-user for ldap * Update ldap.toml.erb * Update resources.ini.erb * Update config.ini.php.erb [16:07:32] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://git.io/vbiAS [16:07:34] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [16:10:19] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.51, 1.78, 1.66 [16:12:16] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.09, 1.99, 1.76 [16:14:15] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.65, 1.86, 1.74 [16:18:18] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.86, 2.12, 1.88 [16:20:17] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.38, 1.85, 1.81 [16:24:13] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.11, 1.38, 1.62 [16:35:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJseW [16:35:15] [02miraheze/services] 07MirahezeSSLBot 0301dcb60 - BOT: Updating services config for wikis [16:49:07] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.03, 1.94, 1.71 [16:51:05] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.76, 2.07, 1.80 [17:03:05] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.49, 1.78, 1.97 [17:11:01] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.23, 1.64, 1.77 [17:14:59] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.84, 1.89, 1.86 [17:26:55] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 4.27, 3.15, 2.31 [17:44:51] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:45:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJsJX [17:45:35] [02miraheze/puppet] 07paladox 034d750bb - Fix password for ldapvi [17:45:37] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://git.io/vbiAS [17:45:38] [02puppet] 07paladox opened pull request 03#1443: Fix password for ldapvi - 13https://git.io/JJsJ1 [17:45:51] [02puppet] 07paladox synchronize pull request 03#1443: Fix password for ldapvi - 13https://git.io/JJsJ1 [17:45:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJsJS [17:45:54] [02miraheze/puppet] 07paladox 0308663f4 - Update openldap.pp [17:46:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJsJ9 [17:46:10] [02miraheze/puppet] 07paladox 038d40c91 - Update ldapvi.conf.erb [17:46:11] [02puppet] 07paladox synchronize pull request 03#1443: Fix password for ldapvi - 13https://git.io/JJsJ1 [17:47:16] [02puppet] 07paladox closed pull request 03#1443: Fix password for ldapvi - 13https://git.io/JJsJ1 [17:47:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJsJd [17:47:19] [02miraheze/puppet] 07paladox 035f2493f - Fix password for ldapvi (#1443) * Fix password for ldapvi * Update openldap.pp * Update ldapvi.conf.erb [17:47:21] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://git.io/vbiAS [17:47:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [17:58:15] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 245s [18:00:34] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.17, 1.36, 1.99 [18:08:27] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.00, 1.18, 1.68 [18:09:16] !log restart mariadb on db7 [18:09:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:20] PROBLEM - db7 Check MariaDB Replication on db7 is UNKNOWN: NRPE: Unable to read output [18:12:35] PROBLEM - db7 MariaDB on db7 is CRITICAL: Can't connect to MySQL server on 'db7.miraheze.org' (111) [18:14:17] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [18:14:35] RECOVERY - db7 MariaDB on db7 is OK: Uptime: 123 Threads: 11 Questions: 234 Slow queries: 6 Opens: 98 Flush tables: 1 Open tables: 92 Queries per second avg: 1.902 [18:14:37] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.87, 2.06, 1.92 [18:16:36] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.67, 1.93, 1.89 [18:18:34] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.78, 2.04, 1.94 [18:20:34] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.20, 1.76, 1.85 [18:22:33] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.88, 2.27, 2.06 [18:28:33] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJsks [18:28:34] [02miraheze/RottenLinks] 07paladox 03d1bf541 - Make "rl_externallink" a primary key [18:28:36] [02RottenLinks] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fxBrr [18:28:37] [02RottenLinks] 07paladox opened pull request 03#18: Make "rl_externallink" a primary key - 13https://git.io/JJskG [18:30:32] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/JJskn [18:30:33] [02miraheze/RottenLinks] 07paladox 03bca8325 - Create patch- rl_externallink.sql [18:30:34] [02RottenLinks] 07paladox synchronize pull request 03#18: Make "rl_externallink" a primary key - 13https://git.io/JJskG [18:30:37] miraheze/RottenLinks/paladox-patch-3/d1bf541 - paladox The build passed. https://travis-ci.org/miraheze/RottenLinks/builds/708088026 [18:31:26] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJskl [18:31:28] [02miraheze/RottenLinks] 07paladox 0338d270a - Update RottenLinksHooks.php [18:31:29] [02RottenLinks] 07paladox synchronize pull request 03#18: Make "rl_externallink" a primary key - 13https://git.io/JJskG [18:31:57] [02RottenLinks] 07paladox closed pull request 03#18: Make "rl_externallink" a primary key - 13https://git.io/JJskG [18:31:59] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JJskB [18:32:01] [02miraheze/RottenLinks] 07paladox 03c50e3d4 - Make "rl_externallink" a primary key (#18) [18:32:02] [02RottenLinks] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fxBrr [18:32:04] [02miraheze/RottenLinks] 07paladox deleted branch 03paladox-patch-3 [18:32:28] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/JJskR [18:32:28] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.93, 1.55, 1.85 [18:32:29] [02miraheze/RottenLinks] 07paladox 03e9eccf4 - Rename patch- rl_externallink.sql to patch-rl_externallink.sql [18:33:04] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJsku [18:33:06] [02miraheze/mediawiki] 07paladox 03beaaf91 - Update RottenLinks [18:33:23] paladox: no CHANEGLOG entry? [18:33:33] oh [18:33:35] bugger [18:33:37] miraheze/RottenLinks/paladox-patch-3/38d270a - paladox The build has errored. https://travis-ci.org/miraheze/RottenLinks/builds/708088942 [18:33:38] adding one [18:34:47] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJskw [18:34:49] [02miraheze/RottenLinks] 07paladox 037c96cc3 - Bump version to 1.0.9, also add changelog entry [18:34:50] [02RottenLinks] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fxBrr [18:34:52] [02RottenLinks] 07paladox opened pull request 03#19: Bump version to 1.0.9, also add changelog entry - 13https://git.io/JJskr [18:35:18] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJskK [18:35:19] [02miraheze/RottenLinks] 07paladox 03148cf3a - Update extension.json [18:35:21] [02RottenLinks] 07paladox synchronize pull request 03#19: Bump version to 1.0.9, also add changelog entry - 13https://git.io/JJskr [18:35:35] [02RottenLinks] 07paladox synchronize pull request 03#19: Bump version to 1.0.9, also add changelog entry - 13https://git.io/JJskr [18:35:37] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJskP [18:35:38] [02miraheze/RottenLinks] 07paladox 03d10f5ad - Update CHANGELOG [18:35:44] [02RottenLinks] 07paladox closed pull request 03#19: Bump version to 1.0.9, also add changelog entry - 13https://git.io/JJskr [18:35:46] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJskX [18:35:48] [02miraheze/RottenLinks] 07paladox 03b396714 - Bump version to 1.0.9, also add changelog entry (#19) [18:35:49] [02RottenLinks] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fxBrr [18:35:50] [02miraheze/RottenLinks] 07paladox deleted branch 03paladox-patch-3 [18:36:13] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJsky [18:36:14] [02miraheze/mediawiki] 07paladox 03cb42cc3 - Update RottenLinks [18:36:23] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.89, 1.21, 1.65 [18:37:58] miraheze/RottenLinks/paladox-patch-3/d10f5ad - paladox The build has errored. https://travis-ci.org/miraheze/RottenLinks/builds/708090376 [18:38:57] miraheze/RottenLinks/paladox-patch-3/7c96cc3 - paladox The build passed. https://travis-ci.org/miraheze/RottenLinks/builds/708090123 [18:41:37] [02mw-config] 07dmehus opened pull request 03#3168: Update LocalSettings.php - 13https://git.io/JJskA [18:43:36] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJsIT [18:43:37] [02miraheze/RottenLinks] 07paladox 03a382218 - Make "rl_respcode" an index instead of "rl_externallink" [18:43:39] [02RottenLinks] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fxBrr [18:43:40] [02RottenLinks] 07paladox opened pull request 03#20: Make "rl_respcode" an index instead of "rl_externallink" - 13https://git.io/JJsIk [18:44:16] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-1/±0] 13https://git.io/JJsIO [18:44:17] [02miraheze/RottenLinks] 07paladox 038c02f38 - Update and rename patch-rl_externallink.sql to patch-rl_respcode.sql [18:44:19] [02RottenLinks] 07paladox synchronize pull request 03#20: Make "rl_respcode" an index instead of "rl_externallink" - 13https://git.io/JJsIk [18:44:33] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJsIG [18:44:35] [02miraheze/RottenLinks] 07paladox 03b947714 - Update RottenLinksHooks.php [18:44:37] [02RottenLinks] 07paladox synchronize pull request 03#20: Make "rl_respcode" an index instead of "rl_externallink" - 13https://git.io/JJsIk [18:45:09] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJsIZ [18:45:10] [02miraheze/RottenLinks] 07paladox 037a6e8ed - Update CHANGELOG [18:45:12] [02RottenLinks] 07paladox synchronize pull request 03#20: Make "rl_respcode" an index instead of "rl_externallink" - 13https://git.io/JJsIk [18:45:20] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJsIn [18:45:21] [02miraheze/RottenLinks] 07paladox 0382537dc - Update extension.json [18:45:23] [02RottenLinks] 07paladox synchronize pull request 03#20: Make "rl_respcode" an index instead of "rl_externallink" - 13https://git.io/JJsIk [18:45:39] miraheze/RottenLinks/paladox-patch-3/a382218 - paladox The build passed. https://travis-ci.org/miraheze/RottenLinks/builds/708093170 [18:45:44] paladox: hm? [18:45:56] JohnLewis it failed with: [18:46:04] Error: 1170 BLOB/TEXT column 'rl_externallink' used in key specification without a key length (db12.miraheze.org) [18:46:44] mhm, but that doesn't explain why you're making a non unique column a primary key [18:47:27] miraheze/RottenLinks/paladox-patch-3/82537dc - paladox The build passed. https://travis-ci.org/miraheze/RottenLinks/builds/708093765 [18:47:40] Oh,didn't know it was a non unique column. I'll revert my change then [18:48:32] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JJsIF [18:48:33] [02miraheze/RottenLinks] 07paladox 034867a7a - Revert "Make "rl_externallink" a primary key" [18:48:35] [02RottenLinks] 07paladox created branch 03paladox-patch-4 - 13https://git.io/fxBrr [18:48:36] [02RottenLinks] 07paladox opened pull request 03#21: Revert "Make "rl_externallink" a primary key" - 13https://git.io/JJsIb [18:49:02] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-1/±0] 13https://git.io/JJsIN [18:49:03] [02miraheze/RottenLinks] 07paladox 0341a3b41 - Delete patch-rl_externallink.sql [18:49:05] [02RottenLinks] 07paladox synchronize pull request 03#21: Revert "Make "rl_externallink" a primary key" - 13https://git.io/JJsIb [18:49:13] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JJsIx [18:49:14] [02miraheze/RottenLinks] 07paladox 03b5c03bd - Update RottenLinksHooks.php [18:49:16] [02RottenLinks] 07paladox synchronize pull request 03#21: Revert "Make "rl_externallink" a primary key" - 13https://git.io/JJsIb [18:49:40] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JJsIj [18:49:42] [02miraheze/RottenLinks] 07paladox 03103cd16 - Update CHANGELOG [18:49:43] [02RottenLinks] 07paladox synchronize pull request 03#21: Revert "Make "rl_externallink" a primary key" - 13https://git.io/JJsIb [18:49:56] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:50:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.80, 25.60, 15.08 [18:50:43] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_34 [+0/-0/±2] 13https://git.io/JJsLt [18:50:45] [02miraheze/mediawiki] 07paladox 033b4ea30 - Revert "Update RottenLinks" This reverts commit cb42cc33ef896eb046cefd2b0365809a36dad3fc. [18:50:46] [02miraheze/mediawiki] 07paladox 03eb65e37 - Revert "Update RottenLinks" This reverts commit beaaf91b711746186456ea153f5b073b382eaec3. [18:51:24] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:51:33] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JJsLn [18:51:34] [02miraheze/RottenLinks] 07paladox 031966e8c - Update rottenlinks.sql [18:51:35] [02RottenLinks] 07paladox synchronize pull request 03#21: Revert "Make "rl_externallink" a primary key" - 13https://git.io/JJsIb [18:52:27] miraheze/RottenLinks/paladox-patch-4/b5c03bd - paladox The build passed. https://travis-ci.org/miraheze/RottenLinks/builds/708094733 [18:52:27] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [18:52:51] [02RottenLinks] 07paladox closed pull request 03#21: Revert "Make "rl_externallink" a primary key" - 13https://git.io/JJsIb [18:52:52] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03master [+0/-1/±4] 13https://git.io/JJsL8 [18:52:54] [02miraheze/RottenLinks] 07paladox 031d3aaa2 - Revert "Make "rl_externallink" a primary key" (#21) * Revert "Make "rl_externallink" a primary key" * Delete patch-rl_externallink.sql * Update RottenLinksHooks.php * Update CHANGELOG * Update rottenlinks.sql [18:52:55] [02miraheze/RottenLinks] 07paladox deleted branch 03paladox-patch-4 [18:52:57] [02RottenLinks] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/fxBrr [18:53:29] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:54:02] miraheze/RottenLinks/paladox-patch-4/103cd16 - paladox The build passed. https://travis-ci.org/miraheze/RottenLinks/builds/708094870 [18:54:10] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:55:30] paladox: incrementing the version by 2 for no reason is bad practise btw [18:55:55] If you unrelease something, you're removing the release, not re-releasing an older version as a new releae [18:56:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.37, 20.22, 16.64 [18:56:30] ok, will remember that (e.g reverting back to the original version). Thanks. [18:56:36] [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsLP [18:56:38] [02miraheze/RottenLinks] 07JohnFLewis 03dac2cdc - remove 2 recent releases [18:56:57] [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsLM [18:56:59] [02miraheze/RottenLinks] 07JohnFLewis 038417c0e - v1.0.8 is the version [18:58:33] [02mw-config] 07paladox commented on pull request 03#3168: Update LocalSettings.php - 13https://git.io/JJsLS [19:02:43] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.52, 2.03, 1.70 [19:05:01] [02RottenLinks] 07paladox closed pull request 03#20: Make "rl_respcode" an index instead of "rl_externallink" - 13https://git.io/JJsIk [19:05:04] [02RottenLinks] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fxBrr [19:05:06] [02miraheze/RottenLinks] 07paladox deleted branch 03paladox-patch-3 [19:06:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JJstL [19:06:30] [02miraheze/puppet] 07paladox 0376cc04a - jobrunner: Don't nice scripts Not needed anymore. [19:06:32] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://git.io/vbiAS [19:06:33] [02puppet] 07paladox opened pull request 03#1444: jobrunner: Don't nice scripts - 13https://git.io/JJstt [19:06:51] paladox: resp code a primary key? [19:06:55] huh? [19:07:02] SPF|Cloud i never did it :) [19:07:14] see above, i reverted my above changes reverting [19:07:24] I know, but you did submit a PR for it [19:07:54] oh, yeh. [19:08:21] I thought primary keying any column would get rid of the that message from the error log on mysql [19:09:36] [02mw-config] 07RhinosF1 commented on pull request 03#3168: Update LocalSettings.php - 13https://git.io/JJst4 [19:09:51] paladox: ^ [19:15:31] ok [19:20:19] RhinosF1 you're around and you have few mins? I have problem with doc for https://meta.miraheze.org/wiki/Template:WikiAdoptionRequest [19:20:21] [ Template:WikiAdoptionRequest - Miraheze Meta ] - meta.miraheze.org [19:20:23] !log root@gluster1:/home/paladox# gluster volume set mvol performance.io-cache off [19:20:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:33] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.87, 1.60, 1.95 [19:20:41] (problem is background) [19:21:27] I will find time [19:21:38] * RhinosF1 needs a mini me to act as a Pa [19:21:43] !log root@gluster1:/home/paladox# gluster volume set mvol performance.write-behind off [19:21:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:31] !log regenerating sitemaps [19:24:34] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.72, 1.96, 1.99 [19:24:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsqr [19:25:15] [02miraheze/services] 07MirahezeSSLBot 034bd8137 - BOT: Updating services config for wikis [19:26:34] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.33, 1.67, 1.88 [19:28:17] !log install fio on gluster1 [19:28:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:25] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 11.20, 9.86, 5.72 [19:34:44] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.96, 2.00, 1.91 [19:35:35] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.96, 20.25, 16.29 [19:37:39] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.74, 18.06, 16.01 [19:38:41] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.82, 1.88, 1.88 [19:39:26] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 2.72, 6.72, 5.82 [19:40:42] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.13, 2.33, 2.04 [19:46:51] !log root@gluster1:/home/paladox# gluster volume set mvol performance.cache-max-file-size 2MB [19:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:13] !log root@gluster1:/home/paladox# gluster volume set mvol performance.cache-refresh-timeout 4 [19:47:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:29] !log root@gluster1:/home/paladox# gluster volume set mvol performance.cache-max-file-size 6MB [19:50:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:59] !log root@gluster1:/mnt/mediawiki-static# gluster volume set mvol performance.write-behind on [20:04:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:53] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.75, 1.37, 1.99 [20:08:51] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.73, 1.74, 2.01 [20:10:50] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.86, 1.39, 1.84 [20:12:50] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.12, 1.61, 1.86 [20:14:48] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.96, 1.63, 1.83 [20:20:49] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 15.76, 5.16, 2.95 [20:44:50] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.14, 1.70, 1.96 [20:50:44] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.65, 1.99, 1.95 [21:13:03] RhinosF1: JohnLewis or RhinosF1 update foreachwikiindblist [21:14:25] JohnLewis did it MirahezeBot [21:18:30] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.74, 1.28, 1.92 [21:18:54] ugh [21:19:09] cp8's load check is going nuts [21:19:40] !log enable flap detection on 'cp8 Current Load' [21:19:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:20:02] and now let's see what the issue is [21:20:29] SPF|Cloud high i/o [21:20:57] but is it serving so much traffic or is OVH's performance so terrible? [21:22:25] it seems typical almost every performance issue we have with OVH's services are i/o related [21:22:29] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.03, 1.52, 1.86 [21:22:51] is related* and that means upgrading the vps won't help [21:24:27] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.45, 1.51, 1.82 [21:30:27] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.85, 1.39, 1.68 [21:33:30] > wonders if Linode or DigitalOcean would be better to OVH [21:38:20] @Doug replacement server already on its way [21:39:25] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.84, 1.72, 1.72 [21:41:23] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.88, 1.39, 1.60 [21:59:17] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 6.71, 6.91, 3.97 [22:04:12] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsZy [22:04:13] [02miraheze/dns] 07Southparkfan 0379e259c - Add cp9.miraheze.org [22:08:38] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JJsZA [22:08:39] [02miraheze/puppet] 07Southparkfan 0381b730a - Add cp9 [22:24:55] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.06, 1.41, 1.99 [22:25:16] !log installing cp9 [22:25:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:20] PROBLEM - cp9 Puppet on cp9 is CRITICAL: connect to address 51.222.27.129 port 5666: Connection refusedconnect to host 51.222.27.129 port 5666: Connection refused [22:28:21] PROBLEM - cp9 Stunnel Http for mon1 on cp9 is CRITICAL: connect to address 51.222.27.129 port 5666: Connection refusedconnect to host 51.222.27.129 port 5666: Connection refused [22:28:21] PROBLEM - cp9 Stunnel Http for mw7 on cp9 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:28:32] PROBLEM - cp9 Stunnel Http for mw6 on cp9 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:28:35] PROBLEM - cp9 Stunnel Http for mw4 on cp9 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:28:36] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is UNKNOWN: NRPE: Unable to read output [22:28:37] PROBLEM - cp9 Stunnel Http for test2 on cp9 is CRITICAL: connect to address localhost and port 8091: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:28:39] PROBLEM - cp9 Varnish Backends on cp9 is UNKNOWN: NRPE: Unable to read output [22:28:55] PROBLEM - cp9 Stunnel Http for mw5 on cp9 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:29:01] ^ downtimed cp9 [22:29:39] ty SPF|Cloud [22:31:31] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.91, 2.14, 2.14 [22:35:43] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.95, 1.69, 1.97 [22:41:56] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.06, 1.48, 1.75 [22:43:58] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.33, 1.39, 1.68 [22:48:18] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.85, 3.73, 2.73 [22:53:43] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2112 MB (10% inode=93%); [22:57:45] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsc7 [22:57:47] [02miraheze/puppet] 07Southparkfan 035c6aecb - Fix race condition in varnish module [23:04:15] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsC3 [23:04:17] [02miraheze/puppet] 07Southparkfan 0335c493a - Explicitly mention requires in varnish module [23:09:03] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.63, 1.21, 1.94 [23:16:11] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsC1 [23:16:13] [02miraheze/mw-config] 07Southparkfan 037fba7e4 - Add cp9 [23:17:39] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsCQ [23:17:40] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.37, 1.31, 1.69 [23:17:41] [02miraheze/puppet] 07Southparkfan 03476fc73 - Update own .bash_aliases to reflect new Salt setup [23:20:09] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsCA [23:20:11] [02miraheze/dns] 07Southparkfan 030f75e23 - Switch cp8 to cp9 [23:24:39] paladox: load is high on gluster1 [23:25:01] hmm [23:25:50] seems to be going down [23:26:48] yep, icinga problems clearing again [23:28:09] wait.. was that the effect of the switchover? [23:29:00] why is cp9 using cp6? [23:29:03] timing seems perfect [23:29:06] "x-cache: cp6 HIT (11)" [23:29:26] you're in the UK, cp6 is closer [23:30:00] I did the curl on cp9 :) [23:30:12] cp9's IP geolocates to France [23:30:43] well, apparently their VPSs don't [23:31:47] quirk in the GeoLite database? [23:33:07] and yep, indeed looks like the cp switchover was the culprit for that load [23:33:10] damn [23:33:55] !log update /usr/share/GeoIP/GeoLite2-Country.mmdb on ns1 & ns2 and restart gdnsd [23:33:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:19] Ok works now "x-cache: cp9 HIT (21)" [23:34:21] SPF|Cloud ^ [23:34:25] cool [23:36:01] i/o looking fine on cp9 [23:36:21] yup [23:36:35] grafana graphs report otherwise for some reason, but iotop doesn't indicate issues [23:37:36] i/o looks fine for me on grafana [23:38:23] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?viewPanel=287&orgId=1&var-job=node&var-node=cp9.miraheze.org&var-port=9100 [23:38:24] [ Grafana ] - grafana.miraheze.org [23:39:59] oh [23:43:55] I guess we can decom cp8? [23:44:53] unless you want to reuse the server for something else [23:47:14] ugh, it's still receiving traffic [23:51:07] !log cp8 is depooled, replaced by cp9. however, since cp8 is still receiving traffic, and server has been paid for until July, I'll leave nginx/varnish running overnight - if traffic does not stop after 24h (to allow rogue DNS caches to be refreshed), services can be shutdown [23:51:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:31] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JJs8e [23:57:33] [02miraheze/ManageWiki] 07paladox 033711ad6 - Add a ChangeDbCluster script Allows you to update multiple wikis and set their db cluster to a different value. [23:57:34] [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns [23:57:36] [02ManageWiki] 07paladox opened pull request 03#176: Add a ChangeDbCluster script - 13https://git.io/JJs8f