[00:01:33] [02puppet] 07paladox closed pull request 03#776: Update dumps to use yaml file for cron - 13https://git.io/fNGF3 [00:01:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±1] 13https://git.io/fNZj3 [00:01:36] [02miraheze/puppet] 07paladox 03744281e - Update dumps to use yaml file for cron (#776) * Update dumps to use yaml file for cron This makes things easier to maintain and cleaner. * Create xml_dump.yaml * Create image_dump.yaml * Update dumps.pp * Update xml_dump.yaml * Update dumps.pp [00:01:37] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [01:47:23] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [01:49:24] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [01:49:27] !log update cp5, cp4, cp2 to stretch 9.5 (point release) (and reboot them one by one) [01:50:24] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 7.85, 7.24, 5.54 [01:50:34] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:50:44] !log update cp5, cp4, cp2 to stretch 9.5 (point release) (and reboot them one by one) [01:50:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:52:24] PROBLEM - Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.92, 7.91, 5.99 [01:55:56] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:56:08] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [01:59:40] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:00:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [02:01:30] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 7.26, 8.00, 6.91 [02:01:40] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:05:40] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [02:06:31] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [02:35:40] PROBLEM - Puppet on swift2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [02:37:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNnU1 [02:37:31] [02miraheze/puppet] 07paladox 030b436c0 - Switch static.miraheze.org to use lizardfs [03:07:40] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:16:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNnT1 [03:16:52] [02miraheze/puppet] 07paladox 03a146739 - Update bacula-dir.conf [03:33:40] PROBLEM - Current Load on test1 is CRITICAL: CRITICAL - load average: 2.56, 1.83, 1.02 [03:34:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNnkf [03:34:06] [02miraheze/mw-config] 07paladox 03fad93ad - Update LocalSettings.php [03:35:40] RECOVERY - Current Load on test1 is OK: OK - load average: 1.36, 1.65, 1.05 [03:39:40] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:41:40] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:45:41] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:50:41] PROBLEM - Current Load on test1 is CRITICAL: CRITICAL - load average: 4.01, 2.38, 1.53 [03:51:02] paladox: why you loading all the servers man [03:51:22] PuppyKun what do you mean? It's swift [03:51:41] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 6.65, 4.24, 2.73 [03:51:42] PuppyKun /me running swift download and swift delete [03:51:45] swift is causing 8+ 1 min loads and 5+ 15 minute load averages on mw1? [03:51:53] PuppyKun yeh [03:51:55] oh [03:51:56] well [03:51:59] rn usually bitches about that [03:52:04] but as long as its not ongoing i guess [03:52:07] PuppyKun yeh it's a one time thing [03:52:11] we are migrating [03:52:13] to lizardfs [03:52:14] now [03:52:21] (right at this second) [03:52:44] PuppyKun lizardfs seems to be stable [03:52:58] though i am not going to gynx it :) [03:57:39] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 2.69, 3.52, 2.96 [03:59:39] RECOVERY - Current Load on misc4 is OK: OK - load average: 3.00, 3.31, 2.95 [04:23:31] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.61, 3.52, 2.85 [04:27:29] RECOVERY - Current Load on swift1 is OK: OK - load average: 3.11, 3.36, 2.93 [04:42:40] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 7.71, 6.68, 5.18 [04:44:40] PROBLEM - Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.22, 7.49, 5.64 [04:46:41] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 6.99, 7.25, 5.78 [04:49:11] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [04:51:11] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [05:34:33] Hello Guest39045! If you have any questions feel free to ask and someone should answer soon. [05:35:16] Eip [05:35:18] Rip [05:35:26] My nickname is unavailable even tho it's offline [05:59:41] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:05:41] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:44:39] PROBLEM - Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.26, 6.64, 5.78 [06:46:40] RECOVERY - Current Load on mw1 is OK: OK - load average: 6.63, 6.56, 5.85 [06:49:40] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:50:40] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 7.50, 6.95, 6.16 [06:57:40] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:09:41] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:13:32] Hey, same issue with the Maia Miraheze again, with the Error creating thumbnail: File missing Leodia [07:13:43] As can be exampled on this page https://maiasongcontest.miraheze.org/wiki/Maia_Song_Contest_20 [07:13:44] Title: [ Maia Song Contest 20 - Maia Song Contest ] - maiasongcontest.miraheze.org [07:24:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [07:28:10] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [07:36:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [07:36:31] PROBLEM - Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [07:36:41] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [07:38:31] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [07:38:41] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [07:47:31] PROBLEM - Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [07:47:41] PROBLEM - Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [07:56:50] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [07:58:50] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:21:39] PROBLEM - Disk Space on cp5 is WARNING: DISK WARNING - free space: / 4992 MB (20% inode=95%): [08:49:40] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:50:40] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [08:50:50] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [08:51:30] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [08:52:40] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [08:52:50] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:55:30] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [09:04:31] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [09:04:41] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [09:04:51] PROBLEM - MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:41] RECOVERY - MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 31111 bytes in 0.101 second response time [09:07:41] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [09:29:40] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:35:41] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [09:40:51] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [09:42:49] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [09:42:51] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:01:40] PROBLEM - Disk Space on swift2 is WARNING: DISK WARNING - free space: / 32148 MB (20% inode=97%): [10:29:40] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [10:37:41] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:49:40] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [11:24:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [11:26:40] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 7.56, 6.47, 5.47 [11:28:40] RECOVERY - Current Load on mw1 is OK: OK - load average: 5.40, 6.18, 5.50 [11:30:10] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [11:31:38] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNnr9 [11:31:39] [02miraheze/mw-config] 07MacFan4000 03c5c0f42 - Fix spelling [11:37:28] Hmm, cp5 has low disk space [11:45:31] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [11:45:41] RECOVERY - Disk Space on cp5 is OK: DISK OK - free space: / 5164 MB (21% inode=95%): [11:48:12] deleted some misc files on it soo [11:48:39] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [11:49:29] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [11:49:39] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [11:52:39] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [11:54:39] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [11:56:55] im going to depool mw1 [11:58:40] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 3.42, 3.03, 2.04 [12:00:13] !log depool mw1 [12:00:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:00:40] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 4.09, 3.26, 2.23 [12:01:15] JohnLewis i've disabled puppet on mw1 and stopped nginx to stop varnish repool it. I didn't set a disabled message so it is blank. [12:01:17] that was me [12:01:40] PROBLEM - Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 1 failures [12:02:00] if its depooled, varnish wont repool it. [12:02:40] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 3.50, 3.34, 2.38 [12:03:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [12:03:30] PROBLEM - HTTPS on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refused [12:03:32] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [12:03:40] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [12:05:06] JohnLewis hmm ok. [12:05:40] RECOVERY - Current Load on swift1 is OK: OK - load average: 3.32, 3.17, 2.57 [12:07:30] RECOVERY - HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.090 second response time [12:09:10] RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 31113 bytes in 0.441 second response time [12:09:40] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 4.20, 6.75, 7.89 [12:13:41] RECOVERY - Current Load on mw1 is OK: OK - load average: 2.24, 4.20, 6.56 [12:20:23] @paladox my apologies but the spambots are back (on the gerrit wiki). I know who the bot operator is and I'll contact them for the spam to be dealt with [12:20:36] ah [12:20:37] thanks [12:21:01] Can you use your server access to install AbuseFilter or some sort of anti spam tool? [12:21:06] Or use cleanupspam.php [12:21:14] It uses regex to delete pages [12:23:52] just revoke edit [12:25:31] sau226 it uses mw vagrant [12:25:44] so i would have to figure out how to do that with mw vagrant puppet [12:30:40] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 30% [12:34:21] I can give you a sample command [12:35:13] mwscript cleanupSpam.php --delete [12:35:19] After using vagrant ssh [12:35:58] Replace with the casiono link. Takes 1 wildxard at the start @paladox [12:39:36] Can you do that? [12:43:22] The easiest thing would probs be to use Extension:Nuke to clean up... [12:43:31] But bots work too [12:44:39] I dont have ssh access. Only @paladox does. I do have an wikitech account if it wver becomes necessary [12:44:59] Though the bot actions aren’t being marked bot [12:45:08] That I know [12:46:30] Its flagged [12:47:31] Has the password for the admin account been reset? [12:48:04] No. You appear to have an admin account though [12:48:23] It says macfan4000 is an admin on wiki [12:48:34] I wonder what the default password is? [12:48:46] You have admin and crat [12:48:51] Yes [12:49:07] I'll try pm you on irc [12:49:38] Bot's done [12:51:36] paladox: realised your site notice is poor now [12:51:49] implies maintenance tonight [13:03:49] JohnLewis oh [13:04:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNnDG [13:05:01] [02miraheze/mw-config] 07paladox 031b2e8d5 - Update Sitenotice.php [13:05:03] JohnLewis ^^ [13:05:03] done [13:12:10] PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refused [13:13:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [13:13:30] PROBLEM - HTTPS on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refused [13:13:32] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [13:13:40] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [13:28:41] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 3.69, 2.98, 1.99 [13:32:40] RECOVERY - Current Load on misc4 is OK: OK - load average: 2.99, 2.88, 2.14 [13:40:40] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 4.56, 3.75, 2.77 [13:47:30] PROBLEM - Disk Space on swift1 is WARNING: DISK WARNING - free space: / 23637 MB (15% inode=96%): [13:54:01] Eh you might want to check [14:07:30] PROBLEM - fikcyjnatv.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate fikcyjnatv.pl expires in 15 day(s) (Sat 04 Aug 2018 02:04:59 PM GMT +0000). [14:09:30] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate adadevelopersacademy.wiki expires in 15 day(s) (Sat 04 Aug 2018 02:07:16 PM GMT +0000). [15:09:29] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2667 [15:15:30] RECOVERY - Disk Space on swift1 is OK: DISK OK - free space: / 32375 MB (21% inode=96%): [15:16:36] hello to everyone [15:19:40] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 3.74, 3.46, 2.35 [15:21:40] RECOVERY - Current Load on misc4 is OK: OK - load average: 2.93, 3.17, 2.37 [15:47:30] PROBLEM - Disk Space on swift1 is WARNING: DISK WARNING - free space: / 31268 MB (20% inode=96%): [15:49:30] RECOVERY - Disk Space on swift1 is OK: DISK OK - free space: / 32878 MB (21% inode=96%): [15:53:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcJ1 [15:53:12] [02miraheze/mw-config] 07paladox 036cad8e4 - Update LocalWiki.php [15:56:40] PROBLEM - Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.48, 7.72, 5.99 [15:58:40] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 7.34, 7.47, 6.10 [16:00:40] PROBLEM - Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.15, 8.04, 6.46 [16:31:31] PROBLEM - Disk Space on swift1 is WARNING: DISK WARNING - free space: / 31506 MB (20% inode=96%): [16:37:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcIW [16:37:09] [02miraheze/mw-config] 07paladox 03a7b47c7 - Update LocalWiki.php [16:37:13] JohnLewis ^^ [16:37:16] Is that ok? [16:43:29] RECOVERY - Disk Space on swift1 is OK: DISK OK - free space: / 33037 MB (21% inode=96%): [16:44:39] PROBLEM - Current Load on test1 is CRITICAL: CRITICAL - load average: 3.49, 2.16, 1.15 [16:44:41] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 5.90, 4.06, 2.33 [16:46:40] PROBLEM - Current Load on test1 is WARNING: WARNING - load average: 1.44, 1.96, 1.21 [16:46:41] RECOVERY - Current Load on misc4 is OK: OK - load average: 1.39, 3.02, 2.16 [16:48:40] RECOVERY - Current Load on test1 is OK: OK - load average: 0.51, 1.44, 1.11 [16:49:40] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:55:40] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:03:57] Hi! Here is the list of currently open high priority tasks on Phabricator [17:04:03] No updates for 3 days - https://phabricator.miraheze.org/T3361 - Migrate from swift to lizardfs - authored by Paladox, assigned to None [17:04:10] No updates for 11 days - https://phabricator.miraheze.org/T3328 - Blank avatars in user page and forum, UploadAvatar doesn't work - authored by SelfCloak, assigned to Paladox [17:04:29] ^ paladox [17:04:40] Reception123 yep i got pinged :) [17:04:51] Heh yeah :D [17:04:57] anyways all those issues should be resolved soon as the migration to lizardfs is ongoing. [17:05:14] paladox: Ok, even the UploadAvatar issue? [17:05:19] yeh [17:05:31] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:18:39] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 5.71, 3.35, 2.13 [17:30:40] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 2.18, 3.55, 3.15 [17:32:40] RECOVERY - Current Load on misc4 is OK: OK - load average: 2.13, 3.02, 3.00 [18:46:41] PROBLEM - Current Load on mw1 is WARNING: WARNING - load average: 7.74, 6.65, 5.56 [18:48:41] RECOVERY - Current Load on mw1 is OK: OK - load average: 4.94, 6.06, 5.48 [18:59:02] [02puppet] 07JohnFLewis closed pull request 03#775: optimising php-fpm - 13https://git.io/fNmIk [20:08:40] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 3.98, 3.44, 2.28 [20:10:40] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 5.39, 3.97, 2.60 [20:27:13] [02miraheze/mw-config] 07NDKilla pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcEW [20:27:15] [02miraheze/mw-config] 07NDKilla 03f830623 - Disable MobileFrontend on horizon.miraheze.org This closes T3369 as resolved [20:29:29] Pull of 'mediawiki-config' failed: Command failed with error #128! COMMAND git fetch origin '+refs/*:refs/*' --prune STDOUT (empty) STDERR error: insufficient permission for adding an object to repository database ./objects fatal: failed to write object fatal: unpack-objects failed [20:29:41] uh [20:29:51] PuppyKun which mw? [20:29:54] misc1 [20:30:00] phabricator isn't tracking github [20:30:05] er, it is, but it errors [20:30:11] uh misc1? [20:30:17] or wherever phab is hosted [20:30:21] * PuppyKun forgets [20:30:21] PuppyKun misc4 [20:30:23] oh [20:30:30] misc4 then [20:30:33] * PuppyKun didnt know we had a misc4 [20:30:39] ok /me will fix that [20:30:40] * PuppyKun one of those "why am i ops" moments [20:30:57] PuppyKun yeh we have a misc4 for phab (and was for swift proxy but now lizardfs master) [20:30:59] lol [20:31:08] so [20:31:14] is lizardfs replacing swift already [20:31:18] or do they go hand in hand [20:31:30] PuppyKun yep it's replacing swift for us [20:31:34] migration in progress [20:31:46] swift was short lived [20:32:15] PuppyKun yeh, lizardfs comes with a ui :D [20:32:16] https://lizard.miraheze.org/mfs.cgi?sections=HD [20:32:17] Title: [ 401 Unauthorized ] - lizard.miraheze.org [20:32:17] create your user on misc4 [20:32:18] in /etc/lizardfs/ [20:32:19] (like icinga) [20:44:40] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 3.11, 3.39, 3.83 [20:50:40] RECOVERY - Current Load on misc4 is OK: OK - load average: 2.57, 2.60, 3.31 [20:58:41] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 5.22, 3.99, 3.58 [21:23:40] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 2.75, 3.49, 3.32 [21:27:40] PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.25, 3.67, 3.41 [21:29:40] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.24, 3.47, 3.36 [22:03:16] !log deleting "static" pool from bacula [22:03:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:15:41] RECOVERY - Disk Space on bacula1 is OK: DISK OK - free space: / 205184 MB (43% inode=99%): [22:49:29] PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:49:39] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:49:49] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:56:00] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcXc [22:56:01] [02miraheze/MirahezeMagic] 07paladox 0397b266c - Remove unused create container from hook [22:56:48] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNcX8 [22:56:49] [02miraheze/mediawiki] 07paladox 03c40cfa9 - Update MirahezeMagic [22:57:24] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNcXE [22:57:25] [02miraheze/mediawiki] 07paladox 03d7ee2ae - Update CreateWiki [22:58:29] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcXo [22:58:31] [02miraheze/MirahezeMagic] 07paladox 033b85131 - Remove createwiki-error-swiftcontainernot [22:58:49] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/fNcX6 [22:58:50] [02miraheze/MirahezeMagic] 07paladox 033f53b7e - Remove setZoneAccess.php script [22:59:17] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNcXM [22:59:19] [02miraheze/mediawiki] 07paladox 0318750d9 - Update MirahezeMagic [23:01:40] PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 5.92, 5.69, 5.45 [23:13:41] PROBLEM - Current Load on misc4 is WARNING: WARNING - load average: 3.75, 3.42, 2.55 [23:15:41] RECOVERY - Current Load on misc4 is OK: OK - load average: 3.06, 3.36, 2.64 [23:23:30] PROBLEM - Puppet on swift1 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-9102],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9100] [23:27:39] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:40] PROBLEM - Current Load on misc4 is CRITICAL: CRITICAL - load average: 5.09, 3.89, 3.12 [23:37:40] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:40:32] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcMb [23:40:33] [02miraheze/mw-config] 07Reception123 0388aabee - change msupload drag and drop to true by default