[00:02:07] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3821 MB (19% inode=93%); [03:20:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5Tm [03:20:22] [02miraheze/services] 07MirahezeSSLBot 03db49689 - BOT: Updating services config for wikis [04:00:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5kT [04:00:25] [02miraheze/services] 07MirahezeSSLBot 03cfcf148 - BOT: Updating services config for wikis [06:25:43] Hi Reception123 [06:39:23] hi RhinosF1 [06:45:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5q3 [06:45:17] [02miraheze/services] 07MirahezeSSLBot 039f80cd1 - BOT: Updating services config for wikis [07:14:47] Hello hd1! If you have any questions, feel free to ask and someone should answer soon. [07:15:57] Silly SigmaBot [07:16:45] Reception123: remove the ban and I'll add him to the userfile [07:21:27] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+2/-0/±1] 13https://git.io/Jf5mP [07:21:29] [02miraheze/ssl] 07Reception123 03587441a - add hololive.wiki cert hopefully the new ZeroSSL cert bundle works... [07:22:58] RhinosF1: ^ (ban removed) [07:24:05] Reception123: fixed sigma, you can talk to him [07:43:36] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jf5Yr [07:43:38] [02miraheze/ssl] 07Reception123 036388a62 - add wiki.thehall.xyz cert [07:47:12] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jf5Yy [07:47:13] [02miraheze/ssl] 07Reception123 0381da537 - switch kkutu.wiki to LE per request: https://phabricator.miraheze.org/T5759#112292 [07:47:14] [ ⚓ T5759 SSL Request for kkutu.wiki ] - phabricator.miraheze.org [07:50:09] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jf5YF [07:50:10] [02miraheze/ssl] 07Reception123 0385a6427 - add archive.stellurgists.wiki cert [09:10:18] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Thu 02 Jul 2020 09:07:43 GMT +0000). [09:15:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5ZI [09:15:46] [02miraheze/ssl] 07MirahezeSSLBot 037d2da8d - Bot: Update SSL cert for guia.cineastas.pt [09:22:19] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Mon 14 Sep 2020 08:15:37 GMT +0000. [10:18:16] paladox: check UBN on phab pls [10:21:46] !log root@jobrunner1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --all --removeperms oathauth-api-all [10:21:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:22:00] RhinosF1 that's not UBN [10:22:34] paladox: I'd disagree [10:23:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5CI [10:23:03] [02miraheze/mw-config] 07paladox 03751bd60 - Blacklist 'oathauth-api-all' [10:23:28] paladox: now it isn't [10:23:40] It wasn't before... [10:24:02] personally, given what it exposes, I disagree [10:24:05] RhinosF1: well since it was there forever we didn't really need it to be an emergency [10:24:14] and it didn't expose PII either [10:40:07] !log deleted and dropped wizpedia101wiki [10:40:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:40:22] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5CV [10:40:23] [02miraheze/services] 07MirahezeSSLBot 0320c8bdb - BOT: Updating services config for wikis [11:04:10] * Sario wanders in [11:05:01] hi Sario [11:05:30] Reception123: Good morning [11:29:33] Hi Sario [11:32:58] I wonder if you can communicate through the discord. [11:33:07] Guess not, oop [12:44:16] [02mw-config] 07Amanda-Catherine commented on pull request 03#3115: Blacklist subdomain.miraheze - 13https://git.io/Jf5Bl [12:50:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5B1 [12:50:23] [02miraheze/services] 07MirahezeSSLBot 0377ac2a8 - BOT: Updating services config for wikis [13:21:42] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.27, 3.17, 2.31 [13:23:45] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.08, 2.60, 2.20 [14:06:06] Hello yaxxon[m]! If you have any questions, feel free to ask and someone should answer soon. [16:05:08] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wizpedia101wiki --report=1 Wizpedia101-20200615213415.xml [16:05:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:23:31] paladox: job queue is stuck [16:23:35] Can someone run jobs [16:36:12] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php [16:36:15] ^ RhinosF1 [16:36:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:47:24] Reception123: ty [16:50:21] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.58, 4.77, 3.18 [16:50:37] RhinosF1: there were a few jobs here and there but not that many [16:51:33] Reception123: how many? [16:51:48] I just ran it and saw I didn't count [16:52:03] and I'm not sure if there's a way to see how many? [16:53:08] Reception123: ah [16:53:23] I saw a very slow email for echo again [16:54:26] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.18, 3.77, 3.17 [16:56:26] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.36, 3.05, 2.99 [17:20:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.31, 19.42, 14.82 [17:22:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 20.03, 18.87, 15.12 [17:37:03] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.76, 3.62, 3.20 [17:38:08] Reception123: can you make that ban longer? ^ [17:38:22] yes [17:39:02] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.59, 2.86, 2.97 [17:39:22] RhinosF1: If he tries to come back, both his nick and ?id is gbanned [17:39:42] Texas: i know [17:39:47] Reception123: and the ?id ban [17:41:35] Yep [17:43:29] .gj Reception123 [17:43:29] You're doing good work, Reception123! [17:43:45] :) [17:46:30] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1696 MB (8% inode=93%); [17:48:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [17:48:53] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.80, 23.19, 19.45 [17:48:54] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.87, 8.11, 6.31 [17:50:24] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.92, 5.67, 4.27 [17:51:00] Hello poomba! If you have any questions, feel free to ask and someone should answer soon. [17:51:09] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.98, 7.89, 6.49 [17:52:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:53:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.94, 20.29, 19.31 [17:53:18] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2528 MB (13% inode=93%); [17:55:16] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.45, 5.86, 5.98 [17:56:45] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.11, 3.62, 3.81 [18:01:39] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2046 MB (10% inode=93%); [18:03:31] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.67, 18.14, 16.53 [18:03:42] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2512 MB (13% inode=93%); [18:05:02] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.26, 3.23, 3.45 [18:05:47] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.79, 18.42, 16.92 [18:07:04] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.74, 3.73, 3.63 [18:09:03] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 10.67, 5.06, 4.06 [18:10:34] SPF|Cloud: Are you around? [18:11:43] Hello avengium! If you have any questions, feel free to ask and someone should answer soon. [18:11:45] hi [18:11:52] Unexpected error (Expecting value: line 1 column 1 (char 0)) from avengium at 2020-06-16 18:11:52.425579. Message was: https://webchat.freenode.net [18:15:14] my wiki was recently imported by an admin, the articles and the categories are up in miraheze, they got uploaded, but most of the categories are not linked to the articles [18:16:27] avengium: hi again, which wiki is it? [18:16:53] this one: https://tuscriaturas.miraheze.org/wiki/Especial:Categor%C3%ADas [18:16:57] [ Categorías - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [18:17:15] ^ Texas paladox do you guys have any ideas? [18:17:22] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 1.05, 2.95, 3.64 [18:17:36] hmm let me look [18:17:46] maybe you need to run the script to link them Reception123 [18:18:00] paladox: oh which one? [18:18:44] https://github.com/wikimedia/mediawiki/blob/master/maintenance/refreshLinks.php [18:18:45] [ mediawiki/refreshLinks.php at master · wikimedia/mediawiki · GitHub ] - github.com [18:19:02] ah that one [18:19:07] ok I'll give it a try then [18:19:24] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.62, 2.48, 3.38 [18:20:05] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php refreshLinks.php --wiki=tuscriaturaswiki [18:20:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:22:20] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.87, 20.56, 18.35 [18:22:38] avengium: please check now [18:22:59] ok [18:23:37] yes. now i see the categories has articles inside. thanks [18:24:21] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.47, 18.56, 17.89 [18:24:27] thanks again. bye [18:24:38] np :) [18:25:40] :) [18:33:17] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Thu 02 Jul 2020 18:30:35 GMT +0000). [18:34:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5iQ [18:34:40] [02miraheze/ssl] 07MirahezeSSLBot 035788aee - Bot: Update SSL cert for wiki.s23.org [18:39:41] PROBLEM - warmage.necrocity.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'warmage.necrocity.co.uk' expires in 15 day(s) (Thu 02 Jul 2020 18:36:53 GMT +0000). [18:41:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5Pe [18:41:52] [02miraheze/ssl] 07MirahezeSSLBot 03b29430f - Bot: Update SSL cert for warmage.necrocity.co.uk [18:43:36] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Mon 14 Sep 2020 17:34:31 GMT +0000. [18:44:11] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.33, 22.45, 18.86 [18:46:16] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.51, 19.37, 18.18 [18:54:06] RECOVERY - warmage.necrocity.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'warmage.necrocity.co.uk' will expire on Mon 14 Sep 2020 17:41:44 GMT +0000. [18:59:18] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.61, 3.54, 3.05 [19:01:18] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.65, 2.94, 2.89 [19:01:40] PROBLEM - towarzystwo.prawa.kredytowego.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'towarzystwo.prawa.kredytowego.pl' expires in 15 day(s) (Thu 02 Jul 2020 18:58:44 GMT +0000). [19:05:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5P7 [19:05:17] [02miraheze/ssl] 07MirahezeSSLBot 03edd1965 - Bot: Update SSL cert for towarzystwo.prawa.kredytowego.pl [19:12:57] Hi again [19:13:09] Hello [19:13:33] RECOVERY - towarzystwo.prawa.kredytowego.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'towarzystwo.prawa.kredytowego.pl' will expire on Mon 14 Sep 2020 18:05:08 GMT +0000. [19:13:34] another admin from my wiki https://tuscriaturas.miraheze.org/wiki/El_Bestiario_del_Hypogripho_Dorado [19:13:36] [ Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [19:14:00] asked me if we can put a "create a new page" link in the navigation menu [19:14:53] but i'm not sure if that is possible, also don't know the link for that page. Do you know if that is something doable? [19:30:57] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1972 MB (10% inode=93%); [19:32:58] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2377 MB (12% inode=93%); [19:33:26] i'm not sure how this irc works but shouldn't someone answer? [19:34:51] Avengium: It works better if you put our nick in front of your message. That way it will notify us that you have sent us a message :) [19:36:08] oh, ok. in other platforms people don't like pings [19:37:22] i have two questions. One is about CSS colors and personalization. The other is a request of another admin that wants a "create a new page" link. [19:38:46] about the CSS colors. The wiki tuscriaturas is currently on https://tuscriaturas.miraheze.org/w/index.php?title=MediaWiki:Timeless.css&action=edit [19:38:47] [ Ver el código de «MediaWiki:Timeless.css» - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [19:40:58] Texas Timeless theme that has dark colors. In this theme "index" boxes and image boxes has a white backgrounds, that don't look on a dark background. How can i change the CSS colors of both my index box and image box. [19:41:21] paladox: can you help him? [20:11:21] there's https://tuscriaturas.miraheze.org/wiki/MediaWiki:Wiki-navigation [20:11:22] [ MediaWiki:Wiki-navigation - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [20:11:35] hi [20:12:36] that's old content. what do you mean with that? [20:12:47] asked me if we can put a "create a new page" link in the navigation menu [20:13:39] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.18, 3.16, 2.88 [20:13:56] i'm not sure what you mean by "index box" and "image box"? [20:14:08] oh, so we can put new pages modifying that wiki navigation [20:14:22] wait a second for the box explanation [20:15:13] a wiki page with 4 or more section creates and index of contents [20:15:21] like this page: https://tuscriaturas.miraheze.org/wiki/Escorpiosauros_Negros [20:15:22] [ Escorpiosauros Negros - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [20:16:48] you can use the browsers developer tools to find the css classes [20:16:50] for exampleul#filetoc, .toc, .mw-warning [20:16:52] * ul#filetoc, .toc, .mw-warning [20:16:54] on a dark theme like Timeless, the index of contents has a white background that doesn't sit well with the rest of the colors of the page. How can an admin change that? What is the precise CSS code to change the color of index background [20:17:10] ok, i will try to find that [20:17:17] ul#filetoc, .toc, .mw-warning { background: } [20:17:47] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.41, 3.72, 3.21 [20:18:43] ok, thanks [20:18:55] =D (y) [20:19:50] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.84, 3.33, 3.13 [20:29:24] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.48, 3.92, 3.55 [20:31:29] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.19, 3.32, 3.37 [20:32:37] .seen JohnLewis [20:32:38] Texas: I last saw JohnLewis in here 4 days, 6 minutes ago, saying: It would need to do a lot more functionality for me to agree to a feature flag when an existing feature can handle it [20:32:52] @seen JohnLewis [20:32:52] Texas: Last time I saw JohnLewis they were quitting the network with reason: Quit: Connection closed for inactivity N/A at 6/13/2020 8:08:45 PM (3d24m6s ago) [20:42:07] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.64, 3.42, 3.23 [20:44:09] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.05, 3.39, 3.25 [20:47:22] !log increase cp6 cores to 4 and reboot [20:47:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:47:54] !log change cp6 cpu to host [20:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:49:44] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5Mo [20:49:45] [02miraheze/dns] 07paladox 031962934 - Depool cp7 [20:50:35] !log apt-get install ca-certificates - ns2 [20:50:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:52:43] paladox: I think we can slightly reduce the load on some cache proxies [20:52:55] will test my patch for that later [20:52:57] today [20:53:13] SPF|Cloud thank you! [20:53:30] SPF|Cloud should i wait on you testing before rebooting cp7? [20:53:39] I'm going to increase cp6 ram to match cp7 [20:53:43] have you changed something? [20:53:47] how much old/new? [20:53:54] i've only just depooled cp7 [20:54:12] i've made changes in proxmox but requires restarting to take affect [20:54:16] got that [20:54:27] increasing ram is probably necessary for my change, so good timing and go ahead [20:54:34] ok [20:54:49] !log increase cp6 ram to 3gb matches cp7 [20:55:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:59:20] tell me when done, so I can test the change right away [21:00:25] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.12, 1.92, 1.44 [21:01:50] SPF|Cloud done [21:02:41] alright [21:02:48] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.54, 1.79, 1.46 [21:04:11] SPF|Cloud: while you are around, would you mind closing https://meta.miraheze.org/wiki/Meta:Requests_for_permissions#Examknow_.28sysop.29 [21:04:12] [ Meta:Requests for permissions - Miraheze Meta ] - meta.miraheze.org [21:04:54] sure [21:05:01] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.26, 1.58, 1.43 [21:07:44] Texas: and done [21:07:58] Woo! [21:08:05] SPF|Cloud: Thanks! [21:08:06] :) [21:10:42] !log disable puppet on cp6 for T5762 [21:10:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:13:48] PROBLEM - cp6 Puppet on cp6 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 minute ago with 0 failures [21:14:31] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.21, 1.75, 1.65 [21:16:42] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.66, 1.66, 1.62 [21:18:46] !log enable puppet on cp6, instead locally applied change to puppet2's git [21:18:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:20:15] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:21:51] SPF|Cloud should i reboot cp7 and repool? [21:22:05] repool what? [21:22:08] cp7 [21:22:13] cp7 currently depooled [21:22:20] per https://github.com/miraheze/dns/compare/5a3162467b69...1962934b2baf [21:22:21] [ Comparing 5a3162467b69...1962934b2baf · miraheze/dns · GitHub ] - github.com [21:22:46] is there something cp7 needs to be rebooted for? [21:23:49] for the slight ram change i did [21:23:55] 2.95gb -> 3.02gb [21:24:05] uh, sure [21:24:14] as long as you don't repool cp6 since I'm working on that one now [21:25:20] sure [21:25:35] PROBLEM - cp6 HTTPS on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.005 second response time [21:25:36] PROBLEM - cp6 Varnish Backends on cp6 is WARNING: No backends detected. If this is an error, see readme.txt [21:25:47] !log up cp7 cores to 4 [21:25:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:26:58] !log change cpu type to host on cp7 [21:27:01] !log reboot cp7 [21:27:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:27:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:27:43] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf5DS [21:27:44] [02miraheze/dns] 07paladox 038cc8ab4 - Revert "Depool cp7" This reverts commit 1962934b2bafceee390016d9a5e5defa79cf4e07. [21:29:44] * Sario528 heads home [21:32:14] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [21:33:26] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [21:33:59] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1692 bytes in 0.025 second response time [21:43:41] paladox: can we remove the references to the new_servers variable? [21:43:49] yeh [21:44:51] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jf5yX [21:44:51] [02miraheze/puppet] 07Southparkfan 03faf7399 - Put varnish shm log in RAM [21:44:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yM [21:44:51] [02miraheze/puppet] 07paladox 03e2efdef - Remove new_servers variable [21:44:51] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [21:44:51] [02puppet] 07paladox opened pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yS [21:47:04] [02miraheze/puppet] 07paladox 03ccebd29 - Update init.pp [21:47:04] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5y9 [21:47:04] [02miraheze/puppet] 07paladox 03450845c - Update init.pp [21:47:04] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yH [21:47:04] [02miraheze/puppet] 07paladox 0381fc5f5 - Update PrivateSettings.php [21:47:04] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5y7 [21:47:04] [02miraheze/puppet] 07paladox 03c1a985a - Update ns2.yaml [21:47:04] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yd [21:47:12] [02miraheze/puppet] 07paladox 03421ca1a - Update rdb2.yaml [21:47:12] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yF [21:47:21] [02miraheze/puppet] 07paladox 03ebe11a6 - Update rdb1.yaml [21:47:21] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yb [21:47:33] [02miraheze/puppet] 07paladox 039ab00fe - Update cloud1.yaml [21:47:34] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yN [21:47:51] [02miraheze/puppet] 07paladox 0384cd24e - Update cloud2.yaml [21:47:51] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:47:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5yA [21:47:52] [02miraheze/puppet] 07paladox 03b197db1 - Update mw4.yaml [21:47:53] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [21:52:41] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jf5Se [21:52:47] [02miraheze/dns] 07Southparkfan 0319e138c - Add cp6, depool cp[78] for maintenance [21:54:31] PROBLEM - cp6 Varnish Backends on cp6 is WARNING: No backends detected. If this is an error, see readme.txt [21:56:14] holy spam [21:56:33] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [21:56:34] uh oh [21:56:41] PROBLEM - cp6 HTTPS on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [21:56:59] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jf5Sk [21:57:00] [02miraheze/dns] 07Southparkfan 03c72e9bc - Revert "Add cp6, depool cp[78] for maintenance" This reverts commit 19e138c166f1c119b76d2ac196b759becea24a24. [22:00:50] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 41% [22:02:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5S3 [22:02:21] [02miraheze/puppet] 07paladox 03c60a31c - Update mw5.yaml [22:02:21] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:02:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5Sn [22:02:36] [02miraheze/puppet] 07paladox 036f86886 - Update mw6.yaml [22:02:36] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:02:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5SC [22:02:42] [02miraheze/puppet] 07paladox 031e21cee - Update mw7.yaml [22:02:42] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:02:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5SW [22:02:54] [02miraheze/puppet] 07paladox 031cf43ac - Update test2.yaml [22:02:55] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:03:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5Sl [22:03:08] [02miraheze/puppet] 07paladox 03eca1c23 - Update services1.yaml [22:03:08] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:03:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5S8 [22:03:31] [02miraheze/puppet] 07paladox 03740d48d - Update services2.yaml [22:03:31] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:03:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5SR [22:03:49] [02miraheze/puppet] 07paladox 0303b4edf - Update services2.yaml [22:03:51] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:04:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5S0 [22:04:04] [02miraheze/puppet] 07paladox 03d852291 - Update phab1.yaml [22:04:11] [02puppet] 07paladox synchronize pull request 03#1407: Remove new_servers variable - 13https://git.io/Jf5yD [22:04:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf5SE [22:04:23] that better get squashed :P [22:07:54] lol [22:08:21] #puppetsucks [22:09:25] haha [22:09:34] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2115 MB (10% inode=93%); [22:09:38] oh... wow [22:09:48] SPF|Cloud, paladox: https://dreamsit.com.br/wiki/Main_Page [22:09:48] [ 502 Bad Gateway ] - dreamsit.com.br [22:10:00] that goes through cp6 [22:10:04] they are not using mw-lib [22:10:09] paladox: ah [22:12:09] paladox: I assume cp6 is being messed with and they should use mw-lib [22:12:09] mw-lb, to be exact [22:12:17] yup [22:12:22] paladox: https://phabricator.miraheze.org/T5634#112550 [22:12:23] [ ⚓ T5634 Page Preview no longer working on dreamsitwiki ] - phabricator.miraheze.org [22:12:30] yup i saw [22:12:33] and yes, cp6 is currently broken - we might have had a few clients getting errors until their dns cache was flushed, but that should have been fixed 10 minutes ago [22:12:40] i've already looked into that [22:12:46] but i cannot find out why it doesn't work. [22:13:07] SPF|Cloud: got it [22:14:45] PROBLEM - misc1 APT on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:54] looked into what? [22:14:55] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:13] PROBLEM - misc1 Disk Space on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:21] PROBLEM - misc1 Current Load on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:39] SPF|Cloud https://phabricator.miraheze.org/T5634 [22:15:42] [ ⚓ T5634 Page Preview no longer working on dreamsitwiki ] - phabricator.miraheze.org [22:15:45] aha [22:15:46] All the settings are set [22:15:52] PROBLEM - misc1 SMTP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:53] but some how the page previews are not working. [22:16:03] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [22:16:04] My suspecion is maybe the new MW system? [22:16:22] PROBLEM - misc1 IMAP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:35] misc1 down [22:16:39] load sky rocketed >25 [22:16:59] PROBLEM - misc1 SSH on misc1 is CRITICAL: Server answer [22:17:08] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1693 bytes in 0.009 second response time [22:17:16] paladox: I'll look once I've cleared some wikimedia site requests in the morning at page previewd [22:17:23] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:17:33] ok [22:18:26] !log reboot misc1 [22:18:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:19:32] RECOVERY - misc1 APT on misc1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:19:37] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:19:46] RECOVERY - misc1 Disk Space on misc1 is OK: DISK OK - free space: / 34698 MB (85% inode=99%); [22:20:18] RECOVERY - misc1 SMTP on misc1 is OK: SMTP OK - 0.220 sec. response time [22:20:52] RECOVERY - misc1 IMAP on misc1 is OK: IMAP OK - 0.040 second response time on 185.52.1.76 port 143 [* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.] [22:21:18] !log apt-get upgrade (puppet-agent) - misc1 [22:21:25] RECOVERY - misc1 SSH on misc1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [22:21:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:21:49] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.20, 0.22, 0.10 [22:23:44] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 24% [22:25:38] !log install apache2-utils on cp6 and cp7 [22:25:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:33:20] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 11394 MB (59% inode=93%); [22:38:46] Unexpected error (list index out of range) from catbeard at 2020-06-16 22:38:45.392532. Message was: nope [22:46:22] Texas: ^ [22:46:42] RhinosF1: She does that sometimes [22:46:50] it is a little strange [22:50:57] *user lastmikoi add [22:50:57] 04Error!04 RhinosF1, lastmikoi already exists on userfile. [22:51:32] * lastmikoi waves [22:51:45] Hi lastmikoi [22:51:51]