[00:03:29] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:40] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:50] PROBLEM - Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:30] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [00:11:50] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [00:13:40] RECOVERY - Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:13:50] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:40] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:18:33] [02services] 07The-Voidwalker opened pull request 03#145: services on gua per T3355 - 13https://git.io/fNcyb [00:19:29] [02services] 07paladox closed pull request 03#145: services on gua per T3355 - 13https://git.io/fNcyb [00:19:31] [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcyh [00:19:32] [02miraheze/services] 07The-Voidwalker 039d3d05b - services on gua per T3355 (#145) [00:29:41] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:31:41] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 1.73, 2.89, 3.79 [00:35:39] RECOVERY - Current Load on swift1 is OK: OK - load average: 1.29, 1.97, 3.20 [00:38:15] [02mw-config] 07The-Voidwalker opened pull request 03#2286: userrights config fixes T3371 - 13https://git.io/fNcSD [00:38:51] ^please don't merge just yet [00:41:06] [02mw-config] 07The-Voidwalker synchronize pull request 03#2286: userrights config fixes T3371 - 13https://git.io/fNcSD [00:41:35] it's good now [00:43:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcSb [00:43:06] [02miraheze/mw-config] 07The-Voidwalker 03dac4262 - userrights config fixes T3371 (#2286) * userrights config fixes T3371 * add templateeditor restriction level [00:43:07] [02mw-config] 07paladox closed pull request 03#2286: userrights config fixes T3371 - 13https://git.io/fNcSD [00:46:20] Hello NeoNamo[m]! If you have any questions feel free to ask and someone should answer soon. [01:07:38] !log repool mw1 [01:07:39] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:09:29] RECOVERY - HTTP on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 406 bytes in 0.002 second response time [01:09:31] RECOVERY - HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.017 second response time [01:09:39] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [01:09:49] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [01:10:09] RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 31130 bytes in 0.100 second response time [01:11:09] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [01:17:50] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:22:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcHO [01:22:23] [02miraheze/puppet] 07paladox 03afe906f - Remove swift from swift[12] [01:25:51] PROBLEM - Puppet on swift2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:27:40] PROBLEM - Puppet on swift2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 22 hours ago with 0 failures [01:27:46] uh [01:29:41] RECOVERY - Puppet on swift2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:29:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcHl [01:29:57] [02miraheze/puppet] 07paladox 03d83173e - Remove swift from misc4 [01:31:31] RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [01:33:51] PROBLEM - Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): File[/etc/swift/account.builder],File[/etc/swift/account.ring.gz],File[/etc/swift/container.builder],File[/etc/swift/container.ring.gz] [01:34:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcHV [01:34:16] [02miraheze/puppet] 07paladox 03173d895 - Update packages.pp [01:34:41] PROBLEM - Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:34:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcHo [01:34:43] [02miraheze/puppet] 07paladox 0330cb79a - Update init.pp [01:34:51] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [01:35:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcHi [01:35:29] [02miraheze/puppet] 07paladox 039fe6b50 - Update data.yaml [01:35:31] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [01:35:51] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [01:36:41] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [01:36:50] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:36:51] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:37:29] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 4% [01:38:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcHD [01:38:12] [02miraheze/mw-config] 07paladox 030102540 - Update Sitenotice.php [01:38:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcHy [01:38:37] [02miraheze/mw-config] 07paladox 039d572d7 - Re enable uploads [01:38:39] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 9% [01:41:43] !log rm -rf /srv/node on swift[12] [01:41:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:00:22] Hello BobbyL! If you have any questions feel free to ask and someone should answer soon. [02:01:51] Hey there. I requested a Wikia a couple hours ago. [02:02:02] BobbyL hi which wiki request? [02:02:28] BobbyL this https://meta.miraheze.org/wiki/Special:RequestWikiQueue/5413 one? [02:02:29] Title: [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [02:02:49] It's called Converse. https://meta.miraheze.org/wiki/Special:RequestWikiQueue/5413 [02:02:50] Title: [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [02:03:15] Yeah [02:04:12] BobbyL do you have a custom domain? [02:04:24] coniverse.miraheze.org [02:04:28] is that fine as the url? [02:04:40] that's perfect. [02:04:41] (just that you have a invalid url) so just making sure [02:04:42] ok [02:05:51] PROBLEM - Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [02:06:27] BobbyL done [02:06:30] wiki is up now [02:06:35] https://converse.miraheze.org/wiki/Main_Page [02:06:35] Title: [ Coniverse ] - converse.miraheze.org [02:07:04] thanks [02:07:16] Hello Refpga[m]! If you have any questions feel free to ask and someone should answer soon. [02:21:04] uh, having an issue concerning templates on my recently created wiki (https://converse.miraheze.org) [02:21:09] "Script error: No such module "Infobox". Script error: No such module "documentation"." [02:21:36] is there an extension I must enable? [02:23:04] BobbyL: hi yes [02:23:12] You need module:documentation [02:23:28] (Which is the lua script) [02:24:15] BobbyL: not sure if you got my last message as having network problems [02:24:37] correction, there is no extension required [02:24:43] you simply need to import a page [02:24:47] ^^ [02:24:51] oh, okay I get it [02:35:40] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [02:36:32] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNc5H [02:36:34] [02miraheze/ssl] 07paladox 0341d385c - Update adadevelopersacademy.wiki.crt [02:37:13] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNc57 [02:37:14] [02miraheze/ssl] 07paladox 039a18037 - Update fikcyjnatv.pl.crt [02:38:19] Another script error. Where do I to get one? [02:38:21] "Script error: No such module "InfoboxImage"." [02:38:41] Same thing [02:38:51] Import that module [02:39:00] BobbyL: ^^ [02:39:56] !log depool mw3 to reclone mw [02:39:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:41:31] RECOVERY - fikcyjnatv.pl - LetsEncrypt on sslhost is OK: OK - Certificate fikcyjnatv.pl will expire on Thu 18 Oct 2018 01:36:53 AM GMT +0000. [02:43:31] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate adadevelopersacademy.wiki will expire on Thu 18 Oct 2018 01:36:00 AM GMT +0000. [02:43:41] RECOVERY - Disk Space on swift2 is OK: DISK OK - free space: / 32310 MB (21% inode=97%): [02:44:41] PROBLEM - Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [02:45:11] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3 [02:46:27] mw3 seems to be down [02:46:33] wont come back up [02:48:28] This would be the correct link, right? "module:InfoboxImage" I imported the module but still seems to be getting the same error as before. [02:48:31] Hasn't happened with the previous modules I imported [02:49:29] PROBLEM - Disk Space on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refused [02:49:31] PROBLEM - SSH on mw3 is CRITICAL: connect to address 81.4.121.113 and port 22: Connection refused [02:49:39] PROBLEM - JobRunner Service on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refused [02:49:40] * paladox opens a ticket with ramnode [02:49:41] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:49:49] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:50:10] BobbyL, can I have a link to a page displaying the error? [02:50:22] because what you're doing sounds right [02:50:27] https://converse.miraheze.org/wiki/Template:Infobox_legislature [02:50:30] Title: [ Template:Infobox legislature - Coniverse ] - converse.miraheze.org [02:50:31] (of course it might be a caching issue) [02:50:59] or this https://converse.miraheze.org/wiki/Module:InfoboxImage [02:51:07] Title: [ Module:InfoboxImage - Coniverse ] - converse.miraheze.org [02:52:50] BobbyL, fixed, purged page cache [02:53:12] well it's showing mw3 as up in ramnode control panel [02:53:16] alright [02:53:18] i guess network is at play here [02:53:22] * paladox files task [02:53:49] also, I'd suggest adding MediaWiki:Common.css so that you get all the styling you need [02:54:02] (makes display work properly) [02:56:22] issue filed [02:59:30] Hello PuppyKun[m]! If you have any questions feel free to ask and someone should answer soon. [03:05:30] RECOVERY - SSH on mw3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [03:05:40] PROBLEM - MediaWiki Rendering on mw3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 199 bytes in 0.018 second response time [03:05:57] Im going to reinstall mw3 [03:08:26] !log reimaging mw3 due to it stopping all services instead of them comming back on boot [03:08:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:11:41] PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refused [03:13:31] PROBLEM - HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refused [03:16:54] This user is now online in #wikimedia-dev. I'll let you know when they show some activity (talk, etc.) [03:16:54] @notify Zppix [03:21:29] PROBLEM - JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [03:21:31] PROBLEM - Puppet on mw3 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [03:23:29] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_jobqueue) or PHP errors [03:23:31] PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 38 seconds ago with 8 failures. Failed resources (up to 3 shown): Package[postfix],Package[lilypond],Package[ocaml],Package[salt-minion] [03:23:39] RECOVERY - php7.2-fpm on mw3 is OK: PROCS OK: 5 processes with command name php-fpm7.2 [03:27:29] RECOVERY - JobChron Service on mw3 is OK: PROCS OK: 1 process with args redisJobChronService [03:27:40] RECOVERY - JobRunner Service on mw3 is OK: PROCS OK: 1 process with args redisJobRunnerService [03:37:30] RECOVERY - HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.029 second response time [03:37:40] PROBLEM - MediaWiki Rendering on mw3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 208 bytes in 0.025 second response time [03:38:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcFc [03:38:17] [02miraheze/puppet] 07paladox 03270ee45 - Increase git timeout to 600 secs MediaWiki is a large wiki and is hitting the timeout. [03:51:50] [02services] 07The-Voidwalker opened pull request 03#146: add horizon per T3373 - 13https://git.io/fNcFX [03:56:10] [02services] 07paladox closed pull request 03#146: add horizon per T3373 - 13https://git.io/fNcFX [03:56:11] [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNcFQ [03:56:13] [02miraheze/services] 07The-Voidwalker 03f4fd7d2 - add horizon per T3373 (#146) [04:22:22] okay, successfully set up the templates after all the module trouble, but again I have run into problems regarding this specific infobox being on the other side of the screen [04:22:34] https://converse.miraheze.org/wiki/Template:Infobox_legislature [04:22:37] Title: [ Template:Infobox legislature - Coniverse ] - converse.miraheze.org [04:33:09] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [04:34:39] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [04:58:41] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [04:59:11] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3 [04:59:41] PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.020 second response time [04:59:53] Sysadmins? [05:33:41] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [05:35:49] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:49:40] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:49:50] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:55:50] PROBLEM - Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [06:11:39] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:23:40] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:23:50] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:25:50] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [11:35:40] PROBLEM - MediaWiki Rendering on mw3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 208 bytes in 0.016 second response time [11:36:21] hello! [11:37:40] PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1272 bytes in 0.022 second response time [11:43:31] PROBLEM - JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [11:43:41] RECOVERY - MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 1101 bytes in 0.018 second response time [11:44:41] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [11:45:11] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [11:53:29] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_jobqueue) or PHP errors [11:55:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3 [11:56:40] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [11:57:30] RECOVERY - Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:58:40] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [11:59:10] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [12:07:11] I'm having intermittent problems similar to https://phabricator.miraheze.org/T2434#46005 [12:07:12] Title: [ ⚓ T2434 Error: Unable to change language preferences ] - phabricator.miraheze.org [12:07:47] Wiki-1776 re building the lc [12:08:22] It because I just fixed mw3 [12:08:38] But didn’t rebuild LC [12:09:17] !log paladox is rebuilding LC on mw* [12:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:38:41] !log rm /srv/phab/images/images.tar.gz on misc4 [12:38:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:47:51] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2a00:d880:5:8ea::ebc7/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [12:49:51] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:09:20] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±10] 13https://git.io/fNC4J [13:09:22] [02miraheze/ManageWiki] 07translatewiki 03524ce10 - Localisation updates from https://translatewiki.net. [13:09:50] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:39:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNCRj [13:39:37] [02miraheze/mw-config] 07paladox 033de0637 - Update GlobalExtensions.php [13:46:06] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNC0o [13:46:07] [02miraheze/mediawiki] 07paladox 034c4de92 - Update CookieWarning to 4494d8095b8cf3b57702cea022cd095768dbb1f9 [13:49:41] PROBLEM - MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 624 bytes in 0.370 second response time [13:50:30] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNCEI [13:50:31] [02miraheze/mediawiki] 07paladox 032baa792 - Revert "Update CookieWarning to 4494d8095b8cf3b57702cea022cd095768dbb1f9" This reverts commit 4c4de929ab2717f5c1646432e7533ae80ab05a5d. [13:52:33] why is cookiewarning constantly being changed? [13:54:14] esp because the last commit to REL1_31 was December [13:54:49] JohnLewis because i have changes pending review! [13:54:56] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CookieWarning/+/435689/ [13:54:56] Title: [ Gerrit Code Review ] - gerrit.wikimedia.org [13:55:18] there in master branch [13:55:25] I still don't get why it's changing [13:55:36] yeh we are using the master branch [13:55:41] RECOVERY - MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 31036 bytes in 0.286 second response time [13:55:47] otherwise all users will have the cookie notice shown for them [13:55:55] since my change is not on the REL branch [13:55:58] backport changes [13:56:23] we should always use REL unless REL is broken [13:56:45] and with the amount of reverts, master doesn't seem stable either [13:56:57] JohnLewis i carn't, i got told of the last time i did that [13:57:08] i can only backport if it's been merged [13:57:19] then oh well [13:57:36] lack of function isn't a real reason to break our deployment guidelines [13:57:36] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNCES [13:57:38] [02miraheze/mediawiki] 07paladox 03054bb0e - Update CookieWarning to 0ce1bc17f1fad4518fa93bbd07bdcd5e6b3c5a2f [13:57:41] if it's not broken, don't use master. [13:57:49] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:58:01] JohnLewis well users complaning is. [13:58:38] it's not broken. it's not against the law to have it in the US therefore it's not an issue [13:58:46] lack of upstream functionality isn't our fault [13:59:34] yep not broken [13:59:54] JohnLewis we updated to the master branch when using 1.30 as the cookie notice woulden't go away after pressing ok. [14:00:10] sounds like REL is broken then? [14:00:38] JohnLewis REL1_30 was yeh [14:00:47] does 1.31 work? [14:00:58] REL1_31? yes, i think so. [14:01:05] then go back to 1.31 [14:01:28] using master is a slippery slope esp with the Wikimedia deployment practises [14:03:57] JohnLewis yes i think so [14:05:07] JohnLewis https://gerrit.wikimedia.org/r/#/q/I5f7505dbd650867155485d17fff58fb92800581b [14:05:07] Title: [ Gerrit Code Review ] - gerrit.wikimedia.org [14:07:50] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:18:26] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNCgR [14:18:28] [02miraheze/MirahezeMagic] 07paladox 0359fb4b5 - Remove unused hook [14:18:29] [02MirahezeMagic] 07paladox opened pull request 03#44: Remove unused hook - 13https://git.io/fNCg0 [14:18:49] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNCgg [14:18:51] [02miraheze/MirahezeMagic] 07paladox 036231277 - Update MirahezeMagicHooks.php [14:18:52] [02MirahezeMagic] 07paladox synchronize pull request 03#44: Remove unused hook - 13https://git.io/fNCg0 [14:19:02] [02MirahezeMagic] 07paladox closed pull request 03#44: Remove unused hook - 13https://git.io/fNCg0 [14:19:03] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fNCgV [14:19:05] [02miraheze/MirahezeMagic] 07paladox 03acdcce7 - Remove unused hook (#44) * Remove unused hook * Update MirahezeMagicHooks.php [14:19:07] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [14:19:08] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [14:19:41] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNCgK [14:19:42] [02miraheze/mediawiki] 07paladox 032b0931c - Update MirahezeMagic [14:20:52] miraheze/MirahezeMagic/paladox-patch-1/59fb4b5 - paladox The build failed. https://travis-ci.com/miraheze/MirahezeMagic/builds/79579537 [14:22:12] miraheze/MirahezeMagic/paladox-patch-1/6231277 - paladox The build failed. https://travis-ci.com/miraheze/MirahezeMagic/builds/79579546 [14:29:51] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:37:57] !log rebooted mw1 due to high load (seeing if this will help relieve it) [14:38:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:39:10] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [14:39:24] mw2? [14:39:30] PROBLEM - HTTP on mw1 is CRITICAL: connect to address 185.52.1.75 and port 80: Connection refused [14:39:32] PROBLEM - SSH on mw1 is CRITICAL: connect to address 185.52.1.75 and port 22: Connection refused [14:39:40] PROBLEM - php7.2-fpm on mw1 is CRITICAL: connect to address 185.52.1.75 port 5666: Connection refused [14:40:50] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:40:52] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:41:30] RECOVERY - HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.024 second response time [14:41:32] RECOVERY - SSH on mw1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [14:41:40] RECOVERY - php7.2-fpm on mw1 is OK: PROCS OK: 5 processes with command name php-fpm7.2 [14:41:50] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [14:42:50] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:43:10] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [14:49:50] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:50:40] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [14:50:42] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [14:51:11] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [14:57:41] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:58:39] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [14:58:41] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [15:00:37] !log depool mw1 [15:00:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:05:39] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:49] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:51] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 25 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:38:20] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNCiR [15:38:22] [02miraheze/dns] 07paladox 03e8cb4b0 - Update config [15:38:56] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNCig [15:38:57] [02miraheze/dns] 07paladox 03053012f - Update miraheze.org [15:40:24] miraheze/dns/master/e8cb4b0 - paladox The build was broken. https://travis-ci.org/miraheze/dns/builds/406305149 [15:42:13] That should be fixed [15:42:17] In the second commit [15:54:50] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [15:56:26] What’s that ip to server? [15:56:30] SPF|Cloud: ^^ [15:56:49] that's cp5 [15:57:20] Ah thanks [15:58:51] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:53:30] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [18:07:36] How do I uninstall scribunto? [18:08:32] MWUnknownContentModelException The content model 'Scribunto' is not registered on this wiki. [18:08:41] (Not mh) [18:16:24] stop using it, then disable [18:19:25] JohnLewis: ? [18:19:41] paladox: in reply to PuppyKun [18:20:25] Oh, I must have missed that [18:20:25] Heh [18:21:25] JohnLewis: isent scribuntu enables on all sites? [18:21:29] Scribuntu [18:21:42] yes [18:22:25] “MWUnknownContentModelException The content model 'Scribunto' is not registered on this wiki.” [18:22:33] * paladox wonders how he managed to hit that! [18:22:42] PuppyKun> (Not mh) [18:22:47] by disabling it while it's in use [18:24:06] How so i stop using it. Someone in #mediawiki said change the content model [18:24:08] *googles* [18:24:30] I have file pages erroring. I feel like those shouldn't have scribunto content models but eh [18:25:19] Oh it's not the file page [18:25:26] Special:WhatLinksHere [18:29:34] PuppyKun: could you join -staff? [18:30:05] Just noticed I'm not in it. Fuck irccloud [18:41:58] 503 "interminent" [18:41:58] Wiki-1776: The servers should return back to normal shortly. [18:44:39] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [18:45:11] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [18:45:49] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [18:48:14] [02mw-config] 07The-Voidwalker opened pull request 03#2287: pagelanguseDB on nvcwiki fixes T3234 - 13https://git.io/fNCNl [18:49:09] PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refused [18:49:30] PROBLEM - SSH on mw1 is CRITICAL: connect to address 185.52.1.75 and port 22: Connection refused [18:50:30] PROBLEM - HTTP on mw1 is CRITICAL: connect to address 185.52.1.75 and port 80: Connection refused [18:50:32] PROBLEM - HTTPS on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refused [18:50:42] PROBLEM - php7.2-fpm on mw1 is CRITICAL: connect to address 185.52.1.75 port 5666: Connection refused [18:58:26] JohnLewis: sshd and nginx didn’t start ^^ [18:58:35] I do know [19:00:40] RECOVERY - Current Load on mw1 is OK: OK - load average: 0.00, 0.00, 0.00 [19:01:23] Nice load! Let's keep it like that [19:01:30] lol [19:01:30] RECOVERY - Disk Space on mw1 is OK: DISK OK - free space: / 48713 MB (63% inode=99%): [19:12:14] [02mw-config] 07Reception123 closed pull request 03#2287: pagelanguseDB on nvcwiki fixes T3234 - 13https://git.io/fNCNl [19:12:16] [02miraheze/mw-config] 07Reception123 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fNCxb [19:12:17] [02miraheze/mw-config] 07The-Voidwalker 03d6d6696 - pagelanguseDB on nvcwiki fixes T3234 [19:12:19] [02miraheze/mw-config] 07Reception123 0334ef83f - Merge pull request #2287 from The-Voidwalker/patch-40 pagelanguseDB on nvcwiki fixes T3234 [19:22:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fNCp5 [19:22:56] [02miraheze/puppet] 07paladox 03bf5559b - Remove loading ssh_host_dsa_key OpenSSH the version in stretch disables dsa by default for security reasons. [19:22:58] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:22:59] [02puppet] 07paladox opened pull request 03#777: Remove loading ssh_host_dsa_key - 13https://git.io/fNCpd [19:44:39] [02puppet] 07paladox closed pull request 03#777: Remove loading ssh_host_dsa_key - 13https://git.io/fNCpd [19:44:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNCjr [19:44:42] [02miraheze/puppet] 07paladox 0345cf5b3 - Remove loading ssh_host_dsa_key (#777) OpenSSH the version in stretch disables dsa by default for security reasons. [19:45:40] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:45:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [20:31:40] PROBLEM - Puppet on db4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [21:10:31] PROBLEM - Disk Space on mw1 is CRITICAL: connect to address 185.52.1.75 port 5666: Connection refused [21:10:41] PROBLEM - Current Load on mw1 is CRITICAL: connect to address 185.52.1.75 port 5666: Connection refused [21:28:02] !log reinstalling mw1 [21:28:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:28:14] PuppyKun you should get xshellz.com [21:28:31] you need to remember to press the renew button every 2 weeks though [21:31:30] RECOVERY - SSH on mw1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u1 (protocol 2.0) [21:34:22] IPv6 ips are disabled from creating free shells. [21:34:22] k [21:35:31] heh really? [21:35:42] PuppyKun i've been with them for years now :) [21:35:45] your znc client [21:35:56] you have to ssh in to start your znc client though [21:42:31] RECOVERY - Disk Space on mw1 is OK: DISK OK - free space: / 75563 MB (98% inode=99%): [21:42:40] RECOVERY - Current Load on mw1 is OK: OK - load average: 0.62, 0.28, 0.10 [21:42:50] PROBLEM - Puppet on mw1 is UNKNOWN: NRPE: Unable to read output [21:54:29] RECOVERY - HTTP on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.002 second response time [21:54:39] RECOVERY - php7.2-fpm on mw1 is OK: PROCS OK: 5 processes with command name php-fpm7.2 [21:54:49] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 14 seconds ago with 15 failures. Failed resources (up to 3 shown): Package[lilypond],Exec[Math texvccheck],Exec[wikibase_composer],Exec[maps_composer] [21:58:29] RECOVERY - HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.012 second response time [22:05:10] PROBLEM - MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 199 bytes in 0.018 second response time [22:07:10] PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1272 bytes in 0.012 second response time [22:14:33] [02services] 07The-Voidwalker opened pull request 03#147: add epicerieissy per T3375 - 13https://git.io/fNWm5 [22:16:14] [02services] 07paladox closed pull request 03#147: add epicerieissy per T3375 - 13https://git.io/fNWm5 [22:16:15] [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNWmj [22:16:17] [02miraheze/services] 07The-Voidwalker 03bd7264f - add epicerieissy per T3375 (#147) [22:17:47] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNWYT [22:17:48] [02miraheze/mediawiki] 07paladox 03ba7c579 - Update AJAXPoll to REL1_31 [22:33:10] PROBLEM - MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2864 bytes in 0.014 second response time [22:34:40] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [22:35:10] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [22:35:50] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [22:42:50] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:44:21] Wow, I haven't used this in so long. [22:45:08] hello [22:57:53] !log repool mw1 [22:57:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:09:50] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:15:51] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:43:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNWGe [23:43:47] [02miraheze/puppet] 07paladox 03a0ae61d - Fix perms on /var/lib/nagios/ssl-acme [23:50:41]