[05:21:21] [02mw-config] 07JohnFLewis commented on pull request 03#2615: Set wgGlobalBlockingDatabase in GlobalExtension - 13https://git.io/fhzkz [05:23:25] [02mw-config] 07JohnFLewis edited a comment on pull request 03#2615: Set wgGlobalBlockingDatabase in GlobalExtension - 13https://git.io/fhzkz [08:24:12] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03revert-2615-paladox-patch-1 [+0/-0/±2] 13https://git.io/fhzOI [08:24:13] [02miraheze/mw-config] 07JohnFLewis 0319299fa - Revert "Set wgGlobalBlockingDatabase in GlobalExtension (#2615)" This reverts commit 8a93d7b51821fb518bdef25c84e96f1e1db7d8aa. [08:24:15] [02mw-config] 07JohnFLewis created branch 03revert-2615-paladox-patch-1 - 13https://git.io/vbvb3 [08:24:16] [02mw-config] 07JohnFLewis opened pull request 03#2616: Revert "Set wgGlobalBlockingDatabase in GlobalExtension" - 13https://git.io/fhzOL [08:24:25] [02mw-config] 07JohnFLewis closed pull request 03#2616: Revert "Set wgGlobalBlockingDatabase in GlobalExtension" - 13https://git.io/fhzOL [08:24:27] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fhzOq [08:24:28] [02miraheze/mw-config] 07JohnFLewis 03b6723d2 - Revert "Set wgGlobalBlockingDatabase in GlobalExtension (#2615)" (#2616) This reverts commit 8a93d7b51821fb518bdef25c84e96f1e1db7d8aa. [08:24:30] [02mw-config] 07JohnFLewis deleted branch 03revert-2615-paladox-patch-1 - 13https://git.io/vbvb3 [08:24:31] [02miraheze/mw-config] 07JohnFLewis deleted branch 03revert-2615-paladox-patch-1 [09:05:34] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±10] 13https://git.io/fhzsP [09:05:36] [02miraheze/ManageWiki] 07translatewiki 038eafb7d - Localisation updates from https://translatewiki.net. [09:05:37] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±12] 13https://git.io/fhzsX [09:05:39] [02miraheze/CreateWiki] 07translatewiki 03a1a3aca - Localisation updates from https://translatewiki.net. [09:20:42] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 640 [09:36:42] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [09:42:42] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 378 [09:46:42] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [10:20:49] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1147 [11:20:42] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [12:28:51] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhz20 [12:28:53] [02miraheze/mw-config] 07JohnFLewis 03e7c0f7f - remove NS settings in MWN [12:29:22] -1157 lines :P [12:30:58] JohnLewis: did you add those ns into MWN? :) [12:31:16] yes? obviously [12:31:24] those settings haven't done anything for weeks [12:31:54] Sorry I didn’t see you log it JohnLewis ;) [12:35:06] Ah [12:35:08] Nvm [13:08:06] JohnLewis: hi. Just a curiosity question but if we ever for whatever reason needed to override a setting in ManageWiki for all wikis, how would that be done? [13:08:31] Reception123: what do you mwan? [13:08:33] *mean [14:02:57] JohnLewis: for example, if for some reason we needed to override all settings by setting something to false [14:03:07] how could we do that? [14:13:03] In LS, bottom of file [14:24:28] PROBLEM - misc2 Redis Process on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:25:09] PROBLEM - misc2 Disk Space on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:25:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:25:17] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:25:21] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [14:25:26] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:42] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:25:47] PROBLEM - misc2 SSH on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:56] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [14:25:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:26:26] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw2 mw3 [14:26:47] PROBLEM - Host misc2 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:58] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 59% [14:29:32] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 50% [14:30:57] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [14:32:54] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 54% [14:34:55] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 57% [14:35:32] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 37% [14:36:49] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 37% [14:36:53] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [14:38:52] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 57% [14:42:45] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 58% [14:42:50] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [14:47:54] You guys know about the 503 errors right? [14:48:29] I do now. JohnLewis ^^ (If you have access) [14:48:37] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [14:48:46] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 54% [14:50:07] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 49% [14:50:35] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 31% [14:50:45] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [14:52:02] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 32% [14:52:58] dealing [14:53:11] Oh misc2 [14:54:21] susp [14:54:33] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [14:54:43] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 47% [14:54:54] Ah ok [14:55:12] JohnLewis: we could request additional bandwith right because mw* connects to it? [14:55:55] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 42% [14:56:31] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 58% [14:57:50] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 35% [14:58:43] yes [14:58:52] but bandwidth usage is getting rather ridiculous now [14:59:03] there is no way this month we've increased usage this much [14:59:17] Indeed [14:59:34] But honestly I have no idea what the cause of it is. [15:03:25] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhzyR [15:03:27] [02miraheze/mw-config] 07JohnFLewis 034b46455 - drop redis caching to DB temp [15:04:37] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [15:06:18] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [15:06:42] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_jobqueue) or PHP errors [15:06:56] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3830 bytes in 0.984 second response time [15:07:32] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [15:07:42] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3817 bytes in 0.019 second response time [15:08:14] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3819 bytes in 0.391 second response time [15:11:18] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhzS5 [15:11:19] [02miraheze/mw-config] 07JohnFLewis 030137f9f - use correct name [15:12:08] I expect everyone on Wikia has been checking out Miraheze, but hasn't necessarily become active yet. [15:12:45] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:12:45] There is no way to be sure how much of that traffic will persist, it could be mainly temporary interest. [15:12:57] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23711 bytes in 1.475 second response time [15:13:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:13:32] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 12% [15:13:42] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23711 bytes in 0.028 second response time [15:13:50] !log falling back to Redis for session and main caching temporarily [15:13:56] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:13:58] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 23745 bytes in 0.488 second response time [15:14:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:14:08] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 37% [15:14:26] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [15:14:31] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 9% [15:14:52] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [15:15:11] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:15:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:16:30] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 9.22, 5.40, 2.77 [15:17:02] we up? [15:18:05] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhzHo [15:18:07] [02miraheze/mw-config] 07JohnFLewis 03f3f7912 - SN [15:18:23] Voidwalker: depends on your definition of "up" but readable, maybe editable, maybe functionable [15:18:55] k [15:19:18] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40384 MB (11% inode=95%); [15:23:03] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:18] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40296 MB (10% inode=95%); [15:27:02] RECOVERY - Host misc2 is UP: PING OK - Packet loss = 54%, RTA = 0.58 ms [15:27:19] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 0.30, 0.09, 0.03 [15:27:30] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 45587 bytes in 1.654 second response time [15:27:32] RECOVERY - misc2 Redis Process on misc2 is OK: PROCS OK: 1 process with args 'redis-server' [15:28:12] RECOVERY - misc2 SSH on misc2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [15:28:22] RECOVERY - misc2 Disk Space on misc2 is OK: DISK OK - free space: / 34536 MB (84% inode=99%); [15:28:55] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet last ran 1 hour ago [15:32:40] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhz7v [15:32:41] [02miraheze/mw-config] 07JohnFLewis 033ec3f07 - Revert "drop redis caching to DB temp" This reverts commit 4b46455c36754760abd8b912733214ed1c3e8997. [15:32:52] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:36:30] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 2.30, 7.01, 7.35 [15:38:30] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 1.95, 5.39, 6.72 [16:09:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [16:09:10] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [17:02:11] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhzxw [17:02:12] [02miraheze/mw-config] 07JohnFLewis 0385d6950 - rm SN [18:36:44] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3380 [18:38:26] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhgTd [18:38:27] [02miraheze/mw-config] 07Southparkfan 03f8ae982 - Set $wgUseLocalMessageCache to true [20:58:42] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [21:09:33] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhgZZ [21:09:35] [02miraheze/ManageWiki] 07JohnFLewis 0314fb3d8 - convert arrays to objects before jsonifying [21:10:39] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhgZC [21:10:41] [02miraheze/mediawiki] 07JohnFLewis 03d9fc540 - upd MW [21:16:44] JohnLewis: isent that suppose to be is_object? [21:16:58] no [21:17:56] Ok [21:19:16] afterall, if it's an object why would we need to cast it as an object? [21:19:57] Yup, just realised that [21:20:38] that doesn't appear to have fixed it [21:20:48] no, it has [21:20:52] that's why I said code wise [21:22:13] then what needs to be done to make it work properly? [21:22:35] cache needs to be updated [21:22:57] ^^ [21:23:01] Purge with fire! [21:23:14] uh [21:23:34] I did that on a test instance and it did nothing [21:23:38] if anyone can purge server side files that's a massive security risk [21:24:22] Uh, who said anything about deleting files JohnLewis ? [21:24:35] I said purge which could have ment using the hook [21:24:37] me when I said "cache needs to be updated" [21:24:56] again, if someone without server access can do that :) [21:25:03] Oh [21:25:18] Yeh I didn’t mean non ops purge :P [21:29:16] [02ManageWiki] 07The-Voidwalker opened pull request 03#69: fix globals - 13https://git.io/fhgn6 [21:29:36] that's the fix :P [21:30:35] [02ManageWiki] 07paladox closed pull request 03#69: fix globals - 13https://git.io/fhgn6 [21:30:36] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhgn1 [21:30:37] [02miraheze/ManageWiki] 07The-Voidwalker 03325f22a - fix globals (#69) [21:30:40] oh... [21:30:53] JohnLewis: or SPF|Cloud pls deploy ^^ [21:31:05] note to self: create ticket for deprecating using globals and instead use Config object [21:31:09] * SPF|Cloud tries [21:31:43] though my change actually is a fix [21:32:00] as otherwise 0 only would break settings (as we've experienced in the past) [21:33:02] Oh yay [21:33:09] So does that mean 0 works now? [21:33:20] yes [21:33:34] Woo thanks JohnLewis! [21:34:20] something is horribly broken somewhere [21:34:40] I've got the entirety of wgGroupPermissions displayed on the top of RecentChanges [21:35:11] Heh [21:35:15] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhgcT [21:35:17] [02miraheze/mediawiki] 07Southparkfan 034bf49d7 - Update ManageWiki [21:35:21] * paladox does too [21:35:34] Thanks SPF|Cloud [21:37:03] deployed [21:37:13] Thanks! [21:38:29] that did the trick [21:38:54] and I'm no longer getting that debug info on the top of RC [22:45:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhg4s [22:45:47] [02miraheze/puppet] 07paladox 03c555a5e - Update config.yaml.erb [22:45:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhg4G [22:46:00] [02miraheze/puppet] 07paladox 0321c37df - Update config.yaml.erb