[00:04:12] !log depool mw1 for upgrade to mw1 [00:04:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:04:20] uh, /me fixes that [00:04:25] that's suppose to say mw 1.33 [00:05:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjKbJ [00:06:01] [02miraheze/puppet] 07paladox 03ba78cff - mw1: Upgrade to mediawiki 1.33 [00:08:28] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: mw upgrade - paladox, last run 5 minutes ago with 0 failures [00:08:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [00:09:17] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [00:10:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [00:12:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:12:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:13:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:14:28] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 36 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:20:28] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:22:14] !log repool mw1 [00:22:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:22:25] !log depool mw3 for mw upgrade to 1.33 [00:22:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:24:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjKbY [00:24:06] [02miraheze/puppet] 07paladox 03ab08df0 - mw3: Upgrade to mediawiki 1.33 [00:25:24] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: mw upgrade - paladox, last run 2 minutes ago with 0 failures [00:26:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [00:26:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [00:27:09] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_jobqueue) or PHP errors [00:27:17] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [00:29:09] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [00:30:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:30:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:31:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:31:23] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:35:03] !log repool mw2 [00:35:06] !loguh [00:35:09] bad [00:35:13] *bah [00:35:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:35:40] !log depool mw2 to upgrade to mw 1.33 [00:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:36:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjKbz [00:36:08] [02miraheze/puppet] 07paladox 03b8bd32f - mw2: Upgrade to mediawiki 1.33 [00:39:12] !log running lc on mw1 and mw3 [00:39:17] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [00:39:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:39:55] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: mw upgrade - paladox, last run 7 minutes ago with 0 failures [00:40:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [00:40:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [00:42:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:42:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:42:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.95, 7.39, 5.44 [00:43:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:45:55] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:48:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.24, 7.16, 6.07 [00:50:20] !log repool mw2 [00:50:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:50:33] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.20, 6.40, 5.90 [00:54:27] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKbP [00:54:29] [02miraheze/mediawiki] 07paladox 035c72fb9 - Update pivot [00:57:56] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKbD [00:57:58] [02miraheze/mediawiki] 07paladox 03e862064 - Update foreground [00:58:44] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKby [00:58:46] [02miraheze/mediawiki] 07paladox 037b2e20c - Update SocialProfile [01:08:19] !log running update.php for all wikis (on mw1 in a screen) [01:08:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:15:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKbp [01:15:41] [02miraheze/mediawiki] 07paladox 036ba9d92 - Update DiscordNotifications [01:21:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjKNe [01:21:11] [02miraheze/puppet] 07paladox 0391a74bc - matomo: Upgrade to 3.10.0 [01:42:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjKNs [01:43:00] [02miraheze/puppet] 07paladox 03fe9a524 - matomo: Use file lock for core:archive command This will prevent multiple consoles from running. Also run it as root so that flock works. [01:54:15] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjKNW [01:54:16] [02miraheze/MatomoAnalytics] 07paladox 038102daf - Add missing "formDescriptor" variable [01:55:08] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKNl [01:55:10] [02miraheze/mediawiki] 07paladox 035bde0a2 - Update MA [01:56:22] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±2] 13https://git.io/fjKNB [01:56:24] [02miraheze/mediawiki] 07paladox 033422b9d - Update Cargo and PF [02:16:14] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKNo [02:16:15] [02miraheze/mediawiki] 07paladox 03e7cec78 - Update ModernSkylight [02:17:10] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKNK [02:17:12] [02miraheze/mediawiki] 07paladox 032b56c2f - Update ArticleFeedbackv5 [02:18:13] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjKNP [02:18:14] [02miraheze/mediawiki] 07paladox 03f5e264e - Updatem ArticleFeedbackv5 [02:59:43] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CRITICAL: Full, 671533 files, 76.35GB, 2019-06-02 02:58:00 (4.4 weeks ago) [04:09:04] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.69, 7.07, 4.86 [04:11:05] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.32, 6.56, 4.95 [04:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjKAK [04:35:12] [02miraheze/services] 07MirahezeSSLBot 0307a9b90 - BOT: Updating services config for wikis [04:41:27] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.77, 3.63, 3.19 [04:43:27] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 1.41, 2.87, 2.97 [04:50:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.28, 5.97, 4.10 [04:54:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.74, 7.44, 5.17 [04:56:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.56, 6.36, 5.05 [05:41:01] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 1.74, 0.91, 0.35 [05:43:01] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 1.05, 0.95, 0.43 [06:01:19] 503 on Testwiki [06:01:19] PROBLEM - misc3 Lizardfs Master Port 1 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9419: Connection refused [06:01:40] PROBLEM - misc3 Lizardfs Master Port 2 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9420: Connection refused [06:02:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:02:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:02:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [06:02:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [06:03:11] https://phabricator.miraheze.org/T4509 [06:03:11] [ ⚓ T4509 503 backend fetch failed on public test wiki on cp4 ] - phabricator.miraheze.org [06:03:17] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [06:03:19] RECOVERY - misc3 Lizardfs Master Port 1 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9419 [06:03:30] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [06:03:36] JohnLewis, Paladox: ^^^^^ UNBREAK NOE [06:03:40] RECOVERY - misc3 Lizardfs Master Port 2 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9420 [06:03:40] NOW [06:04:02] no. do it later [06:04:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:04:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:04:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [06:04:34] Mikeee: looks like everything went down [06:04:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [06:04:39] take a nap and a long lunch [06:05:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [06:05:30] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 6% [06:06:20] And restored [06:36:28] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [06:42:18] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.58, 3.27, 2.19 [06:44:15] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 2.75, 3.06, 2.25 [06:44:27] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:02:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 14.11, 7.98, 4.61 [07:03:43] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.93, 3.57, 3.01 [07:05:40] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.03, 3.39, 3.00 [07:09:28] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.31, 6.28, 3.44 [07:09:58] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.44, 6.74, 4.79 [07:11:28] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 2.49, 4.76, 3.23 [07:11:56] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.44, 5.42, 4.53 [07:12:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.32, 7.28, 6.47 [07:14:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.31, 5.94, 6.07 [07:23:21] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:25:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.17, 6.35, 6.02 [07:25:18] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [07:27:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.82, 5.59, 5.80 [08:13:05] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.42, 6.17, 4.53 [08:15:05] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.26, 5.08, 4.33 [08:34:09] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.71, 3.20, 2.92 [08:36:05] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.37, 3.57, 3.08 [08:38:02] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.43, 3.57, 3.14 [08:51:37] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.24, 3.99, 3.61 [08:57:27] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.54, 3.90, 3.70 [09:01:27] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.49, 4.04, 3.77 [09:07:27] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.26, 3.81, 3.76 [09:11:27] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.41, 4.04, 3.85 [09:15:27] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.81, 3.97, 3.87 [09:31:27] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.32, 3.85, 3.76 [09:37:27] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.36, 3.87, 3.83 [09:39:27] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.87, 4.11, 3.91 [09:43:27] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.72, 3.94, 3.88 [09:49:27] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 1.11, 2.53, 3.35 [10:22:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 11.84, 7.29, 4.41 [10:25:06] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 11.16, 7.57, 4.82 [10:27:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.79, 6.59, 4.81 [10:28:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.35, 7.24, 5.58 [10:30:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.95, 6.36, 5.45 [11:25:06] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 11.66, 6.83, 4.42 [11:25:31] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.15, 5.53, 3.20 [11:26:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [11:26:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [11:27:17] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [11:28:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 12.41, 8.57, 5.20 [11:28:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [11:28:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:29:05] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.38, 6.94, 5.11 [11:29:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [11:29:28] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.36, 6.07, 4.01 [11:30:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.63, 7.58, 5.26 [11:31:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.52, 6.08, 5.01 [11:33:29] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.60, 6.94, 4.83 [11:34:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.72, 6.18, 5.22 [11:35:28] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.71, 6.08, 4.76 [12:32:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.70, 6.14, 4.33 [12:34:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.97, 6.34, 4.62 [12:36:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.82, 7.40, 5.22 [12:38:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.55, 5.55 [12:40:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.62, 6.41, 5.37 [12:52:48] !log removing a users PII [12:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:00:33] [02ManageWiki] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vpSns [13:00:35] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fj6Tk [13:00:36] [02miraheze/ManageWiki] 07paladox 03f4f4219 - Fix "Field 'perm_addgroupstoself' doesn't have a default value" [13:00:38] [02ManageWiki] 07paladox opened pull request 03#111: Fix "Field 'perm_addgroupstoself' doesn't have a default value" - 13https://git.io/fj6TI [13:01:20] [02ManageWiki] 07JohnFLewis closed pull request 03#111: Fix "Field 'perm_addgroupstoself' doesn't have a default value" - 13https://git.io/fj6TI [13:01:22] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj6Tm [13:01:23] [02miraheze/ManageWiki] 07paladox 03d0a0b59 - Fix "Field 'perm_addgroupstoself' doesn't have a default value" (#111) [13:14:25] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fj6Ta [13:14:26] [02miraheze/mediawiki] 07paladox 03b1fc67a - Update MW [13:21:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj6TP [13:21:06] [02miraheze/mw-config] 07paladox 033607a55 - Remove sitenotice Change-Id: I911c51358e1a98717b5bd6bbf6fead0c3a7da387 [13:28:41] [02mw-config] 07Amanda-Catherine opened pull request 03#2692: Enable partial blocks on quircwiki and bipolartownwiki T4508 - 13https://git.io/fj6Td [13:29:16] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.64, 6.61, 4.64 [13:29:29] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.34, 5.61, 3.42 [13:30:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 11.61, 6.94, 4.47 [13:31:28] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.40, 6.04, 3.85 [13:33:29] paladox: what's up with all the deleted stuff in RC? [13:33:43] ToU enforcement (that's all i can say) :) [13:34:20] Based on the looks it would seem that some username violated the ToU, but I'm not aware of any such clause [13:35:13] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.02, 7.65, 5.83 [13:38:18] So wait a minute... if the violation was anything other than a username, why would the global account log (for presumably a lock) be deleted? [13:39:09] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.11, 5.73, 5.50 [13:39:11] It's not a username violation, but i carn't really discuss why we did it. [13:40:11] Oh great you're acting just like the WMF now [13:42:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.73, 7.21, 6.96 [13:42:24] WMF: "we banned a long-term enwiki admin for good reasons but we can't say why" [13:42:29] AmandaCath we are not, only that we have legal obligations. [13:42:49] Miraheze: "we banned and deleted all logs of a user but can't say why" [13:43:26] paladox: so if that's the case, why does the summary include "contact staff@ with queries" [13:44:02] because we can disclose to the user or other appropriate people(s) [13:44:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.91, 6.23, 6.63 [13:45:49] See, this is why IMHO we should have a separate legal team separate from the overall sysadmin team [13:45:57] Especially if we get incorporated [13:46:18] Currently that's not possible [13:46:29] we need to have a volunteer willing to do that [13:47:06] I know a couple lawyers - not sure if they would be interested though [13:47:38] well atm all legal matters more or less pass through myself or labsters, despite not being qualified we oversee pretty much anything labelled under ToU [13:50:20] Ok, but considering that I didn't know that until now, on first impression I see paladox seemingly trying to erase all evidence of someone even existing, and then saying "ToU violation, we can't discuss it" sounds very WMF-ish [13:50:51] talk to the US legislative [13:51:43] Oh, so now we're dealing with a potential federal law violation? [13:51:56] That's definitely something for a legal team [13:52:23] well, it's a case of we're complying with US law [13:52:37] we've done our part within the outlines of the law so it's all okay [13:53:34] it's not theoretically a violation of law, but rather a violation of our ToU taken from US law which is slightly more strict that law to protect ourselves [13:54:24] I would have just done centralauth-oversight and hideuser to take care of everything in two clicks and without a public log that would raise questions [13:55:11] paladox chose to handle it how he did, all I can say really [13:56:20] Meh... on a different topic.... [13:56:33] how come my PR 2692 isn't showing up on Phab? [14:08:10] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2692: Enable partial blocks on quircwiki and bipolartownwiki T4508 - 13https://git.io/fj6Td [14:09:50] Paladox: Can I get a favour? https://github.com/miraheze/mw-config/pull/2692 [14:09:51] [ Enable partial blocks on quircwiki and bipolartownwiki T4508 by Amanda-Catherine · Pull Request #2692 · miraheze/mw-config · GitHub ] - github.com [14:13:20] JohnLewis: should we enable that ^^ for all wikis? [14:13:33] If not that should be in managewiki [14:13:53] all wikis sounds good, plus I've been saying that since before 1.33 was even finished :) [14:13:56] Also i ’m currently mobile [14:14:29] I would oppose doing a global enable simply for procedural reasons - anything that's not required for site functionality should be opt-in or opt-out able [14:14:51] I would do it through ManageWiki but I don't know how [14:15:04] * AmandaCath needs to figure out how MW works [14:15:58] [02mw-config] 07paladox reviewed pull request 03#2692 commit - 13https://git.io/fj6k1 [14:16:56] afaik it's a temp flag [14:17:05] ^ [14:17:33] and in 1.34 it'll be default behaviour [14:17:45] JohnLewis: That sounds right [14:18:00] I think it was mentioned in the release email [14:20:10] AmandaCath: can u do the GitHub patch? See review comments [14:21:15] I can do that patch once my network stops crashing :P [14:22:27] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2692: Enable partial blocks on quircwiki and bipolartownwiki T4508 - 13https://git.io/fj6Td [14:22:38] [02mw-config] 07Amanda-Catherine edited pull request 03#2692: Enable partial blocks on all wikis by default - 13https://git.io/fj6Td [14:23:58] [02mw-config] 07JohnFLewis closed pull request 03#2692: Enable partial blocks on all wikis by default - 13https://git.io/fj6Td [14:23:59] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj6kF [14:24:01] [02miraheze/mw-config] 07Amanda-Catherine 03f37c603 - Enable partial blocks on all wikis by default (#2692) * Enable partial blocks on quircwiki and bipolartownwiki T4508 * Add publictestwiki per request * Enable partial blocks for all wikis by default [14:24:29] Thanks John [14:24:34] & Amanda [14:30:43] AmandaCath: [14:30:44] https://phabricator.miraheze.org/T4510 we are already on MW 1.33 [14:30:46] [ ⚓ T4510 Install MediaWiki 1.33 ] - phabricator.miraheze.org [14:30:48] since last night :) [14:33:47] Ah ok [14:34:19] Just out of curiosity, how many of MH staff are on the mailing list? [14:34:23] * Hispano76 saluda [14:34:47] There’s me so one :P [14:39:07] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.70, 6.19, 4.65 [14:40:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.44, 6.83, 4.84 [14:41:58] JohnLewis: can you take a look at ^^ please? [14:42:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.11, 5.85, 4.73 [14:43:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.66, 5.93, 4.94 [14:45:30] paladox, JohnLewis: can one of you undo autoblock #297 on publictestwiki.com pls [14:51:15] fixed [15:01:10] also can someone look at https://publictestwiki.com/wiki/TestWiki:Request_permissions#RhinosF1_.28Status:_Open.29 [15:01:11] [ TestWiki:Request permissions - TestWiki ] - publictestwiki.com [15:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj6Lr [15:25:13] [02miraheze/services] 07MirahezeSSLBot 037bdc54c - BOT: Updating services config for wikis [15:54:20] RECOVERY - wiki.pyxxl.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pyxxl.tk' will expire on Tue 10 Sep 2019 12:46:18 PM GMT +0000. [16:05:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj6t0 [16:05:14] [02miraheze/services] 07MirahezeSSLBot 03251831a - BOT: Updating services config for wikis [17:18:24] ping paladox [17:18:29] hi [17:19:22] because several deleted wikis (centralwiki..) are still listed in https://meta.miraheze.org/w/index.php?title=Special%3ACentralAuth&target=Hispano76 curiosity [17:19:23] [ Global account information for Hispano76 - Miraheze Meta ] - meta.miraheze.org [17:19:25] paladox, [17:20:11] probably because they haven't been completely deleted yet [17:20:13] Hispano76 the wikis are technically not deleted, in so far as they are only hidden from the public (accessing the url) [17:21:34] hum? [17:21:58] will it be like this now? paladox [17:22:07] Hispano76 the wiki is deleted so that it's hidden (acts like it's deleted), it will be deleted later. [17:22:40] aah, ok [17:22:59] thanks paladox :) [17:23:04] your welcome :) [17:42:11] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj6qb [17:42:13] [02miraheze/CreateWiki] 07JohnFLewis 03c09806b - introduce another database connection [17:42:16] paladox: ^ [17:42:26] * paladox looks [17:42:56] Thanks! [17:43:04] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.43, 5.88, 4.34 [17:43:19] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fj6qA [17:43:34] ok, what is causing the high load on mw1.. [17:43:35] [02miraheze/mediawiki] 07paladox 031bb83ee - Update CW [17:43:45] was going to do it by using selectDomain then checking for an exception for that's cleaner [17:44:17] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 14.08, 8.22, 5.26 [17:45:05] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.18, 6.08, 4.62 [17:45:16] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [17:45:28] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.92, 5.88, 4.15 [17:47:28] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.85, 5.54, 4.24 [17:54:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.15, 7.25, 6.43 [17:55:05] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.37, 7.93, 6.04 [17:56:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.26, 6.14, 6.12 [17:57:04] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.74, 7.53, 6.13 [17:59:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.50, 6.43, 5.89 [18:03:19] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [18:05:18] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:35:28] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.84, 6.63, 4.94 [18:37:29] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.49, 5.77, 4.83 [18:55:28] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.69, 7.95, 6.48 [18:56:28] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.60, 6.85, 5.11 [18:57:06] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.40, 6.40, 4.70 [19:00:23] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.52, 6.61, 5.43 [19:01:05] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.77, 7.28, 5.50 [19:03:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.31, 6.21, 5.33 [19:03:29] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.47, 7.75, 7.08 [19:05:29] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.54, 7.97, 7.23 [19:07:32] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.21, 7.63, 7.20 [19:10:35] just noting here that we are looking into ^^ [19:15:29] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.85, 8.21, 7.43 [19:19:29] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.87, 7.63, 7.39 [19:23:28] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.11, 7.43, 7.32 [19:27:31] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.23, 7.71, 7.48 [19:34:35] !log installing https://github.com/martin-helmich/prometheus-nginxlog-exporter on test1 [19:34:36] [ GitHub - martin-helmich/prometheus-nginxlog-exporter: Export metrics from Nginx access log files to Prometheus ] - github.com [19:34:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:35:35] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.56, 8.23, 7.71 [20:01:34] Error 503 site-wide [20:02:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:02:36] Paladox: ^^ [20:02:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:02:42] hmm [20:02:44] * paladox checks [20:02:55] AmandaCath: Known, was reported this morning and been happening all day [20:02:56] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:02:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:03:09] Paladox: u closed my phab task reporting it. [20:03:10] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 14.40, 8.04, 4.91 [20:03:58] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:04:03] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 48% [20:04:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 12.16, 8.99, 5.54 [20:04:31] https://phabricator.miraheze.org/T4509 For the record [20:04:32] [ ⚓ T4509 503 backend fetch failed on Most/all servers ] - phabricator.miraheze.org [20:04:48] !log restarting php-fpm across mw* [20:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:04:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:06:03] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 17% [20:06:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.14, 7.09, 5.27 [20:06:48] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:06:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+5/-0/±0] 13https://git.io/fj6Ou [20:06:57] [02miraheze/puppet] 07paladox 0326e456c - prometheus: Add nginx exporter [20:07:07] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.10, 7.99, 5.75 [20:07:56] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:08:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.46, 5.90, 5.06 [20:08:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:08:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:08:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+5/-0/±0] 13https://git.io/fj6Og [20:08:48] [02miraheze/puppet] 07paladox 0351fe8fe - prometheus: Add nginx exporter [20:09:01] On a different topic.. why do we have global abuse filter with blocking enabled (30)? [20:09:05] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.28, 6.33, 5.40 [20:09:25] There aren't any globally banned users besides LP that I'm aware of [20:09:42] Ping Voidwalker as most recent filter editor [20:12:42] !log hacking puppet manifest to enable nginx exporter on test1 [20:13:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:16:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj6OP [20:16:02] [02miraheze/puppet] 07paladox 032d9dda0 - prometheus: Fix nginx systemd service [20:19:50] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [20:27:32] !log adding nginx exporter to mw1 [20:27:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:28:14] !log make that all mw* [20:28:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:29:34] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.81, 6.70, 7.95 [20:43:28] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.21, 7.11, 7.28 [20:49:31] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.78, 7.87, 7.60 [20:55:31] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.53, 8.15, 7.70 [20:57:29] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 3.99, 6.93, 7.33 [20:57:38] paladox: given the load issues is it best I hold off bot testing [20:57:52] RF1 yes please. [20:59:25] paladox: thx [20:59:28] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 2.18, 5.26, 6.67 [21:01:04] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.04, 6.51, 4.52 [21:03:04] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.99, 6.84, 4.90 [21:05:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.47, 5.92, 4.79 [21:18:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.93, 5.97, 4.69 [21:20:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.60, 5.97, 4.83 [22:23:04] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.18, 6.09, 4.62 [22:25:05] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.35, 5.41, 4.55 [23:15:10] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [23:15:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/fj6Ge [23:15:13] [02miraheze/puppet] 07paladox 03b2a0c60 - prometheus: Include nginx exporter in nginx class [23:15:14] [02puppet] 07paladox opened pull request 03#1023: prometheus: Include nginx exporter in nginx class - 13https://git.io/fj6Gv [23:16:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/fj6Gf [23:16:48] [02miraheze/puppet] 07paladox 03f9c12c4 - Update prometheus.yml.erb [23:16:49] [02puppet] 07paladox synchronize pull request 03#1023: prometheus: Include nginx exporter in nginx class - 13https://git.io/fj6Gv [23:17:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/fj6GU [23:17:54] [02miraheze/puppet] 07paladox 034ea41b5 - Update init.pp [23:17:55] [02puppet] 07paladox synchronize pull request 03#1023: prometheus: Include nginx exporter in nginx class - 13https://git.io/fj6Gv [23:18:35] [02puppet] 07paladox closed pull request 03#1021: Temporarily blacklist MwClient user agent - 13https://git.io/fjocC [23:18:39] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:18:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:26:00]