[00:06:18] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.49, 1.91, 1.19 [00:08:16] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.71, 1.43, 1.10 [00:26:10] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.58, 2.90, 2.28 [00:30:06] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 5.99, 4.19, 2.91 [00:32:04] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 2.83, 3.80, 2.94 [00:34:02] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.20, 3.21, 2.82 [00:35:55] PROBLEM - cp11 HTTP 4xx/5xx ERROR Rate on cp11 is WARNING: WARNING - NGINX Error Rate is 44% [00:37:55] RECOVERY - cp11 HTTP 4xx/5xx ERROR Rate on cp11 is OK: OK - NGINX Error Rate is 2% [00:48:06] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 107s [00:50:06] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [01:11:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.93, 3.85, 4.00 [01:13:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.44, 4.10, 4.07 [01:14:10] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 293s [01:21:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.83, 3.61, 3.91 [01:24:08] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 128s [01:25:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.02, 3.74, 3.88 [01:26:08] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [01:29:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.60, 3.83, 3.90 [01:31:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.06, 4.30, 4.06 [01:37:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.51, 3.84, 3.95 [01:39:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.85, 4.15, 4.04 [01:43:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.50, 3.87, 3.95 [01:45:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.44, 4.47, 4.16 [01:48:09] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 263s [01:56:08] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 136s [02:02:07] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [02:17:23] JohnLewis, if you're still up and around, every wiki I've approved in the past few hours, or even ones I've declined, has had an approval score of 0.15. That is very odd as that's never happened before and looking at the most recent one for `30xxwiki`, it probably should've between 0.65-0.90 I'd say, based on similar gaming informational wikis with comparable descriptions. Not sure if there might be a tweak we need to make to the AI scoring [02:17:23] code or something [02:20:07] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 336s [02:21:07] AI, as in Artificial Intelligence? [02:21:16] DarkMatterMan4500, yeah [02:21:41] Also, in case you didn't see my message, I apologize for any mishaps I may have caused. [02:22:50] Idea is CreateWiki will learn from and adapt to our approvals and declines, so that eventually, it'll be able to auto-approve or auto-decline wiki requests that meet certain thresholds [02:23:44] Regarding latter part of your comment, yeah, I'll follow up with you on the talk page of one of your wikis with some tips and guidance. Which wiki are you most active on...Crappy Games Wiki? [02:24:37] Yes, Crappy Games Wiki. I did state that I should be a bit more careful from here on out. [02:25:09] ok, sounds good. Yeah, I'll try and follow up tomorrow morning, as it's been a long day [02:25:44] I could see why you haven't been seen on Discord all day or why your Discord profile was offline. [02:26:16] yeah... I have to check my Discord still lol [02:26:57] Yeah, but hopefully your follow-up will probably still be positive either way. [02:27:28] 👍 [02:28:12] And besides, you live in my timezone, which if I'm not mistaken, it's probably just before 9:30PM in your area, right? [02:30:07] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [03:32:07] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 333s [03:36:07] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 12s [03:59:54] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.95, 2.96, 1.90 [04:00:09] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 284s [04:01:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.15, 3.44, 2.21 [04:03:46] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqW4o [04:03:48] [02miraheze/MirahezeMagic] 07Universal-Omega 030162fb4 - Use WikiForum new schema with removePII [04:03:49] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [04:03:51] [02MirahezeMagic] 07Universal-Omega opened pull request 03#219: Use WikiForum new schema with removePII - 13https://git.io/JqW4K [04:04:46] miraheze/MirahezeMagic - Universal-Omega the build passed. [04:07:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.97, 3.94, 2.85 [04:13:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.02, 3.89, 3.17 [04:15:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.83, 3.89, 3.27 [04:17:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.72, 4.29, 3.49 [04:19:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.25, 3.55, 3.96 [04:19:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.29, 3.96, 3.47 [04:21:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.16, 3.89, 3.49 [04:23:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.75, 3.88, 3.54 [04:25:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.75, 3.92, 3.93 [04:27:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.32, 3.65, 3.83 [04:29:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.55, 3.99, 3.93 [04:29:52] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.54, 3.20, 3.35 [04:39:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.77, 3.98, 4.00 [04:41:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.17, 4.45, 4.17 [04:45:22] [02MirahezeMagic] 07Universal-Omega edited pull request 03#219: Use WikiForum new schema with removePII - 13https://git.io/JqW4K [04:51:32] [02MirahezeMagic] 07Universal-Omega edited pull request 03#219: Use WikiForum new schema with removePII - 13https://git.io/JqW4K [04:51:46] [02MirahezeMagic] 07Universal-Omega edited pull request 03#219: Use WikiForum new schema with removePII (T6961) - 13https://git.io/JqW4K [04:53:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.81, 3.81, 3.97 [04:53:32] [02MirahezeMagic] 07Universal-Omega closed pull request 03#219: Use WikiForum new schema with removePII (T6961) - 13https://git.io/JqW4K [04:53:33] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWRv [04:53:35] [02miraheze/MirahezeMagic] 07Universal-Omega 034713176 - Use WikiForum new schema with removePII (#219) [04:53:36] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [04:53:38] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:54:31] miraheze/MirahezeMagic - Universal-Omega the build passed. [04:55:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.83, 4.23, 4.11 [04:56:55] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWRt [04:56:56] [02miraheze/MirahezeMagic] 07Universal-Omega 031fc3505 - Remove cw_requests.cw_ip from removePII (T6961) [04:56:58] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [04:56:59] [02MirahezeMagic] 07Universal-Omega opened pull request 03#220: Remove cw_requests.cw_ip from removePII (T6961) - 13https://git.io/JqWRq [04:57:10] [02MirahezeMagic] 07Universal-Omega closed pull request 03#220: Remove cw_requests.cw_ip from removePII (T6961) - 13https://git.io/JqWRq [04:57:11] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWRY [04:57:13] [02miraheze/MirahezeMagic] 07Universal-Omega 0321876c4 - Remove cw_requests.cw_ip from removePII (T6961) (#220) [04:57:14] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [04:57:16] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:58:06] miraheze/MirahezeMagic - Universal-Omega the build passed. [04:58:16] miraheze/MirahezeMagic - Universal-Omega the build passed. [05:27:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.61, 3.81, 3.99 [05:29:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.45, 4.05, 4.05 [07:42:39] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWuw [07:42:40] [02miraheze/MirahezeMagic] 07Universal-Omega 03c5bb656 - Fix typo [07:43:34] miraheze/MirahezeMagic - Universal-Omega the build passed. [07:52:06] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 80s [10:58:22] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=besttvshowswiki --r "[[phabricator:T6957|Requested]]" /home/reception/BTVSW.txt [10:58:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:00:01] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=crappygameswiki --r "[[phabricator:T6957|Requested]]" /home/reception/CGW.txt [11:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:19:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.07, 3.53, 3.94 [11:21:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.73, 4.07, 4.09 [11:45:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.75, 3.81, 4.00 [11:51:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.29, 4.03, 4.03 [12:17:46] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=loathsomecharacterswiki --r "[[phabricator:T6957|Requested]]" /home/reception/LSCW.txt [12:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:41:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.80, 3.77, 3.97 [12:43:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.50, 4.03, 4.04 [12:45:13] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=greatcharacterswikiwiki --r "[[phab:T6967|Requested]]" /home/reception/GCW.txt [12:45:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:08:06] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 169s [13:10:06] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:28:59] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.43, 2.84, 2.02 [13:30:57] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 3.00, 2.70, 2.06 [13:45:43] !ops Sorry! This site is experiencing technical difficulties. Try waiting a few minutes and reloading. For more information on this error please check out our twitter page (Twitter.com/Miraheze) or contact the system administrators by connecting to #miraheze on irc.freenode.net (Cannot access the database: Cannot access the database: Unknown error (db11.miraheze.org)) [13:45:58] Huh [13:46:17] Which wiki @R4356th [13:46:22] Meta [13:46:30] Trying to review WRs. [13:46:54] Alright. Everything seems normal now. [13:51:56] Okay [13:52:02] paladox: can you check ^ [14:07:41] Reception123: https://phabricator.miraheze.org/T6963 [14:07:42] [ ⚓ T6963 loginwiki not found in matomo ] - phabricator.miraheze.org [14:08:32] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqByu [14:08:33] [02miraheze/CreateWiki] 07paladox 03e4b116b - Fix "Undefined property: RequestWikiAIJob::$description" [14:08:35] [02CreateWiki] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vpJTL [14:08:36] [02CreateWiki] 07paladox opened pull request 03#204: Fix "Undefined property: RequestWikiAIJob::$description" - 13https://git.io/JqByo [14:09:13] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqB9T [14:09:14] [02miraheze/CreateWiki] 07paladox 03928a55a - Update RequestWikiAIJob.php [14:09:16] [02CreateWiki] 07paladox synchronize pull request 03#204: Fix "Undefined property: RequestWikiAIJob::$description" - 13https://git.io/JqByo [14:09:34] miraheze/CreateWiki - paladox the build passed. [14:09:43] ^ paladox looks like that thing is happening again, like it did before with meta where it no longer shows up in Matomo [14:10:05] looks like a bug in the Matomo extension then [14:10:14] miraheze/CreateWiki - paladox the build passed. [14:10:23] it's weird, sometimes it seems wikis just randomly no longer show up [14:10:32] paladox: how did you bring them back? [14:10:42] that's because its a bug when deleting a wiki [14:10:57] i just recreated it [14:11:18] what's loginwiki got to do with deleting a wiki though? [14:11:48] Isn't the script run with --wiki=loginwiki [14:12:00] yes [14:12:07] well iirc before it was metawiki that got randomly deleted [14:12:12] [02CreateWiki] 07paladox closed pull request 03#204: Fix "Undefined property: RequestWikiAIJob::$description" - 13https://git.io/JqByo [14:12:13] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqBxg [14:12:15] [02miraheze/CreateWiki] 07paladox 03f8eec3f - Fix "Undefined property: RequestWikiAIJob::$description" (#204) [14:12:16] [02CreateWiki] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vpJTL [14:12:18] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-2 [14:13:09] miraheze/CreateWiki - paladox the build passed. [14:13:12] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqBjz [14:13:13] [02miraheze/mediawiki] 07paladox 0307fafaa - Update CreateWiki [14:19:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.21, 3.59, 4.00 [14:21:22] For R4356th's error it is: Error connecting to db11.miraheze.org as user mediawiki: :real_connect(): (HY000/1049): Unknown database 'admiraltvwiki' [14:21:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.14, 3.85, 4.04 [14:24:14] JohnLewis: failed creation or? [14:24:43] Wiki seems created and working [14:24:56] paladox: any ideas about a more permanent fix for https://phabricator.miraheze.org/T6963? [14:24:57] [ ⚓ T6963 loginwiki not found in matomo ] - phabricator.miraheze.org [14:25:06] Was created 13:44 [14:25:23] No, i thought i fixed it but apparently not. [14:25:25] JohnLewis: was created one minute before the report [14:25:51] Well the report was because of it, that one minute difference was reporting time [14:39:24] i can have a look [14:47:28] I'm confused on how it can be deleted. Looking at the hook, it doesn't use wgDBname rather the wiki being deleted. [14:52:03] Which one? [14:54:15] looks like this first started on march 11th [14:54:46] @DarkMatterMan4500: whichever was inactive [14:54:54] Oh. [14:55:53] Pal.adox is debugging broken matomo handling [14:59:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.44, 2.67, 1.64 [15:00:06] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 224s [15:01:52] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.15, 3.49, 2.08 [15:03:52] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.00, 2.98, 2.07 [15:04:06] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [15:10:04] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqEmE [15:10:06] [02miraheze/MatomoAnalytics] 07paladox 0333e2572 - Add some more logging and also disable cache when deleting/rename a wiki [15:10:07] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fN4LT [15:10:09] [02MatomoAnalytics] 07paladox opened pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:10:17] think i found the bug [15:10:41] JohnLewis could you review ^ please? [15:10:57] also regarding casting, do we even need it? or is it safe without needing to do the casting [15:11:09] miraheze/MatomoAnalytics - paladox the build passed. [15:11:11] paladox: busy, sorry [15:11:50] ok [15:12:26] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqEGZ [15:12:28] [02miraheze/MatomoAnalytics] 07paladox 031261d57 - Update MatomoAnalytics.php [15:12:29] [02MatomoAnalytics] 07paladox synchronize pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:13:24] miraheze/MatomoAnalytics - paladox the build passed. [15:15:15] confirmed that's the bug [15:15:18] tested [15:20:08] confirmed my change fixes it [15:24:03] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqEiW [15:24:05] [02miraheze/MatomoAnalytics] 07paladox 035dcc977 - Update MatomoAnalytics.php [15:24:06] [02MatomoAnalytics] 07paladox synchronize pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:25:04] miraheze/MatomoAnalytics - paladox the build passed. [15:25:05] Reception123 could you review ^ please? [15:25:24] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqEXh [15:25:25] [02miraheze/MatomoAnalytics] 07paladox 039a3a6d3 - Update MatomoAnalytics.php [15:25:27] [02MatomoAnalytics] 07paladox synchronize pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:26:19] miraheze/MatomoAnalytics - paladox the build passed. [15:26:50] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqEDM [15:26:52] [02miraheze/MatomoAnalytics] 07paladox 03f03dbbd - Update MatomoAnalytics.php [15:26:53] [02MatomoAnalytics] 07paladox synchronize pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:26:56] paladox: what about your other PRs? [15:27:04] what other pr? [15:27:34] Looks good as far as I can tell, but that doesn't mean too much coming from me :P [15:27:49] miraheze/MatomoAnalytics - paladox the build passed. [15:28:06] paladox: 2 from 14/15 days ago [15:28:18] link? [15:28:27] Nothing screams at me but like Reception123 that means it's valid php not that it works [15:28:41] paladox: https://github.com/miraheze/MatomoAnalytics/pulls [15:28:42] [ Pull requests · miraheze/MatomoAnalytics · GitHub ] - github.com [15:29:00] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqEHb [15:29:01] [02miraheze/MatomoAnalytics] 07paladox 03ac15092 - Update CHANGELOG [15:29:03] [02MatomoAnalytics] 07paladox synchronize pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:29:10] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqEQO [15:29:12] [02miraheze/MatomoAnalytics] 07paladox 03112167a - Update extension.json [15:29:13] [02MatomoAnalytics] 07paladox synchronize pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:29:59] miraheze/MatomoAnalytics - paladox the build passed. [15:30:14] miraheze/MatomoAnalytics - paladox the build passed. [15:30:33] oh, that can be closed for now. I may look at https://github.com/miraheze/MatomoAnalytics/pull/40/files in the future. But for now my new pull fixes the current pressing issue. [15:30:33] [ Fix getSiteID so it doesn't return a default site id by paladox · Pull Request #40 · miraheze/MatomoAnalytics · GitHub ] - github.com [15:30:39] [02MatomoAnalytics] 07paladox closed pull request 03#40: Fix getSiteID so it doesn't return a default site id - 13https://git.io/JtxCX [15:30:41] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-2 [15:30:42] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fN4LT [15:30:53] [02MatomoAnalytics] 07paladox closed pull request 03#36: MatomoAnalytics: Cache get within getSiteID - 13https://git.io/JtA0W [15:30:55] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fN4LT [15:30:56] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 [15:35:49] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JqueS [15:35:51] [02miraheze/MatomoAnalytics] 07paladox 03a57209f - Update MatomoAnalytics.php [15:35:53] [02MatomoAnalytics] 07paladox synchronize pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:36:14] [02MatomoAnalytics] 07paladox closed pull request 03#42: Add some more logging and also disable cache when deleting/rename a wiki - 13https://git.io/JqEmo [15:36:16] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jquvo [15:36:17] [02miraheze/MatomoAnalytics] 07paladox 03411142e - Add some more logging and also disable cache when deleting/rename a wiki (#42) [15:36:19] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fN4LT [15:36:20] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-3 [15:36:47] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jquf5 [15:36:49] [02miraheze/mediawiki] 07paladox 030422b21 - Update MatomoAnalytics [15:36:49] miraheze/MatomoAnalytics - paladox the build passed. [15:37:14] miraheze/MatomoAnalytics - paladox the build passed. [15:39:39] !log created zymonicwiki in matomo [15:39:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:47:44] [02mw-config] 07dmehus opened pull request 03#3782: Add Art & Architecture category to CreateWiki - 13https://git.io/Jqu4v [15:48:38] miraheze/mw-config - dmehus the build has errored. [15:50:30] [02mw-config] 07RhinosF1 reviewed pull request 03#3782 commit - 13https://git.io/Jquul [15:50:41] dmehus: ^ [15:50:54] [02mw-config] 07dmehus synchronize pull request 03#3782: Add Art & Architecture category to CreateWiki - 13https://git.io/Jqu4v [15:51:53] miraheze/mw-config - dmehus the build passed. [15:51:53] [02mw-config] 07dmehus reviewed pull request 03#3782 commit - 13https://git.io/JqugM [15:52:33] [02mw-config] 07RhinosF1 reviewed pull request 03#3782 commit - 13https://git.io/Jqu2x [15:53:01] [02mw-config] 07dmehus reviewed pull request 03#3782 commit - 13https://git.io/Jquah [15:54:31] FYI, https://phabricator.miraheze.org/phame/post/view/12/2021_new_year_new_teams_new_focus_same_principles/ is live :) [15:54:31] [ ✩ 2021: New Year, New Teams, New Focus, Same Principles ] - phabricator.miraheze.org [15:54:39] [02mw-config] 07dmehus synchronize pull request 03#3782: Add Art & Architecture category to CreateWiki - 13https://git.io/Jqu4v [15:55:33] miraheze/mw-config - dmehus the build passed. [15:55:36] [02mw-config] 07dmehus reviewed pull request 03#3782 commit - 13https://git.io/JquKg [15:55:45] I've read it. [15:56:11] https://blog.miraheze.org/post/12/2021_new_year_new_teams_new_focus_same_principles/ [15:56:12] [ ✩ 2021: New Year, New Teams, New Focus, Same Principles ] - blog.miraheze.org [15:56:14] Nice url [15:58:25] Reception123: #announcements [15:58:50] You do a MirahezeBot announcement too if you want [15:59:18] I did Discord already, but sure, let's do a MirahezeBot one too [15:59:22] (how do you do that again :P) [15:59:30] .annouce [15:59:38] Do it in -logs [16:00:00] #miraheze-botlogs [16:00:03] [02mw-config] 07Reception123 closed pull request 03#3782: Add Art & Architecture category to CreateWiki - 13https://git.io/Jqu4v [16:00:04] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jquy4 [16:00:06] [02miraheze/mw-config] 07dmehus 03de4a02b - Add Art & Architecture category to CreateWiki (#3782) [16:00:06] ok [16:00:39] [ANNOUNCEMENT] Miraheze has just published a new blog post. Make sure to check it out! https://phabricator.miraheze.org/phame/post/view/12/2021_new_year_new_teams_new_focus_same_principles/ [16:01:04] miraheze/mw-config - Reception123 the build passed. [16:21:52] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=awfulmovieswiki --r "[[phab:T6964|Requested]]" /home/reception/awfulmovies.txt [16:21:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:45] Hello. [16:27:28] Oops, wrong place. [16:32:49] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 133.85 ms [16:33:45] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqzjK [16:33:46] [02miraheze/MirahezeMagic] 07Universal-Omega 037cf4cf6 - Remove Comments user columns from RemovePII (T6961) [16:33:48] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [16:33:49] [02MirahezeMagic] 07Universal-Omega opened pull request 03#221: Remove Comments user columns from RemovePII (T6961) - 13https://git.io/Jqzj9 [16:34:49] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:42:13] [02MirahezeMagic] 07Universal-Omega closed pull request 03#221: Remove Comments user columns from RemovePII (T6961) - 13https://git.io/Jqzj9 [16:42:15] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqg3W [16:42:16] [02miraheze/MirahezeMagic] 07Universal-Omega 03052e566 - Remove Comments user columns from RemovePII (T6961) (#221) [16:42:44] JohnLewis, not sure if you saw my message late last night, but since yesterday, the CreateWiki AI has been scoring every wiki request approved or declined as 0.15 [16:43:05] so wondering if the algorithm got broken with the most recent commits? [16:43:09] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:43:38] dmehus: I did, I think paladox made a commit - but I'm not maintaining the extensions anymore so I'm not the person to report bugs to via IRC [16:44:40] JohnLewis, yeah... I know you're not maintaining it anymore, which is too bad. I guess mainly mention bugs to paladox and Universal_Omega going forward? [16:44:54] Not me [16:45:04] paladox, ack and thanks [16:45:05] i don't maintain the extensions either. Its the mediawiki team [16:45:29] you do still do a bit with DataDump though don't you? [16:45:56] i do bits and pieces with it yeh [16:46:40] but still bugs go to the mediawiki team who can prioritise it seeing as my priority is the infra [16:47:08] okay [16:48:13] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [16:48:14] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:48:28] JohnLewis: should we remove the entries then after the error is fixed? we don't want that much bad data [16:48:56] Reception123: Sounds good for statistical purposes [16:49:20] JohnLewis: though will the script revisit them and give new ones if it sees that there's no entry, or will it not come back to them? [16:49:46] It's only ran on wiki approval [16:49:56] ah [16:53:01] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 113.91 ms [16:59:05] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.26 ms [16:59:09] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqgPZ [16:59:10] [02miraheze/MirahezeMagic] 07Universal-Omega 0320a8534 - Remove all SocialProfile tables from removePII (T6961) [16:59:12] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [16:59:14] [02MirahezeMagic] 07Universal-Omega opened pull request 03#222: Remove all SocialProfile tables from removePII (T6961) - 13https://git.io/JqgP4 [17:01:18] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 109.07 ms [17:01:19] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:01:56] Universal_Omega, Reception123: ping [17:01:57] PROBLEM - files.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.pornwiki.org could not be found [17:02:19] That's down [17:02:21] PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.753 second response time [17:02:21] 502 bad gateway [17:02:24] paladox: 502 [17:02:27] Yeah we know [17:02:29] !sre [17:02:34] fast bois [17:02:36] I half have an idea [17:02:43] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.756 second response time [17:02:44] I got paged in another channel [17:02:52] PROBLEM - cp10 Stunnel Http for mw8 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:02:53] PROBLEM - cp11 Stunnel Http for mw11 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:08] JohnLewis, SPF|Cloud: ^ [17:03:10] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:20] the heck? [17:03:22] What's going on? [17:03:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [17:03:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [17:03:38] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 3 backends are down. mw8 mw9 mw11 [17:03:46] PROBLEM - cp10 Stunnel Http for mw10 on cp10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time [17:03:48] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:53] PROBLEM - cp11 Stunnel Http for mw10 on cp11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time [17:03:55] PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: 3 backends are down. mw8 mw9 mw11 [17:03:56] I'm not sure. I checked the logs and i see nothing. [17:04:13] See -security [17:04:14] mw9 localhost curl works, so not database or php-fpm [17:04:19] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw8 mw9 mw10 [17:04:21] RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15193 bytes in 0.993 second response time [17:04:27] test3 down [17:04:43] now it's just giving a browser timeout for me [17:04:43] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15193 bytes in 1.021 second response time [17:04:48] PROBLEM - cp11 Varnish Backends on cp11 is CRITICAL: 2 backends are down. mw8 mw10 [17:04:49] We know [17:04:51] RECOVERY - cp10 Stunnel Http for mw8 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15193 bytes in 0.004 second response time [17:05:04] but a curl from cp12 to mw9 takes 11 seconds [17:05:36] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 133.62 ms [17:05:37] ooh, 11 seconds, that's quite long [17:05:44] RECOVERY - cp10 Stunnel Http for mw10 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15194 bytes in 0.004 second response time [17:05:47] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20707 bytes in 0.117 second response time [17:05:52] RECOVERY - cp11 Stunnel Http for mw10 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15194 bytes in 0.003 second response time [17:05:55] RECOVERY - cp10 Varnish Backends on cp10 is OK: All 7 backends are healthy [17:06:19] dmehus: it's known [17:06:29] I got alerted before it happened of a possible cause [17:07:21] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15208 bytes in 1.355 second response time [17:08:39] RhinosF1, oh? what's the possible cause? [17:08:42] RECOVERY - files.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - files.pornwiki.org reverse DNS resolves to cp11.miraheze.org [17:08:45] RECOVERY - cp11 Varnish Backends on cp11 is OK: All 7 backends are healthy [17:08:48] dmehus: wikimedia outage [17:08:55] Broke instant commons wikis [17:09:02] looks like site is slowly coming back [17:09:02] RECOVERY - cp11 Stunnel Http for mw11 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15194 bytes in 4.992 second response time [17:09:20] PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 346.74 ms [17:09:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:09:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:09:36] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [17:09:37] It's now fully functional. [17:09:40] back to normal here [17:10:06] InstantCommons, always fun [17:10:26] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [17:10:30] JohnLewis: yeah [17:11:07] SPF|Cloud: they still got an incident so is it worth disabling IC until they declare over [17:11:20] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 249.38 ms [17:11:36] if others agree with that, sure [17:12:01] i have the commons disabled (enabled them before), but the wiki would return 502, back to normal now. [17:12:02] neither option is one to take lightly, either complete loss of functionality or intermittent availability of all functionalities [17:12:06] [02MirahezeMagic] 07Universal-Omega closed pull request 03#222: Remove all SocialProfile tables from removePII (T6961) - 13https://git.io/JqgP4 [17:12:08] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jq2Ts [17:12:09] [02miraheze/MirahezeMagic] 07Universal-Omega 03995b7cd - Remove all SocialProfile tables from removePII (T6961) (#222) [17:12:11] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:12:12]