[00:04:42] [02MirahezeMagic] 07dmehus commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3ZOB [00:15:02] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3Z3S [00:16:08] [02CreateWiki] 07Universal-Omega commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Z3F [00:16:38] [02CreateWiki] 07Universal-Omega edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Z3F [00:17:24] [02CreateWiki] 07Universal-Omega edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Z3F [00:20:32] [02MirahezeMagic] 07dmehus commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3ZG4 [00:22:33] [02CreateWiki] 07dmehus commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZG7 [00:24:17] [02CreateWiki] 07Universal-Omega commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZZU [00:32:34] [02CreateWiki] 07dmehus commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZnL [00:33:01] [02CreateWiki] 07dmehus edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZnL [00:33:07] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3Znm [00:33:37] [02CreateWiki] 07dmehus edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZnL [00:34:17] [02CreateWiki] 07Universal-Omega commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Zn8 [00:34:37] [02CreateWiki] 07Universal-Omega edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Zn8 [00:35:23] [02MirahezeMagic] 07dmehus commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3Zng [00:36:01] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3ZnV [00:36:29] [02CreateWiki] 07dmehus commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZnK [00:36:41] [02CreateWiki] 07dmehus edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZnK [00:37:06] [02CreateWiki] 07dmehus edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZnK [00:38:55] [02MirahezeMagic] 07dmehus commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3ZnF [00:40:01] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3Znj [00:41:59] [02CreateWiki] 07DarkMatterMan4500 commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ZcB [00:42:09] [02MirahezeMagic] 07dmehus commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3Zc0 [00:43:14] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3ZcM [00:43:38] [02MirahezeMagic] 07dmehus commented on pull request 03#248: Add MirahezeMagic global interface messages for Trust and Safety - 13https://git.io/J3ZcH [01:53:22] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.84, 1.64, 1.39 [01:55:19] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.92, 1.34, 1.31 [02:05:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 5.84, 5.16 [02:07:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.34, 5.49, 5.11 [02:54:34] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 168s [02:56:34] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [03:47:04] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 3.54, 3.28, 1.55 [03:49:04] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.51, 2.21, 1.37 [04:07:06] PROBLEM - zh.internetpedia.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.internetpedia.tk All nameservers failed to answer the query. [04:14:00] RECOVERY - zh.internetpedia.tk - reverse DNS on sslhost is OK: rDNS OK - zh.internetpedia.tk reverse DNS resolves to cp11.miraheze.org [05:24:36] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 251s [05:58:44] PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 239.92 ms [06:00:04] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'datacrondatabase.com' expires in 15 day(s) (Mon 17 May 2021 05:51:22 GMT +0000). [06:00:44] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 237.43 ms [06:01:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3nl7 [06:01:12] [02miraheze/ssl] 07MirahezeSSLBot 031f116fb - Bot: Update SSL cert for datacrondatabase.com [06:41:21] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Fri 30 Jul 2021 05:01:06 GMT +0000. [06:48:36] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [07:05:07] [02CreateWiki] 07RhinosF1 commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3na6 [07:05:20] [02CreateWiki] 07RhinosF1 edited a comment on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3na6 [07:05:22] PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 238.54 ms [07:07:23] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 237.52 ms [07:22:36] [02CreateWiki] 07Universal-Omega commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3nK9 [07:47:45] [02CreateWiki] 07RhinosF1 commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3n1y [09:11:59] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [09:12:19] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [09:13:04] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [09:13:34] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [09:16:13] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (11 critical updates). [09:20:53] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 121 packages available for upgrade (11 critical updates). [09:20:55] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (11 critical updates). [09:21:15] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (11 critical updates). [09:21:20] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (11 critical updates). [09:21:23] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (11 critical updates). [09:22:20] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (11 critical updates). [09:22:47] PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (11 critical updates). [09:23:00] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (11 critical updates). [09:23:49] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (11 critical updates). [09:23:55] PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (11 critical updates). [09:24:11] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [09:25:18] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (11 critical updates). [09:26:47] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (13 critical updates). [09:26:59] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (11 critical updates). [09:27:36] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (11 critical updates). [09:33:14] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (11 critical updates). [09:33:18] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [09:34:36] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 49 packages available for upgrade (11 critical updates). [09:34:41] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (11 critical updates). [09:35:17] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (11 critical updates). [09:35:38] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (11 critical updates). [09:36:03] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (11 critical updates). [09:36:08] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (11 critical updates). [09:36:14] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (11 critical updates). [09:36:23] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (11 critical updates). [09:36:57] PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [09:38:08] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (11 critical updates). [10:05:29] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.90, 4.12, 2.13 [10:07:27] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.43, 3.08, 1.99 [12:09:24] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.88, 3.54, 2.17 [12:11:24] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 14.60, 7.68, 3.82 [12:15:24] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.76, 3.69, 3.04 [12:17:24] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.22, 2.49, 2.68 [14:11:24] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 10.63, 5.77, 2.90 [14:11:32] [02CreateWiki] 07dmehus commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Cv4 [14:15:24] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.93, 3.39, 2.60 [15:22:17] [02CreateWiki] 07Reception123 commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Cni [15:26:11] [02CreateWiki] 07RhinosF1 commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3CcP [15:30:18] [02CreateWiki] 07Universal-Omega commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3CCo [15:31:14] [02CreateWiki] 07dmehus commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3CCH [15:32:50] ^I think longer discussions should take place on Discord or IRC, rather than the pull requests [15:33:13] (I feel that "live chat" communicates better as well) [15:33:36] Yeah that's what Reception123 said and I totally agree. [15:34:10] yup, I was quite surprised to see what a huge PR that was, I've never seen that before in almost 6 years as a volunteer at Miraheze [15:35:30] Yeah.. I've never seen that ever on any PR across any organisation. It's rare thing and really should not have been there but it was a large debate then I expected. I thought it'd be non controversial... I was very wrong there. [15:36:42] [02CreateWiki] 07dmehus commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3ClZ [15:37:00] Maybe that PR could be discussed somewhere on Meta for better visibility? [15:38:02] if Wikiforum was more organized, I think it could be enabled on Meta so discussions like this could be longer and public [15:38:26] (I mean, discussions on IRC/Discord are public as well, but it's definitely easier to find something on Meta than here) [15:40:22] [02CreateWiki] 07Universal-Omega commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3C8o [15:40:41] [02CreateWiki] 07RhinosF1 commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3C8M [15:41:55] Almost 50 comments on that PR.... [15:42:57] > Maybe that PR could be discussed somewhere on Meta for better visibility? [15:42:57] Meta wouldn't be the best solution for something that is very wiki creator/steward-specific in terms of their workflow. Best would've been to create a Phabricator task, and have the discussion there before coding it, in my opinion [15:43:47] dmehus: well I didn't think this would be so controversial or I would've done some prior discussion if I thought there would be opposition. [15:44:45] [02CreateWiki] 07dmehus commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3C4A [15:46:11] Universal_Omega, yeah, exactly, no worries :) [15:46:55] [02CreateWiki] 07Universal-Omega commented on pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3CBa [15:47:32] [02CreateWiki] 07Universal-Omega reviewed pull request 03#214 commit - 13https://git.io/J3CBM [15:49:48] dmehus: thanks! But now can we continue this discussion here (50 comments on that PR is to much... almost double what I've seen before has the most, on the mw-config CannedResponses one) a full discussion should've been here not there as Reception mentioned. [15:50:48] yeah, if we do plan on continuing the best would be here :) [15:53:18] @Universal_Omega, yeah if you're going to try and incorporate RhinosF1's idea into the warning / notification message and a link to a prefilled SN request and also create an invalid status type/log action, I'm fine with that, no need to reply on the PR further [15:54:13] dmehus: great, but that's also provided that what RhinosF1 meant and I didn't misunderstand. [15:55:19] Universal_Omega, yeah, feel free to discuss your progress with me via DM on Discord or IRC as well, so we can work through any development roadblocks as may be encountered [16:00:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [16:01:13] miraheze/CreateWiki - Universal-Omega the build passed. [16:05:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.34, 6.69, 5.81 [16:07:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.48, 6.12, 5.71 [16:26:03] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [16:27:12] miraheze/CreateWiki - Universal-Omega the build passed. [16:27:19] paladox, do our GDNSD zone files not use "IN" before the specific DNS operator (i.e., "IN TXT"), so we'd just use "TXT," or do we still need the "IN TXT"? [16:27:49] I'm not sure i understand what your asking [16:28:12] like for adding a TXT record to a user's zone file [16:28:35] oh, you use TXT [16:28:35] normally the syntax per TXT record on enwiki is to use IN TXT, but I don't see any of our zone files using the word "IN" [16:28:41] okay, thanks :) [16:28:59] must be a BIND/GDNSD syntax difference [16:37:18] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [16:38:13] miraheze/CreateWiki - Universal-Omega the build passed. [16:47:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [16:47:28] [02dns] 07dmehus opened pull request 03#203: Add Google site verification to trollpasta.com zone - 13https://git.io/J3CPd [16:48:07] miraheze/CreateWiki - Universal-Omega the build passed. [16:48:30] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [16:48:35] paladox, can you look at #203 ^ when you get a chance, and let me know if any changes are needed? [16:49:28] miraheze/CreateWiki - Universal-Omega the build passed. [16:52:25] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [16:53:29] miraheze/CreateWiki - Universal-Omega the build passed. [16:53:37] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [16:54:34] miraheze/CreateWiki - Universal-Omega the build passed. [17:00:51] [02dns] 07Reception123 reviewed pull request 03#203 commit - 13https://git.io/J3CM2 [17:01:18] [02dns] 07Reception123 reviewed pull request 03#203 commit - 13https://git.io/J3CM2 [17:02:51] * dmehus is looking and fixing the above [17:04:01] [02dns] 07dmehus synchronize pull request 03#203: Add Google site verification to trollpasta.com zone - 13https://git.io/J3CPd [17:04:29] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3CDq [17:04:30] [02miraheze/dns] 07dmehus 03aa52f1d - Add Google site verification to trollpasta.com zone (#203) [17:04:32] [02dns] 07Reception123 closed pull request 03#203: Add Google site verification to trollpasta.com zone - 13https://git.io/J3CPd [17:06:41] paladox: disk space critical on puppet3 [17:07:58] (cc Reception123, if around) [17:08:17] in the meantime I'm looking at a huge list of security patches... [17:10:40] !log install security patches on services3 [17:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:34] RECOVERY - services3 APT on services3 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [17:13:16] It's always nice when icinga is happy after the packages updates are updated :) [17:14:35] yep [17:15:01] even better when the organisation has a team that takes care of 'patching as soon as possible' [17:18:30] !log install security patches on ns1 [17:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:58] RECOVERY - ns1 APT on ns1 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [17:23:43] SPF|Cloud, yep :) [17:24:51] !log puppet3: remove puppetserver-access.log.json.1 (13 GB) to free up disk space [17:24:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:26:33] !log restart puppetserver to remove in-memory reference to aforementioned log file [17:26:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:26:50] RECOVERY - puppet3 Disk Space on puppet3 is OK: DISK OK - free space: / 10640 MB (44% inode=92%); [17:35:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 6.15, 5.23 [17:37:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.21, 5.53, 5.11 [17:43:09] !log installing security patches on all hosts [17:43:20] RECOVERY - db13 APT on db13 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [17:43:23] RECOVERY - db11 APT on db11 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [17:43:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:46] RECOVERY - cp3 APT on cp3 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [17:44:00] RECOVERY - cp10 APT on cp10 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:44:08] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [17:44:16] RECOVERY - mem1 APT on mem1 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [17:44:19] RECOVERY - cp12 APT on cp12 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [17:44:21] RECOVERY - services4 APT on services4 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [17:44:21] RECOVERY - db12 APT on db12 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [17:44:23] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:44:24] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [17:44:40] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:44:46] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [17:44:47] RECOVERY - test4 APT on test4 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [17:44:51] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 110 packages available for upgrade (0 critical updates). [17:44:52] RECOVERY - phab2 APT on phab2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:44:52] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [17:44:53] RECOVERY - test3 APT on test3 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [17:44:55] RECOVERY - mon2 APT on mon2 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [17:44:57] RECOVERY - cp11 APT on cp11 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [17:44:58] RECOVERY - mem2 APT on mem2 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [17:45:16] RECOVERY - ns2 APT on ns2 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:45:19] RECOVERY - mw8 APT on mw8 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:45:35] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [17:45:38] RECOVERY - mw10 APT on mw10 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:45:40] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [17:45:42] RECOVERY - mail2 APT on mail2 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [17:46:03] RECOVERY - mw9 APT on mw9 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:46:07] RECOVERY - mw11 APT on mw11 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:46:14] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [17:47:02] * Reception123 is around now [17:47:15] I did notice that one or two days ago [17:47:25] I think a lot of space it taken up by /var/log [17:47:45] Ah you already removed a log [17:48:21] SPF|Cloud: thanks [17:48:49] SPF|Cloud: do you think the files were not rotating because i need to add reloading/restarting puppetserver to https://github.com/miraheze/puppet/blob/master/modules/puppetserver/files/puppetserver.logrotate.conf? [17:48:50] [ puppet/puppetserver.logrotate.conf at master · miraheze/puppet · GitHub ] - github.com [17:49:21] https://phabricator.miraheze.org/T7224 [17:49:21] [ ⚓ T7224 Uncompressed puppetserver json logs fill up disk ] - phabricator.miraheze.org [17:49:25] uhm [17:49:45] I think that's one cause yes, the other cause being compression [17:50:22] the huge log file as .json.1, but files are only compressed as of .2 [17:51:44] remove 'delaycompress' and reload/restart puppetserver via the postrotate scripts [17:52:12] SPF|Cloud: https://github.com/miraheze/puppet/blob/master/modules/puppetserver/files/puppetserver.logrotate.conf#L1 [17:52:12] [ puppet/puppetserver.logrotate.conf at master · miraheze/puppet · GitHub ] - github.com [17:52:21] we do have a rotating policy for json files [17:52:51] at the moment of task creation, I didn't know that [17:53:14] ah [17:53:25] but please see my comments from 18:49 - 18:51 [17:53:50] with a few tweaks the logrotated configuration can be fixed... [17:54:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/J3CAA [17:54:37] [02miraheze/puppet] 07paladox 03baf71a4 - puppetserver: Rotate log daily not weekly [17:54:39] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [17:54:40] [02puppet] 07paladox opened pull request 03#1749: puppetserver: Rotate log daily not weekly - 13https://git.io/J3CAp [17:54:51] SPF|Cloud: ^ [17:55:36] read my comments [17:59:04] !log bacula2: remove swap (before resizing partition) [17:59:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/J3Cpc [17:59:43] [02miraheze/puppet] 07paladox 03b6f42b0 - Update puppetserver.logrotate.conf [17:59:44] [02puppet] 07paladox synchronize pull request 03#1749: puppetserver: Rotate log daily not weekly - 13https://git.io/J3CAp [18:00:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/J3Cpg [18:00:11] [02miraheze/puppet] 07paladox 03019b3b5 - Update puppetdb.logrotate.conf [18:00:13] [02puppet] 07paladox synchronize pull request 03#1749: puppetserver: Rotate log daily not weekly - 13https://git.io/J3CAp [18:00:31] [02puppet] 07paladox edited pull request 03#1749: puppetserver: Fix rotated logging config for puppetserver and puppetdb - 13https://git.io/J3CAp [18:00:41] SPF|Cloud: this ^? [18:01:42] is the killall -HUP puppetserver necessary? [18:02:17] was following the config we have for gluster [18:02:19] Universal_Omega: so, they'd press submit, they'd get a big box at the top of the form with a message in saying why and how to get help (simple in ext, expand in magic), CW would log the request and auto mark it invalid. Leave a comment saying why but don't trigger echo as that's all done in background. Requestor can simply update the form to a new one or click a button in the error box to get help. [18:02:26] dmehus: ^ [18:02:30] there's already configuration to sighup the puppetserver PID [18:02:36] But no, doesn't look like it's necessary [18:02:41] this sounds like doing it twice [18:03:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/J3ChG [18:03:12] [02miraheze/puppet] 07paladox 03f4c847e - Update puppetdb.logrotate.conf [18:03:13] [02puppet] 07paladox synchronize pull request 03#1749: puppetserver: Fix rotated logging config for puppetserver and puppetdb - 13https://git.io/J3CAp [18:03:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/J3Chl [18:03:24] [02miraheze/puppet] 07paladox 0348f0bf3 - Update puppetserver.logrotate.conf [18:03:24] removed [18:03:26] [02puppet] 07paladox synchronize pull request 03#1749: puppetserver: Fix rotated logging config for puppetserver and puppetdb - 13https://git.io/J3CAp [18:03:39] Would be a bonus to do differentiate between closed/soft-delete/inactive but if not I'm not concerned because a steward will tell them [18:04:05] RhinosF1, yeah I think that's the general idea of the modified solution [18:04:39] > Would be a bonus to do differentiate between closed/soft-delete/inactive but if not I'm not concerned because a steward will tell them [18:04:39] Yeah...I wouldn't mind that new invalid status and logtype as well [18:05:02] dmehus: glad we agree. Balances logging for stewards + fastest response times with easy ability if they simply choose a new one. [18:05:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.44, 6.10, 5.09 [18:05:15] We can have a fancy preload template [18:05:40] RhinosF1, yeah, exactly. and for those that don't bother with the template, we'll catch them patrolling the logs [18:05:55] SPF|Cloud: gonna merge as it all looks good to me. [18:06:12] sure [18:06:29] dmehus: exactly and yes it would be a new invalid status rather than declinded. [18:06:29] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/J3Cjt [18:06:30] [02miraheze/mw-config] 07Reception123 031e0be61 - wgCosmosEnabledRailModules setting for malwiki T7219 [18:06:32] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [18:06:33] [02puppet] 07paladox closed pull request 03#1749: puppetserver: Fix rotated logging config for puppetserver and puppetdb - 13https://git.io/J3CAp [18:06:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J3Cjq [18:06:36] [02miraheze/puppet] 07paladox 038fdd5bd - puppetserver: Fix rotated logging config for puppetserver and puppetdb (#1749) [18:06:38] [02mw-config] 07Reception123 opened pull request 03#3868: wgCosmosEnabledRailModules setting for malwiki T7219 - 13https://git.io/J3Cjm [18:06:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:06:40] I don't think I spelt that right [18:06:41] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:06:51] nice [18:07:04] SPF|Cloud: did you see my ping last night [18:07:05] !log resized / partition on bacula2 to reflect instance resize [18:07:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.90, 5.71, 5.05 [18:07:18] RhinosF1: yes, but you were gone [18:07:36] miraheze/mw-config - Reception123 the build passed. [18:07:52] RhinosF1, yeah minor typo, nbd [18:08:03] (assuming "nbd" is a common abbreviation) [18:08:06] RECOVERY - bacula2 Disk Space on bacula2 is OK: DISK OK - free space: / 341352 MB (26% inode=99%); [18:08:20] [02mw-config] 07Reception123 closed pull request 03#3868: wgCosmosEnabledRailModules setting for malwiki T7219 - 13https://git.io/J3Cjm [18:08:21] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3CjV [18:08:23] [02miraheze/mw-config] 07Reception123 0351517c1 - wgCosmosEnabledRailModules setting for malwiki T7219 (#3868) [18:08:24] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [18:08:26] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [18:08:27] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [18:08:32] SPF|Cloud: just wanted to tell you that I got spam research based on me commiting to the MediaWiki repo. It violates GitHub rules and based on my chat with Owen probably GDPR. I reported it to GitHub and the Uni. [18:08:51] aha.. [18:08:52] RhinosF1: that's exactly what I've been doing in PR now. Thanks! [18:08:59] Universal_Omega: perfect [18:09:05] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 7.15, 5.57, 2.73 [18:09:08] Not sure why it took 40 comments to get there [18:09:20] miraheze/mw-config - Reception123 the build passed. [18:09:22] dmehus: I'm rubbish with abbreviations [18:10:05] RhinosF1: 50 comments... that's way to much... [18:10:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.71, 6.78, 5.67 [18:10:13] Universal_Omega: ye [18:10:25] That's crazy [18:10:36] And I'm saying that, every day is crazy. [18:11:05] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.46, 3.88, 2.45 [18:11:07] RhinosF1, nbd = no big deal, I might've made it up [18:11:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.69, 8.80, 6.41 [18:12:05] > SPF|Cloud: just wanted to tell you that I got spam research based on me commiting to the MediaWiki repo. It violates GitHub rules and based on my chat with Owen probably GDPR. I reported it to GitHub and the Uni. [18:12:05] RhinosF1, why don't you guys just do what I do and disable public identification of your e-mails in co-authorship scenarios to commits? Mine show up as my GH username and like noreply@something.github.com [18:13:04] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.56, 2.70, 2.19 [18:13:49] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3We1 [18:13:51] [02miraheze/puppet] 07Southparkfan 0346250b1 - Install needrestart globally [18:14:40] dmehus: because it's twice been an issue and last time the person got warned over it. [18:15:27] RhinosF1, but how do you know which user is harvesting e-mail addresses? I'm confused [18:16:14] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [18:16:43] dmehus: because it's come from a university. [18:17:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [18:17:10] miraheze/CreateWiki - Universal-Omega the build passed. [18:17:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.21, 7.40, 6.59 [18:17:15] It's just not compliant with both GitHub policy not likely data protection law [18:17:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.89, 19.29, 16.20 [18:18:05] miraheze/CreateWiki - Universal-Omega the build passed. [18:18:20] !log restart various services on phab2 (needrestart test) [18:18:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:18:37] RhinosF1, oh, hrm, I'm just confused how we even knew they're scraping e-mail addresses and which GitHub user it was [18:18:51] dmehus: because I got an email [18:18:58] I'm not sure what's confusing [18:19:11] right, but how'd you know which GitHub user it was? [18:19:28] did they mention their GH username in the spam email? [18:19:28] Because they linked to their profile [18:19:33] ah [18:19:34] yeah [18:19:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.10, 18.28, 16.21 [18:19:39] well that was dumb of them [18:19:47] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [18:19:48] But even if they didn't have an account it would still be against GitHub policy and data laws [18:19:54] but yes likely also violates UK anti-spam laws too [18:20:02] It's dumb of their ethics committee [18:20:40] miraheze/CreateWiki - Universal-Omega the build passed. [18:21:09] RhinosF1 yeah [18:21:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.25, 6.47, 6.39 [18:21:47] Someone at the Uni must have approved it [18:21:54] Despite the rules being very clear [18:22:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 7.98, 7.24 [18:25:05] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Don't allow wiki requests to be submitted if database exists - 13https://git.io/J3Gub [18:26:00] miraheze/CreateWiki - Universal-Omega the build passed. [18:26:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.82, 6.22, 6.69 [18:26:25] [02CreateWiki] 07Universal-Omega edited pull request 03#214: Create new invalid status for databases that already exist - 13https://git.io/J3Gub [18:27:54] > Someone at the Uni must have approved it [18:27:54] They probably don't approve every marketing message. It's incumbent upon you to report it to them as abuse of your services, I'd say [18:29:12] @RhinosF1 and dmehus: I think the PR is finished now, but I have no guarantee with 100% certainty it'll work and I didn't mess up somewhere. [18:36:09] Universal_Omega, LGTM from what I've seen so far. Too bad we can't really test CreateWiki on test3wiki [18:36:28] !log restarted various services (ntp/ssh/syslog-ng/unattended-upgrade/graylog-server & elasticsearch/some systemd related services - skipping systemd-logind, dbus, mariadb and most proxmox related services) on hosts [18:36:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:38:10] dmehus: it can be tested if proxied to test3 from metawiki I think. But maybe not. Not certain. [18:39:20] oh [18:50:32] dmehus: it's research. Uni research gets checked first. [18:52:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.44, 6.76, 6.01 [18:54:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.56, 7.43, 6.34 [18:56:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.02, 7.19, 6.38 [19:00:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.39, 6.79, 6.08 [19:02:47] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.66, 6.70, 6.47 [19:06:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.62, 6.85, 6.04 [19:06:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.22, 6.78, 6.37 [19:08:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.33, 6.50, 6.00 [19:35:16] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.02, 7.38, 6.18 [19:36:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.76, 8.21, 6.73 [19:36:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.72, 6.50, 5.93 [19:37:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.64, 6.70, 6.08 [19:38:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.77, 7.74, 6.73 [19:39:19] PROBLEM - private.revi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for private.revi.wiki could not be found [19:40:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.90, 6.67, 6.47 [19:40:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.89, 6.53, 6.10 [19:46:05] RECOVERY - private.revi.wiki - reverse DNS on sslhost is OK: rDNS OK - private.revi.wiki reverse DNS resolves to cp11.miraheze.org [20:48:36] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.40, 3.93, 2.70 [20:50:31] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.47, 3.00, 2.50 [21:01:07] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.85, 1.39, 1.14 [21:03:01] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 4.81, 2.64, 1.62 [21:06:51] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.30, 1.71, 1.45 [21:06:55] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.78, 6.83, 6.03 [21:08:47] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.84, 1.41, 1.37 [21:08:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.92, 6.50, 6.01 [21:21:15] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Create new invalid status for databases that already exist - 13https://git.io/J3Gub [21:22:17] miraheze/CreateWiki - Universal-Omega the build passed. [21:35:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 7.05, 6.17 [21:37:45] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.28, 6.40, 6.04 [21:48:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.75, 7.12, 6.52 [21:54:53] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Create new invalid status for databases that already exist - 13https://git.io/J3Gub [21:54:54] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Create new invalid status for databases that already exist - 13https://git.io/J3Gub [21:55:51] miraheze/CreateWiki - Universal-Omega the build passed. [21:55:52] miraheze/CreateWiki - Universal-Omega the build passed. [21:56:40] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 198s [21:56:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.78, 6.24, 6.39 [21:58:38] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 205s [21:59:27] [02CreateWiki] 07JohnFLewis commented on pull request 03#214: Create new invalid status for databases that already exist - 13https://git.io/J3WPk [22:08:05] Now it looks like MirahezeLogbot (21 days), Majavah (19 days), and me (17 days) have the best uptime in this channel [22:15:59] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Create new invalid status for databases that already exist - 13https://git.io/J3Gub [22:16:01] [02CreateWiki] 07Universal-Omega synchronize pull request 03#214: Create new invalid status for databases that already exist - 13https://git.io/J3Gub [22:16:58] miraheze/CreateWiki - Universal-Omega the build passed. [22:17:03]