[00:01:02] miraheze/mw-config/paladox-patch-1/f43ccb2 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/708533879 [00:02:11] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3551 MB (14% inode=93%); [00:05:40] [02mw-config] 07paladox closed pull request 03#3175: Switch default db to db11 - 13https://git.io/JJGlG [00:05:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGlw [00:05:43] [02miraheze/mw-config] 07paladox 0338cc6ca - Switch default db to db11 (#3175) [00:05:44] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:05:46] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:06:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGlo [00:06:11] [02miraheze/mw-config] 07paladox 03959a075 - Take batch 5 out of read only [00:08:51] !log sudo -u www-data php changeDBCluster.php --wiki loginwiki --db-cluster c2 --file batch [00:08:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:09:03] !log stopped mysql on db9 [00:09:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJGl7 [00:11:15] [02miraheze/mw-config] 07paladox 035acf2f5 - Remove sitenotice [00:11:17] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:11:18] [02mw-config] 07paladox opened pull request 03#3176: Remove sitenotice - 13https://git.io/JJGl5 [00:11:27] [02mw-config] 07paladox closed pull request 03#3176: Remove sitenotice - 13https://git.io/JJGl5 [00:11:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGlF [00:11:30] [02miraheze/mw-config] 07paladox 033d9c2f0 - Remove sitenotice (#3176) [00:11:32] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:11:33] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:11:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-3173-paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGlN [00:11:55] [02miraheze/mw-config] 07paladox 03abe3e41 - Revert "Disable createwiki temporarily (#3173)" This reverts commit eb7a8253fb4519a4d87a4f1f76880c5fc1ec6860. [00:11:57] [02mw-config] 07paladox created branch 03revert-3173-paladox-patch-2 - 13https://git.io/vbvb3 [00:11:58] [02mw-config] 07paladox opened pull request 03#3177: Revert "Disable createwiki temporarily" - 13https://git.io/JJGlA [00:12:04] [02mw-config] 07paladox closed pull request 03#3177: Revert "Disable createwiki temporarily" - 13https://git.io/JJGlA [00:12:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGlx [00:12:07] [02miraheze/mw-config] 07paladox 038831318 - Revert "Disable createwiki temporarily (#3173)" (#3177) This reverts commit eb7a8253fb4519a4d87a4f1f76880c5fc1ec6860. [00:12:09] [02miraheze/mw-config] 07paladox deleted branch 03revert-3173-paladox-patch-2 [00:12:10] [02mw-config] 07paladox deleted branch 03revert-3173-paladox-patch-2 - 13https://git.io/vbvb3 [00:12:21] PROBLEM - db9 MariaDB on db9 is CRITICAL: Can't connect to MySQL server on 'db9.miraheze.org' (111) [00:12:50] miraheze/mw-config/paladox-patch-1/5acf2f5 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/708536865 [00:12:59] paladox: can you ackdb9? [00:13:09] s/ckd/ck d [00:13:09] RhinosF1 meant to say: paladox: can you ack db9? [00:13:12] I can in a moment [00:14:20] miraheze/mw-config/revert-3173-paladox-patch-2/abe3e41 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/708536968 [00:14:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JJGlj [00:14:51] [02miraheze/puppet] 07paladox 0398b9058 - Switch grafana to db11 [00:14:53] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:14:54] [02puppet] 07paladox opened pull request 03#1447: Switch grafana to db11 - 13https://git.io/JJG8e [00:19:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJG8U [00:19:35] [02miraheze/ssl] 07paladox 036a92a18 - Redirect dcmultiverse.miraheze.org to dc-multiverse.dcwikis.com [00:19:37] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [00:19:38] [02ssl] 07paladox opened pull request 03#341: Redirect dcmultiverse.miraheze.org to dc-multiverse.dcwikis.com - 13https://git.io/JJG8T [00:20:00] [02ssl] 07paladox closed pull request 03#341: Redirect dcmultiverse.miraheze.org to dc-multiverse.dcwikis.com - 13https://git.io/JJG8T [00:20:02] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG8k [00:20:03] [02miraheze/ssl] 07paladox 0311c1f9e - Redirect dcmultiverse.miraheze.org to dc-multiverse.dcwikis.com (#341) [00:20:05] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [00:20:06] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [00:22:46] RECOVERY - db9 MariaDB on db9 is OK: Uptime: 562 Threads: 10 Questions: 2942 Slow queries: 28 Opens: 1443 Flush tables: 1 Open tables: 1437 Queries per second avg: 5.234 [00:24:30] [02puppet] 07paladox closed pull request 03#1447: Switch grafana to db11 - 13https://git.io/JJG8e [00:24:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG8Y [00:24:33] [02miraheze/puppet] 07paladox 0376e8fe7 - Switch grafana to db11 (#1447) [00:24:34] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:24:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [00:38:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JJG8K [00:38:26] [02miraheze/puppet] 07paladox 032186100 - Remove db9 [00:38:27] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:38:29] [02puppet] 07paladox opened pull request 03#1448: Remove db9 - 13https://git.io/JJG86 [00:38:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JJG8i [00:38:53] [02miraheze/puppet] 07paladox 03974ff57 - Update db.pp [00:38:54] [02puppet] 07paladox synchronize pull request 03#1448: Remove db9 - 13https://git.io/JJG86 [00:39:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JJG8P [00:39:10] [02miraheze/puppet] 07paladox 03f47b476 - Update nrpe.cfg.erb [00:39:11] [02puppet] 07paladox synchronize pull request 03#1448: Remove db9 - 13https://git.io/JJG86 [00:39:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JJG8X [00:39:29] [02miraheze/puppet] 07paladox 0381c123d - Update director.pp [00:39:30] [02puppet] 07paladox synchronize pull request 03#1448: Remove db9 - 13https://git.io/JJG86 [00:39:41] [02puppet] 07paladox closed pull request 03#1448: Remove db9 - 13https://git.io/JJG86 [00:39:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JJG81 [00:39:44] [02miraheze/puppet] 07paladox 032ddf43d - Remove db9 (#1448) * Remove db9 * Update db.pp * Update nrpe.cfg.erb * Update director.pp [00:39:45] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:39:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [00:40:20] PROBLEM - db9 MariaDB on db9 is CRITICAL: Can't connect to MySQL server on 'db9.miraheze.org' (111) [00:44:46] !log shutdown cp8 [00:44:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:46:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG8F [00:46:11] [02miraheze/dns] 07paladox 03794b7d2 - Update miraheze.org [00:47:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG4e [00:47:16] [02miraheze/puppet] 07paladox 03040471c - Remove cp8 [00:47:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG4T [00:47:37] [02miraheze/mw-config] 07paladox 03dcf219b - Remove cp8 [00:49:16] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG4Y [00:49:17] [02miraheze/dns] 07paladox 03c18a245 - cp8 -> cp9 [00:51:36] PROBLEM - espiri.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:43] Do you follow the maintenance? [01:12:37] hispano76 not sure what you mean? [03:34:50] > @paladox Was just going to go in and check for wiki requests, but it's still in read only mode. Just wanted to check and make sure that CreateWiki is still supposed to be offline. [03:36:21] I reverted it in https://github.com/miraheze/mw-config/commit/8831318b016fcbbe0816e6f6fcc711ec5b4ce3db [03:36:22] [ Revert "Disable createwiki temporarily (#3173)" (#3177) · miraheze/mw-config@8831318 · GitHub ] - github.com [03:37:33] Yeah, I saw that back then, but just wondering if maybe it was still supposed to be down or not? [03:38:03] It should be back up [03:38:59] Screenshot of CreateWiki as at 8:38 PM UTC -07:00 on 15 July 2020. Let me know if it comes through to IRC [03:38:59] https://cdn.discordapp.com/attachments/435711390544560128/733165756212052098/2020-07-15_20.38.19_meta.miraheze.org_4fffc4c1456d.jpg [03:39:21] I'm hoping the IRC relay bot passes through the screenshot to IRC. [03:40:10] Did the screenshot show up on your end? No tabs, and no big blue "create wiki" button 😦 [03:41:11] you're on Discord anyway, so if not, you'll see the screenshot there [03:47:37] RECOVERY - db9 MariaDB on db9 is OK: Uptime: 443 Threads: 8 Questions: 2858 Slow queries: 9 Opens: 1406 Flush tables: 1 Open tables: 1400 Queries per second avg: 6.451 [03:50:33] @paladox Your revert commit LGTM. Not sure what the issue is. Could it db related, possibly? [03:50:40] Or maybe a chmod issue? [03:53:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGu1 [03:53:10] [02miraheze/puppet] 07paladox 03d3407d4 - mariadb: Add survey to mediawiki grants [03:53:43] @Doug Try now? [03:54:07] ok [03:54:53] @paladox nope, still just the "request" tab and it's read only [03:55:28] should I try changing skins to, say, MonoBook (although I used Vector before with no issue)? [03:56:28] Nope, not a skin issue. Switching back to Vector now. [03:57:47] @Doug try now [03:58:56] okay [03:59:14] okay the tabs and blue create wiki button are back [03:59:20] let's if we've got one I can approve [04:00:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJGuy [04:00:19] [02miraheze/mw-config] 07paladox 034524b73 - Add 'c2' to wgCreateWikiDatabaseClusters This will allow CW to create on either db11/12. [04:00:20] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [04:00:22] [02mw-config] 07paladox opened pull request 03#3178: Add 'c2' to wgCreateWikiDatabaseClusters - 13https://git.io/JJGu9 [04:00:31] [02mw-config] 07paladox closed pull request 03#3178: Add 'c2' to wgCreateWikiDatabaseClusters - 13https://git.io/JJGu9 [04:00:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGuH [04:00:34] [02miraheze/mw-config] 07paladox 0340eb3df - Add 'c2' to wgCreateWikiDatabaseClusters (#3178) This will allow CW to create on either db11/12. [04:00:36] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [04:00:37] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [04:00:46] @paladox, I'm just going to create a wiki, there's nothing I can approve...can you delete it after? [04:01:04] I mean i can set it as "deleted" [04:01:34] PROBLEM - db9 MariaDB on db9 is CRITICAL: Can't connect to MySQL server on 'db9.miraheze.org' (111) [04:01:52] miraheze/mw-config/paladox-patch-1/4524b73 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/708583326 [04:03:37] @paladox Okay, seems to be good now. By "deleted" is that a more advanced phase than "closed"? [04:03:45] cwwikitestwiki is the dbname [04:03:55] yeh, it'll be deleted at some stage, it's just hidden from the UI [04:04:00] ah, okay [04:04:11] sure, yeah, you can delete that one now [04:04:21] done, deleted [04:05:16] Sounds good. Just noticed something, when I create a wiki, I guess it doesn't link to my request, which makes sense because it wasn't requested by a user different than the creator. [04:08:45] !log MariaDB [doctorwhofanfilmsdatabasewiki]> INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); -db1 [04:08:47] *db11 [04:08:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:11:13] !log MariaDB [doctorwhofanfilmsdatabasewiki]> INSERT INTO content_models (model_id, model_name) VALUES (5, 'json'); -db11 [04:11:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:11:28] @paladox Returned an error for KHddwiki, is that the case that the dbname can't contain capital letters? [04:11:31] !log fixed content_models on doctorwhofanfilmsdatabasewiki [04:11:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:11:36] just making sure that's as designed [04:11:37] yeh [04:11:43] okay, thanks [04:12:21] hmm [04:12:25] it allows KHddwiki [04:12:45] what error did you get @Doug? [04:15:29] Oh, shoot, didn't save it. It was a red error with the usual random hash and an error that it couldn't contain all capital letters, I think? [04:16:05] oh [04:16:07] I know we're to avoid editing wiki requests, but this is such a good case where we should, so I renamed the sitename to "khdd" and it renamed the dbname behind the scenes, presumably, to khddwiki [04:16:11] seems to have created it [04:16:20] yeah that was after [04:16:21] oh [04:16:28] I edited it [04:16:51] Could you file a bug report? [04:16:59] 14:47, 15 July 2020 Xigbanort13 talk contribs requested the wiki "Kingdom Hearts [Deep Dive]" (language: en; private: 0) in #13225 (I want to make a 100% canon only wiki for the Kingdom Hearts series, that doesn't focus on gameplay mechanics at all, like the other Kingdom Hearts wikis do) <--- original request [04:17:09] sure [04:17:22] anyways i'm off to bed [04:17:24] *now [04:17:28] okay, night [05:01:59] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mrlove.wiki' expires in 15 day(s) (Sat 01 Aug 2020 04:57:44 GMT +0000). [05:02:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGzu [05:02:38] [02miraheze/ssl] 07MirahezeSSLBot 031d41c8c - Bot: Update SSL cert for mrlove.wiki [05:15:49] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Wed 14 Oct 2020 04:02:30 GMT +0000. [08:38:10] Reception123: you have go [08:38:14] [02dns] 07RhinosF1 edited pull request 03#163: Pool tools1 into bots.miraheze.wiki - 13https://git.io/JJGCl [08:39:02] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2645 MB (10% inode=93%); [08:40:59] k [08:41:33] [02dns] 07Reception123 closed pull request 03#163: Pool tools1 into bots.miraheze.wiki - 13https://git.io/JJGCl [08:41:35] [02miraheze/dns] 07Reception123 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JJGrz [08:41:36] ^ RhinosF1 [08:41:36] [02miraheze/dns] 07RhinosF1 036162c6a - [DO NOT MERGE] Pool tools1 into bots.miraheze.wiki This server will shortly be configured to serve static web [08:41:38] [02miraheze/dns] 07Reception123 03cb13580 - Merge pull request #163 from RhinosF1/patch-8 Pool tools1 into bots.miraheze.wiki [08:42:10] Reception123: ty, now we wait 10 mins [08:42:24] .in 10mins add LE cert for tools1 [08:42:24] RhinosF1: Okay, will remind at 2020-07-16 - 09:52:24BST [08:42:25] yup [08:44:05] Reception123: if you see an ssl error on bots.miraheze.wiki, it's worked! [08:46:02] heh we'll find out soon [08:52:25] RhinosF1: add LE cert for tools1 [08:54:12] Reception123: it's working [09:30:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGK8 [09:30:22] [02miraheze/services] 07MirahezeSSLBot 0377fa916 - BOT: Updating services config for wikis [11:17:56] PROBLEM - misc1 APT on misc1 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [11:59:14] [02dns] 07RhinosF1 opened pull request 03#164: [DNM] Add phabdigests tool - 13https://git.io/JJG1w [12:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGDh [12:45:14] [02miraheze/services] 07MirahezeSSLBot 03e2a2843 - BOT: Updating services config for wikis [13:09:52] !log increase cpu units to 2048 on mw 4,5,6,7 [13:09:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:13:43] !log set cpu cores to 2 for puppet2 [13:13:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:14:34] !log revert that change [13:14:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:14:53] [02dns] 07RhinosF1 edited pull request 03#164: Add phabdigests tool - 13https://git.io/JJG1w [13:14:57] paladox: ^ pls [13:15:52] [02dns] 07paladox synchronize pull request 03#164: Add phabdigests tool - 13https://git.io/JJG1w [13:16:00] PROBLEM - puppet2 Current Load on puppet2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:07] [02dns] 07paladox closed pull request 03#164: Add phabdigests tool - 13https://git.io/JJG1w [13:16:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG99 [13:16:10] [02miraheze/dns] 07RhinosF1 03cf475b1 - Add phabdigests tool (#164) * Add phabdigests tool Using phabdigests.bots.miraheze.wiki * Update miraheze.wiki Co-authored-by: paladox [13:16:48] PROBLEM - puppet2 puppetserver on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8140: Connection refused [13:16:55] ty paladox [13:17:01] and well done for last night [13:17:10] * RhinosF1 is impressed with api speed [13:17:25] !log set cpu to 2 for puppet2 [13:17:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:37] PROBLEM - puppet2 SSH on puppet2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:52] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:18:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJG9d [13:18:22] [02miraheze/puppet] 07paladox 031da7a8a - puppet2: Increase heap to 2g [13:19:43] RECOVERY - puppet2 SSH on puppet2 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [13:20:29] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 2.71, 1.10, 0.41 [13:22:26] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:03] RECOVERY - puppet2 puppetserver on puppet2 is OK: TCP OK - 0.005 second response time on 51.89.160.129 port 8140 [13:23:03] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:18] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:18] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:19] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:22] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:24] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:28] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:35] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:35] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:42] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:42] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:42] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:50] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:59] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:02] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:11] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:12] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:13] !log only changed cpu units on mw 4,5 not 6,7 [13:24:14] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:23] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:23] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:23] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:23] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:29] PROBLEM - db9 Puppet on db9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:34] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:43] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:44] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:45] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:50] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:51] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:51] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:26:11] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:26:18] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:26:19] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:26:22] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:26:23] RECOVERY - db9 Puppet on db9 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:26:44] PROBLEM - puppet2 Current Load on puppet2 is CRITICAL: CRITICAL - load average: 4.87, 3.05, 1.44 [13:26:47] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:26:49] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:26:56] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:32] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:33] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:57] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:58] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:27:58] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:25] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:46] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 2.27, 0.54, 0.18 [13:33:13] PROBLEM - puppet2 Current Load on puppet2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:34:51] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [13:34:58] PROBLEM - db9 Puppet on db9 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [13:35:03] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:35:10] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 2 minutes ago with 21 failures. Failed resources (up to 3 shown) [13:35:12] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [13:35:14] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:35:21] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.63, 0.68, 0.25 [13:35:36] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [13:35:43] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [13:36:10] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:36:20] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [13:36:39] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [13:36:48] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 4 minutes ago with 7 failures. Failed resources (up to 3 shown) [13:36:49] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:36:51] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [13:36:52] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 4 minutes ago with 20 failures. Failed resources (up to 3 shown) [13:37:09] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 4 minutes ago with 8 failures. Failed resources (up to 3 shown) [13:39:05] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:39:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:39:27] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:39:35] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:39:45] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:39:49] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:39:54] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:40:29] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:31] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:40:55] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:41:31] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:42:22] PROBLEM - puppet2 puppetserver on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8140: Connection refused [13:43:21] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:43:27] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:43:43] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:43:45] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:44:03] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:44:04] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:44:04] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:44:11] !log depool, repool and reboot mw4 & 5 [13:44:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:44:24] RECOVERY - puppet2 puppetserver on puppet2 is OK: TCP OK - 0.000 second response time on 51.89.160.129 port 8140 [13:44:33] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:44:35] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:45:58] If any Miraheze user wants a digest of Miraheze or MirahezeBots phabricator, Please inform me. [13:47:02] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:49:51] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:50:37] paladox: can you reboot the irc feed [13:50:48] 08:21:28 ⇐ +MirahezeRC quit (~MirahezeR@miraheze/bots) Excess Flood [13:51:05] done [13:51:10] !log restart ircrcbot on mon1 [13:51:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:51:19] yep [13:56:00] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 517 failures. Last run 2 minutes ago with 517 failures. Failed resources (up to 3 shown) [13:57:10] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 517 failures. Last run 3 minutes ago with 517 failures. Failed resources (up to 3 shown) [13:59:06] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/home/salt-user],File[/etc/sysctl.d] [13:59:16] paladox: ^ [13:59:31] yup [13:59:43] already aware :) [14:02:37] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:02:39] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:02:42] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:02:44] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:02:48] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:02:48] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:02:52] RECOVERY - db9 Puppet on db9 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [14:02:58] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:03:03] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:03:10] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:03:12] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:03:12] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:03:13] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:03:17] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [14:03:23] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [14:03:45] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:03:46] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:03:53] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:03:54] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:03:58] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:03:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:04:01] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:04:05] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:04:11] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:04:22] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:04:31] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:04:40] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:04:41] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:04:57] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:05:04] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:06:04] !log revoked cp8 puppet cert [14:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:59] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JJGFt [15:13:51] [02miraheze/puppet] 07paladox 03c59bc72 - Allows users from matomo-users to login [15:13:52] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [15:13:54] [02puppet] 07paladox opened pull request 03#1449: Allows users from matomo-users to login - 13https://git.io/JJGFq [15:17:29] [02puppet] 07paladox closed pull request 03#1449: Allows users from matomo-users to login - 13https://git.io/JJGFq [15:17:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGFR [15:17:32] [02miraheze/puppet] 07paladox 033600f57 - Allows users from matomo-users to login (#1449) [15:17:34] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [15:17:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [15:18:10] !log lower cpu units to 900 for puppet2 [15:18:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:08]