[00:00:17] RECOVERY - services1 APT on services1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:00:31] [02ManageWiki] 07paladox synchronize pull request 03#176: Add a ChangeDbCluster script - 13https://git.io/JJs8f [00:00:33] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJs8t [00:00:34] [02miraheze/ManageWiki] 07paladox 03bd6ab0c - Update changeDbCluster.php [00:00:53] !log install security updates for chromium chromium-common chromium-sandbox on test1, restarted proton service [00:01:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:13] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3829 MB (19% inode=93%); [00:02:27] [02ManageWiki] 07paladox synchronize pull request 03#176: Add a ChangeDbCluster script - 13https://git.io/JJs8f [00:02:29] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJs8m [00:02:30] [02miraheze/ManageWiki] 07paladox 03c1ea515 - Update changeDbCluster.php [00:05:14] time to sleep! [00:42:03] [02ManageWiki] 07JohnFLewis commented on pull request 03#176: Add a ChangeDbCluster script - 13https://git.io/JJs48 [00:45:08] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/JJs44 [00:45:09] [02miraheze/CreateWiki] 07paladox 032b78087 - Add a ChangeDbCluster script Allows you to update multiple wikis and set their db cluster to a different value. [00:45:11] [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL [00:45:12] [02CreateWiki] 07paladox opened pull request 03#154: Add a ChangeDbCluster script - 13https://git.io/JJs4B [00:45:21] [02ManageWiki] 07paladox commented on pull request 03#176: Add a ChangeDbCluster script - 13https://git.io/JJs4R [00:45:23] [02ManageWiki] 07paladox closed pull request 03#176: Add a ChangeDbCluster script - 13https://git.io/JJs8f [00:45:24] [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns [00:45:26] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1 [01:01:00] [02CreateWiki] 07paladox synchronize pull request 03#154: Add a ChangeDbCluster script - 13https://git.io/JJs4B [01:01:01] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-1/±0] 13https://git.io/JJs4d [01:01:03] [02miraheze/CreateWiki] 07paladox 031f79443 - Update and rename changeDbCluster.php to changeDBCluster.php [01:01:06] [02CreateWiki] 07paladox edited pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [05:00:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsad [05:00:21] [02miraheze/services] 07MirahezeSSLBot 037ed12be - BOT: Updating services config for wikis [05:05:37] "Error loading data from server: apierror-visualeditor-docserver-http: HTTP 404. Would you like to retry?Error loading data from server: apierror-visualeditor-docserver-http: HTTP 404. Would you like to retry?" Hi, I am getting this error when I try to use VisualEditor in my wiki, what do i do? [05:15:14] Ah nevermind, worked after a bit [07:47:52] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://git.io/JJsiO [07:47:53] [02miraheze/ssl] 07Reception123 039379314 - add dc-multiverse.dcwikis.com cert [07:47:55] [02ssl] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vxP9L [07:47:56] [02ssl] 07Reception123 opened pull request 03#339: add dc-multiverse.dcwikis.com cert - 13https://git.io/JJsi3 [07:48:36] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJsin [07:48:37] [02miraheze/ssl] 07Reception123 03f27253f - add dc-multiverse.dcwikis.com cert [07:48:39] [02ssl] 07Reception123 synchronize pull request 03#339: add dc-multiverse.dcwikis.com cert - 13https://git.io/JJsi3 [07:48:48] [02ssl] 07Reception123 closed pull request 03#339: add dc-multiverse.dcwikis.com cert - 13https://git.io/JJsi3 [07:48:50] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJsil [07:48:51] [02miraheze/ssl] 07Reception123 0341e7247 - add dc-multiverse.dcwikis.com cert (#339) * add dc-multiverse.dcwikis.com cert * add dc-multiverse.dcwikis.com cert [07:48:52] [02miraheze/ssl] 07Reception123 deleted branch 03Reception123-patch-1 [07:48:54] [02ssl] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vxP9L [07:55:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsig [07:55:16] [02miraheze/services] 07MirahezeSSLBot 0318c2b81 - BOT: Updating services config for wikis [12:39:50] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 136s [12:41:36] paladox, SPF|Cloud: ^ [12:52:36] Naleksuh_: approved your bot, shut StreamBot up (we have #miraheze-feed-test. Please note that rights are auto-removed after 3 months inactivity. [12:57:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.30, 23.44, 18.58 [12:58:45] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:01:40] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.92, 19.38, 18.17 [13:07:40] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 275s [13:16:59] PROBLEM - wiki.arcolinuxclub.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.arcolinuxclub.org' expires in 15 day(s) (Fri 31 Jul 2020 13:09:42 GMT +0000). [13:20:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsdZ [13:20:41] [02miraheze/ssl] 07MirahezeSSLBot 03890904b - Bot: Update SSL cert for wiki.arcolinuxclub.org [13:25:17] seconds behind master seems clearing now [13:25:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.86, 25.08, 20.39 [13:25:35] ack SPF|Cloud [13:25:52] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:26:04] (for the record, I did nothing) [13:26:42] ok [13:27:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.20, 19.23, 18.83 [13:37:40] RECOVERY - wiki.arcolinuxclub.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.arcolinuxclub.org' will expire on Tue 13 Oct 2020 12:20:29 GMT +0000. [13:38:55] !log disable puppet on mon1 to test a new check [13:38:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:41:44] PROBLEM - mon1 Puppet on mon1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [13:42:12] > [miraheze/ssl] MirahezeSSLBot 890904b - Bot: Update SSL cert for wiki.arcolinuxclub.org Cool. We've got a few open-source software wikis we host. 🙂 [13:42:51] Is Not-3033 @Hispano76 on Discord? [13:43:39] PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: Icinga configuration contains errors [13:43:47] SPF|Cloud: ^ [13:44:00] don't have to ping me for that [13:44:07] but I'll ack it [13:44:14] @Doug: hispano76 is @Hispano76 [13:47:51] yeah, I realized that after, sorry, guess my deleted posts on Discord aren't deleted by the IRC relay bot...dang, no second chances on IRC 😛 [13:48:51] Not-3033 must just be one of the bots. [13:50:59] They're not deleted [13:51:09] Not-3033 is an IRC Bot [13:51:30] The only bot that's also on discord is MH-Discord [13:57:06] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [13:58:44] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsFp [13:58:45] [02miraheze/puppet] 07Southparkfan 035cc6209 - Add DNS check for custom domains [13:59:10] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:59:59] !log enable puppet on mon1 [14:00:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:00:17] 269 checks pending, right :P [14:04:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJsbG [14:04:26] [02miraheze/ssl] 07paladox 03513b0b9 - Remove wiki.nvda-nl.org Not pointing at us. [14:04:27] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [14:04:29] [02ssl] 07paladox opened pull request 03#340: Remove wiki.nvda-nl.org - 13https://git.io/JJsbZ [14:05:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JJsbn [14:05:07] [02miraheze/ssl] 07paladox 03eecc4a3 - Delete wiki.nvda-nl.org.crt [14:05:08] [02ssl] 07paladox synchronize pull request 03#340: Remove wiki.nvda-nl.org - 13https://git.io/JJsbZ [14:05:14] [02ssl] 07paladox closed pull request 03#340: Remove wiki.nvda-nl.org - 13https://git.io/JJsbZ [14:05:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JJsbW [14:05:17] [02miraheze/ssl] 07paladox 0357bacc9 - Remove wiki.nvda-nl.org (#340) * Remove wiki.nvda-nl.org Not pointing at us. * Delete wiki.nvda-nl.org.crt [14:05:19] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [14:05:20] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [14:25:48] PROBLEM - mon1 NTP time on mon1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:26:08] ^ that's a new check, but it's failing due to an unknown reaosn [14:27:12] PROBLEM - db7 NTP time on db7 is CRITICAL: NTP CRITICAL: No response from NTP server [14:29:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JJsNz [14:29:14] [02miraheze/puppet] 07paladox 031bfb18f - openldap: Monitor ldap [14:29:15] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS [14:29:17] [02puppet] 07paladox opened pull request 03#1445: openldap: Monitor ldap - 13https://git.io/JJsNg [14:29:30] [02puppet] 07paladox closed pull request 03#1445: openldap: Monitor ldap - 13https://git.io/JJsNg [14:29:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsN2 [14:29:33] [02miraheze/puppet] 07paladox 035733669 - openldap: Monitor ldap (#1445) [14:29:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [14:29:36] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://git.io/vbiAS [14:31:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsN1 [14:31:44] [02miraheze/puppet] 07paladox 03f57cc8a - mon1: Increase MaxConcurrentChecks to 6 [14:33:47] PROBLEM - test2 NTP time on test2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:33:50] PROBLEM - cp8 NTP time on cp8 is CRITICAL: NTP CRITICAL: No response from NTP server [14:33:52] PROBLEM - cloud1 NTP time on cloud1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:33:56] PROBLEM - db13 NTP time on db13 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:09] PROBLEM - db9 NTP time on db9 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:28] PROBLEM - cp9 NTP time on cp9 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:29] PROBLEM - gluster2 NTP time on gluster2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:31] PROBLEM - misc1 NTP time on misc1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:38] PROBLEM - cp6 NTP time on cp6 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:40] PROBLEM - mw5 NTP time on mw5 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:47] PROBLEM - jobrunner1 NTP time on jobrunner1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:47] PROBLEM - rdb2 NTP time on rdb2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:48] PROBLEM - cloud3 NTP time on cloud3 is CRITICAL: NTP CRITICAL: No response from NTP server [14:34:49] RECOVERY - mon1 NTP time on mon1 is OK: NTP OK: Offset -3.772974014e-05 secs [14:34:56] eh? [14:35:02] PROBLEM - services2 NTP time on services2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:04] PROBLEM - ldap1 NTP time on ldap1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:10] PROBLEM - ns2 NTP time on ns2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:15] PROBLEM - phab1 NTP time on phab1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:18] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:19] PROBLEM - mw7 NTP time on mw7 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:24] PROBLEM - mail1 NTP time on mail1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:26] PROBLEM - mw6 NTP time on mw6 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:31] PROBLEM - cloud2 NTP time on cloud2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:35] SPF|Cloud you added the check to base [14:35:36] PROBLEM - bacula2 NTP time on bacula2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:41] PROBLEM - rdb1 NTP time on rdb1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:47] PROBLEM - mw4 NTP time on mw4 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:49] PROBLEM - puppet2 NTP time on puppet2 is CRITICAL: NTP CRITICAL: No response from NTP server [14:35:52] I am aware, but the check fails for an unknown reason [14:36:01] PROBLEM - cp3 NTP time on cp3 is CRITICAL: NTP CRITICAL: No response from NTP server [14:36:01] oh [14:36:07] PROBLEM - services1 NTP time on services1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:36:08] PROBLEM - db11 NTP time on db11 is CRITICAL: NTP CRITICAL: No response from NTP server [14:36:11] PROBLEM - gluster1 NTP time on gluster1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:36:13] PROBLEM - cp7 NTP time on cp7 is CRITICAL: NTP CRITICAL: No response from NTP server [14:36:21] PROBLEM - db12 NTP time on db12 is CRITICAL: NTP CRITICAL: No response from NTP server [14:36:39] ah [14:37:06] if I add a check using monitoring::services {}, do I really need to put it in nrpe.cfg.erb? [14:37:54] Yes if you’re using nrpe to Conduct the check [14:38:06] SPF|Cloud fixed. [14:38:09] https://www.irccloud.com/pastebin/W5qoeQh4/ [14:38:10] [ Snippet | IRCCloud ] - www.irccloud.com [14:38:23] ntp_time is an ObjectCommand defined in icinga by default, btw [14:38:51] RECOVERY - db7 NTP time on db7 is OK: NTP OK: Offset -0.0002461969852 secs [14:39:07] SPF|Cloud ^ [14:39:09] however, unless I add ntp_address => 'time.cloudflare.com', Icinga seems to ignore the '-H time.cloudflare.com' [14:39:26] RECOVERY - mw4 NTP time on mw4 is OK: NTP OK: Offset -0.003310233355 secs [14:40:10] how did you fix it? [14:41:54] SPF|Cloud changed the command to "nrpe" [14:41:55] removing ntp_address => 'time.cloudflare.com', to see what happens(?) [14:41:57] RECOVERY - mw5 NTP time on mw5 is OK: NTP OK: Offset -0.0008963346481 secs [14:41:59] RECOVERY - db9 NTP time on db9 is OK: NTP OK: Offset -0.001548022032 secs [14:42:02] RECOVERY - cp6 NTP time on cp6 is OK: NTP OK: Offset 5.078315735e-05 secs [14:42:03] RECOVERY - cloud2 NTP time on cloud2 is OK: NTP OK: Offset -0.001282155514 secs [14:42:05] RECOVERY - gluster1 NTP time on gluster1 is OK: NTP OK: Offset -0.001971483231 secs [14:42:08] RECOVERY - ldap1 NTP time on ldap1 is OK: NTP OK: Offset -0.0006856620312 secs [14:42:08] RECOVERY - phab1 NTP time on phab1 is OK: NTP OK: Offset -0.001157224178 secs [14:42:10] RECOVERY - db13 NTP time on db13 is OK: NTP OK: Offset -0.003688812256 secs [14:42:12] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.001731067896 secs [14:42:13] RECOVERY - db12 NTP time on db12 is OK: NTP OK: Offset -0.000652641058 secs [14:42:21] RECOVERY - db11 NTP time on db11 is OK: NTP OK: Offset -0.001980900764 secs [14:42:23] RECOVERY - services1 NTP time on services1 is OK: NTP OK: Offset 6.738305092e-05 secs [14:42:29] RECOVERY - bacula2 NTP time on bacula2 is OK: NTP OK: Offset -0.02058562636 secs [14:42:31] RECOVERY - mail1 NTP time on mail1 is OK: NTP OK: Offset 0.0003772079945 secs [14:42:47] RECOVERY - gluster2 NTP time on gluster2 is OK: NTP OK: Offset 4.425644875e-05 secs [14:42:48] RECOVERY - misc1 NTP time on misc1 is OK: NTP OK: Offset -0.0007551312447 secs [14:42:59] RECOVERY - rdb2 NTP time on rdb2 is OK: NTP OK: Offset -0.0009284317493 secs [14:43:05] RECOVERY - cp7 NTP time on cp7 is OK: NTP OK: Offset 0.004067718983 secs [14:43:05] RECOVERY - cp9 NTP time on cp9 is OK: NTP OK: Offset -0.002217620611 secs [14:43:23] RECOVERY - cloud3 NTP time on cloud3 is OK: NTP OK: Offset -0.001489460468 secs [14:43:23] RECOVERY - services2 NTP time on services2 is OK: NTP OK: Offset -0.0004368722439 secs [14:43:31] RECOVERY - rdb1 NTP time on rdb1 is OK: NTP OK: Offset -0.0001754462719 secs [14:43:33] RECOVERY - cloud1 NTP time on cloud1 is OK: NTP OK: Offset -0.00126311183 secs [14:43:33] RECOVERY - puppet2 NTP time on puppet2 is OK: NTP OK: Offset -0.001221656799 secs [14:43:38] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0002754032612 secs [14:43:52] RECOVERY - test2 NTP time on test2 is OK: NTP OK: Offset -0.002479046583 secs [14:45:37] SPF|Cloud no, you do not need to put it in nrpe.cfg if using monitoring::services [14:45:42] it depends on what you do [14:45:57] if it's check_command => 'nrpe', [14:45:58] then yes [14:46:11] alright [14:46:30] otherwise you can define a check in https://github.com/miraheze/puppet/blob/master/modules/monitoring/files/commands.conf [14:46:31] [ puppet/commands.conf at master · miraheze/puppet · GitHub ] - github.com [14:47:20] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJsAC [14:47:22] [02miraheze/puppet] 07Southparkfan 0390776b6 - Add NTP check [14:49:06] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsA4 [14:49:07] [02miraheze/puppet] 07Southparkfan 03a448260 - Fix spelling mistake [14:52:32] RECOVERY - mw6 NTP time on mw6 is OK: NTP OK: Offset -0.000270396471 secs [14:52:50] RECOVERY - jobrunner1 NTP time on jobrunner1 is OK: NTP OK: Offset 0.0006007552147 secs [14:52:55] RECOVERY - cp8 NTP time on cp8 is OK: NTP OK: Offset -0.0006347000599 secs [14:53:23] RECOVERY - mw7 NTP time on mw7 is OK: NTP OK: Offset 0.0009813308716 secs [14:53:47] RECOVERY - cp3 NTP time on cp3 is OK: NTP OK: Offset 0.0003127157688 secs [14:56:48] https://github.com/wikimedia/puppet/blob/production/modules/base/manifests/monitoring/host.pp#L121 I like this, though [14:56:49] [ puppet/host.pp at production · wikimedia/puppet · GitHub ] - github.com [14:57:07] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2646 MB (10% inode=93%); [15:05:54] !log install nagios-plugins-contrib on mon1 [15:05:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:08:27] !log removed nagios-plugins-contrib, does not support raid checks for mdadm [15:08:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:26:39] [02puppet] 07paladox opened pull request 03#1446: Backup db11/12 - 13https://git.io/JJsxy [15:33:35] [02puppet] 07paladox synchronize pull request 03#1446: Backup db11/12 - 13https://git.io/JJsxy [15:38:25] [02puppet] 07paladox closed pull request 03#1446: Backup db11/12 - 13https://git.io/JJsxy [15:38:29] [02puppet] 07paladox reopened pull request 03#1446: Backup db11/12 - 13https://git.io/JJsxy [15:40:19] [02puppet] 07paladox closed pull request 03#1446: Backup db11/12 - 13https://git.io/JJsxy [15:41:44] !log root@gluster1:/home/paladox# gluster volume set mvol network.inode-lru-limit 600 [15:41:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:50:19] !log increase i/o threads to 40 on gluster [15:50:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:02] !log root@gluster1:/home/paladox# gluster volume set mvol performance.cache-max-file-size 20MB [15:52:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:53:47] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: no terminated jobs [15:53:59] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: CRITICAL: no terminated jobs [15:58:02] [02CreateWiki] 07paladox synchronize pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:00:23] [02CreateWiki] 07JohnFLewis reviewed pull request 03#154 commit - 13https://git.io/JJshJ [16:00:43] [02CreateWiki] 07JohnFLewis reviewed pull request 03#154 commit - 13https://git.io/JJshT [16:00:46] [02CreateWiki] 07paladox synchronize pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:01:22] [02CreateWiki] 07paladox reviewed pull request 03#154 commit - 13https://git.io/JJshI [16:01:44] [02CreateWiki] 07paladox synchronize pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:02:22] [02CreateWiki] 07JohnFLewis reviewed pull request 03#154 commit - 13https://git.io/JJshm [16:10:18] [02CreateWiki] 07paladox synchronize pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:11:21] [02CreateWiki] 07paladox reviewed pull request 03#154 commit - 13https://git.io/JJshg [16:11:51] [02CreateWiki] 07paladox synchronize pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:21:27] [02CreateWiki] 07paladox closed pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:21:30] [02CreateWiki] 07paladox reopened pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:22:03] [02CreateWiki] 07paladox closed pull request 03#154: Add a ChangeDBCluster script - 13https://git.io/JJs4B [16:26:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JJsjO [16:27:23] rhinosf1 : awesome [16:27:25] its run now [16:30:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JJsjg [16:31:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJsjV [16:32:35] [02miraheze/puppet] 07paladox created branch 03paladox-patch-10 13https://git.io/JJsjw [16:32:36] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS [16:33:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JJsji [16:33:20] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJsjP [16:33:21] [02miraheze/CreateWiki] 07paladox 033d36432 - Update changeDBCluster.php [16:33:22] SOMEONE PLEASE +Q THIS BOT [16:33:35] No [16:35:43] [02miraheze/puppet] 07paladox pushed 035 commits to 03master [+0/-0/±5] 13https://git.io/JJsj7 [16:36:11] Naleksuh: you can /ignore it [16:36:34] miraheze/CreateWiki/paladox-patch-3/3d36432 - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/175732933 [16:39:09] [02miraheze/puppet] 07paladox pushed 032 commits to 03paladox-patch-10 [+0/-0/±2] 13https://git.io/JJsjh [16:39:42] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±2] 13https://git.io/JJGev [16:39:43] [02miraheze/mediawiki] 07paladox 03eaa56ca - Update CreateWiki [16:39:47] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:39:56] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:40:21] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJGeL [16:40:23] [02miraheze/mediawiki] 07paladox 03c7bd520 - Update RottenLinks [16:40:24] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJGet [16:40:26] [02miraheze/CreateWiki] 07paladox 03b949bde - Update changeDBCluster.php [16:40:45] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:41:41] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:41:56] miraheze/CreateWiki/paladox-patch-3/b949bde - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/175734960 [16:43:38] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:43:46] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:44] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:22] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJGea [16:46:23] [02miraheze/CreateWiki] 07paladox 030761acc - Update changeDBCluster.php [16:47:18] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJGeK [16:47:19] [02miraheze/CreateWiki] 07paladox 03590de37 - Update changeDBCluster.php [16:47:55] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:00] miraheze/CreateWiki/paladox-patch-3/0761acc - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/175736482 [16:49:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGeH [16:49:19] [02miraheze/services] 07MirahezeSSLBot 030ae2f90 - BOT: Updating services config for wikis [16:49:43] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJGeQ [16:49:44] [02miraheze/CreateWiki] 07paladox 03da0abfc - Update changeDBCluster.php [16:49:54] miraheze/CreateWiki/paladox-patch-3/590de37 - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/175736738 [16:50:03] Naleksuh: how long does it wait before clearing? [16:51:13] [02mw-config] 07paladox opened pull request 03#3169: Switch wgCreateWikiDatabaseClusters to db12 (c3) - 13https://git.io/JJGe5 [16:51:27] miraheze/CreateWiki/paladox-patch-3/da0abfc - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/175737382 [16:53:25] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3 [16:54:31] [02mw-config] 07paladox edited pull request 03#3169: Switch wgCreateWikiDatabaseClusters to db12 (c3) - 13https://git.io/JJGe5 [16:54:41] [02mw-config] 07paladox closed pull request 03#3169: Switch wgCreateWikiDatabaseClusters to db12 (c3) - 13https://git.io/JJGe5 [16:55:15] [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL [16:57:18] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JJGvn [16:57:19] [02miraheze/CreateWiki] 07paladox 0366a250a - Add a ChangeDBCluster script (#154) [16:57:59] rhinosf1 : one hour [16:58:10] Naleksuh: great! [16:58:31] i also shut the bot in -testwiki up [17:01:28] ah nice [17:01:35] except my client seems to forget my ignored users when i restart it [17:03:06] Ah [17:03:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJGv0 [17:03:27] [02miraheze/mw-config] 07paladox 03d4be0e4 - Switch wgCreateWikiDatabaseClusters to db12 (c3) [17:03:32] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [17:05:03] miraheze/mw-config/paladox-patch-1/d4be0e4 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/708403321 [17:06:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGvo [17:06:10] [02miraheze/mw-config] 07paladox 032e02e8b - Switch wgCreateWikiDatabaseClusters to db12 (c3) (#3169) [17:12:55] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [17:12:57] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:01:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JJGJx [18:01:35] [02miraheze/mw-config] 07paladox 0331809b4 - Update and rename DB12Wikis.php to TempDBList.php [18:01:36] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:01:38] [02mw-config] 07paladox opened pull request 03#3170: Update and rename DB12Wikis.php to TempDBList.php - 13https://git.io/JJGJp [18:03:06] miraheze/mw-config/paladox-patch-1/31809b4 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/708422164 [18:10:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGUI [18:10:01] [02miraheze/mw-config] 07paladox 0344d2c50 - Add sitenotice for c1 cluster [18:10:03] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:10:04] [02mw-config] 07paladox opened pull request 03#3171: Add sitenotice for c1 cluster - 13https://git.io/JJGUL [18:11:34] miraheze/mw-config/paladox-patch-2/44d2c50 - paladox The build failed. https://travis-ci.org/miraheze/mw-config/builds/708424601 [18:21:43] [02mw-config] 07paladox synchronize pull request 03#3170: Update and rename DB12Wikis.php to TempDBList.php - 13https://git.io/JJGJp [18:21:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJGU6 [18:21:46] [02miraheze/mw-config] 07paladox 03a39e077 - Update TempDBList.php [18:22:50] [02mw-config] 07paladox closed pull request 03#3170: Update and rename DB12Wikis.php to TempDBList.php - 13https://git.io/JJGJp [18:22:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/JJGUP [18:22:53] [02miraheze/mw-config] 07paladox 0390723de - Update and rename DB12Wikis.php to TempDBList.php (#3170) * Update and rename DB12Wikis.php to TempDBList.php * Update TempDBList.php [18:22:54] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [18:22:56] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:23:22] miraheze/mw-config/paladox-patch-1/a39e077 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/708428920 [18:24:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJGUD [18:24:33] [02miraheze/mw-config] 07paladox 0340f0e6e - Set first batch of wikis into read only [18:24:34] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:24:36] [02mw-config] 07paladox opened pull request 03#3172: Set first batch of wikis into read only - 13https://git.io/JJGUy [18:26:27] miraheze/mw-config/paladox-patch-1/40f0e6e - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/708429668 [18:26:44] [02mw-config] 07paladox synchronize pull request 03#3171: Add sitenotice for c1 cluster - 13https://git.io/JJGUL [18:26:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGUQ [18:26:47] [02miraheze/mw-config] 07paladox 03299d005 - Update Sitenotice.php [18:27:58] [02mw-config] 07paladox synchronize pull request 03#3171: Add sitenotice for c1 cluster - 13https://git.io/JJGUL [18:28:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGU5 [18:28:01] [02miraheze/mw-config] 07paladox 030f976e1 - Update Sitenotice.php [18:28:31] miraheze/mw-config/paladox-patch-2/299d005 - paladox The build is still failing. https://travis-ci.org/miraheze/mw-config/builds/708430357 [18:29:46] miraheze/mw-config/paladox-patch-2/0f976e1 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/708430647 [18:31:32] [02mw-config] 07paladox synchronize pull request 03#3171: Add sitenotice for c1 cluster - 13https://git.io/JJGUL [18:31:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGTt [18:31:35] [02miraheze/mw-config] 07paladox 03f697413 - Update Sitenotice.php [18:31:44] [02mw-config] 07paladox edited pull request 03#3171: Add sitenotice for db maintenance - 13https://git.io/JJGUL [18:32:06] paladox: see my review [18:32:32] [02mw-config] 07paladox synchronize pull request 03#3171: Add sitenotice for db maintenance - 13https://git.io/JJGUL [18:32:32] RhinosF1 done [18:32:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGT8 [18:32:35] [02miraheze/mw-config] 07paladox 03a16b51d - Update Sitenotice.php [18:32:45] [02mw-config] 07paladox commented on pull request 03#3171: Add sitenotice for db maintenance - 13https://git.io/JJGT4 [18:34:20] paladox: left another [18:35:39] RhinosF1 thanks! done [18:35:45] [02mw-config] 07paladox synchronize pull request 03#3171: Add sitenotice for db maintenance - 13https://git.io/JJGUL [18:35:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGTV [18:35:48] [02miraheze/mw-config] 07paladox 032d78258 - Update Sitenotice.php [18:36:21] Approved paladox [18:36:32] Noting 21:00 UTC is 10pm UK timr [18:36:44] So will be online from start for 2.5 hours [18:36:49] yeh [18:36:52] Someone can take the last bit after [18:37:12] paladox: it wouldn't be the first time we've got timezones wrong [18:37:22] yeh :( [18:38:28] Timezones are hard [18:38:48] [02mw-config] 07paladox closed pull request 03#3171: Add sitenotice for db maintenance - 13https://git.io/JJGUL [18:38:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJGT6 [18:38:51] [02miraheze/mw-config] 07paladox 03d7a413a - Add sitenotice for db maintenance (#3171) [18:38:52] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:38:54] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [18:41:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJGTD [18:41:54] [02miraheze/mw-config] 07paladox 033cb496b - Disable createwiki temporarily [18:41:55] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:41:57] [02mw-config] 07paladox opened pull request 03#3173: Disable createwiki temporarily - 13https://git.io/JJGTy [18:50:30] What is c1 wikis? [18:50:34] > [mw-config] paladox opened pull request #3173: Disable createwiki temporarily - https://git.io/JJGTy @paladox The two wikis I just approved seem to have been created without issue, just wanted to make sure this was okay? [18:50:36] [ Disable createwiki temporarily by paladox · Pull Request #3173 · miraheze/mw-config · GitHub ] - git.io [18:50:52] Yes, it was ok. [18:51:01] Oh okay, phew. Thanks! 🙂 [18:51:19] It hasn't been merged yet @Doug [18:51:49] Ah, right. Yeah, he didn't push the commit yet. Thanks, @RhinosF1 🙂 [18:52:13] @Doug: CreateWiki will show access denied when it's merged. [18:52:24] Ah, okay, that's good. [18:52:50] > What is c1 wikis? @MrJaroslavik: All have sitenotices up now. [18:52:51] Fixing bugs, or new features? [18:52:59] Database switch [18:53:02] ah [18:53:23] Ok, so my new wiki too. Thank! [18:57:12] > Ok, so my new wiki too. Thank! @MrJaroslavik: Nearly all wikis are c1. All wikis use c1 for manage/create wiki + some global stuff. [18:57:36] C is cluster, right? [18:58:01] Yes [19:26:55] hi [19:32:51]