[00:01:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JULZS [00:01:18] [02miraheze/puppet] 07paladox 03e7443ee - Update mediawiki.conf [00:01:20] [02puppet] 07paladox synchronize pull request 03#1486: varnish: Fix rate limiter - 13https://git.io/JULZR [00:03:00] [02ManageWiki] 07Universal-Omega synchronize pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [00:04:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULZF [00:04:29] [02miraheze/puppet] 07paladox 0337cfc1e - Update mediawiki.conf [00:05:10] [02puppet] 07paladox synchronize pull request 03#1486: varnish: Fix rate limiter - 13https://git.io/JULZR [00:05:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JULZb [00:05:13] [02miraheze/puppet] 07paladox 032533d21 - Update mediawiki.conf [00:07:51] [02puppet] 07paladox edited pull request 03#1486: varnish: Fix rate limiter - 13https://git.io/JULZR [00:07:56] [02puppet] 07paladox closed pull request 03#1486: varnish: Fix rate limiter - 13https://git.io/JULZR [00:07:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JULZp [00:07:59] [02miraheze/puppet] 07paladox 036888aa2 - varnish: Fix rate limiter (#1486) * varnish: Fix rate limiter * Whitelist all of Miraheze ips * Do not rate limit static.miraheze.org as it can handle bursts. * Update nginx.pp * Update mediawiki.conf * Update mediawiki.conf [00:08:48] [02ManageWiki] 07Universal-Omega synchronize pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [00:09:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1486-paladox-patch-2 [+0/-0/±2] 13https://git.io/JULZj [00:09:50] [02miraheze/puppet] 07paladox 034efafd2 - Revert "varnish: Fix rate limiter (#1486)" This reverts commit 6888aa26eb1d0b43c40dd6a77855b4a0f0e0d780. [00:09:51] [02puppet] 07paladox created branch 03revert-1486-paladox-patch-2 - 13https://git.io/vbiAS [00:09:53] [02puppet] 07paladox opened pull request 03#1487: Revert "varnish: Fix rate limiter" - 13https://git.io/JULne [00:09:56] [02puppet] 07paladox closed pull request 03#1487: Revert "varnish: Fix rate limiter" - 13https://git.io/JULne [00:09:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JULnv [00:09:59] [02miraheze/puppet] 07paladox 033b63504 - Revert "varnish: Fix rate limiter (#1486)" (#1487) This reverts commit 6888aa26eb1d0b43c40dd6a77855b4a0f0e0d780. [00:10:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±2] 13https://git.io/JULnf [00:10:05] [02miraheze/puppet] 07paladox 030bfa2d5 - Revert "Revert "varnish: Fix rate limiter (#1486)" (#1487)" This reverts commit 3b635041779250aabbf3d426a505b8eede8cd155. [00:10:07] [02puppet] 07paladox created branch 03revert-1487-revert-1486-paladox-patch-2 - 13https://git.io/vbiAS [00:10:12] [02puppet] 07paladox opened pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:11:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULnU [00:11:36] [02miraheze/puppet] 07paladox 033648df9 - Update mediawiki.conf [00:12:30] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 41 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [00:13:16] [02puppet] 07paladox synchronize pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:13:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±1] 13https://git.io/JULnI [00:13:19] [02miraheze/puppet] 07paladox 030c87d38 - Update mediawiki.conf [00:14:45] [02puppet] 07paladox synchronize pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:14:46] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [00:14:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±1] 13https://git.io/JULnm [00:14:48] [02miraheze/puppet] 07paladox 035e9b5b9 - Update mediawiki.conf [00:15:35] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [00:16:30] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:17:06] [02miraheze/puppet] 07paladox pushed 032 commits to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±1] 13https://git.io/JULnc [00:17:07] [02miraheze/puppet] 07paladox 03a535c8b - Merge branch 'master' into revert-1487-revert-1486-paladox-patch-2 [00:17:09] [02puppet] 07paladox synchronize pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:17:17] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [00:17:30] [02puppet] 07paladox synchronize pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:17:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±1] 13https://git.io/JULnC [00:17:33] [02miraheze/puppet] 07paladox 03f0db5d9 - Update mediawiki.conf [00:20:42] [02puppet] 07paladox synchronize pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:20:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±1] 13https://git.io/JULnB [00:20:45] [02miraheze/puppet] 07paladox 03e25b8dd - Update mediawiki.conf [00:22:46] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:23:06] [02puppet] 07paladox synchronize pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:23:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±1] 13https://git.io/JULnE [00:23:09] [02miraheze/puppet] 07paladox 032cf74ac - Update mediawiki.conf [00:23:35] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:01] [02puppet] 07paladox synchronize pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [00:24:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1487-revert-1486-paladox-patch-2 [+0/-0/±1] 13https://git.io/JULnu [00:24:04] [02miraheze/puppet] 07paladox 03d9a709c - Update mediawiki.conf [00:25:24] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:30:47] !log restarted nginx on cp7 [00:30:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:36] !log restarted nginx on cp9 [00:32:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:07] !log restarted nginx on cp6 [00:34:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:20] [02mw-config] 07paladox reviewed pull request 03#3215 commit - 13https://git.io/JULn1 [00:41:46] [02mw-config] 07paladox reviewed pull request 03#3215 commit - 13https://git.io/JULny [00:41:48] [02mw-config] 07paladox reviewed pull request 03#3215 commit - 13https://git.io/JULnS [00:43:35] !log restarted nginx on cp3 [00:43:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:51] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JULnA [00:49:52] [02miraheze/ManageWiki] 07paladox 031f70534 - support 'options' under 'userrights' This merges options with userrights if defined. [00:49:54] [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns [00:49:55] [02ManageWiki] 07paladox opened pull request 03#180: support 'options' under 'userrights' - 13https://git.io/JULnx [00:52:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.35, 19.26, 17.42 [00:52:21] miraheze/ManageWiki/paladox-patch-1/1f70534 - paladox The build passed. https://travis-ci.org/miraheze/ManageWiki/builds/722194566 [00:52:37] [02ManageWiki] 07paladox closed pull request 03#180: support 'options' under 'userrights' - 13https://git.io/JULnx [00:52:39] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULcv [00:52:40] [02miraheze/ManageWiki] 07paladox 03c8c466f - support 'options' under 'userrights' (#180) This merges options with userrights if defined. [00:52:42] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1 [00:52:43] [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns [00:54:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.34, 17.81, 17.11 [02:15:23] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.14, 18.77, 15.77 [02:17:23] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.55, 17.34, 15.64 [03:12:39] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:17] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 8.21, 5.69, 2.74 [03:14:36] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:18:14] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.01, 1.22, 1.67 [03:27:26] [02puppet] 07paladox closed pull request 03#1488: varnish: Fix rate limiter - 13https://git.io/JULnJ [03:27:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JUL89 [03:27:29] [02miraheze/puppet] 07paladox 03b2d773c - varnish: Fix rate limiter (#1488) * Revert "Revert "varnish: Fix rate limiter (#1486)" (#1487)" This reverts commit 3b635041779250aabbf3d426a505b8eede8cd155. * Update mediawiki.conf * Update mediawiki.conf * Update mediawiki.conf * Update mediawiki.conf * Update mediawiki.conf * Update mediawiki.conf [03:27:31] [02miraheze/puppet] 07paladox deleted branch 03revert-1487-revert-1486-paladox-patch-2 [03:27:32] [02puppet] 07paladox deleted branch 03revert-1487-revert-1486-paladox-patch-2 - 13https://git.io/vbiAS [06:06:29] RECOVERY - ns2 APT on ns2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:06:46] RECOVERY - mail1 APT on mail1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:08:41] RECOVERY - cp3 APT on cp3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:25:35] RECOVERY - db7 APT on db7 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:25:51] RECOVERY - mon1 APT on mon1 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [06:27:07] RECOVERY - mw6 APT on mw6 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:32:38] RECOVERY - cp7 APT on cp7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:39:04] [02ManageWiki] 07supertassu reviewed pull request 03#179 commit - 13https://git.io/JULzY [06:39:06] [02ManageWiki] 07supertassu reviewed pull request 03#179 commit - 13https://git.io/JULzO [06:39:07] [02ManageWiki] 07supertassu reviewed pull request 03#179 commit - 13https://git.io/JULz3 [06:43:03] !log deleted and dropped hispano76wiki and hispanowiki (reset req) [06:43:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:58] RECOVERY - cp9 APT on cp9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULz8 [06:45:13] [02miraheze/services] 07MirahezeSSLBot 03c62a477 - BOT: Updating services config for wikis [06:45:14] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:47:23] PROBLEM - wiki.thehall.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thehall.xyz' expires in 15 day(s) (Mon 14 Sep 2020 06:42:11 GMT +0000). [06:50:32] RECOVERY - db12 APT on db12 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:52:04] PROBLEM - kkutu.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kkutu.wiki' expires in 15 day(s) (Mon 14 Sep 2020 06:44:01 GMT +0000). [06:52:08] RECOVERY - puppet2 APT on puppet2 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [06:53:11] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.stellurgists.wiki' expires in 15 day(s) (Mon 14 Sep 2020 06:48:45 GMT +0000). [06:53:40] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-11] [06:54:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULzr [06:54:22] [02miraheze/ssl] 07MirahezeSSLBot 03b5a3ec3 - Bot: Update SSL cert for wiki.thehall.xyz [06:57:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULzo [06:57:11] [02miraheze/ssl] 07MirahezeSSLBot 03c4d6703 - Bot: Update SSL cert for archive.stellurgists.wiki [06:59:35] RECOVERY - cp6 APT on cp6 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:59:53] [02ManageWiki] 07Universal-Omega reviewed pull request 03#179 commit - 13https://git.io/JULz6 [07:01:40] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:02:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULzD [07:02:35] [02miraheze/ssl] 07MirahezeSSLBot 03e4b5659 - Bot: Update SSL cert for kkutu.wiki [07:06:44] RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Fri 27 Nov 2020 05:57:03 GMT +0000. [07:08:11] RECOVERY - wiki.thehall.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thehall.xyz' will expire on Fri 27 Nov 2020 05:54:13 GMT +0000. [07:12:13] RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Fri 27 Nov 2020 06:02:26 GMT +0000. [07:26:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.41, 21.54, 16.77 [07:28:16] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.48, 19.27, 16.52 [08:23:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.02, 22.81, 18.61 [08:24:15] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Mon 14 Sep 2020 08:15:37 GMT +0000). [08:24:18] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 8.50, 6.73, 4.42 [08:25:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.70, 20.66, 18.35 [08:25:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULVu [08:25:27] [02miraheze/ssl] 07MirahezeSSLBot 03a292dcd - Bot: Update SSL cert for guia.cineastas.pt [08:26:15] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.60, 5.02, 4.05 [08:27:41] [02ManageWiki] 07supertassu reviewed pull request 03#179 commit - 13https://git.io/JULVa [08:29:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.02, 19.17, 18.42 [08:31:04] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:35:03] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [08:37:55] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Fri 27 Nov 2020 07:25:18 GMT +0000. [09:00:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULwx [09:00:19] [02miraheze/services] 07MirahezeSSLBot 03fd48ccc - BOT: Updating services config for wikis [09:16:12] PROBLEM - wiki.lbcomms.co.za - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lbcomms.co.za' expires in 15 day(s) (Mon 14 Sep 2020 09:10:41 GMT +0000). [09:17:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULr8 [09:17:44] [02miraheze/ssl] 07MirahezeSSLBot 030396a3a - Bot: Update SSL cert for wiki.lbcomms.co.za [09:22:51] RECOVERY - wiki.lbcomms.co.za - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lbcomms.co.za' will expire on Fri 27 Nov 2020 08:17:36 GMT +0000. [10:16:00] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JULoh [10:16:02] [02miraheze/dns] 07Reception123 035462cf3 - add viileapedia.ga zone [10:16:05] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JULoj [10:16:06] [02miraheze/dns] 07Reception123 03cb2552e - add vortekspedia.tk zone [10:16:10] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JULKe [10:16:12] [02miraheze/dns] 07Reception123 0385e5a23 - add debilepedie.gq zone [10:16:16] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JULKv [10:16:18] [02miraheze/dns] 07Reception123 033ee68fe - add kk.uncyc.tk zone [10:28:23] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+4/-0/±1] 13https://git.io/JULKC [10:28:25] [02miraheze/ssl] 07Reception123 03e87e6bd - add 4 certs T6112 viileapedia.ga vortekspedia.tk debilepedie.gq kk.uncyc.tk [13:06:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.17, 20.45, 17.75 [13:08:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.50, 18.16, 17.25 [13:32:05] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.79, 19.93, 17.03 [13:38:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.48, 20.04, 18.41 [14:11:59] [02ManageWiki] 07JohnFLewis commented on commit 03c8c466f00d74b129b6471bf84dbd2c613644ea97 - 13https://git.io/JULD9 [14:21:53] [02ManageWiki] 07Universal-Omega reviewed pull request 03#179 commit - 13https://git.io/JULDj [14:23:38] [02ManageWiki] 07Universal-Omega synchronize pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [14:29:15] [02ManageWiki] 07Universal-Omega synchronize pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [14:32:14] [02ManageWiki] 07Universal-Omega synchronize pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [14:40:15] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.43, 20.94, 18.88 [14:42:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.43, 22.99, 19.84 [14:44:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.64, 20.66, 19.39 [14:46:06] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.44, 19.02, 18.94 [15:21:27] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 14.06, 12.27, 6.09 [15:22:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.55, 26.51, 21.97 [15:24:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.41, 23.43, 21.40 [15:25:27] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.43, 7.50, 5.60 [15:27:27] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.64, 5.60, 5.13 [15:30:14] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.50, 18.91, 19.97 [15:54:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.73, 20.54, 19.62 [15:56:05] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.46, 21.34, 20.00 [15:58:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.96, 19.60, 19.53 [16:02:05] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.35, 21.83, 20.48 [16:04:04] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.69, 20.54, 20.17 [16:06:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.92, 19.86, 19.95 [16:06:23] [02ManageWiki] 07paladox commented on commit 03c8c466f00d74b129b6471bf84dbd2c613644ea97 - 13https://git.io/JULQc [16:19:52] jdstroy: hi! [16:20:32] JohnLewis: * [16:21:08] hi [16:22:48] JohnLewis: I had a question, would admin consensus be enough on Meta to change the "activeusers" definition to 5 edits in 30 days? It doesn't change any policy or anything so I would personally think admin consensus would be enough for such a minor change [16:23:10] if so, could you please change it as Dmehus agrees and RhinosF1 also said they had no issue with that [16:23:13] [02ManageWiki] 07Universal-Omega reviewed pull request 03#179 commit - 13https://git.io/JULQy [16:28:13] [02miraheze/ManageWiki] 07RhinosF1 pushed 031 commit to 03travis-fixes-and-php-support [+0/-0/±1] 13https://git.io/JULQb [16:28:14] [02miraheze/ManageWiki] 07RhinosF1 03dc780e2 - travis: drop php7.2 test and explicitly state os PHP7.2 is dropped in REL1_35 OS parameter should be stated, fixes a build info thing. Similar PR should probably done on all php repos. [16:28:16] [02ManageWiki] 07RhinosF1 created branch 03travis-fixes-and-php-support - 13https://git.io/vpSns [16:28:18] [02ManageWiki] 07RhinosF1 opened pull request 03#181: travis: drop php7.2 test and explicitly state os - 13https://git.io/JULQN [16:30:31] @paladox, JohnLewis: ^ if you want, can't deploy as no access / busy [16:30:46] miraheze/ManageWiki/travis-fixes-and-php-support/dc780e2 - RhinosF1 The build passed. https://travis-ci.org/miraheze/ManageWiki/builds/722318759 [16:33:51] Reception123: what's the relevant settings? [16:34:53] JohnLewis: oh, that's odd actually, it should be $wgActiveUserDays (which also includes how many edits are made in those days) but in ManageWiki there only seems to be the option of days not edits [16:34:56] let me double check that [16:36:16] I thought there was definitely a setting that allowed you to set how many edits one needs to make [16:36:34] https://www.mediawiki.org/wiki/Manual:$wgActiveUserEditCount [16:36:35] [ Manual:$wgActiveUserEditCount - MediaWiki ] - www.mediawiki.org [16:37:06] yeah, I saw they removed that but I thought I remembered a way to change that [16:37:50] so I guess there's no other way to really change it then [16:38:04] thanks Reception123 [16:38:05] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.23, 21.80, 19.88 [16:38:22] no problem hispano76 [16:39:14] yeah, http://phabricator.wikimedia.org/T234798 [16:39:15] [ ⚓ T234798 Special:ActiveUsers and magic word NUMBEROFACTIVEUSERS should support a threshold for very active users ] - phabricator.wikimedia.org [16:39:15] so nevermind then [16:42:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.99, 19.48, 19.49 [17:25:21] !log sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki nevillepediawiki --overwrite [17:25:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:09] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Mon 14 Sep 2020 17:34:31 GMT +0000). [17:48:49] PROBLEM - warmage.necrocity.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'warmage.necrocity.co.uk' expires in 15 day(s) (Mon 14 Sep 2020 17:41:44 GMT +0000). [17:51:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULF6 [17:51:25] [02miraheze/ssl] 07MirahezeSSLBot 03ab86e1c - Bot: Update SSL cert for wiki.s23.org [17:51:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULFX [17:51:57] [02miraheze/ssl] 07MirahezeSSLBot 03b88a6a8 - Bot: Update SSL cert for warmage.necrocity.co.uk [18:02:33] RECOVERY - warmage.necrocity.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'warmage.necrocity.co.uk' will expire on Fri 27 Nov 2020 16:51:47 GMT +0000. [18:03:34] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Fri 27 Nov 2020 16:51:17 GMT +0000. [18:10:18] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULbo [18:10:20] [02miraheze/services] 07MirahezeSSLBot 036d45faa - BOT: Updating services config for wikis [18:10:49] PROBLEM - towarzystwo.prawa.kredytowego.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'towarzystwo.prawa.kredytowego.pl' expires in 15 day(s) (Mon 14 Sep 2020 18:05:08 GMT +0000). [18:12:11] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JULbS [18:12:12] [02miraheze/dns] 07Reception123 03ae8060e - add monarchia.wiki zone [18:12:52] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.25, 23.77, 20.18 [18:13:22] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 128s [18:17:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULNJ [18:17:14] [02miraheze/ssl] 07MirahezeSSLBot 031c00b2c - Bot: Update SSL cert for towarzystwo.prawa.kredytowego.pl [18:17:21] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [18:20:40] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.71, 21.46, 20.99 [18:22:37] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.91, 19.61, 20.32 [18:22:58] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JULNc [18:23:00] [02miraheze/ssl] 07Reception123 033ecddc9 - chouverse.ncps.xyz -> chouverse.cludisciples.cyou [18:24:25] RECOVERY - towarzystwo.prawa.kredytowego.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'towarzystwo.prawa.kredytowego.pl' will expire on Fri 27 Nov 2020 17:17:06 GMT +0000. [18:25:11] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JULN0 [18:25:12] [02miraheze/ssl] 07Reception123 03c3b0020 - add monarchia.wiki cert [18:26:34] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.80, 22.83, 21.67 [18:28:22] Ooops! bug on Hispano76wiki https://usercontent.irccloud-cdn.com/file/dA95GM6g/Screenshot.png [18:28:32] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.46, 18.66, 20.28 [18:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULNh [18:35:12] [02miraheze/services] 07MirahezeSSLBot 03bb5ab74 - BOT: Updating services config for wikis [18:36:23] @System Administrators ^ See @Hispano76's fatal error report when you get a chance. Be curious what the trace shows on that. [18:37:40] will try to look [18:38:41] probably tomorrow [18:39:34] @Reception123 Okay, sounds good. 🙂 [18:56:34] PROBLEM - mg.gyaanipedia.co.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mg.gyaanipedia.co.in could not be found [19:03:30] RECOVERY - mg.gyaanipedia.co.in - reverse DNS on sslhost is OK: rDNS OK - mg.gyaanipedia.co.in reverse DNS resolves to cp7.miraheze.org [19:15:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.36, 21.28, 18.74 [19:17:08] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.97, 19.14, 18.25 [19:28:37] .mhca TaraMondragon82 [19:28:37] https://meta.miraheze.org/wiki/Special:CentralAuth/TaraMondragon82 [19:29:01] @CVT Low-ish priority, but when you get a chance, spam-only account spamming the abuselog on Meta. [19:29:44] Oops, wrong channel, sorry. 🙂 [19:29:54] locked, though just noting those spambots basically do nothing since the abusefilters always catch them but on the safe side we usually lock them just in case [19:30:56] > locked, though just noting those spambots basically do nothing since the abusefilters always catch them but on the safe side we usually lock them just in case @Reception123 Yeah, that was my original question why he locked them if they don't get past the abuse filters, but you said you guys just do that to be safe. Also, if they're spamming the abuse filters, it does create clutter and "noise," so I see the point now. 🙂 [19:31:16] yeah, it's really not the most useful thing but it doesn't hurt either [19:31:40] What's not, the abuse filters? [19:33:42] the locks [19:33:49] the abuse filters are really useful without them we'd be filled with spam [19:37:55] okay [19:38:15] yeah, that's why I did say "low-ish priority"; wasn't expecting you to lock so quickly 😛 [19:38:37] I should just make a list and send them to you once a day; that might be better [20:07:16] heh, at least then I could use MultiLock [20:18:58] @Reception123 Ah, cool! I will do that, then. Does that give you a series of check boxes or radio boxes to select the accounts to which you wish to set as locked? [20:19:50] Well you paste a list and then yeah it lets you decide what to lock all the accounts for [20:20:05] https://commons.wikimedia.org/wiki/File:Centralauth_multilock_detail.png [20:20:06] [ File:Centralauth multilock detail.png - Wikimedia Commons ] - commons.wikimedia.org [20:20:25] Was just going to send that :P [20:20:30] @MrJaroslavik (OFF) is too fast for me [20:20:41] 😆 [20:21:52] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±17] 13https://git.io/JULjM [20:21:53] [02miraheze/ManageWiki] 07JohnFLewis 036e62f41 - Convert ManageWiki to ConfigRegistry (T4400) [20:40:27] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUtei [20:40:28] [02miraheze/dns] 07paladox 036138997 - Add acme challenge for m.miraheze.org [20:42:25] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUteS [20:42:27] [02miraheze/ssl] 07paladox 038e5dc7f - Update m.miraheze.org.crt [20:44:00] [02puppet] 07paladox closed pull request 03#1485: fix reloading nginx after certificate update - 13https://git.io/JUIO6 [20:44:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JUte7 [20:44:03] [02miraheze/puppet] 07paladox 03ef3bfa8 - fix reloading nginx after certificate update (#1485) * fix reloading nginx after certificate update * Update init.pp [20:44:05] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:44:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [20:48:17] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:49:44] ñ [20:52:42] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:53:27] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:53:44] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:54:31] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:55:12] hmm [20:56:23] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:57:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUtv2 [20:57:47] [02miraheze/puppet] 07paladox 030b3545b - Fix relationship chain [21:00:42] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:02:31] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [21:02:50] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUtv5 [21:02:52] [02miraheze/dns] 07paladox 03ec39aa3 - Miraheze.wiki add acme challenge [21:02:57] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Fri 27 Nov 2020 19:41:53 GMT +0000. [21:03:45] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:04:17] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUtvp [21:04:19] [02miraheze/ssl] 07paladox 0318fba7d - Update miraheze.wiki.crt [21:05:28] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [21:08:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUtfI [21:08:02] [02miraheze/puppet] 07paladox 0380a7368 - Phabricator: Reload nginx when certificate updates [21:09:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUtfY [21:09:22] [02miraheze/puppet] 07paladox 03852f688 - Update init.pp [21:14:11] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 27 Nov 2020 20:03:47 GMT +0000. [21:25:27] [02ManageWiki] 07paladox closed pull request 03#181: travis: drop php7.2 test and explicitly state os - 13https://git.io/JULQN [21:25:29] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUtfh [21:25:30] [02miraheze/ManageWiki] 07RhinosF1 03d8cbf6f - travis: drop php7.2 test and explicitly state os (#181) PHP7.2 is dropped in REL1_35 OS parameter should be stated, fixes a build info thing. Similar PR should probably done on all php repos. [21:26:48] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:38:36] !log root@gluster1:/home/paladox# gluster volume set mvol performance.client-io-threads on [21:38:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:13:40] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.07, 18.26, 15.38 [22:15:40] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.51, 17.79, 15.54 [23:14:57] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 10.40, 10.45, 5.52 [23:15:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.82, 25.70, 19.49 [23:16:56] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.57, 7.74, 5.11 [23:17:07] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.99, 23.01, 19.25 [23:20:56] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.10, 4.79, 4.58 [23:21:03] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.34, 18.66, 18.42