[00:32:47] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.97, 18.19, 13.82 [00:34:46] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.46, 15.92, 13.51 [02:10:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUyPo [02:10:20] [02miraheze/services] 07MirahezeSSLBot 03bc16a5c - BOT: Updating services config for wikis [04:34:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-2001:41d0:800:105a::6-to-any-port-5666] [04:38:57] PROBLEM - wiki.8b8t.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.8b8t.com could not be found [04:42:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUyFE [05:35:11] [02miraheze/services] 07MirahezeSSLBot 037c9651f - BOT: Updating services config for wikis [05:35:53] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JUyF2 [05:35:55] [02miraheze/dns] 07Reception123 03acfd960 - add savagepedia.wiki zone [05:40:43] Hello Purpose! If you have any questions, feel free to ask and someone should answer soon. [05:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUybt [05:45:12] [02miraheze/services] 07MirahezeSSLBot 03a314354 - BOT: Updating services config for wikis [05:50:53] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JUyb4 [05:50:55] [02miraheze/ssl] 07Reception123 03b627429 - add savagepedia.wiki cert [06:06:12] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JUyNc [06:06:14] [02miraheze/ssl] 07Reception123 03df2bfa4 - add wiki.anglish.info cert [06:16:07] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php dumpBackup.php --current --wiki allthetropeswiki > /home/reception/allthetropes01102020.xml [06:16:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:05:17] !log reception@jobrunner1:~$ sudo /usr/bin/zip /mnt/mediawiki-static/dumps/allthetropeswiki/allthetropeswiki.zip -r /mnt/mediawiki-static/allthetropeswiki/ [07:05:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:12] PROBLEM - jobrunner2 Current Load on jobrunner2 is WARNING: WARNING - load average: 5.26, 5.08, 4.61 [07:35:12] RECOVERY - jobrunner2 Current Load on jobrunner2 is OK: OK - load average: 5.03, 5.04, 4.66 [07:48:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.49, 20.76, 18.02 [07:49:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.94, 17.74, 17.26 [09:22:01] !log reception@jobrunner1:~$ sudo /usr/bin/zip /mnt/mediawiki-static/dumps/allthetropeswiki/allthetropeswiki.zip -r /mnt/mediawiki-static/allthetropeswiki/ -x mnt/mediawiki-static/allthetropes/archive/ -x mnt/mediawiki-static/allthetropes/deleted/ -x mnt/mediawiki-static/allthetropes/lockdir/ -x mnt/mediawiki-static/allthetropes/temp/ -x mnt/mediawiki-static/allthetropes/thumb/ [09:22:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:20:49] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Sat 17 Oct 2020 10:17:20 GMT +0000). [10:21:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUSsm [10:22:00] [02miraheze/ssl] 07MirahezeSSLBot 03ae5c721 - Bot: Update SSL cert for wiki.mxlinuxusers.de [10:34:10] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Wed 30 Dec 2020 09:21:52 GMT +0000. [11:32:59] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.82, 19.97, 17.19 [11:34:57] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.90, 19.47, 17.32 [12:11:49] !log set i/o threads to 4 on gluster [12:11:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:13:11] Hi [12:23:10] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:37:42] Recommendations for an extension that allows pasting of image from clipboard? [12:38:31] I found this thread which recommends a few, but figured I'd ask before installing a bunch https://www.mediawiki.org/wiki/Topic:Rmjodiwykt7uzcmw [12:38:31] [ Copy paste images with clipboard on Project:Support desk ] - www.mediawiki.org [12:38:48] :O good bot [12:53:46] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.35, 19.73, 17.49 [12:59:36] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.86, 19.31, 17.98 [13:03:33] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.75, 20.71, 18.87 [13:05:29] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.49, 22.15, 19.58 [13:07:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.78, 21.30, 19.56 [13:09:21] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.08, 20.05, 19.29 [13:15:13] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.11, 21.53, 20.32 [13:21:05] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.10, 23.91, 21.54 [13:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUSuh [13:25:14] [02miraheze/services] 07MirahezeSSLBot 0360694b0 - BOT: Updating services config for wikis [13:30:56] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.97, 23.80, 22.91 [13:32:52] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.17, 24.63, 23.36 [13:34:49] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.39, 22.30, 22.63 [13:40:37] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.55, 20.89, 21.69 [13:44:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.80, 22.84, 22.42 [13:46:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.23, 23.00, 22.46 [13:48:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.64, 22.09, 22.23 [13:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUSas [13:55:13] [02miraheze/services] 07MirahezeSSLBot 03928fa40 - BOT: Updating services config for wikis [14:02:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.14, 21.21, 21.28 [14:04:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.85, 19.90, 20.79 [14:06:27] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.20, 22.36, 21.51 [14:08:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.91, 20.88, 21.06 [14:14:26] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.53, 19.42, 20.39 [14:18:27] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.61, 24.47, 22.27 [14:26:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.97, 21.47, 22.69 [14:28:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.67, 22.40, 22.83 [14:32:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.04, 23.21, 23.12 [14:35:58] PROBLEM - tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tradwiki.org' expires in 15 day(s) (Sat 17 Oct 2020 14:30:46 GMT +0000). [14:37:45] PROBLEM - www.tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tradwiki.org' expires in 15 day(s) (Sat 17 Oct 2020 14:30:46 GMT +0000). [14:40:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUS6O [14:40:39] [02miraheze/ssl] 07MirahezeSSLBot 03db025d3 - Bot: Update SSL cert for tradwiki.org [14:42:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 36.72, 24.79, 22.97 [14:56:00] RECOVERY - tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tradwiki.org' will expire on Wed 30 Dec 2020 13:40:31 GMT +0000. [14:56:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.98, 23.42, 23.87 [14:57:52] RECOVERY - www.tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tradwiki.org' will expire on Wed 30 Dec 2020 13:40:31 GMT +0000. [14:58:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 20.11, 23.75, 24.03 [15:30:54] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 10.74, 7.51, 5.45 [15:32:02] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 12.07, 7.35, 4.47 [15:32:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [15:33:12] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.13, 6.73, 5.38 [15:34:01] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 5.89, 7.02, 4.74 [15:34:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:34:52] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 5.43, 7.31, 5.92 [15:35:12] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.06, 5.87, 5.22 [15:35:59] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.62, 5.36, 4.40 [15:36:52] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.60, 6.66, 5.84 [15:56:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.80, 20.30, 23.11 [16:00:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUSSd [16:00:22] [02miraheze/services] 07MirahezeSSLBot 03bb29244 - BOT: Updating services config for wikis [16:08:26] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.46, 18.17, 20.32 [16:12:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.31, 21.20, 20.90 [16:14:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.97, 20.34, 20.64 [16:16:26] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.60, 19.68, 20.35 [16:20:12] hello [16:20:25] is there a way to list all the pages that has __DISAMBIG__ tag? [16:20:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.02, 20.50, 20.60 [16:20:46] other than searching "__DISAMBIG__" in the search box, like a special page or something [16:22:27] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.13, 21.93, 21.08 [16:24:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.94, 20.59, 20.71 [16:26:26] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.85, 18.30, 19.84 [16:42:12] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.61, 22.92, 20.99 [16:44:08] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.30, 23.35, 21.40 [16:44:54] Jakeukalane: Se pueden categorizar las páginas de desambiguación mediante una plantilla, por ejemplo es.wikipedia.org/wiki/Plantilla:Desambiguación si es añadida a una página, la categorizará. (Veasé el código de la plantilla) [16:45:28] ajá [16:45:53] y sí uso {{desambiguación}} con _DISAMBIG_ dentro funcionará igual el tag? [16:46:40] sí, porque se transcluirá [16:46:56] puedes ver ejemplos en https://es.wikipedia.org/wiki/Especial:P%C3%A1ginas_de_desambiguaci%C3%B3n [16:49:59] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.90, 22.59, 21.51 [16:57:46] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.43, 22.19, 22.47 [17:01:38] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.61, 24.03, 22.97 [17:05:33] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.35, 23.18, 22.91 [17:06:30] genial [17:13:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.93, 23.20, 22.62 [17:15:17] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.66, 22.39, 22.39 [17:21:08] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.37, 21.92, 22.04 [17:23:05] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.65, 22.18, 22.14 [17:30:51] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.49, 21.11, 21.17 [17:40:34] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.92, 23.19, 23.02 [17:42:32] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.36, 25.19, 23.73 [17:48:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.03, 23.83, 23.81 [17:50:30] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.50, 23.61, 23.69 [17:53:32] PROBLEM - rdb2 APT on rdb2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:55:27] RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [17:56:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.55, 23.13, 23.82 [17:56:56] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw7 [17:58:31] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.46, 25.81, 24.63 [17:58:58] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [18:00:45] PROBLEM - kunwok.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kunwok.org could not be found [18:00:48] PROBLEM - yellowiki.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for yellowiki.xyz could not be found [18:07:25] RECOVERY - kunwok.org - reverse DNS on sslhost is OK: rDNS OK - kunwok.org reverse DNS resolves to cp6.miraheze.org [18:07:29] RECOVERY - yellowiki.xyz - reverse DNS on sslhost is OK: rDNS OK - yellowiki.xyz reverse DNS resolves to cp7.miraheze.org [18:36:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.45, 21.93, 23.87 [18:42:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.42, 22.83, 23.45 [18:46:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.42, 21.99, 23.08 [18:48:27] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.30, 24.71, 23.99 [18:48:45] [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JU9k3 [18:48:46] [02miraheze/ssl] 07Southparkfan 03a26bea5 - Add new wildcard certificate [18:50:24] [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9ku [18:50:25] [02miraheze/ssl] 07Southparkfan 034105a1d - Add extra chain to wildcard [18:50:47] [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9k2 [18:50:48] [02miraheze/ssl] 07Southparkfan 03d7ea431 - Update wildcard.miraheze.org-2020.crt [18:56:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.75, 21.87, 23.36 [18:58:19] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03Southparkfan-patch-1 [+0/-0/±1] 13https://git.io/JU9kh [18:58:20] [02miraheze/puppet] 07Southparkfan 0345b1dab - Push new wildcard certificate for 2020 key rotation [18:58:22] [02puppet] 07Southparkfan created branch 03Southparkfan-patch-1 - 13https://git.io/vbiAS [18:58:23] [02puppet] 07Southparkfan opened pull request 03#1499: Push new wildcard certificate for 2020 key rotation - 13https://git.io/JU9kj [18:58:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.33, 25.49, 24.47 [18:59:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9IJ [18:59:11] [02miraheze/puppet] 07Southparkfan 0367dd378 - Push new wildcard certificate for 2020 key rotation (#1499) [18:59:13] [02puppet] 07paladox closed pull request 03#1499: Push new wildcard certificate for 2020 key rotation - 13https://git.io/JU9kj [18:59:20] [02miraheze/puppet] 07paladox deleted branch 03Southparkfan-patch-1 [18:59:22] [02puppet] 07paladox deleted branch 03Southparkfan-patch-1 - 13https://git.io/vbiAS [19:02:19] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9I0 [19:02:21] [02miraheze/puppet] 07Southparkfan 0370e1de3 - Deploy new wildcard certificate for phabricator [19:11:20] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JU9Ll [19:11:21] [02miraheze/puppet] 07Southparkfan 03198740f - Deploy new wildcard certificate for icinga2, roundcube, matomo [19:14:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.93, 21.23, 23.68 [19:22:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.26, 24.55, 24.05 [19:22:32] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JU9to [19:22:33] [02miraheze/puppet] 07Southparkfan 03cb15055 - Use 2020 wildcard certificate for mediawiki, cache proxies, postfix and dovecot [19:24:41] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [19:26:10] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[dovecot] [19:26:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.02, 20.72, 22.76 [19:26:50] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JU9q3 [19:26:52] PROBLEM - misc1 IMAP on misc1 is CRITICAL: connect to address 185.52.1.76 and port 143: Connection refused [19:26:52] [02miraheze/puppet] 07Southparkfan 034ea382c - Include ssl::wildcard in dovecot and postfix manifests [19:28:11] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:28:52] RECOVERY - misc1 IMAP on misc1 is OK: IMAP OK - 0.028 second response time on 185.52.1.76 port 143 [* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.] [19:40:29] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.85, 18.02, 19.99 [19:41:56] !log disable puppet on mon1 to verify change [19:42:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:48] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.41, 20.59, 20.54 [19:46:53] PROBLEM - mon1 Puppet on mon1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [19:53:17] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.73, 18.74, 20.10 [19:55:05] !log disable puppet on mail1 to verify change [19:55:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:52] !log enable puppet on mail1 [19:57:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:41] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 169s [20:01:11] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JU9Oe [20:01:13] [02miraheze/puppet] 07Southparkfan 0395948bc - Do not use wildcard certificate for authenticating to ldap/mariadb We do check if ldap/mariadb sends a valid certificate, but only the Sectigo.crt file is needed for that. [20:01:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.16, 20.36, 20.04 [20:01:24] !log enable puppet on mon1 [20:01:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:54] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:03:19] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 37.79, 26.42, 22.27 [20:05:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.45, 24.00, 21.91 [20:05:26] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JU9Ol [20:05:28] [02miraheze/puppet] 07Southparkfan 0310003dc - Deploy new TLS wildcard cert on mediawiki services [20:10:27] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9Ob [20:10:29] [02miraheze/puppet] 07Southparkfan 03b2b8256 - Deploy new wildcard cert for grafana.mh.o + remove grafana-new.mh.o [20:11:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.73, 22.81, 21.84 [20:14:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU93m [20:14:21] [02miraheze/puppet] 07paladox 0336734c7 - Deploy new wildcard cert for ldap [20:19:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JU93M [20:19:40] [02miraheze/puppet] 07paladox 035898626 - postgresql: Deploy new cert [20:19:42] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:19:43] [02puppet] 07paladox opened pull request 03#1500: postgresql: Deploy new cert - 13https://git.io/JU93D [20:20:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JU939 [20:20:12] [02miraheze/puppet] 07paladox 03b6da6a3 - Update ssl.conf.erb [20:20:14] [02puppet] 07paladox synchronize pull request 03#1500: postgresql: Deploy new cert - 13https://git.io/JU93D [20:20:25] [02puppet] 07paladox closed pull request 03#1500: postgresql: Deploy new cert - 13https://git.io/JU93D [20:20:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JU935 [20:20:28] [02miraheze/puppet] 07paladox 03c1e4ca6 - postgresql: Deploy new cert (#1500) [20:20:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [20:20:31] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:21:31] !log restart postfix on all servers to resolve mailer-daemon spam [20:22:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:08] !log restart postgresql on puppet2 [20:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:22] !log restart postgresql on puppetdb [20:24:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9sB [20:28:25] [02miraheze/puppet] 07paladox 030ecf7ae - Fix [20:31:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9sH [20:31:38] [02miraheze/puppet] 07paladox 03843e441 - gluster: Deploy new certificate [20:37:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9G3 [20:37:09] [02miraheze/puppet] 07paladox 03799cd3d - Include ssl::wildcard on gluster [20:40:10] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:40:18] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.18, 5.12, 3.82 [20:41:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9Gw [20:41:28] [02miraheze/puppet] 07paladox 03e4126ca - varnish: Increase rate limit Increases rate limit to 50/s, static to 500 and burst to 30 [20:47:10] !log reboot gluster2 [20:47:14] err [20:47:16] jobrunner2 [20:47:18] will fix log [20:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:18] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.26, 5.63, 4.66 [20:49:41] PROBLEM - jobrunner2 JobRunner Service on jobrunner2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [20:50:00] PROBLEM - jobrunner2 JobChron Service on jobrunner2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:50:02] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.07, 6.88, 6.11 [20:51:15] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [20:51:57] RECOVERY - jobrunner2 JobChron Service on jobrunner2 is OK: PROCS OK: 1 process with args 'redisJobChronService' [20:52:00] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 6.79, 6.78, 6.16 [20:53:33] RECOVERY - jobrunner2 JobRunner Service on jobrunner2 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [20:54:18] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 3.84, 4.73, 4.58 [20:54:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9Zi [20:54:27] [02miraheze/puppet] 07paladox 03ba0dcf4 - varnish: Increase rate limit to 120s/r [20:55:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9ZS [20:55:42] [02miraheze/puppet] 07paladox 03100a52e - varnish: Decrease rate limit to 90s/r [21:20:13] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9cx [21:20:14] [02miraheze/mw-config] 07JohnFLewis 0333cb125 - remove duplicate restriction levels [21:21:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.26, 18.52, 23.82 [21:22:07] !log root@jobrunner1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=csydeswiki --report=1 C.Syde_s_Wiki-20200106000305.xml [21:22:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:23:29] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.88, 21.60, 24.32 [21:27:31] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.44, 21.81, 23.79 [21:27:54] [02mw-config] 07RhinosF1 commented on commit 0333cb125e48f25bad2796d05c7534f0056dfc875b - 13https://git.io/JU9C6 [21:29:00] !log root@jobrunner1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=csydeswiki --report=1 C.Sydes+Wiki-20200312083144.xml [21:29:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:25] !log root@jobrunner1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=csydeswiki --report=1 C.Sydes+Wiki-20200317015034.xml [21:30:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:30] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.09, 25.56, 24.24 [21:35:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.40, 23.48, 23.66 [21:37:42] !log depool, reboot and repool mw[4567] [21:37:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:44] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [21:40:13] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw5 [21:41:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.55, 21.95, 22.69 [21:43:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.11, 21.57, 22.50 [21:44:08] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [21:44:15] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [21:46:03] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [21:46:13] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [21:47:30] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.89, 23.73, 23.09 [21:49:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.53, 22.56, 22.73 [21:49:45] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [21:49:48] !log reboot jobrunner1 [21:49:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:14] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [21:50:33] [02landing] 07AntiJawn opened pull request 03#44: Update translations.php - 13https://git.io/JU9lz [21:51:32] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 37.07, 27.19, 24.35 [21:51:51] PROBLEM - jobrunner1 MirahezeRenewSsl on jobrunner1 is CRITICAL: connect to address 51.89.160.135 and port 5000: Connection refused [21:51:52] PROBLEM - jobrunner1 JobChron Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [21:52:21] PROBLEM - jobrunner1 JobRunner Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:53:46] RECOVERY - jobrunner1 MirahezeRenewSsl on jobrunner1 is OK: TCP OK - 0.000 second response time on 51.89.160.135 port 5000 [21:53:49] RECOVERY - jobrunner1 JobChron Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [21:54:21] RECOVERY - jobrunner1 JobRunner Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:55:34] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.50, 22.46, 23.25 [22:01:36] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.16, 20.88, 21.79 [22:03:35] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.19, 21.04, 21.79 [22:05:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.97, 17.07, 20.24 [22:17:38] !log made sensitive comments on Phabricator visible to only Reception123 (public task) [22:17:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:23:35] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.51, 20.79, 19.65 [22:25:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.40, 19.30, 19.24 [22:43:17] !log root@jobrunner1:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php [22:43:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:58] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU90B [22:49:59] > !log root@jobrunner1:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php @paladox Any significant number of jobs outstanding on either of Meta or Test wikis? [22:50:00] [02miraheze/dns] 07paladox 03dee6af3 - Add logger1 to dns [22:50:29] > @paladox Any significant number of jobs outstanding on either of Meta or Test wikis? @Doug that scripts runs it for all wikis. So i'm not sure atm. [22:53:02] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU906 [22:53:04] [02miraheze/dns] 07paladox 037892823 - logger1 -> graylog1 [22:56:58] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikimicrofinanza.it could not be found [22:56:58] PROBLEM - wiki.ombre.io - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ombre.io could not be found [23:03:47] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: rDNS OK - www.wikimicrofinanza.it reverse DNS resolves to cp6.miraheze.org [23:03:48] RECOVERY - wiki.ombre.io - reverse DNS on sslhost is OK: rDNS OK - wiki.ombre.io reverse DNS resolves to cp6.miraheze.org [23:08:49] > @Doug that scripts runs it for all wikis. So i'm not sure atm. @paladox Ah, okay, thanks. 🙂 [23:15:34] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.83, 21.81, 18.33 [23:19:33] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.44, 18.90, 18.03 [23:20:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU9uB [23:20:19] [02miraheze/services] 07MirahezeSSLBot 0397e8565 - BOT: Updating services config for wikis [23:51:03] @System Administrators Jobrunner seems to be backlogged on testwiki. Mass message to two recipients has been queued for more than 10 minutes. Mind checking the jobs outstanding on testwiki and metawiki again? [23:52:31] ran runJobs for those 2 wikis.