[00:01:11] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 5.37, 6.29, 7.81 [00:03:05] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.22, 21.42, 20.79 [00:03:20] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 4.98, 3.33, 2.30 [00:04:22] !log set puppet2 cores to 3 [00:04:30] !log set mw[45] cores to 4 [00:04:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:53] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.91, 6.71, 7.85 [00:05:04] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 18.08, 19.97, 20.35 [00:05:21] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 2.78, 3.23, 2.40 [00:06:50] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 9.09, 7.09, 7.81 [00:06:59] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 9.18, 6.59, 7.34 [00:08:14] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:08:17] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:08:24] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [00:08:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [00:09:13] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JIBVe [00:09:14] [02miraheze/ManageWiki] 07Universal-Omega 03736b24a - add SpecialPageAliases [00:09:14] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 138 failures. Last run 2 minutes ago with 138 failures. Failed resources (up to 3 shown): File[wiki.lefrenchmelee.fr_private],File[n64brew.dev],File[n64brew.dev_private],File[wiki.kali-team.cn] [00:09:16] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [00:09:18] [02ManageWiki] 07Universal-Omega opened pull request 03#208: add SpecialPageAliases - 13https://git.io/JIBVf [00:09:33] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw5 [00:10:22] PROBLEM - mw5 Check Gluster Clients on mw5 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [00:10:39] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 12.90, 6.23, 2.49 [00:11:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [00:11:49] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw5 [00:11:56] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw5 [00:12:11] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIBV0 [00:12:12] [02miraheze/ManageWiki] 07Universal-Omega 03fbe11df - add ManageWiki.i18n.alias.php [00:12:14] [02ManageWiki] 07Universal-Omega synchronize pull request 03#208: add SpecialPageAliases - 13https://git.io/JIBVf [00:12:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [00:12:59] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.50, 20.01, 20.00 [00:13:03] miraheze/mw-config/parsoid-php-take-2/e0ef5dd - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/748194545 [00:13:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.15, 20.34, 23.90 [00:13:27] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:48] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:57] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [00:14:02] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:14:23] RECOVERY - mw5 Check Gluster Clients on mw5 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:14:24] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:31] PROBLEM - hololive.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hololive.wiki could not be found [00:14:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [00:14:35] PROBLEM - wiki.azimuth.navy - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.azimuth.navy could not be found [00:14:56] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 41.28, 26.38, 22.21 [00:15:19] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 5.49, 3.89, 2.97 [00:15:22] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.011 second response time [00:15:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:15:32] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [00:15:48] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [00:15:48] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15963 bytes in 0.009 second response time [00:16:02] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 19742 bytes in 0.623 second response time [00:16:25] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.960 second response time [00:16:44] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [00:16:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:16:52] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.84, 23.11, 21.52 [00:16:52] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [00:16:55] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [00:16:58] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.nonbinary.wiki could not be found [00:17:09] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [00:17:19] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.45, 3.84, 3.08 [00:18:28] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 4.55, 7.17, 4.82 [00:18:37] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 4.09, 6.75, 7.68 [00:18:39] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 5.61, 7.00, 7.89 [00:20:04] [02mw-config] 07paladox commented on pull request 03#3543: add parsoid extension to extension-list - 13https://git.io/JIBwg [00:20:23] [02mw-config] 07paladox closed pull request 03#3542: use $IP for vendor/wikimedia/parsoid/extension.json - 13https://git.io/JIBu3 [00:20:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBww [00:20:25] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.21, 6.34, 4.80 [00:20:26] [02miraheze/mw-config] 07Universal-Omega 036afe663 - use $IP for vendor/wikimedia/parsoid/extension.json (#3542) [00:20:27] [02miraheze/mw-config] 07paladox deleted branch 03Universal-Omega-patch-4 [00:20:29] [02mw-config] 07paladox deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [00:20:44] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.94, 18.66, 20.04 [00:21:19] RECOVERY - hololive.wiki - reverse DNS on sslhost is OK: rDNS OK - hololive.wiki reverse DNS resolves to cp6.miraheze.org [00:21:19] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 4.18, 3.94, 3.29 [00:21:21] RECOVERY - wiki.azimuth.navy - reverse DNS on sslhost is OK: rDNS OK - wiki.azimuth.navy reverse DNS resolves to cp7.miraheze.org [00:22:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBwS [00:22:28] [02miraheze/mw-config] 07paladox 031028827 - Logging: Drop gitinfo channel [00:23:07] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.19, 5.85, 3.22 [00:23:33] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: rDNS OK - wiki.nowchess.org reverse DNS resolves to cp6.miraheze.org [00:23:42] RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - data.nonbinary.wiki reverse DNS resolves to cp6.miraheze.org [00:23:49] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp6.miraheze.org [00:23:54] RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: rDNS OK - guia.esporo.net reverse DNS resolves to cp6.miraheze.org [00:24:46] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.97, 22.05, 21.16 [00:25:06] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.66, 6.17, 3.68 [00:26:37] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 4.50, 5.64, 6.72 [00:26:44] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.25, 18.69, 20.06 [00:27:20] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.26, 3.99, 3.60 [00:28:39] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 3.90, 5.18, 6.56 [00:32:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.97, 17.92, 20.02 [00:33:19] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 2.99, 3.27, 3.38 [00:37:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBos [00:37:27] [02miraheze/puppet] 07paladox 03c09fb88 - base::puppet: Update puppet cron * Randomises puppet runs with a config that can specify a time * Uses new puppet-run script to run puppet [00:37:28] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [00:37:30] [02puppet] 07paladox opened pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [00:37:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBon [00:37:57] [02miraheze/puppet] 07paladox 0392babe9 - Update puppet.pp [00:37:58] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [00:39:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/JIBou [00:39:26] [02miraheze/puppet] 07paladox 03b823aa2 - Create puppet-run.erb [00:39:28] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [00:41:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIBoy [00:41:39] [02miraheze/mw-config] 07Universal-Omega 03b308ea1 - fix flow-parsoid [00:41:40] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [00:41:42] [02mw-config] 07Universal-Omega opened pull request 03#3544: fix flow-parsoid - 13https://git.io/JIBoS [00:42:56] [02mw-config] 07paladox closed pull request 03#3544: fix flow-parsoid - 13https://git.io/JIBoS [00:42:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBoA [00:42:59] [02miraheze/mw-config] 07Universal-Omega 03bade6e6 - fix flow-parsoid (#3544) [00:43:00] [02mw-config] 07paladox deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [00:43:02] [02miraheze/mw-config] 07paladox deleted branch 03Universal-Omega-patch-3 [00:48:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBKK [00:48:44] [02miraheze/mw-config] 07paladox 03bdde3e7 - Fix setting wgVirtualRestConfig [00:52:29] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JIB6q [00:52:31] [02miraheze/mw-config] 07paladox 03fbf26d3 - Revert "Fix setting wgVirtualRestConfig" This reverts commit bdde3e7f951ae0089332a333921d4263223a84e6. [00:52:32] [02miraheze/mw-config] 07paladox 0306c2c4f - Fix setting wgVirtualRestConfig [00:54:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIB6g [00:54:54] [02miraheze/mw-config] 07Universal-Omega 036faaac1 - remove flow from parsoid [00:57:13] Hello chusan! If you have any questions, feel free to ask and someone should answer soon. [01:04:07] Hello chusan1! If you have any questions, feel free to ask and someone should answer soon. [01:04:43] miraheze/MirahezeMagic/revert-169-revert-168-RhinosF1-patch-1/ea46d9f - paladox The build has errored. https://travis-ci.org/miraheze/MirahezeMagic/builds/748195367 [01:06:13] PROBLEM - test2 APT on test2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:06:16] PROBLEM - test2 Current Load on test2 is CRITICAL: CRITICAL - load average: 8.22, 4.29, 1.77 [01:06:20] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 4.38, 3.62, 2.92 [01:06:40] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.37, 22.60, 19.56 [01:07:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBPL [01:07:57] [02miraheze/mw-config] 07paladox 038c83a64 - Logging: Drop http channel [01:08:11] RECOVERY - test2 APT on test2 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [01:08:39] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.47, 22.56, 19.93 [01:10:12] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 1.75, 2.95, 2.83 [01:10:15] RECOVERY - test2 Current Load on test2 is OK: OK - load average: 1.02, 3.26, 2.01 [01:12:37] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.55, 20.37, 19.57 [01:12:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/JIBPr [01:12:49] [02miraheze/puppet] 07paladox 0398e55ca - Create puppet.cron.erb [01:12:50] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [01:14:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBPP [01:14:08] [02miraheze/puppet] 07paladox 03b83d279 - Update puppet.cron.erb [01:14:09] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [01:18:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBX3 [01:18:30] [02miraheze/puppet] 07paladox 03f808474 - Update puppet.pp [01:18:31] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [01:18:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBXZ [01:18:57] [02miraheze/puppet] 07paladox 03ac7d3c7 - Update puppet.cron.erb [01:18:59] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [01:20:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBX0 [01:20:47] [02miraheze/puppet] 07paladox 031dfe7d1 - Update puppet.cron.erb [01:20:48] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [01:22:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.13, 19.81, 18.99 [01:24:29] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.68, 18.87, 18.78 [01:33:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIB1b [01:33:54] [02miraheze/mw-config] 07Universal-Omega 032a4a066 - Logging: drop caches channel [01:33:55] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:33:57] [02mw-config] 07Universal-Omega opened pull request 03#3545: Logging: drop caches channel - 13https://git.io/JIB1x [01:34:38] [02mw-config] 07Universal-Omega edited pull request 03#3545: Logging: Drop caches channel - 13https://git.io/JIB1x [01:34:47] [02mw-config] 07Universal-Omega edited pull request 03#3545: Logging: Drop 'caches' channel - 13https://git.io/JIB1x [01:34:48] [02mw-config] 07paladox closed pull request 03#3545: Logging: Drop 'caches' channel - 13https://git.io/JIB1x [01:34:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBMU [01:34:51] [02miraheze/mw-config] 07Universal-Omega 0363da622 - Logging: drop caches channel (#3545) [01:34:53] [02mw-config] 07paladox deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:34:54] [02miraheze/mw-config] 07paladox deleted branch 03Universal-Omega-patch-3 [01:36:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JIBMs [01:36:55] [02miraheze/mw-config] 07Universal-Omega 03ebbfbe3 - Remove parsoid [01:36:57] [02mw-config] 07Universal-Omega synchronize pull request 03#3543: add parsoid extension to extension-list - 13https://git.io/JIBui [01:37:09] [02mw-config] 07Universal-Omega edited pull request 03#3543: re-order extension-list - 13https://git.io/JIBui [01:37:35] [02mw-config] 07paladox closed pull request 03#3543: re-order extension-list - 13https://git.io/JIBui [01:37:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBMn [01:37:38] [02miraheze/mw-config] 07Universal-Omega 03305bb69 - re-order extension-list (#3543) [01:37:39] [02miraheze/mw-config] 07paladox deleted branch 03Universal-Omega-patch-5 [01:37:41] [02mw-config] 07paladox deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [01:41:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIBMo [01:41:22] [02miraheze/mw-config] 07Universal-Omega 03e837683 - add Report extension [01:41:24] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:41:25] [02mw-config] 07Universal-Omega opened pull request 03#3546: add Report extension - 13https://git.io/JIBMK [01:41:55] [02mw-config] 07Universal-Omega edited pull request 03#3546: add Report extension (T6526) - 13https://git.io/JIBMK [01:42:21] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.92, 20.33, 19.04 [01:42:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIBMy [01:42:49] [02miraheze/mw-config] 07Universal-Omega 03b0c206c - add Report extension [01:42:51] [02mw-config] 07Universal-Omega synchronize pull request 03#3546: add Report extension (T6526) - 13https://git.io/JIBMK [01:46:19] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.05, 18.30, 18.55 [01:48:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIBDB [01:48:58] [02miraheze/mw-config] 07Universal-Omega 039047487 - add report extension [01:49:00] [02mw-config] 07Universal-Omega synchronize pull request 03#3546: add Report extension (T6526) - 13https://git.io/JIBMK [01:51:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIBDS [01:51:03] [02miraheze/mw-config] 07Universal-Omega 038fbac0c - add report extension [01:51:04] [02mw-config] 07Universal-Omega synchronize pull request 03#3546: add Report extension (T6526) - 13https://git.io/JIBMK [01:52:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±1] 13https://git.io/JIBDb [01:52:29] [02miraheze/mediawiki] 07Universal-Omega 03a0f9083 - Install Report [01:52:31] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:53:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIByJ [01:54:00] [02miraheze/mediawiki] 07Universal-Omega 0351157a5 - add ignore = dirty [01:54:46] [02mediawiki] 07Universal-Omega opened pull request 03#669: Install Report extension (T6526) - 13https://git.io/JIByI [01:57:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JIByu [01:57:11] [02miraheze/puppet] 07paladox 03881373a - Allow specifying hiera values for roles [01:57:13] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [01:57:14] [02puppet] 07paladox opened pull request 03#1548: Allow specifying hiera values for roles - 13https://git.io/JIByz [02:01:47] miraheze/mw-config/paladox-patch-1/0564266 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/748200923 [02:01:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JIBSI [02:02:00] [02miraheze/mw-config] 07Universal-Omega 033f137f7 - remove ULS from translate, since translate already requires ULS in MWE [02:02:02] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [02:02:37] [02mw-config] 07Universal-Omega opened pull request 03#3547: remove ULS from translate, since translate already requires ULS in MWE - 13https://git.io/JIBSC [02:07:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+1/-0/±0] 13https://git.io/JIB9f [02:07:58] [02miraheze/puppet] 07paladox 0352f1292 - Create role.rb [02:08:00] [02puppet] 07paladox synchronize pull request 03#1548: Allow specifying hiera values for roles - 13https://git.io/JIByz [02:09:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JIB9l [02:09:44] [02miraheze/puppet] 07paladox 03ec01804 - Update role.rb [02:09:46] [02puppet] 07paladox synchronize pull request 03#1548: Allow specifying hiera values for roles - 13https://git.io/JIByz [02:09:59] [02puppet] 07paladox closed pull request 03#1548: Allow specifying hiera values for roles - 13https://git.io/JIByz [02:10:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JIB9u [02:10:02] [02miraheze/puppet] 07paladox 039d23644 - Allow specifying hiera values for roles (#1548) * Allow specifying hiera values for roles * Create role.rb * Update role.rb [02:10:04] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [02:10:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [02:14:57] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [02:15:04] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 483 failures. Last run 2 minutes ago with 483 failures. Failed resources (up to 3 shown) [02:15:05] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown) [02:15:16] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown) [02:15:23] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [02:15:47] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 495 failures. Last run 2 minutes ago with 495 failures. Failed resources (up to 3 shown) [02:15:56] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 495 failures. Last run 2 minutes ago with 495 failures. Failed resources (up to 3 shown) [02:15:57] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [02:16:09] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 492 failures. Last run 2 minutes ago with 492 failures. Failed resources (up to 3 shown) [02:16:09] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [02:16:12] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 489 failures. Last run 2 minutes ago with 489 failures. Failed resources (up to 3 shown): File[/usr/local/bin/prometheus-phpfpm-exporter],File[/etc/systemd/system/prometheus-php-fpm.service],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [02:16:16] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 498 failures. Last run 2 minutes ago with 498 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/robots.php],File[/usr/local/bin/fileLockScript.sh],File[/usr/local/bin/foreachwikiindblist],File[/usr/local/bin/pushServices.sh] [02:16:17] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [02:16:19] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [02:16:19] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 3 minutes ago with 30 failures. Failed resources (up to 3 shown) [02:16:21] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [02:16:28] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown) [02:16:29] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [02:16:32] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [02:16:38] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 494 failures. Last run 3 minutes ago with 494 failures. Failed resources (up to 3 shown) [02:16:45] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown) [02:16:49] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown) [02:16:58] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 481 failures. Last run 3 minutes ago with 481 failures. Failed resources (up to 3 shown) [02:17:14] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 490 failures. Last run 3 minutes ago with 490 failures. Failed resources (up to 3 shown): File[/usr/local/bin/pushServices.sh],File[/usr/local/bin/prometheus-phpfpm-exporter],File[/etc/systemd/system/prometheus-php-fpm.service],File[/etc/logrotate.d/puppet] [02:18:07] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 216 failures. Last run 3 minutes ago with 216 failures. Failed resources (up to 3 shown): File[/etc/sudoers],File[/home/nagiosre],File[/home/salt-user],File[/etc/nginx/mime.types] [02:18:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/JIBHg [02:18:51] [02miraheze/puppet] 07paladox 03f8f4928 - Create mediawiki.yaml [02:18:53] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:19:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-1/±0] 13https://git.io/JIBHa [02:19:02] [02miraheze/puppet] 07paladox 038626c48 - Delete mw4.yaml [02:19:04] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:19:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-1/±0] 13https://git.io/JIBHr [02:19:10] [02miraheze/puppet] 07paladox 0370fc806 - Delete mw5.yaml [02:19:12] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 65 failures. Last run 2 minutes ago with 65 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [02:19:12] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:19:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-1/±0] 13https://git.io/JIBHK [02:19:19] [02miraheze/puppet] 07paladox 0380d6fbb - Delete mw6.yaml [02:19:20] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:19:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-1/±0] 13https://git.io/JIBH1 [02:19:28] [02miraheze/puppet] 07paladox 030374bbd - Delete mw7.yaml [02:19:29] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:21:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBQU [02:21:29] [02miraheze/puppet] 07paladox 03402b406 - Update services1.yaml [02:21:30] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:21:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBQt [02:21:38] [02miraheze/puppet] 07paladox 039c5ffdf - Update services2.yaml [02:21:40] [02puppet] 07paladox synchronize pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:23:09] [02puppet] 07paladox closed pull request 03#1547: base::puppet: Update puppet cron - 13https://git.io/JIBoG [02:23:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+3/-4/±3] 13https://git.io/JIBQC [02:23:12] [02miraheze/puppet] 07paladox 0367d6454 - base::puppet: Update puppet cron (#1547) * Randomises puppet runs with a config that can specify a time * Uses new puppet-run script to run puppet [02:23:58] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [02:24:10] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [02:24:18] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [02:24:18] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [02:24:18] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [02:24:19] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:24:28] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:24:28] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:24:32] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:24:40] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:24:45] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:24:50] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:24:57] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:24:58] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:25:04] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:05] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:13] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [02:25:15] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:22] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:34] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:25:35] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:25:46] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:55] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:26:08] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:26:09] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:26:15] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:26:19] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:26:54] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.15, 20.62, 18.31 [02:27:09] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:28:54] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.29, 18.81, 17.95 [02:29:09] err bugger [02:29:11] reverting [02:29:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1547-paladox-patch-4 [+4/-3/±3] 13https://git.io/JIBdv [02:29:20] [02miraheze/puppet] 07paladox 038f7c666 - Revert "base::puppet: Update puppet cron (#1547)" This reverts commit 67d645434d6f3459b028d493e3bbe7e998a7f4d9. [02:29:21] [02puppet] 07paladox created branch 03revert-1547-paladox-patch-4 - 13https://git.io/vbiAS [02:29:23] [02puppet] 07paladox opened pull request 03#1549: Revert "base::puppet: Update puppet cron" - 13https://git.io/JIBdf [02:29:27] [02puppet] 07paladox closed pull request 03#1549: Revert "base::puppet: Update puppet cron" - 13https://git.io/JIBdf [02:29:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+4/-3/±3] 13https://git.io/JIBdI [02:29:30] [02miraheze/puppet] 07paladox 03fcbafbb - Revert "base::puppet: Update puppet cron (#1547)" (#1549) This reverts commit 67d645434d6f3459b028d493e3bbe7e998a7f4d9. [02:29:45] [02puppet] 07paladox deleted branch 03revert-1547-paladox-patch-4 - 13https://git.io/vbiAS [02:29:46] [02miraheze/puppet] 07paladox deleted branch 03revert-1547-paladox-patch-4 [02:30:00] does that have anything to do with why I keep getting the "loss of session data" error? also I can't even logout [02:30:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdG [02:30:17] [02miraheze/puppet] 07paladox 037a62a40 - base::puppet: Update puppet cron [02:30:18] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:30:20] [02puppet] 07paladox opened pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:30:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdR [02:30:40] [02miraheze/puppet] 07paladox 03b78f34b - Update mw5.yaml [02:30:41] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:30:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdE [02:30:52] [02miraheze/puppet] 07paladox 03cc74891 - Update mw6.yaml [02:30:53] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:31:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdz [02:31:03] [02miraheze/puppet] 07paladox 03deb066e - Update mw7.yaml [02:31:05] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:31:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdg [02:31:14] [02miraheze/puppet] 07paladox 035960736 - Update jobrunner1.yaml [02:31:16] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:31:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdV [02:31:26] [02miraheze/puppet] 07paladox 03e8aeeb4 - Update jobrunner2.yaml [02:31:28] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:31:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdr [02:31:37] [02miraheze/puppet] 07paladox 036d2d5ba - Update test2.yaml [02:31:39] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:31:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdX [02:31:56] [02miraheze/puppet] 07paladox 03c1b7441 - Update services1.yaml [02:31:58] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:32:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdM [02:32:05] [02miraheze/puppet] 07paladox 03a154e68 - Update services2.yaml [02:32:07] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:32:21] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 28 failures. Last run 3 minutes ago with 28 failures. Failed resources (up to 3 shown): Package[php7.2-common],Package[php7.2-opcache],Package[php7.2-cli],File[/etc/php/7.2/cli/conf.d] [02:33:14] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 28 failures. Last run 3 minutes ago with 28 failures. Failed resources (up to 3 shown): Package[php7.2-common],Package[php7.2-opcache],Package[php7.2-cli],File[/etc/php/7.2/cli/conf.d] [02:33:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIBdF [02:33:18] [02miraheze/puppet] 07paladox 03a9c06dd - Update puppet.pp [02:33:20] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:33:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/JIBdj [02:33:49] [02miraheze/puppet] 07paladox 034e03c34 - Create puppet-run.erb [02:33:50] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:34:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/JIBFJ [02:34:19] [02miraheze/puppet] 07paladox 03ea036c8 - Create puppet.cron.erb [02:34:20] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:34:21] [02puppet] 07paladox synchronize pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:34:42] [02puppet] 07paladox edited pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:35:14] [02puppet] 07paladox closed pull request 03#1550: base::puppet: Update puppet cron - 13https://git.io/JIBdn [02:35:14] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:35:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±10] 13https://git.io/JIBFY [02:35:17] [02miraheze/puppet] 07paladox 034816228 - base::puppet: Update puppet cron (#1550) * Randomises puppet runs with a config that can specify a time * Uses new puppet-run script to run puppet [02:35:18] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:35:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:45:51] musikanimal if you're on a mac, clear history, make sure to check the `Clear Cookies and Site Data` choice [02:46:33] yeah I fixed it by clearing my cookies. But I must be honest... this is extremely disruptive! Can you not force logout of all users? [02:46:36] !log rm /var/spool/cron/crontabs/root on all servers [02:46:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:03] @paladox can you reply to @musikanimal when you get a chance? [02:50:30] musikanimal Hi, yes. Apologies for that. We tried resetting user tokens for specific users (Doug) and found it didn't work. Using the work around we specified should resolve the issue! Again sorry for the distruption :( (we shouldn't need to do this again) [02:51:37] !log increase puppet2 cores to 4 [02:51:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIBNF [02:52:57] [02miraheze/puppet] 07paladox 031d56dd6 - Update hiera.6.yaml [02:53:34] thanks @paladox. yeah, @musikanimal, our cookie configuration should be much more similar to the Wikimedia cookie configuration, with cookies set on specific wiki subdomains and the CentralAuth login cookie set on a common domain/subdomain, afaik [02:57:35] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 163s [02:58:32] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:58:59] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:59:37] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 2s [03:00:32] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:01:00] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:02:23] it looks like some of my users aren't effected, which is a relief! I have site notices hidden via Common.css because for a while the "this wiki has been closed" banner was still showing even though it was marked as open [03:03:51] @musikanimal ah, yeah, we'll probably remove the central notice in 2-3 days if the login issues slow down. As to that inadvertent wiki closure, that was an issue with our inactive wikis script, which has since been corrected, and we reopened all wikis as of a certain date [03:07:01] miraheze/mw-config/Universal-Omega-patch-2/19d1951 - Universal Omega The build passed. https://travis-ci.org/miraheze/mw-config/builds/748211927 [03:10:16] do you have any idea how widespread this issue is? perhaps it only effects users who logged in to other wikis on the farm? [03:11:45] this user has a SUL (with metawiki and loginwiki created when the account was created), but only edited spcodex. They apparently were not effected by the cookie issue https://spcodex.wiki/w/index.php?title=Special%3ACentralAuth/DavidB [03:11:47] [ Global account information for DavidB - SPCodex ] - spcodex.wiki [03:15:46] Hrm, not sure, exactly. If they never edited `metawiki`, which I think was where cookies were previously set (although I'm not entirely sure as I thought they were set on miraheze.org before), then I wonder if that is a possibility that only users who had made at least one contribution to `metawiki` were affected. @paladox would likely know more on that ^ [03:17:25] The cookies were previously set on .miraheze.org, it’s now set on .miraheze.org. [03:21:20] I appear to have cookies set on `spcodex.wiki` and centralauth cookies set on `.spcodex.wiki` (with an extraneous dot) [03:21:55] Yeh, custom domains had the change done years ago [03:22:50] Just for miraheze, when you logged into 2 wikis, those cookies showed up regardless of wiki (had to be under the miraheze.org domain) [03:23:10] This broke safari when you had a large amount of cookies. [03:23:33] ah ^ didn't know that about the custom domains [03:25:51] well one thing I can say for sure, 99% of spcodex.wiki (custom domain) users have not even viewed another wiki on the farm. So are you saying those such people are unaffected by the cookie issue? if so then I think I'm in good shape :) [03:28:12] yeah it definitely seems that way actually, and now that I think about it, the issue for me was when I tried to access publictestwiki.com after logging in to meta.miraheze.org, where I had a lot of contributions, so perhaps the issue was with users that contributed to wikis with miraheze.org subdomains and custom domains? If they only contributed to one of the two, perhaps there was no login cookie issue for them? 🤔 [03:40:24] I was logged in to a different browser too, and I observe the cookie issue there. I think in that browser I've at least browsed to meta.miraheze.org at some point; however on mobile I'm pretty sure I've only ever viewed SPCodex, and no cookie issue there [03:42:11] also got confirmation from another longtime SPCodex editor that they didn't have this issue, so I'm feeling much better now :) Sorry for freaking out! [04:14:41] miraheze/MirahezeMagic/Universal-Omega-patch-1/fee98de - Universal Omega The build has errored. https://travis-ci.org/miraheze/MirahezeMagic/builds/748216120 [04:51:04] miraheze/mw-config/paladox-patch-2/ba008ff - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/748216335 [05:00:29] [02mediawiki] 07Universal-Omega closed pull request 03#669: Install Report extension (T6526) - 13https://git.io/JIByI [05:00:31] [02miraheze/mediawiki] 07Universal-Omega pushed 033 commits to 03REL1_35 [+2/-0/±3] 13https://git.io/JIRqe [05:00:33] [02miraheze/mediawiki] 07Universal-Omega 0306dbcb6 - Merge pull request #669 from miraheze/Universal-Omega-patch-1 Install Report extension (T6526) [05:00:34] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [05:00:36] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [05:11:15] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:11:47] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:12:11] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:12:16] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:14:29] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03revert-669-Universal-Omega-patch-1 [+0/-1/±1] 13https://git.io/JIRmp [05:14:30]