[00:10:46] RECOVERY - test2 HTTPS on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.008 second response time [00:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvyua [00:15:10] [02miraheze/services] 07MirahezeSSLBot 038cba7a2 - BOT: Updating services config for wikis [00:45:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvyze [00:45:09] [02miraheze/services] 07MirahezeSSLBot 033aa2f4d - BOT: Updating services config for wikis [00:56:24] PROBLEM - test2 Puppet on test2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:59:36] PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 hour ago with 1 failures [01:08:24] PROBLEM - test2 Current Load on test2 is CRITICAL: CRITICAL - load average: 5.90, 6.69, 3.62 [01:47:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [01:50:50] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 5 minutes ago with 10 failures. Failed resources (up to 3 shown) [01:51:04] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:53:58] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 8 minutes ago with 7 failures. Failed resources (up to 3 shown) [01:54:34] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:55:40] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 8 minutes ago with 32 failures. Failed resources (up to 3 shown): File[encyclopediaofastrobiology.org],File[encyclopediaofastrobiology.org_private],File[www.evanswiki.org],File[www.evanswiki.org_private] [01:57:39] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [01:59:20] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:42:23] PROBLEM - test2 Current Load on test2 is WARNING: WARNING - load average: 0.52, 1.33, 1.99 [02:46:05] RECOVERY - test2 Current Load on test2 is OK: OK - load average: 0.02, 0.65, 1.57 [02:59:34] PROBLEM - test2 Current Load on test2 is CRITICAL: CRITICAL - load average: 3.14, 2.90, 2.14 [03:06:44] PROBLEM - test2 Current Load on test2 is WARNING: WARNING - load average: 0.21, 1.66, 1.94 [03:09:47] RECOVERY - test2 Current Load on test2 is OK: OK - load average: 0.01, 0.90, 1.59 [03:48:00] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2646 MB (10% inode=93%); [04:27:50] Hello IcyPhantom! If you have any questions, feel free to ask and someone should answer soon. [04:40:18] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:44:23] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.869 second response time [05:01:47] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:05:06] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [05:44:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2607:5300:205:200::17f6/cpweb [05:48:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:01:34] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 6 minutes ago with 13 failures. Failed resources (up to 3 shown) [06:02:51] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 8 minutes ago with 13 failures. Failed resources (up to 3 shown) [06:04:30] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 7 minutes ago with 27 failures. Failed resources (up to 3 shown): File[enc.for.uz],File[enc.for.uz_private],File[wiki.mikrodev.com],File[wiki.mikrodev.com_private] [06:05:41] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:06:49] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:08:41] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:28:25] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3476 MB (14% inode=93%); [07:08:10] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:08:43] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:09:37] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.161.32.127/cpweb [07:13:41] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18673 bytes in 1.178 second response time [07:14:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:16:07] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 7.280 second response time [07:16:21] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 9.294 second response time [10:21:45] Hello abcd! If you have any questions, feel free to ask and someone should answer soon. [10:24:55] hi abcd [10:51:33] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 5 minutes ago with 10 failures. Failed resources (up to 3 shown) [10:55:17] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:09:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [12:12:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:23:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb [14:26:03] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb [14:26:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:34:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:52:29] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:1056::2/cpweb [14:56:36] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.011 second response time [14:57:27] sorry paladox is there a reason for several problems when resetting a wiki? because now he won't let me edit this page when he did days ago. That happens to me on my other wikis. https://privado.miraheze.org/w/index.php?title=MediaWiki:Recentchangestext&action=edit I haven't changed the permissions or name spaces [14:57:28] [ Ver el código de «MediaWiki:Recentchangestext» - Privado ] - privado.miraheze.org [14:58:33] I bet its due to content_models, hispano76 can you delete it and recreate it please? [14:58:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:59:45] paladox: is an unchangeable namespace :P [14:59:57] i ment the page [14:59:59] not the ns [15:02:20] hum, ok [15:02:41] (that means it will be a few pages) :/ [15:05:28] thanks* paladox [15:12:46] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb [15:16:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:25:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [15:26:13] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:26:34] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:34:25] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 8.621 second response time [15:34:48] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 7.043 second response time [15:39:06] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [15:39:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [15:43:09] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:51:46] paladox: MediaWiki\Storage\NameTableAccessException why has this been happening since the reset of my wikis? that's happening to other wikis, too? [15:52:32] I'm not sure [15:53:56] Reception123 want to fix ^? [15:54:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.161.32.127/cpweb [15:55:45] paladox: fix what? [15:55:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:56:02] Reception123 hispano76 error [15:56:04] paladox, Reception123: https://www.mediawiki.org/wiki/Manual:PopulateContentTables.php should do it [15:56:04] [ Manual:populateContentTables.php - MediaWiki ] - www.mediawiki.org [15:56:17] Reception123 ^ [15:56:43] paladox: ok [15:56:51] paladox: do I just run that then and that should fix? [15:57:05] i guess so, i'm only 50% sure on that. [15:57:08] So try it :) [15:57:38] Reception123: yes, then after maybe run https://www.mediawiki.org/wiki/Manual:PopulateContentModel.php for each table in all namespaces [15:57:38] [ Manual:populateContentModel.php - MediaWiki ] - www.mediawiki.org [15:57:46] That should clean it up [15:58:04] hispano76: which wiki? [15:58:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:58:35] VisualEditor (parsoid?) seems to be failing again on nenawiki.org. Does it need frequent restarts? Is that something I can do? [15:59:09] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:56] MikeV: I'm not sure why, maybe paladox can be of assistance. As to restarting, I'm not sure if that's the problem but you would not be able to do it yourself unfortunately either way [16:02:13] I've restarted it and does not seem to work. Guessing may be related to slowness which is not something i can fix today. [16:03:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvyh8 [16:03:05] [02miraheze/puppet] 07paladox 0329b98ee - Revert "parsoid: Decrease timeout" This reverts commit 1cfbb56d2ea64cfe705cab965c390632a0cb8250. [16:03:27] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 3.570 second response time [16:04:25] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:04:56] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:08:55] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.502 second response time [16:09:04] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.779 second response time [16:10:11] Reception123: mis tres wikis Hispanowiki Ucroniaswiki y PricadoWiiki [16:10:21] Privado** [16:10:24] hispano76: ok [16:11:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvyhA [16:11:27] [02miraheze/puppet] 07paladox 0308ce068 - services: Fix parsoid systemd service [16:13:35] MikeV seems to work now [16:15:11] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:39] paladox, That's good. It would be the focus of a conference call this afternoon. Can I depend on it? [16:16:42] Hopefully [16:16:48] paladox: does traffic increase affect the slowness? [16:16:51] or is it unrelated? [16:18:15] !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php populateContentModel.php --wiki [hispanowiki/ucroniaswiki/pricadowiki] --ns "all" --table page/rev/art [16:18:15] Seems its still failing [16:19:06] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 4.163 second response time [16:19:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:19:37] hispano76: done [16:22:07] Reception123: did you PopulateContentTables first as well? [16:22:29] ah, no [16:22:34] guess I'll do that [16:23:00] sudo -u www-data php populateContentTables.php --wiki [hispanowiki/ucroniaswiki/pricadowiki] [16:23:05] !log sudo -u www-data php populateContentTables.php --wiki [hispanowiki/ucroniaswiki/pricadowiki] [16:23:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:24:05] Reception123: I think that would have to be run before the other script so maybe run what you did first again [16:24:13] I did and it said 0 [16:26:07] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:28] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:49] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:27:17] Reception123: perfect! [16:27:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [16:30:46] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 6.130 second response time [16:32:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvyj9 [16:32:19] [02miraheze/puppet] 07paladox 03e0a94e6 - Update config.yaml [16:32:27] Reception123: see RfP [16:32:56] RhinosF1: that's if it will load [16:32:56] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:47] JohnLewis, SPF|Cloud: around? [16:35:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvyjF [16:35:13] [02miraheze/puppet] 07paladox 031f435e9 - Update config.yaml [16:36:22] RhinosF1; what’s up? [16:37:01] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 18730 bytes in 0.554 second response time [16:43:47] MikeV i'm going to say its unstable for now. Our focus is on improving reliability (which will fix parsoid as it won't be hitting timeouts) [16:43:51] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.359 second response time [16:43:57] (we're currently working on fixing reliability issues) [16:43:58] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.297 second response time [16:48:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:50:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvSeu [16:50:09] [02miraheze/services] 07MirahezeSSLBot 03c44e24c - BOT: Updating services config for wikis [16:58:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [17:00:39] !log deleted and dropped testdelete1wiki [17:01:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:08:27] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:08:41] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:58] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:03] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:23:58] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.977 second response time [17:23:59] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.347 second response time [17:25:13] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 1.061 second response time [17:25:49] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 9.162 second response time [17:27:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:28:25] I'm getting so used to seeing icinga's spam that it's starting to become soothing. [17:29:39] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:30:12] sariomobile: heh, I've just got used to ignoring it :P [17:31:16] All things considered, I'm taking any comfort I can find. [17:33:33] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.805 second response time [17:41:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [17:44:21] paladox: Reception123 no, it didn't work. I opened a page and got the same message. https://prnt.sc/rm1p7k The strange thing is that sometimes I can delete a page, edit it or view the history but not see it. [17:44:21] [ Screenshot by Lightshot ] - prnt.sc [17:47:33] Could it be due to a caching error? [17:48:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [17:56:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:57:27] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.002 second response time [18:01:41] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.413 second response time [18:02:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:06:11] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:06:30] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:19] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 7.581 second response time [18:10:43] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 6.777 second response time [18:28:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb, 2607:5300:205:200::17f6/cpweb [18:33:31] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb [18:35:47] !log MariaDB [ucroniaswiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (3, 'javascript'); -db7 [18:35:56] !log MariaDB [ucroniaswiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (4, 'css'); -db7 [18:36:04] !log MariaDB [ucroniaswiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (5, 'Scribunto'); -db7 [18:36:10] hispano76 ^ [18:37:08] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw1 [18:38:48] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.762 second response time [18:39:29] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:39:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:42:02] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [18:44:07] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [18:44:54] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:49:20] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 6.490 second response time [18:50:43] PROBLEM - test2 Current Load on test2 is CRITICAL: CRITICAL - load average: 6.09, 16.82, 9.91 [18:51:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:54:34] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:44] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 9.427 second response time [19:23:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [19:26:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:36:17] paladox: Not one wiki, but all three (Hispanowiki, PrivadoWiki and Ucroniaswiki). That error is intermittent [19:37:41] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [19:38:16] ok [19:39:30] paladox: Would that be easier to solve if you reset my wikis again? I haven't been able to add anything important yet because of the above [19:39:49] It may be, Reception123 want to do ^? [19:40:22] Because as you know, since I reset it, several different errors have appeared. [19:40:33] ok [19:45:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvSke [19:45:43] [02miraheze/puppet] 07paladox 03a4ae114 - matomo: Upgrade to 3.13.4 [19:47:25] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:48:07] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:39] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:20] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.841 second response time [19:54:45] hispano76: so you would like them to be deleted and recreated? [19:58:01] PROBLEM - test2 Current Load on test2 is WARNING: WARNING - load average: 0.34, 1.31, 1.77 [19:58:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [20:01:48] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:02:16] RECOVERY - test2 Current Load on test2 is OK: OK - load average: 0.00, 0.55, 1.33 [20:05:36] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 8.251 second response time [20:05:48] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.126 second response time [20:06:35] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.529 second response time [20:06:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:08:19] Reception123: sí, esperando que eso solucioné esos problemas [20:08:19] JohnLewis: I know you can’t comment there but what is the issue on https://phabricator.wikimedia.org/T248411#5996246 [20:08:21] [ ⚓ T248411 Show what is the current tab upon arrival ] - phabricator.wikimedia.org [20:09:51] RhinosF1: go to https://meta.miraheze.org/wiki/Special:ManageWiki/settings - the Anti-Spam tab should be auto selected, it is not apparently [20:09:52] [ Manage this wiki's additional settings - Miraheze Meta ] - meta.miraheze.org [20:10:25] The implementation to my knowledge is identical to how MediaWiki does it, so I don't believe the issue is the extension. If it is, then I don't know what the problem is [20:11:43] JohnLewis: what part of the code forms it [20:12:55] Forms which part? The whole page? [20:13:24] https://github.com/miraheze/ManageWiki/blob/master/includes/specials/SpecialManageWiki.php#L107-L121 [20:13:25] [ ManageWiki/SpecialManageWiki.php at master · miraheze/ManageWiki · GitHub ] - github.com [20:13:40] Thanks JohnLewis, I’ll try and update it [20:15:26] Though it exactly implements MediaWiki code anyway, so either the way I do it is wrong (which needs to said as I copied this implementation exactly from MediaWiki core) or there's something wrong on the code's said [20:16:11] Wouldn't surprise me if implementation has changed and I've just not gone with the flow as it's in some release notes I didn't observe or was told about [20:18:17] JohnLewis: any idea what project tag FormFactory would go under? [20:18:37] It's OOUI HTMLForm [20:19:47] JohnLewis: I think https://phabricator.wikimedia.org/T248411 makes sense [20:19:47] [ ⚓ T248411 FormFactory doesn’t show the default tab as selected on first view ] - phabricator.wikimedia.org [20:20:23] Looks good [20:20:42] I'll double check I'm not just deviating from MeidaWiki quickly now though... [20:21:35] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:24:15] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:24:51] RhinosF1: it looks like its possible there's a bit of deviation... let me update code and re-test [20:25:03] Ok [20:28:04] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.046 second response time [20:29:44] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.734 second response time [20:34:23] https://phabricator.miraheze.org/T5340#101517 Reception123 paladox As a reminder [20:34:23] [ ⚓ T5340 Recreate wikis HispanoWiki, UcroniasWiki and PrivadoWiki ] - phabricator.miraheze.org [20:48:40] JohnLewis: does conflicts work for https://phabricator.wikimedia.org/T177243? [20:48:43] [ ⚓ T177243 Visual editor does not work with Modern Skin ] - phabricator.wikimedia.org [20:49:00] Could we set conflicts for VisualEditor in ManageWiki as Modern? [20:49:29] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:49:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::17f6/cpweb [20:51:37] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 34 failures. Last run 9 minutes ago with 34 failures. Failed resources (up to 3 shown): File[wiki.opendominion.net_private],File[wiki.x1c7.com],File[wiki.x1c7.com_private],File[warmage.necrocity.co.uk] [20:53:47] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.822 second response time [20:54:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:54:45] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [20:55:28] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:58:38] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [20:59:16] RhinosF1: hm? [20:59:21] oh [20:59:29] Uh, it can do [20:59:38] just needs the config [20:59:59] JohnLewis: cool. Can you push it or shall I add it to my to-do list? [21:00:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvSIK [21:00:10] [02miraheze/services] 07MirahezeSSLBot 03002cf05 - BOT: Updating services config for wikis [21:00:41] I'm just about to eat and probably this'll be me done for the day so add to your todo list pleasr [21:00:52] Worst case I end up doing it before you [21:03:10] Added [21:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvSLq [21:25:09] [02miraheze/services] 07MirahezeSSLBot 036343e0d - BOT: Updating services config for wikis [21:38:49] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:39:08] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:33] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:23] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 18656 bytes in 0.753 second response time [21:47:05] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.607 second response time [21:47:50] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.699 second response time [21:56:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [22:00:36] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:00:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:08:36] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [22:22:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [22:27:58] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:28:22] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:29:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [22:30:13] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw3 [22:30:23] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:20] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:33] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:43] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw2 [22:34:50] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:00] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:37:05] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 18694 bytes in 1.045 second response time [22:38:03] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.753 second response time [22:38:19] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.003 second response time [22:40:24] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:42:01] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw4 [22:42:25] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [22:44:22] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 0.518 second response time [22:44:23] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18694 bytes in 0.562 second response time [22:45:00] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw3 mw4 mw6 [22:47:19] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 8.769 second response time [22:47:24] PROBLEM - cp7 Stunnel Http for mw2 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:50] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:50] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:05] RECOVERY - cp7 Stunnel Http for mw2 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.189 second response time [22:54:05] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [22:54:11] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 6.048 second response time [22:54:21] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 6.340 second response time [22:54:42] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [22:55:41] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [22:56:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:57:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:59:30] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.968 second response time [22:59:31] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [23:21:36] curiosity, which is Wikimedia\Rdbms\DBTransactionStateError [23:23:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:27:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:45:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2607:5300:205:200::17f6/cpweb [23:48:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:49:12] PROBLEM - gluster1 Disk Space on gluster1 is WARNING: DISK WARNING - free space: / 75655 MB (10% inode=83%);