[00:00:40] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [00:00:50] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [00:01:25] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [00:02:00] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [00:02:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZ5J [00:02:03] [02miraheze/puppet] 07paladox 035f5701a - Use mkdir instead of make [00:04:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZ5k [00:04:12] [02miraheze/puppet] 07paladox 03899b11e - Remove Math texvccheck exec Not needed now that we use mathoid [00:05:39] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [00:05:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZ5t [00:05:49] [02miraheze/mw-config] 07paladox 037989d57 - Set all wiki's into Read Only [00:06:22] Hello mirahezeuser1617! If you have any questions feel free to ask and someone should answer soon. [00:06:41] !log putting all wiki's into read only - pending upgrade to 1.32 [00:06:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:07:04] yeah! :D [00:08:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZ5q [00:08:37] [02miraheze/mw-config] 07paladox 03503d766 - Fix support for mw 1.32 in wgActorTableSchemaMigrationStage [00:09:36] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: upgrading to MediaWiki 1.32 - paladox, last run 7 minutes ago with 0 failures [00:10:15] !log upgrading metawiki to 1.32 [00:11:21] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 434 [00:11:30] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: upgrading to MediaWiki 1.32 - paladox, last run 9 minutes ago with 0 failures [00:14:16] !log upgrading all wiki's to 1.32 [00:14:50] paladox: that won’t work in read only :p [00:15:00] i've removed read only on mw1 [00:15:38] won't work though :P [00:15:58] paladox: ^^ [00:16:05] JohnLewis it's working [00:16:06] though? [00:16:09] paladox: I’m talking about the bot [00:16:13] oh [00:16:16] yeh [00:16:18] i know [00:16:25] but JohnLewis said to log even if it carn't log [00:16:46] I know, but your answer implied it would work :P [00:17:06] i didn't realise he was talking about the bot :) [00:26:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [00:30:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:34:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [00:35:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:8902::f03c:91ff:fe07:444e/cpweb [00:35:19] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 58% [00:36:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:36:33] !log changing default branch for @miraheze/mediawiki to REL1_32 [00:37:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:37:19] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 21% [01:32:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZdf [01:32:34] [02miraheze/puppet] 07paladox 03af2bbce - Update matomo to 3.8.0-rc2 [01:36:22] why using rc grade stuff? [01:36:38] JohnLewis for the 2fa [01:36:45] the plugin didn't support resetting 2fa [01:36:47] mkay [01:37:03] JohnLewis and just my luke 2.8 added support for 2fa with a command to reset :P [01:37:08] *luck [02:20:00] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [02:20:19] !log upgrade loginwiki to 1.32 [02:20:40] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:20:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:24:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZdy [02:24:14] [02miraheze/mw-config] 07paladox 03b8d9bf5 - Disable wgReadOnly if using CLI [02:27:34] !log repooling mw1 and depool mw2 and mw3 [02:27:36] JohnLewis ^^ [02:27:47] kk [02:28:02] try and get them online asap ofc [02:28:34] ok [02:29:09] then to be super efficient, split all.dblist in half :D [02:29:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZdQ [02:29:40] [02miraheze/puppet] 07paladox 0341c385e - Update all mw* to mediawiki 1.32 [02:29:45] JohnLewis huh? [02:30:03] split all.dblist in half or thirds and run update.php on all three servers [02:30:08] (even 4 inc. test1) [02:30:39] hmm [02:31:35] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 46% [02:32:58] paladox: if it helps... find out the size of all.dblist, call it x. then do; split -l (x/4) all.dblist - then foreach loop each server over a different split list [02:33:16] ok [02:34:50] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [02:36:00] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2 [02:36:40] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [02:45:20] I met a MediaWiki internal error on zhdel.miraheze.org. (Sorry for disturbing you if you have already known this issue) [02:45:34] Tiger-: yes we know [02:45:36] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:52] We are currently upgrading MediaWiki [02:46:00] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [02:46:22] wow [02:46:41] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:46:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:47:00] I must have missed some notifications. [02:48:12] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZdp [02:48:14] [02miraheze/mw-config] 07MacFan4000 03bd521d1 - Update Sitenotice.php [02:49:35] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 34% [02:50:04] !log repool mw2 [02:53:35] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 48% [02:58:40] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [02:58:50] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [03:00:00] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3 [03:05:30] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:05:35] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 36% [03:06:00] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [03:06:21] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [03:06:40] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [03:06:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [03:07:55] Hello mirahezeuser5485! If you have any questions feel free to ask and someone should answer soon. [03:08:01] [a9d21c39fe1212e1bf39c5e1] 2019-01-11 03:06:26: Fatal exception of type MWException [03:09:58] [a9d21c39fe1212e1bf39c5e1] 2019-01-11 03:06:26: Fatal exception of type MWException some wikis work correctly and some don't [03:10:04] !log repool mw3 [03:10:04] ups [03:10:08] mirahezeuser5485 hi, known [03:10:19] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3770 [03:10:20] we are upgrading schema for all wiki's [03:10:37] and wiki's that have not been upgraded yet will throw that error (if they have images) [03:10:57] ok [03:13:35] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 45% [03:14:17] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [03:23:35] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 36% [03:38:31] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 43% [03:40:30] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 34% [03:53:07] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 39948 MB (10% inode=96%); [03:55:07] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 43874 MB (11% inode=96%); [03:56:13] MediaWiki internal error. [03:56:13] Original exception: [e082c091f6cf84a1b2db359c] 2019-01-11 03:54:43: Fatal exception of type "MediaWiki\Revision\RevisionAccessException" [03:56:13] Exception caught inside exception handler. [03:56:13] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [03:56:26] paladox: [03:56:41] yup, your wiki has not been upgraded yet judging by that error [03:57:10] hmm, special:recentchanges work [03:58:41] some templates or functions also don't work or don't appear. [04:02:55] Do you think it'll be sorted out by tomorrow? paladox [04:03:01] yup [04:03:23] ok [04:05:06] mirahezeuser5485 which is your wiki? [04:38:49] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhZFS [04:38:51] [02miraheze/mediawiki] 07paladox 03fa9af07 - Update foreground to b52bc08bad8ea51819a1e37feb281bfe762990b7 [04:40:42] Oh, I'm sorry, I didn't see your message. I tell you for private paladox [04:40:48] ok [05:25:07] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40246 MB (10% inode=95%); [05:29:07] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 42053 MB (11% inode=95%); [05:29:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZb0 [05:29:21] [02miraheze/mw-config] 07paladox 034c68b8d - Take wiki's out of ReadOnly [05:35:08] !log [23:29:43] <+paladox> !log depool mw1 [05:35:28] !log [00:06:41] <+paladox> !log putting all wiki's into read only - pending upgrade to 1.32 [05:35:34] !log [00:10:15] <+paladox> !log upgrading metawiki to 1.32 [05:35:39] !log [00:14:16] <+paladox> !log upgrading all wiki's to 1.32 [05:35:56] !log [00:36:33] <+MacFan4000> !log changing default branch for @miraheze/mediawiki to REL1_32 [05:36:04] !log [02:27:34] <+paladox> !log repooling mw1 and depool mw2 and mw3 [05:36:20] !log [02:50:04] <+paladox> !log repool mw2 [05:36:29] !log [03:10:04] <+paladox> !log repool mw3 [05:37:19] !log [23:29:43] <+paladox> !log depool mw1 [05:37:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:37:25] !log [00:06:41] <+paladox> !log putting all wiki's into read only - pending upgrade to 1.32 [05:37:27] !log [00:10:15] <+paladox> !log upgrading metawiki to 1.32 [05:37:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:37:32] !log [00:14:16] <+paladox> !log upgrading all wiki's to 1.32 [05:37:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:37:36] !log [00:36:33] <+MacFan4000> !log changing default branch for @miraheze/mediawiki to REL1_32 [05:37:38] !log [02:27:34] <+paladox> !log repooling mw1 and depool mw2 and mw3 [05:37:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:37:41] !log [02:50:04] <+paladox> !log repool mw2 [05:37:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:37:46] !log [03:10:04] <+paladox> !log repool mw3 [05:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:37:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:39:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZbz [05:39:13] [02miraheze/mw-config] 07paladox 0331b983a - Update Sitenotice.php [05:45:46] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhZb2 [05:45:47] [02miraheze/mediawiki] 07paladox 03cf1121e - Update pivot [05:46:26] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhZba [05:46:27] [02miraheze/mediawiki] 07paladox 03a4a938a - Update math to e7ec2d4a48571d2cc6e9c1ad4c894b6144389eb3 [05:49:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZbV [05:49:46] [02miraheze/puppet] 07paladox 0315fdbe0 - Update whitelist.yaml [05:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZbr [05:55:12] [02miraheze/services] 07MirahezeSSLBot 0347980ff - BOT: Updating services config for wikis [06:05:02] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhZb6 [06:05:04] [02miraheze/ManageWiki] 07paladox 03faae6d8 - perm_wiki -> perm_dbname [06:05:38] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhZbi [06:05:40] [02miraheze/mediawiki] 07paladox 030426151 - Update ManageWiki [06:07:07] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40262 MB (10% inode=95%); [06:19:16] !log adding "editinterface" to default on sysop group [06:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:43:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [06:44:00] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [06:44:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [06:44:26] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [06:44:40] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [06:44:50] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [06:47:43] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 50% [06:48:18] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [06:51:43] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 25% [06:52:00] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [06:52:16] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 5% [06:52:40] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [06:52:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [06:53:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:53:38] !log ran modifyGroupPermission.php to add editsite* to all users in sysop group [06:53:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:54:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:54:23] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [09:02:40] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [09:04:40] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [12:41:44] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 370 [12:43:42] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [14:13:20] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [14:13:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:13:40] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fhnvx [14:13:42] [02miraheze/puppet] 07paladox 031d5bd78 - Revert "Increase redis max memory to 660mb" This reverts commit 7daab66543d3aa6dc315dd4ff440f40b4c014f95. [14:13:43] [02miraheze/puppet] 07paladox 03ae14ce9 - Revert "Update redis.pp" This reverts commit 341ca7ec03636b6bd371f53fd6653f557811f550. [14:13:45] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:14:05] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [14:14:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:14:50] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [14:15:14] !log reboot misc2 to clean ram [14:16:07] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [14:17:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:18:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:18:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [14:19:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [14:19:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:19:43] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [14:21:30] !log [14:15:14] <+paladox> !log reboot misc2 to clean ram [14:21:44] Hello mirahezeuser9084! If you have any questions feel free to ask and someone should answer soon. [14:22:06] !log [14:15:14] <+paladox> !log reboot misc2 to clean ram [14:22:49] A few minutes ago I couldn't log in. I was saying something about an expired session. Why is that? [14:23:23] mirahezeuser9084 hi, please try again. [14:23:34] !log 14:15:14 <+paladox> !log reboot misc2 to clean ram [14:25:12] !log 14:15:14 <+paladox> !log reboot misc2 to clean ram [14:25:12] I'm already connected, but it took me about 5 minutes without being able to do it. paladox [14:25:17] oh [14:25:25] it may have been redis [14:27:04] !log 14:15:14 <+paladox> !log reboot misc2 to clean ram [14:43:16] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhnf8 [14:43:17] [02miraheze/mediawiki] 07Southparkfan 03faac43d - Disable redis caching in diffs memory usage is way too high right now. [14:50:25] !log disable puppet on misc2 to resolve redis trouble [14:52:18] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: redis issues, last run 10 minutes ago with 0 failures [14:54:47] great, redis back online [14:55:20] !log test [14:55:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:55:39] !log 15:50:27 <+SPF|Cloud> !log disable puppet on misc2 to resolve redis trouble [14:55:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:56:27] !log 14:15:14 <+paladox> !log reboot misc2 to clean ram [14:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:57:09] !log deleted all redis keys matching pattern *:diff:* [14:57:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:59:00] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhnf1 [14:59:01] [02miraheze/puppet] 07Southparkfan 03784d8e2 - Fix read-only filesystem issue in redis [14:59:21] great SPF|Cloud! [15:03:19] !log deleted all redis keys matching pattern *:highlight:* [15:03:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:03:42] it almost looks like.. memory usage isn't decreasing no matter how much I'm deleting [15:05:10] !log enabled puppet on misc2 again [15:05:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:05:34] (fix is now in puppet so redis won't fail again, but memory usage still seems to be an issue) [15:06:05] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:39:36] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhnJ5 [15:39:37] [02miraheze/mediawiki] 07Southparkfan 03672269b - DifferenceEngine: fix PHP error retrieving undefined varbiable [16:03:31] Hello giletjaune! If you have any questions feel free to ask and someone should answer soon. [16:31:29] Hello gjpedia! If you have any questions feel free to ask and someone should answer soon. [16:31:31] Some French have created a wiki at https://giletjaune.miraheze.org/ for knowledge about the Yellow vest revolt. (https://en.wikipedia.org/wiki/Yellow_vests_movement) I’d like to ALERT you about the French government, even if folks are just sharing knowledge, it will be fought by the gov, they won’t tolerate it, they declared a TOTAL war on its people. => They will ask for the users’ IP if they don’t already have them. [16:31:32] Title: [ GJpédia ] - giletjaune.miraheze.org [16:34:43] doesn't that depend a lot on what kind of "knowledge" is shared? [16:34:57] so like.. with any other website [16:35:30] => What will be your reaction? Will you handle them the IP logs ? They will say content is illegal even it is not (no pornography, no call for murder, etc. only call for peacefull protest). [16:36:16] We will fight any request for public data in accordance to our privacy policy. [16:36:33] it seems to be knowledge about what happened, who did what, this kind of things [16:37:13] *private [17:18:55] [02mediawiki] 07The-Voidwalker opened pull request 03#117: add arr.png - 13https://git.io/fhnkq [17:19:15] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_32 [+2/-0/±0] 13https://git.io/fhnkm [17:19:17] [02miraheze/mediawiki] 07The-Voidwalker 03b36ee9c - add arr.png forgot to add this to 1.32 earlier [17:19:18] [02miraheze/mediawiki] 07paladox 036893a2d - Merge pull request #117 from The-Voidwalker/REL1_32 add arr.png [17:19:20] [02mediawiki] 07paladox closed pull request 03#117: add arr.png - 13https://git.io/fhnkq [17:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhnkd [17:55:13] [02miraheze/services] 07MirahezeSSLBot 038da4557 - BOT: Updating services config for wikis [17:58:11] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhnkh [17:58:13] [02miraheze/MatomoAnalytics] 07paladox 03d8d1236 - Make matomoScript a static function as it's called statically Per report here https://www.mediawiki.org/wiki/Topic:Ur98bh2957xvcrm2 [17:58:13] Title: [ Error after activating Extension on Extension talk:MatomoAnalytics ] - www.mediawiki.org [18:00:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhnIf [18:00:41] [02miraheze/mediawiki] 07paladox 03b7a189d - Update MatomoAnalytics [19:55:39] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [19:58:05] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:03:36] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:04:23] wasen't me who fixed puppet on mw2 [20:06:19] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:19:14] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhnL5 [20:19:15] [02miraheze/mediawiki] 07paladox 03507a8b5 - Update Tweeki [20:35:27] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_phabricator-extensions] [20:35:38] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [20:43:07] [02mw-config] 07lionelcamara opened pull request 03#2598: updated year to 2019 - 13https://git.io/fhntm [20:43:26] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:44:02] [02mw-config] 07JohnFLewis closed pull request 03#2597: Grant interface-admin to crats - 13https://git.io/fhZ7X [20:44:24] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:44:25] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:44:47] yay GitHub being inconsistent [20:44:52] yup [20:45:30] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 134 failures. Last run 2 minutes ago with 134 failures. Failed resources (up to 3 shown) [20:45:35] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [20:45:40] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 168 failures. Last run 2 minutes ago with 168 failures. Failed resources (up to 3 shown) [20:45:55] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [20:46:13] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [20:46:46] !log restarted puppetserver (semmed stuck on 100% cpu) [20:46:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:47:45] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 107 failures. Last run 3 minutes ago with 107 failures. Failed resources (up to 3 shown): File[adadevelopersacademy.wiki_private],File[wiki.macc.nyc],File[wiki.macc.nyc_private],File[sdiy.info] [20:51:29] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:30] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:31] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:36] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:44] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:55] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:52:13] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:56:06] [02mw-config] 07JohnFLewis closed pull request 03#2598: updated year to 2019 - 13https://git.io/fhntm [20:56:07] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhntn [20:56:09] [02miraheze/mw-config] 07lionelcamara 032da450e - updated year to 2019 (#2598) [21:29:23] [02mw-config] 07lionelcamara opened pull request 03#2599: add IDEAS to ContentNS and NSToBeSearchedDefault - 13https://git.io/fhntP [21:30:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhntM [21:30:39] [02miraheze/puppet] 07paladox 0311ab51f - Use hiera value for icinga php-fpm check [21:34:06] [02mw-config] 07paladox commented on pull request 03#2599: add IDEAS to ContentNS and NSToBeSearchedDefault - 13https://git.io/fhntH [21:37:20] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:37:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhntN [21:37:23] [02miraheze/puppet] 07paladox 03e477f6f - Update check_php_fpm check [21:37:24] [02puppet] 07paladox opened pull request 03#968: Update check_php_fpm check - 13https://git.io/fhntA [21:38:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhntx [21:38:09] [02miraheze/puppet] 07paladox 03790ef57 - Update monitoring.pp [21:38:10] [02puppet] 07paladox synchronize pull request 03#968: Update check_php_fpm check - 13https://git.io/fhntA [21:40:01] PROBLEM - mw1 php7.2-fpm on mw1 is CRITICAL: NRPE: Command 'check_php_fpm_7_2' not defined [21:40:30] [02puppet] 07paladox synchronize pull request 03#968: Update check_php_fpm check - 13https://git.io/fhntA [21:40:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhntj [21:40:33] [02miraheze/puppet] 07paladox 03b20b36a - Update nrpe.cfg.erb [21:41:00] PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: connect to address 172.245.38.205 port 5666: Connection refusedconnect to host 172.245.38.205 port 5666: Connection refused [21:43:00] PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Diff, 148035 files, 7.376GB, 2018-12-16 03:21:00 (3.8 weeks ago) [21:43:08] [02puppet] 07paladox closed pull request 03#968: Update check_php_fpm check - 13https://git.io/fhntA [21:43:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fhnqU [21:43:11] [02miraheze/puppet] 07paladox 03f8452a0 - Update check_php_fpm check (#968) * Update check_php_fpm check * Update monitoring.pp * Update nrpe.cfg.erb [21:43:16] PROBLEM - mw2 php7.2-fpm on mw2 is CRITICAL: NRPE: Command 'check_php_fpm_7_2' not defined [21:44:15] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:44:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:44:55] PROBLEM - test1 php7.2-fpm on test1 is CRITICAL: NRPE: Command 'check_php_fpm_7_2' not defined [21:45:01] PROBLEM - mw3 php7.2-fpm on mw3 is CRITICAL: NRPE: Command 'check_php_fpm_7_2' not defined [21:49:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fhnqO [21:49:46] [02miraheze/puppet] 07paladox 03cce25c1 - mediawiki: Remove old php module support [21:51:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhnq3 [21:51:04] [02miraheze/puppet] 07paladox 03b2312a0 - test1: Switch to php 7.3 [21:51:05] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:51:07] [02puppet] 07paladox opened pull request 03#969: test1: Switch to php 7.3 - 13https://git.io/fhnqs [21:53:27] [02puppet] 07paladox closed pull request 03#969: test1: Switch to php 7.3 - 13https://git.io/fhnqs [21:53:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhnqW [21:53:30] [02miraheze/puppet] 07paladox 0386e96ca - test1: Switch to php 7.3 (#969) [21:53:31] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:53:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:53:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhnql [21:53:58] [02miraheze/puppet] 07paladox 03eca384c - Update test1.yaml [22:05:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhnq6 [22:05:13] [02miraheze/services] 07MirahezeSSLBot 0395d6eae - BOT: Updating services config for wikis [22:05:59] PROBLEM - misc4 Disk Space on misc4 is WARNING: DISK WARNING - free space: / 12759 MB (20% inode=99%); [22:06:28] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40532 MB (11% inode=95%); [22:17:35] !log deleting files on misc4 [22:17:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:19:53] RECOVERY - misc4 Disk Space on misc4 is OK: DISK OK - free space: / 13055 MB (21% inode=99%); [22:27:36] [02miraheze/ManageWiki] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhnqN [22:27:37] [02miraheze/ManageWiki] 07MacFan4000 0307d128f - Expand script [22:29:53] [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhnmf [22:29:54] [02miraheze/mediawiki] 07MacFan4000 03815b42e - Update mw [22:33:46] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40272 MB (10% inode=95%); [22:44:48] How does one remove the ""Jump to: navigation, search"" that's shown up since 1.32? [22:45:42] How does one remove the ""Jump to: navigation, search"" that's shown up since 1.32? [22:46:38] odd, that doesn't turn up on other wikis [22:46:49] and appears to be a preference option that doesn't exist [22:47:07] It's in the "tagline" again [22:47:23] https://concordance.miraheze.org/wiki/MediaWiki:Tagline [22:47:24] Title: [ MediaWiki:Tagline - The Concordance ] - concordance.miraheze.org [22:47:44] you could do [22:47:52] #jump-to-nav { display: none } [22:48:02] where would i put that? [22:48:16] mediawiki:common.css [22:48:23] Thanks! [22:48:57] Carithekitty or MediaWiki:Darkvector.css [22:50:05] I can't eddit common.css anymore, and I'm logged in. [22:50:24] You do not have permission to edit this page, for the following reason: You do not have permission to edit this CSS page because it may affect all visitors. Permissions for editing of sitewide CSS/JS/JSON files were recently separated from the editinterface right. If you do not understand why you are getting this error, see mw:MediaWiki_1.32/interface-admin. [22:52:23] Carithekitty hi, you can grant the editsite* right to your self [22:52:33] through Special:ManageWikiPermissions/sysop [22:52:50] editsitecss [22:52:52] editsitejson [22:52:55] editsitejs [22:53:42] shouldn't we have added this to more users? [22:54:12] uh, I mean, shouldn't we have adjusted the permissions on more wikis? [22:54:41] or did we not because of custom usergroup designs? [22:55:09] We can do it with modifyGroupPermission.php [22:56:39] Putting #jump-to-nav { display: none } in common.css didn't change anythign. [22:56:47] I used that ^^ [22:56:54] all new wiki's should have it [22:57:05] and granted it to sysop (so they can grant it to them selfs) [22:57:29] !I granted it to sysop so they can grant it them selfs" uh, what? [22:58:01] paladox: you only added the rights to sysop on meta [22:58:08] nope [22:58:12] all users [22:58:24] see JohnLewis msg [22:58:34] paladox: ^ [22:58:39] JohnLewis ? [22:58:55] I'm quoting what you put but I don't understand it [22:59:03] *because I don't [22:59:50] ok, well i thought it would grant it to all users, but a my thinking was wrong. '(so they can grant it to them selfs)" was the actual behavour. [23:00:08] I doin't know how to do it that dosen't break existing perms for all wiki's [23:00:11] okay, I'm still confused [23:00:24] nothing you do allows them to add it to themselves [23:00:35]