[00:00:09] <icinga-miraheze>	 PROBLEM - espiral.org - Gandi on sslhost is CRITICAL: CRITICAL - Certificate 'espiral.org' expires in 7 day(s) (Fri 23 Aug 2019 11:59:59 PM GMT +0000).
[02:02:55] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%);
[02:40:48] <ZppixBot>	 Hello tom32! If you have any questions feel free to ask and someone should answer soon.
[02:42:47] <tom32>	 about four or five months ago I was working on two miraheze wikis - and was very sick for a few months and lost the wkis due to inaction lack of editing, and I woulkd like to find out the best way of seeing if I can re-start them again if at all possible, and who the best person(s) to contact are
[02:46:38] <tom32>	 it would be great if someone knew the bestway to re-start or restore
[03:16:30] <tom32>	 have managed to re-enter and request adoptions internally, sorry to have bothered
[05:16:19] <Not-f817>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjdge
[05:16:21] <Not-f817>	 [02miraheze/ssl] 07Reception123 03da3730f - add wiki.opendominion.net cert
[06:26:46] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3292 MB (13% inode=94%);
[10:52:31] <Reception123>	 !log UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.opendominion.net "', '"wgServer":"https:\/\/wiki.opendominion.net"') WHERE wiki_dbname = 'opendominionwiki';
[10:52:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[11:10:45] <Reception123>	 !log used update set to perform same replace command as above on opendominionwiki 
[11:10:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[12:10:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjdrE
[12:10:13] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03616b1f2 - BOT: Updating services config for wikis
[16:40:53] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.94, 5.05, 3.11
[16:41:39] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.74, 6.84, 5.59
[16:42:53] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 2.86, 4.06, 2.97
[16:43:39] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.02, 6.18, 5.50
[16:46:12] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 22240 MB (6% inode=96%);
[17:02:21] <Not-f817>	 [02mediawiki] 07RhinosF1 opened pull request 03#120: Adding MyVariables (T4610) - 13https://git.io/fjdPF
[17:02:33] <RhinosF1>	 ^ Reception123: did it
[17:03:14] <Reception123>	 RhinosF1: I'll have a look. In the meantime you can do the mw-config part :)
[17:03:22] <RhinosF1>	 Reception123: doing
[17:03:52] <Reception123>	 RhinosF1: https://github.com/miraheze/mediawiki/pull/120/files#diff-8903239df476d7401cf9e76af0252622R1164 this should just be extensions/MyVariables so you'll have to redo :( (since you can't edit it directly, it won't work)
[17:03:52] <ZppixBot>	 [ Adding MyVariables (T4610) by RhinosF1 · Pull Request #120 · miraheze/mediawiki · GitHub ] - github.com
[17:11:47] <Not-f817>	 [02mediawiki] 07RhinosF1 closed pull request 03#120: Adding MyVariables (T4610) - 13https://git.io/fjdPF
[17:16:18] <Not-f817>	 [02mw-config] 07RhinosF1 opened pull request 03#2730: add wmgUseMyVariables - 13https://git.io/fjdPj
[17:17:01] <RhinosF1>	 ^ ignore that for now
[17:17:33] <Reception123>	 ok
[17:33:00] <Not-f817>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2730: add wmgUseMyVariables - 13https://git.io/fjdPj
[17:38:13] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21979 MB (5% inode=96%);
[17:39:41] <Not-f817>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2730: add wmgUseMyVariables - 13https://git.io/fjdPj
[17:42:45] <Not-f817>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2730: add wmgUseMyVariables - 13https://git.io/fjdPj
[17:43:29] <Not-f817>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2730: add wmgUseMyVariables - 13https://git.io/fjdPj
[17:43:55] <Not-f817>	 [02mw-config] 07RhinosF1 edited pull request 03#2730: add MyVariables (T4610) - 13https://git.io/fjdPj
[17:44:05] <RhinosF1>	 JohnLewis, Reception123: ^
[17:44:18] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --no-local-users --username-prefix "오사위키" --wiki osaindexwiki /home/reception/dump_original.xml 
[17:44:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:45:42] <Reception123>	 RhinosF1: LGTM waiting on travis
[17:45:58] <RhinosF1>	 Reception123: yeah cool, let me see what it's doing
[17:46:17] <RhinosF1>	 Reception123: all green
[17:47:00] <Not-f817>	 [02mw-config] 07Reception123 closed pull request 03#2730: add MyVariables (T4610) - 13https://git.io/fjdPj
[17:47:02] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fjdXu
[17:47:03] <Not-f817>	 [02miraheze/mw-config] 07RhinosF1 0348e8f8d - add MyVariables (T4610) (#2730) * add wmgUseMyVariables * Update ManageWikiExtensions.php * make restricted * add MyVariables * Update extension-list
[17:48:28] <RhinosF1>	 Reception123: if you enable it then i'll give it a quick test
[17:48:57] <Reception123>	 RhinosF1: Ok, thanks. First I need to do MessageFiles and LC cache
[17:49:30] <RhinosF1>	 Reception123: okay, apparently, I can't create pages on test1 so might need rights or something. is it locked down?
[17:49:36] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki
[17:49:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:49:52] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki loginwiki on mw*
[17:49:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:49:59] <RhinosF1>	 no im just logged out for some reason
[17:49:59] <Reception123>	 RhinosF1: probably is
[17:50:05] <Reception123>	 RhinosF1: I'll check
[17:50:05] <Reception123>	 ok
[17:51:11] <Reception123>	 RhinosF1: even so, * has edit it seems
[17:51:16] <RhinosF1>	 Reception123: works now i'm logged in
[17:51:29] <RhinosF1>	 email confirmed in MW or something?
[17:52:09] <RhinosF1>	 Reception123: now saying sent to moderation
[17:52:29] <Reception123>	 RhinosF1: oh, it's because every extension (almost) is enabled on test1... meh
[17:52:42] <Reception123>	 for now I don't get why MyVariables is not in MW
[17:52:52] <Reception123>	 ok was just added
[17:52:56] <RhinosF1>	 Reception123: puppet?
[17:53:25] <RhinosF1>	 Reception123: [53ecdeed19f1b09a5ad0199b] 2019-08-16 17:53:08: Fatal exception of type MWException on show preview
[17:53:29] <Reception123>	 yup just saw that
[17:53:32] <Reception123>	 checking logs
[17:53:48] <RhinosF1>	 Reception123: whole of test1 is down
[17:54:07] <RhinosF1>	 everything else is up luckily
[17:54:45] <Reception123>	 well yes, it's only enabled on test1 :)
[17:55:11] <Reception123>	 2019-08-16 17:53:31 test1 test1wiki: [407421997f66ee693e2c0467] /w/index.php?title=TestMyVariables&action=edit   MWException from line 187 of /srv/mediawiki/w/includes/MagicWord.php: Error: invalid magic word 'MAG_CURRENTUSER'
[17:56:33] <paladox>	 Are you sure extension.json exists for that extension?
[17:56:48] <Reception123>	 paladox: yes, checked
[17:56:54] <RhinosF1>	 paladox: er, yeah
[17:56:55] <Reception123>	 paladox: https://github.com/wikimedia/mediawiki-extensions-MyVariables/blob/master/extension.json
[17:56:56] <ZppixBot>	 [ mediawiki-extensions-MyVariables/extension.json at master · wikimedia/mediawiki-extensions-MyVariables · GitHub ] - github.com
[17:57:03] <JohnLewis>	 if it didn't, you'd get a fatal not an exception
[17:59:40] <paladox>	 Ok
[18:04:28] <Reception123>	 RhinosF1: as usual, my fault 
[18:04:29] <Reception123>	 fixed
[18:04:33] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on test1
[18:04:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:14:39] <Not-f817>	 [02mw-config] 07RhinosF1 opened pull request 03#2731: move MyVariables to prod - 13https://git.io/fjdXF
[18:15:02] <Not-f817>	 [02mw-config] 07RhinosF1 commented on pull request 03#2731: move MyVariables to prod - 13https://git.io/fjdXb
[18:15:29] <RhinosF1>	 Reception123: ^
[18:16:04] <Not-f817>	 [02mw-config] 07Reception123 closed pull request 03#2731: move MyVariables to prod - 13https://git.io/fjdXF
[18:16:06] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjdXN
[18:16:07] <Not-f817>	 [02miraheze/mw-config] 07RhinosF1 03a79e6e4 - move MyVariables to prod (#2731)
[19:42:10] <Reception123>	 !log deleted 0h4rkwiki; 2019frst352wiki;  3mwiki; aceteccwiki
[19:42:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:42:50] <Reception123>	 !log DROPPED DATABASE for 0h4rkwiki; 2019frst352wiki;  3mwiki; aceteccwiki
[19:42:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:57:41] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.47, 7.74, 6.10
[20:59:40] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.51, 7.35, 6.14
[21:05:39] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.03, 6.66, 6.27
[22:08:17] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjdyr
[22:08:19] <Not-f817>	 [02miraheze/puppet] 07paladox 03f6a23da - Matomo: Add cron back but adjust time to every 6 hours
[22:09:06] <RhinosF1>	 paladox: can you get a status on an import for me?
[22:09:36] <paladox>	 I’m currently mobile
[22:09:45] <paladox>	 JohnLewis: can you look please^ :)
[22:09:48] <RhinosF1>	 paladox: so no
[22:10:01] <JohnLewis>	 probably but I need a bit more than just that
[22:10:27] <RhinosF1>	 JohnLewis: T4651 and see our channel as well pls
[22:11:14] <JohnLewis>	 Still need more info, all I get is Reception123 started it
[22:11:28] <JohnLewis>	 where and how usually helps when trying to try status updates :)
[22:11:40] <RhinosF1>	 cool JohnLewis - the wiki owner has said it's been stuck for a bit now - 4hrs would target when we broke test1
[22:12:12] <JohnLewis>	 so it's being ran on test1 then?
[22:13:28] <RhinosF1>	 JohnLewis: not on SAL so maybe not - that was a guess although I think Reception123 sometimes changes to test1 cause of https://phabricator.miraheze.org/T4638
[22:13:29] <ZppixBot>	 [ ⚓ T4638 Job Queue Fatal Exceptions ] - phabricator.miraheze.org
[22:16:30] <JohnLewis>	 as far as I can see it's running?
[22:16:48] <RhinosF1>	 JohnLewis: i'll let them know but they say nothing has happened in 4hrs
[22:17:10] <JohnLewis>	 only 2 screens on test1, both are running
[22:17:25] <RhinosF1>	 JohnLewis: okay, thx!
[22:19:19] <JohnLewis>	 yep only screens across the cluster are on test1 so if it's not on test1 then I have no clue where it is and what he's doing :)
[22:19:44] <RhinosF1>	 JohnLewis: okay, super and I think meta just woke up
[23:37:08] <Hispano76_>	 hi