[00:00:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:00:06] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:00:36] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:01:39] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [00:02:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:03:39] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 5% [00:03:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:04:05] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [03:55:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjcLG [03:55:12] [02miraheze/services] 07MirahezeSSLBot 032b320b7 - BOT: Updating services config for wikis [04:16:36] MacFan4000: I see. But is it hard to request such exemption? [07:51:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:53:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:16:57] Sorry, someone who is wiki creators, can check farmer log, see a more wiki requested thanks :) [10:25:05] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet],Exec[git_pull_services],Exec[git_pull_ssl] [10:25:18] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[git_pull_libphutil],Exec[git_pull_arcanist],Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core] [10:25:22] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [10:25:34] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [10:25:39] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [10:25:58] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [10:26:01] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [11:03:21] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:13:47] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [11:14:02] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:14:37] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:15:22] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [11:23:04] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:23:21] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:23:34] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:24:12] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:25] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:27] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:28] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:29] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:30] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:40] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:48] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:50] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:51] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:04] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:04] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:11] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:15] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:19] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:21] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:21] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:34] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:49] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:51] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:46] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjcWP [15:22:47] [02miraheze/ManageWiki] 07paladox 036752275 - Fix setting a wiki to private [15:23:04] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:23:16] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:23:19] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:23:21] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjcW1 [15:23:21] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [15:23:23] [02miraheze/mediawiki] 07paladox 03ebb4763 - Update MW [15:23:34] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:24:25] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:27] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:29] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:24:30] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:40] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:24:48] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:50] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:51] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:04] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:27:12] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [15:27:49] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:27:51] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [15:29:21] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:42:27] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:18:18] !log ban bch.miraheze.org from varnish - T4350 [16:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:45:13] O_o [16:45:24] paladox: someone being naughty? [16:45:31] huh? [16:45:37] .t 4350 [16:45:38] 2019-05-07 - 16:45:37UTC [16:45:50] .task 4350 [16:45:53] https://phabricator.miraheze.org/T4350 - Cannot set private wiki back to private! [Resolved] authored by MrSimonC, assigned to Paladox [16:46:13] yup, no, the user wanted that wiki to be private [16:46:25] but some how was being made public after he did changes to mw. [16:46:45] Oh im stupid varnish is caching [16:47:14] Ignore me while my brain has a lapse in software knowledge [16:47:20] heh [17:00:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjc46 [17:00:17] [02miraheze/services] 07MirahezeSSLBot 0335ef1ec - BOT: Updating services config for wikis [17:26:04] JohnLewis: did you just delete my access or wiped the Linode account in full? (just wondering) [17:44:49] revi: deleted accounts then cancelled it [17:45:44] JohnLewis: thx [18:34:18] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjc0s [18:34:20] [02miraheze/CreateWiki] 07JohnFLewis 0374668e7 - delete wiki stuff [18:35:35] miraheze/CreateWiki/master/74668e7 - John Lewis The build was broken. https://travis-ci.com/miraheze/CreateWiki/builds/110940801 [18:36:58] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjc0C [18:36:59] [02miraheze/CreateWiki] 07JohnFLewis 032e4e672 - delete wiki stuff [18:38:31] miraheze/CreateWiki/master/2e4e672 - John Lewis The build was fixed. https://travis-ci.com/miraheze/CreateWiki/builds/110941179 [19:58:39] I seem to remember that a short while ago there was mention of the static backup thing being discontinued and that wiki admins can make their own backups. Now, I have been pretty darn sick recently, and I can;t trust anything that I think I remember from around that time. So is this true, or has my illness been messing with me? [20:00:14] And if true, what details should I be aware of? [20:03:13] Fritigern_; correct memory - Special:DataDump on your wiki [20:03:34] * Fritigern_ runs over to gawk at the shiny new stuff [20:06:52] The static dump comes as a tarred xml file for the data and a zip file for the files. Does Special:DataDump dump the files as well? [20:09:37] a file dump can be generated, but this is limited to sysadmins [20:12:22] Poop :( [20:12:44] you can request it [20:14:43] Request it as in "each time I want a file dump I request one"? On static, it is just being generated and I download it using wget and a cron job. [20:16:46] hmm, might be useful to build in a dump scheduler so that folks don't have to request/manually generate a new one repetitively [20:17:40] yeh [20:24:18] I would very much welcome such a scheduler. One of my two wikis contains quite a few images, the other one has the potential for a large amount of images in due time. Having the assurance that an up-to-date dump is within reach without having to distract a sysadmin from other, more important, tasks would be very welcome. [21:03:21] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: MacFan4000, last run 10 minutes ago with 0 failures [21:23:28] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+7/-5/±3] 13https://git.io/fjcg5 [21:23:29] [02miraheze/ManageWiki] 07JohnFLewis 03e014a97 - delete/undelete check box, move classes around + add deletable wiki list [21:34:25] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjc2L [21:34:26] [02miraheze/CreateWiki] 07JohnFLewis 033ae96db - fix db column name [21:39:47] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjc23 [21:39:48] [02miraheze/CreateWiki] 07JohnFLewis 03e38ceb4 - fix remaining [21:46:55] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-1/±2] 13https://git.io/fjc2E [21:46:57] [02miraheze/ManageWiki] 07JohnFLewis 0308c2adb - update with fixes [22:15:28] [02miraheze/mediawiki] 07JohnFLewis pushed 032 commits to 03REL1_32 [+1/-0/±7] 13https://git.io/fjc2h [22:15:29] [02miraheze/mediawiki] 07JohnFLewis 0368c9c64 - update CW/MW [22:15:31] [02miraheze/mediawiki] 07JohnFLewis 03975ca4d - Merge branch 'REL1_32' of www.github.com:miraheze/mediawiki into REL1_32 [22:42:24] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjca4 [22:42:26] [02miraheze/ManageWiki] 07JohnFLewis 030f99567 - add logs [22:43:18] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjcaB [22:43:19] [02miraheze/mediawiki] 07JohnFLewis 0334b5728 - update MW [23:14:55] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjca5 [23:14:57] [02miraheze/ManageWiki] 07JohnFLewis 03ac4a896 - logID needed for publish [23:15:54] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjcad [23:15:55] [02miraheze/mediawiki] 07JohnFLewis 035e36f2e - update MW [23:18:48] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjcaF [23:18:50] [02miraheze/ManageWiki] 07JohnFLewis 0323dab09 - fix variable calls [23:19:33] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjcaN [23:19:35] [02miraheze/mediawiki] 07JohnFLewis 0368b44e4 - update MW [23:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjcaj [23:25:12] [02miraheze/services] 07MirahezeSSLBot 03045090e - BOT: Updating services config for wikis [23:26:43] and with the many many commits, come https://meta.miraheze.org/wiki/Special:DeletedWikis and the ability to delete wikis via Special:ManageWiki [23:26:44] [ Deleted Wikis - Miraheze Meta ] - meta.miraheze.org [23:27:05] ohh [23:27:08] wow JohnLewis!! [23:40:53] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjcVY [23:40:54] [02miraheze/mw-config] 07JohnFLewis 030c6a4d8 - poll and pollny conflict [23:40:59] massively undetected issue there