[00:09:12] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:11:15] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Mon 03 Jun 2019 08:18:41 PM GMT +0000.
[02:27:08] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40296 MB (10% inode=96%);
[02:53:00] <ZppixBot>	 Hello JohnSmith! If you have any questions feel free to ask and someone should answer soon.
[02:53:42] <JohnSmith>	 hello, anyone here?
[02:53:58] <Voidwalker>	 hi
[02:54:06] <JohnSmith>	 (sorry for annoying)
[02:54:37] <JohnSmith>	 It seems that i didn't get bureaucrat permission
[02:54:42] <JohnSmith>	 on my own wiki
[02:55:07] <JohnSmith>	 https://udcjapan.miraheze.org/
[02:55:08] <ZppixBot>	 Title: [ 後退国、日本 ] - udcjapan.miraheze.org
[02:55:43] <Voidwalker>	 huh, now that's weird
[02:56:16] <JohnSmith>	 as far as i checked other wikis that has created recently, wiki creation requesters become bureaucrats
[02:56:16] <Voidwalker>	 oooohh
[02:56:20] <Voidwalker>	 I know what happened
[02:56:39] <Voidwalker>	 there was a server error on that request, but I thought everything had gone through
[02:56:45] <Voidwalker>	 guess not
[02:56:56] <JohnSmith>	 i see
[02:57:33] <Voidwalker>	 you should be good to go now
[02:57:58] <JohnSmith>	 yeah! thanks. I totally get the permission
[02:58:36] <Voidwalker>	 thanks for letting me know
[03:22:09] <icinga-miraheze>	 PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[03:22:51] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[03:24:08] <icinga-miraheze>	 PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 86872 MB (18% inode=99%);
[03:24:50] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 2896 files, 4.422MB, 2019-02-24 03:58:00 (1.7 weeks ago)
[05:12:23] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:14:23] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Mon 03 Jun 2019 08:18:41 PM GMT +0000.
[05:20:12] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhpjY
[05:20:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03b8381f0 - BOT: Updating services config for wikis
[05:25:10] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhpjG
[05:25:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03945cd3a - BOT: Updating services config for wikis
[05:50:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhpj0
[05:50:14] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03f16a1ec - BOT: Updating services config for wikis
[06:19:24] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:21:24] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Mon 03 Jun 2019 08:18:41 PM GMT +0000.
[06:51:18] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:53:17] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Mon 03 Jun 2019 08:18:41 PM GMT +0000.
[08:31:41] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:33:40] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Mon 03 Jun 2019 08:18:41 PM GMT +0000.
[10:57:18] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhUw
[10:57:19] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07JohnFLewis 03522dddc - fix getUniqueVisits
[10:57:23] <JohnLewis>	 Reception123: ^
[10:59:36] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhhU1
[10:59:38] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 031a331f6 - update MA
[11:32:12] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhTW
[11:32:13] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07JohnFLewis 03c8f4c36 - introduce flat option
[11:32:46] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhhT8
[11:32:47] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 039e95163 - MA
[11:55:10] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhTN
[11:55:12] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 033714994 - BOT: Updating services config for wikis
[11:57:54] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 3482 files, 6.330MB, 2019-03-08 11:57:00 (54.0 seconds ago)
[12:00:31] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fhhTj
[12:00:33] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07JohnFLewis 03447580b - remove uniquevisits
[12:00:57] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhhke
[12:00:59] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 03cbefe71 - MA
[12:09:28] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[12:34:15] <Not-8d6a>	 [02mw-config] 07JohnFLewis commented on pull request 03#2643: configure wgRCLinkDays for T4144 - 13https://git.io/fhhko
[12:34:20] <Not-8d6a>	 [02mw-config] 07JohnFLewis closed pull request 03#2643: configure wgRCLinkDays for T4144 - 13https://git.io/fhxpG
[12:34:22] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhkK
[12:34:23] <Not-8d6a>	 [02miraheze/mw-config] 07The-Voidwalker 03565b936 - configure wgRCLinkDays for T4144 (#2643)
[12:59:03] <Not-8d6a>	 [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fhhIZ
[12:59:04] <Not-8d6a>	 [02miraheze/RottenLinks] 07JohnFLewis 032c6547f - correct time handling
[12:59:28] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhhIc
[12:59:29] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 0336dd5ca - update RL
[13:37:49] <Hispano76>	 hi
[13:41:25] <Zppix>	 .uptime
[13:41:25] <ZppixBot>	 I've been sitting here for 9 days, 17:49:57 and I keep going!
[13:41:31] <Zppix>	 Sweet
[13:57:54] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhLF
[13:57:55] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis 03285c63a - add RL setting to MW
[14:11:44] <Not-8d6a>	 [02ManageWiki] 07Olena23 opened pull request 03#77: fix typo - 13https://git.io/fhhtO
[14:18:00] <Not-8d6a>	 [02ManageWiki] 07paladox closed pull request 03#77: fix typo - 13https://git.io/fhhtO
[14:18:02] <Not-8d6a>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhtW
[14:18:03] <Not-8d6a>	 [02miraheze/ManageWiki] 07Olena23 03386376d - fix typo (#77)
[14:18:05] <Not-8d6a>	 [02ManageWiki] 07paladox commented on pull request 03#77: fix typo - 13https://git.io/fhhtl
[14:20:50] <JohnLewis>	 haha great spot. Shows someone reads code :P should probably move that to i18n at some point in time
[14:22:45] <Zppix>	 JohnLewis:  we're supposed to read code? Damn, i just code with my eyes closed
[14:23:02] <JohnLewis>	 I mean peer review obvs :P
[14:25:23] <Zppix>	 JohnLewis:  I just flip a coin if its heads its fine :P
[14:25:53] <JohnLewis>	 I deploy, if it doesn't bring the site down it's probably okay #Don'tDeployLikeMe
[14:26:30] <Zppix>	 JohnLewis:  but you get a sticker if you break and fix it according to WMF so... i say deploy all teh things
[14:37:53] <Zppix>	 paladox:  if i were to say run a test that has like 98% chance of just flat out denying performance would you be willing to be on standby to fix my mess if needed?
[14:38:36] <paladox>	 huh?
[14:39:25] <JohnLewis>	 best reply ever
[14:39:36] <Zppix>	 paladox:  me and JohnLewis  were talking we were wanting to know what happened if a wiki had less than 2500 pages and LinkTItles enabled, and then it reached over 2500 pages
[14:39:45] <Zppix>	 and so I wanted to test that
[14:39:49] <paladox>	 oh
[14:40:03] <paladox>	 well based on labster comment, i say bad things :P
[14:40:19] <Zppix>	 paladox:  hence is why you would do sysadmin thingys
[14:40:28] <paladox>	 disable it?
[14:40:38] <JohnLewis>	 nah recover :P
[14:40:46] <paladox>	 JohnLewis recover what?
[14:40:47] <paladox>	 :D
[14:40:50] <Zppix>	 paladox:  life
[14:40:53] <Zppix>	 paladox:  everything
[14:40:57] <Zppix>	 the usual
[14:41:04] <paladox>	 hmm, my life is currently very happy atm :P
[14:41:21] <Zppix>	 paladox:  well if the test does what labster says the db wont be saying that
[14:41:38] <paladox>	 oh, you mean the db going down, ok.
[14:42:51] <Zppix>	 JohnLewis:  should i do the test on publictestwiki or should i make a clean slate?
[14:43:23] <paladox>	 oh, you were not joking
[14:43:26] * paladox looks at JohnLewis 
[14:43:28] <JohnLewis>	 if you're importing 2500 pages, I'd say create a wiki and then let us delete it after
[14:43:34] <JohnLewis>	 paladox: hey, science!
[14:43:39] * JohnLewis goes back to ManageWiki
[14:43:52] <Zppix>	 ok
[14:44:02] <paladox>	 JohnLewis well if the db goes down, i know who to blame :D
[14:44:55] <JohnLewis>	 yeah Zppix. He's literally just told you what he's doing :P
[14:45:27] <JohnLewis>	 so paladox is standby... I guess I'm the standby standby
[14:45:52] <Zppix>	 if you two die then i guess im the standby standby standby JohnLewis 
[14:46:32] <Zppix>	 i just created the wiki give me a few mins ill tell you when im ready to actually start
[14:47:55] <paladox>	 JohnLewis standby standby?
[14:55:13] <paladox>	 JohnLewis "He's literally just told you what he's doing :P"?
[14:56:22] <Zppix>	 paladox: the db name is mirahezedisastertestwiki, im ready for you to enable the ext whenever then ill start the import
[14:56:44] <paladox>	 what i great name :D
[14:56:51] <paladox>	 If anything goes wrong blame JohnLewis pls
[14:57:18] <paladox>	 done
[15:02:33] <Zppix>	 paladox:  im exporting from enwiki now ill tell you when i start import
[15:02:41] <paladox>	 ok
[15:02:55] <paladox>	 JohnLewis seems to have disapeared so who's the standby standby now?
[15:03:48] <Zppix>	 i mean paladox  i guess since i was the standby standby standby i am... so lets hope one of my userrights can do something lol
[15:04:28] <paladox>	 lol
[15:04:37] <paladox>	 well if the db goes down
[15:04:46] <paladox>	 there is no way your userrights are gonna work :D
[15:04:54] <Zppix>	 paladox:  i dont think it will go down it should only affect performance according to labster
[15:05:01] <paladox>	 ok
[15:05:21] <Zppix>	 paladox:  in theory even if the db goes down i think the api should work still
[15:05:31] <paladox>	 the api requires the db
[15:07:20] <Zppix>	 paladox:  in case its needed heres the list of what im importing https://pastebin.com/ts2sTG60
[15:07:21] <ZppixBot>	 Title: [ 'Ajam_of_Kuwait User:(CA)Giacobbe/sandbox19 Ababda_people Abaga_Mongols Abag - Pastebin.com ] - pastebin.com
[15:07:28] <paladox>	 ok
[15:07:34] <Zppix>	 paladox:  im starting import now
[15:07:40] <paladox>	 ok
[15:08:39] <Zppix>	 paladox:  it wont let me import lmao its too big
[15:08:44] <paladox>	 lol
[15:08:52] <Zppix>	 paladox:  let me try a file without full rev
[15:08:54] <paladox>	 your gonna have to do it in smaller batches :P
[15:08:55] <Zppix>	 history
[15:09:57] <Zppix>	 paladox:  if trying to not import the whole rev history doesnt work ill try that
[15:10:29] <paladox>	 ok
[15:12:26] <Zppix>	 paladox:  here goes nothing 
[15:12:30] <paladox>	 ok
[15:15:12] <Zppix>	 paladox:  dang this import is taking a while to process :P
[15:15:21] <paladox>	 heh
[15:15:27] <paladox>	 you will hit the timeout likly
[15:15:32] <Zppix>	 paladox:  i imagine thats what 3k pages will do
[15:15:43] <paladox>	 heh
[15:15:45] <Zppix>	 paladox:  i got a 503?
[15:15:57] <paladox>	 a 503 is a backend error
[15:16:10] <Zppix>	 paladox:  i wonder if the import went through
[15:16:31] <Zppix>	 nope
[15:16:36] <Zppix>	 paladox:  ill do it in batches
[15:16:39] <paladox>	 ok
[15:19:56] <Zppix>	 i got a 53 but it imported some things lol
[15:20:39] <Zppix>	 503
[15:21:48] <paladox>	 ok
[15:22:12] <Zppix>	 paladox:  219 pages doing second batch
[15:22:17] <paladox>	 ok
[15:26:12] <Zppix>	 paladox:  whats the timeout time?
[15:26:20] <paladox>	 230 i think
[15:27:47] <Zppix>	 paladox:  im going to try it from apisandbox cause im not conviced that special:Import is going to cut it lol
[15:28:07] <Zppix>	 i guess not api sandbox hates me
[15:28:25] <paladox>	 ok
[15:29:17] <Zppix>	 paladox:  lol -feed is finally reporting my imports 
[15:29:42] <paladox>	 heh
[15:29:52] <Not-8d6a>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhms
[15:29:53] <Not-8d6a>	 [02miraheze/ManageWiki] 07JohnFLewis 035744cf4 - drop $current override, if requires aren't met, it's bad
[15:30:24] <JohnLewis>	 Zppix: ^ there one less potential issue to deal with
[15:30:39] <Zppix>	 JohnLewis:  what'd you od
[15:31:27] <JohnLewis>	 make it so requirements are always checked and if they're not met, the extension is removed automatically
[15:31:54] <Zppix>	 JohnLewis:  (in theory)
[15:32:19] <Zppix>	 JohnLewis:  we've got this far might as well continute testing anyway
[15:32:47] <JohnLewis>	 yeah, I'm just about to start working on increasing requires to allow wiki requirements (like article count)
[15:33:36] <Zppix>	 JohnLewis:  we should also allow certain usergroups access to longer timeout times so i can do imports a lot easier lol
[15:34:30] <JohnLewis>	 if only MediaWiki could relay that to nginx so easily
[15:34:56] <Zppix>	 JohnLewis:  could someone make a system that allows easier ip whitelisting and just whitelist ips
[15:35:05] <Zppix>	 somehow
[15:35:12] <paladox>	 JohnLewis nginx?
[15:35:18] <paladox>	 you must mean varnish :)
[15:35:40] <paladox>	 oh sorry the timeout is set nginx side
[15:35:48] <paladox>	 but it like goes through 2 layers of nginx
[15:36:03] <Zppix>	 halfway to a thousand pages paladox 
[15:36:20] <paladox>	 lol
[15:42:03] <Zppix>	 paladox:  i may have to continue this test later today (around noon my time) or  monday, as im running out of time i have free i didnt expect imports to be so restrictive
[15:42:09] <paladox>	 ok
[15:42:29] <Zppix>	 paladox:  1pm i mean sorry
[15:42:34] <paladox>	 ok
[15:44:29] <Zppix>	 ill be back later\
[16:31:41] <Not-8d6a>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhY2
[16:31:42] <Not-8d6a>	 [02miraheze/ManageWiki] 07JohnFLewis 038dee129 - add article and pages requirements
[16:32:18] <JohnLewis>	 ^^ paladox requires now has "article" => "< 300" and "page" => "< 300"
[16:32:27] <JohnLewis>	 meaning less than 300 pages or articles to be enabled
[16:32:33] <paladox>	 heh
[16:32:44] <paladox>	 well done JohnLewis!
[16:33:19] <paladox>	 JohnLewis why are you using eval()?
[16:33:50] <JohnLewis>	 why not? :P
[16:35:07] <paladox>	 JohnLewis i thought it was recommened not to use eval in prod? :)
[16:36:21] <JohnLewis>	 feel free to code it out if you fancy
[17:45:49] <paladox>	 !log starting gdb on mw1 (to debug segfaults with php)
[17:45:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:10:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhhnx
[21:10:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 0391c497a - BOT: Updating services config for wikis
[21:18:31] <paladox>	 !log starting import on mw2
[21:18:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:57:02] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:06:10] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:08:09] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Mon 03 Jun 2019 08:18:41 PM GMT +0000.