[07:04:51] <snitch>	 [[Tech]]; 220.73.0.228; /* I wanna make Wikipedia about on Chef in Corea.  */ new section; https://meta.wikimedia.org/w/index.php?diff=13413499&oldid=13364942&rcid=6747742
[11:35:48] <snitch>	 [[Tech]]; -revi; Reverted changes by [[Special:Contributions/220.73.0.228|220.73.0.228]] ([[User talk:220.73.0.228|talk]]) to last version by Tbayer (WMF); https://meta.wikimedia.org/w/index.php?diff=13416264&oldid=13413499&rcid=6748178
[18:06:55] <qgil>	 Hi, we are starting a meeting in #wikimedia-office about the Web APIs hub in mediawiki.org. You are invited! See https://phabricator.wikimedia.org/T110108
[21:18:39] <Iridium_>	 Hello
[21:18:48] <Iridium_>	 A few weeks ago, Wikipedia was down worldwide for a bit
[21:19:05] <Iridium_>	 Someone told me that there would be an incident report log on wikipedia, but I can' find it
[21:19:11] <Iridium_>	 If anyone knows, please ping me
[21:19:21] <Iridium_>	 I just wanted to get some info on it
[21:22:27] <Krenair>	 Iridium_, yeah
[21:22:59] <Krenair>	 Iridium_, do you remember which date?
[21:23:36] <Iridium_>	 Let me see
[21:23:38] <Iridium_>	 One moment
[21:23:54] <Bsadowski1>	 Krenair: It was 2 weeks ago I think
[21:24:25] <Krenair>	 There have been multiple things in the past month
[21:24:28] <Krenair>	 Iridium_, https://wikitech.wikimedia.org/wiki/Incident_documentation/20150814-MediaWiki
[21:24:29] <legoktm>	 Iridium_: all of our outage reports are posted at https://wikitech.wikimedia.org/wiki/Incident_documentation unless they're related to a security issue
[21:24:45] <Iridium_>	 Awesome, thanks
[21:24:46] <Iridium_>	 That's the one
[21:25:08] <Iridium_>	 I just kept finding the wikipedia page for incident reporting
[21:25:10] <Iridium_>	 Thank you!
[21:26:21] <Krenair>	 e.g. there was an OCSP issue that caused a lot of users to not be able to access our sites
[21:27:19] <Iridium_>	 Oh, I see
[21:27:43] <Iridium_>	 Well, that's all I was looking for, thanks again
[21:27:58] <Iridium_>	 I'm not quite certain which report is that specific outage, but I'll find it eventually
[21:28:30] <Bsadowski1>	 That was it..
[21:28:30] <Krenair>	 The OCSP one is at https://wikitech.wikimedia.org/wiki/Incident_documentation/20150820-OCSP
[21:28:45] <Krenair>	 The one I think you were asking about based on my logs is https://wikitech.wikimedia.org/wiki/Incident_documentation/20150814-MediaWiki
[21:31:49] <Iridium_>	 August 14th
[21:31:59] <Iridium_>	 Around there
[21:32:17] <Iridium_>	 So I don't think it's the OCSP one
[21:32:43] <marioxcc>	 Hi.
[21:32:44] <marioxcc>	 What approach does the WMF uses to handle leap seconds in its servers?.
[21:34:09] <Iridium_>	 Wow, I've never thought of that for some reason
[21:35:32] <Iridium_>	 Krenair, Yeah, it might've been something a little more serious. I had a log I kept for myself because why not, wikipedia went down
[21:35:46] <marioxcc>	 Iridium_: A log of leap seconds?.
[21:35:48] <Iridium_>	 Log was created on the 14th and there's nothing that day or the day before
[21:36:05] <Iridium_>	 marioxcc, No, how servers would actually deal with leap seconds
[21:36:15] <marioxcc>	 Iridium_: Ok.
[21:36:52] <marioxcc>	 Iridium_: Some ways are described here <https://developerblog.redhat.com/2015/06/01/five-different-ways-handle-leap-seconds-ntp/> for GNU/Linux if you are interested, but I am curious about which way the WMF uses.
[21:37:21] <Krenair>	 marioxcc, I think akosiaris might know about that?
[21:37:49] <Iridium_>	 marioxcc, awesome, thanks
[21:37:57] <marioxcc>	 Iridium_: No problem :)
[21:38:08] <marioxcc>	 Krenair: I do not know about him.
[21:38:28] <Krenair>	 and Moritz, who is not here
[21:38:38] <Krenair>	 I found a thread
[21:38:59] <Krenair>	 marioxcc, https://lists.wikimedia.org/pipermail/wikitech-l/2015-June/082206.html
[21:39:09] <marioxcc>	 Krenair: Thanks. I am taking a look.
[21:44:34] <marioxcc>	 Krenair: It seems to be quite a hack.
[21:45:47] <marioxcc>	 Krenair: I thought they'd do something more systematic, like letting the NTP daemon do its job normally (having tested systems beforehand), or adjusting frequency to lose 1 second (that is detailed in the RedHat web page).
[22:05:51] <[Crow]>	 Hello, is there a way for a crashed bot on Wikimedia Labs to be restarted if the bot owner is MIA?
[22:06:23] <Platonides>	 what kind of bot?
[22:06:36] <[Crow]>	 CorenSearchBot
[22:06:39] <Platonides>	 (#wikimedia-labs would be more appropiate btw)
[22:06:50] <Platonides>	 is that a multimantainer project?
[22:06:52] <[Crow]>	 Aha, I didn't know of that one :)
[22:07:05] <[Crow]>	 No idea
[22:07:10] <Platonides>	 perhaps the owner prefers the bot to die hard in case of errors...
[22:08:04] <[Crow]>	 That may very well be true, though its code has been static for some time.
[22:08:20] <guillom>	 Coren is hardly MIA; probably just on vacation.
[22:09:05] <[Crow]>	 Yes I didn't mean to imply otherwise, just that he doesn't appear to be immediately available to kick it back on :)
[22:09:24] <guillom>	 Fair enough :)
[22:10:50] <guillom>	 https://tools.wmflabs.org/ says Ryan Lane is also a maintainer, but he's even less around these days.
[22:11:48] <[Crow]>	 Hmmm
[22:15:59] <Platonides>	 he was probably affected by some of the recent labs outages
[22:16:42] <Platonides>	 it'd probably be acceptable to restart it... provided that whoever did it knew it
[22:16:58] <Platonides>	 often these bots need arcane parameters...
[22:17:02] <[Crow]>	 I saw the NFS lag issue, I can see that killing a bot
[22:17:06] <greg-g>	 guillom: Ryan is probably more contactable right now than Coren
[22:17:11] <greg-g>	 guillom: you know where to ping him ;0
[22:17:12] <greg-g>	 ;)
[22:17:32] <Platonides>	 or just send an email to both of them
[22:17:48] <guillom>	 greg-g: Oh, right.
[22:17:57] <[Crow]>	 I think Madmanbot is still approved to mirror CSB, though that function is off at the moment.... 
[22:18:47] <[Crow]>	 user page says yes, labs page says no. so... :\
[22:18:55] <[Crow]>	 confuse-a-crow
[22:21:05] <Platonides>	 well, what did you expect from a mad man bot? :)
[22:21:18] <[Crow]>	 :)
[22:21:30] <[Crow]>	 drummer, bummers, and indians in the summer