[00:14:17] <wikibugs>	 06Operations, 10IRCecho: Make ircecho much better - https://phabricator.wikimedia.org/T95052#2467604 (10Dzahn)
[00:14:19] <wikibugs>	 06Operations, 10IRCecho: Make ircecho run as its own user - https://phabricator.wikimedia.org/T76203#2467603 (10Dzahn)
[00:14:43] <wikibugs>	 06Operations, 10IRCecho: Make ircecho run as its own user - https://phabricator.wikimedia.org/T76203#792998 (10Dzahn)
[00:14:45] <wikibugs>	 06Operations, 10IRCecho: Make ircecho much better - https://phabricator.wikimedia.org/T95052#1179271 (10Dzahn)
[00:40:28] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: puppet fail
[01:07:04] <icinga-wm>	 RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:11:14] <icinga-wm>	 PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 1 failures
[01:30:17] <wikibugs>	 06Operations, 10Traffic: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2467654 (10Krinkle)
[01:37:20] <icinga-wm>	 RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:21:17] <logmsgbot>	 !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 22s)
[02:21:23] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:27:01] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 16 02:27:00 UTC 2016 (duration 5m 44s)
[02:27:05] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:10:05] <wikibugs>	 06Operations, 06Performance-Team, 10scap, 07Epic: During deployment old servers may populate new cache URIs - https://phabricator.wikimedia.org/T47877#2467788 (10Danny_B)
[03:33:10] <wikibugs>	 06Operations, 10Traffic, 07HTTPS, 07Tracking: SSL related (tracking) - https://phabricator.wikimedia.org/T29946#2467808 (10Danny_B)
[03:34:45] <wikibugs>	 06Operations, 10Wikimedia-General-or-Unknown, 07Documentation: Add a wiki on wikitech is out of date, incomplete - https://phabricator.wikimedia.org/T87588#2467816 (10Danny_B)
[04:56:33] <icinga-wm>	 PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: puppet fail
[05:23:43] <icinga-wm>	 RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:22:03] <icinga-wm>	 PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:29:53] <icinga-wm>	 PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:30:32] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:32:16] <icinga-wm>	 PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:23] <icinga-wm>	 PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:34:02] <icinga-wm>	 PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:48:03] <icinga-wm>	 RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:56:53] <icinga-wm>	 RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[06:57:23] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:33] <icinga-wm>	 RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:34] <icinga-wm>	 RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:33] <icinga-wm>	 RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:56:24] <icinga-wm>	 PROBLEM - All Flannel etcd nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/flannel - 185 bytes in 1.926 second response time
[08:09:51] <icinga-wm>	 PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures
[08:23:54] <wikibugs>	 06Operations, 06Release-Engineering-Team, 05Gitblit-Deprecate: Clones from git.wikimedia.org are not redirected - https://phabricator.wikimedia.org/T139206#2422729 (10mmodell) IMO this should redirect to diffusion using the existing mappings that we already have set up in {rPHEX}
[08:36:52] <icinga-wm>	 RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:53:37] <apergos>	 I'll be out of reach for a few hours (travelling), then back on irc 
[09:10:16] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468140 (10Peachey88)
[09:12:17] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2456553 (10Peachey88) 05Resolved>03Open Reopening per comment in {T140532} from @G...
[09:19:03] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[09:23:03] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb1002 is OK: All endpoints are healthy
[09:39:13] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468206 (10TTO) I don't have import rights on enwiki, so I wonder if @Legoktm has time...
[10:21:42] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[10:22:53] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[10:23:41] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb1001 is OK: All endpoints are healthy
[10:28:51] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb1002 is OK: All endpoints are healthy
[10:37:10] <Urbanecm>	 Hi, is it possible to turn off Compact language links feature for one user at all WMF wikis at once? I hope there is another solution than changing the settings at 750+ wikis manually... Maybe with direct database changing?
[10:38:16] <p858snake>	 is there someone that really edits all 750+ wikis?
[10:40:00] <Urbanecm>	 p858snake, see https://phabricator.wikimedia.org/T138524#2468233 for the reason why I'm asking. 
[10:41:43] <Nemo_bis>	 global.js can probably help
[10:44:00] <Urbanecm>	 I have no knowledge about JavaScript. Could you create something which will help? Or point me to some person who can help?
[10:54:22] <Nikerabbit>	 I am told there is a script that can go over all wikis with your account and update the preference, but I have not seen it
[10:54:32] <Nikerabbit>	 Nemo_bis: what is this global.js?
[10:55:46] <Nemo_bis>	 it'swhat you just said
[10:56:18] <Nikerabbit>	 do you know where it is?
[10:56:19] <Nemo_bis>	 https://www.mediawiki.org/wiki/Help:Extension:GlobalCssJs
[10:58:03] <Nikerabbit>	 Urbanecm: https://www.mediawiki.org/wiki/Help:Extension:GlobalCssJs#Example:_Set_a_global_interface_language this is almost 100% what you need, you just need to change the preference name and value
[10:58:47] <Nikerabbit>	 (I might note however that the script has a bug, it uses mw.notify without making sure it is loaded first)
[10:59:10] <Nikerabbit>	 oh no it doesn't, just weird
[11:00:40] <Urbanecm>	 Nikerabbit, what is the preference name of compact links feature? ext-uls-compact-language-links-preference? Thanks for the link. 
[11:00:57] <Nikerabbit>	 Urbanecm: let me copy-paste
[11:02:02] <Nikerabbit>	 compact-language-links
[11:02:08] <Urbanecm>	 Thanks Nikerabbit 
[11:02:09] <Nikerabbit>	 default value 1
[11:02:22] <Urbanecm>	 So we want to change it to 0. 
[11:02:29] <Nikerabbit>	 y
[11:03:10] <Urbanecm>	 Thanks
[11:05:11] <Urbanecm>	 Nikerabbit, so https://cs.wikipedia.org/wiki/Wikipedista:Martin_Urbanec/Random_notes/Compact_language_links_turn_off.js will work?
[11:07:47] <Nikerabbit>	 Urbanecm: I tried it, it worked
[11:09:17] <Urbanecm>	 Nikerabbit: Thanks. So when Kusurija will include it to global.js it will check if the feature is turned off and if not it'll disable it? Do you know how to run this script at every WMF wiki?
[11:10:13] <Nikerabbit>	 Urbanecm: as detailed in https://www.mediawiki.org/wiki/Help:Extension:GlobalCssJs
[11:11:57] <Urbanecm>	 Nikerabbit, this isn't what I want. I want a way how to force run this JS at every wiki right now so the feature will be turned off at every WMF wiki where it's available. 
[11:12:33] <Nikerabbit>	 Urbanecm: it will be turned off on first visit to that wiki
[11:14:12] <Nemo_bis>	 Urbanecm: that wouldn't work for future wikis
[11:14:55] <Nemo_bis>	 there are multiple scripts that alter preferences on all wikis but some were hidden; maybe https://tools.wmflabs.org/globalprefs/ is one of them
[11:20:03] <MatmaRex>	 Urbanecm: you could just visit every wiki once ;)
[11:24:53] * Nemo_bis uses "open all in tabs" on Firefox
[11:25:34] <Nemo_bis>	 http://blog.mattes-groeger.de/chrome-open-links/ is decent too
[11:45:49] <Urbanecm>	 Thanks. 
[12:12:03] <wikibugs>	 06Operations, 06Commons: Please fix broken thumbnails - https://phabricator.wikimedia.org/T140536#2468277 (10DaBPunkt)
[12:45:32] <wikibugs>	 06Operations, 06Commons: Please fix broken thumbnails - https://phabricator.wikimedia.org/T140536#2468318 (10DaBPunkt) Forgot to mention: I tried to purge the pictures already.
[12:56:12] <icinga-wm>	 PROBLEM - puppet last run on maerlant is CRITICAL: CRITICAL: puppet fail
[13:19:05] <Bsadowski1>	 Anybody here?
[13:19:14] <Bsadowski1>	 Just got a database query error
[13:19:39] <Bsadowski1>	 "A database query error has occurred"
[13:19:46] <Bsadowski1>	 "This may indicate a bug in the software."
[13:19:49] <Bsadowski1>	 Hmm
[13:20:03] <Bsadowski1>	 "Function: IndexPager::buildQueryInfo (contributions page unfiltered)"
[13:20:18] <Bsadowski1>	 "Error: 2013 Lost connection to MySQL server during query"
[13:21:49] <Bsadowski1>	 On English Wikipedia, by the way.
[13:23:35] <Bsadowski1>	 Oh, well, it fixed itself.
[13:23:52] <icinga-wm>	 RECOVERY - puppet last run on maerlant is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:43:47] <MatmaRex>	 hi Bsadowski1
[13:44:34] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [1000.0]
[13:44:50] <MatmaRex>	 Bsadowski1: that indicates a very slow SQL query. what is the exact URL you were trying to view?
[13:45:43] <Bsadowski1>	 Oh hello.
[13:45:49] <Bsadowski1>	 Hold on. :)
[13:46:08] <Bsadowski1>	 https://en.wikipedia.org/w/index.php?title=Special:Contributions&dir=prev&contribs=newbie&target=newbies
[13:46:31] <Bsadowski1>	 Old contributions from the past 30 days.
[13:46:36] <Bsadowski1>	 (basically)
[13:47:44] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [1000.0]
[13:48:51] <Bsadowski1>	 _O
[13:49:05] <Bsadowski1>	 Another SQL query error, MatmaRex.
[13:49:26] <Bsadowski1>	 Oh and there is a server IP next to it
[13:59:33] <icinga-wm>	 RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[14:00:22] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[14:06:37] <MatmaRex>	 Bsadowski1: hmm, interesting. are you intentionally viewing them from the oldest edits first? (dir=prev) in the other order it's fast
[14:10:45] <Bsadowski1>	 No.
[14:10:49] <Bsadowski1>	 I click on prev
[14:16:38] <icinga-wm>	 PROBLEM - puppet last run on mw2164 is CRITICAL: CRITICAL: puppet fail
[14:17:25] <MatmaRex>	 Bsadowski1: i think the problem is that the further in the past you look, the fewer edits there are by users who are still "newbies" today, and the database has to scan more rows to find the 50 (or whatever number) edits you requested
[14:18:14] <MatmaRex>	 Bsadowski1: i'll file a task about this, perhaps there is some way to make this better. but it's possible that there isn't, in which case we'd probably have to put some limit on how far in the past you can look for newbie contribs
[14:24:59] <icinga-wm>	 PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: puppet fail
[14:37:05] <wikibugs>	 06Operations: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2468345 (10matmarex)
[14:43:59] <icinga-wm>	 RECOVERY - puppet last run on mw2164 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[14:46:34] <wikibugs>	 06Operations: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2468357 (10matmarex) The slow queries are: ``` SELECT rev_id,rev_page,rev_text_id,rev_timestamp,rev_comment,rev_user_text,rev_user,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,...
[14:52:28] <icinga-wm>	 RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[14:55:35] <wikibugs>	 06Operations: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2468358 (10matmarex) The query plans look about the same… I think the problem is that the further in the past you look, the fewer edits there are by users who are still "newbies" t...
[15:30:34] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [1000.0]
[15:32:44] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [1000.0]
[15:35:53] <icinga-wm>	 PROBLEM - puppet last run on cp2009 is CRITICAL: CRITICAL: Puppet has 1 failures
[15:39:23] <icinga-wm>	 RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[15:39:23] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[15:45:32] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[15:47:23] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb1002 is OK: All endpoints are healthy
[16:01:23] <icinga-wm>	 RECOVERY - puppet last run on cp2009 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[18:21:37] <wikibugs>	 06Operations, 07Performance: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2468441 (10Aklapper)
[18:22:20] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:24:19] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb1002 is OK: All endpoints are healthy
[19:15:08] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468491 (10Legoktm) nostalgiawiki is still protcol-relative on https://meta.wikimedia....
[19:24:32] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468493 (10Legoktm) (Also I'm afk most of today so if someone else wants to take the l...
[19:26:40] <icinga-wm>	 RECOVERY - MegaRAID on dbstore1002 is OK: OK: optimal, 1 logical, 2 physical
[19:29:19] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468495 (10Dereckson) 05Open>03Resolved Okay, I'm going to find someone with the c...
[19:29:29] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468498 (10Dereckson) 05Resolved>03Open a:05TTO>03Dereckson
[19:35:24] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[19:38:23] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[19:39:23] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:41:14] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb1001 is OK: All endpoints are healthy
[19:42:54] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[19:44:12] <icinga-wm>	 RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[19:59:57] <grrrit-wm>	 (03PS1) 10Dereckson: Update interwiki map [mediawiki-config] - 10https://gerrit.wikimedia.org/r/299299 (https://phabricator.wikimedia.org/T140206) 
[20:08:24] <grrrit-wm>	 (03CR) 1020after4: [C: 031] phabricator: re-enable community metrics mail [puppet] - 10https://gerrit.wikimedia.org/r/299093 (https://phabricator.wikimedia.org/T139950) (owner: 10Dzahn)
[20:08:54] <icinga-wm>	 PROBLEM - puppet last run on mw2108 is CRITICAL: CRITICAL: Puppet has 1 failures
[20:09:19] <wikibugs>	 06Operations, 10Phabricator, 13Patch-For-Review: Phabricator weekly report not generated (or at least sent) - https://phabricator.wikimedia.org/T139950#2468536 (10mmodell) The weekly stats should have been sent when I ran the script manually.
[20:15:20] <Dereckson>	 twentyafterfour: hi, could you deploy https://gerrit.wikimedia.org/r/#/c/299299 for https://phabricator.wikimedia.org/T140206#2468491?
[20:34:12] <icinga-wm>	 RECOVERY - puppet last run on mw2108 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[20:36:35] <twentyafterfour>	 Dereckson: is it ready to go? Just merge and deploy now?
[20:36:44] <Dereckson>	 twentyafterfour: yup
[20:36:55] <twentyafterfour>	 I can't stick around too long to babysit but I can deploy if you can test
[20:37:03] <Dereckson>	 I can test it.
[20:37:12] <twentyafterfour>	 moving to #wikimedia-operations
[20:37:21] <twentyafterfour>	 oh wait this is operations
[20:37:22] <twentyafterfour>	 lol
[20:37:30] <twentyafterfour>	 I thought I was in a different channel. ok
[20:37:35] <Dereckson>	 :)
[20:38:17] <Dereckson>	 This file controls our interwiki map. It controls the links "[[nostalgia:Foo]]" to redirect them to nostalgia.wiki..../wiki/Foo
[20:38:56] <grrrit-wm>	 (03CR) 1020after4: [C: 032] Update interwiki map [mediawiki-config] - 10https://gerrit.wikimedia.org/r/299299 (https://phabricator.wikimedia.org/T140206) (owner: 10Dereckson)
[20:38:58] <Dereckson>	 Test is straightforward: http://fr.wikipedia.org/wiki/nostalgia:foo must still go there
[20:39:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: Update interwiki map [mediawiki-config] - 10https://gerrit.wikimedia.org/r/299299 (https://phabricator.wikimedia.org/T140206) (owner: 10Dereckson)
[20:41:15] <twentyafterfour>	 !log deploying interwiki https config change https://gerrit.wikimedia.org/r/#/c/299299 refs T140206
[20:41:16] <stashbot>	 T140206: Transwiki import not working in production - https://phabricator.wikimedia.org/T140206
[20:41:19] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:42:24] <logmsgbot>	 !log twentyafterfour@tin Synchronized wmf-config/interwiki.php: (no message) (duration: 00m 26s)
[20:42:28] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:42:34] <Dereckson>	 Testing.
[20:43:06] <twentyafterfour>	 Thanks
[20:43:08] <Dereckson>	 Works fine.
[20:43:11] <twentyafterfour>	 Cool
[20:43:16] <Dereckson>	 Thanks for the swift deployment.
[20:43:47] <twentyafterfour>	 no problem!  :)  I'll be around for a few minutes if you need anything else. Then I'm AFK for the rest of the day.
[20:47:30] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468555 (10Dereckson) 05Open>03Resolved New interwiki map is live.
[20:47:56] <Dereckson>	 No, I'm fine. Have a nice Saturday.
[21:28:52] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb2001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[21:32:53] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb2001 is OK: All endpoints are healthy
[21:36:00] <wikibugs>	 06Operations: Deploy a PHP and HHVM patch (Exif values retrieved incorrectly if they appear before IFD) - https://phabricator.wikimedia.org/T140419#2468580 (10matmarex)
[21:41:59] <wikibugs>	 06Operations: Deploy a PHP and HHVM patch (Exif values retrieved incorrectly if they appear before IFD) - https://phabricator.wikimedia.org/T140419#2468586 (10matmarex)
[21:42:01] <wikibugs>	 06Operations: Deploy a PHP and HHVM patch (Exif values retrieved incorrectly if they appear before IFD) - https://phabricator.wikimedia.org/T140419#2464014 (10matmarex)
[21:53:33] <icinga-wm>	 PROBLEM - All k8s etcd nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/k8s - 185 bytes in 1.704 second response time
[22:26:33] <icinga-wm>	 PROBLEM - IPv4 ping to eqiad on ripe-atlas-eqiad is CRITICAL: CRITICAL - failed 26 probes of 403 (alerts on 19) - https://atlas.ripe.net/measurements/1790945/#!map
[22:32:23] <icinga-wm>	 RECOVERY - IPv4 ping to eqiad on ripe-atlas-eqiad is OK: OK - failed 1 probes of 403 (alerts on 19) - https://atlas.ripe.net/measurements/1790945/#!map
[23:05:22] <wikibugs>	 06Operations, 10Phabricator, 13Patch-For-Review: Phabricator weekly report not generated (or at least sent) - https://phabricator.wikimedia.org/T139950#2468688 (10Danny_B) @mmodell  Obviously haven't as https://lists.wikimedia.org/mailman/private/phabricator-reports/2016-July/thread.html shows.
[23:10:33] <wikibugs>	 06Operations, 10Phabricator, 13Patch-For-Review: Phabricator weekly report not generated (or at least sent) - https://phabricator.wikimedia.org/T139950#2447582 (10Luke081515) I think the archive of that list is private? Is only the archive and the access privat, or is the "can write a mail to the list" acces...
[23:10:44] <Luke081515>	 meh, wikibugs is a bit slow, actually
[23:10:51] <Luke081515>	 but it works
[23:11:28] <Luke081515>	 Danny_B: like you can see, wikibugs didn't join as you restarted it, but as you forced wikibugs to publish your comment ;)
[23:13:06] <wikibugs>	 06Operations, 10Phabricator, 13Patch-For-Review: Phabricator weekly report not generated (or at least sent) - https://phabricator.wikimedia.org/T139950#2468701 (10Danny_B) The stats generator can obviously send to the list. List archive is accessible to anybody who is maillist member (anybody can sign in).
[23:13:37] <Danny_B>	 Luke081515: like i said - that's undocumented feature ;-)
[23:14:21] <Luke081515>	 yep :)
[23:16:15] * Danny_B updated the doc
[23:16:44] <Krenair>	 !log testing that the SAL bot is still working
[23:16:49] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[23:16:55] <Krenair>	 okay so prod's one is good
[23:31:52] <wikibugs>	 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-07-12_(1.28.0-wmf.10): Transwiki import not working in production - https://phabricator.wikimedia.org/T140206#2468729 (10TTO) Funny thing is, I actually [[https://meta.wikimedia.org/wiki/Talk:Inte...