[00:00:37] <wikibugs>	 6operations, 6Discovery, 10Maps, 3Discovery-Maps-Sprint: Assign varnish memory-only role to maps servers - https://phabricator.wikimedia.org/T105076#1461663 (10Yurik)
[00:00:59] <grrrit-wm>	 (03PS1) 10Matanya: Fix fdcwiki's wgMetaNamespace to not be Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225515 
[00:01:35] <matanya>	 Krenair: ^
[00:01:45] <Krenair>	 thanks
[00:04:47] <icinga-wm>	 PROBLEM - check_puppetrun on heka is CRITICAL Puppet last ran 34446 seconds ago, expected 28800
[00:09:47] <icinga-wm>	 PROBLEM - check_puppetrun on heka is CRITICAL Puppet last ran 34746 seconds ago, expected 28800
[00:14:47] <icinga-wm>	 PROBLEM - check_puppetrun on heka is CRITICAL Puppet last ran 35046 seconds ago, expected 28800
[00:19:47] <icinga-wm>	 RECOVERY - check_puppetrun on heka is OK Puppet is currently enabled, last run 162 seconds ago with 0 failures
[01:21:54] <grrrit-wm>	 (03PS1) 10Alex Monk: Add rollbacker to arwikisource and autopatrolled+rollbacker to arwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225519 (https://phabricator.wikimedia.org/T97271) 
[02:03:29] <logmsgbot>	 !log LocalisationUpdate failed (1.26wmf14) at 2015-07-18 02:03:29+00:00
[02:03:34] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:07:38] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 18 02:07:38 UTC 2015 (duration 7m 37s)
[02:07:44] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:30:52] <logmsgbot>	 !log l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 07m 19s)
[02:30:58] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:34:29] <logmsgbot>	 !log LocalisationUpdate completed (1.26wmf14) at 2015-07-18 02:34:29+00:00
[02:34:34] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:48:48] <icinga-wm>	 PROBLEM - are wikitech and wt-static in sync on silver is CRITICAL: wikitech-static CRIT - wikitech and wikitech-static out of sync (102600s 100000s)
[04:24:40] <grrrit-wm>	 (03PS1) 10Yuvipanda: ores: Install spanish dictionaries as well [puppet] - 10https://gerrit.wikimedia.org/r/225521 
[04:24:53] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Install spanish dictionaries as well [puppet] - 10https://gerrit.wikimedia.org/r/225521 (owner: 10Yuvipanda)
[04:26:29] <icinga-wm>	 RECOVERY - are wikitech and wt-static in sync on silver is OK: wikitech-static OK - wikitech and wikitech-static in sync (12115 100000s)
[04:42:38] <icinga-wm>	 PROBLEM - Restbase endpoints health on restbase1004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[04:44:27] <icinga-wm>	 RECOVERY - Restbase endpoints health on restbase1004 is OK: All endpoints are healthy
[05:00:30] <grrrit-wm>	 (03PS1) 10Yuvipanda: ores: Fix stupid typo [puppet] - 10https://gerrit.wikimedia.org/r/225522 
[05:00:51] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Fix stupid typo [puppet] - 10https://gerrit.wikimedia.org/r/225522 (owner: 10Yuvipanda)
[05:05:05] <grrrit-wm>	 (03PS1) 10Yuvipanda: ores: Fix stupid typo [puppet] - 10https://gerrit.wikimedia.org/r/225523 
[05:05:16] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Fix stupid typo [puppet] - 10https://gerrit.wikimedia.org/r/225523 (owner: 10Yuvipanda)
[05:28:25] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 18 05:28:25 UTC 2015 (duration 28m 24s)
[05:28:30] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[05:35:58] <icinga-wm>	 PROBLEM - Apache HTTP on mw1160 is CRITICAL - Socket timeout after 10 seconds
[05:37:37] <icinga-wm>	 RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.109 second response time
[06:30:18] <icinga-wm>	 PROBLEM - puppet last run on cp1054 is CRITICAL Puppet has 2 failures
[06:31:48] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL Puppet has 1 failures
[06:31:49] <icinga-wm>	 PROBLEM - puppet last run on mw1061 is CRITICAL Puppet has 1 failures
[06:32:19] <icinga-wm>	 PROBLEM - puppet last run on mw2018 is CRITICAL Puppet has 1 failures
[06:32:28] <icinga-wm>	 PROBLEM - puppet last run on holmium is CRITICAL Puppet has 1 failures
[06:32:37] <icinga-wm>	 PROBLEM - puppet last run on mw1119 is CRITICAL Puppet has 1 failures
[06:32:48] <icinga-wm>	 PROBLEM - puppet last run on db2056 is CRITICAL Puppet has 1 failures
[06:33:11] <grrrit-wm>	 (03PS11) 10Madhuvishy: wikilabels: Puppetizing wikilabels infrastructure [puppet] - 10https://gerrit.wikimedia.org/r/225092 
[06:56:08] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK Puppet is currently enabled, last run 3 seconds ago with 0 failures
[06:56:17] <icinga-wm>	 RECOVERY - puppet last run on mw1061 is OK Puppet is currently enabled, last run 28 seconds ago with 0 failures
[06:56:37] <icinga-wm>	 RECOVERY - puppet last run on cp1054 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:56:48] <icinga-wm>	 RECOVERY - puppet last run on holmium is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:56:57] <icinga-wm>	 RECOVERY - puppet last run on mw1119 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:09] <icinga-wm>	 RECOVERY - puppet last run on db2056 is OK Puppet is currently enabled, last run 56 seconds ago with 0 failures
[06:58:38] <icinga-wm>	 RECOVERY - puppet last run on mw2018 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:00:10] <grrrit-wm>	 (03PS1) 10Yuvipanda: ores: Install uwsgi python3 plugin only on web nodes [puppet] - 10https://gerrit.wikimedia.org/r/225526 
[07:00:24] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Install uwsgi python3 plugin only on web nodes [puppet] - 10https://gerrit.wikimedia.org/r/225526 (owner: 10Yuvipanda)
[07:15:31] <SPF|Cloud>	 There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection timed out
[07:15:33] <SPF|Cloud>	 yay.
[07:18:07] <SPF|Cloud>	 Hm, sometimes the web interface works, sometimes it doesn't
[08:31:02] <grrrit-wm>	 (03CR) 10Glaisher: [C: 04-1] "You should also add 'Wikipedia' and 'Wikipedia_talk' as aliases otherwise exisiting links will break. Also add a reference to the task." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225515 (owner: 10Matanya)
[08:38:36] <grrrit-wm>	 (03CR) 10Alex Monk: "It would only break if there were existing links." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225515 (owner: 10Matanya)
[08:38:53] <grrrit-wm>	 (03PS2) 10Alex Monk: Fix fdcwiki's wgMetaNamespace to not be Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225515 (https://phabricator.wikimedia.org/T106188) (owner: 10Matanya)
[08:43:31] <grrrit-wm>	 (03CR) 10Glaisher: "I think we also need custom contactpage-* messages (should probably asked in the task?)" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225509 (https://phabricator.wikimedia.org/T98625) (owner: 10Alex Monk)
[08:46:57] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 14.29% of data above the critical threshold [500.0]
[08:58:17] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[14:43:06] <grrrit-wm>	 (03PS1) 10Alex Monk: Remove old VE namespaces on testwiki and mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225541 (https://phabricator.wikimedia.org/T92797) 
[14:44:00] <wikibugs>	 6operations, 10Labs-Vagrant: Backport Vagrant 1.7+ from Debian experimental to our Trusty apt repo - https://phabricator.wikimedia.org/T93153#1461996 (10bd808)
[14:44:38] <wikibugs>	 6operations, 10Labs-Vagrant: Backport Vagrant 1.7+ from Debian experimental to our Trusty apt repo - https://phabricator.wikimedia.org/T93153#1130642 (10bd808)
[14:47:14] <grrrit-wm>	 (03PS18) 10BryanDavis: Add role::mediawiki_vagrant_lxc [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) 
[15:10:53] <wikibugs>	 6operations, 7user-notice: schedule maintenance for IRC server - https://phabricator.wikimedia.org/T105804#1462042 (10Dzahn) @gpaumier Sorry about that. I was travelling and did not get to actually schedule it this week yet and i will be on vacation for one week after Wikimania. Putting it on the tech news was...
[15:16:07] <wikibugs>	 6operations, 7user-notice: schedule maintenance for IRC server - https://phabricator.wikimedia.org/T105804#1462044 (10gpaumier) @Dzahn no problem, thank you for following up. Enjoy the vacation :)
[15:18:17] <icinga-wm>	 PROBLEM - puppet last run on ms-be2004 is CRITICAL puppet fail
[15:19:00] <subbu>	 when opening ganglia pages, I am encountering: "There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection timed out"
[15:42:57] <Krenair>	 !log Doing T44180
[15:43:01] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:45:18] <icinga-wm>	 RECOVERY - puppet last run on ms-be2004 is OK Puppet is currently enabled, last run 31 seconds ago with 0 failures
[15:48:58] <Krenair>	 what's up with ganglia?
[15:49:37] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL Anomaly detected: 10 data above and 2 below the confidence bounds
[16:09:33] <Krenair>	 !log Ganglia seems down
[16:09:38] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[16:13:53] <wikibugs>	 6operations, 10Traffic, 10fundraising-tech-ops, 5Patch-For-Review: Decide what to do with *.donate.wikimedia.org subdomain + TLS - https://phabricator.wikimedia.org/T102827#1462092 (10CCogdill_WMF) Thanks for this explanation, @BBlack. I got a little further clarification from @faidon so I think I understa...
[16:18:18] <Krenair>	 akosiaris, hi, are ops aware of that?
[16:19:35] <paravoid>	 Krenair: looking
[16:19:42] <paravoid>	 (thanks)
[16:27:07] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: ganglia: Set up robots.txt at doc_root, not ganglia_webdir [puppet] - 10https://gerrit.wikimedia.org/r/225550 
[16:28:55] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: ganglia: Set up robots.txt at doc_root, not ganglia_webdir [puppet] - 10https://gerrit.wikimedia.org/r/225550 
[16:31:18] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] ganglia: Set up robots.txt at doc_root, not ganglia_webdir [puppet] - 10https://gerrit.wikimedia.org/r/225550 (owner: 10Alexandros Kosiaris)
[16:34:38] <Krenair>	 akosiaris, paravoid: it looks better now, thanks
[16:36:01] <paravoid>	 !log Ganglia is up :)
[16:36:05] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[16:37:34] <grrrit-wm>	 (03CR) 10Nikerabbit: "I don't like the inconsistency this introduces compared to other Wikipedias. I don't have enough info to judge whether it is worth it (and" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/224772 (https://phabricator.wikimedia.org/T105124) (owner: 10Dereckson)
[16:39:57] <icinga-wm>	 PROBLEM - Apache HTTP on mw1160 is CRITICAL - Socket timeout after 10 seconds
[16:41:39] <icinga-wm>	 RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.099 second response time
[16:43:27] <grrrit-wm>	 (03PS1) 10Gergő Tisza: Update apache rules for 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/225552 
[16:45:34] <grrrit-wm>	 (03PS1) 10Gergő Tisza: [WIP] Update apache rules for 2.4 [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/225553 
[16:46:12] <grrrit-wm>	 (03PS2) 10Gergő Tisza: [WIP] Update apache rules for 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/225552 
[16:47:27] <icinga-wm>	 PROBLEM - Apache HTTP on mw1159 is CRITICAL - Socket timeout after 10 seconds
[16:48:40] <YuviPanda>	 tgr: hey!
[16:48:56] <YuviPanda>	 tgr: are you working on wikimetrics? I was just going to help madhuvishy rewrite the module completely :)
[16:49:19] <icinga-wm>	 RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.095 second response time
[16:51:36] <grrrit-wm>	 (03CR) 10Gergő Tisza: "Missed all the lowercase files ( ack '^\s*(allow|deny) ' -l )." [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/225553 (owner: 10Gergő Tisza)
[16:57:41] <_joe|afk>	 what's happening with the miagescalers?
[16:59:39] <grrrit-wm>	 (03PS1) 10Gergő Tisza: Update deployment_server rule to support Apache 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/225554 (https://phabricator.wikimedia.org/T84956) 
[16:59:57] <icinga-wm>	 PROBLEM - puppet last run on maps-test2004 is CRITICAL Puppet last ran 17 hours ago
[17:00:28] <grrrit-wm>	 (03CR) 10Gergő Tisza: "This changes the 403 error to a 404. Not sure if there is a problem with the patch or that is an independent issue." [puppet] - 10https://gerrit.wikimedia.org/r/225554 (https://phabricator.wikimedia.org/T84956) (owner: 10Gergő Tisza)
[17:01:58] <icinga-wm>	 RECOVERY - puppet last run on maps-test2004 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:05:39] <grrrit-wm>	 (03PS12) 10Madhuvishy: wikilabels: Puppetize wikilabels infrastructure [puppet] - 10https://gerrit.wikimedia.org/r/225092 (https://phabricator.wikimedia.org/T106218) 
[17:06:57] <icinga-wm>	 RECOVERY - citoid endpoints health on sca1001 is OK: All endpoints are healthy
[17:07:54] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] wikilabels: Puppetize wikilabels infrastructure [puppet] - 10https://gerrit.wikimedia.org/r/225092 (https://phabricator.wikimedia.org/T106218) (owner: 10Madhuvishy)
[17:12:29] <icinga-wm>	 PROBLEM - citoid endpoints health on sca1001 is CRITICAL: /api is CRITICAL: Test bad PMCID returned the unexpected status 200 (expecting: 404)
[17:17:09] <grrrit-wm>	 (03CR) 10Gergő Tisza: "Seems like the /srv/deployment/scap/scap/.git/deploy directory is missing. Not related to the patch." [puppet] - 10https://gerrit.wikimedia.org/r/225554 (https://phabricator.wikimedia.org/T84956) (owner: 10Gergő Tisza)
[17:21:36] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: labsdb1004: Reinstall as jessie [puppet] - 10https://gerrit.wikimedia.org/r/225558 (https://phabricator.wikimedia.org/T101233) 
[17:24:37] <bd808>	 tgr: missing /srv/deployment/scap/scap/.git/deploy sounds like trebuchet didn't setup things right. `salt-call deploy.deployment_server_init` might fix it
[17:25:00] <bd808>	 that would be as root on your deploy host
[17:28:08] <grrrit-wm>	 (03PS1) 10Legoktm: Set $wgVectorResponsive = true on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225559 (https://phabricator.wikimedia.org/T106226) 
[17:39:28] <icinga-wm>	 RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected
[18:24:51] <wikibugs>	 6operations, 10Analytics, 6MediaWiki-Stakeholders-Group, 10Traffic, 10Wikimania-Hackathon-2015: Provide summary of MediaWiki downloads - https://phabricator.wikimedia.org/T104010#1462326 (10Nemo_bis) As for USA, there are a number of very similar lines  ``` /mediawiki/1.10/mediawiki-1.10.0.tar.gz United...
[18:29:17] <icinga-wm>	 PROBLEM - puppet last run on sca1001 is CRITICAL puppet fail
[18:47:13] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Stop labsdb1004 replication to labsdb1005 [puppet] - 10https://gerrit.wikimedia.org/r/225562 (https://phabricator.wikimedia.org/T101233) 
[18:47:54] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: labsdb1004: Reinstall as jessie [puppet] - 10https://gerrit.wikimedia.org/r/225558 (https://phabricator.wikimedia.org/T101233) 
[18:48:02] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] labsdb1004: Reinstall as jessie [puppet] - 10https://gerrit.wikimedia.org/r/225558 (https://phabricator.wikimedia.org/T101233) (owner: 10Alexandros Kosiaris)
[18:48:39] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Stop labsdb1004 replication to labsdb1005 [puppet] - 10https://gerrit.wikimedia.org/r/225562 (https://phabricator.wikimedia.org/T101233) 
[18:48:45] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Stop labsdb1004 replication to labsdb1005 [puppet] - 10https://gerrit.wikimedia.org/r/225562 (https://phabricator.wikimedia.org/T101233) (owner: 10Alexandros Kosiaris)
[18:50:46] <grrrit-wm>	 (03CR) 10Alex Monk: "try it on beta first" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225559 (https://phabricator.wikimedia.org/T106226) (owner: 10Legoktm)
[18:55:42] <grrrit-wm>	 (03PS1) 10MaxSem: postgresql: install pg_top to have a better idea what's going on [puppet] - 10https://gerrit.wikimedia.org/r/225565 
[18:56:49] <akosiaris>	 !log reinstall labsdb1004
[18:56:53] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:04:50] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: maps: give login rights to postgres users [puppet] - 10https://gerrit.wikimedia.org/r/225511 
[19:04:55] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] maps: give login rights to postgres users [puppet] - 10https://gerrit.wikimedia.org/r/225511 (owner: 10Alexandros Kosiaris)
[19:07:41] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: postgresql: install pg_top to have a better idea what's going on [puppet] - 10https://gerrit.wikimedia.org/r/225565 (owner: 10MaxSem)
[19:07:43] <wikibugs>	 7Puppet, 6Analytics-Kanban, 10Analytics-Wikimetrics: Cleanup Wikimetrics puppet module so it can run puppet continuously without own puppetmaster - https://phabricator.wikimedia.org/T101763#1462428 (10madhuvishy) p:5Triage>3Normal a:3madhuvishy
[19:07:47] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] postgresql: install pg_top to have a better idea what's going on [puppet] - 10https://gerrit.wikimedia.org/r/225565 (owner: 10MaxSem)
[19:08:22] <wikibugs>	 7Puppet, 6Analytics-Kanban, 10Analytics-Wikimetrics: Cleanup Wikimetrics puppet module so it can run puppet continuously without own puppetmaster {dove} - https://phabricator.wikimedia.org/T101763#1462432 (10madhuvishy)
[19:33:32] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: maps: Give rights to maps-admins [puppet] - 10https://gerrit.wikimedia.org/r/225566 
[19:34:30] <grrrit-wm>	 (03PS1) 10MaxSem: postgresql: install postgis package for shp2pgsql and similar binaries [puppet] - 10https://gerrit.wikimedia.org/r/225567 
[19:37:43] <grrrit-wm>	 (03PS1) 10Alex Monk: Resume running refreshLinks cron on enwiki [puppet] - 10https://gerrit.wikimedia.org/r/225569 (https://phabricator.wikimedia.org/T44180) 
[19:40:15] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: role::postgres: Update to 9.4 [puppet] - 10https://gerrit.wikimedia.org/r/225575 
[19:40:30] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: role::postgres: Update to 9.4 [puppet] - 10https://gerrit.wikimedia.org/r/225575 
[19:41:13] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] role::postgres: Update to 9.4 [puppet] - 10https://gerrit.wikimedia.org/r/225575 (owner: 10Alexandros Kosiaris)
[19:47:17] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: role::postgres: Update to 9.4 [puppet] - 10https://gerrit.wikimedia.org/r/225575 
[19:49:46] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: maps: Tune postgresql [puppet] - 10https://gerrit.wikimedia.org/r/225576 
[19:49:48] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] role::postgres: Update to 9.4 [puppet] - 10https://gerrit.wikimedia.org/r/225575 (owner: 10Alexandros Kosiaris)
[20:00:17] <grrrit-wm>	 (03PS1) 10Krinkle: scap: Return early with usage note if no arguments passed [puppet] - 10https://gerrit.wikimedia.org/r/225605 
[20:04:39] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 25.00% of data above the critical threshold [500.0]
[20:07:26] <grrrit-wm>	 (03CR) 10Yuvipanda: scap: Return early with usage note if no arguments passed (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/225605 (owner: 10Krinkle)
[20:14:00] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[20:24:10] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: maps: LVS IP for kartotherian and tilerator [dns] - 10https://gerrit.wikimedia.org/r/225640 (https://phabricator.wikimedia.org/T105074) 
[20:24:41] <Krinkle>	 YuviPanda: What are you saying, no tabs or no spaces?
[20:25:04] <YuviPanda>	 Krinkle: I think ops/puppet in general is all spaces, no hard tabs
[20:25:09] <Krinkle>	 k
[20:25:36] <grrrit-wm>	 (03PS2) 10Krinkle: scap: Return early with usage note if no arguments passed [puppet] - 10https://gerrit.wikimedia.org/r/225605 
[20:26:18] <YuviPanda>	 Krinkle: would you like me to merge that now?
[20:26:19] <icinga-wm>	 RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 4 seconds ago with 0 failures
[20:27:43] <Krinkle>	 YuviPanda: Sure :)
[20:27:50] <grrrit-wm>	 (03PS3) 10Yuvipanda: scap: Return early with usage note if no arguments passed [puppet] - 10https://gerrit.wikimedia.org/r/225605 (owner: 10Krinkle)
[20:27:57] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] scap: Return early with usage note if no arguments passed [puppet] - 10https://gerrit.wikimedia.org/r/225605 (owner: 10Krinkle)
[20:28:05] <Krinkle>	 YuviPanda: I actually have a stronger need for  https://gerrit.wikimedia.org/r/225639 – as I want to run a simple select query on all wikis to debug something
[20:28:42] <YuviPanda>	 Krinkle: hmm, why -q?
[20:28:57] <YuviPanda>	 that seems like 'quiet' rather than 'run noninteractively'
[20:29:02] <Krinkle>	 Hm.. right
[20:29:02] <YuviPanda>	 how about -e?
[20:29:09] <YuviPanda>	 or -c, I guess
[20:29:44] <Krinkle>	 or --query
[20:29:53] <YuviPanda>	 Krinkle: oh, hmm...
[20:29:57] <Krinkle>	 the other options are words too
[20:30:13] <YuviPanda>	 yeah
[20:30:18] <YuviPanda>	 --query yes
[20:37:02] <yurik>	 akosiaris, join us upstairs
[20:37:48] <yurik>	 akosiaris, actually no, we are heading to 2nd fl to OSM workshop
[20:39:05] <grrrit-wm>	 (03CR) 10Yurik: [C: 031] "ya ya :)" [dns] - 10https://gerrit.wikimedia.org/r/225640 (https://phabricator.wikimedia.org/T105074) (owner: 10Alexandros Kosiaris)
[20:40:00] <YuviPanda>	 akosiaris: don't join yurik upstairs, I hereby declare you stolen for labsdb :P
[20:40:17] <yurik>	 YuviPanda, you had him for 2+ hrs :-P
[20:42:09] <Nikerabbit>	 etherpad.wikimedia.org just went down
[20:42:42] <ori>	 ♥
[20:44:41] <YuviPanda>	 !log restarted etherpad
[20:44:43] <YuviPanda>	 Nikerabbit: ^
[20:44:46] <YuviPanda>	 back now?
[20:44:46] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:46:04] <Nikerabbit>	 YuviPanda: yes thx
[20:46:16] <YuviPanda>	 Nikerabbit: yw
[20:46:50] <YuviPanda>	 akosiaris: did you restore the dump? :D
[20:55:10] <grrrit-wm>	 (03PS2) 10Legoktm: Set $wgVectorResponsive = true on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225559 (https://phabricator.wikimedia.org/T106226) 
[20:55:12] <grrrit-wm>	 (03PS1) 10Legoktm: Set $wgVectorResponsive = true on all beta cluster wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225642 
[20:56:32] <YuviPanda>	 legoktm: nice
[20:56:37] <legoktm>	 :D
[20:56:45] <YuviPanda>	 legoktm: are you going to sync now?
[20:56:47] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Set $wgVectorResponsive = true on all beta cluster wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225642 (owner: 10Legoktm)
[20:56:53] <grrrit-wm>	 (03Merged) 10jenkins-bot: Set $wgVectorResponsive = true on all beta cluster wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225642 (owner: 10Legoktm)
[20:56:53] <YuviPanda>	 heh
[20:56:54] <YuviPanda>	 ofc
[20:57:46] <akosiaris>	 YuviPanda: etherpad restarts itself
[20:57:51] <akosiaris>	 systemd and all
[20:58:19] <akosiaris>	 YuviPanda: no, not yet. where are you ?
[20:58:39] <YuviPanda>	 akosiaris: it was running but unresponsive
[20:58:43] <logmsgbot>	 !log legoktm Synchronized wmf-config/InitialiseSettings-labs.php: labs only (duration: 00m 12s)
[20:58:47] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:59:04] <akosiaris>	 YuviPanda: sigh, one more bugs for john mclear
[20:59:10] <YuviPanda>	 akosiaris: I'm on floor three middle thing, at lzia's talk
[20:59:13] <akosiaris>	 s/bugs/bug/
[20:59:35] <akosiaris>	 YuviPanda: I am ready to restore the dump
[20:59:41] <YuviPanda>	 akosiaris: w00t! do ittt!
[20:59:44] <legoktm>	 YuviPanda: http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version is responsive
[20:59:49] <legoktm>	 !!
[21:00:43] <akosiaris>	 YuviPanda: ok, starting it in a screen. it's 72 GB, it's gonna take a long time
[21:01:04] <YuviPanda>	 legoktm: <3 nice
[21:01:07] <YuviPanda>	 akosiaris: sweet :)
[21:10:32] <YuviPanda>	 cscott: are you aware of any safe ways to do client side templating with arbitrary user templates? (for context - I am planning on implementing a template based 'reports' setup for quarry.wmflabs.org)
[21:12:35] <cscott>	 YuviPanda: hm, that's an interesting question.
[21:13:15] <cscott>	 spacebars is a handlebars clone that works with the DOM, so you're not going to have weird "little bobby tables" type of attacks
[21:13:15] <YuviPanda>	 cscott: my current solution is to use sandboxed jinja2 templates on the server, but that takes away some interactiveness gains we can get from client side templates...
[21:13:32] <YuviPanda>	 hmm, so we can disallow <script> and friends?
[21:13:40] <YuviPanda>	 it also needs to not lock up the browser by infinite looping
[21:13:44] <cscott>	 but i think you would still have to run the output through some kind of sanitizer to prevent <a href="javascript:..."> sorts of things.
[21:14:11] <cscott>	 and <script>, yes.
[21:15:12] <cscott>	 something like handlebars (assuming you don't allow custom block helpers) should also be safe from a compute perspective.
[21:15:12] <YuviPanda>	 not sure if I can actually limit time it runs...
[21:15:15] <YuviPanda>	 the data sets can be huge
[21:15:19] <YuviPanda>	 oh?
[21:15:26] <YuviPanda>	 like, this is arbitrary SQL
[21:15:33] <cscott>	 i mean it should run in time proportional to the input data set size
[21:15:40] <YuviPanda>	 right
[21:15:46] <YuviPanda>	 but I want absolute limits
[21:15:54] <YuviPanda>	 because input data size can be arbitrarily large
[21:15:55] <cscott>	 but i think you can rely on the user's browser to do that protection
[21:16:08] <cscott>	 it will pop up UI asking the user if they want to continue, etc.
[21:16:30] <YuviPanda>	 right
[21:16:31] <YuviPanda>	 hmm
[21:16:50] <YuviPanda>	 this does mean that it's run on every user's browser rather than run once on server and then cached
[21:16:59] <cscott>	 i would think you'd want to protect against XSS primarily.  and that's probably best done by post-processing the DOM to throw away <script> and bad hrefs and stuff.
[21:17:12] <YuviPanda>	 yeah, XSS is teh biggest problem
[21:17:31] <cscott>	 YuviPanda: right.  you could potentially run it in node.js server side, then you wouldn't have XSS issues but you would have to ulimit the process.
[21:17:56] <YuviPanda>	 cscott: ulimits etc is not a problem - but the problem is that I'll cache them and hence they can't be 'interactive' - they're frozen in time a bit.
[21:17:59] <cscott>	 well, i guess the generated HTML could still have evil tags, so you still have the same xss stuff.
[21:18:12] <cscott>	 YuviPanda: well, maybe try the client side approach and see how it works out.
[21:18:20] <cscott>	 browser sandboxes are pretty good.
[21:18:22] <YuviPanda>	 cscott: yeah, because with client side stuff people can use http://worrydream.com/Tangle/
[21:18:24] <YuviPanda>	 and friends
[21:18:31] <YuviPanda>	 (possibly)
[21:19:43] <grrrit-wm>	 (03PS5) 10Thcipriani: Add service deploy via scap [tools/scap] - 10https://gerrit.wikimedia.org/r/224374 
[21:20:14] <yurik>	 YuviPanda, what's the status of the osm db on labs? i'm in the maps conference, ppl asking about updates to the osm database
[21:20:26] <YuviPanda>	 yurik: ask Coren|MX, I'm not sure.
[21:20:35] <YuviPanda>	 yurik: he's been the one dealing with it
[21:20:39] <wikibugs>	 6operations, 6Release-Engineering, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1462694 (10Krenair) And also nb.wikinews.org -> no.wikinews.org?
[21:23:39] <YuviPanda>	 cscott: I'll experiment and let you know if I get anywhere :)
[21:23:41] <YuviPanda>	 thanks!
[21:24:19] <cscott>	 YuviPanda: sure.  if you need any help on the output sanitizer, poke me.  i've been meaning to write a properly-spec'ed standalone sanitizer for a while.
[21:24:43] <YuviPanda>	 cscott: cool! will do :)
[21:25:02] <YuviPanda>	 cscott: I'm already allowing arbitrary sql injection and surviving, would be fun to see if this can work too :D
[21:25:16] <YuviPanda>	 I wonder if there's a secure way of allowing *JS* execution
[21:25:19] <YuviPanda>	 I guess not
[21:26:33] <YuviPanda>	 no clear definition of 'safe' I guess
[21:28:16] <cscott>	 YuviPanda: v8 is pretty well sandboxed.  v8js (the php extension) likewise.
[21:28:27] <YuviPanda>	 cscott: on the client I meant
[21:28:35] <YuviPanda>	 while avoiding XSS as well
[21:28:47] <ori>	 talk to brion
[21:28:48] <YuviPanda>	 I guess I could have it run on a totally different domain
[21:28:52] <ori>	 it is a problem he has thought a lot about
[21:28:53] <YuviPanda>	 aha!
[21:28:55] <YuviPanda>	 I remmeber
[21:29:17] <ori>	 he had some creative solutions, but I don't remember what they were.
[21:29:56] <ori>	 quick googling reveals https://github.com/asvd/jailed
[21:30:14] <ori>	 ah, it's iframe sorcery: http://www.html5rocks.com/en/tutorials/security/sandboxed-iframes/
[21:31:39] <cscott>	 oh, on the client.  yeah, there's stuff like https://code.google.com/p/google-caja/wiki/SES
[21:31:56] <cscott>	 iframes are probably a good idea.  an iframe is basically a sandbox by design.
[21:32:05] <yurik>	 akosiaris, whereareyouat?
[21:32:17] <cscott>	 there are even extra parameters to iframe to sandbox more tightly
[21:32:41] <YuviPanda>	 oooh interesting
[21:32:59] <cscott>	 https://developer.mozilla.org/en/docs/Web/HTML/Element/iframe the "sandbox" attribute (naturally)
[21:33:16] <YuviPanda>	 so I can actually allow arbitrary JS
[21:33:28] <cscott>	 potentially
[21:34:09] <cscott>	 i think you should pay careful attention to what network connections the sandbox is allowed to make
[21:34:37] <cscott>	 so that people don't launch DOS attacks from inside the iframe.  there might already be some protection there (CORS, etc)
[21:34:46] <YuviPanda>	 right
[21:35:08] <cscott>	 ok, switching to _away so i can network & snack some.
[21:36:27] <YuviPanda>	 I can probably disallow all of JS networks
[21:37:18] <grrrit-wm>	 (03PS1) 10Alex Monk: Set up Norwegian language aliases for remaining sites [puppet] - 10https://gerrit.wikimedia.org/r/225646 (https://phabricator.wikimedia.org/T86924) 
[21:37:43] <grrrit-wm>	 (03PS1) 10Ori.livneh: Add '__version__' and 'USER_AGENT_STRING' [debs/pybal] - 10https://gerrit.wikimedia.org/r/225647 
[21:40:30] <grrrit-wm>	 (03PS1) 10Ori.livneh: WIP: Add etcd configuration client [debs/pybal] - 10https://gerrit.wikimedia.org/r/225649 
[21:45:15] <brion>	 YuviPanda: yeah my plan for sandboxed JS widgets is to inject the JS into an <iframe> that's on a magic random subdomain of a separate domain, thus isolated from anything else
[21:45:39] <brion>	 it's possible to send data in and out via postMessage between the parent document and the iframe safely (JSON messages)
[21:45:56] <brion>	 of course don't do something stupid like taking a string from the untrusted JS and putting it into your HTML document as innerHTML ;)
[21:59:26] <grrrit-wm>	 (03CR) 10John Vandenberg: [C: 04-1] Set up Norwegian language aliases for remaining sites (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/225646 (https://phabricator.wikimedia.org/T86924) (owner: 10Alex Monk)
[22:09:30] <icinga-wm>	 PROBLEM - puppet last run on mw2040 is CRITICAL puppet fail
[22:20:46] <grrrit-wm>	 (03PS2) 10Ori.livneh: WIP: Add etcd configuration client [debs/pybal] - 10https://gerrit.wikimedia.org/r/225649 
[22:33:37] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 031] Add '__version__' and 'USER_AGENT_STRING' [debs/pybal] - 10https://gerrit.wikimedia.org/r/225647 (owner: 10Ori.livneh)
[22:35:59] <icinga-wm>	 RECOVERY - puppet last run on mw2040 is OK Puppet is currently enabled, last run 8 seconds ago with 0 failures
[22:36:36] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Add '__version__' and 'USER_AGENT_STRING' [debs/pybal] - 10https://gerrit.wikimedia.org/r/225647 (owner: 10Ori.livneh)
[22:36:38] <ori>	 thanks
[22:36:52] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add '__version__' and 'USER_AGENT_STRING' [debs/pybal] - 10https://gerrit.wikimedia.org/r/225647 (owner: 10Ori.livneh)
[23:03:24] <YuviPanda>	 akosiaris: no screen?
[23:03:28] <YuviPanda>	 is the dump done?
[23:07:38] <grrrit-wm>	 (03PS19) 10BryanDavis: Add role::mediawiki_vagrant_lxc [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) 
[23:07:57] <yurik>	 akosiaris, around?
[23:08:30] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add role::mediawiki_vagrant_lxc [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) (owner: 10BryanDavis)
[23:09:38] <grrrit-wm>	 (03PS20) 10BryanDavis: Add role::mediawiki_vagrant_lxc [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) 
[23:12:30] <grrrit-wm>	 (03CR) 10BryanDavis: "PS 20 changes back to assuming that we will be using the "official" Vagrant deb from https://www.vagrantup.com/. The rabbit hole of backpo" [puppet] - 10https://gerrit.wikimedia.org/r/193665 (https://phabricator.wikimedia.org/T106213) (owner: 10BryanDavis)
[23:13:41] <YuviPanda>	 bd808: I already did it btw
[23:13:43] <YuviPanda>	 bd808: forgot to tell you
[23:13:55] <bd808>	 cool beans
[23:28:58] <grrrit-wm>	 (03PS10) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles)
[23:29:58] <icinga-wm>	 PROBLEM - puppet last run on sca1001 is CRITICAL Puppet has 121 failures
[23:34:58] <grrrit-wm>	 (03CR) 10Legoktm: [C: 04-2] "Per my comment on I22defb9431d0708d00d8294efc44cc8cf287fbcd" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/207909 (owner: 10Paladox)
[23:35:08] <wikibugs>	 6operations, 6Release-Engineering, 10Wikimedia-Language-setup, 5Patch-For-Review: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1462962 (10Krenair)
[23:58:18] <icinga-wm>	 RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures