[00:00:01] <grrrit-wm>	 (03PS2) 10Ori.livneh: Sync ps_mem.py from origin [puppet] - 10https://gerrit.wikimedia.org/r/264028 (owner: 10John Vandenberg)
[00:00:04] <jouncebot>	 RoanKattouw ostriches Krenair: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160115T0000).
[00:00:04] <jouncebot>	 IoannisKydonis andre__: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process.
[00:00:47] <grrrit-wm>	 (03PS3) 10Ori.livneh: Sync ps_mem.py from origin [puppet] - 10https://gerrit.wikimedia.org/r/264028 (owner: 10John Vandenberg)
[00:02:35] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Sync ps_mem.py from origin [puppet] - 10https://gerrit.wikimedia.org/r/264028 (owner: 10John Vandenberg)
[00:06:56] <mobrovac>	 !log restbase started a dump of enwiki to populate storage with mobileapps renders
[00:07:01] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[00:12:09] <grrrit-wm>	 (03CR) 10Mobrovac: [C: 04-1] Add parsoid::testing role and use it on ruthenium (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/264024 (owner: 10Subramanya Sastry)
[00:14:13] <Krenair>	 hm
[00:14:19] <Krenair>	 I suppose I should swat then
[00:14:46] <Krenair>	 even though no one is here
[00:15:17] <Krenair>	 tgr, around?
[00:15:26] <tgr>	 Krenair: o/
[00:15:41] <Krenair>	 so we have a request to deploy https://gerrit.wikimedia.org/r/#/c/264091/
[00:16:02] <tgr>	 yeah, it's a GCI task
[00:17:06] <Krenair>	 is that all okay with you, and can you check everything is working as expected after deployment?
[00:18:09] <tgr>	 yes
[00:18:37] <grrrit-wm>	 (03PS9) 10Alex Monk: Update MediaViewer configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264091 (owner: 10IoannisKydonis)
[00:19:03] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 032] Update MediaViewer configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264091 (owner: 10IoannisKydonis)
[00:19:37] <grrrit-wm>	 (03Merged) 10jenkins-bot: Update MediaViewer configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264091 (owner: 10IoannisKydonis)
[00:21:27] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Remove use of passwords::openstack::nova::nova_ldap_user_pass [puppet] - 10https://gerrit.wikimedia.org/r/264217 
[00:22:00] <logmsgbot>	 !log krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/264091/ (duration: 00m 32s)
[00:22:06] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[00:22:08] <Krenair>	 tgr, ^
[00:23:14] <grrrit-wm>	 (03PS5) 10Alex Monk: Lift IP rate limit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263905 (https://phabricator.wikimedia.org/T123458) (owner: 10Samtar)
[00:24:00] <logmsgbot>	 !log krenair@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 31s)
[00:24:06] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[00:24:30] <Krenair>	 tgr, all ok?
[00:24:38] <grrrit-wm>	 (03PS5) 10Subramanya Sastry: Add parsoid::testing role and use it on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/264024 
[00:24:40] <grrrit-wm>	 (03PS3) 10Subramanya Sastry: WIP: Add the visualdiff module; instantiate psd visualdiffing services [puppet] - 10https://gerrit.wikimedia.org/r/264032 
[00:24:55] <tgr>	 no effect so far, but it's a ResourceLoader change, those tend to take a few minutes
[00:25:27] <grrrit-wm>	 (03CR) 10Mobrovac: WIP: Add the visualdiff module; instantiate psd visualdiffing services (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/264032 (owner: 10Subramanya Sastry)
[00:26:09] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add parsoid::testing role and use it on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/264024 (owner: 10Subramanya Sastry)
[00:26:11] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 032] Lift IP rate limit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263905 (https://phabricator.wikimedia.org/T123458) (owner: 10Samtar)
[00:26:30] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] WIP: Add the visualdiff module; instantiate psd visualdiffing services [puppet] - 10https://gerrit.wikimedia.org/r/264032 (owner: 10Subramanya Sastry)
[00:26:48] <grrrit-wm>	 (03Merged) 10jenkins-bot: Lift IP rate limit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263905 (https://phabricator.wikimedia.org/T123458) (owner: 10Samtar)
[00:27:03] <grrrit-wm>	 (03PS1) 10Cenarium: Autopromotion: remove deprecated onView event, fix INGROUPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264219 
[00:27:36] <logmsgbot>	 !log krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/263905/ (duration: 00m 32s)
[00:27:41] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[00:29:51] <grrrit-wm>	 (03CR) 10Mobrovac: [C: 04-1] Add parsoid::testing role and use it on ruthenium (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/264024 (owner: 10Subramanya Sastry)
[00:30:54] <icinga-wm>	 PROBLEM - HHVM rendering on mw1204 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:31:25] <icinga-wm>	 PROBLEM - Apache HTTP on mw1204 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:32:26] <tgr>	 Krenair: all good, thanks
[00:32:52] <tgr>	 actually, http://en.wikipedia.beta.wmflabs.org/wiki/Special:ListFiles is broken, but that's unrelated
[00:35:35] <tgr>	 filed as https://phabricator.wikimedia.org/T123695
[00:46:33] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Remove use of passwords::openstack::nova::nova_ldap_user_pass [puppet] - 10https://gerrit.wikimedia.org/r/264217 
[00:48:02] <Krenair>	 might do a backport of a core change to fix an error occurring in prod
[00:49:29] <wikibugs>	 6operations, 7Mail: delete exim alias vpe-staff: eng-mgt - https://phabricator.wikimedia.org/T123667#1935936 (10JKrauska) Spoke to praveena who originally requested this -- safe to remove...
[00:49:52] <wikibugs>	 6operations, 7Mail: delete exim alias vpe-staff: eng-mgt - https://phabricator.wikimedia.org/T123667#1935938 (10JKrauska) a:5JKrauska>3Dzahn
[00:57:07] <grrrit-wm>	 (03PS3) 10Andrew Bogott: Remove use of passwords::openstack::nova::nova_ldap_user_pass [puppet] - 10https://gerrit.wikimedia.org/r/264217 
[01:00:36] <logmsgbot>	 !log krenair@tin Synchronized php-1.27.0-wmf.10/includes/api/ApiQueryWatchlist.php: https://gerrit.wikimedia.org/r/#/c/264224/ (duration: 00m 31s)
[01:00:42] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[01:00:57] <Krenair>	 hmm
[01:01:00] <Krenair>	 I don't think that fully fixed it
[01:01:11] <Krenair>	 pretty sure there's still the RC one causing issues
[01:06:35] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[01:07:54] <icinga-wm>	 PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[01:10:29] <grrrit-wm>	 (03PS1) 10Cmjohnson: Adding piwik-roots group bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264229 
[01:11:15] <icinga-wm>	 PROBLEM - Mobile HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[01:11:37] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Adding piwik-roots group bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264229 (owner: 10Cmjohnson)
[01:12:45] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[01:13:15] <icinga-wm>	 RECOVERY - Mobile HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[01:14:05] <icinga-wm>	 RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[01:25:31] <grrrit-wm>	 (03PS3) 10Mattflaschen: Exclude fishbowl and add computed dblist for Flow [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250460 
[01:38:35] <icinga-wm>	 PROBLEM - puppet last run on cp2005 is CRITICAL: CRITICAL: puppet fail
[01:55:07] <grrrit-wm>	 (03PS6) 10Subramanya Sastry: Add parsoid::testing role and use it on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/264024 
[01:55:10] <grrrit-wm>	 (03PS4) 10Subramanya Sastry: WIP: Add the visualdiff module; instantiate psd visualdiffing services [puppet] - 10https://gerrit.wikimedia.org/r/264032 
[02:00:55] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0]
[02:01:55] <icinga-wm>	 PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [1000.0]
[02:05:14] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[02:05:55] <icinga-wm>	 RECOVERY - puppet last run on cp2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:06:05] <icinga-wm>	 RECOVERY - Ulsfo HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[02:09:45] <icinga-wm>	 PROBLEM - puppet last run on bromine is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[02:23:43] <YuviPanda>	 !log pull annualreport git repo on bromine for Krenair
[02:23:49] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:26:02] <Krenair>	 thanks YuviPanda, also can you add that puppet swat window please?
[02:26:12] <YuviPanda>	 oh yeah
[02:26:14] <YuviPanda>	 sure
[02:26:20] <Krenair>	 or I suppose I can try to guess all the details
[02:26:23] <Krenair>	 based on previous ones
[02:26:25] <icinga-wm>	 PROBLEM - puppet last run on mw1013 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[02:26:27] <YuviPanda>	 thanks for reminding me
[02:26:35] <YuviPanda>	 Krenair: that's what I'm going to do :D
[02:26:40] <YuviPanda>	 Krenair: since I"ve no real memory anyway
[02:27:00] <YuviPanda>	 Krenair: it's the same time as thursday, same people
[02:28:56] <icinga-wm>	 PROBLEM - HHVM rendering on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:29:02] <YuviPanda>	 tables
[02:29:04] <YuviPanda>	 how do they work
[02:29:25] <icinga-wm>	 PROBLEM - Apache HTTP on mw1126 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.039 second response time
[02:29:55] <logmsgbot>	 !log mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 14m 00s)
[02:30:01] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:30:48] <logmsgbot>	 !log krenair@tin Synchronized php-1.27.0-wmf.10/includes/api/ApiQueryRecentChanges.php: https://gerrit.wikimedia.org/r/264231 (duration: 00m 42s)
[02:30:53] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:30:58] <Krenair>	 That seems to have stopped the rc_log_type errors
[02:31:05] <icinga-wm>	 RECOVERY - HHVM rendering on mw1126 is OK: HTTP OK: HTTP/1.1 200 OK - 69502 bytes in 2.881 second response time
[02:31:44] <icinga-wm>	 RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.051 second response time
[02:33:34] <YuviPanda>	 Krenair: I think that's right
[02:33:50] <YuviPanda>	 I just copied the people who are in for thursday
[02:34:26] <Krenair>	 Well.
[02:34:28] <Krenair>	 https://office.wikimedia.org/wiki/Operations/Operations_Meeting_Notes/TechOps-2016-01-13#PuppetSWAT
[02:35:10] <YuviPanda>	 yeah ok
[02:35:13] <Krenair>	 moritzm and mutante did it today/yesterday
[02:35:14] <YuviPanda>	 so the deployments page is wrong
[02:39:38] <grrrit-wm>	 (03CR) 10Krinkle: [WIP] Implement /w/static.php (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263566 (https://phabricator.wikimedia.org/T99096) (owner: 10Krinkle)
[02:41:44] <YuviPanda>	 I don't know how people survive wikitext
[02:41:46] <YuviPanda>	 what a clusterfuck
[02:57:25] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[02:58:44] <icinga-wm>	 PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[03:00:04] <icinga-wm>	 PROBLEM - Mobile HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0]
[03:02:14] <icinga-wm>	 PROBLEM - puppet last run on mw1013 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[03:02:15] <icinga-wm>	 RECOVERY - Mobile HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[03:03:04] <icinga-wm>	 RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[03:03:22] <logmsgbot>	 !log mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 16m 02s)
[03:03:28] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:03:45] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[03:10:10] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Fri Jan 15 03:10:09 UTC 2016 (duration 6m 48s)
[03:10:15] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:21:35] <icinga-wm>	 PROBLEM - puppet last run on mw2055 is CRITICAL: CRITICAL: puppet fail
[03:39:21] <grrrit-wm>	 (03PS2) 10Cmjohnson: Adding piwik-roots group bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264229 
[03:40:34] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Adding piwik-roots group bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264229 (owner: 10Cmjohnson)
[03:48:45] <icinga-wm>	 RECOVERY - puppet last run on mw2055 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[04:03:14] <icinga-wm>	 PROBLEM - puppet last run on mw2088 is CRITICAL: CRITICAL: puppet fail
[04:30:24] <icinga-wm>	 RECOVERY - puppet last run on mw2088 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[04:38:24] <icinga-wm>	 RECOVERY - puppet last run on bromine is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[04:43:04] <wikibugs>	 6operations, 7Mail: remove or update ea@ alias? - https://phabricator.wikimedia.org/T123286#1936163 (10JKrauska) ok to remove -- now rebuilt as a google group.
[04:43:15] <wikibugs>	 6operations, 7Mail: remove or update ea@ alias? - https://phabricator.wikimedia.org/T123286#1936164 (10JKrauska) a:5JKrauska>3Dzahn
[04:43:23] <wikibugs>	 6operations, 7Mail: remove ea@ alias? - https://phabricator.wikimedia.org/T123286#1936165 (10JKrauska)
[05:16:14] <icinga-wm>	 PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [1000.0]
[05:17:25] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[05:18:15] <icinga-wm>	 RECOVERY - Ulsfo HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[05:19:34] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[05:39:31] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Install nova-network and nova-api on labtestnet2001 [puppet] - 10https://gerrit.wikimedia.org/r/264256 
[05:41:36] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Install nova-network and nova-api on labtestnet2001 [puppet] - 10https://gerrit.wikimedia.org/r/264256 (owner: 10Andrew Bogott)
[05:44:07] <grrrit-wm>	 (03CR) 10Tim Starling: [C: 04-1] [WIP] Implement /w/static.php (034 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263566 (https://phabricator.wikimedia.org/T99096) (owner: 10Krinkle)
[06:31:01] <icinga-wm>	 PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 5 failures
[06:31:02] <icinga-wm>	 PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 2 failures
[06:31:02] <icinga-wm>	 PROBLEM - puppet last run on neodymium is CRITICAL: CRITICAL: Puppet has 2 failures
[06:31:02] <icinga-wm>	 PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:42] <icinga-wm>	 PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:53] <icinga-wm>	 PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Puppet has 2 failures
[06:32:43] <icinga-wm>	 PROBLEM - puppet last run on subra is CRITICAL: CRITICAL: Puppet has 1 failures
[06:34:33] <icinga-wm>	 PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 1 failures
[06:35:02] <icinga-wm>	 PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:35:32] <icinga-wm>	 PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 1 failures
[06:36:11] <icinga-wm>	 PROBLEM - puppet last run on mw2016 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:37:22] <icinga-wm>	 PROBLEM - puppet last run on mw2036 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:53:42] <icinga-wm>	 PROBLEM - puppet last run on bromine is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:55:23] <icinga-wm>	 RECOVERY - puppet last run on subra is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[06:55:41] <icinga-wm>	 RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[06:55:42] <icinga-wm>	 RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[06:55:42] <icinga-wm>	 RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[06:56:02] <icinga-wm>	 RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[06:56:22] <icinga-wm>	 RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[06:56:41] <icinga-wm>	 RECOVERY - puppet last run on mw2016 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[06:56:41] <icinga-wm>	 RECOVERY - puppet last run on mw2208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:11] <icinga-wm>	 RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:41] <icinga-wm>	 RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:51] <icinga-wm>	 RECOVERY - puppet last run on bromine is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures
[06:57:52] <icinga-wm>	 RECOVERY - puppet last run on neodymium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:01] <icinga-wm>	 RECOVERY - puppet last run on mw2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:07:41] <wikibugs>	 6operations, 7Mail: remove shop@wikipedia.org -> board@wikipedia.org - https://phabricator.wikimedia.org/T123672#1936232 (10faidon) Probably a remnant from the days when there was little to no staff. Delete.
[07:08:43] <wikibugs>	 6operations, 7Mail: remove/update office@wikipedia.org alias - https://phabricator.wikimedia.org/T123669#1936239 (10faidon) ``` root@mx1001:~# exim4 -bt dphelps@wikimedia.org dphelps@wikimedia.org is undeliverable: Address dphelps@wikimedia.org does not exist root@mx1001:~# exim4 -bt dphelps@wikipedia.org dphe...
[07:09:47] <wikibugs>	 6operations, 7Mail: remove staff@wikipedia.org - https://phabricator.wikimedia.org/T123670#1936241 (10faidon) a:5JKrauska>3Dzahn Well we (ops) receive noc@, and I've never seen an email to staff@ in that mailbox during my tenure here, nor do I think we should be the ones receiving something addressed to "s...
[07:10:31] <icinga-wm>	 PROBLEM - puppet last run on bromine is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[07:13:51] <wikibugs>	 6operations, 10ContentTranslation-cxserver, 6Services, 10Traffic: Remove cxserver from parsoidcache cluster - https://phabricator.wikimedia.org/T110478#1936246 (10santhosh) Note that cxserver uses RESTBase and does not have any references to parsoid
[08:16:21] <grrrit-wm>	 (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/264125 (owner: 10Alex Monk)
[08:34:07] <grrrit-wm>	 (03PS1) 10Catrope: Add wgScriptPath to InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264260 
[08:40:22] <icinga-wm>	 PROBLEM - puppet last run on mw2060 is CRITICAL: CRITICAL: puppet fail
[08:44:38] <grrrit-wm>	 (03PS2) 10Catrope: Add wgScriptPath to InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264260 
[08:44:40] <grrrit-wm>	 (03PS1) 10Catrope: Add wgScriptPath to InitialiseSettings in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264261 
[08:45:30] <grrrit-wm>	 (03CR) 10Catrope: [C: 032] Add wgScriptPath to InitialiseSettings in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264261 (owner: 10Catrope)
[08:45:57] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add wgScriptPath to InitialiseSettings in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264261 (owner: 10Catrope)
[08:47:42] <icinga-wm>	 PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /a 328434 MB (3% inode=99%)
[08:49:05] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 04-2] "niah, we will just use alsafi's public IP. I am already migrating role::url_downloader to allow for that in https://gerrit.wikimedia.org/r" [dns] - 10https://gerrit.wikimedia.org/r/264208 (https://phabricator.wikimedia.org/T122134) (owner: 10Dzahn)
[08:52:34] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 04-1] "let's just use alsafi's IP. It will never be used for anything else anyway. I am already working on a patch to allow that to happen in rol" [puppet] - 10https://gerrit.wikimedia.org/r/264205 (https://phabricator.wikimedia.org/T122134) (owner: 10Dzahn)
[09:00:12] <icinga-wm>	 RECOVERY - Disk space on stat1002 is OK: DISK OK
[09:06:22] <icinga-wm>	 PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There is one unmerged change in mediawiki_config (dir /srv/mediawiki-staging/).
[09:06:52] <icinga-wm>	 PROBLEM - Unmerged changes on repository mediawiki_config on mira is CRITICAL: There is one unmerged change in mediawiki_config (dir /srv/mediawiki-staging/).
[09:07:02] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:07:31] <icinga-wm>	 RECOVERY - puppet last run on mw2060 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[09:17:22] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.861 second response time
[09:17:59] <grrrit-wm>	 (03PS1) 10Muehlenhoff: Remove special case handling for labs realm [puppet] - 10https://gerrit.wikimedia.org/r/264264 
[09:18:23] <moritzm>	 !log installed git security updates on all jessie systems
[09:18:28] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[09:41:11] <icinga-wm>	 PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: puppet fail
[09:50:42] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:52:51] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 8.691 second response time
[10:01:32] <moritzm>	 !log installed ganeti security updates
[10:01:37] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[10:03:42] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:07:52] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.707 second response time
[10:08:42] <icinga-wm>	 RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:23:03] <icinga-wm>	 PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: puppet fail
[10:27:11] <wikibugs>	 6operations: Reduce rpcbind use - https://phabricator.wikimedia.org/T106477#1936292 (10MoritzMuehlenhoff) A similar issue exists with smbclient: nagios-plugins-standard recommends it, which in turn pulls in a lot of Samba-related libraries. Since we neither use CIFS shares nor any Windows systems, I doubt it is...
[10:35:35] <wikibugs>	 6operations, 5Patch-For-Review, 7Swift: swift upgrade plans - https://phabricator.wikimedia.org/T117972#1936296 (10fgiunchedi) I did some `upgrade` + `dist-upgrade` testing in labs with `swift-upgrade-ms-fe01` and `swift-upgrade-ms-be01` to move precise -> trusty and it seems successful
[10:40:11] <icinga-wm>	 PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 886
[10:45:11] <icinga-wm>	 RECOVERY - check_mysql on db1008 is OK: Uptime: 246351 Threads: 2 Questions: 1566646 Slow queries: 1596 Opens: 1101 Flush tables: 2 Open tables: 397 Queries per second avg: 6.359 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0
[10:48:14] <wikibugs>	 6operations, 5Patch-For-Review, 7Swift: swift upgrade plans - https://phabricator.wikimedia.org/T117972#1936338 (10MoritzMuehlenhoff) Is there a specific feature from 2.5 that we need compared to the stock swift version in jessie?   We could also migrated to standard jessie first (which is still a leap forwa...
[10:50:31] <icinga-wm>	 RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:54:36] <wikibugs>	 6operations: reinstall eqiad memcache servers with jessie - https://phabricator.wikimedia.org/T123711#1936342 (10fgiunchedi) 3NEW
[11:02:11] <wikibugs>	 6operations: Reimage hooft with jessie and rename to bast3001 - https://phabricator.wikimedia.org/T123712#1936351 (10MoritzMuehlenhoff) 3NEW
[11:02:43] <wikibugs>	 6operations: Reimage hooft with jessie and rename to bast3001 - https://phabricator.wikimedia.org/T123712#1936359 (10MoritzMuehlenhoff)
[11:02:45] <wikibugs>	 6operations: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936358 (10MoritzMuehlenhoff)
[11:04:25] <wikibugs>	 6operations, 10Continuous-Integration-Infrastructure, 10Gerrit, 10GitHub-Mirrors, and 2 others: [Task] Redirect unused extensions/ValueView repository to data-values/value-view - https://phabricator.wikimedia.org/T123624#1936360 (10thiemowmde) https://wikitech.wikimedia.org/wiki/Gerrit#Deleting_repositorie...
[11:07:09] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936362 (10hashar)
[11:07:12] <icinga-wm>	 PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet last ran 1 day ago
[11:07:33] <_joe_>	 !log re-enabled puppet on mw1013, restarted HHVM to make it pick up our latest changes
[11:07:38] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[11:07:59] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1931694 (10hashar)
[11:08:37] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1931694 (10hashar) Added T95757 as a blocker. `gallium;wikimedia.org` is Precise and hosts Zuul scheduler / Jenkins.
[11:11:22] <icinga-wm>	 RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:27:17] <wikibugs>	 6operations: Reinstall magnesium with jessie - https://phabricator.wikimedia.org/T123713#1936373 (10MoritzMuehlenhoff) 3NEW
[11:27:37] <wikibugs>	 6operations: Reinstall magnesium with jessie - https://phabricator.wikimedia.org/T123713#1936381 (10MoritzMuehlenhoff)
[11:29:58] <wikibugs>	 6operations, 5Patch-For-Review, 7Swift: swift upgrade plans - https://phabricator.wikimedia.org/T117972#1936382 (10fgiunchedi) no critical 2.5 feature afaict, mostly performance related. migrating all swift to trusty first has the added bonus of getting us in a uniform situation, codfw is trusty already and...
[11:32:39] <wikibugs>	 6operations: Reinstall caesium with jessie - https://phabricator.wikimedia.org/T123714#1936383 (10MoritzMuehlenhoff) 3NEW
[11:34:52] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936390 (10MoritzMuehlenhoff)
[11:34:53] <wikibugs>	 6operations: Reinstall caesium with jessie - https://phabricator.wikimedia.org/T123714#1936391 (10MoritzMuehlenhoff)
[11:35:59] <wikibugs>	 6operations, 7Tracking: Reinstall nitrogen with jessie - https://phabricator.wikimedia.org/T123715#1936392 (10fgiunchedi) 3NEW
[11:37:13] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, 10Wikidata: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1936413 (10thiemowmde) p:5Normal>3Unbreak! Latest commits and new tags we created still do not show up on the GitHub mirror.
[11:38:33] <wikibugs>	 6operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#1936415 (10MoritzMuehlenhoff) 3NEW
[11:38:58] <wikibugs>	 6operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#1936422 (10MoritzMuehlenhoff)
[11:39:36] <wikibugs>	 6operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#1936415 (10MoritzMuehlenhoff)
[11:39:37] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936424 (10MoritzMuehlenhoff)
[11:40:08] <wikibugs>	 6operations, 7Tracking: Reinstall protactinium with jessie - https://phabricator.wikimedia.org/T123720#1936432 (10fgiunchedi) 3NEW
[11:40:59] <wikibugs>	 6operations: Reinstall bast1001 with jessie - https://phabricator.wikimedia.org/T123721#1936438 (10MoritzMuehlenhoff) 3NEW
[11:41:25] <wikibugs>	 6operations: Reinstall bast1001 with jessie - https://phabricator.wikimedia.org/T123721#1936446 (10MoritzMuehlenhoff)
[11:41:26] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1931694 (10MoritzMuehlenhoff)
[11:45:19] <wikibugs>	 6operations, 7Tracking: Reinstall erbium with jessie - https://phabricator.wikimedia.org/T123722#1936447 (10fgiunchedi) 3NEW
[11:46:35] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936458 (10MoritzMuehlenhoff)
[11:46:36] <wikibugs>	 6operations: Reinstall bast1001 with jessie - https://phabricator.wikimedia.org/T123721#1936453 (10MoritzMuehlenhoff) 5Open>3Resolved a:3MoritzMuehlenhoff Not needed, it's in the process of being decommissioned, see T123029
[11:47:05] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1931694 (10MoritzMuehlenhoff)
[11:47:06] <wikibugs>	 6operations, 7Tracking: Reinstall erbium with jessie - https://phabricator.wikimedia.org/T123722#1936460 (10MoritzMuehlenhoff) 5Open>3Resolved a:3MoritzMuehlenhoff Not needed, it's in the process of being decommissioned, see T123029
[11:47:37] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1931694 (10MoritzMuehlenhoff)
[11:47:38] <wikibugs>	 6operations: Reinstall bast1001 with jessie - https://phabricator.wikimedia.org/T123721#1936466 (10MoritzMuehlenhoff) 5Resolved>3Open Argh, wrong tab, I meant to close a diffent bug. Reopen
[11:52:44] <wikibugs>	 6operations: Move bacula director and storage daemon off helium? - https://phabricator.wikimedia.org/T123723#1936475 (10MoritzMuehlenhoff) 3NEW
[11:55:42] <wikibugs>	 6operations, 10Dumps-Generation: Migrate dataset1001 and ms1001 to jessie - https://phabricator.wikimedia.org/T123724#1936489 (10MoritzMuehlenhoff) 3NEW
[11:56:06] <grrrit-wm>	 (03PS1) 10Filippo Giunchedi: swift: reinstall ms-fe300* with trusty [puppet] - 10https://gerrit.wikimedia.org/r/264275 (https://phabricator.wikimedia.org/T117972) 
[11:56:14] <wikibugs>	 6operations, 10Dumps-Generation: Migrate dataset1001 and ms1001 to jessie - https://phabricator.wikimedia.org/T123724#1936498 (10MoritzMuehlenhoff)
[11:56:16] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936497 (10MoritzMuehlenhoff)
[11:57:38] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: reinstall ms-fe300* with trusty [puppet] - 10https://gerrit.wikimedia.org/r/264275 (https://phabricator.wikimedia.org/T117972) (owner: 10Filippo Giunchedi)
[11:58:12] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1931694 (10MoritzMuehlenhoff)
[11:58:15] <wikibugs>	 6operations, 10Beta-Cluster-Infrastructure, 7HHVM: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1936501 (10MoritzMuehlenhoff)
[12:01:47] <wikibugs>	 6operations: Migrate titanium to jessie - https://phabricator.wikimedia.org/T123725#1936508 (10MoritzMuehlenhoff)
[12:02:08] <wikibugs>	 6operations: Migrate titanium to jessie - https://phabricator.wikimedia.org/T123725#1936502 (10MoritzMuehlenhoff)
[12:02:09] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936511 (10MoritzMuehlenhoff)
[12:07:59] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, 10Wikidata: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1936516 (10thiemowmde) I'm waiting for ValueView 0.15.7 to be released for a month now, see https://gerrit.wikimedia.org/r/259760.  I tried to m...
[12:09:03] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, and 2 others: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1936518 (10thiemowmde)
[12:09:49] <wikibugs>	 6operations, 7Tracking: Reinstall protactinium with jessie - https://phabricator.wikimedia.org/T123720#1936526 (10MoritzMuehlenhoff)
[12:12:02] <icinga-wm>	 PROBLEM - salt-minion processes on tin is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[12:15:58] <wikibugs>	 6operations, 6Parsing-Team, 10Parsoid, 6Services: Update ruthenium to Ubuntu 14.04 from Ubuntu 12.04 - https://phabricator.wikimedia.org/T122328#1936535 (10MoritzMuehlenhoff)
[12:16:01] <wikibugs>	 6operations, 10Dumps-Generation: Migrate dataset1001 and ms1001 to jessie - https://phabricator.wikimedia.org/T123724#1936534 (10MoritzMuehlenhoff)
[12:16:14] <wikibugs>	 6operations, 6Parsing-Team, 10Parsoid, 6Services: Update ruthenium to Ubuntu 14.04 from Ubuntu 12.04 - https://phabricator.wikimedia.org/T122328#1901606 (10MoritzMuehlenhoff)
[12:16:29] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936538 (10MoritzMuehlenhoff)
[12:16:30] <wikibugs>	 6operations, 6Parsing-Team, 10Parsoid, 6Services: Update ruthenium to Ubuntu 14.04 from Ubuntu 12.04 - https://phabricator.wikimedia.org/T122328#1901606 (10MoritzMuehlenhoff)
[12:17:18] <wikibugs>	 6operations, 10Analytics: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#1936541 (10MoritzMuehlenhoff)
[12:17:20] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1931694 (10MoritzMuehlenhoff)
[12:17:58] <hashar>	 !log restarting Jenkins for plugins updates
[12:18:02] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[12:18:40] <godog>	 ms-fe3001 is reinstalling so icinga will weep shortly
[12:19:53] <wikibugs>	 6operations, 10Dumps-Generation: Migrate dataset1001 and ms1001 to jessie - https://phabricator.wikimedia.org/T123724#1936547 (10ArielGlenn) p:5Triage>3Normal a:3ArielGlenn
[12:21:09] <wikibugs>	 6operations: Migrate hydrogen/chromium to jessie - https://phabricator.wikimedia.org/T123727#1936549 (10MoritzMuehlenhoff) 3NEW
[12:21:29] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936556 (10MoritzMuehlenhoff)
[12:21:30] <wikibugs>	 6operations: Migrate hydrogen/chromium to jessie - https://phabricator.wikimedia.org/T123727#1936557 (10MoritzMuehlenhoff)
[12:24:31] <icinga-wm>	 PROBLEM - salt-minion processes on mira is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[12:35:28] <wikibugs>	 6operations, 10OTRS: upgrade iodine to jessie or find a new host with jessie for OTRS - https://phabricator.wikimedia.org/T105125#1936562 (10MoritzMuehlenhoff)
[12:35:30] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936561 (10MoritzMuehlenhoff)
[12:37:51] <icinga-wm>	 RECOVERY - salt-minion processes on tin is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[12:38:57] <wikibugs>	 6operations: Upgrade fluorine to trusty/jessie - https://phabricator.wikimedia.org/T123728#1936565 (10MoritzMuehlenhoff) 3NEW
[12:39:35] <wikibugs>	 6operations: Upgrade fluorine to trusty/jessie - https://phabricator.wikimedia.org/T123728#1936573 (10MoritzMuehlenhoff)
[12:39:36] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936572 (10MoritzMuehlenhoff)
[12:41:51] <wikibugs>	 6operations: Migrate argon to jessie - https://phabricator.wikimedia.org/T123729#1936574 (10MoritzMuehlenhoff) 3NEW
[12:44:15] <wikibugs>	 6operations: Migrate puppetmaster/backends to jessie - https://phabricator.wikimedia.org/T123730#1936586 (10MoritzMuehlenhoff) 3NEW
[12:44:33] <wikibugs>	 6operations: Migrate puppetmaster/backends to jessie - https://phabricator.wikimedia.org/T123730#1936594 (10MoritzMuehlenhoff)
[12:44:34] <wikibugs>	 6operations: Migrate argon to jessie - https://phabricator.wikimedia.org/T123729#1936595 (10MoritzMuehlenhoff)
[12:48:26] <wikibugs>	 6operations: Migrate labsdb1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#1936600 (10MoritzMuehlenhoff) 3NEW
[12:50:12] <icinga-wm>	 RECOVERY - salt-minion processes on mira is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[12:51:33] <wikibugs>	 6operations: Migrate nitrogen to jessie - https://phabricator.wikimedia.org/T123732#1936607 (10MoritzMuehlenhoff) 3NEW
[12:51:55] <wikibugs>	 6operations: Migrate nitrogen to jessie - https://phabricator.wikimedia.org/T123732#1936615 (10MoritzMuehlenhoff)
[12:51:57] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936614 (10MoritzMuehlenhoff)
[12:51:57] <wikibugs>	 6operations: Migrate labsdb1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#1936616 (10MoritzMuehlenhoff)
[12:56:15] <wikibugs>	 6operations: Migrate carbon to jessie - https://phabricator.wikimedia.org/T123733#1936623 (10MoritzMuehlenhoff) 3NEW
[12:56:34] <wikibugs>	 6operations: Migrate carbon to jessie - https://phabricator.wikimedia.org/T123733#1936632 (10MoritzMuehlenhoff)
[12:56:36] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936631 (10MoritzMuehlenhoff)
[13:00:52] <wikibugs>	 6operations: Migrate pool counters to trusty/jessie - https://phabricator.wikimedia.org/T123734#1936635 (10MoritzMuehlenhoff) 3NEW
[13:02:23] <wikibugs>	 6operations: Migrate pool counters to trusty/jessie - https://phabricator.wikimedia.org/T123734#1936643 (10MoritzMuehlenhoff) Also, consider T123723 for helium
[13:03:04] <wikibugs>	 6operations: Migrate pool counters to trusty/jessie - https://phabricator.wikimedia.org/T123734#1936646 (10MoritzMuehlenhoff)
[13:03:05] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1936645 (10MoritzMuehlenhoff)
[13:10:32] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:14:51] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 7.594 second response time
[13:18:59] <Steinsplitter>	 can somone merge https://gerrit.wikimedia.org/r/#/c/263842/  ?   it is not deployment time, but at 00:00 i am not here O_O
[13:21:40] <ostriches>	 Is there a phab ticket for it?
[13:24:53] <Steinsplitter>	 for addinga import source? no.
[13:26:08] <grrrit-wm>	 (03CR) 10Chad: [C: 032] adding w:hewiki to wgImportSources. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263842 (owner: 10Steinsplitter)
[13:26:16] <ostriches>	 Let's do it :)
[13:26:37] <Steinsplitter>	 cool, thanks :-)
[13:27:00] <grrrit-wm>	 (03Merged) 10jenkins-bot: adding w:hewiki to wgImportSources. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263842 (owner: 10Steinsplitter)
[13:27:52] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL: CRITICAL: 87.50% of data above the critical threshold [24.0]
[13:28:32] <logmsgbot>	 !log demon@tin Synchronized wmf-config/InitialiseSettings.php: w:he as import source for commonswiki (duration: 00m 49s)
[13:28:36] <ostriches>	 Steinsplitter: All done ^
[13:28:37] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[13:28:46] <Steinsplitter>	 thx
[13:28:56] <ostriches>	 yw
[13:29:21] <icinga-wm>	 RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge.
[13:29:41] <icinga-wm>	 RECOVERY - Unmerged changes on repository mediawiki_config on mira is OK: No changes to merge.
[13:30:27] <grrrit-wm>	 (03PS3) 10Cmjohnson: Adding piwik-roots group bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264229 
[13:33:49] <grrrit-wm>	 (03PS1) 10Filippo Giunchedi: admin: report nonexistant users and duplicate gid [puppet] - 10https://gerrit.wikimedia.org/r/264281 
[13:34:32] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK: OK: Less than 50.00% above the threshold [16.0]
[13:35:24] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 032 V: 032] admin: report nonexistant users and duplicate gid [puppet] - 10https://gerrit.wikimedia.org/r/264281 (owner: 10Filippo Giunchedi)
[13:43:02] <icinga-wm>	 PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 1 failures
[13:44:31] <icinga-wm>	 PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: Puppet has 1 failures
[13:44:51] <icinga-wm>	 PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: puppet fail
[13:46:42] <grrrit-wm>	 (03PS3) 10Muehlenhoff: Prevent access to hidden directories [puppet] - 10https://gerrit.wikimedia.org/r/217794 (https://phabricator.wikimedia.org/T94570) 
[13:47:02] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:51:21] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.824 second response time
[13:53:12] <ostriches>	 moritzm: Hehe, wasn't sure if anyone was still interested which is why I bumped it :)
[13:59:29] <moritzm>	 ostriches: i was lurking in the darker corners of my inbox :-)
[14:02:12] <icinga-wm>	 PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures
[14:03:52] <mark>	 !log set sync_speed_min to 5000 for md126 on labstore1001
[14:03:58] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[14:04:21] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:07:16] <grrrit-wm>	 (03PS4) 10Cmjohnson: Adding piwik-roots group bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264229 
[14:08:02] <icinga-wm>	 RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[14:08:32] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.351 second response time
[14:08:42] <icinga-wm>	 RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[14:09:49] <chasemp>	 !log phab restart phd (reports as not running in phab itself) seems ok now
[14:09:53] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[14:10:32] <icinga-wm>	 RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[14:12:02] <icinga-wm>	 PROBLEM - salt-minion processes on tin is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[14:12:13] <ostriches>	 chasemp: I saw that too (like 30-45m ago?) but they kicked themselves back into shape
[14:12:43] <chasemp>	 I think there is an illusive bug where the daemons stick on a task and phab thinks they are missing for a bit
[14:13:20] <chasemp>	 I've seen it a few times that were suspect but since it's all stateful on restart should be ok either way
[14:13:25] <mark>	 !log Temporarily paused md126 RAID check on labstore1001 (sync_action idle)
[14:13:29] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[14:15:11] <icinga-wm>	 PROBLEM - DPKG on labmon1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[14:17:02] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:19:22] <icinga-wm>	 RECOVERY - DPKG on labmon1001 is OK: All packages OK
[14:20:41] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[14:20:41] <icinga-wm>	 PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[14:21:45] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] Adding piwik-roots group bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264229 (owner: 10Cmjohnson)
[14:23:21] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 0.924 second response time
[14:24:31] <icinga-wm>	 PROBLEM - Mobile HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[14:24:41] <icinga-wm>	 PROBLEM - salt-minion processes on mira is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[14:26:33] <icinga-wm>	 RECOVERY - Mobile HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[14:27:02] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[14:27:02] <icinga-wm>	 RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[14:27:42] <icinga-wm>	 RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:29:52] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:32:26] <wikibugs>	 6operations: Replace labstore1002 RAID card - https://phabricator.wikimedia.org/T123740#1936716 (10chasemp) 3NEW a:3Cmjohnson
[14:32:40] <wikibugs>	 6operations, 10ops-eqiad: Replace labstore1002 RAID card - https://phabricator.wikimedia.org/T123740#1936716 (10chasemp)
[14:37:01] <wikibugs>	 6operations, 7HHVM: Switch HAT appservers to trusty's ICU (or newer) - https://phabricator.wikimedia.org/T86096#1936727 (10MoritzMuehlenhoff) Since we're using precise's version of icu, working on this is part of ops's quarterly goal to get rid of precise ;-)  I don't really understand how it's linked to icu 4...
[14:38:23] <icinga-wm>	 RECOVERY - salt-minion processes on tin is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[14:41:12] <grrrit-wm>	 (03PS1) 10Cmjohnson: Adding admin role to piwik.yaml bug: task# T122325 [puppet] - 10https://gerrit.wikimedia.org/r/264287 
[14:42:55] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).
[14:43:34] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).
[14:43:45] <moritzm>	 cmjohnson1: ^
[14:44:12] <cmjohnson1>	 mortizm...yep, thanks!
[14:45:43] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge.
[14:47:14] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge.
[14:50:44] <icinga-wm>	 RECOVERY - salt-minion processes on mira is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[14:51:11] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 04-1] Adding admin role to piwik.yaml bug: task# T122325 (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/264287 (owner: 10Cmjohnson)
[14:52:01] <wikibugs>	 6operations: Reimage hooft with jessie and rename to bast3001 - https://phabricator.wikimedia.org/T123712#1936742 (10Southparkfan)
[14:54:19] <godog>	 !log reimage ms-fe3002 with trusty
[14:54:24] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[14:56:23] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.474 second response time
[15:01:14] <grrrit-wm>	 (03PS2) 10Cmjohnson: Adding admin role to piwik.yaml [puppet] - 10https://gerrit.wikimedia.org/r/264287 
[15:03:16] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:07:15] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.669 second response time
[15:11:55] <icinga-wm>	 PROBLEM - salt-minion processes on tin is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[15:18:35] <grrrit-wm>	 (03PS3) 10Filippo Giunchedi: Adding admin role to piwik.yaml [puppet] - 10https://gerrit.wikimedia.org/r/264287 (https://phabricator.wikimedia.org/T122325) (owner: 10Cmjohnson)
[15:18:46] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Adding admin role to piwik.yaml [puppet] - 10https://gerrit.wikimedia.org/r/264287 (https://phabricator.wikimedia.org/T122325) (owner: 10Cmjohnson)
[15:24:25] <icinga-wm>	 PROBLEM - salt-minion processes on mira is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[15:25:17] <wikibugs>	 6operations, 6Analytics-Kanban, 10Wikipedia-iOS-App-Product-Backlog, 6Zero, and 2 others: Request one server to suport piwik analytics - https://phabricator.wikimedia.org/T116312#1936788 (10Cmjohnson)
[15:25:20] <wikibugs>	 10Ops-Access-Requests, 6operations, 10Analytics, 5Patch-For-Review: add mforns, milimetric, nuria,ottomata, madhuvishy and joal to piwik-roots - https://phabricator.wikimedia.org/T122325#1936785 (10Cmjohnson) 5Open>3Resolved a:3Cmjohnson The admin role has been added to and group added to piwki.yaml
[15:31:42] <grrrit-wm>	 (03PS2) 10Filippo Giunchedi: icinga: report atlas measurement url in alert text [puppet] - 10https://gerrit.wikimedia.org/r/263840 
[15:32:15] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 032 V: 032] icinga: report atlas measurement url in alert text [puppet] - 10https://gerrit.wikimedia.org/r/263840 (owner: 10Filippo Giunchedi)
[15:34:48] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 04-1] "https://puppet-compiler.wmflabs.org/1616/chromium.wikimedia.org/ says it's mostly what you 'd expect, but I 'll test it in labs as well fi" [puppet] - 10https://gerrit.wikimedia.org/r/264059 (owner: 10Alexandros Kosiaris)
[15:37:45] <wikibugs>	 6operations, 7Tracking: Reinstall nitrogen with jessie - https://phabricator.wikimedia.org/T123715#1936844 (10fgiunchedi)
[15:38:05] <icinga-wm>	 RECOVERY - salt-minion processes on tin is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[15:41:01] <godog>	 !log reimage ms-be3001 with trusty
[15:41:06] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:41:25] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:43:35] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.416 second response time
[15:49:46] <icinga-wm>	 RECOVERY - salt-minion processes on mira is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[15:59:10] <wikibugs>	 6operations: Update libzmq3/pyzmq - https://phabricator.wikimedia.org/T106093#1936903 (10MoritzMuehlenhoff) 5Open>3declined Closing this bug. I started to work on this a few months ago, but stopped at some point since it would have lots of effects on reverse dependencies. Also, in the mean time Ariel fixed s...
[16:00:04] <wikibugs>	 6operations: Meta task for various security updates - https://phabricator.wikimedia.org/T96545#1936910 (10MoritzMuehlenhoff) 5Open>3Resolved Old meta task, no longer used by now.
[16:00:54] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:01:09] <godog>	 !log bounce hhvm on mw1129 / mw1204
[16:01:15] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[16:02:08] <icinga-wm>	 RECOVERY - Apache HTTP on mw1129 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 2.676 second response time
[16:02:40] <icinga-wm>	 RECOVERY - HHVM rendering on mw1129 is OK: HTTP OK: HTTP/1.1 200 OK - 69773 bytes in 0.969 second response time
[16:03:08] <icinga-wm>	 RECOVERY - HHVM rendering on mw1204 is OK: HTTP OK: HTTP/1.1 200 OK - 69773 bytes in 0.211 second response time
[16:03:09] <icinga-wm>	 RECOVERY - Apache HTTP on mw1204 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.036 second response time
[16:05:51] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] "tested in deployment-prep. works fine both in trusty and jessie, merging" [puppet] - 10https://gerrit.wikimedia.org/r/264059 (owner: 10Alexandros Kosiaris)
[16:07:19] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.938 second response time
[16:07:33] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Make role::url_downloader unparameterized [puppet] - 10https://gerrit.wikimedia.org/r/264059 
[16:07:35] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: Revert "Revert "Add the LVS blocks to url_downloader"" [puppet] - 10https://gerrit.wikimedia.org/r/207490 
[16:07:52] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Make role::url_downloader unparameterized [puppet] - 10https://gerrit.wikimedia.org/r/264059 (owner: 10Alexandros Kosiaris)
[16:10:30] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: "https://gerrit.wikimedia.org/r/#/c/264059/ has been merged and seems to work fine. Let's just apply the role url_downloader on alsafi and " [puppet] - 10https://gerrit.wikimedia.org/r/264205 (https://phabricator.wikimedia.org/T122134) (owner: 10Dzahn)
[16:13:48] <icinga-wm>	 PROBLEM - salt-minion processes on tin is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[16:17:59] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:20:09] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 8.812 second response time
[16:20:16] <grrrit-wm>	 (03CR) 10Muehlenhoff: "Alex confirmed the key in PGP-signed mail:" [puppet] - 10https://gerrit.wikimedia.org/r/264125 (owner: 10Alex Monk)
[16:23:58] <icinga-wm>	 PROBLEM - salt-minion processes on mira is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[16:24:40] <grrrit-wm>	 (03PS2) 10Alex Monk: Add my yubikey [puppet] - 10https://gerrit.wikimedia.org/r/264125 
[16:26:14] <grrrit-wm>	 (03CR) 10Muehlenhoff: [C: 032 V: 032] Add my yubikey [puppet] - 10https://gerrit.wikimedia.org/r/264125 (owner: 10Alex Monk)
[16:35:41] <Krenair>	 moritzm, did you run puppet on a bastion?
[16:36:49] <icinga-wm>	 RECOVERY - salt-minion processes on tin is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[16:36:49] <icinga-wm>	 PROBLEM - swift-account-auditor on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-auditor
[16:36:59] <godog>	 that's me ^ silence expired
[16:37:00] <icinga-wm>	 PROBLEM - swift-account-reaper on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-reaper
[16:37:19] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:37:20] <icinga-wm>	 PROBLEM - swift-account-replicator on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-replicator
[16:39:29] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 8.149 second response time
[16:40:54] <Krenair>	 looks like I can go to all bastions except bast1001
[16:49:39] <icinga-wm>	 RECOVERY - salt-minion processes on mira is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[16:50:10] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:52:15] <Krenair>	 now it works
[16:54:30] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.082 second response time
[16:55:08] <godog>	 poor bromine :( load average: 151.03, 150.73, 150.41
[16:55:09] <icinga-wm>	 RECOVERY - Host ms-be2007 is UP: PING OK - Packet loss = 0%, RTA = 38.07 ms
[16:56:23] <Krenair>	 godog, I imagine it's apache serving annual.wikimedia.org/15.wikipedia.org?
[16:57:29] <godog>	 Krenair: you are indeed correct sir
[16:57:31] <moritzm>	 Krenair: no, the automatic puppet runs will fix that
[16:58:09] <wikibugs>	 6operations, 10ops-codfw: ms-be2007 - System halted!Error:  Integrated RAID - https://phabricator.wikimedia.org/T122844#1937003 (10Papaul) a:5Papaul>3fgiunchedi @fgiunchedi new controller is in place.
[16:59:30] <icinga-wm>	 PROBLEM - configured eth on ms-be2007 is CRITICAL: Connection refused by host
[16:59:30] <icinga-wm>	 PROBLEM - swift-object-replicator on ms-be2007 is CRITICAL: Connection refused by host
[16:59:39] <icinga-wm>	 PROBLEM - puppet last run on ms-be2007 is CRITICAL: Connection refused by host
[16:59:49] <icinga-wm>	 PROBLEM - swift-container-server on ms-be2007 is CRITICAL: Connection refused by host
[16:59:50] <icinga-wm>	 PROBLEM - salt-minion processes on ms-be2007 is CRITICAL: Connection refused by host
[17:00:00] <icinga-wm>	 PROBLEM - dhclient process on ms-be2007 is CRITICAL: Connection refused by host
[17:00:00] <icinga-wm>	 PROBLEM - swift-container-updater on ms-be2007 is CRITICAL: Connection refused by host
[17:00:10] <icinga-wm>	 PROBLEM - swift-account-auditor on ms-be2007 is CRITICAL: Connection refused by host
[17:00:19] <icinga-wm>	 PROBLEM - Check size of conntrack table on ms-be2007 is CRITICAL: Connection refused by host
[17:00:19] <icinga-wm>	 PROBLEM - swift-object-server on ms-be2007 is CRITICAL: Connection refused by host
[17:00:29] <icinga-wm>	 PROBLEM - swift-object-auditor on ms-be2007 is CRITICAL: Connection refused by host
[17:00:31] <icinga-wm>	 PROBLEM - swift-account-reaper on ms-be2007 is CRITICAL: Connection refused by host
[17:00:50] <icinga-wm>	 PROBLEM - swift-account-replicator on ms-be2007 is CRITICAL: Connection refused by host
[17:00:50] <icinga-wm>	 PROBLEM - swift-container-auditor on ms-be2007 is CRITICAL: Connection refused by host
[17:00:51] <icinga-wm>	 PROBLEM - very high load average likely xfs on ms-be2007 is CRITICAL: Connection refused by host
[17:01:00] <icinga-wm>	 PROBLEM - swift-object-updater on ms-be2007 is CRITICAL: Connection refused by host
[17:01:00] <icinga-wm>	 PROBLEM - DPKG on ms-be2007 is CRITICAL: Connection refused by host
[17:01:10] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:01:10] <icinga-wm>	 PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /home 37482 MB (3% inode=98%)
[17:01:10] <icinga-wm>	 PROBLEM - swift-account-server on ms-be2007 is CRITICAL: Connection refused by host
[17:01:20] <icinga-wm>	 PROBLEM - Disk space on ms-be2007 is CRITICAL: Connection refused by host
[17:01:39] <icinga-wm>	 PROBLEM - swift-container-replicator on ms-be2007 is CRITICAL: Connection refused by host
[17:01:40] <icinga-wm>	 PROBLEM - RAID on ms-be2007 is CRITICAL: Connection refused by host
[17:07:40] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 8.036 second response time
[17:07:59] <icinga-wm>	 PROBLEM - Host ms-be2007 is DOWN: PING CRITICAL - Packet loss = 100%
[17:10:14] <bblack>	 annual/15 should be mostly static content and cacheable right?
[17:10:20] <icinga-wm>	 RECOVERY - Host ms-be2007 is UP: PING OK - Packet loss = 0%, RTA = 36.75 ms
[17:11:04] <godog>	 yeah, perhaps size limit? homepage.js is 770k
[17:11:49] <bblack>	 ick
[17:12:34] <bblack>	 our misc-cluster size rules are stream-mode (but still cache) for >=1MB, and no-cache for >=10MB
[17:13:42] <godog>	 heh, so far the only difference I've found, all other assets are getting hits in varnish
[17:16:21] <bblack>	 yeah
[17:16:24] <bblack>	    72 RxURL        c /assets/img/faces_cards/shirley.jpg
[17:16:24] <bblack>	    72 TxHeader     c X-Cache: cp1056 miss(0), cp1070 frontend hit(202)
[17:16:33] <bblack>	 most of the requests are hits like that
[17:17:42] <bblack>	 requests for /piwik.php on piwik.wikimedia.org, with arguments related to the 15 campaign, are some of the heaviest hits on misc right now though, and not cacheable
[17:18:08] <bblack>	 I'm not sure what this is or how it's related, but piwik isn't on bromine
[17:19:51] <bblack>	 hmmm most js is cacheable, e.g.
[17:19:51] <bblack>	    83 RxURL        c /assets/js/masonry.js
[17:19:51] <bblack>	    83 TxHeader     c X-Cache: cp1056 hit(5), cp1070 frontend hit(97)
[17:20:08] <bblack>	 but homepage.js is not, and it's heavy on the top URLs list:
[17:20:09] <bblack>	    77 RxURL        c /assets/js/homepage.js
[17:20:09] <bblack>	    77 TxHeader     c X-Cache: cp1069 pass(0), cp1070 frontend pass(0)
[17:20:28] <godog>	 indeed, re: homepage.js I'm still suspecting the size but the size checks seem correct
[17:20:58] <bblack>	 and it's slow to load at all, because it's blocking and coalescing too
[17:21:08] <bblack>	 hmmmm
[17:22:20] <bblack>	 I've got a curl fetch of it from home that's been hung for nearly 2 minutes now without downloading any content :/
[17:22:39] <godog>	 nothing specific to homepage.js in apache or in the headers sent back from bromine afaics
[17:23:58] <bblack>	 <head><title>504 Gateway Time-out</title></head>
[17:24:03] <bblack>	 on my fetch
[17:25:39] <godog>	 yeah I got that sometime too from a browser
[17:26:50] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:28:00] <greg-g>	 hmm?
[17:28:31] <greg-g>	 (ignore me, that looks like an old alarm, as long as it's known ;) )
[17:28:47] <ostriches>	 that's been flapping all morning
[17:29:09] <bblack>	 it's slow to load even locally with curl to itself on bromine
[17:29:27] <bblack>	 but you're right, no real diff in headers
[17:29:35] <greg-g>	 ostriches: kk
[17:31:53] <wikibugs>	 6operations, 7Mail: Move most (all?) exim personal aliases to OIT - https://phabricator.wikimedia.org/T122144#1937078 (10Dzahn)
[17:31:54] <wikibugs>	 6operations, 7Mail: remove/update office@wikipedia.org alias - https://phabricator.wikimedia.org/T123669#1937076 (10Dzahn) 5Open>3Resolved Thanks @faidon.  done   ``` -office:         dphelps  ```
[17:35:38] <mutante>	 bblack: oh, so the piwik code is in the 15.wp /annual code to create stats and is using the new wmf piwik install at piwik.wikimedia.org
[17:35:44] <mutante>	 nuria: ^
[17:35:57] <nuria>	 hola mutante 
[17:36:28] <Krenair>	 ori, ^
[17:37:15] <nuria>	 bblack: why are those requests not cacheable?
[17:37:32] <nuria>	 bblack: we should only be requesting the beacon
[17:37:53] <nuria>	 bblack: ah wait, maybe we need to add caching headers to the apache that serves piwik trhat we do not have?
[17:39:27] <bblack>	 don't know
[17:39:29] <nuria>	 cc mutante
[17:39:34] <bblack>	 right now I don't think piwik is the problem, just notable in stats
[17:40:06] <nuria>	 bblack: regardless, our js snippet should have cache headers cc milimetric 
[17:40:33] <milimetric>	 yeah, we got close to 1 million requests in the last day for the wikipedia 15 site
[17:40:36] <milimetric>	 goodness gracious
[17:41:58] <nuria>	 bblack, milimetric ; as far as i can see piwik.js is being served with a last-modified
[17:42:14] <mutante>	 eh, and i just merged that change to enable caching for annual.wm yesterday
[17:42:16] <bblack>	 not piwik.js, piwik.php
[17:42:21] <milimetric>	 (reading up, give me a sec to catch up)
[17:42:39] <bblack>	 mutante: a lot of the 15.wp.o hits are cached, it's just not caching homepage.js for some reason
[17:42:41] <nuria>	 bblack: but who requests piwiki.php?
[17:43:16] <nuria>	 bblack: beacon is piwik.js  and that seems to be cached correctly
[17:43:35] <nuria>	 bblack: let me see if i can figure out in piwik docs where those requests come from
[17:44:10] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 9.946 second response time
[17:44:33] <nuria>	 bblack: that is a server side tracker so might be request by non js clients ahhhh
[17:44:36] <nuria>	 bblack: bots!
[17:44:44] <nuria>	 bblack: it's the crawlers who are executing that
[17:44:51] <nuria>	 bblack: * i think*
[17:45:14] <nuria>	 cc milimetric , makes sense , right?
[17:45:51] <nuria>	 bblack, milimetric , i think this is it: <noscript><p><img src="piwik.wikimedia.org/piwik.php?idsite=2" style="border:0;" alt="" /></p></noscript>
[17:46:07] <nuria>	 bblack: we just need to do away with no-js tracking, cc milimetric 
[17:46:28] <mutante>	 ?pk_campaign=kiwi&pk_keyword=anon
[17:47:00] <mutante>	 so much of that pk_campaign stuff
[17:47:02] <milimetric>	 agreed, if piwik.php is causing load, we can remove the <noscript> tag there, I can do that, just let me know if you think that'll help
[17:47:24] <bblack>	 I don't think it's causing the primary problem I'm looking at now, but it's probably not ideal
[17:47:46] <milimetric>	 ok, I'll remove it anyway to be safe, we can add it later
[17:48:00] <nuria>	 milimetric: thank you, i think is best
[17:48:04] <bblack>	 the primary problem right now is that 15.wp.o/assets/js/homepage.js is very slow/unreliable to load, because it's 770K and varnish is not caching it for some unknown reason, unlike most of the assets (images, js) on that site
[17:48:06] <nuria>	 not to have it
[17:48:42] <nuria>	 bblack: let me look at the js there, this site... ahem.. was clearly not done for speed
[17:49:41] <mutante>	 intro animation ..
[17:50:15] <mutante>	 there is some kind of animation and lots of data for that
[17:50:27] <mutante>	 maybe we can disable that?
[17:50:41] <icinga-wm>	 PROBLEM - Static Bugzilla HTTP on bromine is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:51:44] <bblack>	 it might be best to disable it for now, if the site operates fine without it
[17:51:48] <bblack>	 I don't know that it does tbh
[17:52:01] <nuria>	 bblack: i know, 
[17:52:20] <nuria>	 bblack: it's the overlay of a rotating image on top of the background image
[17:52:35] <nuria>	 bblack: let me see if it can be removed entirely
[17:52:37] <milimetric>	 yeah, it's pretty key to the design (I think it's weird, but it's clearly key)
[17:52:49] <icinga-wm>	 RECOVERY - Static Bugzilla HTTP on bromine is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 590 bytes in 4.159 second response time
[17:53:11] <ori>	 Krenair: you pinged?
[17:53:36] <Krenair>	 ori, yeah, there were some caching issues with piwik which I thought you should know about
[17:53:46] <ori>	 can someone summarize?
[17:53:47] <bblack>	 when varnish requests homepage.js from bromine, most of the responses look something like:
[17:53:50] <bblack>	   129 RxProtocol   b HTTP/1.1
[17:53:52] <bblack>	   129 RxStatus     b 200
[17:53:55] <bblack>	   129 RxResponse   b OK
[17:53:57] <bblack>	   129 RxHeader     b Date: Fri, 15 Jan 2016 17:52:48 GMT
[17:54:00] <bblack>	   129 RxHeader     b Server: Apache
[17:54:02] <bblack>	   129 RxHeader     b Vary: X-Forwarded-Proto,Accept-Encoding
[17:54:05] <bblack>	   129 RxHeader     b Strict-Transport-Security: max-age=31536000
[17:54:07] <bblack>	   129 RxHeader     b Last-Modified: Fri, 15 Jan 2016 02:23:33 GMT
[17:54:09] <nuria>	 bblack: let's get rid of it, it is not loading anyways
[17:54:10] <bblack>	   129 RxHeader     b ETag: "bd304-529561704ff76-gzip"
[17:54:12] <bblack>	   129 RxHeader     b Accept-Ranges: bytes
[17:54:14] <nuria>	 cc milimetric 
[17:54:15] <bblack>	   129 RxHeader     b Content-Encoding: gzip
[17:54:17] <bblack>	   129 RxHeader     b Connection: close
[17:54:20] <bblack>	   129 RxHeader     b Content-Type: application/javascript
[17:54:22] <bblack>	   129 Fetch_Body   b 5(eof) cls -1 mklen 1
[17:54:25] <bblack>	   129 BackendClose b be_bromine
[17:54:30] <mutante>	 ori: bromine hosts 15.wp and is under heavy load because homepage.js is not cacheable and we get lots of hits
[17:54:47] <bblack>	 ori: piwik isn't really critical right now, but the main issue is 15.wp.o/assets/js/homepage.js is not caching in varnish (whereas other js on that site does), for unknown reasons, and the requests for it are overwhelming bromine and probably making it worse
[17:54:57] <milimetric>	 mutante: ori: but that's not related to piwik (homepage.js)
[17:55:01] <ori>	 taking a look
[17:55:13] <mutante>	 milimetric: no, homepage.js does an animation for design reasons
[17:55:16] <milimetric>	 mutante: ori: I'm removing the noscript that's hitting piwik.php right now
[17:55:33] <milimetric>	 so nothing further to look at for piwik, piwik.js is cached properly
[17:55:35] <ori>	 why? let's fix one problem at a time, and start with the most severe one
[17:55:48] <mutante>	 ori: var headerContainer=document.getElementById("header-intro_animation");bodymovin.loadAnimation   
[17:56:10] <nuria>	 ori: Let's remove homepage.js 
[17:56:19] <mutante>	 that file is pretty large because it has a whole lot of animation data
[17:56:36] <nuria>	 ori: for now, it doesn't load anyways
[17:56:36] <ori>	 i agree with nuria, let's remove it as a first step
[17:56:38] <ori>	 yeah
[17:56:46] <ori>	 i'll live-hack it on the host
[17:56:49] <bblack>	    30 Debug        c Write error, retval = -1, len = 8096, errno = Broken pipe
[17:56:50] <nuria>	 ori, milimetric , bblack : submitting patch
[17:56:58] <mutante>	 i can do it, already on the host
[17:56:59] <nuria>	 let me test locally
[17:57:01] <milimetric>	 nuria: I'm doing that...
[17:57:10] <nuria>	 mutante: no, i will remote it on teh code
[17:57:16] <bblack>	 something is definitely wrong with the fetch of it from varnish->apache, but I don't know if that's the root cause, or just a symptom of high load now
[17:57:19] <milimetric>	 nuria: I already did it
[17:57:21] <nuria>	 mutante: and we can redeploy, let me test
[17:57:22] <mutante>	 nuria: ok, thanks
[17:57:32] <ori>	 i can't even ssh
[17:57:39] <ori>	 oh, i got in
[17:57:59] <mutante>	  /srv/org/wikimedia/annualreport/2015/assets/js# 
[17:58:28] <mutante>	 deployment = git pull origin
[17:58:34] <bblack>	 this could be related to TE: chunked and/or gzip too on the apache->varnish response of this 770KB js file
[17:58:48] <bblack>	 so another experiment might be to disable some things related to that on the apache
[17:58:50] <nuria>	 milimetric: the piwki.php?
[17:58:56] <nuria>	 milimetric: or the homepage.js?
[17:59:00] <nuria>	 milimetric: both?
[17:59:00] <milimetric>	 nuria: https://gerrit.wikimedia.org/r/264301
[17:59:00] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0]
[17:59:05] <milimetric>	 nuria: piwik.php
[17:59:24] <ori>	 ok, i'm on bromine -- bblack/mutante let me look for a minute
[17:59:47] <ori>	 nuria: hold the patch for a sec
[18:00:00] <nuria>	 ori: k
[18:00:02] <milimetric>	 nuria / ori: I've removed the noscript and self-merged: https://gerrit.wikimedia.org/r/#/c/264301/
[18:00:14] <milimetric>	 (so when we figure out what to do with homepage.js we can deploy)
[18:00:24] <bblack>	 ori: all you, I'm in readonly mode
[18:00:39] <bblack>	 I'm really starting to lean towards some gzip-output problem on bromine for that file though
[18:00:55] <mutante>	 ori: all you, do you want to git pull to deploy milimetric's change?
[18:00:59] <bblack>	 in the access.log, it shows all kinds of random content-length sizes for these supposedly successful requests
[18:01:29] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[18:02:02] <milimetric>	 bblack: that file is crazy, it's like a hard-coded animation where a bunch of images are drawn point by point
[18:02:38] <bblack>	 it is a real static file though right?
[18:02:39] <grrrit-wm>	 (03PS1) 10Ori.livneh: Mirror homepage.js from 15.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264302 
[18:02:50] <nuria>	 bblack: yes, a 700K file
[18:02:53] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Mirror homepage.js from 15.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264302 (owner: 10Ori.livneh)
[18:02:55] <milimetric>	 would it make sense that different browsers are asking for different kinds of compression?  I'm seeing quite a bit of variability in the browser versions that are hitting the site
[18:02:58] <nuria>	 bblack: with a bunch of pixels
[18:02:59] <milimetric>	 bblack: yes, static
[18:03:17] <grrrit-wm>	 (03Merged) 10jenkins-bot: Mirror homepage.js from 15.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264302 (owner: 10Ori.livneh)
[18:03:19] <nuria>	 bblack: so it doesn't explain why it is not cached
[18:03:55] <bblack>	 nuria: well it's not cached well or at all because the requests to bromine for it keep failing in strange ways
[18:04:15] <bblack>	 the backend server closes the connection without transferring it all successfully, where successfully includes correct gzip encoding or whatever too
[18:04:20] <nuria>	 bblack: size? it is 700k uncompressed
[18:04:30] <logmsgbot>	 !log ori@tin Synchronized docroot and w: Ie60638b0: Mirror homepage.js from 15.wikipedia.org (duration: 00m 42s)
[18:04:33] <grrrit-wm>	 (03PS1) 10Niedzielski: Add Xdummy daemon [puppet] - 10https://gerrit.wikimedia.org/r/264303 
[18:04:35] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[18:04:46] <bblack>	 the size on the gzip outputs are kinda random even though apache thinks they're successes (they don't end up in error.log)
[18:05:01] <bblack>	 in any case, it looks like ori's copying the static file to our text cluster stuff and then hacking the link to pull it from there
[18:05:05] <ori>	 milimetric: could you change the url to https://en.wikipedia.org/15/homepage.js ?
[18:05:18] <milimetric>	 (I'm gonna try to fix this by mapping it into a more efficient JSON structure, I think I can save some space)
[18:05:20] <milimetric>	 ori: sure, one sec
[18:05:30] <nuria>	 bblack: ah no, locally it  is not cached either
[18:05:34] <nuria>	 bblack: ok, taht makes sense
[18:05:41] <bblack>	 yeah it loads and caches great from en.wp
[18:05:42] <ori>	 https://en.wikipedia.org/15/homepage.js is a varnish hit
[18:05:43] <ori>	 yeah
[18:05:56] <bblack>	 something's just off about how bromine serves it
[18:06:40] <milimetric>	 ori: https://gerrit.wikimedia.org/r/264304
[18:06:55] <grrrit-wm>	 (03CR) 10Niedzielski: [C: 04-1] "@Hashar, hello! I've made an Xdummy module but I don't really know anything about Puppet... So it's probably busted :) Additionally, I'm p" [puppet] - 10https://gerrit.wikimedia.org/r/264303 (owner: 10Niedzielski)
[18:07:02] <ori>	 milimetric: lgtm, how do we deploy?
[18:07:10] <ori>	 mutante: ^
[18:07:12] <milimetric>	 I think git deploy from the site?
[18:07:17] <milimetric>	 uh git pull i mean
[18:08:21] <mutante>	 ori: just git pull in /srv/org/wikimedia/annualreport
[18:08:40] <mutante>	 it has git clone in puppet but not with auto-update
[18:08:50] <ori>	 done
[18:09:01] <ori>	  27 files changed, 27 insertions(+), 54 deletions(-)
[18:09:04] <ori>	 o.O
[18:09:07] <nuria>	 ori: and bust cache right?
[18:09:10] <nuria>	 cc mutante
[18:09:22] <ori>	 yeah bblack do we need to ban it in varnish?
[18:09:24] <mutante>	 https://phabricator.wikimedia.org/diffusion/WANR/browse/master/2015/assets/js/homepage.js
[18:09:30] <nuria>	 milimetric, ori: homepage.js comes from an after efffects annimation library: https://github.com/bodymovin/bodymovin
[18:10:02] <ori>	 the actual content seems irrelevant, it's big but it should not have the catastrophic effect that it is having, that is a web server misconfiguration on bromine
[18:10:12] <ori>	 optimizing the js won't help
[18:10:46] <milimetric>	 yeah, it's also very hard to optimize, that structure's very complicated and minified anyway
[18:11:14] <bblack>	 ori: ban index.html?
[18:11:19] <milimetric>	 but it's pretty repetitive, zipping should help
[18:11:28] <mutante>	 ori: there were a couple merges that weren't deployed yet it looks, like since "redirect annual.wm.." https://gerrit.wikimedia.org/r/#/q/project:wikimedia/annualreport+status:merged,n,z
[18:11:33] <ori>	 bblack: yes, assuming ban == "purge it once so that it gets refreshed"
[18:11:37] <bblack>	 yeah
[18:11:42] <milimetric>	 bblack: yeah, agreed
[18:12:27] <ori>	 should we do the same thing with the other two js files, bodymovin.js and masonry.js?
[18:12:38] <ori>	 or is this problem really only with homepage.js?
[18:12:52] <bblack>	 the other js seems fine as far as I've seen
[18:12:58] <ori>	 site loads well now
[18:13:01] <bblack>	 so far :)
[18:13:11] <nuria>	 ori: i think if remove homepage.js things will be fine
[18:13:17] <ori>	 nuria: it's fine now
[18:13:26] <nuria>	 ori: ah ok, sorry
[18:13:58] <mutante>	 load is already going down a lot, cool
[18:14:08] <milimetric>	 we'd need to check with those folks if we remove homepage.js though; i know it's fine now, just saying in case we realize we need to do that later
[18:14:29] <ori>	 no need to remove it
[18:14:37] <milimetric>	 I'm guessing IE6 and 7 basically die trying to render this :)
[18:14:39] <icinga-wm>	 RECOVERY - puppet last run on bromine is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:14:53] <milimetric>	 that's a big JSON.parse and those silly browsers die even when parsing like 100K
[18:15:04] <godog>	 loads fast from firefox mobile too, fwiw
[18:15:13] <ori>	 IE10 hit end-of-life earlier this week
[18:15:28] <bd808>	 which means we will only see it in the wild for 10 more years :)
[18:15:29] <chasemp>	 wow, idk why but that makes me feel old
[18:15:38] <milimetric>	 lol
[18:15:53] <bd808>	 IE6 4evr
[18:16:02] <nuria>	 ori: what did we changed on the webserver to fix the caching?
[18:16:18] <bd808>	 moved the content to another apache stack
[18:16:20] <ori>	 nothing; i added it to the docroot for all app servers
[18:16:34] <ori>	 bblack: we have seen varnish decline to cache when the declared and actual content-length mismatch; i bet that is what is going on
[18:16:40] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[18:16:48] <greg-g>	 and then changed the url to point to our app servers instead of that misc cluster server
[18:16:50] <icinga-wm>	 PROBLEM - Codfw HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[18:17:42] <bblack>	 anyways, things seem ok now
[18:17:47] <bblack>	 thanks ori! :)
[18:17:53] <mutante>	 yes, thanks ori!
[18:17:57] <ori>	 np, thanks guys
[18:18:14] <godog>	 thanks ori \o/
[18:20:18] <grrrit-wm>	 (03Abandoned) 10Dzahn: add url-downloader-codfw service IP [dns] - 10https://gerrit.wikimedia.org/r/264208 (https://phabricator.wikimedia.org/T122134) (owner: 10Dzahn)
[18:20:53] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, and 2 others: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1937246 (10demon) Replication is complaining about a missing object `3e588b1f9e4b7e39bf38e43e9b21ccf9f2cb2c94`. I cannot find this object on Git...
[18:21:11] <wikibugs>	 6operations, 7Mail: Move most (all?) exim personal aliases to OIT - https://phabricator.wikimedia.org/T122144#1937249 (10Dzahn)
[18:21:12] <wikibugs>	 6operations, 7Mail: remove exim aliases -  wikimedia brazil - https://phabricator.wikimedia.org/T123682#1937247 (10Dzahn) 5Open>3Resolved done   ``` -## Ação Educativa / Brazilian Education Program -## http://www.acaoeducativa.org/ -## formerly educacao@wm, RT #5436, #6746, #6871, #6921, #6926 :p - -oona.c...
[18:21:22] <godog>	 still some hits for homepage.js on bromine btw, didn't get banned everywhere perhaps?
[18:21:42] <bblack>	 yeah I'm still looking at that
[18:21:59] <bblack>	 it could just be lag here and there in the overall problem, or client-side caching
[18:22:45] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Install nova-compute on labtestvirt2001 [puppet] - 10https://gerrit.wikimedia.org/r/264305 
[18:23:09] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[18:23:11] <bblack>	 we do get some rare hits for index.html, so banning that too (previous ban was on /, which is implicitly index.html)
[18:23:19] <icinga-wm>	 RECOVERY - Codfw HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[18:24:25] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, and 2 others: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1937254 (10demon) Fixed? I killed the replication job and issued a new one. If I had to guess I'd say it was a PR branch open when the job began...
[18:24:47] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Install nova-compute on labtestvirt2001 [puppet] - 10https://gerrit.wikimedia.org/r/264305 (owner: 10Andrew Bogott)
[18:25:41] <wikibugs>	 6operations, 10Continuous-Integration-Infrastructure, 10Gerrit, 10GitHub-Mirrors, and 2 others: [Task] Redirect unused extensions/ValueView repository to data-values/value-view - https://phabricator.wikimedia.org/T123624#1937256 (10demon)
[18:26:33] <bblack>	 I've confirmed directly that all frontend caches serve the new link, so it's not a ban problem
[18:26:44] <bblack>	 it must be client caching or external caching (e.g. for mobile compression stuff)
[18:28:10] <icinga-wm>	 PROBLEM - puppet last run on mw2033 is CRITICAL: CRITICAL: puppet fail
[18:28:40] <ori>	 we could rewrite the URL in VCL..
[18:29:52] <godog>	 301 on bromine's apache could work too
[18:30:40] <ori>	 or we could try to figure out what is wrong with bromine -- the fact that it is still getting hits to homepage.js could make it easy to confirm that a fix (if we come up with one) works
[18:36:02] <bblack>	 it seems to be serving it successfully now, at least in my local curl tests on bromine, even with gzip
[18:36:52] <bblack>	 and the varnish requests going through to bromine for it are now:
[18:36:53] <bblack>	    19 RxHeader     b Content-Type: application/javascript
[18:36:53] <bblack>	    19 Fetch_Body   b 3(chunked) cls 0 mklen 1
[18:36:53] <bblack>	    19 Length       b 245409
[18:36:53] <bblack>	    19 BackendReuse b be_bromine
[18:36:57] <bblack>	 which is correct
[18:37:33] <bblack>	 before I didn't see (chunked) in there, and we got an eof and connection close
[18:37:40] <icinga-wm>	 PROBLEM - swift-container-auditor on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor
[18:37:41] <icinga-wm>	 PROBLEM - puppet last run on mw2187 is CRITICAL: CRITICAL: Puppet has 1 failures
[18:37:49] <icinga-wm>	 PROBLEM - swift-container-replicator on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-replicator
[18:38:11] <icinga-wm>	 PROBLEM - swift-container-server on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-server
[18:38:21] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 031] Add wgScriptPath to InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/264260 (owner: 10Catrope)
[18:38:29] <icinga-wm>	 PROBLEM - swift-container-updater on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-updater
[18:38:44] <godog>	 that's me ^
[18:38:47] <bblack>	 it's still doing pass/hfp in varnish and not caching though
[18:38:50] <icinga-wm>	 PROBLEM - swift-object-auditor on ms-be3001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor
[18:39:26] <bblack>	 so my guess is the primary problem is "for whatever reason, varnish won't cache that", and then the secondary problem was "lack of caching -> overload bromine -> starts sending broken responses due to overload (timeouts or whatever)"
[18:40:22] <bblack>	 maybe we're missing some code we have on e.g. the upload server to handle this case right with chunked+gzip+whatever
[18:44:49] <ori>	 i'm comparing:
[18:44:57] <ori>	 curl -vvv -H 'Accept-encoding: gzip' -H 'X-Forwarded-Proto: https' -H 'host: 15.wikipedia.org' bromine/assets/js/homepage.js -o /dev/null
[18:44:57] <ori>	 to:
[18:45:08] <ori>	 curl -vvv -H 'Accept-encoding: gzip' -H 'X-Forwarded-Proto: https' -H 'host: en.wikipedia.org' mw1042/15/homepage.js -o /dev/null
[18:45:22] <ori>	 i notice the latter response (from mw1042) has Content-Length: 774916
[18:45:25] <ori>	 the former does not
[18:45:39] <ori>	 so maybe varnish doesn't know when the response has ended?
[18:47:04] <ori>	 https://httpd.apache.org/docs/trunk/compliance.html : "When the Content-Length header is present, the size of the body is declared at the start of the response. If this information is missing, an HTTP cache might choose to ignore the response, as it does not know in advance whether the response will fit within the cache's defined limits."
[18:47:10] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[18:47:40] <ori>	 and further:
[18:47:41] <ori>	 "HTTP/1.1 defines the Transfer-Encoding header as an alternative to Content-Length, allowing the end of the response to be indicated to the client without the client having to know the length beforehand. However, when HTTP/1.0 requests are processed, and no Content-Length is specified, the only mechanism available to the server to indicate the end of the request is to drop the connection. In an environment containing load
[18:47:41] <ori>	  balancers, this can cause the keepalive mechanism to be bypassed."
[18:48:13] <bblack>	 yeah
[18:48:21] <bblack>	 I guess mw1042 isn't using chunked, and bromine is
[18:48:36] <bblack>	 and because it's chunked, varnish won't cache it?
[18:49:08] <ori>	 i guess so. apparently apache will send content-length if we don't gzip
[18:49:13] <ori>	 varnish would still gzip, so that seems fine
[18:49:31] <bblack>	 gzip should be fine though, we just need it to be gzip without chunked
[18:49:34] <bblack>	    29 RxHeader     b ETag: "bd304-529561704ff76-gzip"
[18:49:36] <bblack>	    29 RxHeader     b Accept-Ranges: bytes
[18:49:39] <bblack>	    29 RxHeader     b Content-Encoding: gzip
[18:49:41] <bblack>	    29 RxHeader     b Transfer-Encoding: chunked
[18:49:44] <bblack>	    29 RxHeader     b Content-Type: application/javascript
[18:49:46] <bblack>	    29 Fetch_Body   b 3(chunked) cls 0 mklen 1
[18:49:49] <bblack>	    29 Length       b 245409
[18:49:51] <bblack>	    29 BackendReuse b be_bromine
[18:49:52] <bblack>	 that's the brome->varnish response for homepage.js, using chunked encoding and succeeeding (but not cacheable)
[18:50:03] <_joe_>	 bblack: chunked transfers are still supposed to send content-length
[18:50:10] <ori>	 "Apache uses chunked encoding only if the compressed file size is larger than the DeflateBufferSize. Increasing this buffer size will therefore prevent the server using chunked encoding also for larger files, causing the Content-Length to be sent even for zipped data."
[18:50:17] <bblack>	 there we go!
[18:50:23] <ori>	 that explains why homepage.js is an issue but the others are not
[18:50:41] <bblack>	 so we either need to disable gzip (for the site, or for that file) or increase deflatebuffersize
[18:51:15] <_joe_>	 btw, if it's true that chunked transfers are not cached by varnish, we might want to play with DeflateBufferSize for the main site too
[18:51:17] <ori>	 8096 is the default
[18:51:33] <bblack>	 heh
[18:51:40] <_joe_>	 but I'd first verify the hypothesis :)
[18:51:47] <godog>	 the other files are in the 80k range though
[18:52:19] <bblack>	 well, mw1042 isn't even compressing when asked
[18:52:32] <bblack>	 I'm guessing we disable gzip on our main apaches, and thus this problem goes away
[18:52:47] <bblack>	 or at least we do for some statics?
[18:52:52] <_joe_>	 bblack: yeah we do
[18:52:57] <_joe_>	 sorry, brainfart
[18:53:17] <ori>	 hhvm serves a lot of static files
[18:53:40] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[18:53:42] <bblack>	 so if we want to fix this the way we do for the main sites, just disable gzip on bromine's apache I guess
[18:53:55] <ori>	 yeah so comment out: deflate.conf:		AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
[18:54:00] <ori>	 mind if i go ahead?
[18:54:07] <bblack>	 go for it
[18:54:59] <ori>	 !log disabled gzip in apache for javascript mime types and did an apache config reload
[18:55:01] <bblack>	 varnish may have some ephemeral hit-for-pass objects at this point, it may take a bit to react or need another ban
[18:55:03] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[18:55:09] <ori>	 i get Content-Length: 774916 now
[18:55:11] <bblack>	 I see ungzipped repsonses coming out of bromine though
[18:56:29] <ori>	 varnish is caching it
[18:56:29] <icinga-wm>	 RECOVERY - puppet last run on mw2033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:56:37] <ori>	 X-Cache: cp1056 hit(1), cp4003 hit(2), cp4002 frontend hit(29856)
[18:56:45] <bblack>	 nice
[18:56:58] <wikibugs>	 6operations, 10Continuous-Integration-Infrastructure, 10Gerrit, 10GitHub-Mirrors, and 3 others: [Task] Redirect unused extensions/ValueView repository to data-values/value-view - https://phabricator.wikimedia.org/T123624#1937331 (10demon) > # Delete https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki...
[18:57:00] <bblack>	 yeah just took a few for the rate to drop off as the caches caught up to what's happening
[18:57:15] <bblack>	 load on brome dropped from ~8 to ~0.2
[18:57:20] <bblack>	 *bromine
[18:57:46] <ori>	 varnish is correctly gzipping the response too if the req has the right accept-encoding header, so everything checks out
[18:57:55] <bblack>	 awesome
[18:58:20] <ori>	 should we roll back the change to load the js from enwiki or is there no point?
[18:58:44] <bblack>	 yeah we probably should undo the hacks, if we can commit the change to disable gzip
[18:59:23] <ori>	 the content of deflate.conf isn't managed by puppet, so it won't get clobbered, but it should be puppetized anyway, yeah
[18:59:26] <bblack>	 don't ask me how that's puppetized
[18:59:50] <ori>	 puppet only manages the symlinks in -enabled, but not the files the symlinks reference
[19:00:20] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[19:03:16] <mutante>	 ori: should we do this with apache::mod_conf ?
[19:03:51] <mutante>	 there is *loadfile* but not *conffile* hmm
[19:04:12] <ori>	 i'm leaning toward "SetEnvIfNoCase Request_URI \.js$ no-gzip" in a <Location> block
[19:04:41] <ori>	 it's not a big deal, but it'd be overreaching for the 15.wikipedia.org puppetization to mess around with the global config for mod_deflate, because there are other sites on the same host
[19:04:59] <ori>	 of course they're all behind varnish so what is good for 15.wikipedia.org is good for the others too, but i still don't like it
[19:05:30] <icinga-wm>	 RECOVERY - puppet last run on mw2187 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:05:33] <mutante>	 ori: it's a good point about overreaching for that to change global config.. <Location> sounds better
[19:06:09] <ori>	 i'll try it with a live hack before submitting a patch
[19:06:30] <bblack>	 well, arguably (a) virtually all our apache sites are behind varnish and (b) local network bandwidth isn't at a premium, esp vs screwing up caching, so (c) we should disable gzip everywhere by default in apache configs
[19:06:41] <bblack>	 but that doesn't sound like a friday decision->deploy kind of thing heh
[19:06:57] <Steinsplitter>	 action=login will be removed soon?
[19:08:22] <bblack>	 at least, we could puppetize things a little differently so that it's an easy class/define option thing per-site, gzip defaulting off, but can be turned on for some site configs
[19:09:02] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[19:09:38] <mutante>	 bblack: yes, i think we want apache::mod_conf to also have a parameter for the .conf file
[19:12:21] <mutante>	 ah, or apache::conf needs to know {conf,sites,mods} as opposed to juse {conf,sites}-available 
[19:16:27] <grrrit-wm>	 (03PS1) 10Dzahn: apache: add conf_type "mods" [puppet] - 10https://gerrit.wikimedia.org/r/264313 
[19:19:18] <elukey>	 wow nice investigation! 
[19:19:26] <grrrit-wm>	 (03PS1) 10Ori.livneh: 15.wp.org: don't gzip js files [puppet] - 10https://gerrit.wikimedia.org/r/264315 
[19:19:41] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[19:21:36] <wikibugs>	 6operations, 10OTRS: Upgrade OTRS to a more recent stable release - https://phabricator.wikimedia.org/T74109#1937448 (10Rjd0060) Per @akosiaris OTRS upgrade to version 5.0.x has been scheduled for **28 January 2016** and is expected to last 6-8 hours, beginning at **0800 UTC**.  Agents have been notified.
[19:21:54] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] 15.wp.org: don't gzip js files [puppet] - 10https://gerrit.wikimedia.org/r/264315 (owner: 10Ori.livneh)
[19:22:01] <ori>	 thanks mutante
[19:22:03] <grrrit-wm>	 (03PS2) 10Dzahn: apache: add conf_type "mods" [puppet] - 10https://gerrit.wikimedia.org/r/264313 
[19:22:09] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] 15.wp.org: don't gzip js files [puppet] - 10https://gerrit.wikimedia.org/r/264315 (owner: 10Ori.livneh)
[19:22:32] <grrrit-wm>	 (03CR) 10Ori.livneh: [V: 032] 15.wp.org: don't gzip js files [puppet] - 10https://gerrit.wikimedia.org/r/264315 (owner: 10Ori.livneh)
[19:23:05] <ori>	 "Now running conftool-merge to sync any changes to conftool data"
[19:23:05] <ori>	 ooo
[19:23:06] <ori>	 fancy!
[19:25:26] <ori>	 bblack / mutante: https://gerrit.wikimedia.org/r/#/c/264318/
[19:27:33] <wikibugs>	 6operations, 10Traffic, 5Patch-For-Review: HTTP/1.1 keepalive for local nginx->varnish conns - https://phabricator.wikimedia.org/T107749#1937465 (10elukey) Adding some notes after a chat with BBlack.  http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive  "It should be particularly noted that...
[19:28:10] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[19:32:39] * ori goes for it
[19:32:42] <wikibugs>	 6operations, 10hardware-requests: eqiad: (2) servers request for ORES - https://phabricator.wikimedia.org/T119598#1937477 (10RobH) >>! In T119598#1932642, @yuvipanda wrote: > I would suggest oresredis1xxx since they'll be used both as queue and cache machines, no 'db' type things there (we can flush them whene...
[19:36:41] <ori>	 ok, reverted from `src="//en.wikipedia.org/15/homepage.js"` to `src="assets/js/homepage.js"`
[19:36:59] <ori>	 no point in banning the cached homepage in varnish imo
[19:42:13] <ori>	 MaxSem: https://gerrit.wikimedia.org/r/#/c/264319/
[19:43:01] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0]
[19:43:01] <icinga-wm>	 PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0]
[19:45:10] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[19:45:11] <icinga-wm>	 RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[19:47:14] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Specify openstack::nova::partition on labtestvirt2001 [puppet] - 10https://gerrit.wikimedia.org/r/264320 
[19:48:56] <mutante>	 ori: this is very simple but looks like enough to add module config to apache::conf https://gerrit.wikimedia.org/r/#/c/264313/
[19:49:00] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Specify openstack::nova::partition on labtestvirt2001 [puppet] - 10https://gerrit.wikimedia.org/r/264320 (owner: 10Andrew Bogott)
[19:49:08] <grrrit-wm>	 (03CR) 10Mobrovac: [C: 031] Add parsoid::testing role and use it on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/264024 (owner: 10Subramanya Sastry)
[19:52:00] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Revert "Specify openstack::nova::partition on labtestvirt2001" [puppet] - 10https://gerrit.wikimedia.org/r/264321 
[19:53:33] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Revert "Specify openstack::nova::partition on labtestvirt2001" [puppet] - 10https://gerrit.wikimedia.org/r/264321 (owner: 10Andrew Bogott)
[19:55:40] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[19:55:58] <wikibugs>	 6operations, 7Mail: Move most (all?) exim personal aliases to OIT - https://phabricator.wikimedia.org/T122144#1937520 (10Dzahn)
[19:55:59] <wikibugs>	 6operations, 7Mail: remove ea@ alias? - https://phabricator.wikimedia.org/T123286#1937518 (10Dzahn) 5Open>3Resolved thanks. done   ``` -# RT-5019,5657 - ea@ alias for the search for Sue's new EA -# "Exec Admin" -ea: lmartinez -  ```
[19:57:41] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[19:58:31] <wikibugs>	 6operations, 7Mail: Move most (all?) exim personal aliases to OIT - https://phabricator.wikimedia.org/T122144#1937542 (10Dzahn)
[19:58:33] <wikibugs>	 6operations, 7Mail: remove shop@wikipedia.org -> board@wikipedia.org - https://phabricator.wikimedia.org/T123672#1937540 (10Dzahn) 5Open>3Resolved done   ``` -shop:  board  ```
[20:02:08] <wikibugs>	 6operations, 7Mail: remove staff@wikipedia.org - https://phabricator.wikimedia.org/T123670#1937571 (10Dzahn) 5Open>3Resolved done   ``` -staff:  noc  ```
[20:02:09] <wikibugs>	 6operations, 7Mail: Move most (all?) exim personal aliases to OIT - https://phabricator.wikimedia.org/T122144#1937573 (10Dzahn)
[20:06:07] <wikibugs>	 6operations, 7Mail: delete exim alias vpe-staff: eng-mgt - https://phabricator.wikimedia.org/T123667#1937583 (10Dzahn) done   ``` -# T105431 forward for OIT / cajoel -vpe-staff: eng-mgt -  ```
[20:06:14] <wikibugs>	 6operations, 7Mail: Move most (all?) exim personal aliases to OIT - https://phabricator.wikimedia.org/T122144#1937585 (10Dzahn)
[20:06:15] <wikibugs>	 6operations, 7Mail: delete exim alias vpe-staff: eng-mgt - https://phabricator.wikimedia.org/T123667#1937584 (10Dzahn) 5Open>3Resolved
[20:06:20] <subbu>	 ori, https://gerrit.wikimedia.org/r/#/c/264024/ has a +1 from mobrovac ... can you take a look?
[20:09:57] <ori>	 subbu: sure
[20:10:56] <ori>	 subbu: looks ok, at a glance. I can merge it if you are around to see it apply on ruthenium and verify it is working
[20:11:20] <subbu>	 ori, yes, around to verify.
[20:11:38] <grrrit-wm>	 (03PS7) 10Ori.livneh: Add parsoid::testing role and use it on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/264024 (owner: 10Subramanya Sastry)
[20:11:46] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Add parsoid::testing role and use it on ruthenium [puppet] - 10https://gerrit.wikimedia.org/r/264024 (owner: 10Subramanya Sastry)
[20:12:11] <ori>	 subbu: ^
[20:12:21] <ori>	 i did not run puppet, but i can force a run if you like
[20:12:30] <subbu>	 i'll do it now.
[20:13:32] <subbu>	 ori, but, now that i said it .. i forget the incantation .. is it puppet -tv ? 
[20:13:41] <ori>	 close: puppet agent -tv
[20:13:45] <subbu>	 ah, k.
[20:14:41] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[20:16:51] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[20:21:56] <grrrit-wm>	 (03PS5) 10Jforrester: Centralise all VisualEditor feedback pages except for a few wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/258206 (https://phabricator.wikimedia.org/T92661) 
[20:22:51] <gwicke>	 puppet --entertain=me
[20:23:14] <mutante>	 Krenair: i see you are using bast4001 lately (and very few other users), i was going to re-install it .. do you mind?
[20:23:35] <Krenair>	 oh, I only logged in briefly
[20:23:53] <mutante>	 ok, it looks almost unused actually
[20:23:57] <Krenair>	 was just checking which bastions I could get into
[20:23:59] <mutante>	 i think i'll just do it
[20:24:05] <mutante>	 *nod*
[20:24:21] <Krenair>	 (just added a new key)
[20:27:44] <grrrit-wm>	 (03PS1) 10Dzahn: install_server: switch bast4001 to jessie [puppet] - 10https://gerrit.wikimedia.org/r/264330 (https://phabricator.wikimedia.org/T123674) 
[20:28:42] <grrrit-wm>	 (03PS2) 10Dzahn: install_server: switch bast4001 to jessie [puppet] - 10https://gerrit.wikimedia.org/r/264330 (https://phabricator.wikimedia.org/T123674) 
[20:33:17] <grrrit-wm>	 (03PS6) 10Jforrester: Centralise all VisualEditor feedback pages except for a few wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/258206 (https://phabricator.wikimedia.org/T92661) 
[20:34:05] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] install_server: switch bast4001 to jessie [puppet] - 10https://gerrit.wikimedia.org/r/264330 (https://phabricator.wikimedia.org/T123674) (owner: 10Dzahn)
[20:36:10] <icinga-wm>	 PROBLEM - citoid endpoints health on sca1002 is CRITICAL: /api (web site in alternative language) is CRITICAL: Could not fetch url http://10.64.48.29:1970/api: Timeout on connection while downloading http://10.64.48.29:1970/api
[20:36:30] <icinga-wm>	 PROBLEM - citoid endpoints health on sca1001 is CRITICAL: /api (web site in alternative language) is CRITICAL: Could not fetch url http://10.64.32.153:1970/api: Timeout on connection while downloading http://10.64.32.153:1970/api
[20:38:11] <icinga-wm>	 RECOVERY - citoid endpoints health on sca1002 is OK: All endpoints are healthy
[20:38:31] <icinga-wm>	 RECOVERY - citoid endpoints health on sca1001 is OK: All endpoints are healthy
[20:46:10] <grrrit-wm>	 (03CR) 10CSteipp: [C: 031] "Yes for security direction (again), and it also looks like Krenair's comments have been addressed in PS3." [puppet] - 10https://gerrit.wikimedia.org/r/217794 (https://phabricator.wikimedia.org/T94570) (owner: 10Muehlenhoff)
[20:47:05] <wikibugs>	 10Ops-Access-Requests, 6operations, 5WMF-NDA: Requesting access to analytics-privatedata-users for jdcc-berkman - https://phabricator.wikimedia.org/T123785#1937711 (10ZhouZ)
[20:50:41] <icinga-wm>	 PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /home 41018 MB (3% inode=98%)
[20:51:00] <wikibugs>	 10Ops-Access-Requests, 6operations, 5WMF-NDA: Requesting access to analytics-privatedata-users for jdcc-berkman - https://phabricator.wikimedia.org/T123785#1937731 (10ZhouZ) Hi all,  We would like to give Justin access to our pageview_hourly table to allow him to build the anti-censorship software for them....
[20:51:14] <grrrit-wm>	 (03PS1) 10ArielGlenn: dumps multi stream bz2 files: fix corruption in writing end of BZ2 stream [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/264337 (https://phabricator.wikimedia.org/T121348) 
[21:03:51] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[21:07:11] <andrewbogott>	 YuviPanda: are you around?  I have a hiera/labs/realm question
[21:07:45] <mutante>	 !log bast4001 - reinstalling with jessie
[21:07:50] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[21:08:05] <andrewbogott>	 specifically, I need to create a new realm for vms in labtest, wondering how to create that in hiera
[21:08:10] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[21:09:36] <mutante>	 !log (ganglia for ulsfo will be affected, brb)
[21:09:41] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[21:17:23] <grrrit-wm>	 (03PS1) 10Hoo man: Switch back to bzip2 for Wikidata bz2 dumps [puppet] - 10https://gerrit.wikimedia.org/r/264356 
[21:19:36] <mutante>	 eh, lol, guess i can't reinstall the server when it's also the tft-server itself
[21:19:45] <mutante>	 tftp
[21:23:39] <wikibugs>	 6operations, 5Patch-For-Review: reinstall bast4001 with jessie - https://phabricator.wikimedia.org/T123674#1937917 (10Dzahn) eh, yea:)  So this is also the tftp-server so reinstalling it is kind of tricky because it needs itself for install.
[21:25:01] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0]
[21:26:02] <wikibugs>	 10Ops-Access-Requests, 6operations, 5WMF-NDA: Requesting access to analytics-privatedata-users for jdcc-berkman - https://phabricator.wikimedia.org/T123785#1937941 (10Krenair) > "bastiononly" access (I don't know what this means, but it was noted here that I should request it).  `bastiononly` grants you acce...
[21:29:17] <mutante>	 !log protactinium - shut down, unused system with outdated software
[21:29:22] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[21:32:39] <grrrit-wm>	 (03PS1) 10Dzahn: remove protactinium from site and DHCP [puppet] - 10https://gerrit.wikimedia.org/r/264362 (https://phabricator.wikimedia.org/T123720) 
[21:32:58] <grrrit-wm>	 (03PS2) 10Dzahn: remove protactinium from site and DHCP [puppet] - 10https://gerrit.wikimedia.org/r/264362 (https://phabricator.wikimedia.org/T123720) 
[21:33:31] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] remove protactinium from site and DHCP [puppet] - 10https://gerrit.wikimedia.org/r/264362 (https://phabricator.wikimedia.org/T123720) (owner: 10Dzahn)
[21:33:50] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[21:34:00] <wikibugs>	 6operations, 5Patch-For-Review, 7Tracking: decom protactinium (was: Reinstall protactinium with jessie) - https://phabricator.wikimedia.org/T123720#1937968 (10Dzahn)
[21:34:12] <YuviPanda>	 andrewbogott: am now
[21:34:42] <andrewbogott>	 YuviPanda: does my question make sense?
[21:35:19] <YuviPanda>	 ah
[21:35:22] <YuviPanda>	 realms aren't hardcoded
[21:35:27] <YuviPanda>	 so you should be able to jdi
[21:35:48] <YuviPanda>	 actually
[21:35:52] <YuviPanda>	 let's step back
[21:35:57] <YuviPanda>	 what are you trying to do ultimtaely?
[21:36:06] <YuviPanda>	 have a labstest realm that has VMs or?
[21:36:10] <andrewbogott>	 I want to create a vm in labtest that uses the labtest ldap and dns servers
[21:36:22] <grrrit-wm>	 (03PS1) 10Dzahn: remove public IP of protactinium [dns] - 10https://gerrit.wikimedia.org/r/264364 (https://phabricator.wikimedia.org/T123720) 
[21:36:39] <andrewbogott>	 so, somehow puppet needs to notice that these vms aren’t ‘normal’ labs instances and switch accordingly
[21:36:46] <andrewbogott>	 doesn’t have to be by realm, but that seems the obvious choice.
[21:37:09] <grrrit-wm>	 (03PS2) 10Dzahn: remove public IP of protactinium [dns] - 10https://gerrit.wikimedia.org/r/264364 (https://phabricator.wikimedia.org/T123720) 
[21:38:08] <wikibugs>	 6operations, 5Patch-For-Review: reinstall bast4001 with jessie - https://phabricator.wikimedia.org/T123674#1937974 (10Krenair) Is it possible to point it at one of the other DC's tftp servers?
[21:38:28] <mutante>	 realm metalabs?
[21:38:50] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] remove public IP of protactinium [dns] - 10https://gerrit.wikimedia.org/r/264364 (https://phabricator.wikimedia.org/T123720) (owner: 10Dzahn)
[21:40:03] <andrewbogott>	 mutante: I was thinking something like labtestlabs :(
[21:40:06] <mutante>	 andrewbogott: you could do it via regex , just by the hostname 
[21:40:10] <mutante>	 roles would be nicer though
[21:40:49] <mutante>	 hieradata/regex.yaml  on labtest* i guess
[21:41:09] <andrewbogott>	 well, that’s kind of my question.  How do labs instances know they’re labs instances?
[21:41:10] <mutante>	 you could make up any variable
[21:41:21] <wikibugs>	 6operations: Adding/Removing users from enWP Arbcom Mailinglist archives - https://phabricator.wikimedia.org/T123787#1938002 (10Krenair)
[21:42:17] <YuviPanda>	 andrewbogott: so prod and labs puppetmasters are configured to use different hiera configurations
[21:42:26] <mutante>	 andrewbogott: oh, it looks like you already have something
[21:42:32] <mutante>	 codfw/labtest.yaml:# These values are site/realm specific for LABTEST infrastructure
[21:42:46] <mutante>	 same thing?
[21:42:50] <andrewbogott>	 mutante: that’s for labtest servers, not labtest vms
[21:42:54] <mutante>	 ah, right
[21:42:57] <YuviPanda>	 andrewbogott: I already see you have a labstest.hiera.yaml
[21:43:02] <YuviPanda>	 andrewbogott: if the labstest puppetmaster is using that
[21:43:12] <YuviPanda>	 andrewbogott: then any VMs that connect to it will count themselves as being in the labs realm
[21:43:29] <YuviPanda>	 andrewbogott: you can change it by messing with the 'hierarchy' in that file
[21:43:33] <andrewbogott>	 YuviPanda: oh, ok.  So I can change the labtest puppetmaster to put everything in labtestlabs
[21:43:35] <andrewbogott>	 that’ll do
[21:43:44] <andrewbogott>	 YuviPanda: ‘that file’?
[21:43:52] <YuviPanda>	 andrewbogott: labstest.hiera.yaml
[21:44:12] <YuviPanda>	 andrewbogott: in that you see a 'labs', so that's what makes it read hieradata/labs.yaml
[21:44:20] <YuviPanda>	 andrewbogott: if you add a 'labstest' above it, it will also read a labstest.yaml
[21:44:28] <YuviPanda>	 and override anything in labs.yaml to use that
[21:44:30] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[21:44:39] <wikibugs>	 10Ops-Access-Requests, 6operations, 5WMF-NDA: Requesting access to analytics-privatedata-users for jdcc-berkman - https://phabricator.wikimedia.org/T123785#1938036 (10Cmjohnson) HI Justin,  We will also need you to sign the L3 form as well.  It can be found here https://phabricator.wikimedia.org/L3   Thank Y...
[21:45:09] <andrewbogott>	 YuviPanda: ok, that makes sense, somewhat
[21:45:11] <grrrit-wm>	 (03PS3) 10Dzahn: remove public IP of protactinium [dns] - 10https://gerrit.wikimedia.org/r/264364 (https://phabricator.wikimedia.org/T123720) 
[21:45:16] <andrewbogott>	 Once I can build a vm I will tinker.  Thanks
[21:45:26] <YuviPanda>	 andrewbogott: ok :)
[21:46:40] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[21:48:02] <andrewbogott>	 YuviPanda: so modules/puppetmaster/files/labtest.hiera.yaml /only/ dictates the behavior of puppetmasters in the labtest realm?
[21:48:34] <wikibugs>	 6operations, 5Patch-For-Review, 7Tracking: decom protactinium (was: Reinstall protactinium with jessie) - https://phabricator.wikimedia.org/T123720#1938050 (10Dzahn) - shutdown the host to save energy - removed from site.pp and install-server - removed public IP from DNS, kept mgmt - puppetstoredconfigclean...
[21:48:40] <YuviPanda>	 andrewbogott: only whatever puppetmaster is using that file
[21:48:59] <andrewbogott>	 ok
[21:49:06] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] remove public IP of protactinium [dns] - 10https://gerrit.wikimedia.org/r/264364 (https://phabricator.wikimedia.org/T123720) (owner: 10Dzahn)
[21:49:21] <andrewbogott>	 um… actually YuviPanda I’m not sure I understand your response
[21:50:03] <wikibugs>	 6operations, 7Graphite: Wes Moran not able to log into Graphite - https://phabricator.wikimedia.org/T123796#1938063 (10eliza) 3NEW
[21:51:16] <wikibugs>	 6operations, 7Graphite: Wes Moran not able to log into Graphite - https://phabricator.wikimedia.org/T123796#1938075 (10Dzahn)
[21:51:41] <wikibugs>	 10Ops-Access-Requests, 6operations, 5WMF-NDA: Requesting access to analytics-privatedata-users for jdcc-berkman - https://phabricator.wikimedia.org/T123785#1938078 (10Jdcc-berkman) I've already signed the L3 form - sorry I forgot to mention it.
[21:52:20] <wikibugs>	 6operations, 7Graphite: Wes Moran not able to log into Graphite - https://phabricator.wikimedia.org/T123796#1938063 (10Dzahn) @Wes do you have a user on https://wikitech.wikimedia.org/wiki/Main_Page ? That is being used for the login on graphite as well.
[21:52:49] <icinga-wm>	 PROBLEM - salt-minion processes on mira is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[21:54:14] <mutante>	 !log mira - starting salt
[21:54:19] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[21:54:39] <icinga-wm>	 RECOVERY - salt-minion processes on mira is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[21:56:15] <andrewbogott>	 YuviPanda: also, what’s this about?
[21:56:18] <andrewbogott>	 :mwyaml:
[21:56:18] <andrewbogott>	   :host: https://wikitech.wikimedia.org
[21:56:18] <andrewbogott>	   :cache_ttl: 120
[21:57:51] <chasemp>	 andrewbogott: labtest.hiera.yaml comes from $realm.hiera.yaml for puppet master
[21:58:02] <chasemp>	 so yes if you build a puppet master in labtest it should pick that up
[21:58:09] <andrewbogott>	 ok, cool
[21:58:11] <chasemp>	 and only apply for those clients
[21:58:42] <YuviPanda>	 andrewbogott: yeah that's the wikitech.wikimedia.org/wiki/Hiera:<projectname> stuff
[21:58:50] <andrewbogott>	 oh, I see.
[21:58:51] <andrewbogott>	 ok
[21:58:56] <wikibugs>	 6operations, 10ops-eqiad: decom protactinium (datacenter) - https://phabricator.wikimedia.org/T123798#1938107 (10Dzahn) 3NEW
[21:59:29] <wikibugs>	 6operations, 5Patch-For-Review, 7Tracking: decom protactinium (was: Reinstall protactinium with jessie) - https://phabricator.wikimedia.org/T123720#1936432 (10Dzahn)
[21:59:31] <wikibugs>	 6operations, 10ops-eqiad: decom protactinium (datacenter) - https://phabricator.wikimedia.org/T123798#1938123 (10Dzahn)
[22:00:12] <wikibugs>	 6operations, 5Patch-For-Review, 7Tracking: decom protactinium (was: Reinstall protactinium with jessie) - https://phabricator.wikimedia.org/T123720#1936432 (10Dzahn)  I have done all steps on the server lifecycle page up to the "end steps" section.  the rest will be an ops-eqiad ticket a T123798
[22:00:25] <wikibugs>	 6operations, 10ops-eqiad: decom protactinium (datacenter) - https://phabricator.wikimedia.org/T123798#1938107 (10Dzahn)
[22:00:26] <wikibugs>	 6operations, 7Tracking: reduce amount of remaining Ubuntu 12.04 (precise) systems - https://phabricator.wikimedia.org/T123525#1938131 (10Dzahn)
[22:00:27] <wikibugs>	 6operations, 5Patch-For-Review, 7Tracking: decom protactinium (was: Reinstall protactinium with jessie) - https://phabricator.wikimedia.org/T123720#1938128 (10Dzahn) 5Open>3Resolved a:3Dzahn
[22:00:38] <wikibugs>	 6operations, 7Tracking: decom protactinium (was: Reinstall protactinium with jessie) - https://phabricator.wikimedia.org/T123720#1936432 (10Dzahn)
[22:01:09] <grrrit-wm>	 (03PS1) 10Yuvipanda: gridengine: Do not have puppet autostart the master [puppet] - 10https://gerrit.wikimedia.org/r/264390 (https://phabricator.wikimedia.org/T122638) 
[22:05:46] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, and 2 others: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1938147 (10JanZerebecki) My local clone that still has the pre-gerrit-move github origin around does not know that object either.  I did: ``` $...
[22:05:59] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Override a few hiera settings for VMs in labtest. [puppet] - 10https://gerrit.wikimedia.org/r/264392 
[22:06:01] <andrewbogott>	 YuviPanda, chasemp, something like ^ ?
[22:11:42] <chasemp>	 andrewbogott: are you trying to set up things for teh VM's within labtest then
[22:11:53] <andrewbogott>	 right
[22:11:59] <icinga-wm>	 PROBLEM - salt-minion processes on tin is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion
[22:12:00] <andrewbogott>	 so that they don’t hit prod dns and ldap
[22:12:07] <andrewbogott>	 since prod dns and ldap won’t know about them
[22:12:48] <wikibugs>	 6operations, 10Analytics: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#1938161 (10Ottomata)
[22:13:19] <chasemp>	 can I request the path be "%{::site}/%{::realm}-instances.yaml
[22:13:26] <chasemp>	 only because
[22:13:32] <chasemp>	 the config for labtest itself is
[22:13:36] <chasemp>	 $site::$realm
[22:13:43] <chasemp>	 and maybe it makes sense for the isntances within to be
[22:13:48] <andrewbogott>	 sure, makes sense
[22:13:59] <chasemp>	 $site::$realm-something or $site::$realm/something
[22:14:01] <chasemp>	 idk for sure
[22:14:02] <chasemp>	 just a thought
[22:14:12] <chasemp>	 but yeah I get it, what you are doing generally makes sense :)
[22:14:33] <Luke081515>	 btw: I put a part of the SAL to the archiv, the page was very big, so users with a slow connection may have a disadvantage
[22:14:43] <Luke081515>	 so don't wonder ;)
[22:15:08] <YuviPanda>	 andrewbogott: that should work
[22:15:12] <YuviPanda>	 I think
[22:15:32] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] gridengine: Do not have puppet autostart the master [puppet] - 10https://gerrit.wikimedia.org/r/264390 (https://phabricator.wikimedia.org/T122638) (owner: 10Yuvipanda)
[22:15:56] <grrrit-wm>	 (03PS1) 10Dzahn: remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) 
[22:16:39] <grrrit-wm>	 (03PS2) 10Dzahn: remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) 
[22:16:44] <andrewbogott>	 chasemp: is this what you mean?
[22:16:53] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Override a few hiera settings for VMs in labtest. [puppet] - 10https://gerrit.wikimedia.org/r/264392 
[22:16:58] <YuviPanda>	 thanks Luke081515
[22:17:07] <Luke081515>	 np ;)
[22:17:22] <chasemp>	 andrewbogott: yeah I think that works
[22:20:53] <grrrit-wm>	 (03Abandoned) 10Dzahn: remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) (owner: 10Dzahn)
[22:23:50] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, and 2 others: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1938187 (10JanZerebecki) p:5Unbreak!>3High Using the new tag in https://gerrit.wikimedia.org/r/259760 is now working.  >>! In T123521#193651...
[22:24:12] <grrrit-wm>	 (03Restored) 10Dzahn: remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) (owner: 10Dzahn)
[22:25:32] <grrrit-wm>	 (03Abandoned) 10Tim Landscheidt: gridengine: Fix status check for gridengine-master [puppet] - 10https://gerrit.wikimedia.org/r/263242 (owner: 10Tim Landscheidt)
[22:26:07] <grrrit-wm>	 (03Abandoned) 10Hashar: devpi a cache/proxy for python PyPI [puppet] - 10https://gerrit.wikimedia.org/r/250002 (https://phabricator.wikimedia.org/T117207) (owner: 10Hashar)
[22:26:36] <grrrit-wm>	 (03PS2) 10ArielGlenn: Switch back to bzip2 for Wikidata bz2 dumps [puppet] - 10https://gerrit.wikimedia.org/r/264356 (owner: 10Hoo man)
[22:27:42] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] Switch back to bzip2 for Wikidata bz2 dumps [puppet] - 10https://gerrit.wikimedia.org/r/264356 (owner: 10Hoo man)
[22:28:48] <grrrit-wm>	 (03PS1) 10Rush: check-raid: allow a file that specified utility [puppet] - 10https://gerrit.wikimedia.org/r/264395 
[22:31:10] <grrrit-wm>	 (03PS3) 10Dzahn: remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) 
[22:31:17] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Override a few hiera settings for VMs in labtest. [puppet] - 10https://gerrit.wikimedia.org/r/264392 (owner: 10Andrew Bogott)
[22:31:21] <grrrit-wm>	 (03PS2) 10Rush: check-raid: allow a file that specified utility [puppet] - 10https://gerrit.wikimedia.org/r/264395 
[22:31:49] <grrrit-wm>	 (03PS4) 10Andrew Bogott: Override a few hiera settings for VMs in labtest. [puppet] - 10https://gerrit.wikimedia.org/r/264392 
[22:31:51] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Move wmf_scheduler_hosts_pool to hiera. [puppet] - 10https://gerrit.wikimedia.org/r/264397 
[22:33:25] <grrrit-wm>	 (03CR) 10Yuvipanda: "Is the raidutility file a part of nagios? If not shouldn't this be better off as a commandline param?" [puppet] - 10https://gerrit.wikimedia.org/r/264395 (owner: 10Rush)
[22:34:47] <grrrit-wm>	 (03PS2) 10Hashar: contint: rsync server to hold jobs caches [puppet] - 10https://gerrit.wikimedia.org/r/253322 (https://phabricator.wikimedia.org/T116017) 
[22:35:10] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, and 2 others: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1938212 (10JanZerebecki) >>! In T123521#1937246, @demon wrote: > Replication is complaining about a missing object `3e588b1f9e4b7e39bf38e43e9b21...
[22:35:14] <mutante>	 !log erbium - killing from puppet/icinga/salt
[22:35:17] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:37:39] <icinga-wm>	 RECOVERY - salt-minion processes on tin is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[22:38:09] <grrrit-wm>	 (03PS3) 10Rush: check-raid: allow a file that specified utility [puppet] - 10https://gerrit.wikimedia.org/r/264395 
[22:38:10] <mutante>	 !log gadolinium - shutdown -h now 
[22:38:13] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:38:40] <grrrit-wm>	 (03PS4) 10Dzahn: remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) 
[22:39:40] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Move wmf_scheduler_hosts_pool to hiera. [puppet] - 10https://gerrit.wikimedia.org/r/264397 (owner: 10Andrew Bogott)
[22:40:03] <grrrit-wm>	 (03PS3) 10Aaron Schulz: Set $wgCentralAuthUseSlaves for testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263682 (https://phabricator.wikimedia.org/T119689) 
[22:40:12] <mutante>	 aaaa
[22:40:24] <grrrit-wm>	 (03PS5) 10Dzahn: remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) 
[22:40:25] <grrrit-wm>	 (03CR) 10Aaron Schulz: [C: 032] Set $wgCentralAuthUseSlaves for testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263682 (https://phabricator.wikimedia.org/T119689) (owner: 10Aaron Schulz)
[22:40:28] <grrrit-wm>	 (03PS4) 10Rush: check-raid: allow a file that specified utility [puppet] - 10https://gerrit.wikimedia.org/r/264395 
[22:40:32] <mutante>	 ...
[22:40:35] <grrrit-wm>	 (03PS1) 10Madhuvishy: wikimetrics: Make uwsgi serve static files on /static [puppet] - 10https://gerrit.wikimedia.org/r/264400 
[22:40:56] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] remove gadolinium, erbium from site/installserver [puppet] - 10https://gerrit.wikimedia.org/r/264393 (https://phabricator.wikimedia.org/T123029) (owner: 10Dzahn)
[22:40:58] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 031] check-raid: allow a file that specified utility [puppet] - 10https://gerrit.wikimedia.org/r/264395 (owner: 10Rush)
[22:41:43] <grrrit-wm>	 (03Merged) 10jenkins-bot: Set $wgCentralAuthUseSlaves for testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263682 (https://phabricator.wikimedia.org/T119689) (owner: 10Aaron Schulz)
[22:42:06] <grrrit-wm>	 (03PS5) 10Rush: check-raid: allow a file that specified utility [puppet] - 10https://gerrit.wikimedia.org/r/264395 
[22:42:27] <grrrit-wm>	 (03PS2) 10Hoo man: Provide a latest link for the Wikidata JSON dumps [puppet] - 10https://gerrit.wikimedia.org/r/261949 (https://phabricator.wikimedia.org/T72247) 
[22:42:32] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] wikimetrics: Make uwsgi serve static files on /static [puppet] - 10https://gerrit.wikimedia.org/r/264400 (owner: 10Madhuvishy)
[22:43:05] <logmsgbot>	 !log aaron@tin Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthUseSlaves for testwiki (duration: 00m 33s)
[22:43:08] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:43:22] <grrrit-wm>	 (03PS2) 10Madhuvishy: wikimetrics: Make uwsgi serve static files on /static [puppet] - 10https://gerrit.wikimedia.org/r/264400 
[22:43:35] <wikibugs>	 6operations, 7Tracking: Reinstall erbium with jessie - https://phabricator.wikimedia.org/T123722#1938234 (10Dzahn) - server was already shut down but still reported in icinga - removed from puppet/ install-server (T123029) - puppetstoredconfigclean to kill from icinga
[22:43:42] <grrrit-wm>	 (03CR) 10Rush: [C: 032 V: 032] check-raid: allow a file that specified utility [puppet] - 10https://gerrit.wikimedia.org/r/264395 (owner: 10Rush)
[22:43:47] <wikibugs>	 6operations, 7Tracking: decom erbium (was: Reinstall erbium with jessie) - https://phabricator.wikimedia.org/T123722#1938235 (10Dzahn)
[22:44:10] <grrrit-wm>	 (03PS3) 10Hashar: contint: rsync server to hold jobs caches [puppet] - 10https://gerrit.wikimedia.org/r/253322 (https://phabricator.wikimedia.org/T116017) 
[22:44:13] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] wikimetrics: Make uwsgi serve static files on /static [puppet] - 10https://gerrit.wikimedia.org/r/264400 (owner: 10Madhuvishy)
[22:44:49] <wikibugs>	 7Blocked-on-Operations, 6operations, 10ops-eqiad, 5Patch-For-Review: reclaim erbium, gadolinium into spares - https://phabricator.wikimedia.org/T123029#1938245 (10Dzahn)
[22:44:50] <wikibugs>	 6operations, 7Tracking: decom erbium (was: Reinstall erbium with jessie) - https://phabricator.wikimedia.org/T123722#1936447 (10Dzahn)
[22:45:01] <grrrit-wm>	 (03PS3) 10ArielGlenn: Provide a latest link for the Wikidata JSON dumps [puppet] - 10https://gerrit.wikimedia.org/r/261949 (https://phabricator.wikimedia.org/T72247) (owner: 10Hoo man)
[22:45:58] <grrrit-wm>	 (03PS3) 10Madhuvishy: wikimetrics: Make uwsgi serve static files on /static [puppet] - 10https://gerrit.wikimedia.org/r/264400 
[22:46:37] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] Provide a latest link for the Wikidata JSON dumps [puppet] - 10https://gerrit.wikimedia.org/r/261949 (https://phabricator.wikimedia.org/T72247) (owner: 10Hoo man)
[22:47:30] <grrrit-wm>	 (03PS1) 10Dzahn: remove public IP of gadolinium [dns] - 10https://gerrit.wikimedia.org/r/264402 (https://phabricator.wikimedia.org/T123029) 
[22:48:19] <grrrit-wm>	 (03PS2) 10Dzahn: remove public IP of gadolinium [dns] - 10https://gerrit.wikimedia.org/r/264402 (https://phabricator.wikimedia.org/T123029) 
[22:48:24] <grrrit-wm>	 (03PS4) 10Yuvipanda: wikimetrics: Make uwsgi serve static files on /static [puppet] - 10https://gerrit.wikimedia.org/r/264400 (owner: 10Madhuvishy)
[22:48:37] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] remove public IP of gadolinium [dns] - 10https://gerrit.wikimedia.org/r/264402 (https://phabricator.wikimedia.org/T123029) (owner: 10Dzahn)
[22:48:44] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] wikimetrics: Make uwsgi serve static files on /static [puppet] - 10https://gerrit.wikimedia.org/r/264400 (owner: 10Madhuvishy)
[22:50:00] <wikibugs>	 6operations, 7Tracking: decom erbium/gadolinium (was: Reinstall erbium with jessie) - https://phabricator.wikimedia.org/T123722#1938271 (10Dzahn)
[22:50:24] <wikibugs>	 6operations, 7Tracking: decom erbium/gadolinium (was: Reinstall erbium with jessie) - https://phabricator.wikimedia.org/T123722#1936447 (10Dzahn) gadolinium:  - shut down server - removed public IP https://gerrit.wikimedia.org/r/#/c/264402/ - removed from puppet/installserver
[22:52:18] <wikibugs>	 7Blocked-on-Operations, 6operations, 10ops-eqiad, 5Patch-For-Review: reclaim erbium, gadolinium into spares - https://phabricator.wikimedia.org/T123029#1938278 (10Dzahn) @cmjohnson per IRC talk, all done except wiping and the usual final steps from the lifecycle
[22:59:55] <wikibugs>	 6operations, 6Discovery, 10Wikimedia-Logstash, 7Elasticsearch: Upgrade ElasticSearch to 1.7.4 - https://phabricator.wikimedia.org/T122697#1938305 (10Florian) Hmm, the latest release of Elastica (which can be used on the wmf cluster[1]) seems to support 1.7.3 only (I don't know, which features don't work):...
[23:01:16] <wikibugs>	 6operations, 10Gerrit, 10GitHub-Mirrors, 10ValueView, and 2 others: [Bug] ValueView GitHub mirror not updated any more - https://phabricator.wikimedia.org/T123521#1938325 (10JanZerebecki) It seems this repo was before moved from gerrit to github. During the migration to gerrit I left refs/meta/config on th...
[23:03:02] <grrrit-wm>	 (03PS1) 10Madhuvishy: wikimetrics: Allow CORS for /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264405 
[23:03:55] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] wikimetrics: Allow CORS for /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264405 (owner: 10Madhuvishy)
[23:04:27] <grrrit-wm>	 (03PS2) 10Madhuvishy: wikimetrics: Allow CORS for /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264405 
[23:05:21] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] wikimetrics: Allow CORS for /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264405 (owner: 10Madhuvishy)
[23:08:02] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on labcontrol1002 is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).
[23:08:29] <grrrit-wm>	 (03PS3) 10Madhuvishy: wikimetrics: Allow CORS for /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264405 
[23:08:57] <wikibugs>	 6operations, 6Parsing-Team, 10Parsoid, 6Services: Update ruthenium to Ubuntu 14.04 from Ubuntu 12.04 - https://phabricator.wikimedia.org/T122328#1938339 (10Dzahn) @ssastry Would it also be ok if we used Debian and re-installed this with jessie?
[23:09:26] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] wikimetrics: Allow CORS for /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264405 (owner: 10Madhuvishy)
[23:16:40] <wikibugs>	 6operations, 6Parsing-Team, 10Parsoid, 6Services: Update ruthenium to Ubuntu 14.04 from Ubuntu 12.04 - https://phabricator.wikimedia.org/T122328#1938358 (10ssastry) >>! In T122328#1938339, @Dzahn wrote: > @ssastry Would it also be ok if we used Debian and re-installed this with jessie?   What perfect timin...
[23:21:08] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: Upgrade Mailman to version 3 - https://phabricator.wikimedia.org/T52864#1938389 (10RobLa-WMF) @legoktm and I both replied to on [meta:Talk:Discourse](https://meta.wikimedia.org/wiki/Talk:Discourse#Mailman_3) about this upgrade.  Update?
[23:21:30] <grrrit-wm>	 (03PS1) 10Madhuvishy: wikimetrics: Quote route-if param for uwsgi correctly for parameter expansion [puppet] - 10https://gerrit.wikimedia.org/r/264408 
[23:21:48] <wikibugs>	 6operations, 6Discovery, 10Wikimedia-Logstash, 7Elasticsearch: Upgrade ElasticSearch to 1.7.4 - https://phabricator.wikimedia.org/T122697#1938390 (10Reedy) We're on 2.2.0 atm  https://gerrit.wikimedia.org/r/#/c/261768/  https://github.com/ruflin/Elastica/releases/tag/2.2.1  2.2.1 supports PHP >= 5.3.3 http...
[23:22:32] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] wikimetrics: Quote route-if param for uwsgi correctly for parameter expansion [puppet] - 10https://gerrit.wikimedia.org/r/264408 (owner: 10Madhuvishy)
[23:23:05] <YuviPanda>	 mutante: hashar ^ jenkins is wrong here
[23:23:12] <wikibugs>	 6operations: Reinstall magnesium with jessie - https://phabricator.wikimedia.org/T123713#1938392 (10Dzahn) The plan is already to move RT to a VM as well and all other services have been removed to be able to reclaim magnesium.
[23:23:25] <YuviPanda>	 we want to pass ${PATH_INFO} as a raw string
[23:23:27] <YuviPanda>	 in puppet
[23:23:32] <YuviPanda>	 any idea how to do that without pissing puppet-lint off?
[23:23:47] <hashar>	 YuviPanda: blame puppet-lint 1.1.0 :-}
[23:24:15] <hashar>	 there other occurrences though
[23:24:24] <hashar>	 not sure how they got fixed
[23:24:35] <hashar>	 but i think the gerrit role has a similar need
[23:24:54] <wikibugs>	 6operations, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation, 6Services, and 2 others: Migrate CXServer to Node 4.2 and Jessie - https://phabricator.wikimedia.org/T107307#1938393 (10mobrovac)
[23:24:55] <mutante>	 YuviPanda: you can put # lint:ignore: ... 
[23:25:03] <mutante>	 around that one line
[23:25:05] <YuviPanda>	 >                     # lint:ignore:single_quote_string_with_variables
[23:25:06] <YuviPanda>	 right
[23:25:09] <mutante>	 that
[23:25:22] <YuviPanda>	 awesome
[23:25:25] <YuviPanda>	 thanks :D
[23:25:54] <mutante>	 quite welcome. i used # lint:ignore:quoted_booleans in a few place for similar reasons
[23:26:30] <wikibugs>	 6operations, 6Services: Migrate SCA cluster to SCB (Jessie and Node 4.2) - https://phabricator.wikimedia.org/T96017#1938404 (10mobrovac)
[23:26:37] <grrrit-wm>	 (03PS2) 10Madhuvishy: wikimetrics: Quote route-if param for uwsgi correctly for parameter expansion [puppet] - 10https://gerrit.wikimedia.org/r/264408 
[23:27:30] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] wikimetrics: Quote route-if param for uwsgi correctly for parameter expansion [puppet] - 10https://gerrit.wikimedia.org/r/264408 (owner: 10Madhuvishy)
[23:28:03] <wikibugs>	 6operations, 5Patch-For-Review: move RT off of magnesium - https://phabricator.wikimedia.org/T119112#1818428 (10Dzahn)
[23:28:04] <wikibugs>	 6operations: Reinstall magnesium with jessie - https://phabricator.wikimedia.org/T123713#1938418 (10Dzahn)
[23:29:57] <grrrit-wm>	 (03PS3) 10Madhuvishy: wikimetrics: Quote route-if param for uwsgi correctly for parameter expansion [puppet] - 10https://gerrit.wikimedia.org/r/264408 
[23:30:51] <wikibugs>	 6operations, 6Services: Migrate SCA cluster to SCB (Jessie and Node 4.2) - https://phabricator.wikimedia.org/T96017#1938431 (10mobrovac)
[23:31:02] <wikibugs>	 6operations, 6Parsing-Team, 10Parsoid, 6Services: Update ruthenium to Ubuntu 14.04 from Ubuntu 12.04 - https://phabricator.wikimedia.org/T122328#1938432 (10ssastry) @mobrovac tells me that the default nodejs package in jessie is node 4.2 ... so, that is fine.   If we find that we are not ready to migrate t...
[23:31:33] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] wikimetrics: Quote route-if param for uwsgi correctly for parameter expansion [puppet] - 10https://gerrit.wikimedia.org/r/264408 (owner: 10Madhuvishy)
[23:33:04] <wikibugs>	 6operations, 6Services: Migrate SCA cluster to SCB (Jessie and Node 4.2) - https://phabricator.wikimedia.org/T96017#1938447 (10mobrovac)
[23:35:25] <wikibugs>	 6operations: Reinstall magnesium with jessie - https://phabricator.wikimedia.org/T123713#1938459 (10Dzahn) This is also waiting on T118176
[23:35:30] <wikibugs>	 7Puppet: Module uwsgi doesn't allow passing multiple config params of same name - https://phabricator.wikimedia.org/T123809#1938463 (10madhuvishy) 3NEW
[23:35:46] <wikibugs>	 7Puppet, 6operations: Module uwsgi doesn't allow passing multiple config params of same name - https://phabricator.wikimedia.org/T123809#1938471 (10madhuvishy)
[23:35:57] <wikibugs>	 6operations, 5Patch-For-Review: move RT off of magnesium - https://phabricator.wikimedia.org/T119112#1938473 (10Dzahn)
[23:44:43] <grrrit-wm>	 (03PS1) 10Madhuvishy: wikimetrics: Fix allow CORS rule to apply to requests starting with /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264412 
[23:49:31] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] wikimetrics: Fix allow CORS rule to apply to requests starting with /static/public [puppet] - 10https://gerrit.wikimedia.org/r/264412 (owner: 10Madhuvishy)
[23:50:51] <grrrit-wm>	 (03PS5) 10Subramanya Sastry: Add the visualdiff module + instantiate visualdiffing services [puppet] - 10https://gerrit.wikimedia.org/r/264032 
[23:52:09] <grrrit-wm>	 (03CR) 10Subramanya Sastry: "Waiting on the integration/visualdiff repository to be created. Request filed yesterday @ https://www.mediawiki.org/wiki/Git/New_repositor" [puppet] - 10https://gerrit.wikimedia.org/r/264032 (owner: 10Subramanya Sastry)
[23:53:03] <grrrit-wm>	 (03PS6) 10Subramanya Sastry: Add the visualdiff module + instantiate visualdiffing services [puppet] - 10https://gerrit.wikimedia.org/r/264032 (https://phabricator.wikimedia.org/T118778) 
[23:59:11] <grrrit-wm>	 (03PS1) 10Madhuvishy: wikimetrics: Re-enable SSL redirection [puppet] - 10https://gerrit.wikimedia.org/r/264417