[00:01:11] I hate https://gerrit.wikimedia.org/r/#/c/204536 [00:02:53] (03CR) 10MZMcBride: "I'm sorry, but I hate this." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204536 (https://phabricator.wikimedia.org/T94856) (owner: 10Glaisher) [00:03:33] That's about as close as I get to a -1. [00:06:26] Oh, you were asking about the bad CAPTCHA limits. [00:06:35] I always have such trouble reading that part of the config. [00:07:30] A particular IP can trigger a bad CAPTCHA error 1000 times in an hour? [00:07:42] And new users can trigger the error 1000 times in an hour? [00:07:50] That's per-wiki? [00:08:24] Oh, right, there's $wgRateLimits and then a separate array()... [00:09:45] Fiona, I would encourage you to file a new task about the commons repository opt-out thing you hate :) [00:15:10] jamesofur, did you guys see https://gerrit.wikimedia.org/r/#/c/194075/1 ? [00:16:10] pretty sure we should keep the wikipedia logos, icons, etc. on the special wikipedia sites [00:17:40] * jamesofur looks [00:18:29] do we know how many are dropped out from that? [00:18:36] ahh I see comments [00:18:42] * jamesofur adds not to look this evening [00:23:03] *adds note [00:25:02] (03CR) 10Alex Monk: [C: 031] Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [00:27:51] PROBLEM - Outgoing network saturation on labstore1001 is CRITICAL 20.69% of data above the critical threshold [100000000.0] [00:42:41] PROBLEM - puppet last run on zirconium is CRITICAL puppet fail [00:43:41] PROBLEM - puppet last run on erbium is CRITICAL Puppet has 1 failures [00:44:12] PROBLEM - puppet last run on mw1214 is CRITICAL Puppet has 3 failures [00:44:21] PROBLEM - puppet last run on mw1013 is CRITICAL Puppet has 1 failures [00:44:31] PROBLEM - puppet last run on praseodymium is CRITICAL Puppet has 2 failures [00:44:51] PROBLEM - puppet last run on mw1094 is CRITICAL Puppet has 2 failures [00:44:51] PROBLEM - puppet last run on mw1234 is CRITICAL Puppet has 1 failures [00:44:51] PROBLEM - puppet last run on elastic1016 is CRITICAL Puppet has 1 failures [00:45:10] PROBLEM - puppet last run on mw1191 is CRITICAL Puppet has 3 failures [00:45:11] PROBLEM - puppet last run on mw1184 is CRITICAL Puppet has 1 failures [00:45:11] PROBLEM - puppet last run on mw1127 is CRITICAL Puppet has 1 failures [00:48:58] 6operations: Drop database table "optin_survey" from Wikimedia wikis - https://phabricator.wikimedia.org/T54934#1219229 (10Krenair) [00:49:14] 6operations: Drop database table "namespaces" from Wikimedia wikis - https://phabricator.wikimedia.org/T54929#1219232 (10Krenair) [00:49:15] (03CR) 10Jforrester: [C: 04-1] "I feel like this should be (at minimum) done in an extension, ideally in core. This isn't remotely "config"." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/205013 (owner: 10Ricordisamoa) [00:49:30] 6operations: Drop database table "hashs" from Wikimedia wikis - https://phabricator.wikimedia.org/T54927#1219235 (10Krenair) [00:53:09] 6operations: Investigate dropping "edit_page_tracking" database table from Wikimedia wikis after archiving it - https://phabricator.wikimedia.org/T57385#1219244 (10Krenair) [00:53:38] 6operations: Drop database table "email_capture" from Wikimedia wikis - https://phabricator.wikimedia.org/T57676#1219250 (10Krenair) [00:54:18] (03PS6) 10Alex Monk: Add namespace aliases for shwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/118654 (https://phabricator.wikimedia.org/T58169) (owner: 10Gerrit Patch Uploader) [00:54:29] (03PS7) 10Alex Monk: Add namespace aliases for shwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/118654 (https://phabricator.wikimedia.org/T58169) (owner: 10Gerrit Patch Uploader) [00:55:21] RECOVERY - Outgoing network saturation on labstore1001 is OK Less than 10.00% above the threshold [75000000.0] [00:56:40] 6operations: Drop *_old database tables from Wikimedia wikis - https://phabricator.wikimedia.org/T54932#1219254 (10Krenair) [00:58:10] RECOVERY - puppet last run on erbium is OK Puppet is currently enabled, last run 3 seconds ago with 0 failures [00:58:13] 6operations: Drop PovWatch extension-related database tables from Wikimedia wikis - https://phabricator.wikimedia.org/T54924#1219258 (10Krenair) [00:59:01] RECOVERY - puppet last run on praseodymium is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:59:20] RECOVERY - puppet last run on mw1094 is OK Puppet is currently enabled, last run 0 seconds ago with 0 failures [00:59:30] RECOVERY - puppet last run on elastic1016 is OK Puppet is currently enabled, last run 55 seconds ago with 0 failures [00:59:42] RECOVERY - puppet last run on mw1191 is OK Puppet is currently enabled, last run 46 seconds ago with 0 failures [00:59:50] RECOVERY - puppet last run on mw1184 is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:59:50] RECOVERY - puppet last run on mw1127 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [01:00:31] RECOVERY - puppet last run on mw1214 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [01:00:31] RECOVERY - puppet last run on zirconium is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [01:00:40] RECOVERY - puppet last run on mw1013 is OK Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:01:01] RECOVERY - puppet last run on mw1234 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [01:43:39] (03CR) 10Manybubbles: "Reindex is not required for turning it off. I'm not 100% sure if you turn it back on though. You might have to do a reindex then. I'd have" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204536 (https://phabricator.wikimedia.org/T94856) (owner: 10Glaisher) [02:02:35] Krenair: I filed a separate task about foundationwiki. [02:19:54] !log l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 36s) [02:20:11] Logged the message, Master [02:24:25] !log LocalisationUpdate completed (1.26wmf1) at 2015-04-20 02:23:22+00:00 [02:24:29] Logged the message, Master [02:39:13] !log l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 19s) [02:39:21] Logged the message, Master [02:43:20] !log LocalisationUpdate completed (1.26wmf2) at 2015-04-20 02:42:17+00:00 [02:43:24] Logged the message, Master [02:58:53] (03PS2) 10Alex Monk: Run updateArticleCount on the 21st of each month, not 29th [puppet] - 10https://gerrit.wikimedia.org/r/205187 (https://phabricator.wikimedia.org/T68867) [02:59:43] (03CR) 10Alex Monk: "OK, 21st. I don't think any other cron jobs try to run specifically on the 21st." [puppet] - 10https://gerrit.wikimedia.org/r/205187 (https://phabricator.wikimedia.org/T68867) (owner: 10Alex Monk) [03:31:51] PROBLEM - puppet last run on ms-fe3002 is CRITICAL puppet fail [03:33:30] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL Anomaly detected: 10 data above and 0 below the confidence bounds [03:40:41] PROBLEM - puppet last run on es2006 is CRITICAL Puppet has 1 failures [03:44:52] PROBLEM - puppet last run on uranium is CRITICAL Puppet has 1 failures [03:46:00] PROBLEM - puppet last run on mw1161 is CRITICAL Puppet has 1 failures [03:46:22] PROBLEM - puppet last run on mw1252 is CRITICAL Puppet has 3 failures [03:46:41] PROBLEM - puppet last run on mw1132 is CRITICAL Puppet has 1 failures [03:47:01] PROBLEM - puppet last run on mw1233 is CRITICAL Puppet has 1 failures [03:47:11] PROBLEM - puppet last run on mw2074 is CRITICAL Puppet has 1 failures [03:47:20] PROBLEM - puppet last run on mw1138 is CRITICAL Puppet has 1 failures [03:47:30] PROBLEM - puppet last run on mw1028 is CRITICAL Puppet has 1 failures [03:47:31] PROBLEM - puppet last run on mw1005 is CRITICAL Puppet has 1 failures [03:49:40] RECOVERY - puppet last run on ms-fe3002 is OK Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:58:22] RECOVERY - puppet last run on es2006 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:00:20] RECOVERY - puppet last run on mw1138 is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:00:31] RECOVERY - puppet last run on mw1161 is OK Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:01:00] RECOVERY - puppet last run on mw1252 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:01:20] RECOVERY - puppet last run on mw1132 is OK Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:01:31] RECOVERY - puppet last run on mw1233 is OK Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:01:32] (03CR) 10Yuvipanda: [C: 032] Tools: Ensure that webservice is not called by normal users [puppet] - 10https://gerrit.wikimedia.org/r/205184 (https://phabricator.wikimedia.org/T66219) (owner: 10Tim Landscheidt) [04:02:01] RECOVERY - puppet last run on mw1028 is OK Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:02:01] RECOVERY - puppet last run on mw1005 is OK Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:02:41] RECOVERY - puppet last run on uranium is OK Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:03:22] RECOVERY - puppet last run on mw2074 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:43:51] PROBLEM - puppet last run on stat1002 is CRITICAL puppet fail [04:48:55] (03PS2) 10KartikMistry: CX: Enable Content Translation in given wikis [puppet] - 10https://gerrit.wikimedia.org/r/204725 (https://phabricator.wikimedia.org/T95848) [04:53:57] (03PS2) 10KartikMistry: CX: Enable Content Translation in given wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204722 (https://phabricator.wikimedia.org/T95848) [05:06:08] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 20 05:05:04 UTC 2015 (duration 5m 3s) [05:06:14] Logged the message, Master [05:12:21] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected [05:27:37] <_joe_> morning [05:33:51] RECOVERY - puppet last run on stat1002 is OK Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:38:50] PROBLEM - puppet last run on mw1215 is CRITICAL Puppet has 1 failures [05:50:34] (03PS1) 10Yuvipanda: tools: Default uwsgi tools to having 4 worker processes [puppet] - 10https://gerrit.wikimedia.org/r/205198 [05:50:45] (03PS2) 10Yuvipanda: tools: Default uwsgi tools to having 4 worker processes [puppet] - 10https://gerrit.wikimedia.org/r/205198 [05:50:52] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Default uwsgi tools to having 4 worker processes [puppet] - 10https://gerrit.wikimedia.org/r/205198 (owner: 10Yuvipanda) [05:51:20] (03PS1) 10Yuvipanda: tools: Clean up unused imports in tool-nodejs [puppet] - 10https://gerrit.wikimedia.org/r/205199 [05:51:44] (03PS2) 10Yuvipanda: tools: Clean up unused imports in tool-nodejs [puppet] - 10https://gerrit.wikimedia.org/r/205199 [05:52:01] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Clean up unused imports in tool-nodejs [puppet] - 10https://gerrit.wikimedia.org/r/205199 (owner: 10Yuvipanda) [05:56:31] RECOVERY - puppet last run on mw1215 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:06:50] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [06:07:01] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [06:24:51] <_joe_> !log testing new pybal package on lvs2003 [06:24:56] Logged the message, Master [06:27:04] (03PS1) 10Yuvipanda: tools: Set $PORT as well for tool-nodejs [puppet] - 10https://gerrit.wikimedia.org/r/205200 [06:27:17] (03PS2) 10Yuvipanda: tools: Set $PORT as well for tool-nodejs [puppet] - 10https://gerrit.wikimedia.org/r/205200 [06:27:24] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Set $PORT as well for tool-nodejs [puppet] - 10https://gerrit.wikimedia.org/r/205200 (owner: 10Yuvipanda) [06:29:31] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [06:29:51] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [06:30:10] PROBLEM - puppet last run on cp3042 is CRITICAL Puppet has 1 failures [06:30:41] PROBLEM - puppet last run on cp3037 is CRITICAL Puppet has 1 failures [06:30:51] PROBLEM - puppet last run on ms-fe1004 is CRITICAL Puppet has 1 failures [06:32:10] PROBLEM - puppet last run on mw1228 is CRITICAL Puppet has 3 failures [06:35:32] PROBLEM - puppet last run on mw2016 is CRITICAL Puppet has 1 failures [06:37:10] PROBLEM - puppet last run on mw2096 is CRITICAL Puppet has 1 failures [06:45:10] RECOVERY - puppet last run on mw1228 is OK Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:45:30] RECOVERY - puppet last run on ms-fe1004 is OK Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:47:01] RECOVERY - puppet last run on cp3037 is OK Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:47:01] RECOVERY - puppet last run on mw2016 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:00] RECOVERY - puppet last run on cp3042 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:31] RECOVERY - puppet last run on mw2096 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:56] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/185329 (owner: 10Dzahn) [07:02:54] (03PS3) 10Giuseppe Lavagetto: runcommand: Twisted 13.x compatibility [debs/pybal] - 10https://gerrit.wikimedia.org/r/204035 [07:04:23] (03CR) 10Giuseppe Lavagetto: [C: 032] "Tested on trusty in this last version." [debs/pybal] - 10https://gerrit.wikimedia.org/r/204035 (owner: 10Giuseppe Lavagetto) [07:04:42] (03Merged) 10jenkins-bot: runcommand: Twisted 13.x compatibility [debs/pybal] - 10https://gerrit.wikimedia.org/r/204035 (owner: 10Giuseppe Lavagetto) [07:34:21] (03PS1) 10Giuseppe Lavagetto: ipvs: fixup for I2f58010d2e19004b2524e0858254f5b846478f6b [debs/pybal] - 10https://gerrit.wikimedia.org/r/205201 [07:47:38] (03PS1) 10Yuvipanda: tools: Fix typo in tool-uwsg-python [puppet] - 10https://gerrit.wikimedia.org/r/205202 [07:47:50] (03CR) 10jenkins-bot: [V: 04-1] tools: Fix typo in tool-uwsg-python [puppet] - 10https://gerrit.wikimedia.org/r/205202 (owner: 10Yuvipanda) [07:48:07] (03PS2) 10Yuvipanda: tools: Fix typo in tool-uwsg-python [puppet] - 10https://gerrit.wikimedia.org/r/205202 [07:48:45] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Fix typo in tool-uwsg-python [puppet] - 10https://gerrit.wikimedia.org/r/205202 (owner: 10Yuvipanda) [07:53:14] (03CR) 10Giuseppe Lavagetto: [C: 032] "Tested that this fixes ipvsadm invocation" [debs/pybal] - 10https://gerrit.wikimedia.org/r/205201 (owner: 10Giuseppe Lavagetto) [07:53:29] (03Merged) 10jenkins-bot: ipvs: fixup for I2f58010d2e19004b2524e0858254f5b846478f6b [debs/pybal] - 10https://gerrit.wikimedia.org/r/205201 (owner: 10Giuseppe Lavagetto) [08:04:43] (03PS1) 10Giuseppe Lavagetto: New version of the package, for trusty [debs/pybal] - 10https://gerrit.wikimedia.org/r/205205 [08:06:13] (03CR) 10Giuseppe Lavagetto: [C: 032] New version of the package, for trusty [debs/pybal] - 10https://gerrit.wikimedia.org/r/205205 (owner: 10Giuseppe Lavagetto) [08:06:29] (03Merged) 10jenkins-bot: New version of the package, for trusty [debs/pybal] - 10https://gerrit.wikimedia.org/r/205205 (owner: 10Giuseppe Lavagetto) [08:15:32] 6operations: Meta task for various security updates - https://phabricator.wikimedia.org/T96545#1220127 (10MoritzMuehlenhoff) p:5Triage>3Normal [08:15:40] godog: hey! going to sleep but not sure the extended counters on labmon took hold. tools.tools-services-02.WebServiceMonitor.* are still simple counters [08:15:49] do I need to turn something else on? [08:17:18] ugh, time to go sleep. stupid timezones [08:17:19] * YuviPanda waves [08:19:28] <_joe_> !log installed pybal 1.07 on lvs2003 [08:19:34] Logged the message, Master [08:20:11] YuviPanda: ack, I'll take a look [08:20:27] !log swift ms-be101[678] weight to 2600 [08:20:30] Logged the message, Master [08:22:30] 6operations, 5codfw-appserver-setup, 5wikis-in-codfw: Pybal RunCommand monitor doesn't work correctly on ubuntu trusty - https://phabricator.wikimedia.org/T94822#1220137 (10Joe) New package with various fixes is now on lvs2003, and works correctly. Installing the new pybal on all the trusty loadbalancers, th... [08:22:39] 6operations, 5codfw-appserver-setup, 5wikis-in-codfw: Set up the mediawiki application layer in codfw - https://phabricator.wikimedia.org/T86894#1220140 (10Joe) [08:22:39] 6operations, 5codfw-appserver-setup, 5wikis-in-codfw: Pybal RunCommand monitor doesn't work correctly on ubuntu trusty - https://phabricator.wikimedia.org/T94822#1220138 (10Joe) 5Open>3Resolved [08:35:20] PROBLEM - puppet last run on mw2139 is CRITICAL puppet fail [08:43:59] <_joe_> !log upgrading pybal on codfw loadbalancers [08:44:04] Logged the message, Master [08:46:57] YuviPanda: bah, I think graphite-web get confused when a metric exists as a .wsp file and a directory, data was there but I had to remove old names http://graphite.wmflabs.org/render/?width=604&height=402&_salt=1429519680.797&from=-4days&target=tools.tools-services-02.WebServiceMonitor.manifestscollected.upper&target=secondYAxis(tools.tools-services-02.WebServiceMonitor.manifestscollected.rate)&target=secondYAxis(tools.tools-services-02. [08:53:20] RECOVERY - puppet last run on mw2139 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [09:03:01] PROBLEM - puppet last run on stat1002 is CRITICAL puppet fail [09:12:42] RECOVERY - puppet last run on stat1002 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [09:20:28] 6operations, 10Datasets-General-or-Unknown: Dumps (or dump progress page) are stuck since Apr 19 Apr 07:24 UTC - https://phabricator.wikimedia.org/T96550#1220201 (10Ankry) 3NEW [09:21:57] 6operations, 10MediaWiki-General-or-Unknown, 10MediaWiki-JobRunner, 7Graphite: jobrunner metrics audit - https://phabricator.wikimedia.org/T95913#1220211 (10fgiunchedi) spoke with @ori and @aaron, we can drop per-type/per-wiki counters and leave only per-type (currently at 116) [09:33:12] hashar said I should ask here, can anybody recommend a book to get started with puppet? [09:33:36] what I have found so far is the learning vm https://puppetlabs.com/download-learning-vm and a list of recommended books https://puppetlabs.com/resources/books [09:34:02] my plan is to start with the VM, if nobody has book to recommend [09:36:40] zeljkof: on labs you can follow this guide to get your own puppet master to play with too https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [09:37:04] godog: thanks, will take a look [09:43:43] <_joe_> the best book on puppet I know is http://amzn.to/1aJXtTC [09:44:00] <_joe_> followed by "pro puppet", which is quite outdated tbh [09:45:26] zeljkof: I tried the VM from puppet labs, but it is based on puppet enterprise, while we use the open source version of puppet. they differ quite a bit wrt installation paths/command names (nothing one cannot figure out, but just as a heads up) [09:45:41] PROBLEM - puppet last run on eventlog1001 is CRITICAL puppet fail [09:47:07] <_joe_> zeljkof: I'd strongly suggest to get your own self-hosted puppetmaster in labs and work with it [09:47:11] <_joe_> it's way easier [09:47:52] <_joe_> zeljkof: then, most of my testing nowadays involves puppet agent --modulepath=$WMF_ROOT/puppet/modules test.pp [09:48:13] <_joe_> where test.pp is a test file where I test whatever I want to [09:48:32] <_joe_> usally with --noop directly on my computer [09:49:16] 6operations, 10RESTBase, 10RESTBase-Cassandra, 5Patch-For-Review: enable authenticated access to Cassandra JMX - https://phabricator.wikimedia.org/T92471#1220265 (10fgiunchedi) sharing the passwords file with cassandra seems easy enough, instead of group permissions I'd go for limited sudo nodetool as user... [09:51:27] 6operations, 7Graphite, 5Patch-For-Review: Counters now only provide rates (multiplied by 1000?) - https://phabricator.wikimedia.org/T95703#1220269 (10fgiunchedi) mw counters will be reduced a lot by T95913 (ETA 2-3d) so we'll be able to enable extended counters [09:52:47] _joe_: the best book on puppet is Necronomicon? :) [09:53:11] <_joe_> zeljkof: save this advice, and look back at it in one year [09:53:26] <_joe_> you'll see how good it was :P [09:53:47] _joe_: :D will do [09:54:02] moritzm: thanks for the heads up, will have that in mind [09:54:46] _joe_: I plan to use vagrant and local VMs to play with puppet, at least for start [09:55:34] <_joe_> zeljkof: mh, vagrant messes with the puppet installation on the guest VM AFAIR [09:56:01] _joe_: will have that in mind [09:56:02] <_joe_> oh sorry only in the case where the provider is puppet as well [09:56:17] <_joe_> so, don't use puppet as a provider if you want to use puppet in the VM [09:56:42] <_joe_> (is the necronomicon suggestion starting to make sense, eh?) [09:57:03] _joe_: slowly :D [09:59:26] 6operations: Meta task for various security updates - https://phabricator.wikimedia.org/T96545#1220282 (10MoritzMuehlenhoff) unzip: CVE-2015-1315 -> Installed on analytics1003,analytics1027,einsteinium,iridium,labsdb1001,ocg100[1-3],stat100[1-3],labnet1001,tin,antimony,iron,virt100[0-4],virt100[6-9],virt101[10],... [10:03:41] RECOVERY - puppet last run on eventlog1001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [10:23:11] 6operations, 10RESTBase, 10RESTBase-Cassandra: Set up multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1220318 (10fgiunchedi) I took a look at running multiple cassandra hosts on the same host, it looks like we'd need a different ip address for each instance since a n... [10:38:15] 6operations, 10RESTBase, 10hardware-requests: Expand RESTBase cluster capacity - https://phabricator.wikimedia.org/T93790#1220334 (10fgiunchedi) >>! In T93790#1217256, @GWicke wrote: > Update: [Current growth rates](http://grafana.wikimedia.org/#/dashboard/db/cassandra-restbase-eqiad?panelId=12&fullscreen) a... [10:41:41] 6operations: Choose a consistent, distributed k/v storage for configuration management/discovery - https://phabricator.wikimedia.org/T95656#1220342 (10Joe) ZK needs for analytics are completely different from the ones we have here, or I would have surely followed that path, @hashar. But given analytics is basic... [10:48:08] 6operations, 10RESTBase, 10RESTBase-Cassandra: Set up multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1220365 (10mobrovac) >>! In T95253#1220318, @fgiunchedi wrote: > I took a look at running multiple cassandra hosts on the same host, it looks like we'd need a differ... [12:04:50] (03PS3) 10KartikMistry: CX: Enable Content Translation in given wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204722 (https://phabricator.wikimedia.org/T95848) [13:04:06] 6operations, 10RESTBase, 10RESTBase-Cassandra: Set up multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1220476 (10fgiunchedi) >>! In T95253#1220365, @mobrovac wrote: >>>! In T95253#1220318, @fgiunchedi wrote: >> I took a look at running multiple cassandra hosts on the... [13:04:12] 6operations, 10RESTBase, 10RESTBase-Cassandra, 5Patch-For-Review: enable authenticated access to Cassandra JMX - https://phabricator.wikimedia.org/T92471#1220477 (10MoritzMuehlenhoff) Running nodetool through sudo also seems preferable to me. The upstream documentation https://wiki.apache.org/cassandra/J... [13:20:21] PROBLEM - puppet last run on cp3048 is CRITICAL puppet fail [13:24:30] (03PS2) 10Filippo Giunchedi: graphite: enable locking writes [puppet] - 10https://gerrit.wikimedia.org/r/199636 (https://phabricator.wikimedia.org/T86316) [13:24:40] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] graphite: enable locking writes [puppet] - 10https://gerrit.wikimedia.org/r/199636 (https://phabricator.wikimedia.org/T86316) (owner: 10Filippo Giunchedi) [13:24:45] 6operations: Choose a consistent, distributed k/v storage for configuration management/discovery - https://phabricator.wikimedia.org/T95656#1220486 (10hashar) @joe thank you for the explanation. [13:29:23] !log bounce carbon-cache on graphite1001 [13:29:32] Logged the message, Master [13:30:49] _joe_: so consul ships a single binary in 0.5.0_linux_amd64.zip [13:30:51] kind of lol [13:31:05] <_joe_> akosiaris_: yeah [13:31:08] well s/consul/hashicorp/ [13:31:15] <_joe_> akosiaris_: but a lot of things about consul are very nice [13:31:21] <_joe_> at least on paper [13:31:32] yeah, I am reading up on it... seems nice [13:31:36] <_joe_> I'm not sure I'm fond of how many things it does [13:32:37] <_joe_> health checks, dns, gossip, agent/server... [13:32:47] I like the health checks in theory [13:33:03] <_joe_> we already have pybal doing that [13:33:10] I am not sure about the implementation... anyway... still reading .. [13:33:28] <_joe_> yeah me too in fact [13:33:58] <_joe_> it's very well designed, form people that understand how a modern infrastructure should work [13:36:32] RECOVERY - puppet last run on cp3048 is OK Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:37:50] PROBLEM - Varnishkafka Delivery Errors per minute on cp4014 is CRITICAL 11.11% of data above the critical threshold [20000.0] [13:40:15] 6operations, 10RESTBase, 10RESTBase-Cassandra, 5Patch-For-Review: enable authenticated access to Cassandra JMX - https://phabricator.wikimedia.org/T92471#1220504 (10csteipp) What Moritz said sounds sane. [13:41:40] PROBLEM - Varnishkafka Delivery Errors per minute on cp4005 is CRITICAL 11.11% of data above the critical threshold [20000.0] [13:42:51] RECOVERY - Varnishkafka Delivery Errors per minute on cp4014 is OK Less than 1.00% above the threshold [0.0] [13:46:31] RECOVERY - Varnishkafka Delivery Errors per minute on cp4005 is OK Less than 1.00% above the threshold [0.0] [13:56:32] (03PS2) 10Giuseppe Lavagetto: 3.6.1: fix compilation errors [debs/hhvm] - 10https://gerrit.wikimedia.org/r/200513 [14:01:38] (03PS6) 10Hashar: contint: move zuul_merger_hosts to hiera, use in ferm [puppet] - 10https://gerrit.wikimedia.org/r/201882 (https://phabricator.wikimedia.org/T87519) (owner: 10Dzahn) [14:03:01] PROBLEM - Varnishkafka Delivery Errors per minute on cp4015 is CRITICAL 11.11% of data above the critical threshold [20000.0] [14:03:27] <_joe_> akosiaris_: do you remember how to push tags to gerrit? [14:03:40] <_joe_> I had some notes that seem to fail in this case [14:06:14] git push --tags [14:06:20] RECOVERY - Varnishkafka Delivery Errors per minute on cp4015 is OK Less than 1.00% above the threshold [0.0] [14:06:26] but you need the Push Annotated Tags permission _joe_ [14:06:30] <_joe_> uhm, only for annotated tags right? [14:06:38] yup [14:06:51] <_joe_> I'm pretty sure I had those [14:09:19] <_joe_> mh strange, my problem was with gerrit permissions, but I'm pretty sure they were different, as I already pushed a tag in the past [14:09:26] <_joe_> so I don't really get what happened there [14:11:33] (03PS1) 10Hashar: Trying ferm/hiera integration [puppet] - 10https://gerrit.wikimedia.org/r/205250 [14:13:04] 6operations: Choose a consistent, distributed k/v storage for configuration management/discovery - https://phabricator.wikimedia.org/T95656#1220576 (10mobrovac) Probably one thing we should also think about is integrating config management with rolling deployments, where a subset of nodes might need different (n... [14:13:53] <_joe_> akosiaris_: I can't find your HHVM patch in my reviews [14:14:58] _joe_: https://gerrit.wikimedia.org/r/#/c/200855/ [14:15:33] (03PS7) 10Hashar: contint: move zuul_merger_hosts to hiera, use in ferm [puppet] - 10https://gerrit.wikimedia.org/r/201882 (https://phabricator.wikimedia.org/T87519) (owner: 10Dzahn) [14:15:47] (03Abandoned) 10Hashar: Trying ferm/hiera integration [puppet] - 10https://gerrit.wikimedia.org/r/205250 (owner: 10Hashar) [14:16:11] <_joe_> I'm gonna manually rebase it and go on to build the new hhvm package on copper [14:17:21] 6operations, 10RESTBase, 10RESTBase-Cassandra, 5Patch-For-Review: enable authenticated access to Cassandra JMX - https://phabricator.wikimedia.org/T92471#1220602 (10GWicke) We can enforce localhost-only access from a specific group or root with iptables. @Dzahn already looked into that, and it seems that f... [14:25:28] (03PS2) 10Giuseppe Lavagetto: Copy build-dependencies to hhvm-dev dependencies [debs/hhvm] - 10https://gerrit.wikimedia.org/r/200855 (owner: 10Alexandros Kosiaris) [14:26:30] (03PS3) 10Giuseppe Lavagetto: Copy build-dependencies to hhvm-dev dependencies [debs/hhvm] - 10https://gerrit.wikimedia.org/r/200855 (owner: 10Alexandros Kosiaris) [14:29:23] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Copy build-dependencies to hhvm-dev dependencies [debs/hhvm] - 10https://gerrit.wikimedia.org/r/200855 (owner: 10Alexandros Kosiaris) [14:29:41] (03PS2) 10BBlack: ishmael - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198457 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:29:43] (03CR) 10Dzahn: [C: 032] "http://www-01.sil.org/iso639-3/documentation.asp?id=gom" [dns] - 10https://gerrit.wikimedia.org/r/205009 (https://phabricator.wikimedia.org/T96468) (owner: 10Glaisher) [14:30:10] 6operations, 6Labs, 7network: Slow network between virt nodes - https://phabricator.wikimedia.org/T96561#1220645 (10Andrew) 3NEW [14:30:41] (03CR) 10BBlack: [C: 032] ishmael - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198457 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:30:44] (03CR) 10Hashar: [C: 04-1] contint: move zuul_merger_hosts to hiera, use in ferm (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/201882 (https://phabricator.wikimedia.org/T87519) (owner: 10Dzahn) [14:31:10] (03PS3) 10BBlack: RT - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198455 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:31:24] (03CR) 10BBlack: [C: 032 V: 032] RT - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198455 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:31:37] (03CR) 10Dzahn: "for technical reasons if we touch only the lang list but nothing else in the zone templates it won't be created yet" [dns] - 10https://gerrit.wikimedia.org/r/205009 (https://phabricator.wikimedia.org/T96468) (owner: 10Glaisher) [14:32:00] (03CR) 10Dzahn: [C: 032] dbtree - Raise HSTS max-age to 1 year and add always flag [puppet] - 10https://gerrit.wikimedia.org/r/202267 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:32:08] (03PS3) 10BBlack: Add "always" flag when add HSTS header in Apache [puppet] - 10https://gerrit.wikimedia.org/r/199319 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:32:14] (03PS8) 10Hashar: contint: move zuul_merger_hosts to hiera, use in ferm [puppet] - 10https://gerrit.wikimedia.org/r/201882 (https://phabricator.wikimedia.org/T87519) (owner: 10Dzahn) [14:32:23] (03PS2) 10Dzahn: dbtree - Raise HSTS max-age to 1 year and add always flag [puppet] - 10https://gerrit.wikimedia.org/r/202267 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:32:49] (03CR) 10Ottomata: Add alerts for missing hours in pagecounts_all_sites and pagecounts_raw (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/205067 (owner: 10QChris) [14:32:52] (03CR) 10BBlack: [C: 032] Add "always" flag when add HSTS header in Apache [puppet] - 10https://gerrit.wikimedia.org/r/199319 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:33:08] 6operations, 6Labs: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220665 (10Andrew) 3NEW [14:33:42] (03PS3) 10BBlack: gdash - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198469 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:34:09] (03CR) 10Dzahn: [C: 032] scholarships - Increase HSTS max-age to 1 year [puppet] - 10https://gerrit.wikimedia.org/r/199126 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:34:13] 6operations, 6Labs: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220677 (10Andrew) Things I've checked already: - I can still create working instances on virt100x boxes. - Firewall settings don't seem to distinguish between virt100x and labvirt100x [14:34:21] (03PS3) 10Dzahn: scholarships - Increase HSTS max-age to 1 year [puppet] - 10https://gerrit.wikimedia.org/r/199126 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:34:31] mutante: I'm working through the same set of patches heh [14:34:47] bblack: i got inspired :) [14:34:49] but we seem to be going in opposite order [14:35:05] there's some I'm not sure about yet, I'm only hitting the ones that are already HTTPS-only [14:35:07] these 2 were sitting in my queue [14:35:28] (03CR) 10BBlack: [C: 032] gdash - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198469 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:35:31] same here, these were just increasing max-age [14:35:55] (03PS3) 10BBlack: donate - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199200 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:36:01] he's doing the "first 7 days then 1 year"-thing because chris once suggested it to be careful in the beginning [14:36:27] 6operations, 10RESTBase, 10RESTBase-Cassandra: Set up multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1220690 (10GWicke) > multiple instances per machine can also influence each other by e.g. slowing down or go down completely, is this something that cassandra takes... [14:37:01] (03PS4) 10BBlack: donate - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199200 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:37:04] endless rebase wars :P [14:37:11] (03CR) 10BBlack: [C: 032 V: 032] donate - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199200 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:37:33] (03PS2) 10BBlack: integration - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198458 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:38:51] 6operations, 6Labs: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220695 (10Andrew) My instinct is to suspect https://gerrit.wikimedia.org/r/#/c/205093/ but so far I can't provide any evidence that that's the culprit. [14:39:09] (03CR) 10Dzahn: [C: 032] doc - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198819 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:40:48] (03PS3) 10BBlack: integration - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198458 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:41:14] (03PS4) 10BBlack: integration - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198458 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:41:38] (03CR) 10BBlack: [C: 032 V: 032] "(added "always" in PS3 as well)" [puppet] - 10https://gerrit.wikimedia.org/r/198458 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [14:50:54] (03PS9) 10Hashar: contint: move zuul_merger_hosts to hiera, use in ferm [puppet] - 10https://gerrit.wikimedia.org/r/201882 (https://phabricator.wikimedia.org/T87519) (owner: 10Dzahn) [14:51:31] ^demon|away, thcipriani, marktraceur: Who wants to SWAT this morning? Only one change, and seems very trivial, but it'll need a full scap. [14:51:51] prtksxna: Ping for SWAT in 8 minutes [14:51:52] (03CR) 10Dzahn: [C: 032] Run updateArticleCount on the 21st of each month, not 29th [puppet] - 10https://gerrit.wikimedia.org/r/205187 (https://phabricator.wikimedia.org/T68867) (owner: 10Alex Monk) [14:52:03] anomie: o/ [14:52:31] anomie: can do [14:52:45] thcipriani: ok! [14:53:04] Yup, just an i18n change. [14:54:01] PROBLEM - puppet last run on mw1034 is CRITICAL Puppet has 1 failures [14:54:10] I need to backport to wmf/1.26wmf1 for the Wikipedias, right? [14:54:30] prtksxna: Yes. [14:54:53] James_F: Cool. Thanks! [14:56:15] (03CR) 10Hashar: [C: 031] "In the previous patches I:" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/201882 (https://phabricator.wikimedia.org/T87519) (owner: 10Dzahn) [14:56:39] (03CR) 10Hashar: [C: 04-1] "Pending review on https://gerrit.wikimedia.org/r/#/c/201882/" [puppet] - 10https://gerrit.wikimedia.org/r/204876 (https://phabricator.wikimedia.org/T96426) (owner: 10Hashar) [14:57:10] 6operations, 5Continuous-Integration-Isolation, 5Patch-For-Review: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1220728 (10hashar) Hiera / ferm is not as smooth as expected some work in progress is being done on https://gerrit.w... [14:59:00] 6operations, 6Labs: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220737 (10Andrew) Andrews-MBP-2:~ andrew$ ssh virt1000.wikimedia.org root@virt1000:~# source ~/novaenv.sh root@virt1000:~# nova boot --image e9587207-1fa2-4faf-926b-d9efc41b176c --flavo... [15:00:00] thcipriani: o/ [15:00:37] prtksxna: +2ing [15:00:43] manybubbles, anomie, ^d, thcipriani: Dear anthropoid, the time has come. Please deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150420T1500). [15:01:46] "just an i18n change" heh [15:02:28] (03CR) 10Giuseppe Lavagetto: [C: 031] "Very good patch, I'm not sure if we don't want to shuffle some variables around, but please merge :)" [puppet] - 10https://gerrit.wikimedia.org/r/204514 (owner: 10BBlack) [15:02:28] bd808: which will take a full deploy window :P [15:02:28] bd808: I meant no harm [15:02:46] prtksxna: not your fault that we haven't made scap faster yet [15:03:07] What is scap? [15:03:26] https://wikitech.wikimedia.org/wiki/Wikimedia_binaries#scap [15:03:41] https://doc.wikimedia.org/mw-tools-scap/ [15:03:54] It is the tool that we use to deploy on the cluster [15:04:05] one of the tools* [15:04:19] (03CR) 10Giuseppe Lavagetto: [C: 031] "Not sure I love depending on the hostname instead than on the realm, but this is nonetheless correct and simplifies the code, so +1 from m" [puppet] - 10https://gerrit.wikimedia.org/r/204552 (owner: 10BBlack) [15:04:32] That sure does a lot of things! [15:04:33] and an i18n change requires a "full scap" run which for various reasons takes ~30 minutes of wall clock time to complete [15:04:50] last one I ran took 41m 12s [15:05:02] *nod* [15:05:05] and jenkins can take 20 minutes [15:05:11] or more [15:05:13] > rebuilds localization caches of core and extensions [15:05:14] jenkins is much better now [15:05:19] oh :) [15:05:28] * prtksxna waits expectantly for jenkins [15:05:29] jenkins looks like it's on the road to 5min [15:06:00] it's now running mysql on tmpfs [15:07:21] (03CR) 10Giuseppe Lavagetto: [C: 031] "since these are just labels, it makes sense to have them being words; it also makes the templates slightly easier to read" [puppet] - 10https://gerrit.wikimedia.org/r/204773 (owner: 10BBlack) [15:07:57] okie doke, looks like jenkins bot has merged. [15:09:03] anomie: I'm unsure what standard procedure is for submodule updates the specific releases: do I just make the patch for 1.26wmf1 after scap? [15:10:11] to update the submodule to the latest commit [15:10:21] RECOVERY - puppet last run on mw1034 is OK Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:10:23] thcipriani: you need it before scap [15:10:59] kk, lemme make that then really quickly [15:11:03] thcipriani: Huh? You should make the patch for 1.26wmf1 mediawiki/core updating the submodule, and deploy that. [15:11:06] you need to bump the submodule on the 1.26wmf1 branch, upload to gerrit, merg, pull to tin and then scap [15:11:28] kk [15:13:28] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Small omission in labs (I think), otherwise LGTM" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/204545 (owner: 10BBlack) [15:13:36] (03CR) 10Alexandros Kosiaris: [C: 032] make carbon a ganglia_new aggregator for eqiad [puppet] - 10https://gerrit.wikimedia.org/r/204978 (https://phabricator.wikimedia.org/T93776) (owner: 10Dzahn) [15:14:07] 6operations, 7Graphite: scale graphite deployment (tracking) - https://phabricator.wikimedia.org/T85451#1220751 (10RobH) Host uranlum has two SSDs installed, and the R610 planform can usually support 4 2.5" disks installed. If you guys wanted to upgrade the disks, please create a hardware-request task linked... [15:16:35] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "The values of RAM you choose seem pulled a bit out of thin air without any comment in the puppet manifest." [puppet] - 10https://gerrit.wikimedia.org/r/204755 (owner: 10BBlack) [15:17:35] :) [15:17:48] <_joe_> bblack: hehe did you ask for a review? :P [15:18:05] yes, it's all good stuff [15:18:17] I'm gonna merge up through the ones that passed so far to reduce all the rebase/push traffic first [15:19:00] thcipriani: can I also add two other config patches for SWAT? [15:19:01] PROBLEM - puppet last run on carbon is CRITICAL puppet fail [15:19:07] actually I take that back, I'll fix these first and then wait to merge them all. Because I only really compiler-tested prod with the complete set. [15:19:33] Glaisher: just mediawiki-config patches? [15:19:38] yeah [15:19:59] if they're small, go for it [15:21:10] thcipriani: https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=154589&oldid=154574 [15:21:16] kk [15:23:04] thcipriani: Let me know when I should test :) [15:23:15] 6operations, 7Graphite: scale graphite deployment (tracking) - https://phabricator.wikimedia.org/T85451#1220764 (10RobH) So, I looked at uranium, not graphite1001. graphite1001 is an R420 with 4 disks already installed. So this host actually CANNOT add additional SSDs without removing existing disks. robh@g... [15:23:15] kk, going to be a few [15:23:24] Sure [15:23:49] (03PS1) 10Alexandros Kosiaris: Followup commit for 8a06ac1 [puppet] - 10https://gerrit.wikimedia.org/r/205263 [15:24:44] I'm trying to figure out why tin 1.26wmf1 has a different submodule checkout than extensions/popups 1.26wmf1 [15:33:15] (03CR) 10Alexandros Kosiaris: [C: 032] Followup commit for 8a06ac1 [puppet] - 10https://gerrit.wikimedia.org/r/205263 (owner: 10Alexandros Kosiaris) [15:33:31] Glaisher: going to do yours quickly, then get back to popups [15:33:43] ok! [15:33:55] (03CR) 10Thcipriani: [C: 032] Remove $wgSpamRegex in favor of AbuseFilter and SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/196782 (https://phabricator.wikimedia.org/T50491) (owner: 10Glaisher) [15:34:06] (03Merged) 10jenkins-bot: Remove $wgSpamRegex in favor of AbuseFilter and SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/196782 (https://phabricator.wikimedia.org/T50491) (owner: 10Glaisher) [15:35:20] RECOVERY - puppet last run on carbon is OK Puppet is currently enabled, last run 40 seconds ago with 0 failures [15:36:41] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [15:36:51] (03CR) 10BBlack: Apply class-param + hiera pattern to base+2layer (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/204545 (owner: 10BBlack) [15:37:06] !log thcipriani Synchronized wmf-config/InitialiseSettings.php: scap [[gerrit:196782]] (duration: 00m 11s) [15:37:11] Logged the message, Master [15:37:13] ^ Glaisher [15:37:19] thanks [15:37:48] lmk if it looks good. [15:41:03] thcipriani: ok.. the edit went through [15:41:11] kk, merging the other one [15:41:46] (03CR) 10BBlack: "Well the comment I could add on the formula-vs-1 cutoffs was "this is to avoid the case where floor() ends up returning a value of zero", " [puppet] - 10https://gerrit.wikimedia.org/r/204755 (owner: 10BBlack) [15:41:51] RECOVERY - Kafka Broker Messages In on analytics1021 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 5159.92324322 [15:41:53] (03CR) 10Thcipriani: [C: 032] Enable Other Projects Links by default on svwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/205081 (https://phabricator.wikimedia.org/T96502) (owner: 10Glaisher) [15:42:34] !log analytics1014 offline, due to cpu temp?? attempting to reboot. [15:42:40] Logged the message, Master [15:43:57] SWAT is going to run long today, sorry prtksxna submodules got me twisted up :( still waiting on core to merge the submodule bump [15:44:12] thcipriani: No worries, I am up for another hour or so :) [15:44:25] ugh [15:44:38] thcipriani: Anything I can do to help? [15:44:41] !log Updated iegreview to 7303e5a (Update wikitext report for Inspire) [15:44:47] Logged the message, Master [15:44:57] prtksxna: nope, just waiting for zuul right now https://integration.wikimedia.org/zuul/ [15:45:02] (03PS2) 10Papaul: added primary network MAC address for db2043-db2070 [puppet] - 10https://gerrit.wikimedia.org/r/204829 [15:45:21] RECOVERY - SSH on analytics1014 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 (protocol 2.0) [15:45:21] RECOVERY - dhclient process on analytics1014 is OK: PROCS OK: 0 processes with command name dhclient [15:45:22] RECOVERY - Disk space on analytics1014 is OK: DISK OK [15:45:22] RECOVERY - Hadoop NodeManager on analytics1014 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [15:45:22] RECOVERY - Hadoop DataNode on analytics1014 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [15:45:22] RECOVERY - salt-minion processes on analytics1014 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [15:45:22] RECOVERY - DPKG on analytics1014 is OK: All packages OK [15:45:30] RECOVERY - configured eth on analytics1014 is OK - interfaces up [15:45:30] RECOVERY - RAID on analytics1014 is OK no disks configured for RAID [15:45:31] RECOVERY - Host analytics1014 is UPING OK - Packet loss = 0%, RTA = 2.49 ms [15:46:04] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "One small doubt, otherwise correct." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/204756 (owner: 10BBlack) [15:48:05] (03Merged) 10jenkins-bot: Enable Other Projects Links by default on svwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/205081 (https://phabricator.wikimedia.org/T96502) (owner: 10Glaisher) [15:49:14] (03CR) 10Giuseppe Lavagetto: [C: 031] remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 (owner: 10BBlack) [15:49:57] (03CR) 10Giuseppe Lavagetto: [C: 031] hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 (owner: 10BBlack) [15:50:07] !log thcipriani Synchronized wmf-config/InitialiseSettings.php: scap [[gerrit:205081]] (duration: 00m 12s) [15:50:14] Logged the message, Master [15:50:14] ^ Glaisher patch 2 [15:51:08] working :) [15:51:11] thanks [15:51:32] (03CR) 10Giuseppe Lavagetto: [C: 031] drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 (owner: 10BBlack) [15:52:51] !log thcipriani Started scap: Morning swat for [[gerrit:205219]] [15:52:54] Logged the message, Master [15:52:55] (03PS2) 10Alexandros Kosiaris: base: is_virtual comparison fix [puppet] - 10https://gerrit.wikimedia.org/r/201445 [15:53:12] ^ prtksxna started, way way late :( [15:53:31] (03CR) 10BBlack: hieraize bits/upload domainname options for labs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/204756 (owner: 10BBlack) [15:53:55] \o/ [15:53:55] (03PS11) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [15:53:57] (03PS16) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [15:53:59] (03PS11) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [15:54:01] (03PS11) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [15:54:03] (03PS10) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [15:54:05] (03PS7) 10BBlack: normalize on $cluster_tier as string words [puppet] - 10https://gerrit.wikimedia.org/r/204773 [15:54:07] (03PS9) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [15:54:09] (03PS9) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [15:54:11] (03PS18) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [15:54:13] (03PS1) 10BBlack: backport ceiling() func from puppetlabs-stdlib v4.6.0 [puppet] - 10https://gerrit.wikimedia.org/r/205272 [15:54:15] (03PS1) 10BBlack: re-arrange to align classes a bit... [puppet] - 10https://gerrit.wikimedia.org/r/205273 [15:54:53] (03PS1) 10Hashar: (WIP) ferm: lame tests for service template (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/205274 [15:57:47] (03CR) 10Giuseppe Lavagetto: [C: 031] base: is_virtual comparison fix (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/201445 (owner: 10Alexandros Kosiaris) [16:00:39] 7Puppet, 6Phabricator: Initialize phd working directory for phd services - https://phabricator.wikimedia.org/T96567#1220826 (10Negative24) 3NEW a:3Negative24 [16:01:07] (03PS1) 10Giuseppe Lavagetto: Disable mysql socket support [debs/hhvm] - 10https://gerrit.wikimedia.org/r/205276 [16:01:30] scap proxies updated, updating apaches started [16:02:08] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] "sigh." [debs/hhvm] - 10https://gerrit.wikimedia.org/r/205276 (owner: 10Giuseppe Lavagetto) [16:02:10] (03PS3) 10Alexandros Kosiaris: base: is_virtual comparison fix [puppet] - 10https://gerrit.wikimedia.org/r/201445 [16:04:05] _joe_: I suppose export DEB_BUILD_OPTIONS=nocheck would not work, right ? [16:04:22] <_joe_> akosiaris_: this check is done in ./configure [16:04:34] oh, really ? [16:04:35] <_joe_> they choke the friggin build [16:04:37] come on... [16:04:43] <_joe_> really. [16:05:00] * _joe_ <3 Technology [16:05:08] here's a nice way to abuse autotools [16:05:29] <_joe_> oh they use cmake, and some obscure tool in ocaml too [16:05:34] <_joe_> so it's not autotools [16:05:36] (03CR) 10Alexandros Kosiaris: [C: 032] base: is_virtual comparison fix [puppet] - 10https://gerrit.wikimedia.org/r/201445 (owner: 10Alexandros Kosiaris) [16:06:11] * akosiaris_ is oblivious to cmake [16:06:21] never had to use it... and I don't regret it [16:06:33] I did not know it had a ./configure too [16:08:08] <_joe_> well it's there for retrocompat I suppose [16:08:12] so -C main for jessie wikimedia and etherpad-lite I suppose [16:08:18] no reason for backports or thirdparty [16:09:36] !log upload etherpad-lite 1.5.4-1 on apt.wikimedia.org. Not to be used in production yet, testing purposes [16:09:40] Logged the message, Master [16:10:54] prtksxna: apache sync done, waiting on rebuild cdb [16:12:45] 6operations, 10ops-fundraising, 7network: pfw-eqiad JunOS upgrade - https://phabricator.wikimedia.org/T96569#1220888 (10faidon) 3NEW [16:14:14] (03PS1) 10Faidon Liambotis: Remove pfw2-* loopback addresses, introduce pfw-* [dns] - 10https://gerrit.wikimedia.org/r/205279 [16:14:14] (03PS1) 10Faidon Liambotis: Assign subnet for pfw-eqiad<->pfw-codfw (tunnel) [dns] - 10https://gerrit.wikimedia.org/r/205280 [16:14:14] (03CR) 10jenkins-bot: [V: 04-1] Remove pfw2-* loopback addresses, introduce pfw-* [dns] - 10https://gerrit.wikimedia.org/r/205279 (owner: 10Faidon Liambotis) [16:14:14] (03CR) 10jenkins-bot: [V: 04-1] Assign subnet for pfw-eqiad<->pfw-codfw (tunnel) [dns] - 10https://gerrit.wikimedia.org/r/205280 (owner: 10Faidon Liambotis) [16:14:14] !log thcipriani Finished scap: Morning swat for [[gerrit:205219]] (duration: 21m 13s) [16:14:14] PROBLEM - puppet last run on db1057 is CRITICAL Puppet has 1 failures [16:14:14] ^ prtksxna [16:14:21] Logged the message, Master [16:14:21] PROBLEM - puppet last run on wtp1004 is CRITICAL Puppet has 1 failures [16:14:21] PROBLEM - puppet last run on es2007 is CRITICAL Puppet has 1 failures [16:14:31] PROBLEM - puppet last run on rdb2002 is CRITICAL Puppet has 1 failures [16:14:34] * prtksxna refreshes [16:14:40] PROBLEM - puppet last run on db1054 is CRITICAL Puppet has 1 failures [16:14:41] PROBLEM - puppet last run on mc1007 is CRITICAL Puppet has 1 failures [16:14:48] (03PS2) 10Faidon Liambotis: Remove pfw2-* loopback addresses, introduce pfw-* [dns] - 10https://gerrit.wikimedia.org/r/205279 [16:14:50] (03PS2) 10Faidon Liambotis: Assign subnet for pfw-eqiad<->pfw-codfw (tunnel) [dns] - 10https://gerrit.wikimedia.org/r/205280 [16:14:50] PROBLEM - puppet last run on wtp1015 is CRITICAL Puppet has 1 failures [16:14:50] PROBLEM - puppet last run on rdb1002 is CRITICAL Puppet has 1 failures [16:14:51] PROBLEM - puppet last run on analytics1014 is CRITICAL Puppet has 1 failures [16:14:51] PROBLEM - puppet last run on lvs2003 is CRITICAL Puppet has 1 failures [16:14:52] PROBLEM - puppet last run on db1001 is CRITICAL Puppet has 1 failures [16:14:52] PROBLEM - puppet last run on rcs1002 is CRITICAL Puppet has 1 failures [16:15:02] 6operations, 10ops-fundraising, 7network: pfw-eqiad JunOS upgrade - https://phabricator.wikimedia.org/T96569#1220906 (10Jgreen) [16:15:10] PROBLEM - puppet last run on titanium is CRITICAL Puppet has 1 failures [16:15:11] PROBLEM - puppet last run on elastic1002 is CRITICAL Puppet has 1 failures [16:15:11] PROBLEM - puppet last run on labsdb1004 is CRITICAL Puppet has 1 failures [16:15:11] PROBLEM - puppet last run on protactinium is CRITICAL Puppet has 1 failures [16:15:12] PROBLEM - puppet last run on ganeti1001 is CRITICAL Puppet has 1 failures [16:15:12] PROBLEM - puppet last run on wtp1007 is CRITICAL Puppet has 1 failures [16:15:15] Jeff_Green, mark:https://gerrit.wikimedia.org/r/205279 might be of interest [16:15:16] meh? [16:15:20] PROBLEM - puppet last run on wtp1013 is CRITICAL Puppet has 1 failures [16:15:20] PROBLEM - puppet last run on mc1013 is CRITICAL Puppet has 1 failures [16:15:21] PROBLEM - puppet last run on es1002 is CRITICAL Puppet has 1 failures [16:15:24] <_joe_> greg-g: looking [16:15:30] PROBLEM - puppet last run on wtp1011 is CRITICAL Puppet has 1 failures [16:15:30] PROBLEM - puppet last run on db1064 is CRITICAL Puppet has 1 failures [16:15:31] PROBLEM - puppet last run on analytics1001 is CRITICAL Puppet has 1 failures [16:15:31] PROBLEM - puppet last run on mw2144 is CRITICAL Puppet has 1 failures [16:15:31] PROBLEM - puppet last run on mw2193 is CRITICAL Puppet has 1 failures [16:15:31] PROBLEM - puppet last run on mw2187 is CRITICAL Puppet has 1 failures [16:15:31] PROBLEM - puppet last run on mw2155 is CRITICAL Puppet has 1 failures [16:15:40] PROBLEM - puppet last run on virt1000 is CRITICAL Puppet has 1 failures [16:15:40] PROBLEM - puppet last run on cp3034 is CRITICAL Puppet has 1 failures [16:15:41] PROBLEM - puppet last run on elastic1005 is CRITICAL Puppet has 1 failures [16:15:41] PROBLEM - puppet last run on nitrogen is CRITICAL Puppet has 1 failures [16:15:41] PROBLEM - puppet last run on cp1060 is CRITICAL Puppet has 1 failures [16:15:41] PROBLEM - puppet last run on ms-be1018 is CRITICAL Puppet has 1 failures [16:15:41] PROBLEM - puppet last run on mw1074 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1243 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1248 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on rdb1003 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1148 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1239 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on analytics1011 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on db1063 is CRITICAL Puppet has 1 failures [16:16:17] <_joe_> E: Unable to locate package microcode [16:16:17] PROBLEM - puppet last run on mw1077 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on ocg1003 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1032 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1225 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mc2009 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on db2003 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mc2002 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw2058 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mc2012 is CRITICAL Puppet has 1 failures [16:16:17] <_joe_> did someone remove it? [16:16:17] PROBLEM - puppet last run on mw2026 is CRITICAL Puppet has 1 failures [16:16:17] <_joe_> or moved it around? [16:16:17] PROBLEM - puppet last run on mw1108 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1105 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on graphite1001 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on ganeti2006 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on ganeti2002 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on rdb2004 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on mw1229 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on elastic1023 is CRITICAL Puppet has 1 failures [16:16:17] PROBLEM - puppet last run on rcs1001 is CRITICAL Puppet has 1 failures [16:16:20] PROBLEM - puppet last run on cp3049 is CRITICAL Puppet has 1 failures [16:16:20] PROBLEM - puppet last run on maerlant is CRITICAL Puppet has 1 failures [16:16:30] thcipriani: I don't see the change yet. [16:16:30] PROBLEM - puppet last run on elastic1017 is CRITICAL Puppet has 1 failures [16:16:31] PROBLEM - puppet last run on db1047 is CRITICAL Puppet has 1 failures [16:16:31] PROBLEM - puppet last run on pc1003 is CRITICAL Puppet has 1 failures [16:16:31] PROBLEM - puppet last run on mw1201 is CRITICAL Puppet has 1 failures [16:16:32] PROBLEM - puppet last run on db1030 is CRITICAL Puppet has 1 failures [16:16:32] PROBLEM - puppet last run on db1006 is CRITICAL Puppet has 1 failures [16:16:40] PROBLEM - puppet last run on mw1024 is CRITICAL Puppet has 1 failures [16:16:40] PROBLEM - puppet last run on mw1033 is CRITICAL Puppet has 1 failures [16:16:41] PROBLEM - puppet last run on dbstore1002 is CRITICAL Puppet has 1 failures [16:16:43] sigh that must be me [16:16:50] PROBLEM - puppet last run on db1070 is CRITICAL Puppet has 1 failures [16:16:51] PROBLEM - puppet last run on mw1022 is CRITICAL Puppet has 1 failures [16:16:51] PROBLEM - puppet last run on mw1121 is CRITICAL Puppet has 1 failures [16:16:52] PROBLEM - puppet last run on wtp1003 is CRITICAL Puppet has 1 failures [16:16:52] PROBLEM - puppet last run on fluorine is CRITICAL Puppet has 1 failures [16:17:01] PROBLEM - puppet last run on ganeti1004 is CRITICAL Puppet has 1 failures [16:17:01] PROBLEM - puppet last run on ms-fe2002 is CRITICAL Puppet has 1 failures [16:17:01] PROBLEM - puppet last run on db1061 is CRITICAL Puppet has 1 failures [16:17:02] PROBLEM - puppet last run on ms-fe3001 is CRITICAL Puppet has 1 failures [16:17:02] PROBLEM - puppet last run on mw1001 is CRITICAL Puppet has 1 failures [16:17:08] <_joe_> akosiaris_: so there was a bug that your patch fixed? [16:17:10] PROBLEM - puppet last run on mw2167 is CRITICAL Puppet has 1 failures [16:17:10] PROBLEM - puppet last run on wtp2013 is CRITICAL Puppet has 1 failures [16:17:10] PROBLEM - puppet last run on mw2186 is CRITICAL Puppet has 1 failures [16:17:11] PROBLEM - puppet last run on wtp2006 is CRITICAL Puppet has 1 failures [16:17:11] PROBLEM - puppet last run on mw2175 is CRITICAL Puppet has 1 failures [16:17:11] PROBLEM - puppet last run on mw2205 is CRITICAL Puppet has 1 failures [16:17:11] PROBLEM - puppet last run on mw2209 is CRITICAL Puppet has 1 failures [16:17:12] PROBLEM - puppet last run on mw2204 is CRITICAL Puppet has 1 failures [16:17:12] PROBLEM - puppet last run on mw2201 is CRITICAL Puppet has 1 failures [16:17:13] PROBLEM - puppet last run on mw2159 is CRITICAL Puppet has 1 failures [16:17:13] PROBLEM - puppet last run on ms-be1011 is CRITICAL Puppet has 1 failures [16:17:14] PROBLEM - puppet last run on mw2139 is CRITICAL Puppet has 1 failures [16:17:14] PROBLEM - puppet last run on mw2140 is CRITICAL Puppet has 1 failures [16:17:14] yeah [16:17:15] PROBLEM - puppet last run on mw2153 is CRITICAL Puppet has 1 failures [16:17:18] fixing [16:17:21] installing package microcode [16:17:30] PROBLEM - puppet last run on analytics1004 is CRITICAL Puppet has 1 failures [16:17:31] PROBLEM - puppet last run on db1044 is CRITICAL Puppet has 1 failures [16:17:31] PROBLEM - puppet last run on mw1043 is CRITICAL Puppet has 1 failures [16:17:31] PROBLEM - puppet last run on db1033 is CRITICAL Puppet has 1 failures [16:17:32] PROBLEM - puppet last run on mw2116 is CRITICAL Puppet has 1 failures [16:17:32] PROBLEM - puppet last run on mw2091 is CRITICAL Puppet has 1 failures [16:17:34] paravoid: how ? [16:17:40] PROBLEM - puppet last run on mw1185 is CRITICAL Puppet has 1 failures [16:17:41] PROBLEM - puppet last run on db2012 is CRITICAL Puppet has 1 failures [16:17:41] PROBLEM - puppet last run on mw1167 is CRITICAL Puppet has 1 failures [16:17:41] PROBLEM - puppet last run on copper is CRITICAL Puppet has 1 failures [16:17:41] PROBLEM - puppet last run on mw2031 is CRITICAL Puppet has 1 failures [16:17:41] PROBLEM - puppet last run on elastic1020 is CRITICAL Puppet has 1 failures [16:17:41] PROBLEM - puppet last run on mw2044 is CRITICAL Puppet has 1 failures [16:17:42] PROBLEM - puppet last run on mw2068 is CRITICAL Puppet has 1 failures [16:17:42] PROBLEM - puppet last run on mw2071 is CRITICAL Puppet has 1 failures [16:17:43] PROBLEM - puppet last run on db2011 is CRITICAL Puppet has 1 failures [16:17:43] PROBLEM - puppet last run on mw2009 is CRITICAL Puppet has 1 failures [16:17:44] PROBLEM - puppet last run on mw2064 is CRITICAL Puppet has 1 failures [16:17:44] PROBLEM - puppet last run on db2030 is CRITICAL Puppet has 1 failures [16:17:50] PROBLEM - puppet last run on mw1122 is CRITICAL Puppet has 1 failures [16:17:50] PROBLEM - puppet last run on analytics1027 is CRITICAL Puppet has 1 failures [16:17:50] PROBLEM - puppet last run on ganeti2003 is CRITICAL Puppet has 1 failures [16:17:50] PROBLEM - puppet last run on ganeti2001 is CRITICAL Puppet has 1 failures [16:17:51] PROBLEM - puppet last run on ganeti2004 is CRITICAL Puppet has 1 failures [16:17:51] PROBLEM - puppet last run on rdb2003 is CRITICAL Puppet has 1 failures [16:17:51] PROBLEM - puppet last run on mw1142 is CRITICAL Puppet has 1 failures [16:17:52] PROBLEM - puppet last run on cp4020 is CRITICAL Puppet has 1 failures [16:17:52] PROBLEM - puppet last run on virt1008 is CRITICAL Puppet has 1 failures [16:18:00] PROBLEM - puppet last run on iodine is CRITICAL Puppet has 1 failures [16:18:00] PROBLEM - puppet last run on cp3039 is CRITICAL Puppet has 1 failures [16:18:00] PROBLEM - puppet last run on calcium is CRITICAL Puppet has 1 failures [16:18:01] PROBLEM - puppet last run on es1010 is CRITICAL Puppet has 1 failures [16:18:01] PROBLEM - puppet last run on cp1052 is CRITICAL Puppet has 1 failures [16:18:01] PROBLEM - puppet last run on mc1018 is CRITICAL Puppet has 1 failures [16:18:10] PROBLEM - puppet last run on labvirt1002 is CRITICAL Puppet has 1 failures [16:18:11] PROBLEM - puppet last run on mw1010 is CRITICAL Puppet has 1 failures [16:18:11] PROBLEM - puppet last run on iridium is CRITICAL Puppet has 1 failures [16:18:11] PROBLEM - puppet last run on analytics1003 is CRITICAL Puppet has 1 failures [16:18:11] PROBLEM - puppet last run on analytics1018 is CRITICAL Puppet has 1 failures [16:18:12] PROBLEM - puppet last run on restbase1001 is CRITICAL Puppet has 1 failures [16:18:15] (03CR) 10Jgreen: [C: 031] Remove pfw2-* loopback addresses, introduce pfw-* [dns] - 10https://gerrit.wikimedia.org/r/205279 (owner: 10Faidon Liambotis) [16:18:20] PROBLEM - puppet last run on mw1223 is CRITICAL Puppet has 1 failures [16:18:21] PROBLEM - puppet last run on labvirt1004 is CRITICAL Puppet has 1 failures [16:18:21] PROBLEM - puppet last run on wtp1010 is CRITICAL Puppet has 1 failures [16:18:22] oh damn... intel-microcode [16:18:23] (03PS1) 10Faidon Liambotis: Remove invalid "microcode" package [puppet] - 10https://gerrit.wikimedia.org/r/205282 [16:18:31] PROBLEM - puppet last run on db1045 is CRITICAL Puppet has 1 failures [16:18:31] PROBLEM - puppet last run on chromium is CRITICAL Puppet has 1 failures [16:18:31] PROBLEM - puppet last run on virt1011 is CRITICAL Puppet has 1 failures [16:18:31] PROBLEM - puppet last run on mw1209 is CRITICAL Puppet has 1 failures [16:18:31] PROBLEM - puppet last run on mw1016 is CRITICAL Puppet has 1 failures [16:18:32] PROBLEM - puppet last run on mw1231 is CRITICAL Puppet has 1 failures [16:18:33] <_joe_> akosiaris_: you reversed the meaning of the check [16:18:34] <_joe_> fixing [16:18:40] PROBLEM - puppet last run on ganeti1002 is CRITICAL Puppet has 1 failures [16:18:41] PROBLEM - puppet last run on dbstore2002 is CRITICAL Puppet has 1 failures [16:18:41] PROBLEM - puppet last run on mw1093 is CRITICAL Puppet has 1 failures [16:18:41] PROBLEM - puppet last run on db1072 is CRITICAL Puppet has 1 failures [16:18:46] _joe_: nope, that's not it [16:18:46] everyone's fixing, apparently [16:18:48] see above [16:18:51] PROBLEM - puppet last run on wtp2020 is CRITICAL Puppet has 1 failures [16:18:51] PROBLEM - puppet last run on mw2188 is CRITICAL Puppet has 1 failures [16:18:52] PROBLEM - puppet last run on cp4009 is CRITICAL Puppet has 1 failures [16:18:52] PROBLEM - puppet last run on mw2148 is CRITICAL Puppet has 1 failures [16:18:52] PROBLEM - puppet last run on mw2154 is CRITICAL Puppet has 1 failures [16:18:52] PROBLEM - puppet last run on mw2180 is CRITICAL Puppet has 1 failures [16:18:55] _joe_: no that is actually ok [16:19:00] PROBLEM - puppet last run on mw1139 is CRITICAL Puppet has 1 failures [16:19:00] PROBLEM - puppet last run on mw2185 is CRITICAL Puppet has 1 failures [16:19:00] PROBLEM - puppet last run on mw2157 is CRITICAL Puppet has 1 failures [16:19:00] PROBLEM - puppet last run on mw2141 is CRITICAL Puppet has 1 failures [16:19:00] PROBLEM - puppet last run on cp4012 is CRITICAL Puppet has 1 failures [16:19:01] PROBLEM - puppet last run on mw1152 is CRITICAL Puppet has 1 failures [16:19:01] PROBLEM - puppet last run on ms-be1015 is CRITICAL Puppet has 1 failures [16:19:02] PROBLEM - puppet last run on mw1219 is CRITICAL Puppet has 1 failures [16:19:02] PROBLEM - puppet last run on mw1091 is CRITICAL Puppet has 1 failures [16:19:03] PROBLEM - puppet last run on labvirt1006 is CRITICAL Puppet has 1 failures [16:19:11] PROBLEM - puppet last run on cp3047 is CRITICAL Puppet has 1 failures [16:19:11] PROBLEM - puppet last run on wtp1001 is CRITICAL Puppet has 1 failures [16:19:20] PROBLEM - puppet last run on mw2075 is CRITICAL Puppet has 1 failures [16:19:21] PROBLEM - puppet last run on achernar is CRITICAL Puppet has 1 failures [16:19:21] PROBLEM - puppet last run on strontium is CRITICAL Puppet has 1 failures [16:19:21] PROBLEM - puppet last run on baham is CRITICAL Puppet has 1 failures [16:19:21] (03PS2) 10Faidon Liambotis: Remove invalid "microcode" package [puppet] - 10https://gerrit.wikimedia.org/r/205282 [16:19:21] PROBLEM - puppet last run on mw2004 is CRITICAL Puppet has 1 failures [16:19:21] PROBLEM - puppet last run on ms-fe1002 is CRITICAL Puppet has 1 failures [16:19:21] <_joe_> yes sorry, so they weren't installed and it was an error [16:19:21] PROBLEM - puppet last run on mw1107 is CRITICAL Puppet has 1 failures [16:19:22] <_joe_> right [16:19:30] PROBLEM - puppet last run on mc2007 is CRITICAL Puppet has 1 failures [16:19:30] PROBLEM - puppet last run on mw2001 is CRITICAL Puppet has 1 failures [16:19:31] PROBLEM - puppet last run on mc1017 is CRITICAL Puppet has 1 failures [16:19:31] PROBLEM - puppet last run on mw2069 is CRITICAL Puppet has 1 failures [16:19:31] PROBLEM - puppet last run on mc2005 is CRITICAL Puppet has 1 failures [16:19:31] PROBLEM - puppet last run on mw2020 is CRITICAL Puppet has 1 failures [16:19:31] PROBLEM - puppet last run on mw2010 is CRITICAL Puppet has 1 failures [16:19:32] PROBLEM - puppet last run on mc2016 is CRITICAL Puppet has 1 failures [16:19:32] PROBLEM - puppet last run on mw1204 is CRITICAL Puppet has 1 failures [16:19:51] paravoid: just change the package name [16:19:59] to intel-microcode, do not remove it [16:19:59] (03CR) 10Faidon Liambotis: [C: 032] Remove invalid "microcode" package [puppet] - 10https://gerrit.wikimedia.org/r/205282 (owner: 10Faidon Liambotis) [16:20:12] (03CR) 10Faidon Liambotis: [V: 032] Remove invalid "microcode" package [puppet] - 10https://gerrit.wikimedia.org/r/205282 (owner: 10Faidon Liambotis) [16:20:15] akosiaris_: see the commit msg [16:20:15] it's that the old version of that code was "intel-microcode" not "microcode" [16:20:36] oh, so we had it already [16:20:41] also, I don't think it was moot [16:20:41] as intel-microcode [16:20:44] 6operations, 6Labs: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220936 (10Andrew) On labnet1001, we get dhcp requests for precise on labvirt, trusty on virt, but not for trusty on labvirt. Here's trusty on virt: Apr 20 16:13:08 labnet1001 dnsmasq-d... [16:20:47] we already did the fleet a while back [16:20:48] prtksxna: huh, everything looks to be on the latest version... bd808 is there ever any delay in localization updates post-scap? [16:20:58] it's just the refactor also renamed the package name [16:21:04] yeah just saw that [16:21:05] fixing [16:21:16] thcipriani: Hm. I've been refreshing at clearing my cache. [16:21:20] yes, I am to blame for this one :-( [16:21:28] still, I'll make it ensure => present [16:21:31] thcipriani: Do you have hovercards enabled by any chance? [16:21:38] ok [16:21:42] and I actually went through the lint:ignore stuff to insert a different bug :-( [16:21:43] ensuring latest CPU microcodes scares me a little tbh :) [16:21:48] prtksxna: yes [16:22:06] paravoid, afraid of being owned by intel? :P [16:22:12] paravoid: seems a little paranoid :) [16:22:13] thcipriani: What is the tooltip that you see when you hover over the megaphone icon on the card? [16:22:13] but sure [16:22:20] of someone pushing a broken CPU microcode [16:22:30] thcipriani: https://gerrit.wikimedia.org/r/#/c/205219/1/i18n/en.json [16:23:05] see https://lists.debian.org/debian-release/2015/02/msg00036.html [16:23:17] Intel botched a microcode update in the 20150107 release, currently in Debian jessie (testing). This broken microcode update causes core hangs and boot issues on boxes with Intel Xeon E5v3 processors (Debian bug #776431). [16:23:24] imagine this getting out to the whole fleet [16:23:25] ouch! [16:23:50] prtksxna: yeah, not updated for me either, definitely updated on tin though [16:24:09] it seems insane that could pass testing... [16:24:21] faidon@copper:~$ dpkg -l |grep intel- [16:24:21] faidon@copper:~$ [16:24:29] so I think it is moot [16:24:36] thcipriani: So, we wait? [16:24:46] same for e.g. cp3005 [16:25:00] I'm not convinced this Package was applied before [16:25:13] copper is a bad example [16:25:18] just reimaged [16:25:22] exactly :) [16:25:38] bblack: was meaning he has done it manually I believe [16:25:46] no, I don't think I did [16:25:53] prtksxna: lets take this to #wikimedia-releng real quick so we stay out of the way [16:25:55] but I thought I saw it working on many hosts back when I first merged that [16:26:01] maybe I didn't! looking... [16:26:37] (07:20:47 μμ) bblack: we already did the fleet a while back ??? [16:27:00] and I should change to 24H I know :-) [16:27:05] I meant, I thought the conditional worked originally, even if it broke for labs or whatever [16:27:15] I guess not! [16:27:21] I see it installed on a bunch of Ubuntu hosts, though [16:27:22] ah, OK [16:27:22] hmmmm [16:27:27] interesting [16:28:28] 372 hosts don't have it installed [16:28:32] ah yes, I figured it out now [16:28:44] I originally added both packages to standard-packages unconditionally [16:28:47] ah! [16:28:54] they were moved to the is_virtual thing later at paravoid's request, which didn't work at all [16:28:56] 639 hosts have it installed [16:28:58] that explains it :) [16:28:59] but it was already on all hosts from that time [16:29:11] just not on newer ones since [16:29:19] which is all of codfw [16:29:29] and probably nearly-all of the new caches [16:29:36] yup [16:29:40] (newly-reinstalled, I should say) [16:29:49] all, actually [16:29:50] 72f55d62627a4f177d084643f594355ce0b76645 is dated Jan 12 [16:29:54] so, yeah [16:31:02] so, what do we do? intel-microcode => installed ? [16:31:04] I still think this whole thing with how boolean facter facts translate to boolean conditionals in manifests is just abhorrent :P [16:31:18] bblack: you 're not alone in that group [16:31:19] bblack: it is, but they fixed it [16:31:29] paravoid: only if you also got puppetDB [16:31:37] which we dont [16:31:51] that matters only for exported resources [16:32:27] anyways, let's leave it how it is now (-microcode package install) for a bit then, and actually test installing it on some jessie hosts + reboots first? [16:32:57] doesn't need a reboot, the microcode gets applied immediately when you install the package [16:33:10] yeah but the bug you mentioned caused reboot issues [16:33:19] well that bug was fixed [16:33:22] :) [16:33:31] my point was that intel has shipped bad microcode before [16:33:34] 6operations, 10RESTBase, 10RESTBase-Cassandra: Set up multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1220979 (10fgiunchedi) >>! In T95253#1220690, @GWicke wrote: >> multiple instances per machine can also influence each other by e.g. slowing down or go down complete... [16:33:43] now you've made me scared of screwing the CPUs on every global cache at the same time, though [16:33:48] and I can imagine a situation where they do that, somehow it slips past the debian maintainer [16:33:50] your paranoia is infectious [16:33:57] then we get bad microcode to the whole fleet [16:34:20] and since that has happened before, I stand by my paranoia ;) [16:34:51] (03PS1) 10Faidon Liambotis: Re-add Package['intel-microcode'] [puppet] - 10https://gerrit.wikimedia.org/r/205289 [16:34:52] although to be fair, we are running an unrelease distro here [16:35:01] maybe it would've have passed review into a stable repo [16:35:07] *wouldn't have [16:35:10] microcodes get updated even in stable [16:35:19] yeah but that one didn't [16:35:27] sure, but it did reach testing ;) [16:36:01] which was the point anyway, wasn [16:36:06] wasn'it it ? [16:36:17] someone getting bitten by it [16:36:55] I didn't say that it will necessarily happen again, just that it /may/ happen and it'd be best if we didn't just roll updates across the fleet in one go [16:37:08] it's the standard ensure => latest argument, basically :) [16:37:29] sure [16:37:31] yup [16:37:42] (03CR) 10Alexandros Kosiaris: [C: 032] Re-add Package['intel-microcode'] [puppet] - 10https://gerrit.wikimedia.org/r/205289 (owner: 10Faidon Liambotis) [16:37:49] merging [16:38:36] anomie: for whatever reason it seems like extension localization has yet to take hold after a full scap, I can see the updates on live servers via ssh but I don't see the updates live on the site—is there another step here? [16:39:32] ok.. [16:39:37] thcipriani: Not that I know of [16:39:39] I just checked, we don't have that broken version anywhere [16:40:00] it'd be plausible, since we installed the package once then stopped updating on Jan 12nd [16:40:32] paravoid: User must install Facter ≥ 2.0 on all nodes, upgrade to PuppetDB ≥ 2.2, then setstringify_facts = false in puppet.conf on all nodes (details here) [16:40:40] https://docs.puppetlabs.com/puppet/latest/reference/lang_facts_and_builtin_vars.html [16:40:55] and the broken one was 20150107, fixed is 20150121 [16:40:57] what's with puppetDB 2.2 requirement ? [16:41:20] akosiaris_: it's for exporting (non-stringified) facts [16:41:50] stringify_facts = false (all nodes) — This enables full data type support for facts, allowing facts to contain arrays, hashes, and booleans instead of just strings. This requires Facter ≥ 2.0. [16:41:52] facter 2.0 has data types, a fact can return e.g. a hash [16:41:57] yup [16:42:10] (I know, I've written such a fact some time ago) [16:42:13] we should be past most of the icinga-wm spam now right? [16:42:27] well I guess maybe another few minutes for the slower-running hosts [16:43:07] bblack: not even close, look at the web interface... [16:43:21] 323 Matching Service Entries Displayed [16:43:22] I'm tailing the log now heh [16:45:25] anomie: so, not sure how to troubleshoot from here: /srv/mediawiki/php-1.26wmf1/extensions/Popups/i18n/en.json shows the correct string, ran scap, en.json is the same live as on tin, but the string is the same as before the update. For context the change was https://gerrit.wikimedia.org/r/#/c/205219/1/i18n/en.json and it's the megaphone icon in the corner of hovercards. [16:47:22] (03CR) 10Faidon Liambotis: [C: 032] Remove pfw2-* loopback addresses, introduce pfw-* [dns] - 10https://gerrit.wikimedia.org/r/205279 (owner: 10Faidon Liambotis) [16:47:27] (03CR) 10Faidon Liambotis: [C: 032] Assign subnet for pfw-eqiad<->pfw-codfw (tunnel) [dns] - 10https://gerrit.wikimedia.org/r/205280 (owner: 10Faidon Liambotis) [16:48:27] thcipriani: does the string show in something that is loaded by resourceloader (eg javascript ui) [16:48:48] bd808: Yes. [16:50:41] _joe_: what's with mw2128? [16:50:58] <_joe_> paravoid: ach it lost the acknowledgement, of course [16:51:06] <_joe_> it's being reinstalled [16:51:15] ok [16:51:28] <_joe_> and I think papaul managed to bring it back on friday night, and that removed the ack [16:51:39] papaul said it was reinstalled [16:51:43] ( https://phabricator.wikimedia.org/T95264 ) [16:51:59] <_joe_> he installed the OS [16:52:19] <_joe_> didn't run puppet nor removed old facts from puppet etc [16:52:55] oh... [16:54:10] 6operations, 10Graphoid, 6Services, 10service-template-node, 7service-runner: Deploy graphoid service into production - https://phabricator.wikimedia.org/T90487#1221052 (10mobrovac) [16:56:17] bd808: yes, but clearing cache/adding ?debug=true doesn't seem to correct the issue [16:56:17] andrewbogott: I am closing https://phabricator.wikimedia.org/T96291 [16:56:37] akosiaris_: yep, thanks. [16:58:08] 6operations: setup/deploy ganeti2001-2006 - https://phabricator.wikimedia.org/T94042#1221070 (10akosiaris) 5Open>3Resolved [16:58:16] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1221072 (10vshchepakina) @Dzahn I am doing the store announcement tomorrow and I was hoping to have a proper URL by now. Is there any chance it can be do... [17:01:27] bd808: also, fwiw, files in the l10n cache were updated for php-1.26wmf1/cache/l10n were updated when scap ran, both on tin and live [17:10:57] springle: db1019 disk space alert [17:11:39] 6operations, 6Labs: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1221112 (10Andrew) When I try to mount the drive of a failed instance on labvirt, I get: root@labvirt1001:/var/lib/nova/instances/7fc33e19-b002-4965-b7a4-886ea824fd9b# mount /dev/nbd0 /t... [17:11:40] (03CR) 10Dzahn: [C: 032] "springle: look, lots of db's" [puppet] - 10https://gerrit.wikimedia.org/r/204829 (owner: 10Papaul) [17:12:11] RECOVERY - puppet last run on mw2171 is OK Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:12:37] (03Restored) 10Mjbmr: Add autopatrolled for fawikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201977 (owner: 10Mjbmr) [17:13:03] (03Restored) 10Mjbmr: Add draft namespace for fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204467 (https://phabricator.wikimedia.org/T92760) (owner: 10Mjbmr) [17:13:18] (03Restored) 10Mjbmr: Rename project namespace for tewikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204464 (https://phabricator.wikimedia.org/T89332) (owner: 10Mjbmr) [17:14:41] (03Restored) 10Mjbmr: Add autopatrolled, patroller and rollbacker user groups for svwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204473 (https://phabricator.wikimedia.org/T93339) (owner: 10Mjbmr) [17:14:53] 6operations: Allow access to https://archiva.wikimedia.org from analytics nodes. - https://phabricator.wikimedia.org/T95712#1221133 (10akosiaris) 5Open>3Resolved Just poked a hole in the ACLs for the analytics VLAN. Curl is now working fine, resolving this [17:16:05] 6operations: Create apertium-admins group on sca1001/sca1002 - https://phabricator.wikimedia.org/T89222#1221140 (10akosiaris) 5Open>3stalled Until the comments above have been answered [17:16:20] 6operations: Create apertium-admins group on sca1001/sca1002 - https://phabricator.wikimedia.org/T89222#1221142 (10akosiaris) a:5akosiaris>3None [17:17:00] PROBLEM - Varnishkafka Delivery Errors per minute on cp4014 is CRITICAL 11.11% of data above the critical threshold [20000.0] [17:18:01] ^ I'm pretty sure that's because I just did an "apt-get -y upgrade there", which use a lot of network bandwidth and I guess that mucks with kafka traffic? [17:18:14] s/ there"/" there/ [17:19:29] (either that or one of the package updates hurt varnishkafka or one of the many other dependent bits in that data-delivery chain) [17:20:30] RECOVERY - Varnishkafka Delivery Errors per minute on cp4014 is OK Less than 1.00% above the threshold [0.0] [17:21:12] can I get a moratorium on +2 merges to ops/puppet for a few minutes? [17:21:33] (03PS12) 10BBlack: get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 [17:21:35] (03PS17) 10BBlack: Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 [17:21:35] for merging up all of this together: [17:21:37] (03PS12) 10BBlack: hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 [17:21:39] (03PS12) 10BBlack: creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 [17:21:41] (03PS11) 10BBlack: remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 [17:21:43] (03PS2) 10BBlack: re-arrange to align classes a bit... [puppet] - 10https://gerrit.wikimedia.org/r/205273 [17:21:45] (03PS8) 10BBlack: normalize on $cluster_tier as string words [puppet] - 10https://gerrit.wikimedia.org/r/204773 [17:21:47] (03PS10) 10BBlack: hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 [17:21:49] (03PS2) 10BBlack: backport ceiling() func from puppetlabs-stdlib v4.6.0 [puppet] - 10https://gerrit.wikimedia.org/r/205272 [17:21:51] (03PS10) 10BBlack: drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 [17:21:53] (03PS19) 10BBlack: Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 [17:22:31] 6operations, 10Datasets-General-or-Unknown: Dumps (or dump progress page) are stuck since Apr 19 Apr 07:24 UTC - https://phabricator.wikimedia.org/T96550#1221171 (10Ankry) It seems that dumps are online again; so forget this report [17:22:32] (03CR) 10BBlack: [C: 032] backport ceiling() func from puppetlabs-stdlib v4.6.0 [puppet] - 10https://gerrit.wikimedia.org/r/205272 (owner: 10BBlack) [17:23:25] (03CR) 10BBlack: [V: 032] backport ceiling() func from puppetlabs-stdlib v4.6.0 [puppet] - 10https://gerrit.wikimedia.org/r/205272 (owner: 10BBlack) [17:25:09] (03CR) 10BBlack: [C: 032] Switch inheritance to include; refactor storage conf [puppet] - 10https://gerrit.wikimedia.org/r/204514 (owner: 10BBlack) [17:25:22] (03CR) 10BBlack: [C: 032] creatively kill $realm storage conditionals [puppet] - 10https://gerrit.wikimedia.org/r/204552 (owner: 10BBlack) [17:25:34] (03CR) 10BBlack: [C: 032] normalize on $cluster_tier as string words [puppet] - 10https://gerrit.wikimedia.org/r/204773 (owner: 10BBlack) [17:25:39] !log remove wmf PHP5 for lucid from apt.wikimedia.org [17:25:44] Logged the message, Master [17:26:18] (03CR) 10BBlack: [C: 032] Apply class-param + hiera pattern to base+2layer [puppet] - 10https://gerrit.wikimedia.org/r/204545 (owner: 10BBlack) [17:26:34] (03CR) 10BBlack: [C: 032] get rid of realm conditionals on $memory_storage_size [puppet] - 10https://gerrit.wikimedia.org/r/204755 (owner: 10BBlack) [17:26:50] (03CR) 10BBlack: [C: 032] hieraize bits/upload domainname options for labs [puppet] - 10https://gerrit.wikimedia.org/r/204756 (owner: 10BBlack) [17:27:03] (03CR) 10BBlack: [C: 032] remove realm conditional from parsoid lvs::realserver [puppet] - 10https://gerrit.wikimedia.org/r/204757 (owner: 10BBlack) [17:27:21] (03CR) 10BBlack: [C: 032] hieraize mobile $zero_site [puppet] - 10https://gerrit.wikimedia.org/r/204759 (owner: 10BBlack) [17:27:34] (03CR) 10BBlack: [C: 032] drop ESI from labs-mobile [puppet] - 10https://gerrit.wikimedia.org/r/204760 (owner: 10BBlack) [17:27:38] (03Restored) 10Mjbmr: Add alias namespace for previous project namespace fawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/195017 (owner: 10Mjbmr) [17:28:20] done merging! [17:29:54] lucid, heh [17:33:59] 6operations: Choose a consistent, distributed k/v storage for configuration management/discovery - https://phabricator.wikimedia.org/T95656#1221186 (10Joe) @marko I think all what you state here is something that will be enabled by this software, once integrated with our tools (salt, puppet, pybal, etc.) [17:34:06] 6operations, 10RESTBase, 10RESTBase-Cassandra: Set up multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1221187 (10Eevans) > I took a look at running multiple cassandra hosts on the same host, it looks like we'd need a different ip address for each instance since a nod... [17:36:12] 6operations, 10RESTBase, 10RESTBase-Cassandra: Set up multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1221194 (10GWicke) >>! In T95253#1220979, @fgiunchedi wrote: >> Regarding isolation: A single instance is not very good at isolating requests for parts of the keyspa... [17:37:41] (03CR) 10Dzahn: [C: 031] "didn't check the actual numbers but worth it i'd say" [puppet] - 10https://gerrit.wikimedia.org/r/204616 (owner: 10Alex Monk) [17:45:56] !opshelp is https://wikitech.wikimedia.org/wiki/Operations_requests [17:45:57] Key was added [17:46:22] !opshelp [17:46:23] https://wikitech.wikimedia.org/wiki/Operations_requests [17:46:25] nice. [17:46:51] Please note all questions that are addressed on that page will result in one word answers from the ops clinic person (meee ;) [17:50:01] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1221261 (10greg) {F100025} [17:50:51] (03PS3) 10Rush: phabricator: chad and mukunda as root [puppet] - 10https://gerrit.wikimedia.org/r/204862 [17:51:12] 6operations: Create apertium-admins group on sca1001/sca1002 - https://phabricator.wikimedia.org/T89222#1221272 (10GWicke) 5stalled>3Open @kartikmistry, https://github.com/wikimedia/service-runner provides a logger instance which by default logs to logstash. It might be worth adopting service-runner in gener... [17:51:20] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1221274 (10chasemp) https://gerrit.wikimedia.org/r/#/c/204862/1 [17:51:41] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1221281 (10RobH) This is now on the ops agenda for approval during our meeting. [17:51:53] (03CR) 10BBlack: [C: 032] "Validated no-op in compiler" [puppet] - 10https://gerrit.wikimedia.org/r/205273 (owner: 10BBlack) [17:52:17] (03CR) 10RobH: [C: 04-2] "My -2 is only a blocker until review of this request during our operations meeting today. I'll remove post-meeting, pending approval." [puppet] - 10https://gerrit.wikimedia.org/r/204862 (owner: 10Rush) [18:01:29] 6operations: Multiple PHP security issues - https://phabricator.wikimedia.org/T96586#1221385 (10MoritzMuehlenhoff) [18:02:20] RoanKattouw: having a little trouble with RL message caching post scap. After swat-ing https://gerrit.wikimedia.org/r/#/c/205219/ the change doesn't show up live but does show up in https://en.wikipedia.org/wiki/MediaWiki:Popups-send-feedback can you shed any light on next steps? [18:04:26] Ugh RL messages [18:04:28] I hate that system [18:04:32] But I wrote it so I don't get to complain [18:04:50] thcipriani: You can try running mwscript extensions/WikimediaMaintenance/clearMessageBlobs.php enwiki [18:05:15] kk, will try, thank you! [18:05:20] Which will take forever (I don't know why it's gotten so slow) and probably die with a lock error eventually :S [18:05:47] RoanKattouw: oh good. [18:06:02] thcipriani: If there's a specific wiki you have in mind (enwiki it looks like) you can also manually truncate the msg_resource and msg_resource_links (sp?) tables on that wiki [18:06:16] Like, sql enwiki, TRUNCATE msg_resource; [18:06:30] The script is supposed to run that on every wiki [18:06:43] At least that's what it did when I first wrote it, God knows what it does know because it's slow as hell [18:06:45] (patches welcome :D ) [18:09:12] alright, I'm going to try truncate msg_resource on enwiki [18:09:15] OK cool [18:09:25] Again, I apologize for this meess [18:09:38] There's a task somewhere about how we need to move msg_resource out of the DB [18:13:12] godog: yay for recreating. that seems to have fixed some (http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1429553580.331&target=tools.tools-services-02.WebServiceMonitor.manifestscollected.mean has only whole numbers!) [18:13:12] !log truncate msg_resource on enwiki to refresh RL messages, seems to have fixed the issue [18:13:17] Logged the message, Master [18:13:39] godog: but! [18:13:52] godog: http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1429553612.563&target=tools.tools-services-02.WebServiceMonitor.startsuccess.sum is a ‘sum’ and has fractions when I never send more than integers to it [18:16:09] RoanKattouw: thanks for the assist, issue seems to be fixed. [18:16:43] hmm, ops. puppet is based on ruby, right? what version of ruby is it using in production? [18:17:03] ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux] [18:17:05] MatmaRex: ^ [18:17:09] thanks [18:17:31] (i'm trying to guess what causes https://phabricator.wikimedia.org/T548 ) [18:18:13] (03CR) 10Faidon Liambotis: [C: 04-1] "restbase-roots?" [puppet] - 10https://gerrit.wikimedia.org/r/204862 (owner: 10Rush) [18:25:09] MatmaRex: modules/phabricator/manifests/init.pp: $fixed_settings = loadyaml("${module_path}/data/fixed_settings.yaml") [18:25:44] MatmaRex: I think that + merging is what’s fucking that up [18:26:16] Might look at https://github.com/wikimedia/operations-puppet/blob/production/modules/wmflib/lib/puppet/parser/functions/ordered_json.rb and https://github.com/wikimedia/operations-puppet/blob/production/modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb to see if they can help [18:27:51] (03PS4) 10Rush: phabricator: chad and mukunda as root [puppet] - 10https://gerrit.wikimedia.org/r/204862 [18:28:01] PROBLEM - puppet last run on mw2164 is CRITICAL Puppet has 1 failures [18:28:02] (03CR) 10Rush: "faidon, I thought I changed this already but it was sitting un pushed to gerrit thanks :)" [puppet] - 10https://gerrit.wikimedia.org/r/204862 (owner: 10Rush) [18:28:08] (03PS5) 10Rush: phabricator: chad and mukunda as root [puppet] - 10https://gerrit.wikimedia.org/r/204862 [18:29:12] 6operations, 7Graphite: revisit what percentiles are calculated by txstatsd - https://phabricator.wikimedia.org/T88662#1221532 (10faidon) We're off txstatsd, so that ticket should be adjusted now, no? Do we have median & 95p/98p with statsite now? These are absolutely needed, IMHO. [18:34:00] (03PS13) 10BBlack: Set up /api/v1/ entry point for restbase [puppet] - 10https://gerrit.wikimedia.org/r/203871 (https://phabricator.wikimedia.org/T95229) (owner: 10GWicke) [18:40:24] YuviPanda: bd808: i think it's the ordered_json, actually [18:40:30] in /modules/phabricator/templates/local.json.erb [18:40:51] that seems to be a custom function in our puppet [18:40:56] "Serialize a hash into JSON with lexicographically sorted keys." [18:41:31] documented as a workaround for Ruby 1.8, and you're saying we're running 1.9 now [18:43:10] MatmaRex: we are indeed on ruby 1.9 there, but I don't know for sure if that resolves things [18:43:24] as I haven't tested it, in theory that function is not needed now and it's all testing to confirm? [18:43:31] basically someone needs to do the leg work [18:44:20] chasemp: i haven't worked with this for a while, but IIRC starting with ruby 1.9 iterating over a hash (so also serializing it to json) will iterate over the keys in insertion order [18:44:31] RECOVERY - puppet last run on mw2164 is OK Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:44:40] previously iteration order was undefined [18:45:33] chasemp: so yeah, i think we can replace `scope.function_ordered_json([@phab_settings])` with `[@phab_settings].to_pson` in /modules/phabricator/templates/local.json.erb ? or whatever the mangled puppet syntax is [18:45:56] very possibly :) but someone needs to validate and put up the change and I won't have time to do it [18:45:57] krenair@silver:~$ mwscript eval.php labswiki [18:45:58] mkdir: cannot create directory �/sys/fs/cgroup/memory/mediawiki/job/2546�: No such file or directory [18:45:58] limit.sh: failed to create the cgroup. [18:45:59] urgh, it's back [18:46:07] if you want to get onto phab-01 seems good to me [18:46:31] i can submit the proposed patch, but i'm fairly sure i don't have permissions/ability to deploy it anywhere [18:46:53] 6operations: Investigate why cgroup on silver was only mounted with 4k of space - https://phabricator.wikimedia.org/T92712#1221714 (10Krenair) Looks like it's back. [18:48:29] YuviPanda: can you add MatmaRex to the project phab-01 is in? [18:49:07] yeah [18:49:08] moment [18:49:33] MatmaRex: what’s your wikitech username? [18:50:19] MatmaRex: added you [18:50:32] :o [18:50:45] chasemp: ^ done [18:52:34] (03PS1) 10Bartosz Dziewoński: Do no order Phabricator settings JSON lexicographically [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) [18:52:53] (03CR) 10Bartosz Dziewoński: "Not tested." [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) (owner: 10Bartosz Dziewoński) [18:53:50] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1221739 (10RobH) The meeting result is approved, but with some conditionals: As these guys are taking over the maintaining of... [18:54:09] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1221740 (10RobH) ie: no root until they also get paged, so dont merge until both are ready. [18:54:47] YuviPanda: thanks, but i still have no idea how to deploy it [18:55:02] :D [18:56:49] 6operations: Choose a consistent, distributed k/v storage for configuration management/discovery - https://phabricator.wikimedia.org/T95656#1221751 (10mobrovac) >>! In T95656#1221186, @Joe wrote: > @marko I think all what you state here is something that will be enabled by this software, once integrated with our... [19:04:32] 6operations, 6Labs, 7network: Slow network between virt nodes - https://phabricator.wikimedia.org/T96561#1221815 (10Andrew) 5Open>3Invalid a:3Andrew Two things were happening here -- half the problem was I forgot the difference between Mb and MB, which makes the picture much better than I thought. 56M... [19:10:37] 6operations, 10Datasets-General-or-Unknown, 10Wikidata, 3§ Wikidata-Sprint-2015-03-24, 3Wikidata-Sprint-2015-04-07: Wikidata dumps contain old-style serialization. - https://phabricator.wikimedia.org/T74348#1221882 (10daniel) a:5ArielGlenn>3hoo Assigning to hoo, who said he'd look into this some more. [19:11:10] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 35.71% of data above the critical threshold [500.0] [19:27:26] 6operations, 6Labs: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1222008 (10Andrew) 5Open>3Resolved a:3Andrew This was a result of corrupt files in the base-image cache in /var/lib/nova/instances/_base. I've cleaned out the base images for trust... [19:27:31] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [19:31:53] (03PS1) 10Andrew Bogott: Switch nova scheduler pool to use new labvirt* nodes [puppet] - 10https://gerrit.wikimedia.org/r/205349 [19:33:29] (03CR) 10Andrew Bogott: [C: 032] Switch nova scheduler pool to use new labvirt* nodes [puppet] - 10https://gerrit.wikimedia.org/r/205349 (owner: 10Andrew Bogott) [19:41:06] (03PS1) 10Mobrovac: WIP: Graphoid: Puppet bits [puppet] - 10https://gerrit.wikimedia.org/r/205350 (https://phabricator.wikimedia.org/T90487) [19:43:03] mobrovac: config => template [19:43:10] you accidentally [19:43:29] ori: euh? [19:43:30] (03CR) 10Ori.livneh: WIP: Graphoid: Puppet bits (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/205350 (https://phabricator.wikimedia.org/T90487) (owner: 10Mobrovac) [19:43:38] (03CR) 10jenkins-bot: [V: 04-1] WIP: Graphoid: Puppet bits [puppet] - 10https://gerrit.wikimedia.org/r/205350 (https://phabricator.wikimedia.org/T90487) (owner: 10Mobrovac) [19:43:41] https://gerrit.wikimedia.org/r/#/c/205350/1/modules/graphoid/manifests/init.pp line 27 [19:43:53] ah damn, forgot it [19:43:55] should be content => template('graphoid/config.yaml.erb') presumably [19:43:55] ori: thnx [19:43:58] np [19:44:24] PROBLEM - puppet last run on cp3015 is CRITICAL puppet fail [19:47:17] (03PS2) 10Mobrovac: WIP: Graphoid: Puppet bits [puppet] - 10https://gerrit.wikimedia.org/r/205350 (https://phabricator.wikimedia.org/T90487) [19:47:59] (03CR) 10Mobrovac: WIP: Graphoid: Puppet bits (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/205350 (https://phabricator.wikimedia.org/T90487) (owner: 10Mobrovac) [19:48:19] (03CR) 10jenkins-bot: [V: 04-1] WIP: Graphoid: Puppet bits [puppet] - 10https://gerrit.wikimedia.org/r/205350 (https://phabricator.wikimedia.org/T90487) (owner: 10Mobrovac) [19:48:40] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [19:49:26] (03PS2) 10Bartosz Dziewoński: Do not order Phabricator settings JSON lexicographically [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) [19:50:32] (03PS3) 10Mobrovac: WIP: Graphoid: Puppet bits [puppet] - 10https://gerrit.wikimedia.org/r/205350 (https://phabricator.wikimedia.org/T90487) [19:54:12] (03PS1) 10Andrew Bogott: Add virt-what to base images. [puppet] - 10https://gerrit.wikimedia.org/r/205357 [19:57:10] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1222180 (10mmodell) I don't have a problem with getting paged really. How does that work? I just give you guys a phone number?... [20:00:05] gwicke, cscott, arlolra, subbu: Dear anthropoid, the time has come. Please deploy Services – Parsoid / OCG / Citoid / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150420T2000). [20:02:03] (03PS2) 10Hashar: (WIP) ferm: lame tests for service template (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/205274 [20:02:30] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1222203 (10greg) >>! In T96425#1222180, @mmodell wrote: > Is there an alternative to SMS because I'm a lot more likely to wake... [20:03:51] RECOVERY - puppet last run on cp3015 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [20:04:06] jouncebot: okay! [20:04:08] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1222208 (10mmodell) >>! In T96425#1222203, @greg wrote: > Way to wade right into a religious war within Ops, there ;) I seem... [20:04:14] i'm going to deploy parsoid! [20:04:35] doesn't look like there are any operations fires at the moment [20:04:50] (03PS1) 10Alexandros Kosiaris: Add quilt to package_builder packages [puppet] - 10https://gerrit.wikimedia.org/r/205359 [20:05:56] (03CR) 10Andrew Bogott: [C: 032] Add virt-what to base images. [puppet] - 10https://gerrit.wikimedia.org/r/205357 (owner: 10Andrew Bogott) [20:08:10] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [20:11:24] (03CR) 10Alexandros Kosiaris: "Actually, we still run ruby 1.8 in production. That is puppetmaster production and it is still on precise and hence ruby 1.8 where hashes " [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) (owner: 10Bartosz Dziewoński) [20:12:02] ;_; [20:13:16] (03CR) 10Bartosz Dziewoński: ";_; Ruby 1.8 is like, PHP-4-old. Do you have any plans about upgrading?" [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) (owner: 10Bartosz Dziewoński) [20:18:11] !log updated Parsoid to version 0cabb5b2 [20:18:17] Logged the message, Master [20:18:38] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Heh, PHP 5.3 as well ;-). Yes we do have plans for upgrading. Not only that, there is also a thought out plan as well. There are a number " [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) (owner: 10Bartosz Dziewoński) [20:20:19] (03CR) 10Alexandros Kosiaris: [C: 032] Update package_builder/README.md [puppet] - 10https://gerrit.wikimedia.org/r/204776 (owner: 10Alexandros Kosiaris) [20:20:31] (03CR) 10Alexandros Kosiaris: [C: 032] Add quilt to package_builder packages [puppet] - 10https://gerrit.wikimedia.org/r/205359 (owner: 10Alexandros Kosiaris) [20:20:36] (03CR) 10Bartosz Dziewoński: [C: 04-1] "Yeah, if we're still on 1.8, this will likely not work as intended anyway. The bug is still a very annoying (yet tiny) one, I'll try writi" [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) (owner: 10Bartosz Dziewoński) [20:20:46] (03CR) 10Alexandros Kosiaris: [C: 032] Improve package_builder tests [puppet] - 10https://gerrit.wikimedia.org/r/204821 (owner: 10Alexandros Kosiaris) [20:20:54] (03PS2) 10Alexandros Kosiaris: Improve package_builder tests [puppet] - 10https://gerrit.wikimedia.org/r/204821 [20:21:01] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Improve package_builder tests [puppet] - 10https://gerrit.wikimedia.org/r/204821 (owner: 10Alexandros Kosiaris) [20:21:27] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1222264 (10Dzahn) @vshchepakina yes, it is on the deployment calendar for today now. in a 14:00–15:00 PDT slot. https://wikitech.wikimedia.org/wiki/Dep... [20:22:00] 6operations, 10Continuous-Integration, 5Continuous-Integration-Isolation: Provide Debian package python-pymysql for jessie-wikimedia - https://phabricator.wikimedia.org/T96131#1222265 (10hashar) And it works :) ``` labnodepool1001:~$ apt-cache policy python-pymysql python-pymysql: Installed: 0.6.2-1 Cand... [20:23:13] MatmaRex: I agree with you it is a bug on phab btw [20:26:59] (03CR) 10John F. Lewis: [C: 031] shop/store: switch redirects to store.wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/204559 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [20:28:29] (03CR) 10John F. Lewis: [C: 031] remove dropped shop/store redirects [puppet] - 10https://gerrit.wikimedia.org/r/204558 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [20:33:23] (03CR) 10RobH: [C: 031] shop/store: switch redirects to store.wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/204559 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [20:33:50] (03CR) 10RobH: [C: 031] shop/store: switch old URL to cluster for redirect [dns] - 10https://gerrit.wikimedia.org/r/204810 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [20:33:55] (03CR) 10Gage: [C: 031] Change BZ references to Phabricator tickets [puppet] - 10https://gerrit.wikimedia.org/r/204626 (owner: 10Alex Monk) [20:34:08] (03PS2) 10Hashar: contint: allow gearman/zeromq from labnodepool1001 [puppet] - 10https://gerrit.wikimedia.org/r/204876 (https://phabricator.wikimedia.org/T96426) [20:35:19] !log upload php5_5.3.10-1ubuntu3.18+wmf1 on precise-wikimedia distribution precise-wikimedia [20:35:23] _joe_: paravoid ^ [20:35:24] Logged the message, Master [20:35:36] moritzm: ^ [20:36:43] (03CR) 10Dzahn: [C: 031] phabricator: chad and mukunda as root [puppet] - 10https://gerrit.wikimedia.org/r/204862 (owner: 10Rush) [20:36:57] (03CR) 10Hashar: "Daniel proposed to use hiera instead of networks.pp for the Zuul mergers at https://gerrit.wikimedia.org/r/#/c/201882/ so this patch does" [puppet] - 10https://gerrit.wikimedia.org/r/204876 (https://phabricator.wikimedia.org/T96426) (owner: 10Hashar) [20:37:10] (03CR) 10RobH: [C: 032] "this was approved in meeting, merging live" [puppet] - 10https://gerrit.wikimedia.org/r/204862 (owner: 10Rush) [20:37:54] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1222296 (10chasemp) [20:37:58] (03PS1) 10Yuvipanda: tools: Make active proxy name available everywhere [puppet] - 10https://gerrit.wikimedia.org/r/205434 [20:38:07] (03PS2) 10Yuvipanda: tools: Make active proxy name available everywhere [puppet] - 10https://gerrit.wikimedia.org/r/205434 [20:38:15] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Make active proxy name available everywhere [puppet] - 10https://gerrit.wikimedia.org/r/205434 (owner: 10Yuvipanda) [20:38:19] twentyafterfour: i just pushed live you and chad's updated/expanded rights for the phab box [20:38:22] fyi [20:39:05] now we need the oreilly meme in phab [20:39:19] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1222307 (10RobH) 5Open>3Resolved This was discussed and approved in our operations meeting. Since these two now will be s... [20:39:20] (03CR) 10Gage: [C: 031] Change RT references to Phabricator tickets [puppet] - 10https://gerrit.wikimedia.org/r/204616 (owner: 10Alex Monk) [20:41:43] (03PS1) 10Yuvipanda: tools: Set tools.wmflabs.org to resolve internally to -webproxy-01 [puppet] - 10https://gerrit.wikimedia.org/r/205436 [20:41:57] (03PS2) 10Yuvipanda: tools: Set tools.wmflabs.org to resolve internally to -webproxy-01 [puppet] - 10https://gerrit.wikimedia.org/r/205436 [20:42:04] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Set tools.wmflabs.org to resolve internally to -webproxy-01 [puppet] - 10https://gerrit.wikimedia.org/r/205436 (owner: 10Yuvipanda) [20:47:02] (03PS1) 10Alexandros Kosiaris: include base::firewall in role::package::builder [puppet] - 10https://gerrit.wikimedia.org/r/205437 [20:47:08] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1222328 (10RobH) T96613 handles paging stuff [20:47:26] 6operations: Add Chad & Mukunda for phabricator paging - https://phabricator.wikimedia.org/T96613#1222310 (10RobH) [20:48:08] twentyafterfour: so you are chatting with daniel [20:48:13] you should totally talk to me since im doing it right now [20:48:14] ;D [20:48:29] https://phabricator.wikimedia.org/T96425 [20:48:33] arggh [20:48:34] wrong link [20:48:40] https://phabricator.wikimedia.org/T96613 [20:48:52] (03CR) 10Alexandros Kosiaris: [C: 032] include base::firewall in role::package::builder [puppet] - 10https://gerrit.wikimedia.org/r/205437 (owner: 10Alexandros Kosiaris) [20:50:52] (03CR) 10Andrew Bogott: [C: 031] contint: allow gearman/zeromq from labnodepool1001 [puppet] - 10https://gerrit.wikimedia.org/r/204876 (https://phabricator.wikimedia.org/T96426) (owner: 10Hashar) [20:53:29] twentyafterfour: need to know what timezone you are in as well [20:53:56] robh: central, but he keeps non-central hours :) [20:53:58] robh: us central [20:54:45] 10ops-fundraising, 10Fundraising-Backlog: Need Civi access for Donor Services agent - https://phabricator.wikimedia.org/T95011#1222349 (10Jgreen) >>! In T95011#1217495, @CCogdill_WMF wrote: > Hey @Jgreen, was Kristie included in the list of people who you send the cert update to earlier this week? If you could... [20:56:15] I'm around roughly 07:00 to 19:00 utc [20:57:42] (03PS2) 10Dzahn: Change RT references to Phabricator tickets [puppet] - 10https://gerrit.wikimedia.org/r/204616 (owner: 10Alex Monk) [20:59:11] (03CR) 10Dzahn: [C: 032] Change RT references to Phabricator tickets [puppet] - 10https://gerrit.wikimedia.org/r/204616 (owner: 10Alex Monk) [20:59:46] (03PS1) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [21:00:04] mutante, robh: Respected human, time to deploy Apache Redirects Update (apache restart/reload) for shop/store (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150420T2100). Please do the needful. [21:00:10] (03CR) 10jenkins-bot: [V: 04-1] Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (owner: 10Yuvipanda) [21:01:03] (03PS2) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [21:02:14] !log disabling puppet on mw servers for deployment [21:02:18] Logged the message, Master [21:04:13] (03CR) 10RobH: [C: 031] remove dropped shop/store redirects [puppet] - 10https://gerrit.wikimedia.org/r/204558 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [21:05:03] (03PS2) 10Dzahn: remove dropped shop/store redirects [puppet] - 10https://gerrit.wikimedia.org/r/204558 (https://phabricator.wikimedia.org/T92438) [21:06:32] (03CR) 10Dzahn: [C: 032] remove dropped shop/store redirects [puppet] - 10https://gerrit.wikimedia.org/r/204558 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [21:06:50] (03PS4) 10Dzahn: shop/store: switch redirects to store.wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/204559 (https://phabricator.wikimedia.org/T92438) [21:07:40] (03CR) 10Dzahn: [C: 032] shop/store: switch redirects to store.wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/204559 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [21:08:36] (03PS3) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [21:11:14] (03PS4) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [21:12:34] (03PS2) 10Alex Monk: Change BZ references to Phabricator tickets [puppet] - 10https://gerrit.wikimedia.org/r/204626 [21:13:54] (03CR) 10Alex Monk: "Only file I've left out is modules/toollabs/manifests/exec_environ.pp" [puppet] - 10https://gerrit.wikimedia.org/r/204626 (owner: 10Alex Monk) [21:19:23] 6operations, 10Analytics-Cluster: Set up ops kafkatee instance as part of udp2log transition - https://phabricator.wikimedia.org/T96616#1222404 (10Ottomata) 3NEW a:3Ottomata [21:23:37] (03PS1) 10RobH: adding chad and mukunda to contactgroups for phabricator paging [puppet] - 10https://gerrit.wikimedia.org/r/205441 [21:25:23] !log re-enabling puppet on mw servers for Apache change [21:25:30] Logged the message, Master [21:29:01] !log tagged puppet run on appservers , --tags mw-apache-config [21:29:05] Logged the message, Master [21:31:46] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1222464 (10Dzahn) ``` [tin:~] $ apache-fast-test T92438 mw1022 testing 7 urls on 1 servers, totalling 7 requests spawning threads.. http://shop.wikimed... [21:34:51] (03PS1) 10RobH: adding phabricator paging group [puppet] - 10https://gerrit.wikimedia.org/r/205442 [21:35:00] (03CR) 10RobH: [C: 032] adding chad and mukunda to contactgroups for phabricator paging [puppet] - 10https://gerrit.wikimedia.org/r/205441 (owner: 10RobH) [21:35:55] (03CR) 10RobH: [C: 032] adding phabricator paging group [puppet] - 10https://gerrit.wikimedia.org/r/205442 (owner: 10RobH) [21:36:22] (03PS3) 10Dzahn: LVS: add text and bits for codfw [puppet] - 10https://gerrit.wikimedia.org/r/196036 (https://phabricator.wikimedia.org/T92377) [21:36:38] (03CR) 10Gage: [C: 031] adding phabricator paging group [puppet] - 10https://gerrit.wikimedia.org/r/205442 (owner: 10RobH) [21:38:01] (03PS14) 10BBlack: Set up /api/v1/ entry point for restbase [puppet] - 10https://gerrit.wikimedia.org/r/203871 (https://phabricator.wikimedia.org/T95229) (owner: 10GWicke) [21:38:50] (03CR) 10BBlack: [C: 031] "PS13/14 are just rebases onto other recent varnish work. I think this looks ok on a technical level at this point..." [puppet] - 10https://gerrit.wikimedia.org/r/203871 (https://phabricator.wikimedia.org/T95229) (owner: 10GWicke) [21:41:37] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [21:42:11] mutante: ^ Your favorite [21:43:03] hoo: :p restarting [21:43:36] dont even login .. ssh antimony -C .. gitblit restart .. grbml [21:46:27] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60621 bytes in 0.237 second response time [21:47:19] (03PS2) 10Dzahn: LVS: add api,apaches and rendering for codfw [puppet] - 10https://gerrit.wikimedia.org/r/196067 (https://phabricator.wikimedia.org/T92377) [21:47:58] (03PS3) 10Dzahn: LVS: add api and apaches for codfw [puppet] - 10https://gerrit.wikimedia.org/r/196067 (https://phabricator.wikimedia.org/T92377) [21:52:00] (03CR) 10Dzahn: [C: 031] "asked Ryan, it's fine either way, go ahead" [puppet] - 10https://gerrit.wikimedia.org/r/202759 (owner: 10Andrew Bogott) [21:53:06] PROBLEM - Check correctness of the icinga configuration on neon is CRITICAL: Icinga configuration contains errors, please check! [21:53:41] 6operations, 10Wikimedia-Apache-configuration, 5Patch-For-Review: wikibooks.org redirects to en.wikibooks.org - https://phabricator.wikimedia.org/T87039#1222588 (10Dzahn) when i just deployed unrelated changes for the shop redirect, this change showed up as well, as merged but undeployed. so it's kind of con... [21:54:15] (03CR) 10MaxSem: [C: 031] Set up /api/v1/ entry point for restbase [puppet] - 10https://gerrit.wikimedia.org/r/203871 (https://phabricator.wikimedia.org/T95229) (owner: 10GWicke) [21:55:27] PROBLEM - puppet last run on mw2204 is CRITICAL puppet fail [21:55:51] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1222608 (10RobH) [21:56:53] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1222616 (10RobH) This was discussed during the operations meeting. The resulting discussion (ops & Marko) is that Marko will detail what they need to do with shell access (as in, exactl... [21:57:16] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1222623 (10RobH) a:5ArielGlenn>3None [21:57:53] 7Blocked-on-Operations, 10Ops-Access-Requests, 6operations: Access to francium - https://phabricator.wikimedia.org/T94093#1155079 (10RobH) a:3mobrovac [21:58:45] 6operations, 10Datasets-General-or-Unknown: Dumps (or dump progress page) are stuck since Apr 19 Apr 07:24 UTC - https://phabricator.wikimedia.org/T96550#1222631 (10Aklapper) 5Open>3declined a:3Aklapper [21:59:49] (03CR) 10Dzahn: [C: 032] shop/store: switch old URL to cluster for redirect [dns] - 10https://gerrit.wikimedia.org/r/204810 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [22:00:11] (03CR) 10Dzahn: "tested by hacking my /etc/hosts, apache config was deployed" [dns] - 10https://gerrit.wikimedia.org/r/204810 (https://phabricator.wikimedia.org/T92438) (owner: 10Dzahn) [22:00:58] (03PS1) 10Ottomata: Puppetize impala [puppet/cdh] - 10https://gerrit.wikimedia.org/r/205446 [22:04:07] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1222649 (10Dzahn) ``` curl -vvv http://shop.wikimedia.org

Moved Permanently

The document has moved he... [22:11:07] 10ops-fundraising, 10Fundraising-Backlog: Need Civi access for Donor Services agent - https://phabricator.wikimedia.org/T95011#1222674 (10CCogdill_WMF) Done! I need to confirm the number is her cell, as I'm not sure. https://office.wikimedia.org/w/index.php?title=Contact_list [22:13:17] RECOVERY - puppet last run on mw2204 is OK Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:20:05] (03PS5) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [22:21:29] 6operations, 6Security, 10Wikimedia-Shop, 7HTTPS, 5Patch-For-Review: Changing the URL for the Wikimedia Shop - https://phabricator.wikimedia.org/T92438#1222700 (10Dzahn) 5Open>3Resolved works for me now. you may have to restart your browser to see it. [22:22:23] (03CR) 1020after4: "see my comments on T548, we should probably submit an upstream patch to make the configuration order depend on an explicit ordering proper" [puppet] - 10https://gerrit.wikimedia.org/r/205338 (https://phabricator.wikimedia.org/T548) (owner: 10Bartosz Dziewoński) [22:22:31] (03PS6) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [22:22:48] (03CR) 10jenkins-bot: [V: 04-1] Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (owner: 10Yuvipanda) [22:23:57] (03PS7) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [22:26:26] (03PS1) 10Ori.livneh: coal: Turn into a module; add uWSGI app [puppet] - 10https://gerrit.wikimedia.org/r/205452 [22:34:46] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 780.348333947 [22:35:43] 10ops-fundraising, 10Fundraising-Backlog: Need Civi access for Donor Services agent - https://phabricator.wikimedia.org/T95011#1222730 (10CCogdill_WMF) I can confirm her cell number is up there and Kristie is ready! [22:40:26] * jgage looks at kafka on analytics1021 [22:41:06] PROBLEM - puppet last run on elastic1010 is CRITICAL puppet fail [22:43:01] 6operations, 10RESTBase, 10VisualEditor, 7Performance: Set up an API base path for REST and action APIs - https://phabricator.wikimedia.org/T95229#1222742 (10BBlack) >>! In T95229#1210586, @csteipp wrote: > Yep, being able to setup a new $foo.wikimedia.org domain, and not get penalized in performance when... [22:45:47] PROBLEM - puppet last run on mw1130 is CRITICAL Puppet has 1 failures [22:48:56] (03PS15) 10BBlack: Set up /api/v1/ entry point for restbase [puppet] - 10https://gerrit.wikimedia.org/r/203871 (https://phabricator.wikimedia.org/T95229) (owner: 10GWicke) [22:49:09] !log analytics1021: kafka disconnected from zk at 21:40; preferred-replica-election initiated at 22:48 to bring it back into service [22:49:13] Logged the message, Master [22:49:38] RECOVERY - Kafka Broker Messages In on analytics1021 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 3698.99499282 [22:51:30] (03PS8) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 [22:55:54] (03PS1) 10Dzahn: contacts: remove role from zirconium and delete it [puppet] - 10https://gerrit.wikimedia.org/r/205457 (https://phabricator.wikimedia.org/T90679) [22:56:00] (03PS9) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (https://phabricator.wikimedia.org/T96625) [22:57:37] RECOVERY - puppet last run on elastic1010 is OK Puppet is currently enabled, last run 33 seconds ago with 0 failures [22:58:58] (03CR) 10Ori.livneh: [C: 032] "Let's try it." [puppet] - 10https://gerrit.wikimedia.org/r/205452 (owner: 10Ori.livneh) [23:00:04] RoanKattouw, ^d, Krenair, Krenair: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150420T2300). Please do the needful. [23:01:05] Oh I need to do those submodule updates [23:01:30] ebernhardson, hey [23:01:40] Krenair: ho [23:01:46] your deployment calendar entry is broken [23:01:49] :S [23:01:51] sec [23:02:01] two different lines point to the same place [23:02:26] RECOVERY - puppet last run on mw1130 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [23:02:37] Krenair: thats what the (same core bump as above) is :P [23:03:01] Krenair: wasn't sure how else to format it, didn't like making one neverending line where the second patch gets lost by people skimming [23:03:25] they are separate patches, but i rolled them into the same core bump [23:04:37] (03PS1) 10Ori.livneh: coal: provision on graphite1001, not eventlog1001 [puppet] - 10https://gerrit.wikimedia.org/r/205459 [23:05:06] (03CR) 10Ori.livneh: [C: 032 V: 032] coal: provision on graphite1001, not eventlog1001 [puppet] - 10https://gerrit.wikimedia.org/r/205459 (owner: 10Ori.livneh) [23:05:32] OK so the effective patches are https://gerrit.wikimedia.org/r/#/c/205369/ and https://gerrit.wikimedia.org/r/#/c/205453/ [23:06:39] Krenair: yes [23:07:57] (03PS1) 10Ori.livneh: coal: set{ug}id coal [puppet] - 10https://gerrit.wikimedia.org/r/205460 [23:08:08] (03CR) 10Ori.livneh: [C: 032 V: 032] coal: set{ug}id coal [puppet] - 10https://gerrit.wikimedia.org/r/205460 (owner: 10Ori.livneh) [23:09:25] (03CR) 10Legoktm: [C: 04-1] Restart webservice if it is unregistered (032 comments) [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (https://phabricator.wikimedia.org/T96625) (owner: 10Yuvipanda) [23:09:48] legoktm: is’nt .keys() more explicit? [23:10:01] (the values() are the active / passive statuses of the tools) [23:10:09] legoktm: also without the , is it still a tuple? [23:10:22] oh python [23:10:23] * ori looks [23:10:25] > Calling d.keys() will return a dictionary view object. It supports operations like membership test and iteration, but its contents are not independent of the original dictionary – it is only a view. [23:10:33] YuviPanda: % doesn't require a tuple [23:10:45] hmm, I always try to pass it a tuple [23:11:05] YuviPanda: I meant remove the parenthesis and comma [23:11:05] legoktm: oh, hmm. in this case it doesn’t matter, no? I’m not changing the dictionary at all... [23:11:20] hmm [23:11:37] I’ve always passed tuples to % formatting, even if it’s just one... [23:11:58] I like typing less :P [23:12:06] I wrote a year of Java :P [23:12:43] for view vs list, it doesn't matter much here, it's just so later down the road you don't try using list operations on the view and wonder why it doesn't work :P [23:13:04] (03PS10) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (https://phabricator.wikimedia.org/T96625) [23:13:13] right :) [23:13:20] legoktm: I still like .keys() better because more explicit [23:13:25] actually [23:13:29] why do I even need a list [23:13:37] I’m just checking for membership [23:13:40] might as well use the dict [23:14:01] (03PS11) 10Yuvipanda: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (https://phabricator.wikimedia.org/T96625) [23:14:01] legoktm: ^ [23:14:05] .keys is gone [23:14:14] :P [23:14:23] legoktm: it’s the nodejs eating into my brain :P [23:14:27] (or not) [23:14:32] (03CR) 10Legoktm: [C: 032] Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (https://phabricator.wikimedia.org/T96625) (owner: 10Yuvipanda) [23:14:36] I warned you!!! [23:14:40] (https://tools.wmflabs.org/wikistreak) [23:14:51] I can do ‘git push deploy master’ and it just gets deployed!!!!1 [23:14:53] (03Merged) 10jenkins-bot: Restart webservice if it is unregistered [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/205439 (https://phabricator.wikimedia.org/T96625) (owner: 10Yuvipanda) [23:15:59] * YuviPanda is also super groggy today, on account of having woken up early [23:16:53] ebernhardson, syncing [23:17:01] If it decides to wake up today, of course. [23:17:05] :) [23:17:39] !log krenair Synchronized php-1.26wmf2: php-1.26wmf2/extensions/Flow https://gerrit.wikimedia.org/r/#/c/205432/ (duration: 01m 11s) [23:17:43] Logged the message, Master [23:17:58] ebernhardson, please check [23:18:26] Krenair: sure, sec [23:18:59] Krenair: works a charm, thanks [23:27:35] ugh. [23:29:29] Looks like https://gerrit.wikimedia.org/r/#/c/202792/ was merged but there was no submodule update [23:36:38] !log krenair Synchronized php-1.26wmf2/extensions/AbuseFilter: https://gerrit.wikimedia.org/r/205462 (duration: 00m 15s) [23:36:42] Logged the message, Master [23:38:36] Krenair: Also, I merged a couple things about an hour ago (just in case Jenkins was taking forever), but I forgot to add them to the SWAT schedule. I can deploy them myself if you like. [23:39:06] https://gerrit.wikimedia.org/r/#/c/205415/ and https://gerrit.wikimedia.org/r/#/c/205409/ [23:39:07] !log krenair Synchronized php-1.26wmf1/extensions/AbuseFilter: https://gerrit.wikimedia.org/r/205463 (duration: 00m 14s) [23:39:11] Logged the message, Master [23:40:14] do you have submodule updates made for these kaldari? [23:41:27] Krenair: no. [23:42:00] I'm gonna do a couple of config changes, if you'd like to get those up in the mean time.. [23:42:10] Krenair: ok [23:42:44] (03CR) 10Alex Monk: [C: 032] New user groups on fr.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198822 (https://phabricator.wikimedia.org/T90979) (owner: 10Dereckson) [23:42:54] (03Merged) 10jenkins-bot: New user groups on fr.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198822 (https://phabricator.wikimedia.org/T90979) (owner: 10Dereckson) [23:43:49] !log krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198822/ (duration: 00m 13s) [23:43:54] Logged the message, Master [23:45:19] (03CR) 10Alex Monk: [C: 032] Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [23:45:32] (03CR) 10Yuvipanda: "*bump*" [puppet] - 10https://gerrit.wikimedia.org/r/199267 (https://phabricator.wikimedia.org/T85606) (owner: 10coren) [23:46:03] (03Merged) 10jenkins-bot: Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [23:47:10] !log krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/201915/ (duration: 00m 24s) [23:47:13] Logged the message, Master [23:48:27] Krenair: submodule updates ready: https://gerrit.wikimedia.org/r/#/c/205469/ and https://gerrit.wikimedia.org/r/#/c/205470/ [23:50:37] Krenair: also updated the deployment schedule to add them [23:52:05] (03CR) 10Alex Monk: [C: 032] Disable Commons file search on officewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204536 (https://phabricator.wikimedia.org/T94856) (owner: 10Glaisher) [23:52:11] (03Merged) 10jenkins-bot: Disable Commons file search on officewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/204536 (https://phabricator.wikimedia.org/T94856) (owner: 10Glaisher) [23:52:53] (03PS1) 10Ori.livneh: Serve coal from performance.wikimedia.org/coal [puppet] - 10https://gerrit.wikimedia.org/r/205471 [23:53:10] !log krenair Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/#/c/204536/ - disable commons file search on officewiki, per erik (duration: 00m 12s) [23:53:14] Logged the message, Master [23:54:53] (03PS2) 10Ori.livneh: Serve coal from performance.wikimedia.org/coal [puppet] - 10https://gerrit.wikimedia.org/r/205471 [23:55:14] (03CR) 10Ori.livneh: [C: 032 V: 032] Serve coal from performance.wikimedia.org/coal [puppet] - 10https://gerrit.wikimedia.org/r/205471 (owner: 10Ori.livneh)