[00:14:08] (03PS5) 10Krinkle: grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 [00:15:48] (03PS6) 10Krinkle: grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 [00:15:54] (03PS7) 10Krinkle: grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 [00:16:10] (03CR) 10Krinkle: "OK. https://grafana.wikimedia.org/#/dashboard/db/home is now working." [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [00:17:11] (03CR) 10Hoo man: [C: 031] "Thanks" [puppet] - 10https://gerrit.wikimedia.org/r/225902 (https://phabricator.wikimedia.org/T106045) (owner: 10Filippo Giunchedi) [00:21:15] (03CR) 10Krinkle: "@Dzahn: None of those warnings are related to operations/puppet or this patch. That html quirks are generated by the CodePen preview becau" [puppet] - 10https://gerrit.wikimedia.org/r/223012 (owner: 10Krinkle) [01:47:35] PROBLEM - puppet last run on mw2188 is CRITICAL puppet fail [02:03:12] !log LocalisationUpdate failed (1.26wmf14) at 2015-07-21 02:03:11+00:00 [02:03:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:07:22] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 21 02:07:22 UTC 2015 (duration 7m 21s) [02:07:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:08:55] Reedy: I poked ori IRL about the double l10nupdate stuff [02:09:13] hopefully he will remember to figure out what he did and undo it [02:09:30] lol [02:16:04] RECOVERY - puppet last run on mw2188 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [02:16:54] 6operations, 10Wikimedia-Logstash: Update Elasticsearch on logstash* to elasticsearch-1.7.0.deb - https://phabricator.wikimedia.org/T106126#1466192 (10bd808) [02:18:54] 6operations, 10Wikimedia-Logstash: Update Elasticsearch on logstash* to elasticsearch-1.7.0.deb - https://phabricator.wikimedia.org/T106126#1459787 (10bd808) Upgraded deployment-logstash2 in beta cluster to elasticsearch-1.7.0.deb [02:23:17] !log l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 06m 55s) [02:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:26:59] !log LocalisationUpdate completed (1.26wmf14) at 2015-07-21 02:26:59+00:00 [02:27:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:31:05] PROBLEM - Host mw2027 is DOWN: PING CRITICAL - Packet loss = 100% [02:32:15] RECOVERY - Host mw2027 is UPING WARNING - Packet loss = 86%, RTA = 92.76 ms [03:04:55] PROBLEM - puppet last run on mw1216 is CRITICAL Puppet has 1 failures [03:12:04] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [03:13:45] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 61466 bytes in 0.426 second response time [03:31:25] RECOVERY - puppet last run on mw1216 is OK Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:32:17] 6operations, 7Database: new external storage cluster(s) - https://phabricator.wikimedia.org/T105843#1466232 (10matthiasmullie) [04:47:44] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL Anomaly detected: 10 data above and 1 below the confidence bounds [04:51:45] PROBLEM - puppet last run on mw2084 is CRITICAL Puppet has 1 failures [04:54:18] 6operations, 6Commons: Commons thumbnail of Pluto photo is broken at 500px - https://phabricator.wikimedia.org/T105793#1466263 (10MZMcBride) I just noticed that someone did, in fact, upload a version of this image that was smaller than 500px: !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 21 05:08:32 UTC 2015 (duration 8m 31s) [05:08:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [05:11:50] (03PS2) 10MaxSem: postgresql: install postgis package for shp2pgsql and similar binaries [puppet] - 10https://gerrit.wikimedia.org/r/225567 [05:18:24] RECOVERY - puppet last run on mw2084 is OK Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:25:00] urandom: yeah? [05:25:07] * YuviPanda is in line at airport security [05:29:24] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected [05:30:15] PROBLEM - puppet last run on cp3008 is CRITICAL puppet fail [05:53:02] (03CR) 10Gilles: [C: 031] grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [05:56:36] RECOVERY - puppet last run on cp3008 is OK Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:04:07] 10Ops-Access-Requests, 6operations: tjones needs access to stat1002 - https://phabricator.wikimedia.org/T106175#1466290 (10Matanya) you can put it here. via web, not email. [06:31:15] PROBLEM - puppet last run on cp2001 is CRITICAL Puppet has 1 failures [06:31:24] PROBLEM - puppet last run on mw2207 is CRITICAL Puppet has 1 failures [06:31:25] PROBLEM - puppet last run on mw1135 is CRITICAL Puppet has 1 failures [06:32:24] PROBLEM - puppet last run on mw2050 is CRITICAL Puppet has 1 failures [06:33:04] PROBLEM - puppet last run on mw1170 is CRITICAL Puppet has 1 failures [06:57:24] RECOVERY - puppet last run on mw1170 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:57:34] RECOVERY - puppet last run on cp2001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:35] RECOVERY - puppet last run on mw1135 is OK Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:58:35] RECOVERY - puppet last run on mw2050 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:26] RECOVERY - puppet last run on mw2207 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:09] (03CR) 10Jcrespo: [C: 032] postgresql: install postgis package for shp2pgsql and similar binaries [puppet] - 10https://gerrit.wikimedia.org/r/225567 (owner: 10MaxSem) [07:26:21] (03CR) 10Jcrespo: "A couple of comments on the patch." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/225702 (owner: 10MaxSem) [07:36:01] (03CR) 10Giuseppe Lavagetto: "@SMalyshev where do you get that error? I tried to compile the catalog on a self-hosted puppetmaster and had no issues; maybe there is som" [puppet] - 10https://gerrit.wikimedia.org/r/223663 (owner: 10Smalyshev) [07:40:27] (03PS1) 10Yuvipanda: celery: Add celery::flower setup for setting up a flower monitor [puppet] - 10https://gerrit.wikimedia.org/r/226041 [07:42:07] _joe_, I think you may know this: is service::node compatible with jessie's systemd? [07:42:38] 10Ops-Access-Requests, 6operations: Access request to stat1002 - https://phabricator.wikimedia.org/T106370#1466371 (10dcausse) 3NEW [07:42:54] I am ge3tting a "Provider upstart is not functional on this host" [07:44:22] (03CR) 10Yuvipanda: [C: 032 V: 032] celery: Add celery::flower setup for setting up a flower monitor [puppet] - 10https://gerrit.wikimedia.org/r/226041 (owner: 10Yuvipanda) [07:45:36] (03PS1) 10Yuvipanda: celery: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/226042 [07:45:44] ^ classic [07:45:50] (03CR) 10Yuvipanda: [C: 032 V: 032] celery: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/226042 (owner: 10Yuvipanda) [07:55:14] 6operations, 6Discovery, 10Maps, 6Services, and 2 others: Puppetize Kartotherian & Tilerator for deployment - https://phabricator.wikimedia.org/T105074#1466379 (10jcrespo) Getting the error right now: `Error: /Stage[main]/Kartotherian/Service::Node[kartotherian]/Service[kartotherian]: Provider upstart is... [07:58:54] <_joe_> jynus: it is AFAIR [07:58:59] <_joe_> but lemme verify [07:59:19] <_joe_> jynus: it's not, and I need it as well [07:59:23] I think it does not, it on creates an init.conf file [07:59:24] <_joe_> lemme fix this [07:59:33] oh, you can do that? [07:59:43] that would be great! [08:02:17] (03PS12) 10Giuseppe Lavagetto: Add definitions for WDQS service [puppet] - 10https://gerrit.wikimedia.org/r/223663 (owner: 10Smalyshev) [08:05:28] (03PS1) 10Yuvipanda: ores: Add flower class / role [puppet] - 10https://gerrit.wikimedia.org/r/226043 [08:07:01] (03CR) 10Yuvipanda: [C: 032] ores: Add flower class / role [puppet] - 10https://gerrit.wikimedia.org/r/226043 (owner: 10Yuvipanda) [08:13:05] (03PS1) 10Yuvipanda: ores: Add missing file [puppet] - 10https://gerrit.wikimedia.org/r/226044 [08:13:18] (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Add missing file [puppet] - 10https://gerrit.wikimedia.org/r/226044 (owner: 10Yuvipanda) [08:21:29] jynus, ping [08:21:45] what server are you getting the error on? [08:22:40] yurik, do not worry, it is not because of the code [08:22:49] (03PS1) 10Yuvipanda: ores: Use include style for redisproxy [puppet] - 10https://gerrit.wikimedia.org/r/226045 [08:23:02] (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Use include style for redisproxy [puppet] - 10https://gerrit.wikimedia.org/r/226045 (owner: 10Yuvipanda) [08:23:05] jynus, all servers seem to be working ok last time i checked [08:23:07] I am getting it on maps-test2001 [08:23:45] there seems to some missing functionality on the deploy script [08:24:00] probaby it was started manually [08:24:15] (it is a puppet error, not an application error) [08:24:27] jynus, could have been, even though i did do a git deploy sync [08:24:33] i meant git deploy restart [08:24:54] but later i connected to the server a number of times for various updates [08:25:09] i just checked - service on 2001 runs fine [08:25:10] (03PS1) 10Yuvipanda: ores: Fix duplicate declaration of redisproxy in worker defs [puppet] - 10https://gerrit.wikimedia.org/r/226046 [08:25:30] (03CR) 10Yuvipanda: [C: 032 V: 032] ores: Fix duplicate declaration of redisproxy in worker defs [puppet] - 10https://gerrit.wikimedia.org/r/226046 (owner: 10Yuvipanda) [08:25:56] jynus, will you have time to install varnish on 200{1-4} ? [08:26:00] still, I have to fix puppet [08:26:19] yurik, sure, I think alex is on vacation this week [08:26:33] I will take over his tasks [08:26:39] 6operations, 10CirrusSearch, 6Discovery, 3Discovery-Cirrus-Sprint: Validate Cirrus against Elasticsearch 1.7.0 - https://phabricator.wikimedia.org/T106160#1466386 (10dcausse) a:3dcausse [08:27:54] jynus, thx! let me know if you have any questions - i will be on the ground for the next few hours, will be able to reply to emails. We could set up varnish in simple mem-only, or can do a double layer with consistent hashing, but still probably better to keep 2nd level mem-only [08:27:56] (03PS2) 10Filippo Giunchedi: Revert "Don't killall $DAEMON" [puppet] - 10https://gerrit.wikimedia.org/r/225932 (owner: 10GWicke) [08:28:02] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Revert "Don't killall $DAEMON" [puppet] - 10https://gerrit.wikimedia.org/r/225932 (owner: 10GWicke) [08:30:04] 6operations, 6Discovery, 10Maps, 3Discovery-Maps-Sprint: Assign varnish memory-only role to maps servers - https://phabricator.wikimedia.org/T105076#1466388 (10jcrespo) a:3jcrespo [08:35:43] (03PS1) 10Yuvipanda: celery: Fix formatting of commandline arguments to flower [puppet] - 10https://gerrit.wikimedia.org/r/226047 [08:35:54] (03PS2) 10Yuvipanda: celery: Fix formatting of commandline arguments to flower [puppet] - 10https://gerrit.wikimedia.org/r/226047 [08:36:02] (03CR) 10Yuvipanda: [C: 032 V: 032] celery: Fix formatting of commandline arguments to flower [puppet] - 10https://gerrit.wikimedia.org/r/226047 (owner: 10Yuvipanda) [08:41:26] (03PS1) 10Giuseppe Lavagetto: service::node: convert to base::service_unit, add systemd support [puppet] - 10https://gerrit.wikimedia.org/r/226048 [08:41:45] <_joe_> jynus: ^^ needs some testing [08:42:34] <_joe_> which I'm going to do now :) [08:43:20] nice, thank you [08:48:28] <_joe_> jynus: is this for maps, right? [08:48:35] _joe_, yes [08:52:55] (03CR) 10Gilles: [C: 031] Tessera: base config.py.erb on Tessera's config.py [puppet] - 10https://gerrit.wikimedia.org/r/222365 (owner: 10Ori.livneh) [09:06:31] (03CR) 10Filippo Giunchedi: [C: 031] service::node: convert to base::service_unit, add systemd support [puppet] - 10https://gerrit.wikimedia.org/r/226048 (owner: 10Giuseppe Lavagetto) [09:07:30] 10Ops-Access-Requests, 6operations: Access request to stat1002 for dcausse - https://phabricator.wikimedia.org/T106370#1466438 (10fgiunchedi) [09:09:22] (03PS2) 10Giuseppe Lavagetto: service::node: convert to base::service_unit, add systemd support [puppet] - 10https://gerrit.wikimedia.org/r/226048 [09:10:06] (03PS1) 10DCausse: Upgrade plugins to elasticsearch 1.7.0 [software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/226049 (https://phabricator.wikimedia.org/T106165) [09:10:48] (03CR) 10DCausse: [C: 04-1] "This must not be merged now." [software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/226049 (https://phabricator.wikimedia.org/T106165) (owner: 10DCausse) [09:11:34] (03CR) 10Giuseppe Lavagetto: [C: 032] service::node: convert to base::service_unit, add systemd support [puppet] - 10https://gerrit.wikimedia.org/r/226048 (owner: 10Giuseppe Lavagetto) [09:11:40] 10Ops-Access-Requests, 6operations: Access request to stat1002 for dcausse - https://phabricator.wikimedia.org/T106370#1466448 (10fgiunchedi) p:5Triage>3Normal hi @dcausse, we'd need manager approval for this, there's also 3 days grace/waiting period. [09:18:34] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466459 (10Gilles) a:3Gilles I'll add 400px, since it seems like the best compromise in the current thumbnailing environment.... [09:23:07] <_joe_> jynus: it should now work on maps-test2001 [09:24:24] _joe_, it does, thank you! [09:25:47] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466466 (10fgiunchedi) @gilles I take it new defaults will affect only new uploads? also plans on removing some smaller sizes l... [09:30:27] (03PS1) 10Gilles: Offer 400px as a thumbnail size available in Special:Preferences [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226051 (https://phabricator.wikimedia.org/T65440) [09:31:34] _joe_: buongiorno ! Have you managed to look at the conftool tests related patches I sent a couple weeks ago? :D https://gerrit.wikimedia.org/r/#/q/status:open+topic:221087,n,z [09:31:50] the changes are idling in my Gerrit dashboard :D [09:36:08] 6operations, 10Continuous-Integration-Infrastructure, 6Multimedia, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1466477 (10MoritzMuehlenhoff) >>! In T103335#1465013, @brion wrote: > As long as whatever we switc... [09:39:32] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466478 (10Gilles) @fgiunchedi no, it affects all files. Basically users can opt into viewing all thumbnai... [09:45:27] <_joe_> hashar: nope, but I will between today and tomorrow [09:45:42] <_joe_> if I get to the bottom of the ganglia affair today, I might tackle them :) [09:45:47] _joe_: great! They probably need some rebasing / retest though [09:46:00] what about Gangia? Are we dropping it ? :D [09:46:27] <_joe_> https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/puppet+branch:production+topic:burn_ganglia,n,z [09:46:39] 7Puppet: Write, publish and deploy puppet-lint plug-in for ensure attribute bareword check - https://phabricator.wikimedia.org/T95377#1466494 (10Matanya) [09:47:19] _joe_: ah getting rid of some legacy tech debt great! [09:47:33] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466497 (10Gilles) It would make sense to keep 200 and 300, given their overlap with each other and with 4... [09:58:01] 6operations, 10RESTBase-Cassandra: setup an alertable threshold for Cassandra heap dumps - https://phabricator.wikimedia.org/T106346#1466499 (10fgiunchedi) indeed they can add up quickly, what about trimming their size periodically but leave some behind? the rationale being that under normal circumstances ther... [09:58:16] 6operations, 10RESTBase-Cassandra: setup an alertable threshold for Cassandra heap dumps - https://phabricator.wikimedia.org/T106346#1466500 (10fgiunchedi) p:5Triage>3Normal [09:59:28] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466501 (10Gilles) The default is 220, btw. Defined in a very weird way by wmgThumbsizeIndex, which is the... [10:00:47] 7Puppet, 6operations, 5Patch-For-Review: Make Puppet repository pass lenient and strict lint checks - https://phabricator.wikimedia.org/T87132#1466504 (10Matanya) from my POV: yes. [10:01:11] (03PS2) 10Giuseppe Lavagetto: ganglia: use ganglia_new by default, disable otherwise [puppet] - 10https://gerrit.wikimedia.org/r/225872 [10:05:36] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466505 (10fgiunchedi) thanks for the detailed explanation @gilles! If it is a cheap (space wise) additio... [10:05:40] (03CR) 10Filippo Giunchedi: [C: 031] Offer 400px as a thumbnail size available in Special:Preferences [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226051 (https://phabricator.wikimedia.org/T65440) (owner: 10Gilles) [10:11:24] (03CR) 10Giuseppe Lavagetto: [C: 032] ganglia: use ganglia_new by default, disable otherwise [puppet] - 10https://gerrit.wikimedia.org/r/225872 (owner: 10Giuseppe Lavagetto) [10:19:39] (03PS2) 10Giuseppe Lavagetto: ganglia: move ganglia::plugins::python to the correct location [puppet] - 10https://gerrit.wikimedia.org/r/225873 [10:20:32] 10Ops-Access-Requests, 6operations, 6Analytics-Backlog: Provide daniel (Daniel Kinzler) with Hive access - https://phabricator.wikimedia.org/T106047#1466515 (10fgiunchedi) +ops-access-requests to get this triaged/processed [10:20:48] 10Ops-Access-Requests, 6operations, 6Analytics-Backlog: Provide daniel (Daniel Kinzler) with Hive access - https://phabricator.wikimedia.org/T106047#1466518 (10fgiunchedi) p:5Triage>3Normal [10:21:49] (03PS3) 10Filippo Giunchedi: admin: add hoo to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/225902 (https://phabricator.wikimedia.org/T106045) [10:22:00] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] admin: add hoo to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/225902 (https://phabricator.wikimedia.org/T106045) (owner: 10Filippo Giunchedi) [10:22:54] 10Ops-Access-Requests, 10Ops-Access-Reviews, 6operations, 5Patch-For-Review: Provide hoo (Marius Hoch) with Hive access - https://phabricator.wikimedia.org/T106045#1466526 (10fgiunchedi) 5Open>3Resolved [10:28:14] (03PS2) 10Muehlenhoff: add ferm rules for memcached [puppet] - 10https://gerrit.wikimedia.org/r/222556 [10:28:27] (03CR) 10Muehlenhoff: [C: 032 V: 032] add ferm rules for memcached [puppet] - 10https://gerrit.wikimedia.org/r/222556 (owner: 10Muehlenhoff) [10:29:35] (03PS3) 10Giuseppe Lavagetto: ganglia: move ganglia::plugins::python to the correct location [puppet] - 10https://gerrit.wikimedia.org/r/225873 [10:29:51] 10Ops-Access-Requests, 6operations, 7Icinga: give John Lewis permissions to send commands in icinga - https://phabricator.wikimedia.org/T105229#1466532 (10fgiunchedi) 5Open>3stalled [10:30:01] 10Ops-Access-Requests, 6operations, 6Services, 7Icinga, 7Monitoring: give services team permissions to send commands in icinga - https://phabricator.wikimedia.org/T105228#1466534 (10fgiunchedi) 5Open>3stalled [10:33:54] (03PS1) 10Filippo Giunchedi: admin: add daniel to analytics-privatedata-users group [puppet] - 10https://gerrit.wikimedia.org/r/226055 (https://phabricator.wikimedia.org/T106047) [10:35:16] 10Ops-Access-Requests, 6operations: Requesting access to stat1003 and eventlogging for legoktm - https://phabricator.wikimedia.org/T106184#1466550 (10fgiunchedi) a:5Joe>3None [10:35:19] 10Ops-Access-Requests, 6operations, 6Analytics-Backlog, 5Patch-For-Review: Provide daniel (Daniel Kinzler) with Hive access - https://phabricator.wikimedia.org/T106047#1466551 (10fgiunchedi) a:5Ottomata>3None [10:35:21] 10Ops-Access-Requests, 6operations, 10Analytics-Cluster: Sudo permissions for hdfs user madhuvishy on analytics-hadoop - https://phabricator.wikimedia.org/T104020#1466552 (10fgiunchedi) a:5Ottomata>3None [10:36:49] (03PS4) 10Giuseppe Lavagetto: ganglia: move ganglia::plugins::python to the correct location [puppet] - 10https://gerrit.wikimedia.org/r/225873 [10:37:09] 7Puppet, 6operations, 5Patch-For-Review: Resource attributes are quoted inconsistently - https://phabricator.wikimedia.org/T91908#1466556 (10fgiunchedi) p:5Low>3Lowest [10:39:35] anyone here? [10:40:42] probably [10:41:51] I was just wondering why wikibase doesn't have a wmf/1.26wmf14 branch? [10:44:54] (03PS5) 10Giuseppe Lavagetto: ganglia: move ganglia::plugins::python to the correct location [puppet] - 10https://gerrit.wikimedia.org/r/225873 [10:45:20] (03CR) 10Giuseppe Lavagetto: [C: 032] "Noop, confirmed by the puppet compiler" [puppet] - 10https://gerrit.wikimedia.org/r/225873 (owner: 10Giuseppe Lavagetto) [10:45:38] (03CR) 10Giuseppe Lavagetto: [V: 032] ganglia: move ganglia::plugins::python to the correct location [puppet] - 10https://gerrit.wikimedia.org/r/225873 (owner: 10Giuseppe Lavagetto) [10:50:48] 6operations: Rename 'restricted' group? - https://phabricator.wikimedia.org/T104671#1466568 (10fgiunchedi) p:5Triage>3Low agreed we could revisit, I'll leave it to folks with more context than me on why `restricted` exists and what it does. Note also that access to bastions is granted (bast1001/hooft) [10:52:40] (03PS2) 10Muehlenhoff: add ferm rules for redis [puppet] - 10https://gerrit.wikimedia.org/r/222554 [10:53:25] <_joe_> some puppet errors can happen, will be my fault [10:53:27] <_joe_> oh btw [10:53:33] <_joe_> no icinga-wm ? [10:54:03] 6operations, 10Wikimedia-Git-or-Gerrit, 7HTTPS: Chromium says "Your connection to gerrit.wikimedia.org is encrypted with obsolete cryptography" - https://phabricator.wikimedia.org/T104649#1466576 (10fgiunchedi) 5Open>3declined a:3fgiunchedi @polybuildr I'm going to resolve this in favor of {T55259}, pl... [10:54:05] PROBLEM - puppet last run on erbium is CRITICAL Puppet has 2 failures [10:54:38] Mjbmr: wikibase? I don't even see that in make-wmf-branch, so that's probably why it doesn't have a release branch [10:54:40] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466580 (10Nemo_bis) >>! In T65440#1466501, @Gilles wrote: > The default is 220, btw. Isn't 440 better as... [10:54:53] 6operations: Create instrumentation to monitor load on geoiplookup.wikimedia.org - https://phabricator.wikimedia.org/T104258#1466581 (10fgiunchedi) p:5Triage>3Normal [10:56:48] (03PS2) 1020after4: Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226031 (https://phabricator.wikimedia.org/T104904) [11:01:14] PROBLEM - puppet last run on gallium is CRITICAL Puppet has 2 failures [11:02:01] 6operations, 10Wikimedia-DNS, 7Mail: DNS Change for GreenHouse - https://phabricator.wikimedia.org/T103893#1466589 (10fgiunchedi) 5Open>3declined a:3fgiunchedi @jgulingan I'm going to decline the ticket for the reasons @faidon explained, if there are updates feel free to reopen! [11:02:43] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466593 (10Gilles) >>! In T65440#1466580, @Nemo_bis wrote: >>>! In T65440#1466501, @Gilles wrote: >> The d... [11:03:02] 6operations, 10Wikimedia-Git-or-Gerrit: Get rid of the gerrit Debian package and migrate to puppet - https://phabricator.wikimedia.org/T103735#1466594 (10fgiunchedi) p:5Triage>3Low [11:03:30] twentyafterfour: can you fix it manually please? [11:03:37] (03PS1) 10Giuseppe Lavagetto: udp2log: use ganglia::plugins::python [puppet] - 10https://gerrit.wikimedia.org/r/226058 [11:03:38] 6operations, 10Wikimedia-Git-or-Gerrit: Remove Java 6 from ytterbium.wikimedia.org (Gerrit production host) - https://phabricator.wikimedia.org/T103668#1466597 (10fgiunchedi) p:5Triage>3Low [11:44:33] (03PS2) 10Muehlenhoff: Update to 3.19.8-ckt3 [debs/linux] - 10https://gerrit.wikimedia.org/r/225850 [11:45:34] (03CR) 10Muehlenhoff: [C: 032 V: 032] Update to 3.19.8-ckt3 [debs/linux] - 10https://gerrit.wikimedia.org/r/225850 (owner: 10Muehlenhoff) [12:00:48] (03PS1) 10Muehlenhoff: Update to 3.19.8-ckt4 [debs/linux] - 10https://gerrit.wikimedia.org/r/226062 [12:00:52] (03PS2) 10Giuseppe Lavagetto: udp2log: use ganglia::plugins::python [puppet] - 10https://gerrit.wikimedia.org/r/226058 [12:02:50] (03PS1) 10Giuseppe Lavagetto: misc: remove misc::monitoring::htcp-loss [puppet] - 10https://gerrit.wikimedia.org/r/226063 [12:03:12] (03CR) 10Giuseppe Lavagetto: [C: 032] udp2log: use ganglia::plugins::python [puppet] - 10https://gerrit.wikimedia.org/r/226058 (owner: 10Giuseppe Lavagetto) [12:18:39] (03CR) 10Muehlenhoff: [C: 032 V: 032] Update to 3.19.8-ckt4 [debs/linux] - 10https://gerrit.wikimedia.org/r/226062 (owner: 10Muehlenhoff) [12:29:27] 6operations, 5Patch-For-Review: Mediawiki font packages: switch to Jessie - https://phabricator.wikimedia.org/T102623#1466754 (10fgiunchedi) [12:29:29] 6operations: Investigate Ubuntu fork of ttf-indic-fonts and bring it in Jessie - https://phabricator.wikimedia.org/T103328#1466751 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi so it looks like there were space considerations in Ubuntu for forking the fonts, for jessie I think we're fine with installing `fo... [12:35:19] (03PS1) 10Muehlenhoff: Enable ferm for mc2* systems in codfw [puppet] - 10https://gerrit.wikimedia.org/r/226065 [12:39:44] 6operations: tin doesn't have access to same memcached as terbium and app servers - https://phabricator.wikimedia.org/T103198#1466776 (10fgiunchedi) the fact that there's nutcracker on tin doesn't seem intentional, likely tin used to get some mediawiki roles and thus nutcracker but that stopped a year ago: ```... [12:40:07] 6operations: tin doesn't have access to same memcached as terbium and app servers - https://phabricator.wikimedia.org/T103198#1466777 (10fgiunchedi) p:5Triage>3Normal [12:46:08] 6operations, 10Wikimedia-Git-or-Gerrit, 7HTTPS: Chromium says "Your connection to gerrit.wikimedia.org is encrypted with obsolete cryptography" - https://phabricator.wikimedia.org/T104649#1466785 (10Chmarkine) 5declined>3Resolved Why decline it? It has been resolved! Apache 2.2 now supports ECDHE. See T5... [12:49:46] 6operations, 10Wikimedia-Git-or-Gerrit, 7HTTPS: Chromium says "Your connection to gerrit.wikimedia.org is encrypted with obsolete cryptography" - https://phabricator.wikimedia.org/T104649#1466789 (10fgiunchedi) thanks @chmarkine, I did miss that update! even better [13:15:33] (03PS1) 10Muehlenhoff: Add ferm rules for mariadb labsdb [puppet] - 10https://gerrit.wikimedia.org/r/226068 (https://phabricator.wikimedia.org/T104699) [13:23:49] 6operations: various salt-minions are not replying to test.ping or commands - https://phabricator.wikimedia.org/T102808#1466809 (10fgiunchedi) @arielglenn what's left to do here? still prio high? [13:24:42] 10Ops-Access-Requests, 6operations: tjones needs access to stat1002 - https://phabricator.wikimedia.org/T106175#1466815 (10TJones) ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaSUgjRZTi0djdSkKrqxTUXQisq/OCb9ZlXZNJQ69VD6Gup+GlzLxT5MhZrQ6hlf6p/NVvuc9y7zVI09qVYgbpe1L6EJWjyA2CnqLdukq46JDo31EcE1rhfeVqm7eba3X5DWLR3Tu7+tNu... [13:24:43] (03PS1) 10Muehlenhoff: Add ferm rules for rsync server [puppet] - 10https://gerrit.wikimedia.org/r/226071 [13:25:28] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1466819 (10Nemo_bis) Quoting: ``` 800,1611558739172,12428155 600,240281894064,2708978 440,165087861885,314... [13:25:40] (03PS1) 10Giuseppe Lavagetto: misc::udp2log: use ganglia::plugins::python everywhere [puppet] - 10https://gerrit.wikimedia.org/r/226072 [13:27:47] 6operations, 7Monitoring: icinga log rotation wipes out portions of history - https://phabricator.wikimedia.org/T102397#1466826 (10fgiunchedi) see also {T7} about sending icinga alerts to logstash, not sure if we'd benefit from the complete icinga history though [13:27:50] (03CR) 10Giuseppe Lavagetto: [C: 032] misc::udp2log: use ganglia::plugins::python everywhere [puppet] - 10https://gerrit.wikimedia.org/r/226072 (owner: 10Giuseppe Lavagetto) [13:31:05] RECOVERY - puppet last run on erbium is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:28] (03CR) 10Giuseppe Lavagetto: [C: 032] misc: remove misc::monitoring::htcp-loss [puppet] - 10https://gerrit.wikimedia.org/r/226063 (owner: 10Giuseppe Lavagetto) [13:32:19] (03PS2) 10Giuseppe Lavagetto: misc: remove misc::monitoring::htcp-loss [puppet] - 10https://gerrit.wikimedia.org/r/226063 [13:41:26] YuviPanda: what was it that required that change; what is the conflict you speak of? i'm seeing this: https://phabricator.wikimedia.org/P1029 [13:46:08] 6operations: Investigate smsglobal delivery failures from 2015-06-13 weekend - https://phabricator.wikimedia.org/T102396#1466845 (10fgiunchedi) there seem to be ongoing, I checked mail logs on polonium for my number and it shows two entries: `2015-07-16 10:17:14` and `2015-07-16 10:26:34` respectively a page for... [13:50:36] 6operations, 10Traffic, 7Monitoring: Implement pybal pool state monitoring and alerting via icinga - https://phabricator.wikimedia.org/T102394#1466853 (10fgiunchedi) looks like this should be easier once we have pool state exposed on etcd with {T97029} [13:51:38] 6operations, 10Traffic, 7Pybal: Make pybal accept 30[12] for ProxyFetch - https://phabricator.wikimedia.org/T102393#1466856 (10fgiunchedi) a:3BBlack [13:53:52] 6operations: asw-b-eqiad:ge-5/0/1(nas1001-a:e0a) port saturation - https://phabricator.wikimedia.org/T106181#1466861 (10fgiunchedi) p:5Triage>3Low [13:55:29] 7Blocked-on-Operations, 6operations, 6Commons, 6Multimedia, and 5 others: Convert eqiad imagescalers to HHVM, Trusty - https://phabricator.wikimedia.org/T84842#1466863 (10Joe) FTR, I am going to depool all remaining Zend imagescaler today to test any outstanding problems with those. If none arise, I'm goin... [13:57:49] <_joe_> !log depooling mw1158-60 from the imagescaler pool, to test HHVM-only imagescalers [13:57:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [13:58:31] 6operations, 7Graphite: Upgrade to Grafana v2.x - https://phabricator.wikimedia.org/T104738#1466864 (10fgiunchedi) p:5Triage>3Normal [13:59:00] 6operations, 10CirrusSearch, 6Discovery, 3Discovery-Cirrus-Sprint: Validate Cirrus against Elasticsearch 1.7.0 - https://phabricator.wikimedia.org/T106160#1466866 (10Manybubbles) Ahk! I just finished this! I ran the Cirrus test suite with these instructions: https://gerrit.wikimedia.org/r/#/c/226074/ [14:00:20] (03PS1) 10Muehlenhoff: add ferm rules for postgis [puppet] - 10https://gerrit.wikimedia.org/r/226075 [14:02:21] <_joe_> godog: if any tickets/report fly by about imagescaling keep in mind we're running 100% on HHVM now [14:04:17] (03CR) 10Smalyshev: "@Guiseppe I got it on db01." [puppet] - 10https://gerrit.wikimedia.org/r/223663 (owner: 10Smalyshev) [14:04:55] _joe_: yup, thanks for the heads up [14:06:54] 6operations, 10CirrusSearch, 6Discovery, 3Discovery-Cirrus-Sprint: Validate Cirrus against Elasticsearch 1.7.0 - https://phabricator.wikimedia.org/T106160#1466869 (10dcausse) a:5dcausse>3Manybubbles [14:07:22] 6operations, 10CirrusSearch, 6Discovery, 3Discovery-Cirrus-Sprint: Validate Cirrus against Elasticsearch 1.7.0 - https://phabricator.wikimedia.org/T106160#1460693 (10dcausse) Thanks for the instructions! I just tested locally with a "manual" upgrade. [14:08:11] (03PS2) 10Giuseppe Lavagetto: ganglia: move ganglia::collector to ganglia::deprecated::collector [puppet] - 10https://gerrit.wikimedia.org/r/225874 [14:11:17] <_joe_> SMalyshev: mind if I update your puppetmaster a bit? [14:13:43] _joe_: hi, are you planning on video scalers too ? [14:18:32] <_joe_> matanya: yes, in a not too distant future I do [14:18:40] thanks! [14:19:51] 6operations, 7Mail: Ferm rules for MX mail servers - https://phabricator.wikimedia.org/T104979#1466907 (10MoritzMuehlenhoff) a:3MoritzMuehlenhoff [14:20:06] (03CR) 10Giuseppe Lavagetto: [C: 032] ganglia: move ganglia::collector to ganglia::deprecated::collector [puppet] - 10https://gerrit.wikimedia.org/r/225874 (owner: 10Giuseppe Lavagetto) [14:20:24] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.69% of data above the critical threshold [500.0] [14:22:53] Trey314159: around ? [14:23:50] matanya: here [14:24:09] hi Trey314159 I am setting up you access request, what is you full name? [14:24:14] *your [14:24:20] Trey Jones [14:25:30] (03PS2) 10Giuseppe Lavagetto: ganglia: remove ganglia::aggregator [puppet] - 10https://gerrit.wikimedia.org/r/225875 [14:25:52] (03PS1) 10Matanya: access: shell account for Trey Jones [puppet] - 10https://gerrit.wikimedia.org/r/226077 [14:25:56] Trey314159: ^ [14:26:10] (03CR) 10Giuseppe Lavagetto: [C: 032] ganglia: remove ganglia::aggregator [puppet] - 10https://gerrit.wikimedia.org/r/225875 (owner: 10Giuseppe Lavagetto) [14:26:16] (03PS1) 10Muehlenhoff: Add ferm rules for MX mail servers [puppet] - 10https://gerrit.wikimedia.org/r/226078 (https://phabricator.wikimedia.org/T104979) [14:27:09] matanya: cool—thanks! [14:27:21] sure :) [14:27:34] (03PS2) 10Giuseppe Lavagetto: ganglia: remove unused files [puppet] - 10https://gerrit.wikimedia.org/r/225876 [14:27:47] (03CR) 10Giuseppe Lavagetto: [C: 032] ganglia: remove unused files [puppet] - 10https://gerrit.wikimedia.org/r/225876 (owner: 10Giuseppe Lavagetto) [14:27:49] (03PS1) 10Mjbmr: Touch project domain templates to enable azb sumdomain [dns] - 10https://gerrit.wikimedia.org/r/226079 (https://phabricator.wikimedia.org/T106305) [14:28:40] Trey314159: do you know if you need private data as well? [14:29:25] I haven't hear that I will, but I don't know that I won't. I can ask... [14:30:15] (03PS1) 10Cmjohnson: Adding lvs1006-1012 mgmt dns entries [dns] - 10https://gerrit.wikimedia.org/r/226080 [14:31:08] (03CR) 10Gilles: "It doesn't seem like this was the source of the main problems on hhvm scalers, was it? If we do want correct length headers and disable mo" [puppet] - 10https://gerrit.wikimedia.org/r/222673 (owner: 10Ori.livneh) [14:31:21] (03PS2) 10Mjbmr: Touch project domain templates to enable azb subdomain [dns] - 10https://gerrit.wikimedia.org/r/226079 (https://phabricator.wikimedia.org/T106305) [14:31:23] please do Trey314159 it is two levels of access, one is analytics-users and the other is analytics-privatedata-users [14:31:45] otto or Wes should know [14:31:55] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [14:32:14] (03CR) 10John F. Lewis: [C: 031] Add ferm rules for MX mail servers [puppet] - 10https://gerrit.wikimedia.org/r/226078 (https://phabricator.wikimedia.org/T104979) (owner: 10Muehlenhoff) [14:32:33] 6operations, 10Continuous-Integration-Infrastructure, 6Multimedia, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1466954 (10brion) [14:32:50] 6operations, 10Continuous-Integration-Infrastructure, 6Multimedia, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1387291 (10brion) [14:32:54] 6operations, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: Support VP9 in TMH (Unable to decode) - https://phabricator.wikimedia.org/T55863#1466955 (10brion) [14:32:56] (03CR) 10Cmjohnson: [C: 032] Adding lvs1006-1012 mgmt dns entries [dns] - 10https://gerrit.wikimedia.org/r/226080 (owner: 10Cmjohnson) [14:33:03] (03PS2) 10Giuseppe Lavagetto: ganglia: remove references to ganglia::cname [puppet] - 10https://gerrit.wikimedia.org/r/225877 [14:33:17] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] ganglia: remove references to ganglia::cname [puppet] - 10https://gerrit.wikimedia.org/r/225877 (owner: 10Giuseppe Lavagetto) [14:37:34] 6operations: spare/unused disks on application servers - https://phabricator.wikimedia.org/T106381#1466972 (10fgiunchedi) 3NEW [14:37:46] (03CR) 10Smalyshev: "Still getting the same error when trying to run this on db01:" [puppet] - 10https://gerrit.wikimedia.org/r/223663 (owner: 10Smalyshev) [14:41:23] 6operations, 6Discovery, 7Elasticsearch: unattended elasticsearch restarts - https://phabricator.wikimedia.org/T89845#1466982 (10Manybubbles) [14:41:46] 6operations, 10CirrusSearch, 6Discovery: Release swift-repository for 1.7.0 - https://phabricator.wikimedia.org/T106163#1466986 (10Manybubbles) [14:42:33] 6operations, 10CirrusSearch, 6Discovery: Release wikimedia-extra plugin for Elasticsearch 1.7.0 - https://phabricator.wikimedia.org/T106161#1466991 (10Manybubbles) [14:42:39] 6operations, 10CirrusSearch, 6Discovery: Release experimental-highlighter for 1.7.0 - https://phabricator.wikimedia.org/T106162#1466993 (10Manybubbles) [14:42:43] (03CR) 10Tjones: [C: 031] access: shell account for Trey Jones [puppet] - 10https://gerrit.wikimedia.org/r/226077 (owner: 10Matanya) [14:43:56] 6operations, 10Wikimedia-Apache-configuration, 10Wikimedia-DNS, 7domains: Faulty DNS setup for wikipedia.is - https://phabricator.wikimedia.org/T103915#1466998 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi looks like this is completed ``` $ dig +short wikipedia.is ns ns2.wikimedia.org. ns0.wikimedia.... [14:47:59] 6operations, 10ops-eqiad: db1050 raid degraded - https://phabricator.wikimedia.org/T103110#1467002 (10fgiunchedi) p:5High>3Normal [14:52:24] 6operations: spare/unused disks on application servers - https://phabricator.wikimedia.org/T106381#1467004 (10fgiunchedi) [14:54:38] 6operations, 10ops-eqiad, 10Traffic: rack/setup new eqiad lvs machines - https://phabricator.wikimedia.org/T104458#1467014 (10Cmjohnson) Racked, Cabled, Racktables, Enabled on switch (private vlan). Mgmt dns has been completed but not production. ILO still needs configuration. +lvs1007 1H IN A 10... [14:56:45] Mjbmr: hi i'd prefer speaking in channels instead of private messages if possible. [14:56:47] (03PS13) 10Giuseppe Lavagetto: Add definitions for WDQS service [puppet] - 10https://gerrit.wikimedia.org/r/223663 (owner: 10Smalyshev) [14:57:00] jzerebecki: wmf/1.26wmf13 is no live. [14:57:33] Mjbmr: it is for the wikidata extensions/build [14:57:41] we didn't branch for 24 [14:57:44] err 14 [14:58:50] (03PS1) 10Muehlenhoff: Add ferm rules for syslog-ng [puppet] - 10https://gerrit.wikimedia.org/r/226084 [14:59:20] jouncebot: next [14:59:21] In 0 hour(s) and 0 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150721T1500) [14:59:30] so core 14 contains wikidata 13, which means because of automatic submodule updates either one needs to branch 14 for wikidata or scap 13 and 14 of core [15:00:04] manybubbles anomie ostriches thcipriani marktraceur Krenair: Dear anthropoid, the time has come. Please deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150721T1500). [15:00:04] gilles Mjbmr jzerebecki: A patch you scheduled for Morning SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [15:00:37] OK, I can SWAT today [15:01:21] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225935 (https://phabricator.wikimedia.org/T106323) (owner: 10Gilles) [15:01:29] (03Merged) 10jenkins-bot: Assign thumbnail access log to Monolog debug channel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225935 (https://phabricator.wikimedia.org/T106323) (owner: 10Gilles) [15:01:49] thcipriani: do you prefer to scap wmf13 and 14 or branch 14 of wikidata and only scap that? [15:02:35] (03PS2) 10Giuseppe Lavagetto: ganglia: remove ganglia_class conditionals [puppet] - 10https://gerrit.wikimedia.org/r/225878 [15:03:55] (03PS14) 10Smalyshev: Add definitions for WDQS service [puppet] - 10https://gerrit.wikimedia.org/r/223663 [15:04:04] jzerebecki: it's early, what are you referring to? [15:04:14] the wikibase extension? [15:04:19] thcipriani: yes [15:04:42] and no, i'm refering to the build that includes the wikibase extension [15:05:00] so the extension wikidata is that build [15:05:15] and it was intentionally not branched for 14 [15:05:40] jzerebecki: is that why Wikibase doesn't have 14 branch? [15:05:40] I don't get it: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wikiversions.json [15:05:41] ah, I see, so that's why I can't find Wikibase [15:06:13] that means if we merge into 14 of wikidata it auto updates the submodules of core 13 and 14 [15:06:42] Mjbmr: yes [15:07:05] gilles: I guess i should have pingged you before I merged your patch, are you ready for deploy your first patch? [15:07:10] Mjbmr: wikibase and wikidata don't have it because we intentionally skip every second wmf branch [15:07:17] thcipriani: yes [15:07:24] jzerebecki: Wikibase is used on all other projects, not only Wikidata. [15:07:47] Mjbmr: i'm referring to extensions/Wikidata not wikidata.org [15:08:24] jzerebecki: I'm saying doesn't Wikibase need a 14 branch? [15:08:42] !log thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Assign thumbnail access log to Monolog debug channel [[gerrit:225935]] (duration: 00m 13s) [15:08:46] (03PS15) 10Smalyshev: Add definitions for WDQS service [puppet] - 10https://gerrit.wikimedia.org/r/223663 [15:08:46] ^ gilles please check [15:08:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:09:41] Mjbmr: it is for the wikidata extensions/buildore branch 14 [15:09:52] thcipriani: it will only start doing something once the '14 patch is deployed as well [15:10:19] gilles: kk [15:10:41] matanya: The advice from the Discovery team is that I should get both analytics-users and analytics-privatedata-users access. Do you need anything else from me for that? [15:11:08] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226051 (https://phabricator.wikimedia.org/T65440) (owner: 10Gilles) [15:11:16] (03Merged) 10jenkins-bot: Offer 400px as a thumbnail size available in Special:Preferences [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226051 (https://phabricator.wikimedia.org/T65440) (owner: 10Gilles) [15:11:28] Mjbmr: no 13 of wikibase gets build into 13 of wikidata. that is a submodule of both 13 and 14 of core [15:11:59] 6operations, 10Beta-Cluster, 10MediaWiki-extensions-GettingStarted: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#1467024 (10fgiunchedi) indeed it looks like both beta redis are using aof persistence now, does still show up @mattflaschen ? [15:12:06] wikidata needs a 14 if we do not want to update both 13 and 14 of core, because of automatic submodule updates [15:12:46] sorry, but that's ridiculous [15:12:55] !log thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Offer 400px as a thumbnail size available in Special:Preferences [[gerrit:226051]] (duration: 00m 12s) [15:13:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:13:03] ^ gilles check please [15:13:59] Mjbmr: i didn't create that situation [15:14:05] thcipriani: works as expected [15:14:08] kk [15:15:18] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, 5Patch-For-Review: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1467035 (10Gilles) The change has just been deployed. 400px is now an option in Special:Preferences. [15:15:18] Mjbmr: anyway how do you suggest should the situation be improved? [15:16:10] jzerebecki: I don't suggest anything but Wikibase branches must act like any other extension. [15:17:42] 6operations, 7Monitoring: improve redis master/slave monitoring - https://phabricator.wikimedia.org/T101584#1467038 (10fgiunchedi) [15:17:42] Mjbmr: i'd like that and thus probposed https://www.mediawiki.org/wiki/Requests_for_comment/Streamlining_Composer_usage [15:19:05] (03PS16) 10Giuseppe Lavagetto: Add definitions for WDQS service [puppet] - 10https://gerrit.wikimedia.org/r/223663 (owner: 10Smalyshev) [15:19:15] (03CR) 10Giuseppe Lavagetto: [C: 032] Add definitions for WDQS service [puppet] - 10https://gerrit.wikimedia.org/r/223663 (owner: 10Smalyshev) [15:20:28] 10Ops-Access-Requests, 6operations, 6Discovery, 10SEO, 3Discovery-Analysis-Sprint: Get Oliver Keyes access to Google Webmaster Tools for all Wikimedia domains - https://phabricator.wikimedia.org/T101157#1467047 (10chasemp) a:5chasemp>3Deskana >>! In T101157#1441701, @Deskana wrote: > @chasemp I respo... [15:20:47] !log re-installing mw1090 [15:20:48] 6operations, 6Discovery, 10SEO, 3Discovery-Analysis-Sprint: Get Oliver Keyes access to Google Webmaster Tools for all Wikimedia domains - https://phabricator.wikimedia.org/T101157#1467049 (10chasemp) [15:20:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:20:59] jzerebecki: so, I'm still confused about what you're asking with 226021—wikibase is not currently a branched extension, like it's not in php-1.26wmf13 or php-1.26wmf14/extensions/Wikibase what are you trying to do in SWAT? [15:21:33] jzerebecki: extension/Wikidata is which is a build result that includes Wikibase [15:21:52] so i first need a +2 on wikibase and then can update wikidata [15:22:00] as i don't have deployment rights [15:22:24] ah, ok, got it [15:22:42] jzerebecki: according to Special:Version, this is the last commit: https://github.com/wikimedia/mediawiki-extensions-Wikidata/commit/c30a43d9b115758b28a5cce5a494368ba9b06445 [15:23:01] 6operations, 7HHVM: Custom session handler corrupted by session_destroy, "Failed to initialize storage module" - https://phabricator.wikimedia.org/T97675#1467050 (10Joe) a:3Joe [15:23:26] RECOVERY - Host mw1090 is UPING OK - Packet loss = 0%, RTA = 0.66 ms [15:24:15] jzerebecki: ah, got it, and wmf13 since it's a special branched extensions, OK, I'll +2 and then you'll make the wikidata update? [15:24:19] that is a bug: https://phabricator.wikimedia.org/T74759 [15:24:40] ^ I've noticed that behavior, didn't know the bug, though [15:24:46] thcipriani: ok then you scap php-1.26wmf13 and 14 after that? [15:25:09] does it require a full scap? Can't I sync-dir? [15:25:28] yes sync-dir but for both versions [15:25:34] ? [15:25:35] and also, you've probably explained this, why do you need wmf13? [15:25:53] submodule auto update changes both core branches [15:25:55] I mean, I'll sync-dir on php-1.26wmf14 for extensions wikidata [15:26:03] sure, ok [15:26:23] then you'd leave php-1.26wmf13 dirty [15:27:47] PROBLEM - configured eth on mw1090 is CRITICAL: Connection refused by host [15:27:56] PROBLEM - dhclient process on mw1090 is CRITICAL: Connection refused by host [15:27:57] PROBLEM - nutcracker port on mw1090 is CRITICAL: Connection refused by host [15:28:16] PROBLEM - nutcracker process on mw1090 is CRITICAL: Connection refused by host [15:28:25] !log thcipriani Synchronized php-1.26wmf14/thumb.php: SWAT: Thumbnail logging and stats part I [[gerrit:225936]] (duration: 00m 11s) [15:28:29] 6operations, 7HHVM: Custom session handler corrupted by session_destroy, "Failed to initialize storage module" - https://phabricator.wikimedia.org/T97675#1467056 (10fgiunchedi) p:5High>3Normal [15:28:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:28:37] PROBLEM - salt-minion processes on mw1090 is CRITICAL: Connection refused by host [15:29:16] PROBLEM - DPKG on mw1090 is CRITICAL: Connection refused by host [15:29:16] PROBLEM - RAID on mw1090 is CRITICAL: Connection refused by host [15:29:26] !log thcipriani Synchronized php-1.26wmf14/includes/filerepo/file/File.php: SWAT: Thumbnail logging and stats part II [[gerrit:225936]] (duration: 00m 13s) [15:29:26] PROBLEM - Disk space on mw1090 is CRITICAL: Connection refused by host [15:29:27] (03CR) 10Rush: "I did some of this outside of this change and would like to move the ssh bits to modules/phabricator/manifests/ssh.pp. This can be abando" [puppet] - 10https://gerrit.wikimedia.org/r/222987 (https://phabricator.wikimedia.org/T104827) (owner: 10Negative24) [15:29:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:29:35] ^ gilles check please [15:30:27] jzerebecki: +2'd [15:30:41] thx [15:32:25] thcipriani: not seeing anything yet [15:33:11] thcipriani: I never paid attention to that, is the commit shown on Special:Version supposed to update? or will it still be the branch cut one [15:33:27] gilles: dangit, no, I checked the fetch but didn't rebase, hang on [15:34:21] !log thcipriani Synchronized php-1.26wmf14/thumb.php: SWAT: Thumbnail logging and stats part I [[gerrit:225936]] (duration: 00m 12s) [15:34:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:34:41] !log thcipriani Synchronized php-1.26wmf14/includes/filerepo/file/File.php: SWAT: Thumbnail logging and stats part II [[gerrit:225936]] (duration: 00m 12s) [15:34:46] ok ,now it should be good [15:35:07] 6operations, 6Discovery, 10Wikidata, 10Wikidata-Query-Service, and 3 others: Wikidata Query Service hardware - https://phabricator.wikimedia.org/T86561#1467071 (10Cmjohnson) [15:35:10] 6operations, 10ops-eqiad: wmf3543 can't install from PXE - https://phabricator.wikimedia.org/T106320#1467068 (10Cmjohnson) 5Open>3Resolved a:3Cmjohnson The boot process was set to icsi not pxe. I fixed this and it should work now. I also changed this on wmf3544 (backup) [15:35:56] Mjbmr: looks like you have a -1 on 225214 [15:35:58] thcipriani: looking great, the data is pouring in. thank you! [15:36:34] I don't think Siebrand will remove it soon [15:36:44] 6operations, 10ops-eqiad: nas1001-a environmental failure - https://phabricator.wikimedia.org/T102955#1467081 (10Cmjohnson) 5Open>3declined a:3Cmjohnson The power supplies cables were fine. If it's the PSU, it will have to stay in it's current state. The NETAPP is no longer under a support contract. [15:37:26] !log cleanup ganglia temp files on uranium [15:37:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:38:23] Mjbmr: is 225828 going to be a full scap? [15:38:51] well, it'd be good if both were done for full scap [15:39:12] 6operations: stray ganglia-graph files left in /tmp - https://phabricator.wikimedia.org/T97637#1467085 (10fgiunchedi) indeed there's still temp files (~10/day) back from May on uranium, I'm going to prepare a puppet fix ``` uranium:~$ ls -lat /tmp/ | head -10 total 265976 drwxrwxrwt 2 root root 32358... [15:39:28] thcipriani: you're adding a wiki in swat? might take too long https://wikitech.wikimedia.org/wiki/Add_a_wiki [15:39:52] no, not a wiki. [15:41:11] thcipriani: build result: https://gerrit.wikimedia.org/r/#/c/226086/1 [15:42:44] jzerebecki: kk, I'll +2 that and get it out the door for 14 [15:45:32] Mjbmr: I don't feel comfortable scapping if there's a -1 on a patch that I don't have a good handle on, I can push your 2nd patch out, though [15:45:45] ok [15:45:59] Mjbmr: 2nd one need a full scap? [15:46:07] yes. [15:46:11] (03PS1) 10Filippo Giunchedi: ganglia_new: cleanup old temporary graphs [puppet] - 10https://gerrit.wikimedia.org/r/226087 (https://phabricator.wikimedia.org/T97637) [15:46:21] (03PS3) 10Giuseppe Lavagetto: ganglia: remove ganglia_class conditionals [puppet] - 10https://gerrit.wikimedia.org/r/225878 [15:46:51] Mjbmr: kk, I'll finish up wikidata and then do that [15:47:02] Thanks. [15:47:36] (03CR) 10Ottomata: "I forget, does ferm work in labs?" [puppet] - 10https://gerrit.wikimedia.org/r/226071 (owner: 10Muehlenhoff) [15:47:49] 6operations, 5Patch-For-Review: stray ganglia-graph files left in /tmp - https://phabricator.wikimedia.org/T97637#1467095 (10fgiunchedi) a:3fgiunchedi [15:49:12] 6operations, 10ops-eqiad, 10Incident-20150401-LabsNFS-Overload: Inspect and diagnose labstore1001's H800 controler - https://phabricator.wikimedia.org/T95293#1467097 (10fgiunchedi) did this happen? [15:50:37] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.69% of data above the critical threshold [500.0] [15:51:46] (03PS3) 10Rush: Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226031 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [15:51:56] (03CR) 10Rush: [C: 032 V: 032] Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226031 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [15:52:11] 7Puppet, 6Labs, 6Phabricator, 5Patch-For-Review: On labs phabricator references security extension even though it isn't present - https://phabricator.wikimedia.org/T104904#1467102 (10chasemp) 5Open>3Resolved [15:52:35] <_joe_> argh [15:52:38] <_joe_> rebase race [15:52:42] (03PS4) 10Giuseppe Lavagetto: ganglia: remove ganglia_class conditionals [puppet] - 10https://gerrit.wikimedia.org/r/225878 [15:52:44] 6operations, 10Beta-Cluster, 6Labs, 7Monitoring: Setup (simple) catchpoint monitoring and metrics for enwiki betacluster just like production - https://phabricator.wikimedia.org/T97865#1467105 (10greg) [15:53:06] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] "verified it's a noop with the compiler" [puppet] - 10https://gerrit.wikimedia.org/r/225878 (owner: 10Giuseppe Lavagetto) [15:53:54] (03CR) 10DCausse: "I think we use the default elasticsearch port configuration which is a port range [9200-9300] for http and [9300-9400] for transport. ES w" [puppet] - 10https://gerrit.wikimedia.org/r/224095 (https://phabricator.wikimedia.org/T104962) (owner: 10Muehlenhoff) [15:55:05] (03PS1) 10Rush: Revert "Ensure that phabricator/src/extensions exists" [puppet] - 10https://gerrit.wikimedia.org/r/226090 [15:55:53] (03CR) 10Rush: "change is bad actually but I don't have time to debug atm https://gerrit.wikimedia.org/r/#/c/226090/ so I am reverting" [puppet] - 10https://gerrit.wikimedia.org/r/226031 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [15:56:02] (03CR) 10Rush: [C: 032] Revert "Ensure that phabricator/src/extensions exists" [puppet] - 10https://gerrit.wikimedia.org/r/226090 (owner: 10Rush) [15:56:32] are submodules not autoupdating still? [15:57:08] PROBLEM - puppet last run on iridium is CRITICAL puppet fail [15:57:23] (03CR) 10Rush: "it may already have been in an error state?" [puppet] - 10https://gerrit.wikimedia.org/r/226031 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [15:57:34] jzerebecki: php-1.26wmf14 doesn't seem to notice that wikidata wmf/1.26wmf13 updated [15:57:55] can you do a submodule bump on core? Or do you need me to? [15:58:04] php-1.26wmf13 was auto updated [15:58:08] can do [15:58:19] 6operations, 7Monitoring: Refactor RAID checks (check-raid) - https://phabricator.wikimedia.org/T84050#1467112 (10fgiunchedi) I did ask the same question to Jeff without remembering this ticket, anyways for reference I'm attaching it {F203780} [15:58:21] kk [16:01:28] RECOVERY - Apache HTTP on mw1090 is OK: HTTP OK: HTTP/1.1 200 OK - 11783 bytes in 0.022 second response time [16:01:41] !log thcipriani Synchronized php-1.26wmf13/extensions/Wikidata: SWAT: Update Wikibase: Add api featureLog for ungroupedlist param [[gerrit:226086]] (duration: 00m 20s) [16:01:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:03:55] jzerebecki: lemme know the bump patch when you get it [16:05:35] thcipriani: https://gerrit.wikimedia.org/r/#/c/226091/ [16:05:52] 6operations, 10Wikimedia-Apache-configuration, 10Wikimedia-DNS, 7domains: Faulty DNS setup for wikipedia.is - https://phabricator.wikimedia.org/T103915#1467188 (10Slaporte) The registrar confirmed that this is resolved. Thanks! [16:08:21] (03PS2) 10Giuseppe Lavagetto: ganglia: remove ganglia.pp [puppet] - 10https://gerrit.wikimedia.org/r/225879 [16:13:34] (03CR) 10Giuseppe Lavagetto: "noop again" [puppet] - 10https://gerrit.wikimedia.org/r/225879 (owner: 10Giuseppe Lavagetto) [16:13:43] (03CR) 10Giuseppe Lavagetto: [C: 032] "noop again" [puppet] - 10https://gerrit.wikimedia.org/r/225879 (owner: 10Giuseppe Lavagetto) [16:14:29] <_joe_> ganglia.pp is no more, another one down [16:15:07] <_joe_> ouch I might have missed an inclusion somewhere [16:15:07] RECOVERY - salt-minion processes on mw1090 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:15:36] RECOVERY - DPKG on mw1090 is OK: All packages OK [16:15:37] RECOVERY - RAID on mw1090 is OK no RAID installed [16:15:46] RECOVERY - Disk space on mw1090 is OK: DISK OK [16:15:51] <_joe_> expect a shower of puppet failures [16:16:07] RECOVERY - configured eth on mw1090 is OK - interfaces up [16:16:08] <_joe_> Fixing those anyways [16:16:17] RECOVERY - dhclient process on mw1090 is OK: PROCS OK: 0 processes with command name dhclient [16:16:26] RECOVERY - nutcracker port on mw1090 is OK: TCP OK - 0.000 second response time on port 11212 [16:16:37] RECOVERY - nutcracker process on mw1090 is OK: PROCS OK: 1 process with UID = 109 (nutcracker), command name nutcracker [16:16:46] RECOVERY - HHVM processes on mw1090 is OK: PROCS OK: 6 processes with command name hhvm [16:18:27] PROBLEM - puppet last run on palladium is CRITICAL puppet fail [16:18:36] PROBLEM - puppet last run on mw2212 is CRITICAL puppet fail [16:18:37] PROBLEM - puppet last run on mw1003 is CRITICAL puppet fail [16:18:37] PROBLEM - puppet last run on mw1066 is CRITICAL puppet fail [16:18:57] PROBLEM - puppet last run on mw2176 is CRITICAL puppet fail [16:18:57] PROBLEM - puppet last run on mw2127 is CRITICAL puppet fail [16:19:07] PROBLEM - puppet last run on mw1189 is CRITICAL puppet fail [16:19:08] PROBLEM - puppet last run on mw2113 is CRITICAL puppet fail [16:19:27] PROBLEM - puppet last run on mw2096 is CRITICAL puppet fail [16:19:27] PROBLEM - puppet last run on mw2070 is CRITICAL puppet fail [16:19:27] PROBLEM - puppet last run on logstash1002 is CRITICAL puppet fail [16:19:28] PROBLEM - puppet last run on mw1155 is CRITICAL puppet fail [16:19:36] PROBLEM - puppet last run on mw1166 is CRITICAL puppet fail [16:19:37] PROBLEM - puppet last run on mw2196 is CRITICAL puppet fail [16:19:37] PROBLEM - puppet last run on mw2083 is CRITICAL puppet fail [16:19:37] PROBLEM - puppet last run on mw2019 is CRITICAL puppet fail [16:19:57] PROBLEM - puppet last run on mw2093 is CRITICAL puppet fail [16:19:57] PROBLEM - puppet last run on mw2047 is CRITICAL puppet fail [16:19:57] PROBLEM - puppet last run on mw2067 is CRITICAL puppet fail [16:19:57] PROBLEM - puppet last run on mw2003 is CRITICAL puppet fail [16:20:07] PROBLEM - puppet last run on mw1011 is CRITICAL puppet fail [16:20:07] PROBLEM - puppet last run on mw1104 is CRITICAL puppet fail [16:20:07] PROBLEM - puppet last run on mw1235 is CRITICAL puppet fail [16:20:07] PROBLEM - puppet last run on mw1131 is CRITICAL puppet fail [16:20:10] (03PS1) 10Giuseppe Lavagetto: ganglia: re-define generic ganglia class [puppet] - 10https://gerrit.wikimedia.org/r/226094 [16:20:16] PROBLEM - puppet last run on mw2079 is CRITICAL puppet fail [16:20:16] PROBLEM - puppet last run on mw2110 is CRITICAL puppet fail [16:20:27] PROBLEM - puppet last run on mw2092 is CRITICAL puppet fail [16:20:27] PROBLEM - puppet last run on mw1211 is CRITICAL puppet fail [16:20:27] PROBLEM - puppet last run on mw1128 is CRITICAL puppet fail [16:20:27] PROBLEM - puppet last run on labcontrol1001 is CRITICAL puppet fail [16:20:27] PROBLEM - puppet last run on mw1213 is CRITICAL puppet fail [16:20:27] PROBLEM - puppet last run on mw1118 is CRITICAL puppet fail [16:20:32] jzerebecki: just now syncing wikidata 14 [16:20:37] PROBLEM - puppet last run on mw1137 is CRITICAL puppet fail [16:20:37] PROBLEM - puppet last run on mw2134 is CRITICAL puppet fail [16:20:37] PROBLEM - puppet last run on mw2049 is CRITICAL puppet fail [16:20:37] PROBLEM - puppet last run on mw1194 is CRITICAL puppet fail [16:20:37] (03CR) 10Giuseppe Lavagetto: [C: 032] ganglia: re-define generic ganglia class [puppet] - 10https://gerrit.wikimedia.org/r/226094 (owner: 10Giuseppe Lavagetto) [16:20:38] PROBLEM - puppet last run on mw1021 is CRITICAL puppet fail [16:20:41] !log thcipriani Synchronized php-1.26wmf14/extensions/Wikidata: SWAT: Update Wikibase: Add api featureLog for ungroupedlist param [[gerrit:226086]] (duration: 00m 20s) [16:20:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:20:46] PROBLEM - puppet last run on mw1092 is CRITICAL puppet fail [16:20:46] ^ jzerebecki check please [16:20:46] PROBLEM - puppet last run on mw2143 is CRITICAL puppet fail [16:20:46] PROBLEM - puppet last run on mw2184 is CRITICAL puppet fail [16:20:46] PROBLEM - puppet last run on mw2123 is CRITICAL puppet fail [16:20:47] PROBLEM - puppet last run on mw1129 is CRITICAL puppet fail [16:20:56] PROBLEM - puppet last run on mw2030 is CRITICAL puppet fail [16:20:57] PROBLEM - puppet last run on mw2055 is CRITICAL puppet fail [16:20:57] PROBLEM - puppet last run on mw2090 is CRITICAL puppet fail [16:20:57] PROBLEM - puppet last run on mw1253 is CRITICAL puppet fail [16:20:57] PROBLEM - puppet last run on etherpad1001 is CRITICAL puppet fail [16:21:02] (03PS1) 10Ottomata: Using analytics-flex for new Hadoop worker nodes analytics1042-1045 [puppet] - 10https://gerrit.wikimedia.org/r/226095 [16:21:07] PROBLEM - puppet last run on mw1025 is CRITICAL puppet fail [16:21:07] PROBLEM - puppet last run on mw1206 is CRITICAL puppet fail [16:21:16] PROBLEM - puppet last run on mw1154 is CRITICAL puppet fail [16:21:17] PROBLEM - puppet last run on mw1208 is CRITICAL puppet fail [16:21:17] PROBLEM - puppet last run on antimony is CRITICAL puppet fail [16:21:18] PROBLEM - puppet last run on mw1047 is CRITICAL puppet fail [16:21:27] PROBLEM - puppet last run on mw2084 is CRITICAL puppet fail [16:21:27] PROBLEM - puppet last run on mw2039 is CRITICAL puppet fail [16:21:27] PROBLEM - puppet last run on mw2203 is CRITICAL puppet fail [16:21:28] PROBLEM - puppet last run on mw1054 is CRITICAL puppet fail [16:21:28] <_joe_> ottomata: please don't merge [16:21:33] (03PS2) 10Ottomata: Using analytics-flex for new Hadoop worker nodes analytics1042-1045 [puppet] - 10https://gerrit.wikimedia.org/r/226095 [16:21:35] _joe_: ok [16:21:37] PROBLEM - puppet last run on mw2142 is CRITICAL puppet fail [16:21:37] PROBLEM - puppet last run on mw2130 is CRITICAL puppet fail [16:21:37] PROBLEM - puppet last run on mw2101 is CRITICAL puppet fail [16:21:47] PROBLEM - puppet last run on mw2111 is CRITICAL puppet fail [16:21:47] PROBLEM - puppet last run on mw1049 is CRITICAL puppet fail [16:21:57] PROBLEM - puppet last run on mw2085 is CRITICAL puppet fail [16:21:58] PROBLEM - puppet last run on mw2182 is CRITICAL puppet fail [16:21:58] PROBLEM - puppet last run on mw2062 is CRITICAL puppet fail [16:22:07] PROBLEM - puppet last run on mw2172 is CRITICAL puppet fail [16:22:07] PROBLEM - puppet last run on mw1084 is CRITICAL puppet fail [16:22:08] thcipriani: works. thx. [16:22:15] jzerebecki: cool [16:22:16] PROBLEM - puppet last run on mw2168 is CRITICAL puppet fail [16:22:17] PROBLEM - puppet last run on mw1180 is CRITICAL puppet fail [16:22:17] PROBLEM - puppet last run on mw1020 is CRITICAL puppet fail [16:22:17] PROBLEM - puppet last run on mw2131 is CRITICAL puppet fail [16:22:17] PROBLEM - puppet last run on mw2056 is CRITICAL puppet fail [16:22:17] PROBLEM - puppet last run on mira is CRITICAL puppet fail [16:22:22] Mjbmr: still around for scap of yours? [16:22:33] thcipriani: yeah [16:22:36] PROBLEM - puppet last run on mw1094 is CRITICAL puppet fail [16:22:36] PROBLEM - puppet last run on mw1238 is CRITICAL puppet fail [16:22:38] PROBLEM - puppet last run on bromine is CRITICAL puppet fail [16:22:46] PROBLEM - puppet last run on mw1075 is CRITICAL puppet fail [16:22:47] PROBLEM - puppet last run on mw2098 is CRITICAL puppet fail [16:22:47] PROBLEM - puppet last run on mw1078 is CRITICAL puppet fail [16:22:47] PROBLEM - puppet last run on mw1230 is CRITICAL puppet fail [16:22:53] _joe_: lemme know when I can, no hurry [16:22:57] PROBLEM - puppet last run on mw1179 is CRITICAL puppet fail [16:23:05] <_joe_> ottomata: in a sec I can confirm, I hope [16:23:07] PROBLEM - puppet last run on mw1169 is CRITICAL puppet fail [16:23:16] PROBLEM - puppet last run on mw2200 is CRITICAL puppet fail [16:23:17] PROBLEM - puppet last run on mw1083 is CRITICAL puppet fail [16:23:28] PROBLEM - puppet last run on mw1030 is CRITICAL puppet fail [16:23:28] PROBLEM - puppet last run on mw2152 is CRITICAL puppet fail [16:23:36] Mjbmr: can you make a submodule bump on core for that one? Looks like 14 is not autoupdating [16:23:36] PROBLEM - puppet last run on argon is CRITICAL puppet fail [16:23:37] PROBLEM - puppet last run on mw2094 is CRITICAL puppet fail [16:23:37] PROBLEM - puppet last run on mw2048 is CRITICAL puppet fail [16:23:37] PROBLEM - puppet last run on mw2174 is CRITICAL puppet fail [16:23:37] PROBLEM - puppet last run on hafnium is CRITICAL puppet fail [16:23:40] <_joe_> ottomata: it's ok [16:23:46] PROBLEM - puppet last run on mw2150 is CRITICAL puppet fail [16:23:47] PROBLEM - puppet last run on stat1001 is CRITICAL puppet fail [16:23:47] PROBLEM - puppet last run on mw2107 is CRITICAL puppet fail [16:23:47] PROBLEM - puppet last run on mw1138 is CRITICAL puppet fail [16:23:48] PROBLEM - puppet last run on sodium is CRITICAL puppet fail [16:23:56] PROBLEM - puppet last run on mw1136 is CRITICAL puppet fail [16:23:56] PROBLEM - puppet last run on mw2202 is CRITICAL puppet fail [16:23:57] PROBLEM - puppet last run on mw2133 is CRITICAL puppet fail [16:24:10] <_joe_> uhm [16:24:17] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: tjones needs access to stat1002 - https://phabricator.wikimedia.org/T106175#1467324 (10fgiunchedi) >>! In T106175#1465782, @TJones wrote: > Signed https://phabricator.wikimedia.org/L3 > wikitech profile: https://wikitech.wikimedia.org/wiki/User:Tjones >... [16:24:17] PROBLEM - puppet last run on mw2027 is CRITICAL puppet fail [16:24:18] PROBLEM - puppet last run on mw2106 is CRITICAL puppet fail [16:24:18] PROBLEM - puppet last run on mw2046 is CRITICAL puppet fail [16:24:27] PROBLEM - puppet last run on mw1062 is CRITICAL puppet fail [16:24:27] RECOVERY - puppet last run on palladium is OK Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:24:37] PROBLEM - puppet last run on mw1132 is CRITICAL puppet fail [16:24:37] PROBLEM - puppet last run on mw1116 is CRITICAL puppet fail [16:24:47] PROBLEM - puppet last run on mw1165 is CRITICAL puppet fail [16:24:47] PROBLEM - puppet last run on mw2053 is CRITICAL puppet fail [16:24:47] PROBLEM - puppet last run on mw1240 is CRITICAL puppet fail [16:24:47] PROBLEM - puppet last run on mw1181 is CRITICAL puppet fail [16:24:53] (03CR) 10Filippo Giunchedi: [C: 04-1] "same key as labs, see https://phabricator.wikimedia.org/T106175" [puppet] - 10https://gerrit.wikimedia.org/r/226077 (owner: 10Matanya) [16:25:07] PROBLEM - puppet last run on mw1053 is CRITICAL puppet fail [16:25:07] PROBLEM - puppet last run on mw1198 is CRITICAL puppet fail [16:25:17] PROBLEM - puppet last run on mw1134 is CRITICAL puppet fail [16:25:17] PROBLEM - puppet last run on mw1040 is CRITICAL puppet fail [16:25:27] RECOVERY - puppet last run on iridium is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [16:25:27] PROBLEM - puppet last run on mw2167 is CRITICAL puppet fail [16:25:36] PROBLEM - puppet last run on mw1218 is CRITICAL puppet fail [16:25:37] PROBLEM - puppet last run on mw1074 is CRITICAL puppet fail [16:25:37] PROBLEM - puppet last run on mw2058 is CRITICAL puppet fail [16:25:46] PROBLEM - puppet last run on mw1178 is CRITICAL puppet fail [16:25:46] PROBLEM - puppet last run on krypton is CRITICAL puppet fail [16:25:46] 6operations, 10MediaWiki-Database: Compress data at external storage - https://phabricator.wikimedia.org/T106386#1467339 (10jcrespo) 3NEW [16:26:46] (03PS3) 10Ottomata: Using analytics-flex for new Hadoop worker nodes analytics1042-1045 [puppet] - 10https://gerrit.wikimedia.org/r/226095 [16:26:46] <_joe_> and sorry everyone for the spam [16:26:52] <_joe_> but it's fixed now [16:26:53] 6operations, 10MediaWiki-Database: Compress data at external storage - https://phabricator.wikimedia.org/T106386#1467358 (10jcrespo) [16:27:16] thcipriani: maybe let the patch done for now, will do the full scap when other patch is approved. [16:27:29] the problem must be fixed tho. [16:27:35] 6operations, 7Database: new external storage cluster(s) - https://phabricator.wikimedia.org/T105843#1452813 (10jcrespo) [16:27:37] 6operations, 10MediaWiki-Database: Compress data at external storage - https://phabricator.wikimedia.org/T106386#1467339 (10jcrespo) [16:27:39] Mjbmr: ok [16:27:48] 6operations, 6Release-Engineering, 7Database: Re-compress External Storage in production using trackBlobs.php and recompressTracked.php - https://phabricator.wikimedia.org/T106387#1467370 (10Jdforrester-WMF) 3NEW [16:27:49] SWAT is complete then! [16:28:00] Thanks! [16:28:07] 6operations, 6Release-Engineering, 7Database: Audit all existing code to ensure that any extension currently or previously adding blobs to ES has been registering a reference in the text table - https://phabricator.wikimedia.org/T106388#1467385 (10Jdforrester-WMF) 3NEW [16:28:16] 6operations, 6Release-Engineering, 7Database: Audit all existing code to ensure that any extension currently or previously adding blobs to ES has been registering a reference in the text table - https://phabricator.wikimedia.org/T106388#1467385 (10Jdforrester-WMF) [16:28:18] 6operations, 6Release-Engineering, 7Database: Re-compress External Storage in production using trackBlobs.php and recompressTracked.php - https://phabricator.wikimedia.org/T106387#1467392 (10Jdforrester-WMF) [16:28:28] 6operations, 6Release-Engineering, 7Database: Audit all existing code to ensure that any extension currently or previously adding blobs to ES has been registering a reference in the text table (and fix up if wrong) - https://phabricator.wikimedia.org/T106388#1467385 (10Jdforrester-WMF) [16:28:35] 6operations, 6Release-Engineering, 7Database: Audit all existing code to ensure that any extension currently or previously adding blobs to ES has been registering a reference in the text table (and fix up if wrong) - https://phabricator.wikimedia.org/T106388#1467385 (10Jdforrester-WMF) [16:28:54] (03PS1) 1020after4: Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) [16:29:22] (03CR) 1020after4: "trying this again..." [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [16:30:06] (03PS1) 10Cmjohnson: Adding mw1090 back to dsh group [puppet] - 10https://gerrit.wikimedia.org/r/226098 [16:30:10] (03CR) 1020after4: "previous related (failed & reverted) patch:" [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [16:31:14] (03CR) 10Ottomata: [C: 032] Using analytics-flex for new Hadoop worker nodes analytics1042-1045 [puppet] - 10https://gerrit.wikimedia.org/r/226095 (owner: 10Ottomata) [16:31:43] (03PS2) 10Cmjohnson: Adding mw1090 back to dsh group [puppet] - 10https://gerrit.wikimedia.org/r/226098 [16:32:01] 6operations, 6Release-Engineering, 7Database: Audit all existing code to ensure that any extension currently or previously adding blobs to ES has been registering a reference in the text table (and fix up if wrong) - https://phabricator.wikimedia.org/T106388#1467446 (10Jdforrester-WMF) [16:32:06] 6operations, 10MediaWiki-Database: Compress data at external storage - https://phabricator.wikimedia.org/T106386#1467445 (10Jdforrester-WMF) [16:32:29] 6operations, 6Release-Engineering, 7Database: Re-compress External Storage in production using trackBlobs.php and recompressTracked.php - https://phabricator.wikimedia.org/T106387#1467452 (10Jdforrester-WMF) [16:32:31] 6operations, 10MediaWiki-Database: Compress data at external storage - https://phabricator.wikimedia.org/T106386#1467339 (10Jdforrester-WMF) [16:34:59] (03CR) 10Cmjohnson: [C: 032] Adding mw1090 back to dsh group [puppet] - 10https://gerrit.wikimedia.org/r/226098 (owner: 10Cmjohnson) [16:35:05] 6operations, 5codfw-appserver-setup, 5wikis-in-codfw: install/deploy codfw appservers - https://phabricator.wikimedia.org/T85227#1467467 (10fgiunchedi) anything left to be done here? afaik all appservers in codfw might be up now [16:35:51] 6operations, 10ops-eqiad, 5Patch-For-Review: mw1090 has a read-only filesystem - https://phabricator.wikimedia.org/T105835#1467478 (10Cmjohnson) Swapped the disk with a new one, reinstalled and added puppet certs. Added back to pybal and dsh group https://gerrit.wikimedia.org/r/#/c/226098/ [16:36:00] 6operations, 10ops-eqiad, 5Patch-For-Review: mw1090 has a read-only filesystem - https://phabricator.wikimedia.org/T105835#1467479 (10Cmjohnson) 5Open>3Resolved [16:36:11] 6operations, 7Database: new external storage cluster(s) - https://phabricator.wikimedia.org/T105843#1467480 (10jcrespo) [16:36:25] (03PS2) 10Giuseppe Lavagetto: ganglia: standardize has_ganglia [puppet] - 10https://gerrit.wikimedia.org/r/225880 [16:36:41] 6operations, 7Database: new external storage cluster(s) - https://phabricator.wikimedia.org/T105843#1452813 (10jcrespo) Removing "blocked by" as this will be done first. [16:39:29] (03PS3) 10Giuseppe Lavagetto: ganglia: standardize has_ganglia [puppet] - 10https://gerrit.wikimedia.org/r/225880 [16:39:44] _joe_: btw since you're also pruning ganglia, I have https://gerrit.wikimedia.org/r/#/c/226087/ lined up [16:40:12] 6operations, 10MediaWiki-Database: Compress data at external storage - https://phabricator.wikimedia.org/T106386#1467504 (10jcrespo) [16:42:39] thcipriani: is this it https://github.com/wikimedia/mediawiki/commits/wmf/1.26wmf14 [16:43:47] Mjbmr: oh, I guess it did update core... [16:45:27] locked db at it.wiki [16:45:43] now it's just lagged [16:46:02] [rollback di 0 modifiche] <-- lol [16:46:17] (03CR) 1020after4: "chase: I tested this on labs phab-pup + phab-01 and it seems to work this time." [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [16:46:23] (03PS2) 1020after4: Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) [16:46:31] (03CR) 1020after4: [C: 031] Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [16:47:44] cmjohnson1: hiya! thanks for racking those nodes. Do you remember which NIC MAC I should assign? [16:47:55] NIC.Integrated.1-3-1 Ethernet = 44:A8:42:24:92:37 [16:47:55] NIC.Integrated.1-4-1 Ethernet = 44:A8:42:24:92:38 [16:47:55] NIC.Integrated.1-1-1 Ethernet = 44:A8:42:24:92:35 [16:47:56] NIC.Integrated.1-2-1 Ethernet = 44:A8:42:24:92:36 [16:48:01] 1-1-1 [16:48:04] great danke [16:48:10] yw [16:48:49] cmjohnson1: what do you think about moving the 3 ciscos we have to row D [16:48:57] to make space for other nodes [16:49:27] these are the ciscos in row B..correcT? [16:49:43] 6operations, 6Multimedia, 6Performance-Team, 10Wikimedia-Site-requests, and 2 others: Please offer larger image thumbnail sizes in Special:Preferences - https://phabricator.wikimedia.org/T65440#1467541 (10Glaisher) [16:50:06] checking [16:50:06] i think so [16:50:28] 1003/4/7 and 1010 are in B3 [16:50:29] (03PS1) 10Ori.livneh: ops: +aaron [puppet] - 10https://gerrit.wikimedia.org/r/226101 (https://phabricator.wikimedia.org/T106051) [16:50:36] godog: ^ [16:50:37] cmjohnson1: yes [16:50:54] B3 [16:51:14] woudl that only free up space for 3? [16:51:29] i remember the ciscos being huge, but maybe they are the same size as the dells [16:51:37] hrm..IDK, lemme talk with Rob about that first. [16:51:47] they're 2u just like the Dells [16:52:00] ok, maybe there are is a slot or two in B3 free anyway? at least it looks like it in RackTables [16:52:13] yeah that slot is only 1u [16:52:42] I have some db's going away in row A....I also need to talk to Andrew B about moving or decom'ing virts [16:52:49] the cisco virts [16:53:08] the goal is start replacing and removing them [16:54:10] aye, we are using these 3 still, but not for production purposes. we use them to evaluate streaming techs and eventlogging staging. if/when we want to actually set up a production streaming services we will allocate new nodes for it i thikn [16:54:16] but we do use our 3 ciscos, so I"d like to keep them for now [16:54:17] 6operations, 6Discovery, 6Security, 7Elasticsearch: wait longer in es-tool before enabling replication - https://phabricator.wikimedia.org/T99500#1467562 (10demon) [16:54:55] 6operations, 6Discovery, 6Security, 7Elasticsearch: Update Logstash Elasticsearch to 1.3.8+ - https://phabricator.wikimedia.org/T92854#1467568 (10demon) [16:54:57] 6operations, 6Discovery, 6Security, 7Elasticsearch: Upgrade CirrusSearch's Elasticsearch cluster to 1.3.8+ - https://phabricator.wikimedia.org/T92853#1467571 (10demon) [16:55:38] ottomata: okay, will get back to you about them [16:55:52] <_joe_> Coren: did you ban icinga-wm or it's just dumb and not rejoining? [16:55:59] 6operations, 6Discovery, 6Security, 7Elasticsearch: Upgrade all ElasticSearch clusters to 1.3.8+ (RCE vulnerability) - https://phabricator.wikimedia.org/T92770#1467597 (10demon) [16:56:14] _joe_: I think it's just being dumb - I only kicked it. [16:56:39] <_joe_> ok, restarting it then [16:57:35] (03CR) 10Filippo Giunchedi: [C: 031] ops: +aaron [puppet] - 10https://gerrit.wikimedia.org/r/226101 (https://phabricator.wikimedia.org/T106051) (owner: 10Ori.livneh) [16:57:50] godog: could you merge? [16:58:30] 6operations, 7Database: new external storage cluster(s) - https://phabricator.wikimedia.org/T105843#1467615 (10jcrespo) From the high level point of view, there are 2 main options here: * Keeping the old servers, which have no warranty and right now no replacements (see for example T103843). Buy only a batch... [16:58:35] 10Ops-Access-Requests, 6operations: Access request to stat1002 for dcausse - https://phabricator.wikimedia.org/T106370#1467619 (10Tfinc) Approved [16:59:10] 6operations, 10RESTBase-Cassandra: setup an alertable threshold for Cassandra heap dumps - https://phabricator.wikimedia.org/T106346#1467624 (10Eevans) >>! In T106346#1466499, @fgiunchedi wrote: > indeed they can add up quickly, what about trimming their size periodically but leave some behind? the rationale b... [16:59:21] (03PS1) 10Ottomata: Add host entries for analytics1042-analytics1045 [puppet] - 10https://gerrit.wikimedia.org/r/226102 (https://phabricator.wikimedia.org/T104463) [16:59:54] godog: I'd prefer if you (or someone else from ops) merged that [17:01:07] (03CR) 10Ottomata: [C: 032] Add host entries for analytics1042-analytics1045 [puppet] - 10https://gerrit.wikimedia.org/r/226102 (https://phabricator.wikimedia.org/T104463) (owner: 10Ottomata) [17:01:38] Mjbmr: does this mean we ought to revert that submodule bump? [17:01:41] ori: sure, I'd like another +1 so it isn't a one man show :) [17:01:49] chasemp: could you? [17:02:02] <_joe_> godog: is the 3 day period over? [17:02:09] eating lunch what did I miss [17:02:15] not sure which patch you mean [17:02:17] <_joe_> I'd like to allow people coming back from wikimania to comment if they whish [17:02:23] https://gerrit.wikimedia.org/r/#/c/226101/ [17:02:33] chase is coming back from wikimania ;) [17:02:34] 6operations, 10hardware-requests, 7Database: new external storage cluster(s) - https://phabricator.wikimedia.org/T105843#1467639 (10jcrespo) [17:02:36] thcipriani: no, it's ok. [17:02:51] kk [17:03:05] _joe_: technically it is over yeah [17:03:56] JFDI [17:04:10] (03CR) 10Giuseppe Lavagetto: [C: 031] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/226101 (https://phabricator.wikimedia.org/T106051) (owner: 10Ori.livneh) [17:04:52] (03PS2) 10Filippo Giunchedi: ops: +aaron [puppet] - 10https://gerrit.wikimedia.org/r/226101 (https://phabricator.wikimedia.org/T106051) (owner: 10Ori.livneh) [17:05:00] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] ops: +aaron [puppet] - 10https://gerrit.wikimedia.org/r/226101 (https://phabricator.wikimedia.org/T106051) (owner: 10Ori.livneh) [17:06:05] thanks guys [17:08:07] ori: is it time to start chanting 'rm -rf /' to AaronSchulz? :) [17:08:20] <_joe_> !log restarted logmsgbot, ircecho on neon [17:08:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:09:22] JohnFLewis: sadly that doesn't work anymore on recent coreutils :( [17:09:36] ori: np [17:09:39] <_joe_> no I think it's time to chant "improve your opsec" :) [17:09:47] godog: really? why do people have to take the fun out of things :( [17:13:41] 6operations, 5Patch-For-Review: Allow rsync traffic between analytics VLAN and fluorine - https://phabricator.wikimedia.org/T99245#1467664 (10Ottomata) Fluorine's udp2log instance is different than the one used for webrequest logs. It is used directly (I think) by mediawiki to log. We should deprecate it, bu... [17:14:17] 7Blocked-on-Operations, 3Discovery-Analysis-Sprint, 5Patch-For-Review: Create rsync connector to fluorine - https://phabricator.wikimedia.org/T98383#1467670 (10Ottomata) [17:14:18] 6operations, 5Patch-For-Review: Allow rsync traffic between analytics VLAN and fluorine - https://phabricator.wikimedia.org/T99245#1467668 (10Ottomata) 5Open>3Resolved a:3Ottomata [17:14:25] 7Puppet, 6operations, 5Patch-For-Review: Make Puppet repository pass lenient and strict lint checks - https://phabricator.wikimedia.org/T87132#1467671 (10scfc) I think as well that strict lint checks should be enforced, and as https://integration.wikimedia.org/ci/job/operations-puppet-puppetlint-strict/24810... [17:15:15] 7Blocked-on-Operations, 6operations, 6Commons, 6Multimedia, and 5 others: Convert eqiad imagescalers to HHVM, Trusty - https://phabricator.wikimedia.org/T84842#1467672 (10ori) >>! In T84842#1466863, @Joe wrote: > FTR, I am going to depool all remaining Zend imagescaler today to test any outstanding problem... [17:15:31] JohnFLewis: hehe apparently so, it was too blunt perhaps [17:15:39] (03PS1) 10Giuseppe Lavagetto: ci: fix ganglia plugins hotlinking [puppet] - 10https://gerrit.wikimedia.org/r/226104 [17:16:31] (03CR) 10Giuseppe Lavagetto: [C: 032] ci: fix ganglia plugins hotlinking [puppet] - 10https://gerrit.wikimedia.org/r/226104 (owner: 10Giuseppe Lavagetto) [17:18:44] (03PS1) 10Giuseppe Lavagetto: ci: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/226106 [17:21:46] (03CR) 10Giuseppe Lavagetto: [C: 032] ci: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/226106 (owner: 10Giuseppe Lavagetto) [17:24:28] (03PS4) 10Giuseppe Lavagetto: ganglia: standardize has_ganglia [puppet] - 10https://gerrit.wikimedia.org/r/225880 [17:24:57] RECOVERY - puppet last run on gallium is OK Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:29:33] bd808: should we just cancel the 1:1s going forward? we can talk at the weekly manager meeting, i should think? [17:32:25] (03CR) 10Giuseppe Lavagetto: [C: 031] "Will merge tomorrow" [puppet] - 10https://gerrit.wikimedia.org/r/225880 (owner: 10Giuseppe Lavagetto) [17:33:31] _joe_: i know your always busy, but it would be nice to get https://gerrit.wikimedia.org/r/#/c/219125/ on your schedule somewhere :) [17:33:42] s/your/you're/ [17:33:57] ebernhardson: submit a task for that if there isn't one already [17:34:40] jzerebecki: Wikibase and Wikidata extensions don't have a wmf/1.26wmf14 branch, that's why they don't automatically update wmf/1.26wmf14 branch of mediawiki/core. [17:34:44] ori: there is https://phabricator.wikimedia.org/T102937 where joe mentioned he was going to take care of it, but its been a few weeks [17:34:55] (few = 3) [17:34:59] he's very overloaded :( maybe other opsen can help [17:36:00] hmm, cmjohnson1, need some help with these installs [17:37:24] <_joe_> ebernhardson: sigh, you are right, I also have to build the package for T97675 and a security patch [17:37:28] RECOVERY - mediawiki-installation DSH group on mw1090 is OK [17:37:35] <_joe_> so, "tomorrow" I hope [17:37:46] _joe_: don't burn out! [17:37:48] <_joe_> ebernhardson: seriously, I have to roll that out [17:38:19] <_joe_> ori: no it's sitting since 3 weeks, it needs to be done :) [17:39:33] * _joe_ off now :) [17:40:20] ottomata: what problems are you getting? [17:40:27] You have ordered a Dell System with no OS installed. If you have ordered [17:40:28] direct attach 3TB or larger drives, please be aware that not all OSs have [17:40:28] support for these larger drives. Please consult the following blog for [17:40:28] support levels for various OS's and choose your OS to install accordingly. [17:40:28] Boot Failed: PXE Device 1: Integrated NIC 1 Port 1 Partition 1 [17:40:28] http://en.community.dell.com/dell-blogs/enterprise/b/tech-center/archive/2010/ [17:40:29] 12/16/breaking-through-the-2tb-partition-limitation-3tb-hard-drives-and-beyond [17:40:29] .aspx [17:40:40] i think there is a message i am missing that gets overwritten on the console by that [17:40:48] on carbon i see [17:40:51] Jul 21 17:34:15 carbon dhcpd: DHCPDISCOVER from 44:a8:42:24:cc:34 via 10.64.48.3: network 10.64.48.0/22: no free leases [17:41:14] cc:34 is analytics1043 [17:41:18] but i see that for all of them [17:41:33] k [17:41:49] if it says no free leases does that mean I misconfigured dhcpd somehow? [17:41:54] did I put it in the wrong file maybe? [17:42:01] i put it in ttyS1-115200 [17:42:07] same file the other analytics nodes are in [17:42:07] that's correct [17:44:54] 7Puppet, 6operations, 6Discovery, 10Wikidata, and 2 others: Make a puppet role that sets up a query service and loads it - https://phabricator.wikimedia.org/T95679#1467805 (10Smalyshev) 5Open>3Resolved [17:45:44] ottomata: problem was me...didn't setup the switch ports...give me a few mins [17:46:00] ah, ok thanks [17:48:51] ottomata: try now [17:50:32] k... [17:50:44] 6operations, 10Wikimedia-DNS, 10Wikimedia-Language-setup: nan and minnan subdomain redirects are a mess - https://phabricator.wikimedia.org/T86915#1467837 (10Glaisher) OK so, I think we should remove zh-cfr and minnan sites. After that, we should redirect from nan.$project.org to zh-min-nan until wiki rename... [17:53:58] cmjohnson1: same thing [17:54:04] godog: JohnFLewis rm -rf / does not work on recent core utils and os x does not have recent core utils [17:54:11] This can have terrible consequences [17:55:11] or let someone have a good few seconds [17:57:10] ottomata: forgot row D was setup with everything defaulted to private vlan...fixed [17:58:11] k trying again [18:00:04] twentyafterfour greg-g: Respected human, time to deploy MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150721T1800). Please do the needful. [18:01:15] hmm, cmjohnson1 getting better [18:01:16] still fail [18:01:23] .aspx filesize is 26720 Bytes [18:01:24] Downloading NBP file... [18:01:24] PXE-E21: Remote boot cancelled. [18:01:24] Boot Failed: PXE Device 1: Integrated NIC 1 Port 1 Partition 1 [18:01:24] Booting from Integrated RAID Controller 1: EFI Fixed Disk Boot Device 1 [18:01:39] i see it trying to serve os [18:01:40] Jul 21 18:00:22 carbon atftpd[8523]: Serving trusty-installer/ubuntu-installer/amd64/pxelinux.0 to 10.64.53.24:1678 [18:01:58] fix the raid cfg [18:02:02] oh [18:02:13] its the partman recipe? [18:02:19] or in bios? [18:02:21] no, h/w raid [18:02:23] oh [18:02:25] k... [18:03:01] cmjohnson1: these have the flex bays with 2 drives, yes? [18:03:14] yes [18:03:17] k [18:05:12] so, hm, cmjohnson1, hw raid shoudl be on, yes? [18:05:19] Integrated RAID Controller [18:05:46] oh i have to configure it? [18:05:48] righ? [18:05:51] yes remmebering... [18:05:51] how do you have it set up? [18:06:01] shoudl be h/w raid on the 2 flex bay drives [18:06:07] raid 1 [18:06:08] I can set it up...i believe raid 1 the 2 small disks [18:06:34] i forget about the 12 disks... [18:06:47] !log ori Synchronized php-1.26wmf14/extensions/WikimediaEvents: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/WikimediaEvents 968890f1a256a08a02925e4bdb53a8e8d64aacea (duration: 00m 13s) [18:06:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:07:17] cmjohnson1: i think i'm in and remmeer this... [18:07:22] !log ori Synchronized php-1.26wmf14/extensions/Scribunto: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/Scribunto 5af0350e2d09444db279f58504967d0e9b154534 (duration: 00m 13s) [18:07:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:07:44] hm [18:07:46] Virtual Disk 0: RAID1, 232GB, Ready [18:07:52] that looks right [18:08:31] Physical Disk 00:01:12: HDD, SATA, 232GB, Online, (512B) [18:08:32] Physical Disk 00:01:13: HDD, SATA, 232GB, Online, (512B) [18:08:46] cmjohnson1: afaict it looks good, yes? [18:08:48] you want to look? [18:09:18] okay...what about PD 00 to 11? you want them non-raid or individual Virtual Disks? [18:09:26] non raid [18:10:12] which servers are you in? can you get out of console? [18:10:23] i'm in all, getting out now ja [18:10:26] i'll leave it in bios mode [18:10:31] just leave me one [18:10:40] i'm out [18:10:44] do analytics1042 [18:11:44] (03PS1) 10Gage: Icinga: fix varnishncsa warning on text & mobile caches [puppet] - 10https://gerrit.wikimedia.org/r/226110 [18:12:13] ottomata: we need to convert each of the 12 PD's to non-raid disk [18:12:38] ah k [18:12:45] cmjohnson1: i can do them all at once [18:12:53] it's under physical disk...click on disk and under operation convert to non-raid [18:12:57] do i just have to select each one and flip [18:12:57] k [18:13:17] shall I do 1042 too? if so lemme in! [18:13:17] exiting console [18:13:27] out [18:14:24] springle: any chance to look at https://gerrit.wikimedia.org/r/#/c/202344/, it's a fairly simple change? [18:15:00] cmjohnson1: k doing that, there is no save confirmation, i assume i just hit that and then esc back? [18:15:12] ah! [18:15:13] yep [18:15:13] go [18:15:18] have to select go [18:15:19] ok [18:19:07] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [18:19:46] dr0ptp4kt: Sure. We can always setup some new meetings when we find a need [18:19:58] * bd808 is finally awake [18:20:50] (03CR) 10Tjones: access: shell account for Trey Jones [puppet] - 10https://gerrit.wikimedia.org/r/226077 (owner: 10Matanya) [18:22:38] (03PS1) 10GWicke: Make compaction alert less sensitive to short-time spikes [puppet] - 10https://gerrit.wikimedia.org/r/226113 [18:24:32] cmjohnson1: a little different [18:24:34] .aspx filesize is 26720 Bytes [18:24:34] Downloading NBP file... [18:24:34] Succeed to download NBP file. [18:24:34] Boot Failed: PXE Device 1: Integrated NIC 1 Port 1 Partition 1 [18:24:35] Booting from Integrated RAID Controller 1: EFI Fixed Disk Boot Device 1 [18:26:07] which preseed cfg are you using? I bet it's trying to install on one of the 3TB disks and not the raided 250GB disk [18:26:35] analytics-flex [18:26:43] same one we used for the other flex bays [18:26:56] i think it will try to install on whatever sda is [18:27:05] assuming sda is the hw raid-1 on the flex drives [18:27:32] booting back to bios... [18:27:42] (03CR) 10Muehlenhoff: "Before creating the rules I checked the existing elastic* systems and they all used 9200/9300. Are the any realistic conditions under whic" [puppet] - 10https://gerrit.wikimedia.org/r/224095 (https://phabricator.wikimedia.org/T104962) (owner: 10Muehlenhoff) [18:27:46] (03Abandoned) 10Smalyshev: Add definitions for WDQS service [puppet] - 10https://gerrit.wikimedia.org/r/216403 (owner: 10Smalyshev) [18:28:00] maybe sda isn't the hw raid [18:28:41] but, even so, cmjohnson1, if sda was one of the physical 3T drives, install would still work, right? it would just install on the wrong drive [18:30:52] ottomata: no, it would have to be setup as a GPT [18:31:14] disk size limitation for MBR is 2TB [18:31:27] ok hm [18:32:24] cmjohnson1: i'm out of 1042 if you want to poke around there [18:32:27] i'm poking aroundin others [18:32:32] okay..thx [18:32:57] cmjohnson1: Select Boot Device RAID1, 232GB, Ready> [18:32:58] that looks good [18:33:17] ah..yeah..forgot about that...need to change that [18:33:24] good catch [18:33:36] oh? that is correct, no? [18:33:42] what needs changed there? [18:34:27] hrm....give me a few mins ..going to check cfg [18:35:33] ok [18:36:15] bd808: cool cool, will cancel series [18:38:02] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: tjones needs access to stat1002 - https://phabricator.wikimedia.org/T106175#1468229 (10TJones) @fgiunchedi @Matanya: take 2! ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFBwfxSdeDwBy5ypWAfsfnmwqJ3R2ks+T9wbnqq30zPyOvZjzequ6vA5JzboCce6OqZ5+mLg3LPQqczlTXbJakWJTj... [18:39:09] ottomata: hi, people that need access to stat1002 need the private user group or the non-private one ? [18:39:40] private [18:39:42] stat1002 is private [18:41:05] ottomata: so what is the purpose of the regular alanytics user group ? [18:41:24] 6operations, 10Wikimedia-DNS, 10Wikimedia-Language-setup: nan and minnan subdomain redirects are a mess - https://phabricator.wikimedia.org/T86915#1468231 (10Purodha) >>! In T86915#1467837, @Glaisher wrote: > OK so, I think we should remove zh-cfr and minnan sites. After that, we should redirect from nan.$pr... [18:41:48] mathttps://wikitech.wikimedia.org/wiki/Analytics/Data_access#Access_Groups [18:42:07] (03PS2) 10Matanya: access: shell account for Trey Jones [puppet] - 10https://gerrit.wikimedia.org/r/226077 [18:43:14] ottomata: not quite clear analytics-users vs analytics-privatedata-users [18:43:22] and the formar has only one user [18:44:06] yeah it hasn't been a common request, since most data in hadoop is private [18:44:29] matanya: private data in hdfs is group readable by analytics-privatedata-users [18:44:41] i see [18:45:05] analytics-users gives access to hadoop (via stat1002, which is not ideal, because they could access the private logs on stat1002), but it doesn't allow access to private data inside hadoop [18:45:13] (03PS8) 10Ori.livneh: grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:45:17] ottomata: so in the ticket https://phabricator.wikimedia.org/T106175 needs analytics-privatedata-users [18:45:20] so, it is for users that want to use hadoop for purposess other than pricate data stuf [18:45:23] (03CR) 10Ori.livneh: [C: 032 V: 032] grafana: Set a default dashboard [puppet] - 10https://gerrit.wikimedia.org/r/224129 (owner: 10Krinkle) [18:45:40] i think not matanya, zero data is just log files on stat1002 [18:45:40] so [18:45:46] statistics-privatedata-users [18:46:30] statistics-privatedata-users vs. analytics-privatedata-users is very confusing for me :) [18:47:14] matanya: easy answer, if it starts with analytics, it is for hadoop cluster [18:47:25] if it starts with statistics, it is for shell access on a specific box [18:47:36] and who needs hadoop? researches ? [18:47:43] yes, but not always [18:47:50] depends on what they want access to [18:47:54] this says zero logs [18:48:20] which are not in hadoop by default. but, if someone wanted to put the zero logs in hadoop and then use it to process them [18:48:29] is there a list of what resides where apart from your amazing brain ? :) [18:48:35] statistics-privatedata-users + analytics-users would be appropriate [18:48:54] like what data is where? not really, maybe? puppet? [18:49:03] there is this matanya [18:49:04] https://wikitech.wikimedia.org/wiki/Category:Data_stream [18:49:06] oh actually [18:49:09] this is a great list! [18:49:36] execlent! [18:49:38] thanks [18:51:35] (03PS1) 10Matanya: access: add Trey Jones to statistics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/226115 [18:52:21] (03CR) 10jenkins-bot: [V: 04-1] access: add Trey Jones to statistics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/226115 (owner: 10Matanya) [18:52:51] (03CR) 10Matanya: "depends on https://gerrit.wikimedia.org/r/226077" [puppet] - 10https://gerrit.wikimedia.org/r/226115 (owner: 10Matanya) [18:55:38] (03PS1) 10Matanya: access: add dcausse to statistics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/226117 [18:58:46] dcausse: ^ [19:01:08] (03PS1) 10Muehlenhoff: Enable ferm on labnodepool [puppet] - 10https://gerrit.wikimedia.org/r/226118 [19:02:26] (03CR) 10Eevans: [C: 031] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/226113 (owner: 10GWicke) [19:06:41] (03PS3) 1020after4: Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) [19:09:17] (03CR) 10Rush: [C: 032 V: 032] Ensure that phabricator/src/extensions exists [puppet] - 10https://gerrit.wikimedia.org/r/226097 (https://phabricator.wikimedia.org/T104904) (owner: 1020after4) [19:09:30] 7Puppet, 6Labs, 6Phabricator, 5Patch-For-Review: On labs phabricator references security extension even though it isn't present - https://phabricator.wikimedia.org/T104904#1468296 (10mmodell) [19:16:12] (03PS3) 10Alex Monk: Disable a bunch of extensions on loginwiki/votewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225840 (https://phabricator.wikimedia.org/T61702) [19:23:53] 6operations: Clean up Hiera - https://phabricator.wikimedia.org/T106404#1468355 (10ori) 3NEW [19:25:30] 6operations: Simplify hiera lookup model - https://phabricator.wikimedia.org/T106404#1468364 (10yuvipanda) [19:26:01] YuviPanda: better :) [19:26:27] ori: :P [19:32:55] one thing I want is a way to test what value will be populated from the hieararchy [19:33:21] this would make exploration and refactor safer I think [19:33:31] maybe just my relative lack of hiera confidence [19:33:59] alexandros suggested doing the refactor and running the catalog compiler for all of production before and after [19:34:01] and then diffing [19:34:12] you can apparently do that, it just takes ~2 hours [19:34:31] I would rather have a local tool that says "values pulled from hiera:" [19:34:38] 6operations: Simplify hiera lookup model - https://phabricator.wikimedia.org/T106404#1468427 (10yuvipanda) This would also work fine with the new labs DNS scheme, which is $hostname.$projectname.eqiad.wmflabs. [19:34:49] chasemp: +1 [19:35:08] utils/hiera_lookup ? [19:35:10] that exists [19:35:18] maybe I just don't know about it / how to use it [19:35:29] I'll have to fit wikitech on this somehow but that seems trivial [19:35:52] chasemp: apart from that, thoughts re: the sanity / insanity of the approach i'm proposing? [19:36:29] I'm mid review on another changeset and then I'll try to be useful? [19:36:38] cool no rush [19:36:46] no pun intended [19:36:51] zin [19:36:52] g [19:37:08] teh zen version of zing [19:37:41] heh [19:37:45] * YuviPanda unrushes [19:41:55] (03CR) 10Rush: "commentary only preceded by ##'s :)" (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/218930 (https://phabricator.wikimedia.org/T101235) (owner: 10Jakob) [19:44:15] (03CR) 10Yuvipanda: Add Phragile module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/218930 (https://phabricator.wikimedia.org/T101235) (owner: 10Jakob) [19:44:25] (03PS2) 10Ori.livneh: Disable AccountAudit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/224936 (https://phabricator.wikimedia.org/T105894) (owner: 10Legoktm) [19:44:30] (03CR) 10Ori.livneh: [C: 032] Disable AccountAudit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/224936 (https://phabricator.wikimedia.org/T105894) (owner: 10Legoktm) [19:44:37] (03Merged) 10jenkins-bot: Disable AccountAudit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/224936 (https://phabricator.wikimedia.org/T105894) (owner: 10Legoktm) [19:44:47] chasemp: dammit I feel nerdsniped and feel the need to lecture someone on how they are using puppet for *deployment* and that's a terrible terrible idea [19:45:33] !log ori Synchronized wmf-config: I3887fd6c: Disable AccountAudit (duration: 00m 12s) [19:45:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:45:52] use puppet for orchestration (setting up your servers, proxies, db servers, etc) and something else for deployment (composer install, git checkout, restarts, upgrades etc) [19:47:13] (03CR) 10Rush: Add Phragile module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/218930 (https://phabricator.wikimedia.org/T101235) (owner: 10Jakob) [19:47:32] I think it's ok in some isntances to do some of this sanely :) [19:47:39] with puppet I mean [19:49:06] is wikidata supposed to still be at wmf13? [19:49:19] maybe it's language, I don't think of puppet as an orchestration tool like fabric or mcollective but generally it's poor suitability for deployment is a lack of procedural underpinnings so it makes it awkward and terrible [19:49:26] but idk it's all difficult [19:49:42] (03CR) 10Yuvipanda: Add Phragile module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/218930 (https://phabricator.wikimedia.org/T101235) (owner: 10Jakob) [19:50:38] YuviPanda: first tripup on using an actual url as a param default. It can't be reused across instances. [19:50:50] so it can't really be a default except in one case, which is to say...it can't be a default [19:50:56] chasemp: fair enough. [19:51:04] chasemp: actually [19:51:09] chasemp: I don't know why they need it at all? [19:51:18] why are they even using vhosts [19:51:22] there's only one domain being served [19:51:34] twentyafterfour: hmmm... seems like they would be cutting a new branch this week for wmf15 [19:51:34] I know...but I wasn't going to try to explain apache [19:51:57] bd808: that's why I asked [19:52:25] chasemp: so since you can't override params per host in labs with hiera yet, they're screwed either way :) [19:52:33] so you can't use that role across multiple instances either [19:52:37] bd: they went from wmf12 to wmf13 so maybe they are trying to get back to an even-number-week branch cadence? [19:52:54] "so since you can't override params per host in labs with hiera yet" didn't know this actually but heh [19:52:57] I guess I'm pushing the branch without them [19:53:29] twentyafterfour: hmmm, maybe. I pinged in #wikidata [19:53:33] (03CR) 10Rush: Add Phragile module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/218930 (https://phabricator.wikimedia.org/T101235) (owner: 10Jakob) [19:53:36] chasemp: like, even if you put any vhost in the role, that can't be changed per-host either [19:53:47] chasemp: so if they put the vhost in puppet at all, they're screwed [19:54:02] should be populated corrrectly in the template more or less [19:54:02] yes [19:54:04] there was something about wikibase in SWAT this morning, or at least talk about it [19:54:34] I don't understand the connection between wikibase and wikidata exactly [19:54:37] $::hostname.wmflabs.org? [19:54:59] chasemp: that ties you down to having one particular VM pet name because otherwise you will break URLs [19:54:59] Wikibase updated inside wikidata wmf13 branch this morning [19:55:05] during SWAT [19:55:20] it's a large part of the guts. "wikidata" is a collection of extensions baked into one with Composer. wikibase is the heart of that beast [19:55:21] YuviPanda: yes agreed but we do in phabricator case as it's simple and so far has been fine [19:55:30] but I understand the point [19:55:31] twentyafterfour: Wikibase is the extension, Wikidata is the site - literally that :) [19:55:40] chasemp: 'solution' is to not use a vhost because they do not need one, or not care about that the VM name will have to be the same and if they recreate it there is going to be downtime [19:55:48] JohnFLewis: ah, that makes more sense then [19:55:59] and re branch, let me look at the calendar [19:56:02] YuviPanda: sounds good to me [19:56:09] !log Dropping AccountAudit table on all wikis (T105894) [19:56:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:56:27] I have no objection but I thought you didn't want to fall down this rabbit hole :) [19:56:34] I'm basically picking my battles here [19:56:36] chasemp: I know man, it's the post wikimania energy. [19:56:42] It will pass! [19:57:09] twentyafterfour: Monday has the wmf26 branch on the calendar so I assume it will be branched for it [19:57:18] *wmf16 [19:57:24] (03CR) 10Yuvipanda: Add Phragile module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/218930 (https://phabricator.wikimedia.org/T101235) (owner: 10Jakob) [19:57:30] I agree the hsotname => url tie is tenuous as best and it's slightly awkward especially here [19:57:32] but I'm ok w/ it [19:57:42] and when someone needs to break out of it I would revisit [19:57:43] chasemp: I basically fell into it a bit because I'm doing the same thing for all the research projects (puppetizing them) [19:57:46] YuviPanda: can i ask a dumb question? How do i change the security group of an existing instance on wikitech.wikimedia.org? [19:57:52] I think this will be a 2 or 3 vm thing, 1 for real and 1 for test or so [19:57:56] jdlrobson: you can't :( [19:57:58] jdlrobson: not a dumb question at all! Terrible answer is you can not [19:58:01] noooooooo [19:58:03] jdlrobson: you can edit the default one [19:58:06] which is horrible [19:58:09] so i have to delete and redo everything i've just done?! [19:58:11] but oh well [19:58:25] jdlrobson: 'manage security groups' on sidebar, change the default security group to do what you want [19:58:35] chasemp: yeah, prolly ok in this case. ORES is already 8 VMs [19:59:25] I'm not sure if they'll understand your last comment there [19:59:31] but I do [19:59:58] chasemp: :) ok! my brain's still waking up from all the wikimania [20:00:11] chasemp: sorry to have dragged you down a bit with me, etc :) [20:00:21] I never mind having high standards [20:00:44] but I'll admit to be a pragmatism in this case [20:00:47] bit of even [20:00:48] chasemp: I think I also wasn't sure if you knew about the proxies requiring manual switchover etc [20:01:22] I have some vague idea of how it works in practice for me but no thanks for the clarification [20:02:07] thanks YuviPanda :-/ also bd808 thanks for adding us all as members to the other instance! [20:02:57] (03CR) 10Krinkle: [C: 04-1] "Please announce ahead of time through wikitech, wikitech-ambassadors and Tech News to allow the community time to track down any tools and" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/217858 (owner: 10Faidon Liambotis) [20:04:09] (03CR) 10Krinkle: "Maybe we can drop it from testwiki/test2wiki first so that tools can use that wiki name to verify their fix." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/217858 (owner: 10Faidon Liambotis) [20:06:52] ori: this would effectively eliminate the "role" keyword in use right? [20:06:59] T106404 I mean [20:10:42] (03CR) 10Rush: Add Phragile module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/218930 (https://phabricator.wikimedia.org/T101235) (owner: 10Jakob) [20:14:53] (03PS1) 10Smalyshev: Fix package_dir dependency for wdqs [puppet] - 10https://gerrit.wikimedia.org/r/226216 [20:16:13] urandom: don't remember right now, I think it was diamond including package python-yaml, and some other place also including it with ensure_packages and them failing [20:16:40] (03PS2) 10Smalyshev: Fix package_dir dependency for wdqs [puppet] - 10https://gerrit.wikimedia.org/r/226216 [20:19:25] (03PS3) 10Gage: Fix package_dir dependency for wdqs [puppet] - 10https://gerrit.wikimedia.org/r/226216 (owner: 10Smalyshev) [20:20:42] (03CR) 10Gage: [C: 032] Fix package_dir dependency for wdqs [puppet] - 10https://gerrit.wikimedia.org/r/226216 (owner: 10Smalyshev) [20:21:06] Krinkle: would there be a specific reason why the load.php 500 errors of PHPVersionCheck.php are not marked as no-cache ? Or is it just sloppyness [20:22:07] thedj: In what way is load.php different? [20:22:15] I don't see its call to wfEntryPointCheck being different [20:23:46] http://git.wikimedia.org/blob/mediawiki%2Fcore.git/78e7a1b786c9fa278f6fc087ae4ee6095a0d2fd7/includes%2FPHPVersionCheck.php#L153 [20:24:28] thedj: Oh, right [20:24:33] I think that's significant [20:24:39] i know of no specific reason why that couldn't be exactly the same headers as somewhere higher up. [20:24:44] Caching of 4xx and 5xx responses is special [20:24:49] but there might be one of course :) [20:25:14] I'm not sure if this currently happens but the desired behaviour is that existing non-error cache stays [20:25:17] YuviPanda: hey dude. Am trying to get vagrant installed on a labs instance but don't have access to labs-vagrant command [20:25:18] unless manually purged [20:25:25] getting `Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Resource type mw-extension doesn't exist on node mf-browser-tests.mobile-smoketests.eqiad.wmflabs` for some reason` when running puppet agent [20:25:41] jdlrobson: have you seen https://wikitech.wikimedia.org/wiki/Help:Labs-vagrant [20:25:55] what documentation are you following? [20:25:56] yeh that's what i'm reading through... [20:25:56] Krinkle: so, if you get a broken response, you would still be allowed to keep the old non-broken response [20:26:09] thedj: right [20:26:11] bd808: ^^ (jdlrobson's error) [20:26:13] instance exists and should have role and access to port 80 :/ [20:26:13] thedj: but this doens't change that [20:26:25] jdlrobson: lets take this over to #wikimedia-labs [20:26:31] thedj: i'd move the headers up to the 'else' branch of 'if' cli. And then have index/load.php inside that if -block [20:28:04] so if cli, else set headers (if index.php else if load.php end) end [20:28:26] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.14% of data above the critical threshold [500.0] [20:28:29] !log Zuul no more report any result back to Gerrit :( Fix being deployed [20:28:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:30:39] (03PS1) 1020after4: static symlinks: add 1.26wmf15, remove 1.26wmf8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226221 [20:37:06] 6operations: Simplify hiera lookup model - https://phabricator.wikimedia.org/T106404#1468756 (10chasemp) Does anyone think the main weakness it exposes re: mw2148.yaml, mw2149.yaml, mw2150.yaml, and mw2151.yaml would be better resolved by a better naming scheme to fit this model? [20:41:24] (03CR) 1020after4: [C: 032] static symlinks: add 1.26wmf15, remove 1.26wmf8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226221 (owner: 1020after4) [20:46:40] I'm ready to scap if there are no objections [20:48:37] ori: so the table has been dropped everywhere? [20:51:41] !log twentyafterfour Purged l10n cache for 1.26wmf9 [20:51:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:52:14] !log twentyafterfour Purged l10n cache for 1.26wmf10 [20:52:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:53:02] YuviPanda: ok; i'm trying to understand why this is happening in deployment-prep: https://phabricator.wikimedia.org/P1029 [20:53:03] !log twentyafterfour Purged l10n cache for 1.26wmf11 [20:53:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:53:39] !log twentyafterfour Started scap: sync 1.26wmf15 branch + localization cache, remove wmf8 [20:53:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:54:31] if an ops person would care to double check: https://gerrit.wikimedia.org/r/#/c/222917/ much appreciated [20:54:40] (03PS1) 10MaxSem: Enable tracking of geo features on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226224 (https://phabricator.wikimedia.org/T103017) [20:55:09] (03CR) 10jenkins-bot: [V: 04-1] Enable tracking of geo features on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226224 (https://phabricator.wikimedia.org/T103017) (owner: 10MaxSem) [20:56:14] (03PS2) 10MaxSem: Enable tracking of geo features on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226224 (https://phabricator.wikimedia.org/T103017) [20:57:25] urandom: I see. is that happening on all nodes or? [20:57:54] YuviPanda: it's happening on deployment-restbase02 (i haven't tried others) [20:58:53] (03PS1) 10Yuvipanda: diamond: Use require_package instead of ensure_packages [puppet] - 10https://gerrit.wikimedia.org/r/226225 [20:58:55] urandom: ^ might fix that [20:59:30] (03PS2) 10Yuvipanda: diamond: Use require_package instead of ensure_packages [puppet] - 10https://gerrit.wikimedia.org/r/226225 [20:59:53] YuviPanda: cool; let me give it a try [21:03:03] YuviPanda: aye, that seems to do it [21:03:20] (03CR) 10DCausse: "I agree this should not happen in "normal condition", but recently Nik discovered that we had 2 ES instances running on a single node. And" [puppet] - 10https://gerrit.wikimedia.org/r/224095 (https://phabricator.wikimedia.org/T104962) (owner: 10Muehlenhoff) [21:03:58] (03PS3) 10Yuvipanda: diamond: Use require_package instead of ensure_packages [puppet] - 10https://gerrit.wikimedia.org/r/226225 [21:03:59] urandom: sweet [21:04:07] urandom: I'll merge now [21:04:17] (03CR) 10Yuvipanda: [C: 032 V: 032] diamond: Use require_package instead of ensure_packages [puppet] - 10https://gerrit.wikimedia.org/r/226225 (owner: 10Yuvipanda) [21:04:23] (03CR) 10Eevans: "Fixes the issue I was seeing (https://phabricator.wikimedia.org/P1029)" [puppet] - 10https://gerrit.wikimedia.org/r/226225 (owner: 10Yuvipanda) [21:05:13] YuviPanda: thank you! [21:05:19] urandom: yw! [21:06:09] (03PS2) 10Gage: Symlink .wars for git-fat in Archiva too [puppet] - 10https://gerrit.wikimedia.org/r/223089 (owner: 10Ottomata) [21:07:29] (03CR) 10Gage: [C: 032] Symlink .wars for git-fat in Archiva too [puppet] - 10https://gerrit.wikimedia.org/r/223089 (owner: 10Ottomata) [21:07:31] 6operations, 10Beta-Cluster, 6Labs, 7Monitoring: Setup (simple) catchpoint monitoring and metrics for enwiki betacluster just like production - https://phabricator.wikimedia.org/T97865#1468877 (10greg) We talked about this on ops list: https://lists.wikimedia.org/mailman/private/ops/2015-July/049244.html... [21:16:15] (03PS2) 10Eevans: WIP: Cassanra logstash setup [puppet] - 10https://gerrit.wikimedia.org/r/226025 (https://phabricator.wikimedia.org/T100970) [21:20:44] 6operations: Simplify hiera lookup model - https://phabricator.wikimedia.org/T106404#1468927 (10yuvipanda) Indeed, this hiera schema might force us to a nicer naming scheme as well :) [21:21:11] !log twentyafterfour Finished scap: sync 1.26wmf15 branch + localization cache, remove wmf8 (duration: 27m 32s) [21:21:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:22:27] (03PS3) 10Eevans: WIP: Cassanra logstash setup [puppet] - 10https://gerrit.wikimedia.org/r/226025 (https://phabricator.wikimedia.org/T100970) [21:47:19] (03PS4) 10Eevans: WIP: Cassanra logstash setup [puppet] - 10https://gerrit.wikimedia.org/r/226025 (https://phabricator.wikimedia.org/T100970) [21:48:09] (03PS1) 10Tim Landscheidt: Labs: Let Puppet install mpt-status [puppet] - 10https://gerrit.wikimedia.org/r/226232 (https://phabricator.wikimedia.org/T104779) [21:55:17] PROBLEM - BGP status on cr2-ulsfo is CRITICAL No response from remote host 198.35.26.193 [22:01:36] RECOVERY - BGP status on cr2-ulsfo is OK host 198.35.26.193, sessions up: 45, down: 0, shutdown: 0 [22:03:07] mw1090.eqiad.wmnet returned [127]: bash: /srv/deployment/scap/scap/bin/sync-common: No such file or directory [22:03:20] fun [22:06:22] twentyafterfour: It looks to me like that might be a new host or freshly reimaged [22:06:34] and not synced with all the salt magic yet [22:06:35] yeah [22:06:46] it had a failing hard drive last week [22:06:53] ah [22:17:43] twentyafterfour: [22:17:45] Error: Could not update: Execution of '/usr/bin/salt-call --log-level=quiet --out=json grains.append deployment_target scap/scap' returned 2: Minion failed to authenticate with the master, has the minion key been accepted? [22:17:47] Error: /Stage[main]/Mediawiki::Scap/Package[scap]/ensure: change from absent to latest failed: Could not update: Execution of '/usr/bin/salt-call --log-level=quiet --out=json grains.append deployment_target scap/scap' returned 2: Minion failed to authenticate with the master, has the minion key been accepted? [22:17:49] that's the result of a puppet run on mw1090 [22:18:35] yeah apparently it's broken. Is it in rotation? [22:18:45] dunno, i accepted its key on the salt master [22:18:48] let's try puppet again [22:18:50] ok [22:20:14] !log Accepted mw1090's minion key on palladium [22:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:20:35] (03PS1) 10Ricordisamoa: Don't match Phabricator task IDs inside URLs [puppet] - 10https://gerrit.wikimedia.org/r/226234 [22:22:05] (03PS1) 1020after4: group0 to 1.26wmf15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226235 [22:22:12] (03CR) 10Paladox: "I think there is a bug for this in phabricator." [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:22:16] (03CR) 1020after4: [C: 032] group0 to 1.26wmf15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226235 (owner: 1020after4) [22:23:25] (03CR) 1020after4: [C: 031] Don't match Phabricator task IDs inside URLs [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:24:56] 6operations: Update wikimedia apt repo to include debs for shiny-server - https://phabricator.wikimedia.org/T106435#1469127 (10EBernhardson) 3NEW [22:26:25] (03CR) 10Rush: [C: 031] "seems ok but I would love it if qchris could confirm and thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:26:31] !log twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf15 [22:26:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:27:52] PHP fatal error: [22:27:52] File not found: /srv/mediawiki/php-1.26wmf15/../wmf-config/ExtensionMessages-1.26wmf15.php [22:27:56] !log twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: revert group0 to 1.26wmf15 [22:27:57] twentyafterfour: mw.o down [22:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:28:01] legoktm: reverted [22:28:04] ok [22:28:06] thanks [22:28:09] Yup, back up. [22:28:09] (03CR) 10Ricordisamoa: "Not tested :(" [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:28:18] that's a wtf though [22:28:30] * twentyafterfour doesn't know why that file is missing [22:28:38] (03CR) 10Ricordisamoa: "It looks like e3fe7011e360804f1c7ab4b0c93b9bb5f6cf8b39 should have fixed this but a8999ced55773efd7a93d21bbbf77d52ed3055b2 reverted it." [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:29:31] Couldn't just be mw1090 could it? [22:29:39] I don't think so [22:29:45] If it was, we wouldn't have seen legoktm and James_F noticing mw.org being down independently [22:29:48] 6operations: Update wikimedia apt repo to include debs for shiny-server - https://phabricator.wikimedia.org/T106435#1469156 (10EBernhardson) [22:29:57] yeah was down for me as ewll [22:30:06] and me [22:30:08] :) [22:30:16] I saw it blip but then main page was there [22:30:19] so I wasn't sure [22:31:54] Unless mw.org is pinned to mw1090 somehow? [22:34:12] I don't think mw.org is a pinned wiki [22:34:34] ebernhardson: a .changes isn't mandatory, but would-be-nice - I've imported .debs before (vagrant was a similar package) [22:35:07] RECOVERY - puppet last run on mw1090 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [22:35:37] (03CR) 10Paladox: "Here https://phabricator.wikimedia.org/T75997" [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:39:03] !log twentyafterfour Started scap: test: syncing 1.26wmf15 again [22:39:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:39:19] (03PS1) 10Rush: phab: syntax fixes for extension management [puppet] - 10https://gerrit.wikimedia.org/r/226236 [22:40:19] (03CR) 10Rush: [C: 032 V: 032] phab: syntax fixes for extension management [puppet] - 10https://gerrit.wikimedia.org/r/226236 (owner: 10Rush) [22:41:32] opsen, we need some help downgrading the zuul package on gallium [22:41:54] who can help? [22:42:18] I'm about [22:42:34] coming to -releng [22:42:42] ty [22:44:07] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [22:46:50] (03PS5) 10Eevans: WIP: Cassanra logstash setup [puppet] - 10https://gerrit.wikimedia.org/r/226025 (https://phabricator.wikimedia.org/T100970) [22:47:17] YuviPanda: good to know, thanks [22:51:30] !log gallium 'service zuul stop && service zuul-merger stop && sudo apt-get install zuul=2.0.0-304-g685ca22-wmf1precise1' DOWNGRADE due to errors [22:51:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:53:25] 7Puppet, 6Labs, 3Labs-Sprint-104, 3Labs-Sprint-105: Allow per-host hiera overrides via wikitech - https://phabricator.wikimedia.org/T104202#1469226 (10yuvipanda) [22:53:54] (03CR) 10QChris: [C: 04-1] Don't match Phabricator task IDs inside URLs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:54:21] !log 22:50 < chasemp> "then git reset --hard 9588d0a6844fc9cc68372f4bf3e1eda3cffc8138 in /etc/zuul/wikimedia" [22:54:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:58:47] (03CR) 10Ricordisamoa: Don't match Phabricator task IDs inside URLs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa) [22:59:55] !log twentyafterfour Finished scap: test: syncing 1.26wmf15 again (duration: 20m 51s) [22:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:00:04] RoanKattouw ostriches Krenair: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150721T2300). [23:00:04] James_F: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [23:00:14] * James_F waves. [23:00:52] * MaxSem waves [23:01:10] good thing zuul is back :) [23:01:59] It is? [23:02:06] Oh good [23:02:18] Alright then, I'll do SWAT [23:04:10] (03Abandoned) 10Gage: Install-server: fix partman/logstash.cfg [puppet] - 10https://gerrit.wikimedia.org/r/211931 (https://phabricator.wikimedia.org/T98620) (owner: 10Gage) [23:04:27] (03CR) 10Catrope: [C: 032] Enable tracking of geo features on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226224 (https://phabricator.wikimedia.org/T103017) (owner: 10MaxSem) [23:05:00] (03Merged) 10jenkins-bot: Enable tracking of geo features on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/226224 (https://phabricator.wikimedia.org/T103017) (owner: 10MaxSem) [23:05:27] !log twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: trying this again: group0 to 1.26wmf15 [23:05:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:05:52] uh, mid-air collision? [23:05:58] Ugh [23:06:07] twentyafterfour: You still doing the train deploy? :S [23:06:17] RoanKattouw: just finished it [23:06:30] sorry it went way long again [23:06:36] No worries [23:06:49] I just thought you were done already because of the "finished scap" message [23:06:52] tuesdays are real inconvenient right now but at least the rest of the week is smooth sailing [23:07:20] that first scap was testwiki, second one was mediawiki.org [23:07:44] !log catrope Synchronized wmf-config/InitialiseSettings.php: Enable tracking of geo feature usage on enwiki (duration: 00m 13s) [23:07:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:08:10] !log catrope Synchronized wmf-config/CommonSettings.php: Enable tracking of geo feature usage on enwiki (duration: 00m 12s) [23:08:10] thanks Roan, testing [23:08:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:09:15] Ooh, morebots has a new message, ncie [23:12:56] RoanKattouw, I can't see it to work, but at least I don't see it breaking stuff. You can continue with James' patch [23:23:33] 10Ops-Access-Requests, 6operations: access request for server side uploads - https://phabricator.wikimedia.org/T106447#1469336 (10Matanya) 3NEW [23:25:51] (03PS1) 10Yuvipanda: ssh: Disable LDAP key lookup for HBA [puppet] - 10https://gerrit.wikimedia.org/r/226246 (https://phabricator.wikimedia.org/T101447) [23:25:53] (03PS1) 10Yuvipanda: ssh: Remove ssh_restrict_network LDAP variable [puppet] - 10https://gerrit.wikimedia.org/r/226247 (https://phabricator.wikimedia.org/T101447) [23:26:06] Coren: ^ [23:26:16] matanya: ha, I was just about to ask you about that [23:26:36] 10Ops-Access-Requests, 6operations: access request for server side uploads - https://phabricator.wikimedia.org/T106447#1469355 (10Legoktm) I think adding matanya to the "restricted" group would work? I support this request fwiw. [23:27:11] 10Ops-Access-Requests, 6operations: access request for server side uploads - https://phabricator.wikimedia.org/T106447#1469366 (10Matanya) [23:27:21] thanks legoktm [23:27:35] legoktm [23:27:57] How are ya? [23:30:16] (03CR) 10coren: [C: 031] "I know what it was for: it was for restricting SSH access to certain boxes. :-)" [puppet] - 10https://gerrit.wikimedia.org/r/226247 (https://phabricator.wikimedia.org/T101447) (owner: 10Yuvipanda) [23:30:32] Obvious comment is obvious. :-) [23:30:54] (03PS2) 10Yuvipanda: ssh: Disable LDAP key lookup for HBA [puppet] - 10https://gerrit.wikimedia.org/r/226246 (https://phabricator.wikimedia.org/T101447) [23:31:01] (03CR) 10Yuvipanda: [C: 032 V: 032] ssh: Disable LDAP key lookup for HBA [puppet] - 10https://gerrit.wikimedia.org/r/226246 (https://phabricator.wikimedia.org/T101447) (owner: 10Yuvipanda) [23:31:09] (03PS2) 10Yuvipanda: ssh: Remove ssh_restrict_network LDAP variable [puppet] - 10https://gerrit.wikimedia.org/r/226247 (https://phabricator.wikimedia.org/T101447) [23:31:17] (03CR) 10Yuvipanda: [C: 032 V: 032] ssh: Remove ssh_restrict_network LDAP variable [puppet] - 10https://gerrit.wikimedia.org/r/226247 (https://phabricator.wikimedia.org/T101447) (owner: 10Yuvipanda) [23:37:30] !log catrope Synchronized php-1.26wmf15/extensions/VisualEditor: SWAT (duration: 00m 13s) [23:37:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:39:14] !log catrope Synchronized php-1.26wmf14/extensions/VisualEditor: SWAT (duration: 00m 13s) [23:39:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:40:40] RoanKattouw: can I sneak https://gerrit.wikimedia.org/r/#/c/225559/ into swat? [23:41:02] 6operations, 6Multimedia: Add monitoring of upload rate on commons to icinga alerts - https://phabricator.wikimedia.org/T92322#1469411 (10Tgr) [23:41:31] legoktm: Sure, if you add it to the wiki page [23:41:38] (03CR) 10Catrope: [C: 032] Set $wgVectorResponsive = true on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225559 (https://phabricator.wikimedia.org/T106226) (owner: 10Legoktm) [23:42:04] (03Merged) 10jenkins-bot: Set $wgVectorResponsive = true on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/225559 (https://phabricator.wikimedia.org/T106226) (owner: 10Legoktm) [23:42:48] RoanKattouw: done [23:45:38] !log catrope Synchronized wmf-config/InitialiseSettings.php: Set $wgVectorResponsive = true on testwiki (duration: 00m 12s) [23:45:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:46:39] woot [23:58:20] (03CR) 10QChris: Don't match Phabricator task IDs inside URLs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/226234 (owner: 10Ricordisamoa)