[00:06:33] (03CR) 10Chad: [C: 031] Add /api/ listing to www.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/252863 (https://phabricator.wikimedia.org/T118519) (owner: 10GWicke) [01:00:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [01:03:57] (03PS1) 10Dzahn: base::firewall: remove exec for nf_conntrack [puppet] - 10https://gerrit.wikimedia.org/r/253056 [01:05:12] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [01:10:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [01:12:19] ah, that's in .frack.eqiad, not .eqiad [01:13:00] can icinga not be made to show the full name for misc hosts? [01:14:32] although then I guess you could still get "db1008" which is actually in frack :/ [01:15:12] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [01:15:50] Krenair: it shows me the IP address when i am in web ui [01:16:03] yea, fqdn would be nice [01:16:08] but also an icinga-wm feature [01:16:16] member of "Fundraising Banner Logger" [01:16:17] fwiw [01:17:00] interesting, frack has a separate mgmt domain in codfw but not eqiad? [01:18:08] hmm.. i didnt know [01:18:22] about the fr setup in codfw [01:20:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [01:25:11] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:55:05] err, I'm getting a 503 on enwiki [01:55:48] me too [01:55:49] ping bblack YuviPanda ori [01:55:57] who else can be there [01:56:02] i'm here [01:56:20] but i need to take my son to a doctor. what was the last deployment? [01:56:34] I'm getting 503 on load.php on mediawikiorg and also when diffing things on office wiki [01:56:35] you [01:56:36] 23:30 logmsgbot: ori@tin Synchronized rpc/RunJobs.php: I5e15ec9fb: Disable ChronologyProtector for rpc/RunJobs.php (duration: 00m 30s) [01:57:00] why the fuck is icinga quiet? [01:57:01] checking logstash [01:57:36] logstash is elevated but not outage grade [01:57:42] There's like 10,000 hits for 'Internal error in ApiQueryAllUsers::execute: Saw more duplicate rows than expected' [01:58:01] not related to normal pv's broken [01:58:16] wfm now... [01:58:16] !log restarted HHVM on all app servers [01:58:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:58:41] dafuq happened? [01:59:14] i gotta run in a sec but i tailed fatal.log, saw memory exhaustion, figured things couldn't be much worse than they are already and went for an hhvm restart across the cluster [01:59:15] There's also a lot of A connection error occured. Query: SELECT rev_id,rev_page,rev_text_id,rev_timesta [01:59:25] $ sudo salt -G 'php:hhvm' cmd.run 'service hhvm status | grep running && service hhvm restart' [01:59:28] For various regular view urls [01:59:45] could someone start paging? [01:59:50] even if we're back up, this needs attention [01:59:52] Lost connection to MySQL server during query [02:00:13] call jynus (jaime crespo), bblack and faidon at minimum [02:00:36] anyone on that? [02:00:49] ok, i'm calling [02:00:51] ori: yes [02:00:52] calling brandon [02:00:59] i'll call jaime [02:01:01] calling faidon [02:03:06] hi [02:03:12] what's wrong? [02:03:20] 503s on every page request [02:03:35] I just did one fine, not logged in on enwiki... [02:03:35] i tailed fatal.log, saw memory exhaustion, figured things couldn't be much worse than they are already and went for an hhvm restart across the cluster [02:03:41] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [500.0] [02:03:47] $ sudo salt -G 'php:hhvm' cmd.run 'service hhvm status | grep running && service hhvm restart' [02:03:48] hey [02:03:49] even with cache busting query arg [02:03:49] site came back [02:03:52] there's the alarm [02:03:55] it just showed up now [02:03:56] but no real idea what the cause was [02:04:59] * YuviPanda is here [02:05:01] if you didn't call jaime, don't yet [02:05:08] * YuviPanda reads backlog [02:05:13] I see a big 503 spike so far, but it did go back down [02:05:30] it's 3am there, if there isnt't evidence of a database issue so far and one that we cannot solve, let's shield him from that [02:05:39] k [02:06:04] too late I guess [02:06:18] someone called? [02:07:30] 18:00 < Krinkle> There's like 10,000 hits for 'Internal error in ApiQueryAllUsers::execute: Saw more duplicate rows than expected' [02:07:38] 18:02 < Krinkle> There's also a lot of A connection error occured. Query: SELECT rev_id,rev_page,rev_text_id,rev_timesta [02:08:08] a lot of the 503s in the logs seem to be disproportionately /w/index.php with some query args e.g. ?title=Special:LinkSearch [02:08:09] Yeah, there is also a few hundred hits in the last hour for: 'Connection error: Unknown error (208.80.154.136)' and and 'Error connecting to 208.80.154.136: Can't connect to MySQL server on '208.80.154.136' (4)' in log stash. [02:08:20] !info 208.80.154.136 [02:08:21] https://www.mediawiki.org/wiki/WMF_Projects/Wikimedia_Labs [02:08:23] ?title=MediaWiki:Wikibugs.js&action=raw&ctype=text/javascript [02:08:24] http://bots.wmflabs.org/dump/%23wikimedia-operations.htm [02:08:24] @info 208.80.154.136 [02:08:24] etc [02:08:24] Krinkle: [208.80.154.136: silver] silver [02:08:57] That's wikitech [02:09:00] enwiki: Could not connect to server "10.192.0.199" - /wiki/Main_Page [02:09:04] http://bots.wmflabs.org/dump/%23wikimedia-operations.htm [02:09:04] @info 10.192.0.199 [02:09:04] Krinkle: Unknown identifier (10.192.0.199) [02:09:09] Shouldn't be related to the site outage [02:09:15] !info 10.192.0.199 [02:09:15] https://www.mediawiki.org/wiki/WMF_Projects/Wikimedia_Labs [02:09:21] That's Jobqueu redis [02:09:24] 10.192.0.199 [02:09:28] Hmm I guess neither of those things works [02:09:43] rdb2001 [02:10:00] raw index.php almost always skips caches, hence the abundance [02:10:34] is rdb2001 actually in use? [02:11:05] Can't connect to MySQL server on 10.64.48.21, 10.64.48.20, 10.64.48.26 [02:11:10] those are also in the logs. Not nearly as high though [02:11:16] rdb2001.codfw.wmnet (10.192.0.119) [02:11:16] couple dozen in the last hour [02:11:21] http://bots.wmflabs.org/dump/%23wikimedia-operations.htm [02:11:21] @info 10.64.48.21 [02:11:21] Krinkle: [10.64.48.21: s1] db1066 [02:11:26] 10.192.0.199 does not exist [02:11:26] I do not see mysql issues [02:11:45] no wonder enwiki couldn't connect to it [02:11:46] Krenair: https://github.com/search?q=%2210.192.0.199%22+@wikimedia&type=Code&ref=searchresults [02:11:48] I see segfaults in hhvm log [02:12:09] https://logstash.wikimedia.org/#dashboard/temp/AVEDw1tJptxhN1XaOpwm [02:12:10] 10.192.0.199 isn't legit I don't think [02:12:16] It's 119 [02:12:17] not 199 [02:12:28] https://logstash.wikimedia.org/#/dashboard/elasticsearch/mediawiki-errors [02:12:44] rdb2001.codfw.wmnet [02:12:53] we shouldn't be connecting to anything in codfw for live hits right? [02:12:58] wmnet:rdb2001 1H IN A 10.192.0.119 [02:13:21] analytics was working on an announcement for the page view api, did they turn anything on? the spike of traffic on the app servers coincides with http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Analytics+Query+Service+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [02:13:56] yes, 5000 errors, out of 100000/s [02:14:20] yeah logstash doesn't have a big spike or anything [02:14:24] hmmm [02:14:43] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:14:47] did anyone follow the incident protocol? [02:14:52] (side note, sorry) [02:15:15] ori, wqs has periodic spikes, not likely related [02:15:30] I guess, not, I'll drop a quick note to outage-notifications [02:15:45] i was on the phone with pediatric emergency assessing whether i should bring my son in because he has some massively swolen bite, did not follow protocol, sorry [02:15:49] (03PS1) 10Dzahn: fix wrong IP for codfw redis server [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253063 [02:15:59] ouch, hope he's ok :/ [02:16:27] hmm, segfault only on one host so everything else probably ok [02:16:39] mutante, shouldn't there be reverse dns there too? [02:16:44] i'm actually gonna go now, thanks for responding to the page, and i will review the protocol and do anything i need to when i get back [02:16:55] mutante: The redis server failures are only for enwiki /wiki/Main_Page. I guess those are from health checks of some kind [02:17:00] well I guess there's little point in it now, site works [02:17:06] I've got 1:53 -> 1:59 or so, but honestly the rates in the req logs don't look like a total outage to me [02:17:07] I guess I shouldn't page people for no reason [02:17:18] very steady interval and ditribution among servers [02:17:24] mutante, ignore message above...my bad [02:17:37] 5xx running to ~700/sec, out of !33K/sec gets [02:17:40] ori: take care, hope your son gets ok [02:17:41] apologies if it was an overreaction, every pv for me and everyone around me was 533 [02:17:42] s/!/~/ heh [02:17:45] 503 [02:17:49] bye! [02:17:54] it may have been mostly for logged-in? [02:17:57] bye! [02:18:09] yup, ws like a total outage for me [02:18:19] (03PS2) 10Dzahn: fix wrong IP for codfw redis server [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253063 [02:18:19] and ori's restart of hhvm helped [02:18:19] See spike on https://grafana.wikimedia.org/dashboard/db/varnish-http-errors [02:18:22] it seems to have resolved [02:18:29] I only see 40 failed connections to mysql per server [02:18:36] second graph from the top [02:18:38] i did not see it myself, but a user reported it on wikimedia-tech [02:18:43] a spike indeed, but not too serious [02:18:44] and after the hhvm restarted they said it was back [02:18:47] (third) [02:19:16] https://grafana.wikimedia.org/dashboard/db/varnish-aggregate-client-status-codes?from=1447445941501&to=1447467541501&var-site=All&var-cache_type=mobile&var-cache_type=text&var-status_type=5 [02:19:33] ^ that shows the rates I'm talking about, doesn't seem like most gets were affected [02:20:03] (it was all confined to mobile+text, and it's on all DCs) [02:20:23] !log l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 58s) [02:20:28] these grafana dashboard confuse me [02:20:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:20:31] (03PS3) 10Dzahn: fix wrong IP for codfw redis server [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253063 [02:20:40] very inconsistent reporting of 500 vs. 503 [02:20:45] if I read https://grafana.wikimedia.org/dashboard/db/varnish-http-errors correctly [02:20:46] ? [02:20:54] there were no 500, just 503s [02:20:57] I can't really read it all, try my link [02:21:14] your link doesn't make a distinction between 5xx subcodes I think? [02:21:16] it just says "type 5" [02:21:19] yes, it does [02:21:32] top graph is all reqs, middle graph is 5xx, bottom splits out the various 5xx codes [02:21:39] oh, right [02:21:52] so no 500s [02:21:58] why are we even looking at mediawiki then? [02:22:01] logstash and all that [02:22:28] hrm I guess the hhvm restart did fix, maybe, possibly [02:22:34] did fix it* [02:22:47] that would fit what was reported by users on IRC [02:22:49] well 503s could still mean varnish->mw conns failing [02:23:08] or timing out or dropping before completion, etc [02:23:31] sure, and appservers memory graph is suspicious, but zero 500s is too [02:23:52] ori said he saw memory exhaustion before he restarted them [02:26:26] faidon@oxygen:/srv/log/webrequest$ grep 2015-11-14T01:57 5xx.json | jq .x_cache | sed 's/,.*//' | sort | uniq -c | sort -nr| head -5 [02:26:29] 23500 "cp1066 miss (0) [02:26:32] 839 "cp1065 miss (0) [02:26:34] 671 "cp1055 miss (0) [02:26:37] 625 "cp1052 miss (0) [02:26:39] 624 "cp1067 miss (0) [02:27:31] hmmm [02:28:06] due to URL hash? [02:28:29] hhvm channel in logstash also has a trending 'Lost parent, LightProcess exiting'. I don't know if that's SNAFU though. [02:28:38] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Text+caches+eqiad&h=cp1066.eqiad.wmnet&jr=&js=&event=hide&ts=0&v=8608136&m=varnish.backend_busy&vl=N%2Fs&ti=Backend+conn.+too+many [02:28:42] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Text+caches+eqiad&h=cp1066.eqiad.wmnet&jr=&js=&event=hide&ts=0&v=175&m=varnish.n_sess&vl=N&ti=N+struct+sess [02:28:46] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Text+caches+eqiad&h=cp1066.eqiad.wmnet&jr=&js=&event=hide&ts=0&v=500&m=varnish.n_wrk&vl=N&ti=N+worker+threads [02:29:30] Krinkle: that might just be the restart [02:30:15] is that just because /w/index.php hashes there? :) [02:30:26] surely not, the quewry args count too [02:31:32] POSt request don't have query params though [02:31:38] if that's what it is [02:31:45] but the requests I was trying were GETs though [02:32:39] (03CR) 10Bmansurov: [C: 04-1] "We need to SWAT deploy this on Monday only. -1 until then." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253038 (https://phabricator.wikimedia.org/T118525) (owner: 10Jhobs) [02:33:00] well yeah but once things break the damage spread [02:33:25] https://grafana.wikimedia.org/dashboard/db/save-timing did have a spike as well [02:34:21] anything I can do at all? [02:34:24] or should I go? [02:34:37] you can go YuviPanda [02:35:06] ok. I'll have my phone handy and keep sober in case I'm needed [02:35:13] thanks paravoid [02:35:27] the cp1066 bit is almost certainly because something hashes there, but nothing stands out as e.g. a huge spike in a specific uri_path in the sampled log [02:35:40] Krinkle, look at the scle [02:35:50] from 900ms to 1.1s [02:36:08] paravoid: this gets back to your whole thing about switching to backend_random when total request rate gets too high [02:37:29] jynus: Im faimilar with the scale, I look at it times a day. But the p75 doubled as well. The largest graph there is a median, which doesn't always tell the story. And the other lines show that such anomaly is unusual, save timing is suprisingly stable. [02:38:38] 300ms -> 1s. 510ms -> 2.1s [02:38:44] p50 and p75 respectively [02:38:51] avg -> max [02:38:52] I have s3 recentchanges server depooled [02:38:54] anyway [02:42:13] So yeah, I still have a few of the 503s I got open in my browser [02:42:21] they happen to come from cp1066 [02:42:22] e.g. [02:42:22] https://www.mediawiki.org/w/load.php?debug=false&lang=en&modules=ext.gadget.Edittools%2CcollapsibleTables%2Cenwp-boxes%2Csite%7Cext.uls.nojs%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.raggett%2CsectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles%7Cwikibase.client.i [02:42:22] nit&only=styles&skin=vector [02:42:23] don't get me wrong, I think there was a spike, but related to post-reload [02:42:38] no, definitely not. [02:43:12] It was mediawiki.org/load.php javascript/css responses (pages without styling), and index.php viewing of diffs on enwiki and officewiki. [02:43:37] Request from 10.20.0.103 via cp1066 cp1066 ([10.64.0.103]:3128), Varnish XID 2328639123. Error: 503, Service Unavailable at Sat, 14 Nov 2015 01:56:21 GMT. Forwarded for: **, 10.20.0.105, 10.20.0.105, 10.20.0.103 [02:44:32] Another one had the same error and IPs and hostnames (except different XID of course) for a GET on https://office.wikimedia.org/w/index.php?title=**&diff=next&oldid=** [02:45:14] Request from 10.20.0.166 via cp1066 [02:57:43] (03PS1) 10BBlack: text/mobile: use backend_random for non-GET/HEAD [puppet] - 10https://gerrit.wikimedia.org/r/253069 [02:59:39] (03PS2) 10BBlack: text/mobile: use backend_random for non-GET/HEAD [puppet] - 10https://gerrit.wikimedia.org/r/253069 [03:23:21] (03CR) 10BBlack: [C: 032] text/mobile: use backend_random for non-GET/HEAD [puppet] - 10https://gerrit.wikimedia.org/r/253069 (owner: 10BBlack) [03:38:37] any objections out there to https://gerrit.wikimedia.org/r/#/c/253063/3 ? [03:42:07] will let it sit since it's not apparently critical, but we should probably fix that :) [03:45:50] is there any backup from where to restore an overwritten grafana dashboard? [03:46:04] I don't think so [03:46:29] you can do "export" from the setting icon to save the JSON though [03:46:37] (to backup one up for yourself locally) [03:49:02] there is a cli tool to do that recursively, was thinking of running that in cron with git push [03:50:08] unless someone already has something of similar result [03:50:24] I don't think so [04:08:32] RECOVERY - cassandra-b CQL 10.192.16.163:9042 on restbase2001 is OK: TCP OK - 0.036 second response time on port 9042 [04:52:08] A user wants me to help him get http://is.gd/iaHkmC deleted..but It is reported on en-wiki and commons as not ever existing, despite on the User's end it existing on multiple browsers. Any ideas what's happening? [04:53:26] Nevermind...it is working now...Weird. [04:55:31] cache probably [04:56:48] c, The user tested it on multiple Browsers. I'm guessing there was some sort of server lag. Doesn't matter now as the user wanted it deleted anyway. [06:15:02] (03CR) 10Krinkle: "Can we switch to hostnames here? I heard something about us having HHVM's dns cache enabled. See also I93aa4df." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253063 (owner: 10Dzahn) [06:16:30] jzerebecki: bblack: There are backups for grafana dashboards. [06:16:40] I don't know where, but I recall ori setting htem up [06:17:04] Predating that, I often make exports to here https://wikitech.wikimedia.org/wiki/Grafana.wikimedia.org/ [06:30:22] PROBLEM - puppet last run on db2058 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:31] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:42] PROBLEM - puppet last run on mw2016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:51] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 3 failures [06:31:03] PROBLEM - puppet last run on mw2120 is CRITICAL: CRITICAL: puppet fail [06:31:08] oh dear [06:31:22] PROBLEM - puppet last run on mw2145 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:52] PROBLEM - puppet last run on restbase2006 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:23] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 3 failures [06:32:41] PROBLEM - puppet last run on mw2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:41] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:43] PROBLEM - puppet last run on mw2050 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:51] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:52] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:12] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:23] PROBLEM - puppet last run on wtp2008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:02] robh: [06:34:12] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 2 failures [06:34:43] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 2 failures [06:34:51] PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet has 1 failures [06:44:11] PROBLEM - puppet last run on mw2112 is CRITICAL: CRITICAL: puppet fail [06:49:22] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:01] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:55:23] RECOVERY - puppet last run on wtp2008 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:56:22] RECOVERY - puppet last run on db2058 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:56:22] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:56:23] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:32] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:56:41] RECOVERY - puppet last run on mw2016 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:56:42] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:43] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:56:43] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:56:51] RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:57:11] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:57:22] RECOVERY - puppet last run on mw2145 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:52] RECOVERY - puppet last run on restbase2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:04] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:32] RECOVERY - puppet last run on mw2018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:42] RECOVERY - puppet last run on mw2050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:02] RECOVERY - puppet last run on mw2120 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:13:52] RECOVERY - puppet last run on mw2112 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:17:12] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:01] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 1 failures [07:54:03] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [08:22:52] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: puppet fail [08:50:42] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:01:14] nothing like checking the channel for the morning puppet failures and thinking I'm late for work. on a saturday. [09:02:45] apergos: :) [09:39:13] <_joe_> apergos: ahahaha [10:34:02] PROBLEM - YARN NodeManager Node-State on analytics1030 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:39:31] RECOVERY - YARN NodeManager Node-State on analytics1030 is OK: OK: YARN NodeManager analytics1030.eqiad.wmnet:8041 Node-State: RUNNING [14:49:03] PROBLEM - YARN NodeManager Node-State on analytics1030 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:50:52] RECOVERY - YARN NodeManager Node-State on analytics1030 is OK: OK: YARN NodeManager analytics1030.eqiad.wmnet:8041 Node-State: RUNNING [15:41:11] PROBLEM - YARN NodeManager Node-State on analytics1030 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:44:52] RECOVERY - YARN NodeManager Node-State on analytics1030 is OK: OK: YARN NodeManager analytics1030.eqiad.wmnet:8041 Node-State: RUNNING [16:37:09] 6operations: sitemap.wikimedia.org ? - https://phabricator.wikimedia.org/T101486#1805667 (10Aklapper) > Should http://sitemap.wikimedia.org/ [...] have an actual sitemap on it? Who to make a call? Is this something to bring up on a Wikimedia mailing list or such? [16:56:07] 6operations, 6Security-Team: can we get rid of rsvg security patch? - https://phabricator.wikimedia.org/T104147#1805678 (10Aklapper) > `# Newer librsvg supports a sane security model by default and doesn't need our security patch` Still unclear which "newer" version is refered to and no references provided. h... [17:05:47] 6operations, 10Wikimedia-Apache-configuration, 10Wikimedia-Site-Requests, 5codfw-appserver-setup, 5wikis-in-codfw: Configure mediawiki to operate in the Dallas DC - https://phabricator.wikimedia.org/T91754#1805688 (10Aklapper) @Joe: Could you explain what is missing to close this task as resolved? [17:13:26] 6operations: Google Webmaster Tools - 1000 domain limit - https://phabricator.wikimedia.org/T99132#1805695 (10Aklapper) >>! In T99132#1642500, @dr0ptp4kt wrote: > @aklapper, I'm pinging on the thread. @dr0ptp4kt: Was there any outcome? [18:36:42] PROBLEM - YARN NodeManager Node-State on analytics1030 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:42:13] RECOVERY - YARN NodeManager Node-State on analytics1030 is OK: OK: YARN NodeManager analytics1030.eqiad.wmnet:8041 Node-State: RUNNING [18:51:08] [0e00fe2a] 2015-11-14 18:50:57: Fatal exception of type "MWException" [18:51:13] while trying to move a page on wikitech... [18:51:39] 2015-11-14 18:50:57 silver labswiki exception ERROR: [0e00fe2a] /w/index.php?title=Special:MovePage&action=submit MWException from line 220 of /srv/mediawiki/php-1.27.0-wmf.6/includes/Hooks.php: Detected bug in an extension! Hook SMWParseData::onTitleMoveComplete has invalid call signature; Parameter 3 to SMWParseData::onTitleMoveComplete() expected to be a reference, value given {"exception_id":"0e00fe2a"} [18:51:43] what more info do you need? i mean, it says clearly that the exception was of type mwexception, and you have a randomly generated hash to go with it [18:51:51] should that not be enough for pinpointing the problem? [18:53:01] lol [18:53:09] yeah, I was just pasting here [18:53:13] filed as https://phabricator.wikimedia.org/T118649?workflow=create [18:53:30] * legoktm goes back to the bug he was actually trying to investigate [18:53:30] legoktm: i was just sardonically deriding our error reporting, not actually criticizing you :P [18:53:43] oh :P [18:53:56] ori: "Something broke" [18:54:11] Reedy: no, it's more specific than that [18:54:17] "Something related to MediaWiki broke" [18:54:41] i.e., it was not a bicycle gear that broke, or a gas meter [18:54:52] the breakage was very definitely wiki-related [18:55:37] Do SMW care about 1.8.X? [18:55:46] it's really a UX issue; there may not be more information available that could be disclosed to the user [18:56:04] but in that case we should just say "an error occurred; here's a reference number you can use when asking for help about this problem" [18:56:31] instead of Fatal exception of type "MWException" [18:56:52] !bug 1 [18:56:52] https://bugzilla.wikimedia.org/show_bug.cgi?id=1 [19:04:42] PROBLEM - YARN NodeManager Node-State on analytics1030 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:09:28] 6operations, 6Security-Team: can we get rid of rsvg security patch? - https://phabricator.wikimedia.org/T104147#1805813 (10Reedy) >>! In T104147#1805678, @Aklapper wrote: >> `# Newer librsvg supports a sane security model by default and doesn't need our security patch` > > Still unclear which "newer" version... [19:10:13] RECOVERY - YARN NodeManager Node-State on analytics1030 is OK: OK: YARN NodeManager analytics1030.eqiad.wmnet:8041 Node-State: RUNNING [19:12:51] ori: I do have a patch for that https://gerrit.wikimedia.org/r/190379 [19:15:15] Nemo_bis: I'm not sure that's the right solution, without some kind of throttling / deduplication [19:15:37] That's pre-emptive optimisation. [19:15:53] no, it's capacity planning [19:16:55] well, anyways, let me back up for a second, and before quibbling with particulars, just say thanks for thinking about this issue and proposing a solution [19:17:18] :) [19:18:46] why do you think it's pre-emptive optimisation? isn't there a credible risk that a site outage would spill over to a phabricator overload? [19:19:14] either an actual server overload, bringing phabricator down, or phabricator stays up but there are 5,000 new tasks to clean up [19:19:39] First, if the outage is so severe and widespread, people won't be able to login in phabricator anyway. [19:19:48] Second, most people don't bother even trying to report. [19:20:02] Third, those who try usually don't bother logging in. [19:20:25] Fourth, only a minuscule minority of users manages to file reports even after logging in. [19:21:43] "AaronSchulz committed with atdt 2 days ago" [19:21:45] heh, github [19:23:00] Nemo_bis: [19:23:01] Ah. And at some point we had such a link, perhaps to the webchat for #wikimedia-tech, and the amount of people who ever clicked that was neglible. [19:23:03] The messenger started off at once, a powerful, tireless man. ... [But] how futile are all his efforts. He is still forcing his way through the private rooms of the innermost palace. Never will he win his way through. And if he did manage that, nothing would have been achieved. He would have to fight his way down the steps, and, if he managed to do that, nothing would have been achieved. [19:23:03] He would have to stride through the courtyards, and after the courtyards through the second palace encircling the first, and, then again, through stairs and courtyards, and then, once again, a palace, and so on for thousands of years. [19:23:03] And if he finally burst through the outermost door—but that can never, never happen—the royal capital city, the centre of the world, is still there in front of him, piled high and full of sediment. No one pushes his way through here, certainly not someone with a message from a dead man. [19:23:22] (Though maybe I'm wrong and we never replaced the nast irc:// link.) [19:23:56] (http://www.kafka-online.info/an-imperial-message.html -- second time a technical conversation reminds me of kafka in a week) [19:24:20] Ah yes, that story is nice. [19:25:23] ori: Krinkle said you set up backups of grafana dashboards. where can I find those? [19:27:02] jzerebec1i: I did, but I'm not sure where they end up. This is configured in https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/grafana.pp#L176-L178 [19:27:20] it gets rsync'd to a backup server i think [19:27:33] what did you lose? [19:28:27] ori: nothing yet [19:29:24] if you are worried about that, there is something you could do [19:30:05] there are two places where dashboards can be defined -- in the database (which does not keep a log of changes) but also in git [19:30:12] i added a grafana::dashboard resource [19:31:26] so if you want to combine the convenience of the web ui with the benefits provided by having the configuration in git, what you could do is create the dashboard using the web interface, tweak it to your heart's content, and then once you're reasonably happy with what you have, export the JSON, paste it into a file, and submit it to the puppet repo [19:32:17] * yuvipanda would love reviews on https://gerrit.wikimedia.org/r/#/c/253048/ [19:32:21] ( Nemo_bis -- incidentally, this also provides a means, though admittedly a clunky one, for volunteers who don't have privileged access to create dashboards ) [19:32:55] yuvipanda: looking [19:33:47] jzerebec1i: does that solve your problem? [19:34:14] i would really love for there to be a mediawiki storage backend for grafana [19:34:56] that way you get the versioning / diffing / attribution / protecting features of mediawiki [19:35:33] yuvipanda: is etherpad1001 jessie or ? [19:35:42] ori: jessie [19:36:54] ori: nice. yes that is probably better than what I had in mind: there is a cli tool to make a recursive export of an grafana installation. thought about adding it with git commit&&push to a cron. [19:44:02] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Puppet has 1 failures [19:44:42] PROBLEM - puppet last run on mw2099 is CRITICAL: CRITICAL: puppet fail [20:02:48] (03CR) 10Ori.livneh: [C: 04-1] etherpad: Add an autorestarter (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/253048 (owner: 10Yuvipanda) [20:10:03] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:12:33] RECOVERY - puppet last run on mw2099 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [20:18:59] 6operations, 10Traffic, 10Wikimedia-DNS: Consider DNSSec - https://phabricator.wikimedia.org/T26413#1805883 (10Vertigre) I second this issue. DNSSEC is a major improvement, especially for a site like wikipedia which needs to be constantly on guard for censorship and intimidation. Regarding the DNS server,... [20:30:33] (03CR) 1020after4: [C: 031] scap: Create wrapper script for master-master rsync [puppet] - 10https://gerrit.wikimedia.org/r/253040 (https://phabricator.wikimedia.org/T117016) (owner: 10BryanDavis) [20:40:23] !log reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Email header logging on unsubscribe (duration: 00m 53s) [20:40:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:42:52] PROBLEM - DPKG on mw1153 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [20:44:11] PROBLEM - Apache HTTP on mw1153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.003 second response time [20:44:52] PROBLEM - HHVM rendering on mw1153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.005 second response time [20:46:41] PROBLEM - HHVM processes on mw1153 is CRITICAL: PROCS CRITICAL: 0 processes with command name hhvm [20:47:09] Luke081515: go for the 3 clear rights, you'll be able to amend to add suppressredirect later when they know what they want [20:47:32] Dereckson: Ok :) [20:49:42] PROBLEM - Kafka Broker Replica Max Lag on kafka1018 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [5000000.0] [20:51:22] !log Deleted php-1.26wmf11, php-1.26wmf12 and php-1.26wmf13 from mira /srv/mediawiki-staging [20:51:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:54:45] 6operations: Cleanup root:root owned files on mira:/srv/mediawiki-staging - https://phabricator.wikimedia.org/T118657#1805929 (10Reedy) 3NEW [20:55:13] PROBLEM - Kafka Broker Replica Max Lag on kafka1018 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [5000000.0] [20:55:49] 6operations: Cleanup root:root owned files on mira:/srv/mediawiki-staging - https://phabricator.wikimedia.org/T118657#1805936 (10Reedy) [20:57:02] RECOVERY - Apache HTTP on mw1153 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 1.349 second response time [20:57:32] RECOVERY - DPKG on mw1153 is OK: All packages OK [20:57:42] RECOVERY - HHVM processes on mw1153 is OK: PROCS OK: 11 processes with command name hhvm [20:57:51] RECOVERY - HHVM rendering on mw1153 is OK: HTTP OK: HTTP/1.1 200 OK - 64246 bytes in 7.074 second response time [21:00:53] PROBLEM - Kafka Broker Replica Max Lag on kafka1018 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [5000000.0] [21:03:33] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [5000000.0] [21:07:21] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [5000000.0] [21:08:23] RECOVERY - Kafka Broker Replica Max Lag on kafka1018 is OK: OK: Less than 1.00% above the threshold [1000000.0] [21:09:07] (03PS1) 10Luke081515: Add a rollbacker group for wuuwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253084 (https://phabricator.wikimedia.org/T116270) [21:09:40] Dereckson: Done :) [21:11:41] (03CR) 10Dereckson: Add a rollbacker group for wuuwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253084 (https://phabricator.wikimedia.org/T116270) (owner: 10Luke081515) [21:12:52] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 87.50% of data above the critical threshold [5000000.0] [21:13:55] (03PS2) 10Luke081515: Add a rollbacker group for wuuwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253084 (https://phabricator.wikimedia.org/T116270) [21:20:12] RECOVERY - Kafka Broker Replica Max Lag on kafka1020 is OK: OK: Less than 1.00% above the threshold [1000000.0] [21:46:21] (03CR) 10Steinsplitter: [C: 031] Add a rollbacker group for wuuwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253084 (https://phabricator.wikimedia.org/T116270) (owner: 10Luke081515) [22:02:03] (03PS1) 10Ori.livneh: wmflib: add conflicts() function [puppet] - 10https://gerrit.wikimedia.org/r/253145 [22:02:05] (03PS1) 10Ori.livneh: Add redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/253146 (https://phabricator.wikimedia.org/T100714) [22:03:54] 6operations: Cleanup root:root owned files on mira:/srv/mediawiki-staging - https://phabricator.wikimedia.org/T118657#1805972 (10ArielGlenn) I've reset all of /srv/mediawiki-staging to mwdeploy:wikidev. Hope that wasn't overkill. [22:06:18] 6operations, 10Deployment-Systems: Cleanup root:root owned files on mira:/srv/mediawiki-staging - https://phabricator.wikimedia.org/T118657#1805974 (10Krenair) [22:09:59] !log reedy@tin Synchronized phpunit.xml: noop test (duration: 01m 03s) [22:10:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:10:15] 6operations, 10Deployment-Systems: Cleanup root:root owned files on mira:/srv/mediawiki-staging - https://phabricator.wikimedia.org/T118657#1805976 (10Reedy) 5Open>3Resolved a:3Reedy LGTM. Thanks! [22:22:09] !log reedy@tin Synchronized php-1.26wmf23: remove old localisation cache (duration: 01m 40s) [22:22:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:23:16] ValueError: /srv/mediawiki-staging/php-1.26wmf24/vendor/oyejorge/less.php/lib/Less/Version.php has content before opening 22:22:53 sync-dir failed: /srv/mediawiki-staging/php-1.26wmf24/vendor/oyejorge/less.php/lib/Less/Version.php has content before opening Sigh [22:23:32] unicode thingie? [22:23:39] (random guess) [22:23:42] Possibly [22:23:47] It's not visible [22:23:55] cat -vte is your friend [22:24:05] reedy@tin:/srv/mediawiki-staging$ cat /srv/mediawiki-staging/php-1.26wmf24/vendor/oyejorge/less.php/lib/Less/Version.php [22:24:05] Krenair: any idea how many old versions we actually need to be keeping around? [22:26:23] !log reedy@tin Synchronized php-1.26wmf24: remove old localisation cache (duration: 01m 38s) [22:26:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:26:59] Hmm, that's not doing it [22:31:13] 6operations, 10Traffic, 10Wikimedia-DNS: Consider DNSSec - https://phabricator.wikimedia.org/T26413#1806020 (10BBlack) [22:36:24] wooah, did that check actually catch something? [22:36:53] legoktm: apparently so [22:37:05] !log reedy@tin Synchronized phpunit.xml: (no message) (duration: 00m 30s) [22:37:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:39:28] :D [22:41:05] it must hav ebeen some other file [22:41:18] I fixed it [22:41:19] when I run the function in question directly on the file it does not whine [22:41:20] ah [22:41:20] dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy -- rm -rf /srv/mediawiki/php-STALE_VERSION" [22:41:22] :) [22:41:27] That won't work now... [22:41:28] heh [22:41:31] Ugh [22:41:40] trying to cleanup a load of crap laying around, and I can't [22:42:02] does sync-dir not propogate deletions? [22:42:08] uh [22:43:28] !log deleted more old l10ncaches from tin staging 1.27 wmf1-4 [22:43:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:43:44] Can't run salt as I'm not root either, right? [22:44:56] Reedy: dsh! [22:44:58] * hoo|away hides [22:45:03] hoo|away: I tried [22:45:10] Can't agent forward [22:45:25] You could do it locally [22:45:34] that would be rather awful, though [22:45:34] What, put my private key on the server? [22:45:39] And get shot by opsen? [22:45:48] it has --delete as one of the rsync args [22:45:52] No, from your machine with a local dsh [22:45:53] no salt for you [22:46:05] hoo|away: that sounds awful [22:46:08] (I'm not really serious on this one) [22:46:10] yeah :D [22:46:12] my ssh connections take an age to begin with [22:46:27] I have totally done ssh loops from my laptop on the cluster [22:46:33] sucked big time but it did work [22:46:54] hahaha [22:47:24] 486 /etc/dsh/group/mediawiki-installation [22:48:19] yeah [22:48:28] start the loop go drink coffee, watcha movie, come back [22:48:42] but be prepared that it might want to prompt you for keys you haven't accepted [22:48:47] hahaha [22:49:16] All these extra files laying around presumably just make scap longer than it needs to be [22:49:17] !log reedy@tin Synchronized php-1.26wmf24/: purge l10n cache... test if deletes are propogated (duration: 01m 12s) [22:49:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:49:28] they really should be [22:49:45] Nope, delete isn't being propogated :( [22:50:07] oh groan [22:50:17] well why not, the rsync should be doing it given the args [22:50:58] !log reedy@mira Synchronized phpunit.xml: (no message) (duration: 00m 29s) [22:51:00] is there a verbose option to the script so you can see what it runs? [22:51:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:51:13] 22:50:39 sync-master failed: Command '['sudo', '-u', 'mwdeploy', '-g', 'wikidev', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/cache/l10n/*.cdb', '--exclude=*.swp', '--no-perms', '--verbose', 'mira.codfw.wmnet::common', '/srv/mediawiki-staging']' returned non-zero exit status 23 [22:51:18] it's passing --delete [22:51:25] right [22:51:31] Reedy: you can use the mwdeploy key to use dsh [22:51:43] legoktm: just gotta load it? [22:51:46] https://wikitech.wikimedia.org/wiki/Dsh [22:52:29] aha [22:52:30] <3 [22:54:15] what were you hoping to see had been deleted? [22:54:19] !log dsh delete php-1.26wmf16 [22:54:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:54:28] apergos: I was trying to delete localisation cache files [22:54:35] for ancient versions [22:55:11] !log dsh delete php-1.26wmf17 [22:55:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:56:33] !log dsh delete php-1.26wmf18 [22:56:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:57:14] well looks like the dsh solution is getting it done [22:57:14] !log dsh delete php-1.26wmf19 [22:57:19] yeah [22:57:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:57:38] I'm just clearing the old versions which aren't in staging, but are apparently in /srv/mediawiki on tin [22:57:45] !log dsh delete php-1.26wmf20 [22:57:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:58:12] !log dsh delete php-1.26wmf21 [22:58:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:58:24] so it rsyncs from mira to /srv/mediawiki-staging [22:58:41] yeah, so we can deploy from it [22:58:43] * apergos is probably too sleepy to think about this straight [22:58:44] or that's the plan [22:58:47] not sure how well it works :) [22:58:56] so mira is where you would have to delete things first I guess [22:59:28] it should sync tin -> mira first [22:59:28] 22:36:34 Started sync-masters [22:59:29] sync-masters: 100% (ok: 1; fail: 0; left: 0) [22:59:29] 22:36:45 Finished sync-masters (duration: 00m 10s) [22:59:29] 22:36:45 Started sync-proxies [22:59:29] sync-proxies: 100% (ok: 12; fail: 0; left: 0) [22:59:31] 22:36:48 Finished sync-proxies (duration: 00m 02s) [22:59:33] 22:36:48 Started sync-apaches [22:59:36] sync-common: 100% (ok: 468; fail: 0; left: 0) [22:59:38] 22:37:05 Finished sync-apaches (duration: 00m 17s) [23:00:07] !log dsh delete php-1.26wmf22/cache/l10n/* [23:00:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:00:23] well looking at the sync-master, it seeeemed it was syncing from mira to /srv/mw-staging [23:00:35] er loking at the rsync command you pasted earlier I mean [23:00:51] yeah definitely too brain dead to carry on a coherent conversation about this [23:00:58] 1 am. packing it in for the night [23:00:59] --exclude=**/cache/l10n/*.cdb' [23:01:07] that'll be why it's probably ignoring it :D [23:01:17] !log dsh delete php-1.26wmf23/cache/l10n/* [23:01:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:01:38] shouldn't sync them but shouldn't delete them either [23:03:06] !log dsh delete php-1.26wmf24/cache/l10n/* [23:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:04:27] !log dsh delete php-1.27.0-wmf.1/cache/l10n/* [23:04:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:04:48] !log dsh delete php-1.27.0-wmf.2/cache/l10n/* [23:04:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:04:56] !log I just took a massive shit [23:05:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:06:13] !log dsh delete php-1.27.0-wmf.3/cache/l10n/* [23:06:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:06:40] !log dsh delete php-1.27.0-wmf.4/cache/l10n/* [23:06:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:07:12] !log