[10:01:04] How to track down a specific PoolCounter hit? [10:01:39] I just got a Pool-timeout error on https://meta.wikimedia.org/wiki/Letter_to_Wikimedia_Foundation:_Superprotect_and_Media_Viewer and I have no idea which pool is at fault [10:04:04] Or maybe someone can just look up poolcounter.log to see if there's something weird in the last few days/weeks on Meta :) [10:12:06] metawiki: Pool key 'metawiki:pcache:idhash:5415286-0!*!0!!it!*!*:revid:9761977' (ArticleView): Timeout waiting for the lock [10:12:09] I'm guessing that's you? [10:12:14] at 2014-09-05 09:53:03 [10:12:15] probably [10:12:22] hm [10:12:40] yes, timestamp matches my complaint on another channel [10:13:21] hmmm, there are a bunch across multiple wikis [10:15:10] doesn't look that unusual compared to the logs from earlier this month [10:25:10] Maybe it has to do with LST? [10:31:26] The subpages have been updated only at 9:55 https://meta.wikimedia.org/w/index.php?title=Special:Contributions/FuzzyBot&dir=prev&offset=20140905094952&limit=21&target=FuzzyBot [10:32:58] Unless it's the fault of 10 updated pages at 9:49 https://meta.wikimedia.org/w/index.php?title=Special:Contributions/FuzzyBot&offset=20140905095524&limit=15&target=FuzzyBot (reparsing the LST-transcluded page?) [16:33:09] Am I the only one for which the site CSS is broken? [16:33:37] I get bare HTML formatting without the skin [16:33:57] which wiki? it's working fine for me [16:34:09] meta, wikidata [16:34:12] not the only one [16:34:16] and now enwiki [16:34:26] if it helps, I'm on IPv6 [16:34:39] marktraceur was also seeing it [16:34:43] broken for me now [16:36:45] wfm, and I vpn'd to my home connection (instead of using the WMF SF office's direct line to the ULSFO cache) [16:37:11] Seems to be working now [16:37:28] yep [16:40:26] I see intermittent css problems too [16:40:40] Both on Wikidata and Commons [16:40:50] :/ [16:41:03] me too, on many wikipedia's too [16:41:11] 12:40 <+icinga-wm> PROBLEM - Varnish HTTP bits on cp1069 is CRITICAL: Connection timed out [16:42:19] omg [16:42:23] Carmela, legoktm: Aha, cool. [16:42:25] 12:41 <+icinga-wm> RECOVERY - Varnish HTTP bits on cp1069 is OK: HTTP OK: HTTP/1.1 200 OK - 212 bytes in 4.141 second response time [16:42:43] it's back [16:43:31] another varnish icinga error just now, though [16:44:45] marktraceur: I'm being redirected to ulsfo [16:44:55] (as in I get served by it) [16:45:31] JD|cloud: where are you physically? [16:46:18] greg-g: I'm on the evams cluster and seeing the same problems [16:46:51] me too [16:47:19] I saw it just now. esams [16:48:08] yeah, had some unstyled page too [16:48:09] greg-g: sent in PM [16:48:20] anyone getting this on eqiad? [16:48:23] howdy. What are the typical scenarios causing MW to emit pages with "? [16:51:14] abartov, looks like client-nojs is always given out, then replaced with 'client-js' if/when JS is run [16:52:05] abartov, are you using IE6 or something? [16:53:10] abartov, Actually I think the issue we're having with bits is causing JS to break as well. . . [16:55:16] Krenair: certainly not. Latest Firefox. [16:56:10] Krenair: oh. That's pretty spectacular breakage. The wiki is only semi-usable, quite apart from the looks. [16:56:28] I trust it's being worked on then, so no need for more information from me. [17:06:21] So, I’m getting pretty consistent 503s from bits, if anybody is interested. [17:07:05] yay consistency [17:12:47] Matthew_: yeah, we're on it (see /topic) [17:12:59] Matthew_: thanks though [17:13:25] greg-g: Sweet, thankee :) [17:16:25] greg-g, I'm guessing the consistant issues I get on firefox latest are at my end [17:16:37] had them for several months now [17:17:10] several months? which issues? [17:18:17] greg-g, js is a bit iffy, it rarely loads fully, although it is custom js, so may be something I've installed.... [17:28:54] greg-g: how serious is this? Do we need to communicate to users via tweet etc.? [17:29:26] * guillom was in a meeting and just noticed the IRC chatter. [17:31:34] Well, this looks quite drastic https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Bits%20caches%20eqiad&m=cpu_report&r=4hr&s=by%20name&hc=4&mc=2&st=1409938261&g=network_report&z=large [17:32:09] But over? [17:32:21] wow.... [17:38:21] anyone still see any problems ? [17:38:57] thedj, not really, gimme a sec [17:39:06] nope, all seems to be well here [17:54:46] what's wrong with geohack? [17:57:23] many things, usually [17:57:54] 404: https://tools.wmflabs.org/geohack/geohack.php?params=48.59_N_10.66_E [19:29:22] hey, is there a channel for app development? [19:29:33] (ie. where I can chat with someone involved with it?) [19:30:44] Cookies52: you mean mobile apps? [19:30:52] try #wikimedia-mobile [19:31:26] mutante, thanks