[00:09:58] !log Disabled donations queue consumption on aluminium [00:10:00] Logged the message, Master [00:13:56] AaronSchulz: Reedy: db12 doesn't have indexes on recentchanges or watchlist that aren't on the other enwiki dbs [00:14:17] hah [00:14:20] binasher: check db16 [00:14:46] maybe no one ever set up the query grouping for this after some switch [00:15:19] that is, set up the group to use a db with a covering index [00:15:47] is the creation of the index in a sql file somewhere in svn? [00:15:50] hmm, so db16 probably has a large revision index [00:16:02] no idea about RC/watchlists [00:16:12] db16 is in S7 though [00:16:18] maybe those were grouped for some other reason [00:16:36] 11:00 domas: reenabled db10, added db14 to s1, db9 given away to non-core tasks, added full contributions load to db16 (as it has covering index) [00:16:55] when is that from? [00:17:02] http://wikitech.wikimedia.org/view/Server_admin_log/2008-08 [00:17:25] AFBorche1t: that's 3.5 years old ;) [00:17:25] Guess things aren't suffering too much without it then [00:17:38] AFBorche1t: → AaronSchulz [00:17:56] and four days after domas did that, Tim: returned db16 to general load, a less critical role [00:18:29] http://dom.as/2007/01/26/mysql-covering-index-performance/ [00:18:37] binasher: that was 4 days BEFORE afais [00:18:39] New patchset: Bhartshorne; "shush!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2247 [00:18:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2247 [00:19:02] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2247 [00:19:03] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2247 [00:19:05] then a month later 11:51 Tim: schema update at 04:44 made db7 segfault. Replication stopped, watchlists stopped working after code referencing the new schema was synced. Switched to db16 for watchlist and RCL. Tried INSERT SELECT, that segfaulted too. [00:19:32] DaBPunkt: you're right [00:20:03] domas: copied in mysql build from db16 to db12 - db12 was running gcc-4.2 one, and in crashloop. next crash will bring up proper build :) [00:20:07] these are fun to read [00:21:27] it looks like db16 died in 2009 and was completely rebuilt [00:22:16] seems it went to s4 for a while.. 03:32 domas: new repl positions, s2: db30-bin.000015:1227, s4: db16-bin.019: [00:22:54] ok, so I must have misremembered, it seems it was only revision that had the index [00:23:05] !log asher synchronized wmf-config/db.php 'moving watchlist/recentchanges back to db12, returning db24 to s2' [00:23:07] Logged the message, Master [00:23:08] RC/watchlist were query grouped for some other reason [00:23:39] * AaronSchulz wonders why [00:27:26] does somebody know if that is a bug or a feature? http://commons.wikimedia.org/w/index.php?title=File:AaatestSonnepalmenstrand-portrait_new.jpg&action=history The page is upload=sysop protected and it works (I cannot upload with my test account). But the testaccount can revert to another file version.. [00:27:26] pretty stupid since I thought it is enough at upload edit wars to upload protect... [00:27:37] mediawiki displays the reverts as "uploads" - but apparently doesn't apply the protection status [00:27:58] we used to have covering indexes [00:28:01] then we forgot [00:28:07] then we had too much/too powerful hardware to care [00:28:09] best reason [00:28:11] domas: on RC too? [00:31:37] https://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/specials/SpecialWatchlist.php?r1=18692&r2=18901 [00:31:39] AaronSchulz around? [00:31:40] 'Watchlist query group.' [00:31:51] well that's helpful ;) [00:32:50] heh [00:33:55] gn8 folks [00:34:00] AaronSchulz http://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard#Rename_un-vanished_editor [00:34:09] when will you fix renameuser? [00:37:10] binasher: before wl_notificationtimestamp, watchlist was already covering via PK [00:37:23] so that explains why there was no special box [00:37:31] (for that table) [00:39:26] MBisanz: pester people about bug 31863 [00:40:26] heh [00:41:06] it's mostly tedious whack-a-mole work [00:42:10] !log updated production civicrm to r1293 [00:42:12] Logged the message, Master [00:42:47] thanks AaronSchulz [00:42:59] maybe people like ialex ;) [00:43:59] AaronSchulz, now is the perfect time to do it and get it in trunk [00:44:05] lol [00:44:34] it's a good 1.20 target [00:51:21] !b 31863 [00:51:21] https://bugzilla.wikimedia.org/show_bug.cgi?id=31863 [00:51:36] New patchset: Ottomata; "Scrapped Variable class." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2248 [00:51:38] New patchset: Ottomata; "Removing directories. A bit cluttery, I'll re-add when needed. Also,, backend is not a python package" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2249 [00:51:39] New patchset: Ottomata; "Mm, feeling good! AccessLogPipeline now able to be used without extending the class. Mmmm, prettier interface!" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2250 [00:52:43] thanks for no answer... hmpf [00:53:17] New patchset: Ottomata; "Removing unused user_agent1.py file. user_agent.py is left around for historical purposed until I feel ready to remove it." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2251 [00:57:59] !log re-enabled the donations queue consumer via Jenkins [00:58:00] Logged the message, Master [00:58:58] * jeremyb pushes MBisanz into #wikimedia-northeast [01:05:35] New patchset: Ottomata; "Meant to commit access_log.py before" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2252 [01:05:36] New patchset: Ottomata; "pipeline/base.py - adding documentation" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2253 [01:59:57] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2249 [02:00:21] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2248 [02:00:21] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2249 [02:00:22] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2248 [02:00:39] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2250 [02:00:40] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2250 [02:01:08] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2251 [02:01:08] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2251 [02:01:31] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2253 [02:01:50] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2252 [02:01:51] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2253 [02:01:51] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2252 [02:24:54] !log LocalisationUpdate completed (1.18) at Fri Feb 3 02:24:54 UTC 2012 [02:25:02] Logged the message, Master [03:06:21] RECOVERY - Disk space on srv219 is OK: DISK OK [03:06:21] RECOVERY - Disk space on srv223 is OK: DISK OK [03:17:49] zzz =_= [04:16:20] RECOVERY - Disk space on es1004 is OK: DISK OK [04:21:10] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:43:55] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [09:11:23] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [09:11:23] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [09:11:23] PROBLEM - Puppet freshness on mw65 is CRITICAL: Puppet has not run in the last 10 hours [09:45:38] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [09:54:18] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [10:03:38] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=64%): /var/lib/ureadahead/debugfs 0 MB (0% inode=64%): [10:06:32] New patchset: J; "add timedmediahandler files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2254 [10:08:28] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 451676 MB (3% inode=99%): [10:17:58] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 414417 MB (3% inode=99%): [10:26:58] RECOVERY - Disk space on srv219 is OK: DISK OK [11:38:01] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2216 [11:38:02] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2216 [12:12:07] New patchset: Mark Bergsma; "Revert "squid class not getting included for some reason. maybe this is a workaround?"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2255 [12:12:39] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2255 [12:12:52] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2255 [12:14:47] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2226 [12:14:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2226 [12:17:02] New patchset: Mark Bergsma; "Test what was up with include squid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2256 [12:17:21] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2256 [12:18:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2256 [12:18:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2256 [12:21:12] New patchset: Mark Bergsma; "Fully qualify manifests/swift.pp includes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2257 [12:21:38] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2257 [12:21:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2257 [12:39:01] New patchset: Mark Bergsma; "Create a [volatile] puppet fileserver module for volatile files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2258 [12:39:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2258 [12:39:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2258 [13:16:19] New patchset: Mark Bergsma; "Add support for squid config file serving by Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2259 [13:17:23] New patchset: Mark Bergsma; "Add support for squid config file serving by Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2259 [13:18:01] RECOVERY - Frontend Squid HTTP on cp1001 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.202 seconds [13:21:41] RECOVERY - Backend Squid HTTP on cp1001 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.196 seconds [13:28:15] New patchset: Mark Bergsma; "Add support for squid config file serving by Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2259 [13:52:36] New patchset: ArielGlenn; "move kiwix mirror contents to public/other directory for mirroring" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2260 [13:55:19] New review: ArielGlenn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2260 [13:55:20] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2260 [14:39:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2259 [14:39:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2259 [14:43:00] New patchset: Mark Bergsma; "Make mount volatile readable for the puppetmaster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2261 [14:43:17] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2261 [14:43:36] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2261 [14:43:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2261 [15:08:34] hmm, as this already been signaled ? "SqlBagOStuff::set" "Too many active concurrent transactions (10.0.6.50)" [15:08:43] has* [15:08:45] nope [15:08:50] What are you doing when you get that? [15:09:02] try to access a discussion pages [15:09:06] which site? [15:09:07] db40 [15:09:08] http://fr.wikipedia.org/wiki/Discussion:M%C3%A9ro%C3%AFtique [15:09:44] db40 isn't listed as being used in production... [15:09:51] it's ok now [15:10:01] strange :p a phantom server ? [15:10:25] CPU load has increased quite a lot in the last half an hour [15:10:30] as has network [15:12:14] ah, parsercache [15:13:11] New patchset: Mark Bergsma; "Don't run setup-aufs-cachedirs on squids that don't use aufs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2262 [15:13:36] !log reedy synchronized wmf-config/db.php 'Add comment that db40 is parsercache' [15:13:38] Logged the message, Master [15:14:23] Reedy> thank you for investigating it ;) [15:14:54] New patchset: Mark Bergsma; "Don't run setup-aufs-cachedirs on squids that don't use aufs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2262 [15:15:33] hi. is this supposed to be publicly viewable? i found it in your puppet configuration: http://ganglia.wikimedia.org/latest/ [15:16:02] yes [15:16:12] oh, ok. [15:17:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2262 [15:17:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2262 [15:24:13] New patchset: Mark Bergsma; "It's kinda useful to know that db40 is not "just" a core db" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2263 [15:25:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2263 [15:25:17] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2263 [15:30:32] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2230 [15:30:32] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2230 [15:40:43] New patchset: Dzahn; "enhance purge_all - area code API lookup one-liner :p" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2264 [15:42:01] New patchset: Dzahn; "enhance purge_all - area code API lookup one-liner :p" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2264 [15:44:04] New patchset: Dzahn; "enhance page_all - area code API lookup one-liner :p - option to skip an area" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2264 [15:44:21] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2264 [15:45:06] New patchset: Mark Bergsma; "Retab squid.xml, decommission knsq1-15" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2265 [15:45:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2265 [15:45:32] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2265 [15:45:32] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2265 [15:56:19] New patchset: Mark Bergsma; "Decommission knsq1-15" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2266 [15:56:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2266 [16:00:23] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2266 [16:00:24] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2266 [16:13:39] RECOVERY - Host db41 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [16:15:09] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:15] New patchset: Mark Bergsma; "Assign new ganglia aggregators" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2267 [16:16:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2267 [16:18:37] New patchset: Mark Bergsma; "Assign new ganglia aggregators" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2267 [16:19:01] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2267 [16:19:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2267 [16:26:28] mark: knsq1 - 15 are Dell PowerEdge 1950, aren't they? [16:26:44] yes [16:26:49] PROBLEM - Host db41 is DOWN: PING CRITICAL - Packet loss = 100% [16:29:19] mark: if the wmf doesn't need they anymore: whom I have to ask to get some for the TS? [16:29:35] you can have em [16:29:39] more than half are broken right now [16:29:43] we expect the rest will break soon [16:30:33] no need to talk to anybody, you can take em ;) [16:30:41] but then they're your problem, not mine [16:30:44] I won't service them anymore [16:30:58] mm, that doesn't sound good [16:38:19] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 0.006 seconds [16:46:09] RECOVERY - Backend Squid HTTP on cp1014 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.178 seconds [16:46:59] RECOVERY - Frontend Squid HTTP on cp1011 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.180 seconds [16:47:39] RECOVERY - Frontend Squid HTTP on cp1012 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.179 seconds [16:48:19] RECOVERY - Frontend Squid HTTP on cp1006 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.202 seconds [16:48:47] Hi need help updating side bar on te.wikipedia.org [16:48:59] RECOVERY - Frontend Squid HTTP on cp1019 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.180 seconds [16:48:59] RECOVERY - Backend Squid HTTP on cp1016 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.218 seconds [16:48:59] RECOVERY - Frontend Squid HTTP on cp1013 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.221 seconds [16:49:24] I tried updating mediawiki:Sidebar, but did not work. [16:49:32] Then I posted https://bugzilla.wikimedia.org/show_bug.cgi?id=34181 [16:50:29] RECOVERY - Backend Squid HTTP on cp1009 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.200 seconds [16:50:29] RECOVERY - Frontend Squid HTTP on cp1007 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.219 seconds [16:52:40] Hi Reedy [16:53:29] RECOVERY - Backend Squid HTTP on cp1017 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.178 seconds [16:54:09] RECOVERY - Frontend Squid HTTP on cp1003 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.179 seconds [16:54:58] RECOVERY - Backend Squid HTTP on cp1003 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.161 seconds [16:54:59] RECOVERY - Backend Squid HTTP on cp1015 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.200 seconds [16:54:59] RECOVERY - Frontend Squid HTTP on cp1018 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.179 seconds [16:56:03] Hi Tanvir [16:57:13] New patchset: Dzahn; "add account for Andrew Otto, add to host stat1 per RT 2375" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2268 [16:57:58] RECOVERY - Frontend Squid HTTP on cp1014 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.190 seconds [16:57:58] RECOVERY - Frontend Squid HTTP on cp1020 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.220 seconds [16:59:18] RECOVERY - Frontend Squid HTTP on cp1008 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.189 seconds [16:59:59] RECOVERY - Backend Squid HTTP on cp1005 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.200 seconds [17:00:45] New patchset: Dzahn; "add account for Andrew Otto, add to host stat1 per RT 2375 (alphabetical)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2268 [17:00:58] RECOVERY - Frontend Squid HTTP on cp1015 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.219 seconds [17:01:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2268 [17:01:38] RECOVERY - Frontend Squid HTTP on cp1010 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.163 seconds [17:01:38] RECOVERY - Frontend Squid HTTP on cp1004 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.219 seconds [17:01:38] RECOVERY - Frontend Squid HTTP on cp1016 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.182 seconds [17:02:06] He woosters can you help fix why sidebar is not getting updated on te.wikipedia.org [17:03:28] RECOVERY - Frontend Squid HTTP on cp1005 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.219 seconds [17:03:38] RECOVERY - Backend Squid HTTP on cp1018 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.178 seconds [17:03:38] RECOVERY - Frontend Squid HTTP on cp1009 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.183 seconds [17:03:59] arjunaraoc, when you don't read the language comparing 2 sets of text is difficult as hell [17:04:31] Hi Reedy [17:04:48] RECOVERY - Backend Squid HTTP on cp1020 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.178 seconds [17:05:04] thx. can you let me know where to look as the side bar is not getting updated [17:05:30] Usually a purge/null edit fixes this sort of caching issue [17:06:18] RECOVERY - Frontend Squid HTTP on cp1017 is OK: HTTP OK HTTP/1.0 200 OK - 27535 bytes in 0.189 seconds [17:07:29] Reedy I tried for the main page url with ?action=purge [17:07:34] did not work. [17:07:54] Am i on the right page, i am using MediaWiki:Sidebar [17:07:58] RECOVERY - Backend Squid HTTP on cp1010 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.188 seconds [17:08:56] it won't let me type english :( [17:09:08] RECOVERY - Backend Squid HTTP on cp1004 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.186 seconds [17:09:22] Reedy: Use Esc, then you can type english [17:09:38] RECOVERY - Backend Squid HTTP on cp1013 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.193 seconds [17:09:48] doesn't work [17:09:51] numbers do though [17:10:22] Which browser are you using? [17:10:28] RECOVERY - Backend Squid HTTP on cp1011 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.165 seconds [17:10:29] chrome dev [17:10:40] Can you use Firefox? [17:11:09] !log reedy synchronized wmf-config/InitialiseSettings.php 'touch' [17:11:11] Logged the message, Master [17:11:25] arjunaraoc, that looks correct now [17:12:15] Reedy: I am trying to remove current events as it is not in use. [17:12:28] RECOVERY - Backend Squid HTTP on cp1012 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.219 seconds [17:12:43] It is not getting updated even if I update the page. [17:14:19] I wouldn't recommend commenting stuff out [17:14:21] remove it completely [17:14:58] RECOVERY - Backend Squid HTTP on cp1008 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.187 seconds [17:15:06] !log reedy synchronized wmf-config/InitialiseSettings.php 'touch' [17:15:07] Logged the message, Master [17:15:16] Reedy: thx a lot, it worked. [17:15:26] one more help [17:16:00] Reedy: any help on https://bugzilla.wikimedia.org/show_bug.cgi?id=23804 [17:19:48] PROBLEM - Host db42 is DOWN: PING CRITICAL - Packet loss = 100% [17:20:33] arjunaraoc, that sounds more of a mw bug than a site bug [17:20:56] Reedy: actually the same thing works fine on English wikipedia [17:21:07] I don't know of any config thing for that [17:21:08] RECOVERY - Backend Squid HTTP on cp1007 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.198 seconds [17:22:23] Reedy: Can you bring it to the attention of other experts? I filed it long back. appreciate if it is fixed soon. [17:23:54] New patchset: Mark Bergsma; "Make gmetad restart upon config file changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2269 [17:24:11] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2269 [17:24:21] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2269 [17:24:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2269 [17:24:33] thx Reedy, bye [17:24:58] RECOVERY - Backend Squid HTTP on cp1019 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.199 seconds [17:25:28] RECOVERY - Backend Squid HTTP on cp1006 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.186 seconds [17:31:52] New patchset: Mark Bergsma; "Fix remaining file modes in ganglia::web" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2270 [17:32:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2270 [17:32:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2270 [17:32:17] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2270 [17:34:36] New patchset: Mark Bergsma; "gmetad's init script doesn't support status" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2271 [17:34:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2271 [17:35:01] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2271 [17:35:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2271 [17:48:15] New patchset: Mark Bergsma; "Add new squid servers cp1001-1020 to torrus" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2272 [17:48:46] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2272 [17:48:47] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2272 [17:52:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2268 [17:52:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2268 [17:56:12] New patchset: Mark Bergsma; "Fix indentation, modes of ganglia.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2273 [17:58:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2273 [17:58:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2273 [18:01:20] New patchset: Mark Bergsma; "Merge remote-tracking branch 'origin/production' into test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2274 [18:01:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2274 [18:01:54] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2274 [18:01:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2274 [18:03:52] New patchset: Mark Bergsma; "Revert "Merge remote-tracking branch 'origin/production' into test"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2275 [18:04:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2275 [18:22:35] RECOVERY - Host db41 is UP: PING OK - Packet loss = 0%, RTA = 0.17 ms [18:24:18] saper: btw, from this morning https://github.com/filbertkm/Scholarships/pull/15 [18:25:07] saper: hope that works, off for weekend [18:32:35] RECOVERY - Host db42 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [18:33:53] New patchset: Mark Bergsma; "Revert "Merge remote-tracking branch 'origin/production' into test"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2276 [18:34:21] Change abandoned: Mark Bergsma; "bad revert" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2275 [18:34:47] New review: Mark Bergsma; "THIS REVERT MAY NEED TO BE REVERTED ON THE NEXT test -> production MERGE!" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2276 [18:34:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2276 [18:37:16] mutante: sieht cool aus, danke! [18:41:55] PROBLEM - Host db41 is DOWN: PING CRITICAL - Packet loss = 100% [19:13:04] New patchset: Lcarr; "Changed xmit_hash_policy to a variable so that we can set it to any policy desired" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2277 [19:13:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2277 [19:18:29] New patchset: Bhartshorne; "add nagios to iptables, enable nagios to talk to swift via nrpe" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2278 [19:18:50] New patchset: Asher; "upgrading dbs 13,18,25,33" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2279 [19:19:08] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2278 [19:19:08] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2278 [19:19:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2279 [19:20:49] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2279 [19:20:50] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2279 [19:22:04] RECOVERY - RAID on ms-fe2 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [19:22:44] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [19:22:44] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [19:22:44] PROBLEM - Puppet freshness on mw65 is CRITICAL: Puppet has not run in the last 10 hours [19:23:27] !log asher synchronized wmf-config/db.php 'pulling dbs 13,18,25,26 for upgrades' [19:23:28] Logged the message, Master [19:25:04] RECOVERY - DPKG on ms-fe2 is OK: All packages OK [19:25:44] RECOVERY - Disk space on ms-fe2 is OK: DISK OK [19:30:34] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Fri Feb 3 19:30:10 UTC 2012 [19:32:34] RECOVERY - RAID on ms-fe1 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [19:36:01] hexmode: Jeff_Green: where's a good place to talk otrs? [19:36:28] jeremyb: Um, #wikimedia-otrs [19:36:35] MRB[homework]: um, no [19:36:55] MRB[homework]: unless you want to convince them to join... [19:37:04] RECOVERY - DPKG on ms-be1 is OK: All packages OK [19:37:17] jeremyb: OK, nevermind then :P [19:37:18] MRB[homework]: i'm willing to go to their preferred place to discuss [19:37:25] jeremyb: on the bugzilla ticket ? [19:37:37] hexmode: well, on irc i mean. here? [19:37:54] jeremyb: or maybe -operations [19:37:54] RECOVERY - DPKG on ms-fe1 is OK: All packages OK [19:38:34] RECOVERY - Disk space on ms-fe1 is OK: DISK OK [19:40:02] New patchset: Pyoungmeister; "having /etc/sudoers include sudoers.d will help a lot." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2280 [19:40:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2280 [19:41:04] RECOVERY - RAID on ms-be1 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [19:41:23] hexmode: your pick. idk if Jeff_Green is busy with something else. but maybe at least you have some answers since you were channeling him ;) [19:41:59] Jeff_Green: maybe I can help... what is your q? [19:42:06] hahaha [19:42:06] jeremyb: ^^ [19:42:15] i'm so confused [19:42:28] I am on the phone, though [19:42:33] je^I is more than one person [19:42:34] oh sorry, I see the backscroll now--I was on phone too [19:44:22] hexmode: Jeff_Green: well first of all, i can kind of guess what's in the RT but i don't really know. so, when you say jeff's going to do them but it may be a while i'm not sure what it is that's going to get done [19:46:40] ok--so here's what I know [19:47:03] Jeff_Green: so, idk what you've done to explore feasibility but i was hoping some of this could be done in labs with volunteer time [19:47:29] yeah that's actually something I suggested to someone somewhere along the way [19:47:58] so really all I know is that Philippe talked with the OTRS folks about the upgrade path and was told to expect a very long and involved project [19:48:19] the word "year" flew around [19:48:20] sure, i got the same message [19:48:33] It is somewhat worrying [19:48:35] so i guess there's 3 options [19:48:55] New patchset: Bhartshorne; "changed URL for nagios check to ms-fe to something that exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2281 [19:49:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2281 [19:49:17] 1) we give in and pay them 2) we make some tools to make the upgrade faster/smoother/repeatable and release them 3) we move to something else entirely [19:49:24] yeah [19:49:36] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2281 [19:49:37] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2281 [19:49:43] Is there major data structural changes or something? [19:49:58] yeah, some explanation of the year would be nice [19:50:07] yeah I don't have any of that information yet [19:50:10] You could just start with a new instance, migrate in our patches, and go from there [19:50:23] the idea of me getting involved was deferred while the fundraiser was going on [19:50:47] 5 * 8 * 52... 2000 man hours? [19:50:53] Jeff_Green: also, unrelated: how hard is it to get minor changes made to the current instance? i.e. i want to change something which i suspect is already a local hack [19:51:33] are our local hacks/patches tracked somewhere? [19:51:34] in theory it shouldn't be too hard--it's just perl and Tim made all the changes we've made as proper patches [19:51:36] soooo [19:51:39] WHO INVALIDATED THE CACHE [19:52:01] Jeff_Green: are those patches published? [19:52:06] domas: not I [19:52:12] jeremyb, they're all in svn [19:52:14] jeremyb: they're in SVN [19:52:21] mediawiki/trunk/otrs iirc [19:52:31] http://svn.wikimedia.org/viewvc/mediawiki/trunk/otrs/ [19:52:33] * jeremyb looks [19:52:48] Tim used 'quilt' to deploy them [19:52:53] the modern location for them would be wikimedia not mediawiki right? [19:53:06] Doesn't really matter too much [19:53:09] jeremyb: I was confused by that [19:53:32] my guess is that an upgrade would render those patches useless and we'd start over [19:53:37] but that's just a guess [19:53:47] Change abandoned: Pyoungmeister; "this would conflict with the /etc/sudoers file for all apaches :/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2280 [19:53:48] well at least we'd have a hit list of features that we want [19:54:01] but surely there would be lots of acceptance testing [19:55:29] yeah for sure [19:55:59] Jeff_Green: so, what now? ;) [19:56:03] Pffft, deploy and go to lunch! [19:56:13] jeremyb: cry or nap, you choose [19:56:41] seriously though, I think we need to get the OTRS folks, Philippe, me, and possibly a few others together to get a better understanding of the issues [19:56:57] You could evaluate the patches, and work out what would need reapplying/rewriting ontop of a more current release [19:57:00] and documented it somewhere ;) [19:57:19] sure, but the patches aren't what worries me [19:57:33]