[00:03:08] New review: Pyoungmeister; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49950 [00:03:24] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49950 [00:21:18] TimStarling: http://redis.io/commands/debug-segfault [00:21:34] best command ever [00:24:30] heh [00:37:13] RECOVERY - mysqld processes on db43 is OK: PROCS OK: 1 process with command name mysqld [00:39:37] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [00:41:16] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 2707 seconds [00:41:43] PROBLEM - Puppet freshness on amssq43 is CRITICAL: Puppet has not run in the last 10 hours [00:43:51] New patchset: Asher; "gitreview file" [operations/software/redactatron] (master) - https://gerrit.wikimedia.org/r/49956 [00:44:18] New review: Asher; "Patch Set 1: Verified+2 Code-Review+2" [operations/software/redactatron] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49956 [00:45:01] Change merged: Asher; [operations/software/redactatron] (master) - https://gerrit.wikimedia.org/r/49956 [00:46:12] New patchset: Asher; "Initial crud functionality + schema review + redaction defaults for trigger generation" [operations/software/redactatron] (master) - https://gerrit.wikimedia.org/r/49957 [00:52:22] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [00:54:10] New review: Asher; "Patch Set 1: Verified+2 Code-Review+2" [operations/software/redactatron] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49957 [00:54:10] Change merged: Asher; [operations/software/redactatron] (master) - https://gerrit.wikimedia.org/r/49957 [00:54:59] New patchset: Pyoungmeister; "db-pmtpa.php: db43 back in, db58 out" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49958 [00:55:58] New review: Pyoungmeister; "Patch Set 1: Code-Review+2" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/49958 [00:56:13] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49958 [00:57:27] !log py synchronized wmf-config/db-pmtpa.php 'removing db58 from db-secondary for upgrades and maria, replacing db43' [00:57:28] Logged the message, Master [00:58:34] notpeter: do you know anything about ishmael being broken by any chance? [00:59:14] binasher: nein [01:00:49] !log do-release-upgrade-ing on db58 [01:00:50] Logged the message, notpeter [01:06:30] PROBLEM - mysqld processes on db58 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [01:10:44] ok bedtime for me and mostly not here tomorrow (trading monday hoilday when I worked for off day on wednesday) [01:10:49] see folks later [01:16:40] New patchset: Pyoungmeister; "db58 mariafication" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49961 [01:17:18] PROBLEM - Host db58 is DOWN: PING CRITICAL - Packet loss = 100% [01:17:34] New review: Pyoungmeister; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49961 [01:17:45] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49961 [01:18:39] RECOVERY - Host db58 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [01:28:06] RECOVERY - mysqld processes on db58 is OK: PROCS OK: 1 process with command name mysqld [01:28:37] !log mw1161-1208 have wrong power setting, changing all to system.power.hotspare.enable 0 [01:28:41] Logged the message, RobH [01:30:39] heh, power is normalizing in c6, awesome [01:32:54] PROBLEM - MySQL Slave Running on db58 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Table ./eswiki/page_props is marked as crashed and should be [01:44:37] New review: Ryan Lane; "Patch Set 2: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49916 [02:03:16] New patchset: Faidon; "Revoke preilly's access" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49964 [02:03:37] New review: preilly; "Patch Set 1: Code-Review+1" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/49964 [02:07:16] New review: Faidon; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49964 [02:07:25] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49964 [02:22:32] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [02:24:56] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Wed Feb 20 02:24:39 UTC 2013 [02:29:12] !log LocalisationUpdate completed (1.21wmf9) at Wed Feb 20 02:29:11 UTC 2013 [02:29:15] Logged the message, Master [02:39:22] New patchset: Tim Starling; "New cluster-local key for me" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49966 [02:40:05] apergos: around? [02:40:13] New review: Tim Starling; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49966 [02:40:22] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49966 [02:40:33] 04:40, unlikely :) [02:40:39] i was gonna say... [02:40:53] you should know better than most [02:43:09] heh [02:45:31] TimStarling: your new and old keys are both absent? [02:46:06] jeremyb_: good point [02:47:05] New patchset: Tim Starling; "Maybe let the new key work" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49967 [02:47:54] New review: Tim Starling; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49967 [02:48:03] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49967 [02:50:35] New patchset: Catrope; "New cluster key for me" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49968 [02:53:05] New patchset: Faidon; "Revoke Ariel's second key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49969 [02:53:22] New review: Faidon; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49969 [02:53:33] !log LocalisationUpdate completed (1.21wmf10) at Wed Feb 20 02:53:32 UTC 2013 [02:53:35] Logged the message, Master [02:54:49] New review: Jeremyb; "Patch Set 1: Code-Review+1" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/49968 [02:54:49] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49969 [03:03:05] I have also regenerated my Gerrit/github key, and shredded the old one [03:03:10] And shredded the old fenari key [03:03:24] Which means I'm locked out of the cluster until my key change commit is merged [03:04:40] Patrick R. is leaving WMF? [03:04:46] yes. [03:04:52] Hi Eloquence. [03:05:07] Susan's eager to update wmfwiki? [03:05:07] hi max [03:05:17] jeremyb_: It's already been updated! [03:05:24] oh, wow [03:05:32] i should pay more attention to your bot [03:05:39] jeremyb_: I didn't see it on the lists. That's why I was curious whether it was a mistake. [03:07:01] Susan: You might have noticed the commits above to operations/puppet [03:07:12] I haven't been paying attention in here. [03:08:02] Hi Roan. [03:08:35] damnit, i hate that it's a template. ([[wmfwiki:staff and contractors]]) [03:08:44] RoanKattouw: Did you see wikitech-l re: using wiki pages as databases? [03:08:52] No, I'm behind on wikitech-l [03:08:54] * jeremyb_ went to the history of the wrong page [03:08:55] jeremyb_: It's supposed to be localized. [03:08:56] Haven't read it since Christmas [03:08:58] (bad Roan) [03:09:08] Susan: i know. i saw some recent debate about htat [03:09:09] that [03:09:16] RoanKattouw: http://lists.wikimedia.org/pipermail/wikitech-l/2013-February/066656.html [03:10:03] I have no strong opinions about that and defer to Tim's opinion [03:14:38] New patchset: Ryan Lane; "Changing my production key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49971 [03:16:25] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49971 [03:16:34] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49971 [03:21:49] RoanKattouw: Well, I was just thinking at dinner that maybe we need Starling's Law or something. [03:22:16] New review: Faidon; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49968 [03:22:25] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49968 [03:23:02] RoanKattouw: "No matter how intricately designed or implemented, Wikimedians will find a way to mis-use or abuse a technology in infuriating and horrifying ways and with an astonishing pace." [03:23:09] I'm still working on it. [03:25:29] Susan: Mediawiki is just teh suxx0rz at structured data. We need to code a database extension. AKA: Wikidata? :-) [03:26:02] Heh, I know, right? [03:26:22] That thread was a little painful. [03:34:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.691 seconds [04:14:06] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:20:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.062 seconds [04:25:30] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds [04:32:06] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [04:32:33] PROBLEM - ircecho_service_running on neon is CRITICAL: Connection refused by host [05:06:27] RECOVERY - MySQL disk space on neon is OK: DISK OK [05:07:21] RECOVERY - ircecho_service_running on neon is OK: PROCS OK: 2 processes with args ircecho [05:25:14] New review: Ori.livneh; "Patch Set 1: -Code-Review" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49678 [05:31:25] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [05:32:55] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 0.027 second response time on port 8123 [05:33:01] New patchset: Tim Starling; "Re-enable Scribunto profiler" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49976 [05:33:26] New review: Tim Starling; "Patch Set 1: Verified+2 Code-Review+2" [operations/mediawiki-config] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49976 [05:33:27] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49976 [05:34:43] !log tstarling synchronized wmf-config/CommonSettings.php 're-enable Scribunto profiler' [05:34:43] Logged the message, Master [05:39:13] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [05:52:48] TimStarling: how is lua going? [05:53:12] the wikipedians are finally starting to think about using it, which is nice [05:53:22] yesterday they seemed to be afraid to touch it [05:53:40] I guess Citation usage and ramp up is the next milestone perhaps [05:53:50] yeah, AllPages for Module: has built up a tad [05:53:55] it was quite that first day [05:54:01] and the wiktionarians? [05:54:19] let's see what it looks like there [05:54:21] http://en.wikipedia.org/w/index.php?namespace=828&tagfilter=&title=Special%3ARecentChanges [05:54:38] to judge by that, it won't be long before we have lua versions of citations and convert on enwiki [05:55:09] the wiktionarians were less shy, but one of them uncovered an unfortunate performance edge case, per wikitech-l [05:55:10] https://en.wiktionary.org/w/index.php?namespace=828&tagfilter=&title=Special%3ARecentChanges [05:55:32] oh, i only saw the IRC discussion. let me search wikitech-l [05:55:48] I'm trying to compose a reply to code cat as to why it's faster to write your own parser than to use lua tables [05:56:29] * Aaron|home recalls the day that RC links like that first one would have timed out [06:01:13] I changed the code to use a two queries and UNION the result to make use of an existing index and felt giddy joy when querying the Portal namespace became reasonable [06:01:18] * Aaron|home was easily impressed [06:02:00] TimStarling: anyway, when does CodeEditor start to choke on decent client hardware? [06:02:29] I definitely noticed slowdown on that language module thing, not unusable though [06:03:11] I'd hate to disable the highlighting at some threshold, it seems better to force people to break up the code a bit [06:03:31] * Aaron|home imagines MediaWiki as one big PHP file [06:03:52] CodeEditor seems to do alright [06:04:02] the thing that fails terribly is firefox with geshi [06:04:20] it spends about a minute frozen, trying to apply the CSS rules [06:04:45] I assume it is CSS, chromium has CSS profiling and it was slow enough there to be suspicious [06:05:46] any particular rules stand out? [06:09:11] just running profiling in chromium again, it is saying different things this time [06:09:40] it may actually be JS [06:13:25] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [06:14:04] I don't know, it's hard to make sense of this tool [06:14:55] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 0.027 second response time on port 8123 [06:14:57] need perf or something [06:15:01] JS causing re-rendering with a crapload of tiny DOM changes in a loop? [06:15:30] where is a page that's slow? [06:16:02] https://en.wiktionary.org/wiki/Module:languages [06:16:12] I think I tried it with JS off and found that it was still slow [06:16:16] trying it with CSS off now [06:16:16] yeah, "that language module thing" [06:16:35] maybe firefox is just slow [06:16:36] * Aaron|home hears is fan speed up [06:16:49] *his [06:17:49] huh, and no CodeEditor for view source :/ [06:18:04] my laptop went into swap [06:19:03] it takes a while in Opera (a bit less though) and doesn't "stop the world" so much, though it's still laggy [06:19:38] at the top of the CSS profile in chromium, there is .lua.source-lua .sy0, .lua.source-lua .br0, .lua.source-lua .st0 [06:20:13] 2.77, 2.75 and 1.73 seconds respectively [06:20:25] ugh, Chrome is worse than FF [06:20:29] for scrolling, not for the whole page [06:21:04] * Aaron|home likes how Opera doesn't freeze for this [06:21:13] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [06:21:15] well, the DOM has 437,213 nodes [06:22:41] $('span').length [06:22:41] RangeError: Maximum call stack size exceeded [06:22:52] hm. [06:22:58] have you heard the news? opera is going to become another webkit clone [06:23:07] so anything that's good about it will stop existing [06:23:30] * Aaron|home misses all the rock opera themed picture opera installation had [06:23:52] just what the word needs, another webkit clone :) [06:24:14] Chrome is by far the worst for this page (and I didn't scroll) [06:26:09] * Aaron|home tries IE [06:27:16] meh, in between chrome and FF there [06:29:47] it looks like SyntaxHighlight_GeSHi's lexer has odd ideas about how to handle that code [06:36:50] heh [06:36:55] ]] closes comment blocks [06:36:58] that could be a problem [06:39:03] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 194 seconds [06:39:21] try --[=[ [06:39:39] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 218 seconds [06:39:44] --[=[ hello ]=]-- [06:39:58] which on the plus side, looks very artistic [06:40:45] yes, very FILE_ID.DIZ [06:41:08] exactly [06:44:37] "The Lua comparison operators on strings (< and <=) use the C function strcoll which is locale dependent. This means that two strings can compare in different ways according to what the current locale is. For example, strings will compare differently when using Spanish Traditional sorting to that when using Welsh sorting. " [06:44:39] ugh [06:45:03] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 13 seconds [06:45:05] yeah, you must have missed the gerrit discussion about that [06:45:40] brad pushed a change for review that involved calling Lua's setlocale() to force a C locale [06:46:15] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [06:46:18] I pointed out that it is just a wrapper for the C library function of the same name [06:46:35] and so it will affect everything in the same process, even other threads [06:46:48] bbl [06:48:10] * Aaron|home raises an eyebrow at http://stackoverflow.com/questions/8715980/javascript-strings-utf-16-vs-ucs-2 [06:48:40] the "ugh" was about global state [06:48:52] but yeah I noticed that local thing [06:53:02] * Aaron|home reads http://mathiasbynens.be/notes/javascript-encoding [06:55:42] yes :( [06:56:10] http://stackoverflow.com/questions/3744721/javascript-strings-outside-of-the-bmp [06:56:23] lol the "UTF 16 curse" [06:56:57] maybe it's good that lua waited [07:00:35] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [07:04:50] not at 4:40 am I'm not around, paravoid (but not much today, swapped monday holiday for today) [07:10:28] I sure forgot to change that [07:16:02] RECOVERY - MySQL Slave Delay on db59 is OK: OK replication delay 0 seconds [07:33:53] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [07:36:44] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 26.73 ms [07:37:59] PROBLEM - NTP on mw1085 is CRITICAL: NTP CRITICAL: Offset unknown [07:39:52] RECOVERY - NTP on mw1085 is OK: NTP OK: Offset -0.001426935196 secs [07:46:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:02] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [07:57:28] New review: Danny B.; "Patch Set 2: Code-Review+1" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/49681 [08:03:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [08:05:22] New review: Danny B.; "Patch Set 1: Code-Review-1" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/49802 [08:13:14] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [08:15:20] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [08:17:17] PROBLEM - Puppet freshness on labstore3 is CRITICAL: Puppet has not run in the last 10 hours [08:21:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:47] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 185 seconds [08:31:50] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 210 seconds [08:35:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.055 seconds [08:40:50] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 11 seconds [08:41:44] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [08:58:34] when wikidata goes read-only later today will https://www.wikidata.org/wiki/MediaWiki:Readonlywarning be displayed for users? [09:20:26] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Wed Feb 20 09:20:08 UTC 2013 [09:23:26] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Wed Feb 20 09:23:00 UTC 2013 [09:26:15] Depends how it's set read-only, I suppose. [09:26:43] You can set the variable to a string to display a custom message or you can set it to true to show that MediaWiki page, I think. [09:30:47] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 196 seconds [09:31:04] Susan: Right but normally that only shows up in the edit window right? [09:31:10] Wikidata doesnt have an edit window... [09:31:32] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 214 seconds [09:41:55] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [09:43:07] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [09:43:34] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [09:44:10] PROBLEM - ircecho_service_running on neon is CRITICAL: Connection refused by host [09:48:09] hello [10:13:07] RECOVERY - MySQL disk space on neon is OK: DISK OK [10:13:16] RECOVERY - ircecho_service_running on neon is OK: PROCS OK: 2 processes with args ircecho [10:49:00] New patchset: QChris; "(bug 35802) Stop applying unwarranted ellipses to email subjects" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49987 [10:56:38] qchris: :-] [10:56:53] Hi hashar :-) [10:57:22] qchris: it is good to see Chad having a new Gerrit friend :-] [10:57:35] can't wait to see the bugzilla/gerrit integration [10:57:53] The plugin is there ... [10:58:11] so it cannot take too long until it's done [10:58:20] are you coming to SF next week? [10:58:27] But there are few other things on the agenda before [10:58:34] no. [10:58:40] I suppose you are going? [10:58:44] yeah I am [10:59:00] One less in my time-zone :-( [10:59:02] I have ton of people to interview for continuous integration [10:59:20] I am afraid you will be a bit lonely during european morning : / [10:59:32] though there are the wikidata folks from Wikimedia Deutschland ! [10:59:38] :-) [10:59:39] they hide in #wikimedia-wikidata [10:59:46] So we're not alone after all. [11:15:39] PROBLEM - Apache HTTP on mw1085 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:06] PROBLEM - SSH on mw1085 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:23] New patchset: Ori.livneh; "EventLogging: fix test2wiki configs; +CodeEditor" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49991 [11:19:50] I hate java really [11:29:18] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [11:35:00] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 198 seconds [11:36:48] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [11:38:46] New patchset: Hashar; "(bug 38114) gerrit: alternate change list row colors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49993 [11:39:28] New review: Hashar; "Patch Set 1:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49993 [11:39:49] lunchhh [12:02:12] PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused [12:36:51] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 198 seconds [12:37:09] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 209 seconds [12:38:40] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [12:38:57] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [12:49:51] New review: QChris; "Patch Set 1: Code-Review-1" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/49993 [12:55:36] PROBLEM - ircecho_service_running on neon is CRITICAL: Connection refused by host [12:56:35] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [13:03:11] New review: Hashar; "Patch Set 1:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49993 [13:05:41] New patchset: Hashar; "(bug 38114) gerrit: alternate change list row colors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49993 [13:05:41] qchris: thanks for test of alternate row coloring [13:05:50] qchris: I have amended the change ^^^^ [13:05:58] * qchris looks again :-) [13:07:23] New review: QChris; "Patch Set 2: Code-Review+1" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/49993 [13:07:35] Tables in gerrit are annoying [13:07:43] bah [13:07:44] http://integration.wmflabs.org/gerrit/#/q/status:open,n,z [13:07:50] Somtimes they even come with invisible extra rows/columns :-( [13:07:50] that is not working on my install :( [13:08:18] Nice theme :-) [13:08:35] s/Tables in // [13:08:45] :-D [13:09:37] oh my god [13:09:40] gerrit I hate you [13:10:06] it start up with the wrong jar [13:10:07] :( [13:10:08] /var/lib/gerrit2/review_site/bin/gerrit.war [13:17:48] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 181 seconds [13:18:17] <^demon> It always starts with the one in ./bin [13:18:42] PROBLEM - MySQL Slave Delay on db44 is CRITICAL: CRIT replication delay 187 seconds [13:20:01] ^demon: I start it using the init.d script which loads /etc/default/gerritcodereview that got: GERRIT_WAR="/var/lib/gerrit2/review_site/bin/gerrit.war" [13:20:01] ;-D [13:20:10] not sure why I had the .war in /var/lib/gerrit2 [13:20:26] <^demon> The package installs the war there for installation purposes. [13:21:04] <^demon> (So as to not overwrite the running .war until you do init) [13:21:29] qchris: the table alternance does not work properly :( [13:21:39] ^demon: yeah that make sense :-] [13:22:02] qchris: on the general view http://integration.wmflabs.org/gerrit/#/q/status:open,n,z I got the first change with a white background [13:22:15] * qchris takes a look [13:22:22] but in "My Changes", I got the first one with gray ;-] [13:22:27] not sure how to link you to it hehe [13:22:48] <^demon> You're not supposed to link to those anymore. [13:23:14] ahh [13:23:19] I'll apply it locally again [13:23:43] that is a bit crazy, I am not sure how they count their rows hehe [13:24:20] <^demon> Make sure we're not overriding anything in GerritSite.css [13:24:37] AHHH [13:24:39] <^demon> Stuff in there is !important so would override theme settings. [13:24:46] so turns out that is just a huge table [13:25:23] .changeTable tr:nth-child(odd) { background: #EEE; } [13:25:31] but then you have headers inserted in the .changeTable [13:25:43] and can have either and odd or even number of rows before [13:25:50] so after a header, you could get a odd or even row :( [13:26:39] Can we pick different colors for header and grey rows? [13:27:15] RECOVERY - MySQL disk space on neon is OK: DISK OK [13:28:09] RECOVERY - ircecho_service_running on neon is OK: PROCS OK: 2 processes with args ircecho [13:30:04] New review: Hashar; "Patch Set 2: Code-Review-1" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/49993 [13:30:10] gave a long comment https://gerrit.wikimedia.org/r/#/c/49993/ [13:30:17] qchris: yeah different colors I guess :( [13:30:23] what about pink for headers? :-]]]]]]]]]]]]]]]]]]] [13:30:39] \o/ [13:31:55] http://www.colorhexa.com/eed2ee [13:32:16] everyone is going to hate me [13:35:08] qchris: I will let Krinkle respond on the bug report [13:35:21] ok [13:35:23] * Krinkle hears bleep [13:35:28] though #eed2ee is nice :-] [13:36:11] Krinkle: we were talking about colors alternance in the gerrit changes lists [13:36:38] Right [13:37:05] Krinkle: headers are #EEE, both headers and changes are all part of the same table. By picking up #EEE as an alternate color, you can end up with a header and the change next to it having the same color (#EEE) [13:37:19] When I tested this locally I used :nth-child(odd/even) in the console as well, but I'm assuming that is not what it will use in production since those tables are spanning multiple "user" tables, (they are technically one big table) [13:37:21] I know that [13:39:54] and so that looks odd http://bug-attachment.wikimedia.org/attachment.cgi?id=11812 [13:39:55] ;D [13:41:01] yeah [13:41:11] Niklas was talking about patch lists though, not change lists. [13:41:16] (he even corrected himself) [13:41:24] that's why I suggested #eee as it would be fine in there [13:41:34] They're headings anyway [13:41:38] they're all white now [13:44:54] * qchris will take a look in the aftenoon to see whether we cannot just fix the counting of rows upstream [13:46:08] RECOVERY - MySQL Replication Heartbeat on db44 is OK: OK replication delay 0 seconds [13:47:11] RECOVERY - MySQL Slave Delay on db44 is OK: OK replication delay 0 seconds [13:49:35] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 187 seconds [13:49:44] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 190 seconds [14:18:23] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [14:18:41] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [14:19:54] New review: Mark Bergsma; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49831 [14:20:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49831 [14:22:00] mark: I will boot a new instance and use puppetmaster::self to have a clean change. [14:22:17] mark: I sense that patch is not enough yet to have an upload cache in beta :-/ [14:23:28] which patch? [14:26:37] mark: the one you just merged ( https://gerrit.wikimedia.org/r/#/c/49831/ ) which tweaks the lvs config for labs [14:26:51] hehe [14:26:59] just noticed the upload role has sda / sdb devices [14:27:10] we need to abstract that [14:30:49] something like $role::cache::configuration::disk[$::realm] ? :-D [14:31:04] hmm not necessarily [14:34:43] warning: peer certificate won't be verified in this SSL session [14:34:43] Feb 20 14:23:40 deployment-cache-upload-test puppet-agent[4355]: Did not receive certificate [14:34:44] [0;36mnotice: Did not receive certificate[0m [14:34:45] Feb 20 14:23:51 deployment-cache-upload-test dhclient: DHCPREQUEST of 10.4.1.80 on eth0 to 10.4.0.1 port 67 [14:34:46] bahhh [14:35:08] I should get my own virtual boxes [14:40:38] haha [14:40:43] that's what I dooooooo [14:41:02] PROBLEM - ircecho_service_running on neon is CRITICAL: Connection refused by host [14:41:47] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [15:12:02] RECOVERY - MySQL disk space on neon is OK: DISK OK [15:12:29] RECOVERY - ircecho_service_running on neon is OK: PROCS OK: 2 processes with args ircecho [15:16:03] puppet1.pmtpa.wmflabs died in labs, not sure whether mark / Coren can do anything about :-D [15:16:25] Not yet. Sorry. [15:18:08] hm [15:18:12] i just created that [15:18:26] wait, did you have one called puppet1, hashar? [15:18:32] ohh [15:18:40] that just sounded like the puppet master in labs died :] [15:18:59] oh is puppet1.pmtpa.wmflabs the puppet master? haha [15:19:05] na should be virt0 or something [15:19:09] hm [15:19:15] right [15:19:28] do we both have instances named puppet1 now? [15:20:14] virt0 is just named virt0 [15:20:17] it's not an instance [15:20:21] it's a physical host [15:20:39] New patchset: Platonides; "Configuration for webtools-apache VMs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50011 [15:20:55] ja, right [15:21:36] puppetmaster is not running on virt0 [15:22:16] hmm, maybe it is, service just says it isn't [15:22:49] right, you should check for other webservers [15:23:08] ? [15:24:12] puppetmasters can be their own service or can be run under apache or whatever webserver someone chose [15:24:27] anyway, the entry point is always HTTPS [15:40:23] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [15:50:39] New patchset: Platonides; "Configuration for webtools-apache VMs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50011 [15:56:56] New patchset: Platonides; "Handling of packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50015 [15:57:07] New patchset: Demon; "Update gerrit /etc/default/* for init script" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50016 [15:59:57] New review: Platonides; "Patch Set 1: Code-Review+1" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/50016 [16:22:40] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [16:36:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.034 seconds [16:48:57] New patchset: Demon; "Make the # optional when listing RT and Bugzilla entries" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50017 [16:49:54] <^demon> 50016 and 50017 are super trivial if someone's got a minute. [16:51:55]