[00:08:36] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: NRPE: Unable to read output [00:09:03] PROBLEM - mysqld processes on db1046 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:10:33] RECOVERY - mysqld processes on db1046 is OK: PROCS OK: 1 process with command name mysqld [00:15:59] New patchset: Pyoungmeister; "cleanup of apache.pp role file. merging labs and proc common role class." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15581 [00:16:32] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/15581 [01:35:40] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [01:39:35] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [01:40:46] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 238 seconds [01:41:31] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 282 seconds [01:47:04] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 615s [01:49:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [01:57:16] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 7 seconds [01:57:34] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 26s [02:37:37] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours [03:11:22] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Fri Jul 13 03:11:02 UTC 2012 [04:51:35] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [05:05:32] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [05:29:05] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [05:30:08] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [05:33:26] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [05:49:57] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.082 second response time [06:05:33] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [07:17:32] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [07:28:29] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [07:54:25] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [09:04:33] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [09:47:29] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [10:10:43] !log mw1016 - was down, reinstalling with precise [10:10:44] RECOVERY - Host mw1016 is UP: PING OK - Packet loss = 0%, RTA = 30.89 ms [10:10:52] Logged the message, Master [10:14:11] PROBLEM - SSH on mw1016 is CRITICAL: Connection refused [10:24:50] RECOVERY - SSH on mw1016 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [10:27:40] aren't those unused [10:27:41] ? [10:30:16] yes, they are still unused, but they kept going down because of the kernel bug and i thought they need to be reinstalled some time anyways, vs. just installing kernel upgrades [10:31:24] please don't [10:31:26] i saw some down again and thought now that precise is default installer, i could as well just boot into PXE , no? [10:31:31] there's no point in doing that for unused servers [10:31:34] just shut them down or something [10:31:44] ok [10:31:44] PROBLEM - NTP on mw1016 is CRITICAL: NTP CRITICAL: No response from NTP server [10:31:45] it's much better to reinstall them all at the same time when we're actually gonna use them [10:31:52] alright [10:31:54] there's much more useful stuff to do [10:42:14] RECOVERY - NTP on mw1016 is OK: NTP OK: Offset -0.03559303284 secs [10:45:09] RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Fri Jul 13 10:45:03 UTC 2012 [10:47:42] RECOVERY - DPKG on mw1102 is OK: All packages OK [10:48:00] RECOVERY - Disk space on mw1102 is OK: DISK OK [10:56:51] RECOVERY - Varnish HTCP daemon on cp1041 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [10:58:00] New patchset: Mark Bergsma; "varnishhtcpd depends on certain perl dependencies installed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15551 [10:58:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15551 [11:04:57] RECOVERY - NTP on mw1102 is OK: NTP OK: Offset -0.01451635361 secs [11:22:14] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15551 [11:36:36] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [11:36:54] RECOVERY - RAID on mw1102 is OK: OK: no RAID installed [11:40:39] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [12:38:19] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours [12:46:21] New review: Mark Bergsma; "Yes, ERB is just Ruby, so that would work." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/13304 [13:00:00] New patchset: Mark Bergsma; "Append the auto-install date to /etc/motd.tail even when it doesn't exist yet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15590 [13:00:36] New patchset: Mark Bergsma; "Remove outdated serial console stuff for hardy" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15591 [13:01:10] New patchset: Mark Bergsma; "Disable IPv6 privacy extensions during installation, before first boot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15592 [13:01:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15590 [13:01:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15591 [13:01:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15590 [13:01:43] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15592 [13:01:54] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15591 [13:02:15] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15592 [13:09:17] New review: Mark Bergsma; "Well, just checking for $test_wikipedia is not appropriate. It's not a local variable of varnish::in..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/15445 [13:10:40] New review: Mark Bergsma; "I don't know anything about the Labs setup, so I won't review this." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/15545 [13:12:37] New review: Catrope; "That doesn't work for me:" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15561 [13:13:09] New review: Catrope; "....but following the install steps, then running ./local-lint works. Thanks hashar :)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15561 [13:13:38] New review: Mark Bergsma; "This won't currently work. Right now, $::syslog_server doesn't exist as it's not set globally." [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/14090 [13:17:41] New patchset: Catrope; "Clean up the mess that is SSL certificate installation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15561 [13:18:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15561 [13:18:28] New review: Mark Bergsma; "Why the $labs variable?" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/15581 [13:19:51] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15552 [13:25:06] New patchset: Mark Bergsma; "Use much smaller disk caches to test the new LRU code" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15594 [13:25:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15594 [13:25:58] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15594 [13:29:45] PROBLEM - Varnish HTTP mobile-backend on cp1041 is CRITICAL: Connection refused [13:30:21] New patchset: Mark Bergsma; "Properly sort the hash keys to stop Puppet from changing it every run" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15595 [13:30:56] New patchset: Catrope; "Remove star cert on ekrem, it's unused" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15597 [13:31:15] RECOVERY - Varnish HTTP mobile-backend on cp1041 is OK: HTTP OK HTTP/1.1 200 OK - 698 bytes in 0.063 seconds [13:31:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15595 [13:31:28] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15595 [13:31:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15597 [14:02:57] New patchset: Catrope; "Add rewrites from secure.wikimedia.org to the new HTTPS URLs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15599 [14:03:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15599 [14:18:24] (Cannot contact the database server: Unknown error (10.0.0.221)) [14:18:44] @info 10.0.0.221 [14:18:44] Krinkle: Unknown identifier (10.0.0.221 [14:18:50] @info 10.0.0.221 [14:18:50] Krinkle: Unknown identifier (10.0.0.221 [14:18:52] strange [14:20:40] @info 10.0.0.221 [14:20:40] Krinkle: [10.0.0.221: ] pc1 [14:28:25] New patchset: Mark Bergsma; "Use the chash director for the mobile servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15601 [14:29:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15601 [14:29:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15601 [14:43:21] PROBLEM - Apache HTTP on mw16 is CRITICAL: Connection refused [14:47:51] RECOVERY - Apache HTTP on mw16 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [14:54:52] !log Added cp1041.eqiad.wmnet back into the mobile LVS pool [14:55:01] Logged the message, Master [14:55:04] !log Increased all mobile varnish server weights from 10 to 100 to aid chash [14:55:16] Logged the message, Master [15:06:36] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [15:07:01] !log unknown column 'af_is_featured' on en_labswikimedia@db25 [15:07:10] Logged the message, Master [15:14:24] hasharDeadmau5: do you need to know anything about that 'unknown column'? I can help sort that if it is a problem [15:14:41] chrismcmahon: na just logged it [15:14:52] chrismcmahon: I guess that is Roan forgetting to apply some schema change [15:14:57] he is most probably aware of it [15:15:24] hasharDeadmau5: Matthias Mullie probably knows about it if Roan doesn't [15:16:19] hasharDeadmau5: actually, Matthias is probably the one to talk to, but I don't see him around right now [15:21:06] I'll fix it [15:21:31] just need to go grab a drink [15:25:57] Lol [15:26:04] There's no transient patch for adding that to the schmea [15:28:23] hasharDeadmau5: disabled it for now with a note [15:30:50] Reedy: thanks! [15:52:20] hey ^demon, it seems that i have +2 rights on the analytics/udp-filters.git repo but not merge rights, is that even possible? [16:06:39] drdeem did you press the right button? [16:07:00] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [16:09:23] Platonides: I give +1 for Verified and +2 for CodeReview, it says approved but it didn't merge, see https://gerrit.wikimedia.org/r/#/c/15610/ [16:11:06] Did you tell it to merge? [16:11:18] Says merged though [16:11:42] because i asked ottomata to do +1 and +2 that's why it merged, you can see that in the history [16:11:47] of the comments [16:12:00] There's a +2 and a +2 and merge button IIRC [16:13:19] why would you want to give +2 and not merge? [16:20:37] i don't have a +2 and merge button [16:20:42] me neither [16:20:45] I did the sam that drdee said [16:20:51] +2 and +1 verified, submit review [16:20:54] then it merges [16:20:54] platonides: see https://gerrit.wikimedia.org/r/#/c/15611/1,publish [16:20:57] at least, after drdee [16:20:59] that's a new commit [16:21:12] i can do +1 / 0 / -1 for Verified [16:21:22] i can do +2 .. -2 for Code Review [16:23:25] nevermind, i press the wrong button,why are two publish buttons in gerrit and is the on the left not doing what you would expect it to do [17:18:44] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [17:29:52] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [17:52:02] New patchset: Lcarr; "icinga::monitor::apache class added" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15618 [17:52:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15618 [17:52:53] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15618 [17:54:55] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [17:57:37] RECOVERY - Puppet freshness on ms1 is OK: puppet ran at Fri Jul 13 17:57:24 UTC 2012 [18:18:28] PROBLEM - MySQL Slave Delay on db12 is CRITICAL: CRIT replication delay 208 seconds [18:19:04] PROBLEM - MySQL Replication Heartbeat on db12 is CRITICAL: CRIT replication delay 240 seconds [18:30:19] New patchset: Pyoungmeister; "cleanup of apache.pp role file. merging labs and proc common role class." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15581 [18:30:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15581 [18:44:24] New patchset: Lcarr; "create icinga rw directory" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15628 [18:44:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15628 [18:50:14] New patchset: Pyoungmeister; "renaming imagescaler nagios group so that it's consistent with all the others" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15630 [18:50:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15630 [18:50:59] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15628 [19:01:00] sorry folks [19:01:04] LeslieCarr: i'm getting a ton of pages from icinga [19:01:11] jesus wtf [19:01:12] lucky notpeter isn't though [19:01:16] yeah, turning up new icinga…. sorry ! [19:01:23] ok well [19:01:28] heart attack averted [19:01:30] yeah, fixing that [19:01:30] sorry [19:01:38] icinga's not even up. oh my [19:01:50] what's going on? [19:01:52] snmptt was having a circular crashing [19:01:54] nothing [19:01:56] nothng at all [19:01:56] oh [19:01:56] good [19:01:58] false alarm [19:02:06] db12 has lots of lag [19:02:16] isn't that watchlist? [19:02:17] unrelated, but a problem [19:02:19] it's always sad [19:02:27] not 600+ seconds bad [19:02:54] hah, just gave myself and everyone a heart attack [19:03:35] RoanKattouw, maplebed false alarm [19:03:37] sorry about the pages [19:03:46] hehehe [19:03:49] figured, but had to check, right? [19:03:53] what's the story? [19:03:54] * apergos wonders who else is gonna show up [19:04:46] oh was getting the new icinga up on precise [19:04:48] and it wasn't muted :( [19:04:55] ouch. [19:05:05] db12 is actually in trouble though [19:05:07] well, it works! [19:05:08] Its replag is in the 600s [19:05:14] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14154 [19:05:28] ok, going away again. [19:05:46] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [19:05:59] now only in the 400s [19:06:10] New patchset: Lcarr; "muting all config changes from puppet to prevent pages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15631 [19:06:47] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15631 [19:12:41] LeslieCarr [19:12:44] not sure what just happened [19:12:47] oh she's gone [19:12:59] ah there she is, LeslieCarr [19:13:07] not sure if your change there is related [19:13:07] but [19:13:15] all of our udp2log hosts just starting notifiying us about: [19:13:22] The command defined for service udp2log log age for oxygen does not exist [19:13:26] cmjohnson1: do you have any free time today to help with OS installs on the 15 new db servers that were racked a few weeks ago (that was in https://rt.wikimedia.org/Ticket/Display.html?id=2614) [19:13:31] ottomata: yeah, that was me :( [19:13:44] well that was me and neon [19:13:48] not the real server [19:13:49] sorry [19:14:13] ok [19:14:14] New patchset: Catrope; "Add rewrites from secure.wikimedia.org to the new HTTPS URLs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15599 [19:14:24] ottomata: should have stopped just now [19:14:25] ? [19:14:36] cmjohnson1: ah, ok [19:14:37] seeing if we get the OK notification [19:14:38] ... [19:14:46] RECOVERY - MySQL Replication Heartbeat on db12 is OK: OK replication delay 14 seconds [19:14:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15599 [19:15:13] RECOVERY - MySQL Slave Delay on db12 is OK: OK replication delay 0 seconds [19:26:16] New patchset: Pyoungmeister; "creating jobrunner nagios group" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15633 [19:26:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15633 [19:29:29] New review: Reedy; "Finally!!" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15599 [19:29:56] LeslieCarr [19:29:57] http://nagios.wikimedia.org/nagios/cgi-bin/extinfo.cgi?type=2&host=oxygen&service=udp2log+log+age+for+lucene [19:30:17] New review: Reedy; "Though, can't we just be lazy, and have a rule per project, and wildcard the language codes?" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15599 [19:30:20] wait not that one [19:30:31] oh i take it back [19:30:32] sorry [19:30:40] LeslieCarr all looks well [19:30:49] ok good ottomata [19:31:12] haven't received OK notices [19:31:17] but in web interface it looks fine [19:32:15] New patchset: Platonides; "$wgDBerrorLogInUTC -> $wgDBerrorLogTZ" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/15634 [19:37:01] apergos: see 15599. More progress to singers death! :D [19:38:00] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15555 [19:38:15] New review: Reedy; "And a couple of exceptions (a la Wikimania wikis)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15599 [19:39:46] yay! [19:39:52] kill it with fire [19:40:13] Hmm [19:40:19] I wonder what else needs migrating off it.. [19:40:41] public key list.. [19:45:12] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15581 [19:45:49] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15630 [19:45:59] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15633 [19:48:49] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [19:53:59] New patchset: Reedy; "Add keys for Sam Reed and Brion Vibber" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/15636 [19:54:26] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/15636 [20:20:08] hi ther [20:20:11] there [20:35:27] hey, do we have a spare machine for storing the Wiki Loves Monuments database? [20:38:55] PROBLEM - Puppet freshness on mw1016 is CRITICAL: Puppet has not run in the last 10 hours [20:49:04] RobH, LeslieCarr, or anyone else around? ^^ a bit of context: the mobile team needs to make this volunteer-maintained DB available for our app, but we've no time to reingineer it to make it work with WMF main Db cluster, so we would like to have a separate machine for this thing [20:52:37] MaxSem: do i recall correctly that you had also already talked with asher a bit about this? iirc he had suggested trying lucene for fulltext search (so we souldn't need a separate MySQL instance with MyISAM support) but failing that it might be possible to get a box set up for this purpose [21:37:24] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [21:39:53] MaxSem: the mobile apps will be hitting that directly? [21:40:25] you mean whether it will host Apaches? [21:40:45] idk what you mean [21:40:59] is this for e.g. a cron job or some other background process? [21:41:22] or is it for direct use when serving requests to mobile users? [21:41:27] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [21:41:31] (dynamically, on the fly) [21:41:50] MaxSem: also, MyISAM is evil, try to find another way [21:41:56] the latter [21:42:06] MaxSem: is there something written about what you need? or can I discuss it with maarten? [21:42:09] or who? [21:42:40] jeremyb, Maarten is one of devs [21:42:46] right [21:43:09] yeah myisam but we don't have the time to seriously reingineer it [21:43:34] so is the schema published somewhere? [21:44:14] the request is at https://rt.wikimedia.org/Ticket/Display.html?id=3221 [21:44:22] the schema is https://svn.toolserver.org/svnroot/p_erfgoed/erfgoedbot/sql/fill_table_monuments_all.sql [21:44:34] though it might be tweaked for WMF use [21:45:33] MaxSem: oh, wow just one table? [21:45:36] * jeremyb expected more [21:46:06] there might a be a couple more later but this one is main and most problematic [21:46:33] ideally, this thing should be done in NoSQL [21:47:42] hrmmmm [21:59:47] New patchset: Lcarr; "moving ganglia-web directory to new install" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15645 [22:00:17] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/15645 [22:01:56] New patchset: Lcarr; "moving ganglia-web directory to new install" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15645 [22:02:25] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/15645 [22:04:06] New patchset: Lcarr; "moving ganglia-web directory to new install" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15645 [22:04:41] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15645 [22:39:21] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours