[00:01:41] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 6.215 second response time [00:05:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:07:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:09:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:09:23] <^d> mutante: gitblit process flapping? [00:09:24] <^d> up for me [00:09:36] ^d: well i just saw Icinga above [00:09:47] <^d> I lost my scrollback, bleh [00:09:48] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 248560 bytes in 6.897 second response time [00:09:49] a couple days ago i killed the process [00:09:54] when it did that [00:09:56] there we go:) [00:10:17] ^d: recovered, let's listen to the speech instead:) [00:11:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:11:58] * Coren hates hates hates having to deal with keystores. [00:13:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:15:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:16:40] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [00:17:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:18:34] (03CR) 10Dr0ptp4kt: "@Ori.livneh - per our face-to-face, generic cookie setting out of the C subroutine, if it conferred performance or feature benefits for an" [operations/puppet] - 10https://gerrit.wikimedia.org/r/113935 (owner: 10Ori.livneh) [00:19:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:21:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:23:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:24:24] Oh, FFS, and the layout doesn't /quite/ match the documentation either. [00:25:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:27:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 11:58:06 PM UTC [00:28:08] RECOVERY - Puppet freshness on bast4001 is OK: puppet ran at Sat Feb 22 00:27:58 UTC 2014 [00:30:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 12:27:58 AM UTC [00:32:18] PROBLEM - Puppet freshness on bast4001 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 12:27:58 AM UTC [00:58:06] RECOVERY - Puppet freshness on bast4001 is OK: puppet ran at Sat Feb 22 00:58:03 UTC 2014 [01:05:14] !log Shutting down LDAP briefly on virt0 for a config switch [01:05:24] Logged the message, Master [01:09:32] !log LDAP on virt0 back up with new certificates (man, that was painful) [01:11:19] Oh, FFS. Switching the certificate didn't help. [01:11:52] Ah, maybe with the whole chain. [01:17:00] i dont envy your current task [01:17:09] i feel ssl empathy pain [01:17:41] matanya: I managed to switch the certificates but I can't figure out how to stuff the chain in the fucking keystore. I can force the check to veryfy against the right CA, but I don't know if that suffices. [01:19:45] That configuration is surprisingly intricate. There are four keystores, pointed to by bits in LDAP, with hard-coded cert aliases galore. [01:20:04] It's almost as bad as gridengine. [01:20:23] it's puppetized last I checked [01:20:32] so why not look at how puppet deals with them? [01:20:54] it uses keytool [01:20:55] Ryan_Lane1: Much of it doesn't actually manage to change anything past the initial install AFAICT [01:21:05] Or I missed the right bit. [01:21:47] there's only one keystore actually being used from what I remember [01:21:52] and all the config points to it [01:22:02] ah, there's two [01:22:06] PROBLEM - ElasticSearch health check on logstash1001 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.138 [01:22:19] Yeah, in cn=PKCS12,cn=Key Manager Providers,cn=config but that both hard codes the filename /and/ the certificate alias. [01:22:22] one for admin (4444) and one for the server (636) [01:22:24] add_ca_to_admintruststore [01:22:27] add_ca_to_truststore [01:22:36] PROBLEM - ElasticSearch health check on logstash1002 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.137 [01:22:36] PROBLEM - ElasticSearch health check on logstash1003 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.136 [01:22:42] all you need to do there is trust the CA [01:22:47] The truststores are uninteresting; they're not the *key* [01:23:17] And LDAP trusts the key that I switch to properly, as expected. The issue is that cert_check doesn't. [01:24:07] if the ldap clients and ldap itself works, why not fix the check? [01:24:28] Well, first I had to remove the star certificate. :-) [01:24:29] seems weird to go through so much trouble to have the chain delivered when the clients that use it don't care [01:25:15] The trouble was switching the /key/ :-) [01:25:21] from what I remember you just change out the PKCS12 file in the config [01:25:32] you are just referencing the PKCS12 file, right? [01:25:37] PROBLEM - ElasticSearch health check on logstash1002 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.137 [01:25:51] Yes; but puppet will create it okay but not change which one opendj uses. [01:26:05] you just need to ensure it has the same password [01:26:07] and replace the file [01:26:23] ... and the same alias. [01:26:37] And honestly, calling it 'star.wikimedia.org' would have been wrong on too many levels. :-) [01:26:40] heh [01:26:41] yeah [01:26:50] this is mostly a matter of running dsconfig [01:26:57] * Coren nods. [01:27:03] I did figure it out eventually. [01:27:06] PROBLEM - ElasticSearch health check on logstash1001 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.138 [01:27:06] it has a menu and everything ;) [01:27:21] I'm just annoyed that it didn't properly fix the cert check. [01:27:27] :D [01:27:33] yeah, I can understand that [01:27:36] PROBLEM - ElasticSearch health check on logstash1002 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.137 [01:27:37] PROBLEM - ElasticSearch health check on logstash1003 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.136 [01:27:41] I was hoping moving away from self-signed would have sufficed. [01:27:42] the cert check is kind of shitty if I recall correctly [01:27:59] self-signed? it was using a self-signed cert? [01:28:02] I'll say. It stack traces rather than return a proper value. [01:28:15] is this the python check that I wrote ages ago? [01:28:49] It was using a cert signed by wmf-ca which I presume is self-signed. Last I checked, we didn't have a real CA. :-) [01:29:18] this is what someone changed it to? [01:29:23] Ryan_Lane1: I think it is. I don't remember having ever seen it before, and I was presuming it was homebrew. [01:29:27] it was using the star cert last I checked [01:29:45] no, I've just changed it to the real virt0 cert. [01:29:58] it was using a *.wmnet cert? [01:30:00] The website, at least, was using the real star cert. [01:30:28] hm. I guess puppet indicates it was a fake *.wikimedia.org cert [01:30:47] anyway, you should really update puppet with the new CA that needs to be installed in the trust stores :) [01:30:57] Yeah, face star. [01:30:58] otherwise when we want to build new LDAP servers it'll be harder [01:31:10] Yes, yes I should. But now, it's late and I need to eat. [01:31:20] Also caffeine. [01:31:24] :D [01:44:26] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [02:07:45] !log undid the cert change on the virt0 LDAP; this has subtle impact in some other places because of the RapidSSL cert and will need planning. [02:07:55] Logged the message, Master [02:19:37] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [02:21:50] !log LocalisationUpdate completed (1.23wmf14) at 2014-02-22 02:21:50+00:00 [02:21:58] Logged the message, Master [02:35:25] !log LocalisationUpdate completed (1.23wmf15) at 2014-02-22 02:35:25+00:00 [02:35:33] Logged the message, Master [03:16:15] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-02-22 03:16:14+00:00 [03:16:23] Logged the message, Master [03:26:37] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [04:05:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:07:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:09:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:09:36] (03CR) 10Technical 13: [C: 031] "Unable to test, but the source looks reasonable to me." [operations/puppet] - 10https://gerrit.wikimedia.org/r/106907 (owner: 10Stwalkerster) [04:11:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:13:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:15:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:17:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:19:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:21:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:23:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:25:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:26:37] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [04:27:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 03:58:30 AM UTC [04:28:16] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Sat Feb 22 04:28:08 UTC 2014 [04:30:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 04:28:08 AM UTC [04:30:56] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 2 copy to table, 252 statistics [04:31:46] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 1 copy to table, 15 statistics [04:32:16] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 04:28:08 AM UTC [04:37:47] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 84 statistics [04:38:47] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [04:45:17] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [04:58:38] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Sat Feb 22 04:58:32 UTC 2014 [05:41:18] (03PS1) 10Ori.livneh: Get rid of remaining references to $wmfExtendedVersionNumber [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114915 [05:41:48] oh, lots of exceptions [05:41:49] fun [07:46:17] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [10:23:00] (03CR) 10Guido.iaquinti: [C: 031] ldap: Fix typo in usage messages [operations/puppet] - 10https://gerrit.wikimedia.org/r/114740 (owner: 10Tim Landscheidt) [10:23:46] (03CR) 10Guido.iaquinti: [C: 031] Fix manage-keys-nfs misnomer in usage message [operations/puppet] - 10https://gerrit.wikimedia.org/r/114739 (owner: 10Tim Landscheidt) [10:25:44] (03CR) 10Guido.iaquinti: [C: 031] Fix paths in comments after modularization [operations/puppet] - 10https://gerrit.wikimedia.org/r/114736 (owner: 10Tim Landscheidt) [10:47:17] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [13:24:47] PROBLEM - RAID on es1006 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) [13:48:17] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [13:59:07] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:57] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.164 second response time [14:07:57] PROBLEM - Host virt1002 is DOWN: PING CRITICAL - Packet loss = 100% [14:23:15] Anyone know what the keymap is for f10 on cisco mgmt? [14:34:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:36:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:37:25] RECOVERY - Host virt1002 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [14:38:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:40:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:42:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:44:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:44:55] PROBLEM - Host virt1003 is DOWN: PING CRITICAL - Packet loss = 100% [14:46:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:48:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:50:06] RECOVERY - Host virt1003 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [14:50:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:52:15] PROBLEM - Disk space on virt1003 is CRITICAL: Connection refused by host [14:52:25] PROBLEM - puppet disabled on virt1003 is CRITICAL: Connection refused by host [14:52:25] PROBLEM - RAID on virt1003 is CRITICAL: Connection refused by host [14:52:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:52:45] PROBLEM - SSH on virt1003 is CRITICAL: Connection refused [14:53:05] PROBLEM - DPKG on virt1003 is CRITICAL: Connection refused by host [14:54:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:56:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:58:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:28:36 PM UTC [14:59:25] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Sat Feb 22 14:59:14 UTC 2014 [15:00:45] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:59:14 PM UTC [15:02:47] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:59:14 PM UTC [15:03:34] PROBLEM - Host virt1004 is DOWN: PING CRITICAL - Packet loss = 100% [15:04:25] PROBLEM - NTP on virt1003 is CRITICAL: NTP CRITICAL: No response from NTP server [15:08:32] RECOVERY - Host virt1004 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [15:10:52] PROBLEM - puppet disabled on virt1004 is CRITICAL: Connection refused by host [15:10:53] PROBLEM - DPKG on virt1004 is CRITICAL: Connection refused by host [15:11:02] PROBLEM - RAID on virt1004 is CRITICAL: Connection refused by host [15:11:02] PROBLEM - SSH on virt1004 is CRITICAL: Connection refused [15:11:32] PROBLEM - Disk space on virt1004 is CRITICAL: Connection refused by host [15:12:12] PROBLEM - Host virt1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:14:42] RECOVERY - SSH on virt1003 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [15:14:53] RECOVERY - Host virt1003 is UP: PING OK - Packet loss = 0%, RTA = 2.03 ms [15:22:42] PROBLEM - NTP on virt1004 is CRITICAL: NTP CRITICAL: No response from NTP server [15:29:24] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Sat Feb 22 15:29:01 UTC 2014 [15:30:42] PROBLEM - Host virt1004 is DOWN: PING CRITICAL - Packet loss = 100% [15:32:02] RECOVERY - SSH on virt1004 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [15:32:12] RECOVERY - Host virt1004 is UP: PING OK - Packet loss = 0%, RTA = 0.80 ms [15:37:32] (03CR) 10Andrew Bogott: [C: 032] Add a couple more compute nodes to eqiad labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114941 (owner: 10Andrew Bogott) [16:48:52] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [17:12:02] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [17:13:12] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.35 ms [17:36:52] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:36:40 PM UTC [18:44:14] (03PS1) 10coren: Tool Labs: [operations/puppet] - 10https://gerrit.wikimedia.org/r/114950 [18:44:56] (03PS2) 10coren: Tool Labs: Omnibus fix to gridengine for eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/114950 [18:46:07] (03CR) 10jenkins-bot: [V: 04-1] Tool Labs: Omnibus fix to gridengine for eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/114950 (owner: 10coren) [18:48:05] (03PS3) 10coren: Tool Labs: Omnibus fix to gridengine for eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/114950 [18:50:00] (03CR) 10coren: [C: 032] "If I break it, I get to keep both pieces." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114950 (owner: 10coren) [18:52:19] (03PS1) 10coren: Tool Labs: typo fix in master.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/114951 [18:54:26] (03CR) 10coren: [C: 032] "Simple fix" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114951 (owner: 10coren) [18:56:21] (03PS1) 10coren: Tool Labs: more eqiad fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/114952 [18:57:46] (03CR) 10coren: [C: 032] "Simple fix." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114952 (owner: 10coren) [19:19:39] (03PS1) 10coren: Tool Labs: use bind mount for /var/lib/gridengine [operations/puppet] - 10https://gerrit.wikimedia.org/r/114954 [19:21:01] (03CR) 10coren: [C: 032] "Man, gridengine is finnicky" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114954 (owner: 10coren) [19:25:55] (03CR) 10Tim Landscheidt: Tool Labs: use bind mount for /var/lib/gridengine (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/114954 (owner: 10coren) [19:27:36] (03PS1) 10coren: Tool Labs: make the /var/lib/gridengine mountpoint [operations/puppet] - 10https://gerrit.wikimedia.org/r/114956 [19:28:15] (03CR) 10Matanya: [C: 031] "lgtm, one minor request." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/114734 (owner: 10Tim Landscheidt) [19:29:20] (03CR) 10coren: Tool Labs: use bind mount for /var/lib/gridengine (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/114954 (owner: 10coren) [19:29:37] (03CR) 10coren: [C: 032] Tool Labs: make the /var/lib/gridengine mountpoint [operations/puppet] - 10https://gerrit.wikimedia.org/r/114956 (owner: 10coren) [19:49:08] (03PS1) 10coren: eqiad should not include role::labsnfs::client [operations/puppet] - 10https://gerrit.wikimedia.org/r/114958 [19:49:52] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [19:50:44] (03CR) 10coren: [C: 032] "Moar eqiaditude." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114958 (owner: 10coren) [20:25:24] (03CR) 10Tim Landscheidt: Fix indentation in role::labs::instance (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/114734 (owner: 10Tim Landscheidt) [20:28:21] (03PS2) 10Tim Landscheidt: Fix indentation in role::labs::instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/114734 [20:29:40] (03PS1) 10coren: Tool Labs: eqiad tweaks to master.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/114960 [20:33:55] (03CR) 10coren: "Moar! MOAAAR!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114960 (owner: 10coren) [20:34:00] (03CR) 10coren: [C: 032] Tool Labs: eqiad tweaks to master.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/114960 (owner: 10coren) [20:37:52] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:36:40 PM UTC [22:50:52] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Fri 21 Feb 2014 04:42:42 PM UTC [23:03:26] hi all. Please can you tell me how I can ask to install a python package (internetarchive) into Labs? [23:04:43] Package can be found here: https://pypi.python.org/pypi/internetarchive [23:36:20] (03PS3) 10Hoo man: Make labs' sql command work with -v and remove cruft [operations/puppet] - 10https://gerrit.wikimedia.org/r/113755 [23:37:33] (03CR) 10Hoo man: "Restored b/c and made sure that -v can be given at any point (which required nasty hacks)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/113755 (owner: 10Hoo man) [23:38:01] petan: ^ [23:38:08] hi [23:38:27] I totally love when stuff in bash gets complex... [23:38:42] maybe we can switch to perl? [23:38:43] :P [23:38:52] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Last successful Puppet run was Sat 22 Feb 2014 02:36:40 PM UTC [23:39:03] let's make few hash of arrays of hashes if it's not complex enough [23:39:29] echo ${*:1:`expr $verbose - 1`} ${*:`expr $verbose + 1`:999} [23:40:36] hoo: is it somewhere on server? [23:40:45] hoo: I need to test it [23:41:05] sure /home/hoo/sql [23:41:17] just realized I don't even need the 999 :P [23:41:32] (03PS4) 10Hoo man: Make labs' sql command work with -v and remove cruft [operations/puppet] - 10https://gerrit.wikimedia.org/r/113755 [23:42:41] (03CR) 10Hoo man: "Removed the superfluous upper limit..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/113755 (owner: 10Hoo man) [23:43:08] petrb@tools-login:~$ ./sql -v en [23:43:09] This is unknown db to me, if you don't like that, blame petan on freenode [23:43:15] I don't really care :P [23:43:28] that's intended behavior, I guess [23:43:41] well... it's not but I don't care I am fine with that [23:43:47] the current script fails with that as well [23:44:09] yes I know [23:44:13] current script suck [23:44:31] I could fix it, but that would add even more complexity... [23:44:55] (03CR) 10Petrb: [C: 031] Make labs' sql command work with -v and remove cruft [operations/puppet] - 10https://gerrit.wikimedia.org/r/113755 (owner: 10Hoo man) [23:46:12] hoo complexity and obfuscation... we never have enough of that [23:46:13] :P [23:49:53] Sure :P [23:59:24] (03PS5) 10Hoo man: Make labs' sql command work with -v and remove cruft [operations/puppet] - 10https://gerrit.wikimedia.org/r/113755