[00:00:29] PROBLEM - HTTP on ms-fe1004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.006 second response time [00:01:18] RECOVERY - HTTP on ms-fe1003 is OK: HTTP OK: HTTP/1.1 200 OK - 395 bytes in 0.008 second response time [00:06:27] PROBLEM - MySQL Replication Heartbeat on db35 is CRITICAL: NR [00:06:49] New review: Dzahn; "same as i just did on the blog" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/60431 [00:06:50] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60431 [00:06:51] New patchset: Faidon; "Ceph: fix start/stop/restart for radosgw Service" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61160 [00:08:14] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61160 [00:08:16] !log make SSL on Bugzilla also use SSLCACertificatePath and c_rehash certs on kaulen [00:08:24] Logged the message, Master [00:09:12] and that's it for now .. cya later [00:09:40] Verify return code: 0 (ok) [00:10:18] New review: Dzahn; "openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -connect bugzilla.wikimedia.org:443" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60431 [01:05:47] what's the deal with things like 'node /^db67\.pmtpa\.wmnet/ {' (site.pp, line 518) [01:05:53] like, why is that a regex? [01:06:23] there are at least a couple of them [01:08:13] they were probably more and it got simplified over time [01:08:17] patches are welcome :) [01:11:05] paravoid: are you *sure* :P [01:11:20] about patches being welcome, i mean, heh [01:12:52] patches totally welcome on those two javascript changes in OSM :) [01:18:20] Ryan_Lane: i was actually working on that earlier [01:18:40] they should both pass lint now, at minimum, though I didn't verify that with the second one since jenkins was down [01:18:46] interface behavior acceptable? [01:18:51] oh? [01:19:10] I see some changes pushed into gerrit, but they were the first pass from the other day [01:19:39] yeah, those are the only changes i pushed [01:19:43] other one not done yet [01:19:54] ah. ok [01:27:40] New patchset: Ori.livneh; "Add 'tcpircbot' Puppet class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61078 [01:31:33] New review: Ori.livneh; "PS2: Fix a bug caused by us calling select on the bot's socket when the bot had meanwhile recycled i..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61078 [01:35:24] ori-l: tim had discussed socat wasn't an option because it would basically make a public anonymous irc relay [01:35:34] ori-l: this doesn't seem to have any authentication [01:35:41] how is this not also one? [01:37:07] we could use socat + ircecho and it would do the same thing as this [01:39:52] that said, with authentication this is really useful [02:13:53] !log LocalisationUpdate completed (1.22wmf2) at Sat Apr 27 02:13:53 UTC 2013 [02:14:03] Logged the message, Master [02:25:38] New patchset: Ori.livneh; "Avoid using regexps where string literals would do" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61164 [03:06:47] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 27 03:06:47 UTC 2013 [03:06:55] Logged the message, Master [06:33:06] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 183 seconds [07:06:05] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:06:05] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:06:05] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [07:06:06] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 16 seconds [07:16:05] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:51] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [08:39:52] PROBLEM - Puppet freshness on vanadium is CRITICAL: No successful Puppet run in the last 10 hours [12:42:04] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [12:42:05] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:44] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 203 seconds [12:44:44] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [13:32:55] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 194 seconds [14:00:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [14:14:03] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [14:19:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [14:20:23] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds [14:43:21] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 194 seconds [14:45:20] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [16:08:25] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [16:10:25] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [16:13:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [16:14:24] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 9 seconds [16:24:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 218 seconds [16:25:25] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [16:34:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 216 seconds [16:35:25] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [16:43:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [16:45:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 13 seconds [17:07:02] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:02] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:02] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [17:17:02] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:18:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [17:20:23] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [17:39:30] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.113 second response time [17:50:30] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [17:52:30] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [18:11:30] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [18:19:10] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 238 seconds [18:21:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 22 seconds [18:24:10] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 202 seconds [18:28:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [18:41:00] PROBLEM - Puppet freshness on vanadium is CRITICAL: No successful Puppet run in the last 10 hours [18:59:10] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 201 seconds [19:00:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [19:10:10] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 219 seconds [19:18:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [19:34:10] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 239 seconds [19:35:20] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 9 seconds [19:38:20] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 189 seconds [19:40:20] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [20:15:18] New patchset: Hashar; "contint: tweak apache logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61195 [20:23:17] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 190 seconds [20:25:17] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [20:31:20] New review: RobH; "agreed with daniel's comments, making required changes and will resubmit" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/60766 [20:31:23] New patchset: Hashar; "zuul: pass puppet-lint (whitespaces)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61244 [20:40:04] RobH: next step : transform the rackstables manifest to a module :) [20:43:19] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 190 seconds [20:44:19] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [21:59:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 203 seconds [22:00:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [22:09:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 225 seconds [22:10:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [22:13:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [22:15:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [22:20:34] New review: MZMcBride; "This seems reasonable to me." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61164 [22:42:29] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [22:42:30] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [22:47:59] New review: Jeremyb; "recheck" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61050 [22:48:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [22:50:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [23:08:10] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [23:10:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [23:13:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [23:15:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [23:18:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [23:20:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [23:23:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [23:24:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 16 seconds [23:32:59] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [23:48:33] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 202 seconds [23:50:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds [23:58:33] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 202 seconds