[00:01:40] well, we need to track down all the authors and have them agree to a license [00:01:50] domas often releases public domain [00:02:14] and anyone from WMF is required to release under an open source license, and we generally default to GPL 3 [00:02:27] so, assuming no one outside of WMF has modified it, we're fine [00:02:29] err [00:02:34] we default to GPL 2 or later [00:05:37] okidoki [00:09:19] RECOVERY - Parsoid on wtp1011 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.004 second response time [00:11:14] Ryan_Lane: do you individually retain the copyright or do you transfer it to WMF? [00:11:21] both [00:11:27] it's dual licenced [00:14:40] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [00:22:09] PROBLEM - RAID on mc15 is CRITICAL: Timeout while attempting connection [00:23:00] RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [00:27:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [00:31:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [00:52:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [01:10:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [01:32:13] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset 0.003308296204 secs [01:32:33] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset 0.001198649406 secs [01:51:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [01:57:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [02:00:00] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:01:00] RECOVERY - DPKG on mc15 is OK: All packages OK [02:02:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [02:03:54] !log LocalisationUpdate completed (1.22wmf6) at Sun Jun 16 02:03:54 UTC 2013 [02:04:10] Logged the message, Master [02:06:29] !log LocalisationUpdate completed (1.22wmf7) at Sun Jun 16 02:06:29 UTC 2013 [02:06:37] Logged the message, Master [02:13:03] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 16 02:13:03 UTC 2013 [02:13:12] Logged the message, Master [02:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [02:53:44] PROBLEM - RAID on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:54:34] RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [03:02:04] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:02:56] RECOVERY - DPKG on mc15 is OK: All packages OK [03:32:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:43:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:54:41] New patchset: Parent5446; "Enabling secure login (HTTPS), second attempt" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68937 [03:56:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.578 second response time [04:05:42] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:06:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:42] RECOVERY - DPKG on mc15 is OK: All packages OK [04:07:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [04:26:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [04:32:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [04:43:11] New patchset: Ori.livneh; "$wgEventLoggingSchemaIndexUri => $wgEventLoggingSchemaApiUri" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68938 [04:49:40] New patchset: Ori.livneh; "$wgEventLoggingSchemaIndexUri => $wgEventLoggingSchemaApiUri" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68938 [04:50:11] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68938 [04:51:59] New review: MZMcBride; "Just out of curiosity, is there any reason to prefer HTTP over HTTPS here? I feel that choosing eith..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68938 [04:56:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:59:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [05:08:20] New review: Ori.livneh; "(context: Tim's review of , last November)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68938 [05:10:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:11:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [05:45:17] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:17] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:17] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:17] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:17] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:17] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [05:45:18] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:19] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:19] PROBLEM - Puppet freshness on spence is CRITICAL: No successful Puppet run in the last 10 hours [05:45:20] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:20] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:47] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:46:17] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [05:49:22] New review: MZMcBride; "Ah, neat. Thanks for looking that up." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68938 [06:22:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [06:25:36] PROBLEM - Disk space on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:26:34] RECOVERY - Disk space on mc15 is OK: DISK OK [06:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.187 second response time [07:00:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.140 second response time [07:26:43] PROBLEM - SSH on searchidx1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:32] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [07:27:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [07:56:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [08:32:50] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.003562808037 secs [08:35:59] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.002108573914 secs [09:09:01] New patchset: Nemo bis; "(bug 15434) Periodical run of currently disabled special pages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33713 [09:09:32] ah, it said Can Merge: No but rebase button worked [10:15:23] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [10:43:22] PROBLEM - Parsoid on wtp1013 is CRITICAL: Connection refused [10:49:22] RECOVERY - Parsoid on wtp1013 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time [10:56:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [10:59:01] PROBLEM - Parsoid on wtp1024 is CRITICAL: Connection refused [11:02:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:03:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [11:05:21] PROBLEM - Parsoid on wtp1002 is CRITICAL: Connection refused [11:08:01] RECOVERY - Parsoid on wtp1024 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time [11:18:21] RECOVERY - Parsoid on wtp1002 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time [11:30:41] PROBLEM - Parsoid on wtp1012 is CRITICAL: Connection refused [11:34:51] PROBLEM - Host wtp1008 is DOWN: PING CRITICAL - Packet loss = 100% [11:36:03] RECOVERY - Host wtp1008 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [11:36:13] PROBLEM - NTP on wtp1008 is CRITICAL: NTP CRITICAL: Offset unknown [11:39:13] RECOVERY - NTP on wtp1008 is OK: NTP OK: Offset -0.0008370876312 secs [11:40:53] PROBLEM - Parsoid on wtp1010 is CRITICAL: Connection refused [11:52:53] RECOVERY - Parsoid on wtp1010 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.004 second response time [11:57:43] RECOVERY - Parsoid on wtp1012 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [12:02:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:11:16] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [12:12:49] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.63 ms [12:14:47] PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused [12:15:57] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.649 second response time [12:34:56] PROBLEM - Parsoid on wtp1023 is CRITICAL: Connection refused [12:40:16] New patchset: Odder; "(bug 49639) Change local time zone for ko.wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68947 [12:59:56] RECOVERY - Parsoid on wtp1023 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [13:01:01] Ryan_Lane: around? [13:12:25] operations/debs/wikimedia-keyring seems to lack everything [13:12:46] as wll git-deploy seems to be strange [13:22:58] PROBLEM - Parsoid on wtp1001 is CRITICAL: Connection refused [13:39:15] PROBLEM - Parsoid on wtp1021 is CRITICAL: Connection refused [13:44:54] RECOVERY - Parsoid on wtp1001 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time [13:50:14] RECOVERY - Parsoid on wtp1021 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time [14:15:12] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: Connection timed out [14:15:49] I'm on my way to the airport so I can't have a look now [14:15:51] but don't worry [14:16:01] it's been depooled from production [14:16:02] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.005 second response time [14:40:51] PROBLEM - Parsoid on wtp1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:43] PROBLEM - Parsoid on wtp1007 is CRITICAL: Connection refused [15:10:49] RECOVERY - Parsoid on wtp1004 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time [15:12:09] RECOVERY - Parsoid on wtp1007 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time [15:20:39] PROBLEM - Parsoid on wtp1022 is CRITICAL: Connection refused [15:20:56] paravoid: both? [15:26:49] PROBLEM - Parsoid on wtp1017 is CRITICAL: Connection refused [15:27:09] PROBLEM - Parsoid on wtp1020 is CRITICAL: Connection refused [15:29:09] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:29:59] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [15:30:40] RECOVERY - Parsoid on wtp1022 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time [15:31:49] RECOVERY - Parsoid on wtp1017 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [15:45:24] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [15:45:24] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:24] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:24] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:24] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:25] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:26] PROBLEM - Puppet freshness on spence is CRITICAL: No successful Puppet run in the last 10 hours [15:45:26] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:27] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:27] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [15:53:44] RECOVERY - Parsoid on wtp1020 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [16:06:24] PROBLEM - Parsoid on wtp1003 is CRITICAL: Connection refused [16:09:29] RECOVERY - Parsoid on wtp1003 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.006 second response time [17:10:27] PROBLEM - Parsoid on wtp1014 is CRITICAL: Connection refused [17:26:27] RECOVERY - Parsoid on wtp1014 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.020 second response time [17:44:31] PROBLEM - Parsoid on wtp1011 is CRITICAL: Connection refused [18:09:33] RECOVERY - Parsoid on wtp1011 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [18:31:31] panda, behave [18:31:47] hmm? [18:31:54] what did I do? [18:32:05] Nemo_bis: ^ [18:32:17] Vodka-ise [18:32:21] :-) [18:32:34] hmm? [18:32:47] I can't be VodkaPanda? [18:33:52] PROBLEM - Solr on solr1001 is CRITICAL: Average request time is 402.61258 (gt 400) [18:34:02] YuviPanda: nope. That says me, a Polish person. [18:34:11] why? [18:34:15] with temperance [18:35:09] YuviPanda: you'd need to be officially recognised as honorary Polish (or maybe Russian) to be able to use that particular nick. [18:35:16] Ah [18:35:19] I see [18:35:37] I'll keep that in mind :) [18:35:50] haha, sure :) [18:38:00] I'm sure the IRC GC can help then [18:54:43] PROBLEM - Parsoid on wtp1018 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:53] RECOVERY - Solr on solr1001 is OK: All OK [19:00:34] RECOVERY - Parsoid on wtp1018 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.002 second response time [19:03:53] PROBLEM - Solr on solr1001 is CRITICAL: Average request time is 403.88428 (gt 400) [19:23:40] PROBLEM - Parsoid on wtp1015 is CRITICAL: Connection refused [19:26:42] PROBLEM - Parsoid on wtp1019 is CRITICAL: Connection refused [19:27:41] PROBLEM - Parsoid on wtp1018 is CRITICAL: Connection refused [19:30:21] PROBLEM - Parsoid on wtp1009 is CRITICAL: Connection refused [19:31:40] RECOVERY - Parsoid on wtp1019 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.009 second response time [19:33:20] RECOVERY - Parsoid on wtp1009 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time [19:39:27] RECOVERY - Parsoid on wtp1018 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [19:39:57] PROBLEM - Parsoid on wtp1006 is CRITICAL: Connection refused [19:41:37] RECOVERY - Parsoid on wtp1015 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.017 second response time [19:53:57] RECOVERY - Parsoid on wtp1006 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.002 second response time [20:05:19] PROBLEM - Parsoid on wtp1005 is CRITICAL: Connection refused [20:07:08] PROBLEM - Parsoid on wtp1024 is CRITICAL: Connection refused [20:07:58] PROBLEM - Parsoid on wtp1002 is CRITICAL: Connection refused [20:08:08] RECOVERY - Parsoid on wtp1024 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.013 second response time [20:15:28] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [20:17:58] RECOVERY - Parsoid on wtp1002 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.016 second response time [20:28:48] RECOVERY - Parsoid on wtp1005 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.010 second response time [20:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [21:05:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [21:18:35] New patchset: Jeroen De Dauw; "Update Wikidata and SMW IRC notification repos" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/69010 [21:26:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [21:36:11] PROBLEM - Parsoid on wtp1016 is CRITICAL: Connection refused [21:45:33] Still looking for the conf file that contains the code for auto-sending requests for non-existent thumbs to thumb.php btw [21:45:38] AaronSchulz perhaps? [21:54:04] RECOVERY - Parsoid on wtp1016 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [21:54:54] PROBLEM - Parsoid on wtp1003 is CRITICAL: Connection refused [21:58:03] Oh, is it rewrite.py in Swift? [21:59:29] jarry1250: https://git.wikimedia.org/blob/operations%2Fmediawiki-config.git/a3efb69e2d347960f67fb9bb31f3753933109815/wmf-config%2Ffilebackend.php#L189 [22:00:41] see also http://www.mediawiki.org/wiki/Manual:Thumb.php#404_Handler [22:01:24] ori-l: transformVia404? Yes, I'm just looking for the bit that actually does the redirecting on Wikimedia wikis so I can patch it [22:01:39] If that script is rewrite.py though, then I shouldn't need to patch it [22:03:02] not sure, tbh [22:04:44] No problem, thanks anyway [22:08:54] RECOVERY - Parsoid on wtp1003 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.015 second response time [22:15:56] jarry1250: yes it's in rewrite.py [22:16:20] apergos: Great, thanks :) [22:16:25] I don't know about ceph though [22:16:28] yw [22:16:44] what do you need to patch (or not need to)? [22:17:50] apergos: My patch added a new format for thumbnail names, just need to make sure the redirect to thumb.php would still work [22:18:37] right [22:20:16] linky? (I might need it for something else) [22:20:37] apergos: Link to? The patch? [22:20:42] yeah [22:20:48] https://gerrit.wikimedia.org/r/#/c/25838/ [22:20:52] ok thanks [22:21:51] might affect something I'm doing in labs, I'll check my stuff tomorrow [22:21:53] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: No route to host [22:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:54] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.004 second response time [22:23:05] apergos: Sure (it adds e.g. a "langfr-" prefix for PNG SVG thumb renderings where the rendering was done in French, in case that's not obvious) [22:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.708 second response time [22:23:26] ok, thanks for the info! [22:31:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [22:32:55] PROBLEM - RAID on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:33:55] RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [22:50:02] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:50:52] RECOVERY - DPKG on mc15 is OK: All packages OK [22:55:32] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [23:00:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:02:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [23:05:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [23:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [23:35:45] !log restarting pybal on lvs1003/lvs1006 for ms-fe4.eqiad.wmnet ip change [23:35:54] Logged the message, Master [23:55:36] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server