[00:20:49] hello [00:21:09] apparently TOR exit nodes are no longer blocked by mediawiki. Is torblock not working for some reason? [00:23:40] it's been broken for a while [00:23:45] oh [00:23:55] https://bugzilla.wikimedia.org/show_bug.cgi?id=30716 [00:25:01] I see, thanks. In the mean time, I guess we're going to have to run scripts to block them on our own. [00:36:00] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [01:43:53] gn8 folks [02:05:35] TimStarling: http://www.nscl.msu.edu/~tsang/CMP/scientificamerican.html [02:05:44] that's pretty epic :) [02:17:37] !log LocalisationUpdate completed (1.18) at Tue Feb 21 02:17:36 UTC 2012 [02:17:41] Logged the message, Master [02:34:36] !log LocalisationUpdate completed (1.19) at Tue Feb 21 02:34:36 UTC 2012 [02:34:39] Logged the message, Master [04:02:12] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [04:08:12] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [04:08:12] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [04:42:39] !log on db40: setting innodb-use-purge-thread=4 to test multithreaded purge [04:42:41] Logged the message, Master [06:15:23] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [07:17:48] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [07:19:36] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [07:23:21] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [07:32:48] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.270 second response time [07:40:27] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [08:01:36] PROBLEM - Lucene on search1 is CRITICAL: Connection timed out [08:04:00] PROBLEM - Lucene on search3 is CRITICAL: Connection timed out [08:09:16] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Tue Feb 21 08:08:42 UTC 2012 [08:33:52] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [08:35:49] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [08:39:52] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [09:01:37] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.048 second response time [09:26:09] PROBLEM - DPKG on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:28:42] RECOVERY - DPKG on db1047 is OK: All packages OK [09:31:15] RECOVERY - Lucene on search1 is OK: TCP OK - 2.999 second response time on port 8123 [09:39:39] PROBLEM - Lucene on search1 is CRITICAL: Connection timed out [09:51:21] PROBLEM - udp2log processes on locke is CRITICAL: CRITICAL: filters absent: /a/squid/urjc.awk, [09:58:15] RECOVERY - udp2log processes on locke is OK: OK: all filters present [09:59:54] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 18.7802274783 (gt 8.0) [10:03:39] PROBLEM - udp2log processes on locke is CRITICAL: CRITICAL: filters absent: /a/squid/urjc.awk, [10:05:01] RECOVERY - udp2log processes on locke is OK: OK: all filters present [10:09:03] RECOVERY - Lucene on search3 is OK: TCP OK - 0.006 second response time on port 8123 [10:10:22] RECOVERY - Lucene on search1 is OK: TCP OK - 0.001 second response time on port 8123 [10:13:58] PROBLEM - udp2log processes on locke is CRITICAL: CRITICAL: filters absent: /a/squid/urjc.awk, [10:15:19] RECOVERY - udp2log processes on locke is OK: OK: all filters present [10:27:01] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 0.267108521739 [10:37:13] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [11:07:13] PROBLEM - Puppet freshness on db46 is CRITICAL: Puppet has not run in the last 10 hours [11:07:13] PROBLEM - Puppet freshness on mw1002 is CRITICAL: Puppet has not run in the last 10 hours [12:23:40] New review: Mark Bergsma; "Is the Puppet dependency between the packages necessary, i.e. doesn't APT resolve it?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2614 [12:27:37] New review: Mark Bergsma; "Instead of matching on $hostname to determine which site, please just use $::site, which is pmtpa or..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2670 [12:43:35] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [12:44:29] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [12:47:29] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [13:07:53] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [13:57:03] New patchset: QChris; "Set up .gitignore" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/2683 [13:57:07] New patchset: QChris; "Create directory for FileUtils.writeFile, if it does not exist" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/2684 [14:02:43] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [14:08:43] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [14:08:43] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [14:56:35] anyone around good with javascript who can help me out with a userscript? (or actually, with best practices in javascript) [15:00:00] FooBarMartijn: Post it to my meta talk (Hoo man) if it's non urgent ;) [15:00:21] it's not, thanks [15:06:22] New patchset: Pyoungmeister; "search lvs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2685 [15:10:15] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:15] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:15] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:15] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:20] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:21] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:21] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:21] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:17] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:17] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:17] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:26] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:14] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:14] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:14] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:14] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:29] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:30] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:30] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:30] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:17] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:26] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:27] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:27] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:09] !log reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'Enable AntiSpoof for CentralAuth on testwiki only' [15:37:11] Logged the message, Master [15:39:32] !log reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'Enable AntiSpooof for CentralAuth on all 1.19 wikis again, doesn't break signup with a mass of fail' [15:39:34] Logged the message, Master [15:40:14] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:23] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:24] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:24] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:20] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:20] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:29] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:29] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:17] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:17] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:17] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:26] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:00] New patchset: Pyoungmeister; "allowing eqiad to rsync from home" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2686 [15:55:02] Change abandoned: Pyoungmeister; "going to think about this one some more..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2686 [15:55:14] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:15] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:15] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:50] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:54] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:54] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:54] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:54] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:33] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:33] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:33] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:33] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:08:57] New patchset: Pyoungmeister; "allowing 10.64.0.0/22 - private1-a-eqiad to rsync from home" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2687 [16:10:30] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:30] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:30] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:30] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:27] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:28] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:28] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:28] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:54] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [16:16:19] New patchset: Pyoungmeister; "do sites properly" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2688 [16:20:33] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:34] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:34] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:34] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:30] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:30] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:30] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:30] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:27] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:28] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:28] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:28] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:33] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:33] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:33] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:33] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:30] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:31] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:31] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:31] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:05] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2687 [16:42:05] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2687 [16:45:27] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:28] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:28] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:28] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:09] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2688 [16:49:10] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2688 [16:50:33] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:33] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:33] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:33] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:52] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34560 - Moodbar on ta.wikipedia' [16:50:54] Logged the message, Master [16:51:18] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 8.56340678261 (gt 8.0) [16:53:42] PROBLEM - Disk space on searchidx1001 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=51%): /var/lib/ureadahead/debugfs 0 MB (0% inode=51%): [16:58:47] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:47] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:47] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:47] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:15] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:15] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:15] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:15] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.036 seconds [17:03:21] RECOVERY - Disk space on searchidx1001 is OK: DISK OK [17:03:21] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 1.9399722807 [17:05:50] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:50] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:50] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:50] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:09] !log reedy synchronized php-1.19/extensions/FeaturedFeeds/ 'r112023' [17:08:11] Logged the message, Master [17:08:58] !log reedy synchronized php-1.19/extensions/CentralAuth/CentralAuthHooks.php 'r112023' [17:09:00] Logged the message, Master [17:10:13] !log reedy synchronized php-1.19/includes/ 'r112024' [17:10:15] Logged the message, Master [17:10:30] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:31] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:31] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:31] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:12] New patchset: Pyoungmeister; "needs more root..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2689 [17:14:35] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2689 [17:14:39] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2689 [17:15:28] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:28] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:28] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:28] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:14] New patchset: Diederik; "Added full support for ip address and ip range filtering Added full support for regular expression matching Incorporated feedback from Tim, still struggling around lines 235-240. Change-Id: I8d52bbd84fd4ec39a6d735d802d9b87f95d1b0a0" [analytics/udp-filters] (refactoring) - https://gerrit.wikimedia.org/r/2626 [17:19:58] PROBLEM - Host searchidx1001 is DOWN: PING CRITICAL - Packet loss = 100% [17:20:33] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:34] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:34] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:34] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:21] RECOVERY - Host searchidx1001 is UP: PING OK - Packet loss = 0%, RTA = 26.42 ms [17:25:31] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:31] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:31] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:31] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:13] PROBLEM - Disk space on searchidx1001 is CRITICAL: Connection refused by host [17:28:49] PROBLEM - RAID on searchidx1001 is CRITICAL: Connection refused by host [17:29:06] PROBLEM - SSH on searchidx1001 is CRITICAL: Connection refused [17:29:15] PROBLEM - DPKG on searchidx1001 is CRITICAL: Connection refused by host [17:30:27] PROBLEM - check_minfraud1 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:28] PROBLEM - check_minfraud1 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:28] PROBLEM - check_minfraud1 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:28] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:09] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [17:33:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:15] RECOVERY - check_minfraud1 on payments1 is OK: HTTP OK: HTTP/1.1 200 OK - 8643 bytes in 0.390 second response time [17:35:16] RECOVERY - check_minfraud1 on payments2 is OK: HTTP OK: HTTP/1.1 200 OK - 8643 bytes in 0.314 second response time [17:35:16] RECOVERY - check_minfraud1 on payments4 is OK: HTTP OK: HTTP/1.1 200 OK - 8643 bytes in 0.313 second response time [17:35:16] RECOVERY - check_minfraud1 on payments3 is OK: HTTP OK: HTTP/1.1 200 OK - 8643 bytes in 0.313 second response time [17:35:26] !log reedy synchronized php-1.18/extensions/FeaturedFeeds/FeaturedFeeds.body.php 'r112029' [17:35:28] Logged the message, Master [17:35:52] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.045 seconds [17:36:10] wtf [17:36:14] 400 != OK [17:36:21] :-D [17:36:29] * RoanKattouw files RT ticket [17:36:44] at least it gave a reply [17:37:35] upstream fail ;) [17:38:15] #2492 [17:40:44] !log Going to have secks with fluffernutter today. It will be glorious [17:40:46] Logged the message, Master [17:41:19] Ops [17:41:24] ... [17:41:24] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [17:41:27] !ops [17:41:30] please ban *!*@mobile-166-147-*.mycingular.net [17:42:02] *!*@mobile-166-147-*.mycingular.net is already banned in most other chans [17:42:29] TBloemink: which host was it this time? [17:42:39] * JimJones (~reedy@mobile-166-147-108-245.mycingular.net) has joined #wikimedia-tech [17:42:46] ty [17:43:07] why do only ops people have ops here *rimshot* [17:44:14] I don't [17:45:34] RobH can fix that apergos [17:45:43] he could [17:45:45] but I don't need it [17:47:43] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [17:48:10] RECOVERY - DPKG on searchidx1001 is OK: All packages OK [17:48:18] damn it, how do I ban someone? [17:48:55] RECOVERY - Disk space on searchidx1001 is OK: DISK OK [17:49:09] meh [17:50:18] !log

buttsecks

[17:50:20] Logged the message, Master [17:50:33] marienz: [17:50:53] Thanks [17:50:57] TBloemink: thanks, I was wondering why that beeped [17:51:27] We need someone to undo that one [17:51:35] Reedy maybe? [17:52:04] yup [17:52:10] fixed:position vandalism through IRC O_o [17:52:35] Hi all: office hours with the WMF localization team is starting about now in #wikimedia-office [17:52:44] so, yeah, how do I ban someone again? [17:53:09] You'll need a range ban here, /mode +b *!*@mobile-166-147-*.mycingular.net [17:53:09] Ryan_Lane: /mode +b *!*@mobile-166-147-*.mycingular.net [17:53:11] /mode #wikimedia-tech +b *!*@host [17:53:12] I always have to look it up on freenode [17:53:18] (cause I never use it) [17:53:24] PROBLEM - NTP on searchidx1001 is CRITICAL: NTP CRITICAL: Offset unknown [17:53:27] let's add that to the wikitech page [17:53:38] there we go [17:53:41] banned [17:53:51] will that extra space at the end matter? [17:54:03] probably :) [17:56:01] rats, I was adding that [17:56:07] RECOVERY - NTP on searchidx1001 is OK: NTP OK: Offset 0.001218318939 secs [17:58:42] updated anyways [17:59:34] extra space at the end? [17:59:53] for the banlist [17:59:56] ouch .netÂ [18:09:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.842 seconds [18:20:40] New patchset: Pyoungmeister; "searchidx wants this too" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2690 [18:22:01] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2690 [18:22:02] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2690 [18:40:12] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.36314886957 (gt 8.0) [18:46:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:02] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.31944417391 [18:47:16] New patchset: Lcarr; "Creating new class for new nagios host (aka neon)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2666 [18:51:08] New patchset: Lcarr; "Creating new class for new nagios host (aka neon)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2666 [18:52:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 8.365 seconds [19:03:44] New patchset: Ottomata; "Comments, fixing tests" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2691 [19:04:43] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2691 [19:04:43] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2691 [19:16:45] New patchset: Lcarr; "Creating new class for new nagios host (aka neon)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2666 [19:19:56] New patchset: Lcarr; "Creating new class for new nagios host (aka neon)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2666 [19:24:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 9.192 seconds [19:32:03] New patchset: Ottomata; "Another test for push" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2692 [19:34:24] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2692 [19:34:25] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2692 [19:36:50] New patchset: Lcarr; "Creating new class for new nagios host (aka neon)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2666 [19:41:19] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/2675 [19:42:29] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2666 [19:42:30] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2666 [19:43:35] PROBLEM - SSH on neon is CRITICAL: Connection refused [19:43:53] PROBLEM - RAID on neon is CRITICAL: Connection refused by host [19:43:53] PROBLEM - Disk space on neon is CRITICAL: Connection refused by host [19:44:11] PROBLEM - DPKG on neon is CRITICAL: Connection refused by host [19:59:38] PROBLEM - NTP on neon is CRITICAL: NTP CRITICAL: No response from NTP server [20:04:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:53] RECOVERY - SSH on neon is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [20:08:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.665 seconds [20:38:03] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [20:41:30] can someone help with figuring out how an edit wasn't caught by global SBL? [20:41:37] *spam blacklist [20:42:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:46:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.746 seconds [20:47:39] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [20:49:00] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [20:53:59] am I the ony one having trouble with Bugzilla? [20:56:29] don't know, are you? [20:56:31] what's up? [20:57:18] I noticed there's a couple of bugs with IRC on the new mediawiki 1.19 - but I'm unsure if this one has been reported [20:57:28] on the RC feed for meta, when I protect a page, it gives: [20:55:48] [[Special:Log/protect]] protect * Thehelpfulone * Thehelpfulone protected "[[User:Thehelpfulone]]" ‎[edit=autoconfirmed] (expires 20:55, 21 February 2012 (UTC)): test [20:57:47] the 2nd Thehelpfulone there is not need as the first one is the user doing the action [20:58:00] i believe it was reported [20:58:04] dont know sure [20:59:15] Reedy: Connection's resetting while it's loading. [20:59:15] Thehelpfulone: I believe it was https://bugzilla.wikimedia.org/show_bug.cgi?id=34508 [20:59:41] ah okay, yes a big one [21:08:03] PROBLEM - Puppet freshness on db46 is CRITICAL: Puppet has not run in the last 10 hours [21:08:04] PROBLEM - Puppet freshness on mw1002 is CRITICAL: Puppet has not run in the last 10 hours [21:20:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 1.478 seconds [21:58:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.773 seconds [22:20:07] !log catrope synchronized wmf-config/CommonSettings.php 'Comment out hack that enabled $wgResourceLoaderExperimentalAsyncLoading for logged-in users' [22:20:09] Logged the message, Master [22:36:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.092 seconds [22:44:38] !log catrope synchronized php-1.19/includes/resourceloader/ 'r112055' [22:44:40] Logged the message, Master [22:45:07] robla: --^^ [22:45:29] oh good, thanks! [23:14:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.716 seconds [23:18:40] New patchset: Lcarr; "Changed name of createfirewall.py to match new name of software" [operations/software] (master) - https://gerrit.wikimedia.org/r/2694 [23:19:11] New review: Lcarr; "(no comment)" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2694 [23:19:12] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2694 [23:34:22] New patchset: Pyoungmeister; "adding cron to search hosts to occassionally poll for new mediawiki config files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2695 [23:34:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2695 [23:35:50] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2695 [23:35:51] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2695 [23:43:58] New patchset: Diederik; "Added full support for ip address and ip range filtering Added full support for regular expression matching Incorporated feedback from Tim, still struggling around line 369 - 378. Change-Id: I8d52bbd84fd4ec39a6d735d802d9b87f95d1b0a0" [analytics/udp-filters] (refactoring) - https://gerrit.wikimedia.org/r/2626 [23:47:25] RECOVERY - Lucene on search1007 is OK: TCP OK - 0.027 second response time on port 8123 [23:52:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:27] New patchset: Pyoungmeister; "adding ganglia data sources for eqiad search" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2696 [23:55:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2696 [23:56:01] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2696 [23:56:01] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2696 [23:56:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.933 seconds [23:57:59] gn8 folks