[00:02:23] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [00:05:27] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:08:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 00:08:36 UTC 2013 [00:09:27] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:09:37] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [00:09:47] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 00:09:40 UTC 2013 [00:10:27] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:10:37] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:10:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 00:10:36 UTC 2013 [00:11:27] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:11:27] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [00:11:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 00:11:29 UTC 2013 [00:12:27] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:12:57] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 00:12:49 UTC 2013 [00:13:27] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:14:17] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [00:31:27] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [00:33:18] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 00:33:13 UTC 2013 [00:33:27] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [01:05:00] New review: Nemo bis; "There's already a thwiki above with" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/57515 [01:05:40] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [01:30:00] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [01:49:27] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [02:02:27] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [02:05:43] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [02:11:21] !log LocalisationUpdate completed (1.22wmf1) at Sun Apr 7 02:11:21 UTC 2013 [02:11:29] Logged the message, Master [02:17:09] !log LocalisationUpdate completed (1.21wmf12) at Sun Apr 7 02:17:09 UTC 2013 [02:17:17] Logged the message, Master [02:25:23] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [02:29:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: No successful Puppet run in the last 10 hours [02:32:33] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [03:07:12] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [03:26:04] New review: Reedy; "Note, these came from https://gerrit.wikimedia.org/r/gitweb?p=operations/debs/wikimedia-task-appserv..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57854 [04:06:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:08:46] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 04:08:37 UTC 2013 [04:09:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:09:46] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 04:09:40 UTC 2013 [04:10:07] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [04:10:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:10:46] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 04:10:37 UTC 2013 [04:11:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:11:36] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 04:11:30 UTC 2013 [04:12:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:12:16] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 04:12:10 UTC 2013 [04:13:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:13:26] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 04:13:16 UTC 2013 [04:14:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:32:46] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 04:32:45 UTC 2013 [04:33:07] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [04:33:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [05:33:01] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 05:32:52 UTC 2013 [05:33:11] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [05:40:48] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [05:48:18] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:48:18] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [05:48:18] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:02:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:48] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [06:03:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [06:07:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:28:53] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 06:28:43 UTC 2013 [06:29:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:32:53] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 06:32:47 UTC 2013 [06:33:03] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:48:33] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [07:00:33] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [07:04:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [07:26:20] !log gallium: restarted Zuul just to be sure. Gerrit on manganese is stuck again and need to be restarted {{bug|46917}} [07:26:26] Logged the message, Master [07:26:29] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [07:30:29] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [07:43:32] Anybody else having trouble connecting to Wikipedia? If I try to ping wikipedia-lb.eqiad.wikimedia.org (208.80.154.225), I get "Request timeout for icmp_seq 0", etc. [07:44:51] If I try to connect over the web I get "The server at en.wikipedia.org is taking too long to respond." [07:45:14] Same for any other Wikimedia sites. Everything else seems to be working OK though. [07:54:18] New review: Hashar; "See also:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57854 [07:57:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [08:03:28] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [08:04:29] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [08:06:38] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:08:18] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 08:08:14 UTC 2013 [08:08:38] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:08:58] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 08:08:50 UTC 2013 [08:09:38] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:14:28] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [08:31:28] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [08:46:22] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [09:01:22] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [09:04:57] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [09:18:27] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [09:31:27] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [09:41:28] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [10:02:28] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [10:05:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [10:14:48] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [10:18:28] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [10:25:00] New patchset: Odder; "(bug 46154) abusefilter-log-detail for autoconfirmed on thwiki This commit removes duplicate setting from InitialiseSettings.php and changes it in abusefilter.php from false to true per Nemo bis's comment below; thanks for spotting it!" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57515 [10:31:28] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [10:55:31] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [11:01:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [11:02:31] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [11:05:25] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [11:10:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [11:30:05] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [11:35:09] <^demon> !log restarting gerrit, stream-events seized up again. [11:35:17] Logged the message, Master [11:36:26] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [11:36:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:02:26] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [12:04:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:05:28] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [12:08:38] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 12:08:36 UTC 2013 [12:09:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:09:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 12:09:40 UTC 2013 [12:10:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:10:38] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 12:10:36 UTC 2013 [12:11:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:11:28] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 12:11:27 UTC 2013 [12:12:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:12:58] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 12:12:49 UTC 2013 [12:13:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:13:28] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 12:13:20 UTC 2013 [12:14:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:29:58] PROBLEM - Puppet freshness on virt1000 is CRITICAL: No successful Puppet run in the last 10 hours [12:32:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 12:32:43 UTC 2013 [12:33:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:33:28] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [12:44:34] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: Timeout while attempting connection [12:47:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [13:06:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [13:34:25] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [13:44:25] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:02:25] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [14:05:25] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:06:26] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [14:30:07] PROBLEM - Varnish HTTP mobile-backend on cp1041 is CRITICAL: Connection timed out [14:32:06] RECOVERY - Varnish HTTP mobile-backend on cp1041 is OK: HTTP OK: HTTP/1.1 200 OK - 634 bytes in 1.153 second response time [14:33:26] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [15:05:38] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [15:44:24] PROBLEM - Varnish HTCP daemon on cp1041 is CRITICAL: Timeout while attempting connection [15:45:15] RECOVERY - Varnish HTCP daemon on cp1041 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [15:49:04] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [15:49:04] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:49:04] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [15:49:34] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: Timeout while attempting connection [16:04:24] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [16:05:40] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:08:40] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 16:08:29 UTC 2013 [16:08:40] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:09:30] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 16:09:25 UTC 2013 [16:09:30] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:10:21] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 16:10:15 UTC 2013 [16:10:30] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:11:10] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 16:10:59 UTC 2013 [16:11:30] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:11:40] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 16:11:34 UTC 2013 [16:12:30] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:33:10] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 16:33:05 UTC 2013 [16:33:30] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:33:50] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:37] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [17:17:24] New patchset: Odder; "(bug 46915) Enable the autopatrolled group on ruwikivoyage Enabling the group per bug request & community consensus. Group can be added and removed by administrators." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57927 [17:20:47] New patchset: Odder; "(bug 46915) Enable the autopatrolled group on ruwikivoyage Enabling the group per bug request & community consensus. Group can be added and removed by administrators. (Proper indentation, guess I gotta change my keyboard :)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57927 [17:32:37] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [17:32:57] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 17:32:48 UTC 2013 [17:33:57] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [17:56:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.182 second response time [18:01:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [18:05:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [18:20:37] Susan: spammer [18:22:49] Nemo_bis: Apparently Gerrit has groups that can be added as a reviewer. [18:22:53] I didn't know about this. [18:23:02] I'm also not really sure who's supposed to review config changes. [18:23:06] So I just added a bunch of helpful people. [18:23:36] generally it's useless to review them [18:23:47] but odder likes us to :) [18:24:19] It's unclear to me whether someone besides a shell user can +2 those changes. [18:24:23] Can --> should. [18:24:38] no [18:24:58] So it's basically up to Reedy? [18:29:24] mostly [18:29:56] Hmm. [18:31:21] hey :-] [18:31:25] which change ? [18:31:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [18:34:02] I got cc'd on three. [18:34:18] https://gerrit.wikimedia.org/r/57506 [18:34:27] https://gerrit.wikimedia.org/r/57515 [18:34:47] https://gerrit.wikimedia.org/r/57506 [18:34:52] Oh, already linked that one. [18:35:29] https://gerrit.wikimedia.org/r/57519 [18:35:32] Those three. [18:36:38] * Nemo_bis slaps Susan  [18:36:46] Harder. [18:36:49] What'd I do? [18:36:50] odder: stop including Susan in your review requests, he's useless [18:37:09] Susan: nothing :) [18:37:28] The house is not on fire. [18:37:32] Nemo_bis: This is not what I heard! [18:37:35] Yet. [18:37:54] I remember Susan saying he's happy to help in reviewing stuff on Gerrit [18:38:13] Susan: I can not include you if you don't like it though... [18:39:02] New review: MZMcBride; "I think database names are usually alphabetized in lists like this." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57683 [18:39:08] odder: I'm happy to help if I can. [18:40:12] Susan: ^^ then en would need to be below cs. Or maybe en doesn't count? [18:40:50] New review: MZMcBride; "Is this wiki intending to use namespace 100 as a content namespace? I believe all content namespaces..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57506 [18:41:09] odder: NFI. I think most lists in that file are roughly indexed, though. [18:41:36] With default first. [18:41:44] Then all the individual wiki databases. [18:41:51] New review: Hashar; "I guess that is because CdbWriter_PHP does a lot of tiny writes to the disk. I have noticed on const..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/57774 [18:46:45] New patchset: Nemo bis; "(bug 46915) Enable the autopatrolled group on ru.wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57927 [18:47:21] ? [18:48:30] odder: again, differences between patches that don't affect the end result are better commented in the commentes [18:49:04] Ah, I see. [18:56:39] New review: MaxSem; "At least the system I tested it on had native dba functions, yet writing to /tmp was 8 times faster:)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57774 [19:04:20] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [19:26:40] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: Timeout while attempting connection [19:27:30] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [19:39:59] That's odd. http://meta.wikimedia.org/w/index.php?title=Special%3AGlobalUsers&username=&group=Staff&limit=50 returns a list of users, many with syadmin, but http://meta.wikimedia.org/w/index.php?title=Special%3AGlobalUsers&username=&group=sysadmin&limit=50 returns nobody [19:43:08] But http://meta.wikimedia.org/wiki/Special:GlobalUsers/sysadmin works. [19:43:40] https://meta.wikimedia.org/wiki/Special:GlobalUsers/sysadmin works [19:43:43] oup [19:44:33] probably due to the recent name normalisation mess [19:44:50] Probably. To the bz mobile, Robin! [19:45:39] related to the bug about broken renames [19:45:47] (group renames) [19:46:02] * Coren hunts for that bug [19:46:48] probably [19:51:16] lemme find it [19:51:19] i was cc'd on it [19:53:37] https://bugzilla.wikimedia.org/show_bug.cgi?id=46631 [19:58:32] It's obviously related, but I don't see how the patch fixes it; unless the switch to getUserCaseDBKey() solves it [19:59:20] PROBLEM - DPKG on cp1041 is CRITICAL: Timeout while attempting connection [20:00:10] RECOVERY - DPKG on cp1041 is OK: All packages OK [20:05:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:08:28] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 20:08:24 UTC 2013 [20:09:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:09:17] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 20:09:13 UTC 2013 [20:10:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:10:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 20:10:31 UTC 2013 [20:11:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:15:47] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [20:16:17] PROBLEM - SSH on cp1041 is CRITICAL: Connection timed out [20:17:07] RECOVERY - SSH on cp1041 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [20:29:37] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:31:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:57] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sun Apr 7 20:32:54 UTC 2013 [20:33:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:33:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [20:36:22] PROBLEM - Varnish HTCP daemon on cp1041 is CRITICAL: Timeout while attempting connection [20:37:22] RECOVERY - Varnish HTCP daemon on cp1041 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [20:39:14] PROBLEM - Varnish HTTP mobile-frontend on cp1041 is CRITICAL: Connection timed out [20:42:02] RECOVERY - Varnish HTTP mobile-frontend on cp1041 is OK: HTTP OK: HTTP/1.1 200 OK - 707 bytes in 0.989 second response time [20:56:32] PROBLEM - Disk space on cp1041 is CRITICAL: Timeout while attempting connection [20:57:22] RECOVERY - Disk space on cp1041 is OK: DISK OK [21:01:32] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [21:04:12] PROBLEM - SSH on cp1041 is CRITICAL: Connection timed out [21:05:09] RECOVERY - SSH on cp1041 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [21:05:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [21:18:18] PROBLEM - DPKG on cp1041 is CRITICAL: Timeout while attempting connection [21:18:28] PROBLEM - Varnish HTCP daemon on cp1041 is CRITICAL: Timeout while attempting connection [21:19:18] RECOVERY - Varnish HTCP daemon on cp1041 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [21:21:18] RECOVERY - DPKG on cp1041 is OK: All packages OK [21:30:08] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [21:33:08] PROBLEM - Varnish HTTP mobile-frontend on cp1041 is CRITICAL: Connection timed out [21:34:08] RECOVERY - Varnish HTTP mobile-frontend on cp1041 is OK: HTTP OK: HTTP/1.1 200 OK - 707 bytes in 0.886 second response time [21:41:18] PROBLEM - SSH on cp1041 is CRITICAL: Connection timed out [21:42:08] RECOVERY - SSH on cp1041 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [21:51:09] New patchset: Legoktm; "Restrict creation of new properties to new "propertycreator" group and sysops. The propertycreator group can be added/removed by sysops." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58036 [22:01:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [22:05:28] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [22:06:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [22:20:33] !log reedy synchronized php-1.22wmf1/extensions/CentralAuth [22:20:41] Logged the message, Master [22:20:54] Reedy: wanna merge https://gerrit.wikimedia.org/r/#/c/58036/ :) [22:21:39] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58036 [22:21:59] ty :) [22:22:38] New patchset: Reedy; "(bug 46828) Add patroller and autopatrolled groups on viwiki Added patroller and autopatrolled user groups on viwiki, and modified wgAddGroups, wgRemoveGroups according to the bug." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57497 [22:22:46] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57497 [22:23:00] New patchset: Reedy; "(bug 45638) Add patrol right to autopatrolled on itwikivoyage This change removes patrol right added to autoconfirmed group in I198651d7, and moves it to the autopatrolled group instead, per comment 9 in bug 45638." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57519 [22:23:13] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57519 [22:23:20] New patchset: Reedy; "(bug 46154) abusefilter-log-detail for autoconfirmed on thwiki This commit removes duplicate setting from InitialiseSettings.php and changes it in abusefilter.php from false to true per Nemo bis's comment below; thanks for spotting it!" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57515 [22:23:40] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57515 [22:24:59] New patchset: Reedy; "(bug 46882) Namespace 100 to be searched by default on sewikimedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57506 [22:25:08] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57506 [22:25:12] New patchset: Reedy; "(bug 46915) Enable the autopatrolled group on ru.wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57927 [22:25:34] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57927 [22:26:26] !log reedy synchronized wmf-config/ [22:26:33] Logged the message, Master [22:28:40] Reedy: according to https://www.wikidata.org/wiki/Special:ListGroupRights the right is still given to * even though i explicitly set it to false [22:28:45] https://gerrit.wikimedia.org/r/#/c/58036/1/wmf-config/InitialiseSettings.php [22:29:59] Uggggggggggggh [22:30:05] I think they broke git review again [22:30:13] o.O [22:30:14] New patchset: Reedy; "Remove disabled ReaderFeedback" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47532 [22:30:23] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47532 [22:30:48] PROBLEM - Puppet freshness on virt1000 is CRITICAL: No successful Puppet run in the last 10 hours [22:31:06] !log reedy synchronized wmf-config/ 'Kill readerfeedback config, it aint coming back' [22:31:13] Logged the message, Master [22:32:25] Right [22:32:27] So [22:33:32] legoktm: Overrides? [22:33:37] Not enough diff context ;) [22:33:48] well what could be overriding it? :P [22:33:57] nothing in that file should [22:34:03] * legoktm checks [22:34:13] 'groupOverrides' => array( [22:34:13] // Note: don't change the default setting here, because it won't take [22:34:13] // effect for the wikis which have overrides below. Instead change the [22:35:21] oh [22:35:24] so it has to be in [22:35:28] 'groupOverrides2' => array( [22:35:29] ? [22:35:42] I was just going to look where the original is/was set [22:36:00] wouldnt it be in the extension itself? [22:36:07] Presumably [22:36:11] That's what I was going to confirm [22:36:55] $wgGroupPermissions['*']['property-create'] = true; [22:36:55] $wgGroupPermissions['*']['property-remove'] = true; [22:37:04] Would've thought we should probably kill override too ;) [22:37:40] err, remove [22:37:55] oh, how do we do that? [22:39:15] !log reedy synchronized wmf-config/InitialiseSettings.php [22:39:22] Logged the message, Master [22:40:05] !log reedy synchronized wmf-config/InitialiseSettings.php [22:40:13] Logged the message, Master [22:46:03] !log reedy synchronized wmf-config/CommonSettings.php [22:46:09] Logged the message, Master [22:46:20] legoktm: That fixes it :D [22:46:41] yup! [22:46:46] whats the patchset though? [22:47:05] $wgGroupPermissions['*']['property-create'] = false; in CommonSettings [22:47:27] But like I say, shouldn't * be prevented from removing properties too? [22:47:28] $wgGroupPermissions['*']['property-create'] = false; [22:47:36] err [22:47:36] $wgGroupPermissions['*']['property-remove'] = true; [22:48:34] umm [22:48:39] im not sure what property-remove does [22:48:48] hmm [22:48:49] ok [22:48:56] Well, that's a different issue then ;) [22:49:22] 'right-property-override' => 'Override properties', [22:49:22] 'right-property-create' => 'Create properties', [22:49:22] 'right-property-remove' => 'Remove properties', [22:49:28] Fuck yeah, documentation [22:49:35] haha [22:49:45] i dont know how to override a property either [22:49:56] maybe thats for statements? [22:50:21] Reedy: is your fix for the jobqueue ganglia check still in CR? https://ganglia.wikimedia.org/latest/?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&m=cpu_report&r=hour&s=descending&hc=4&mc=2 [22:50:33] It was merged AFAIK [22:50:34] and is there a patch for that other error in the sh [22:53:29] ie notpeter> /bin/sh: 1: mwscript: not found [22:54:47] New patchset: Odder; "(bug 46990) Add the 'editor' restriction level on pl.wikipedia This change adds the 'editor' restriction level on pl.wikipedia, in between the 'autoconfirmed' and 'sysop' protection levels, so that pages can only be edited by users belonging to the 'edito" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58038 [22:55:43] Nemo_bis: No, I don't think so. I did detail the fix on IRC though [22:56:19] New patchset: Reedy; "$wgGroupPermissions['*']['property-create'] = false;" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58039 [22:56:27] Reedy: yes I remember that one [22:56:48] New review: Nemo bis; "Said to be broken due to: notpeter> /bin/sh: 1: mwscript: not found" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37441 [22:56:52] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58039 [22:57:15] Going to fix it? ;) [23:01:16] Revoked right [23:01:21] Does that even actually get used? :/ [23:01:55] not frequently [23:02:15] I was just thinking I can't recall ever seeing a user righte entry striked out [23:04:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [23:05:44] most wikis use userrights as a way of promotion, rather than as punishment [23:11:25] New patchset: MaxSem; "Prepare infrastructure for resource variance" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58043 [23:12:40] Reedy: fix how? [23:13:46] revoked rights are a recent thing [23:13:54] we mainly use them in private wikis IIRC [23:14:03] Nemo_bis: Fix the path to mwscript so it works [23:14:19] Reedy: and how am I supposed to know what's the right one? :) [23:14:32] Because I told you in IRC when Peter showed the error? [23:14:40] aww [23:14:52] ;) [23:14:59] you didn't say that was the right one [23:15:07] it could as well have been "this is the problem" [23:15:14] so you're sure about user apache? [23:15:16] Yes [23:15:18] And full path [23:15:25] oki