[00:01:55] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 5.365 second response time [00:10:55] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 118101 bytes in 5.543 second response time [00:44:22] can somebody deploy git master of the wikibugs bot? https://wikitech.wikimedia.org/wiki/Wikibugs [00:44:33] it seems to be running a version of code as of 10 months ago [00:45:45] I'm considering doing that [00:45:55] Are there any objections? [00:46:07] * rdwrer looks for mutante|away [00:47:25] Never mind, I don't have access to mchenry anyway [00:58:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [01:40:25] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [01:46:35] greg-g: We have a couple of backports for VE; rather than put them into the already-slightly-worrying Parsoid-enablement, I've added a Lightning Deploy for Roan. Hope this is OK. [01:46:40] (For Monday.) [02:25:01] !log LocalisationUpdate completed (1.23wmf15) at 2014-03-01 02:25:01+00:00 [02:25:10] Logged the message, Master [02:27:01] !log LocalisationUpdate completed (1.23wmf16) at 2014-03-01 02:27:01+00:00 [02:27:13] Logged the message, Master [02:28:25] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [02:32:45] m( it's says "Can Merge No" because I'm not allowed to... not because it needs a rebase -.- [02:33:03] (puppet) [02:33:20] m( is a very strange emoticon. [02:33:41] Gloria: Kind of like Facepalm... picked it up somewhere :P [02:33:49] :P [02:51:25] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [03:00:37] James_F: yeppers [03:02:37] greg-g: Awesome. [03:02:46] greg-g: (We really need to write the second bug fix, then. ;-)) [03:07:02] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-03-01 03:07:02+00:00 [03:07:11] Logged the message, Master [03:44:29] !log Disabled puppet on pmtpa tool labs; ignore puppet whining. [03:44:38] Logged the message, Master [03:47:11] (03PS1) 10Jalexander: Apache config for legalteamwiki [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 [03:53:54] (03CR) 10MZMcBride: "This change is pretty bad... you may want to abandon it in favor of having someone else take care of this." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [03:55:48] Jamesofur: I appreciate the intentions, but I'm not sure any of that is necessary. [03:55:50] Gloria: I'm happy to, but can you expand on bad? [03:55:58] I literally just copied what is up for every other wiki [03:56:08] (either in remnants or wikimedia.conf) [03:56:41] * Gloria peeks. [03:56:42] it seemed odd to have to repeat for each one but it's clear that they have been doing that [03:57:31] (03PS2) 10coren: Labs: Some tweaks and saner defaults for labs LVM [operations/puppet] - 10https://gerrit.wikimedia.org/r/116050 [03:57:59] Jamesofur: Eh, maybe you're right. I'm looking at remnant.conf now and boggling a bit. [03:58:24] UseMod compatibility? [03:58:34] yeah, I know the feeling, though singularly unqualified to judge better ways. I leave that debate to everyone else :) [03:58:45] my guess is that there is a large amount of copy/paste cruft.... [03:58:52] Yeah. [03:59:02] I hadn't realized this was so braindead. [03:59:13] I have no idea why there are different docroots. [03:59:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [03:59:33] (03CR) 10coren: [C: 032] Labs: Some tweaks and saner defaults for labs LVM [operations/puppet] - 10https://gerrit.wikimedia.org/r/116050 (owner: 10coren) [03:59:38] Or why it's necessary to redeclare some of this shit. [04:02:04] (03PS1) 10Jalexander: DNS setup for legalteamwiki [operations/dns] - 10https://gerrit.wikimedia.org/r/116220 [04:02:35] that one is at least much simpler [04:02:57] though also could use a clean up... [04:03:49] (03CR) 10MZMcBride: "Eh, having looked more closely, it seems the Apache configuration is generally a mess and that's outside the scope of this change. This ch" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [04:04:29] Jamesofur: It's weird, the chapter wikis seem to be able to share configuration pretty easily. [04:05:30] yeah, even if they still have all of the usemod cruft [04:06:01] I'd have less of an issue supporting no-op UseMod directives if they were declared once instead of a dozen times. [04:06:14] * Jamesofur nods [04:06:35] though how many wikis would that ever be a thing for? 2-3..? [04:07:11] A few more than that. :-) [04:07:33] I have no memory of exactly how many wikis we had when the change was made ;) [04:07:55] Meta-Wiki is ancient. Many of the large Wikipedias are ancient. [04:08:26] yeah, for some reason I thought even many of the ancient ones were post that but I'm clearly putting the switch far earlier in my mind then it really was [04:08:47] The DNS file is large and overwhelming, but doesn't seem like it needs cleanup. [04:11:44] http://no.wikimedia.org/stats/ <-- what... [04:16:43] Jamesofur: I'm fairly sure you want remnant.conf, BTW. Was there a reason you chose wikimedia.conf? [04:29:43] Gloria: I asked about it in wikimedia-dev, it seems like there is some confusion... there are 2 private wikis (transitionteam and iegcom ) in wikimedia.conf and the rest seem to be in remenant. Since I think those last 2 are the more recent ones and being in wikimedia.conf made sense given how the other operations files are set up I put them there. (and because it was the first one I opened and I didn't get a response i [04:29:44] n -dev ) [04:29:55] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 161 statistics [04:30:19] so basically I just went with the one that made the most sense in my head, little to know care which one it actually is. [04:30:26] *little to no [04:30:27] * Jamesofur sighs [04:31:55] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 15 statistics [04:41:25] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [04:56:13] (03CR) 10TTO: [C: 031] "It's not that bad...! I agree that the UseMod stuff could go though." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [05:13:00] (03CR) 10TTO: "This also requires the addition of symlinks (in /docroot/legalteam) in the mediawiki-config patch. It doesn't seem possible to do this on " [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [05:28:14] (03PS3) 10Jalexander: Initial setup for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 (owner: 10TTO) [05:29:25] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [05:41:34] and time to go home [05:48:29] (03CR) 10MZMcBride: "I filed bug 62093 about cleaning up the Apache configuration files." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [05:52:25] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [06:27:35] PROBLEM - udp2log log age for emery on emery is CRITICAL: CRITICAL: log files /a/log/webrequest/packet-loss.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:28:35] RECOVERY - udp2log log age for emery on emery is OK: OK: all log files active [06:40:41] (03CR) 10Legoktm: "'legoktm' is perfect, but I asked a question about other database access on the RT ticket though." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116133 (owner: 10ArielGlenn) [07:00:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [07:42:25] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [08:30:25] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [08:53:25] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [10:01:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [10:42:30] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [11:27:02] https://github.com/wikimedia/operations-mediawiki-config/pull/6 can somone merge this? [11:30:16] Steinsplitter: is that in gerit somewhere? if it isn't, then probably no one can until it's imported there [11:30:30] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [11:31:14] MatmaRex: i dos not have gerrit on pc, maby you can put the configfile to gerrit? [11:31:20] Steinsplitter: you can probably import it using http://tools.wmflabs.org/gerrit-patch-uploader/ + https://github.com/wikimedia/operations-mediawiki-config/pull/6.patch [11:31:53] good iddea, thx ;:) [11:32:25] Doesn't matter. [11:32:35] 1) No community consensus, 2) It doesn't even work. [11:35:12] odder: BureauOdder, really.... to much b ureaucracy. bether building som js one. [11:35:21] a) license revier have this tool [11:35:32] b) for such minor chanegs no consensus needed [11:35:42] its standard practise for community consensus [11:35:59] if we don't do it for one community, we may as well not do it for any [11:36:24] upload-by-url only works for Flickr, doesn't it? [11:36:30] yes [11:36:46] so why would OTRS-member users need this right? [11:36:48] bether bulding som jstol and enabling ths as default gadged : [11:37:42] why licens revier? [11:37:50] it is simply: becous it is evil to use the tools tools [11:37:58] ? [11:38:00] why using the tools one when we have a own [11:41:11] p858snake|l: you work for the wmf? [11:41:24] nope [11:41:55] good [11:42:13] haha [11:53:30] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [12:20:14] (03CR) 10Dr0ptp4kt: "By the way, I98e38fa651990a4efb5ab10bc34d5377b4edebcf now reflects the changes for ZeroRatedMobileAccess origin cookie Set-Cookie and for " [operations/puppet] - 10https://gerrit.wikimedia.org/r/115669 (owner: 10BBlack) [12:50:41] Steinsplitter, what on earth are you on about? [13:01:30] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [13:06:38] Krenair: ? [13:16:30] PROBLEM - Puppet freshness on labsdb1005 is CRITICAL: Last successful Puppet run was Sat 01 Mar 2014 10:15:50 AM UTC [13:43:30] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [14:31:30] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [14:54:30] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [15:48:45] (03CR) 10Reedy: [C: 04-1] "No, a new docroot folder shouldn't be added!!" (031 comment) [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [15:49:34] (03CR) 10Reedy: [C: 04-1] "A new docroot folder shouldn't be added!!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 (owner: 10TTO) [15:56:54] !log reedy synchronized php-1.23wmf16/includes/specials/SpecialRecentchanges.php 'Id323e3b7daced1e7b6b1e1add4e9e1bf7df05e4e' [15:57:00] Logged the message, Master [15:58:19] thanks Reedy [15:59:55] +1 :) [16:02:30] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [16:17:30] PROBLEM - Puppet freshness on labsdb1005 is CRITICAL: Last successful Puppet run was Sat 01 Mar 2014 10:15:50 AM UTC [16:44:30] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [17:31:10] PROBLEM - SSH on chromium is CRITICAL: Server answer: [17:32:30] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [17:33:10] RECOVERY - SSH on chromium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [17:39:10] PROBLEM - SSH on chromium is CRITICAL: Server answer: [17:40:10] RECOVERY - SSH on chromium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [17:55:30] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [18:28:00] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [19:03:30] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [19:16:18] Anyone else getting 502 Bad Gateway ? [19:16:22] eh, works now [19:18:30] PROBLEM - Puppet freshness on labsdb1005 is CRITICAL: Last successful Puppet run was Sat 01 Mar 2014 10:15:50 AM UTC [19:18:56] (03PS1) 10Odder: Add Wikimedia CH domain to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116263 [19:29:00] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [19:45:30] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [20:33:30] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [20:56:30] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [22:04:30] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [22:19:30] PROBLEM - Puppet freshness on labsdb1005 is CRITICAL: Last successful Puppet run was Sat 01 Mar 2014 10:15:50 AM UTC [22:34:00] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [22:46:05] greg-g: How was the wedding? (if it's happened) :p [22:46:30] PROBLEM - Puppet freshness on lanthanum is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 04:37:38 PM UTC [23:32:00] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [23:34:30] PROBLEM - Puppet freshness on gallium is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 05:25:59 PM UTC [23:56:06] JohnLewis: was fun :) My son ate too much chocolate, but other than that it went well :) [23:56:42] greg-g: Nice [23:57:30] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC