[00:17:00] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [00:17:40] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 2.87 ms [00:33:38] (03PS1) 10Mxn: Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) [00:35:05] (03PS2) 10Alex Monk: Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [00:37:26] (03CR) 10Alex Monk: Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [00:39:56] (03CR) 10Mxn: "I also saw high-resolution logos at , but I had no idea how to handle variants here." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) (owner: 10Mxn) [01:00:39] PROBLEM - Outgoing network saturation on labstore1001 is CRITICAL: CRITICAL: 23.33% of data above the critical threshold [100000000.0] [01:08:09] PROBLEM - RAID on analytics1004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:09:59] RECOVERY - RAID on analytics1004 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [01:17:29] PROBLEM - RAID on analytics1004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:19:19] RECOVERY - RAID on analytics1004 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [01:19:40] lots of NFS access from tools-webgrid-04 [01:19:46] * YuviPanda is attempting to log in there [01:25:01] PROBLEM - puppet last run on mw2028 is CRITICAL: CRITICAL: puppet fail [01:25:09] PROBLEM - RAID on analytics1004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:30:30] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [01:36:10] RECOVERY - Outgoing network saturation on labstore1001 is OK: OK: Less than 10.00% above the threshold [75000000.0] [01:43:59] RECOVERY - puppet last run on mw2028 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [01:45:40] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [01:47:40] RECOVERY - RAID on analytics1004 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [01:49:50] (03PS3) 10Mxn: Set $wgLogoHD for wikis that currently do so in MediaWiki:Common.css [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201915 (https://phabricator.wikimedia.org/T37337) [02:09:21] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:12:59] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 59665 bytes in 0.282 second response time [02:18:04] !log l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 06m 23s) [02:18:27] Logged the message, Master [02:22:49] !log LocalisationUpdate completed (1.25wmf23) at 2015-04-05 02:21:46+00:00 [02:22:56] Logged the message, Master [02:39:14] !log l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 05m 55s) [02:39:22] Logged the message, Master [02:43:51] !log LocalisationUpdate completed (1.25wmf24) at 2015-04-05 02:42:47+00:00 [02:43:58] Logged the message, Master [03:35:09] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Puppet has 1 failures [03:51:50] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:59:59] PROBLEM - puppet last run on mw2024 is CRITICAL: CRITICAL: puppet fail [04:07:29] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:12:19] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 59666 bytes in 0.082 second response time [04:18:19] RECOVERY - puppet last run on mw2024 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:26:56] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 5 04:25:52 UTC 2015 (duration 25m 51s) [04:27:02] Logged the message, Master [04:53:40] PROBLEM - puppet last run on mw2162 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:16] it would be awesome if the mobile site had a download link that autoconverted articles to kindle format [05:04:15] (03Abandoned) 10Ori.livneh: Dedupe severity. Dedupe severity. [debs/ircecho] - 10https://gerrit.wikimedia.org/r/201738 (owner: 10Ori.livneh) [05:05:38] PeperPots: what's "kindle format"? azw? According to https://en.wikipedia.org/wiki/Amazon_Kindle#Format_support_by_device , PDF and HTML is supported by newer devices. [05:05:54] i think so yes [05:06:12] PeperPots: yes to which, AZW? [05:06:24] PDF is available already, and HTML is the native format [05:06:26] yes [05:06:46] (03PS1) 10Yuvipanda: ircecho: Restart if any of the files involved change [puppet] - 10https://gerrit.wikimedia.org/r/201924 [05:06:53] pdf doesn't work on all readers [05:07:14] html and pdf don't scroll corectly [05:07:50] Yeah, AZW or mobi would be nice [05:08:11] azw would be perfect [05:09:54] PeperPots: http://www.ereadups.com/ does what you want, sort of [05:10:30] RECOVERY - puppet last run on mw2162 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [05:26:25] ori: only issue there is ERU won't load over 3g by default [05:27:05] PeperPots: your best bet would be to submit a feature request via https://phabricator.wikimedia.org/ [05:28:32] thanks will do [06:11:39] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:50] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 59698 bytes in 0.069 second response time [06:22:00] PROBLEM - Host mw2072 is DOWN: PING CRITICAL - Packet loss = 100% [06:23:49] RECOVERY - Host mw2072 is UP: PING OK - Packet loss = 0%, RTA = 43.79 ms [06:30:09] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:10] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:30] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:10] PROBLEM - puppet last run on mw2123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:11] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:11] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:11] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:59] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:00] PROBLEM - puppet last run on mw2143 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:09] PROBLEM - puppet last run on mw2022 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:09] PROBLEM - puppet last run on mw2097 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:09] PROBLEM - puppet last run on mw2093 is CRITICAL: CRITICAL: Puppet has 2 failures [06:36:20] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:40] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:46:20] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:46:50] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:46:50] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:48:00] RECOVERY - puppet last run on mw2143 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:48:00] RECOVERY - puppet last run on mw2022 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:48:01] RECOVERY - puppet last run on mw2097 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:01] RECOVERY - puppet last run on mw2093 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:01] RECOVERY - puppet last run on mw2123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:09] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:09] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:30] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:46] !log double ospf/ospf3 metric for eqiad-ulsfo GTT link; switch to other transport link [06:58:53] Logged the message, Master [07:04:58] (03PS1) 10Faidon Liambotis: Silence Phabricator/Bugzilla migration cronspam [puppet] - 10https://gerrit.wikimedia.org/r/201930 [07:32:40] PROBLEM - Host mw2027 is DOWN: PING CRITICAL - Packet loss = 100% [07:33:50] RECOVERY - Host mw2027 is UP: PING WARNING - Packet loss = 54%, RTA = 43.83 ms [08:29:19] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: puppet fail [08:47:49] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:25:50] PROBLEM - Outgoing network saturation on labstore1001 is CRITICAL: CRITICAL: 10.34% of data above the critical threshold [100000000.0] [10:10:00] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [10:26:49] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [10:27:50] RECOVERY - Outgoing network saturation on labstore1001 is OK: OK: Less than 10.00% above the threshold [75000000.0] [10:40:24] 7Puppet, 6Labs: Labs: Could not find dependency File[/usr/lib/ganglia/python_modules] for File[/usr/lib/ganglia/python_modules/gmond_memcached.py] - https://phabricator.wikimedia.org/T95107#1180505 (10Tgr) 3NEW [11:43:42] (03PS1) 10Glaisher: Add 'editeditorprotected' protection level on dewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201940 (https://phabricator.wikimedia.org/T94368) [11:48:09] PROBLEM - puppet last run on mw2015 is CRITICAL: CRITICAL: puppet fail [12:00:25] (03PS1) 10Gergő Tisza: Set has_ganglia=false for labs [puppet] - 10https://gerrit.wikimedia.org/r/201942 (https://phabricator.wikimedia.org/T95107) [12:05:00] RECOVERY - puppet last run on mw2015 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [12:08:00] (03PS1) 10Glaisher: Add $wgUploadNavigationUrl for iswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201943 (https://phabricator.wikimedia.org/T95089) [12:08:16] (03PS2) 10Glaisher: Add $wgUploadNavigationUrl for iswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201943 (https://phabricator.wikimedia.org/T95089) [12:14:43] 48 Lost parent, LightProcess exiting [12:15:31] all from mw1081, again [12:20:03] although looking back in the archives mw1114 has done it before as well, to a lesser extent [12:22:39] strange hhvm thing, apparently: https://github.com/facebook/hhvm/blob/master/hphp/util/light-process.cpp#L600 [12:40:22] Krenair: Interesting. Any lead on a cause or a phabricator task? [12:41:40] PROBLEM - puppet last run on mw2131 is CRITICAL: CRITICAL: Puppet has 1 failures [12:58:20] RECOVERY - puppet last run on mw2131 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:00:30] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: puppet fail [13:18:59] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:28:47] nope [15:11:47] (03CR) 10Nikerabbit: Add pool counter config for Translate (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/199263 (https://phabricator.wikimedia.org/T54728) (owner: 10Nikerabbit) [15:25:19] (03CR) 10JanZerebecki: [C: 04-1] "Needs careful checking, as explained by Yuvipanda." [puppet] - 10https://gerrit.wikimedia.org/r/181949 (owner: 10Hoo man) [16:10:50] 7Puppet, 6Labs: geoipupdate package not found for jessie - https://phabricator.wikimedia.org/T95117#1180714 (10Tgr) 3NEW [16:15:54] 7Puppet, 6Labs: geoipupdate package not found for jessie - https://phabricator.wikimedia.org/T95117#1180733 (10Tgr) [16:15:58] 6operations, 10Staging: Package geoipupdate for jessie - https://phabricator.wikimedia.org/T90229#1180735 (10Tgr) [16:17:07] (03PS1) 10John F. Lewis: labs: update bastion values in network.pp [puppet] - 10https://gerrit.wikimedia.org/r/201957 [16:19:50] (03PS1) 10John F. Lewis: network: add uranium.wm ipv6 to def [puppet] - 10https://gerrit.wikimedia.org/r/201958 [16:47:11] (03CR) 10Steinsplitter: [C: 031] "looks ok" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201940 (https://phabricator.wikimedia.org/T94368) (owner: 10Glaisher) [17:16:21] (03PS1) 10John F. Lewis: Configure EventLogging on wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) [17:18:32] 6operations, 10Analytics-EventLogging, 10Wikimedia-Site-requests, 5Patch-For-Review: wikitech.wikimedia.org error "$wgEventLoggingBaseUri is not set." - https://phabricator.wikimedia.org/T84965#1180799 (10JohnLewis) a:5Andrew>3JohnLewis [17:21:48] Request: POST http://incubator.wikimedia.org/w/index.php?title=Special:Import&action=submit, from 10.20.0.152 via cp1068 cp1068 ([10.64.0.105]:3128), Varnish XID 3538243820 [17:21:52] Forwarded for: 88.101.17.127, 10.20.0.152, 10.20.0.152, 10.20.0.152 [17:21:55] Error: 503, Service Unavailable at Sun, 05 Apr 2015 17:21:19 GMT [17:22:05] continuously, won't let me import [17:22:33] (03PS1) 10Ori.livneh: postgresql: choose distro-appropriate default version [puppet] - 10https://gerrit.wikimedia.org/r/201962 [17:23:32] Danny_B: is this import upload? [17:24:27] (03CR) 10Faidon Liambotis: [C: 04-1] "There are metapackages (postgresql & postgresql-client, respectively) pointing to the right default version per distribution. No need to r" [puppet] - 10https://gerrit.wikimedia.org/r/201962 (owner: 10Ori.livneh) [17:24:55] evening paravoid :) [17:24:59] hello :) [17:25:45] you're right, but that's a bigger change to the module [17:25:50] Steinsplitter: yup [17:26:02] Steinsplitter: known issue? [17:26:11] any workarounds? [17:26:26] Danny B: how much pages? There is a limit. Try maximal 20/30 pages. depends on how much revs [17:26:35] bigger because $pgversion is being used for file paths too you mean? [17:26:50] yeah [17:26:59] and package names [17:27:10] package names is easy [17:27:23] only append the suffix is $pgversion is not undef [17:27:29] right [17:28:16] but puppet strings are immutable so you need variables for each and i'm lazyyyyyyy [17:28:24] haha [17:30:25] it's ok, I don't have a strong opinion [17:30:43] I'd prefer it we didn't hardcode versions of course, but if it's too difficult/not worth the trouble that's fine [17:30:52] (03CR) 10Ori.livneh: "i agree that letting apt do the heavy lifting in figuring out the appropriate version is the right thing to do, but it's a bigger change t" [puppet] - 10https://gerrit.wikimedia.org/r/201962 (owner: 10Ori.livneh) [17:31:19] Steinsplitter: single page with all history. xml has about 670kb [17:32:13] paravoid: yeah i actually had a bigger change staged already, cleaning up whitespace and other minor things, but then i went back and unstaged everything other than the minimum required to unblock tgr (who is experimenting with setting up sentry on jessie in labs) [17:32:14] 670 schouldn't be a problem. i never had problems :-/ [17:32:57] wondering about incompatibility of xml versions? [17:33:44] ori: yeah I'm okay [17:33:50] I don't like the default value though [17:33:51] Danny_B: a useful thing is if you can provide information such as the mediawiki host (look at the page source code, find $wgHostname) as well [17:34:06] I also disagree with the comment, it clearly doesn't default to 9.1 :) [17:34:24] oh yeah, the comment needs to be fixed [17:34:32] what do you suggest for default => ? having it fail if the selector doesn't match? [17:34:33] (03CR) 10Krinkle: "I don't think that would work. Wikitech does not have a Schema namespace. And the extensions installed on wikitech are not referring to is" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) (owner: 10John F. Lewis) [17:34:48] I guess [17:34:55] although fail() in those selectors doesn't work [17:35:08] happy easter btw (is it today? the orthodox church has it on a different date iirc?) [17:35:13] well you can just omit 'default =>' [17:35:15] and it'll fail [17:35:18] yeah [17:35:28] JohnLewis: source of bsod? [17:35:31] but add precise too [17:35:44] (03CR) 10John F. Lewis: "@Krinkle should just removing the whole if statement work in the case then?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) (owner: 10John F. Lewis) [17:35:50] for backwards compat, in case we have a caller somewhere with precise defaulting to 9.1... [17:35:59] orthodox easter is next week [17:36:18] the orthodox church uses a different calendar or something [17:36:41] Danny_B: before you press submit, view the page source code (depends on which browser on how you do it) and look for a "$wghostname" variable [17:37:16] it's even stupider than DST [17:37:25] nothing is stupider than DST [17:38:36] (03PS2) 10John F. Lewis: Configure EventLogging on wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) [17:39:24] ori: double DST perhaps? [17:39:46] (03PS2) 10Ori.livneh: postgresql: choose distro-appropriate default version [puppet] - 10https://gerrit.wikimedia.org/r/201962 [17:40:33] (03PS3) 10Faidon Liambotis: postgresql: choose distro-appropriate default version [puppet] - 10https://gerrit.wikimedia.org/r/201962 (owner: 10Ori.livneh) [17:40:40] (03CR) 10Ori.livneh: [C: 04-1] "First, thanks for this. Second, it'd be better to just remove the if ( $wgDBname === 'labswiki' ) block, so that the catch-all else is use" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) (owner: 10John F. Lewis) [17:40:42] (03CR) 10Faidon Liambotis: [C: 032] postgresql: choose distro-appropriate default version [puppet] - 10https://gerrit.wikimedia.org/r/201962 (owner: 10Ori.livneh) [17:40:59] paravoid: thanks! [17:41:07] tgr|away: voila ^ [17:41:11] thank you :) [17:42:00] JohnLewis: mw.config.set({"wgBackendResponseTime":153,"wgHostname":"mw1243"}); [17:42:03] Request: POST http://incubator.wikimedia.org/w/index.php?title=Special:Import&action=submit, from 10.20.0.152 via cp1068 cp1068 ([10.64.0.105]:3128), Varnish XID 3540267962 [17:42:04] I'll avoid making an inappropriate/controversial comment about religion & stupidity [17:42:07] Forwarded for: 88.101.17.127, 10.20.0.152, 10.20.0.152, 10.20.0.152 [17:42:10] Error: 503, Service Unavailable at Sun, 05 Apr 2015 17:41:38 GMT [17:42:30] (03PS3) 10John F. Lewis: Remove if labswiki clause [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) [17:43:13] wth [17:43:28] some strontium failure [17:43:49] weird [17:46:04] Danny_B: does the revisions actually get imported? [17:46:43] tto was working on this, I think [17:47:56] Glaisher: https://incubator.wikimedia.org/wiki/Special:Log/import nope [17:48:48] Hm, then it's not the same issue. Imports were throwing 503s last week too. [17:48:56] but it actually gets imported [17:49:02] and the log entries were not created [17:49:38] https://phabricator.wikimedia.org/T94325 closed, resolved [17:50:11] let's see then [17:50:44] nope, not there [17:50:50] neither page nor log [17:53:47] https://wikitech.wikimedia.org/wiki/Special:Contributions/WikitechAnio243 [17:53:49] wtf [17:54:02] Danny_B: file a bug? [18:01:30] setting the import source might be faster, if there is anybody here who could scap it [18:02:18] (03PS1) 10John F. Lewis: bugzilla: use https by default for static [puppet] - 10https://gerrit.wikimedia.org/r/201964 [18:02:55] Danny_B: anyone can make a patch but getting it deployed today might be an issue [18:03:56] 7Blocked-on-Operations, 6operations: Package or import mod_pagespeed for Trusty - https://phabricator.wikimedia.org/T95123#1180812 (10ori) 3NEW [18:04:18] paravoid: ^ fyi, if you have any suggestions [18:05:51] if anybody is willing to file the bug on behalf of myself, i'd appreciate that. in a nutshell: "incubator on file import raises 503 bsod" [18:13:44] 7Blocked-on-Operations, 6operations: Package or import mod_pagespeed for Trusty - https://phabricator.wikimedia.org/T95123#1180821 (10ori) [19:49:17] (03PS4) 10Krinkle: Remove if labswiki clause for EventLogging config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) (owner: 10John F. Lewis) [19:51:02] 6operations: Package or import PageSpeed module (mod_pagespeed) - https://phabricator.wikimedia.org/T95123#1180901 (10ori) [19:51:48] (03CR) 10Ori.livneh: [C: 032] Remove if labswiki clause for EventLogging config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) (owner: 10John F. Lewis) [19:51:53] (03Merged) 10jenkins-bot: Remove if labswiki clause for EventLogging config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201961 (https://phabricator.wikimedia.org/T84965) (owner: 10John F. Lewis) [19:54:56] 6operations, 10Analytics-EventLogging, 10Wikimedia-Site-requests, 5Patch-For-Review: wikitech.wikimedia.org error "$wgEventLoggingBaseUri is not set." - https://phabricator.wikimedia.org/T84965#1180914 (10ori) 5Open>3Resolved Thanks, @JohnLewis. [19:55:13] ori: welcome :p [19:57:00] 6operations, 10Continuous-Integration, 6Labs: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1180919 (10Krinkle) As long as the "number of failures" (e.g. errors/warnings, not the complete failures) are counted in Graphite and result in Shinken e-mails. Is that... [19:59:55] 6operations, 10Continuous-Integration, 6Labs: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1180922 (10yuvipanda) They are counted in graphite, and present in the emails just not particularly clearly. [20:06:06] ori, JohnLewis: hey. does virt1000 still run mediawiki? [20:06:57] Krenair: to my knowledge it does not anymore [20:06:58] (03CR) 10Alex Monk: "I87a6d010" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158405 (owner: 10Andrew Bogott) [20:07:11] Krenair: doesn't look like it [20:07:40] I ask because virt1000 still produces entries in eventlogging.log [20:07:44] rather than silver [20:08:46] shouldn't [20:09:02] but maybe apache / jobrunner weren't shut down? [20:09:38] it is always at one second past every minute, same pair of errors twice [20:09:40] * YuviPanda is still in bed on phone cant check [20:09:54] cron or jobrunner i guess? [20:10:05] probably cron [20:10:43] maybe apache hadn't been restarted [20:10:53] because it did at one point run mediawiki, and still has a mediawiki source tree [20:11:37] apache has been running since Fri Mar 13 12:17:35 2015 [20:54:55] (03CR) 10JanZerebecki: [C: 04-1] "Having this the other way around would also be uniform. By using a newline one could also make it visible that the module name is the same" [puppet] - 10https://gerrit.wikimedia.org/r/201884 (https://phabricator.wikimedia.org/T87132) (owner: 10Tim Landscheidt) [21:36:11] ori: thanks a lot! [21:42:04] 6operations, 10Continuous-Integration, 6Labs: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1180998 (10Krinkle) 5Open>3Resolved a:3Krinkle [22:09:32] (03CR) 10Yuvipanda: [C: 031] "(I'll merge monday)" [puppet] - 10https://gerrit.wikimedia.org/r/201957 (owner: 10John F. Lewis) [22:09:45] (03Abandoned) 10Yuvipanda: Make ircecho run as ircecho user [debs/ircecho] - 10https://gerrit.wikimedia.org/r/176333 (https://phabricator.wikimedia.org/T76203) (owner: 10Yuvipanda) [22:32:38] (03CR) 10Tim Landscheidt: "@JanZerebecki: Do you want to submit a change to illustrate your idea?" [puppet] - 10https://gerrit.wikimedia.org/r/201884 (https://phabricator.wikimedia.org/T87132) (owner: 10Tim Landscheidt) [22:37:35] (03PS1) 10Mjbmr: Add autopatrolled for fawikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201977 [22:57:49] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: puppet fail [23:14:40] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:28:35] (03CR) 10Alex Monk: [C: 04-2] "There's only one vote there..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/201977 (owner: 10Mjbmr) [23:33:51] 6operations, 10Staging: Package geoipupdate for jessie - https://phabricator.wikimedia.org/T90229#1181077 (10ori) >>! In T90229#1054901, @faidon wrote: > It should be trivial to build for jessie and include into jessie-wikimedia, at least until jessie-backports comes to life. Done. [23:34:05] 6operations, 10Staging: Package geoipupdate for jessie - https://phabricator.wikimedia.org/T90229#1181078 (10ori) 5Open>3Resolved [23:34:10] tgr: ^ [23:35:50] ori: thanks again! [23:36:02] tgr: is https://phabricator.wikimedia.org/T95107 blocking you? [23:36:54] just out of curiosity, how did you do that? I tried to figure out from wikitech how to repackage it, but there are at least four different tutorials and none of them complete [23:38:27] T95107 might be a blocker when we try to deploy sentry, for now I can just apply the patch locally [23:51:29] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: puppet fail [23:56:35] tgr: https://phab.wmfusercontent.org/file/data/ba35mumgwqtp762ssav3/PHID-FILE-w6no76llthdochs5ozv2/e75v5lvrnvswxz3z/building_geoipupdate