[00:01:22] 3operations, Analytics-Engineering, Analytics-Cluster: Install Fortran packages on stat1002 and stat1003 - https://phabricator.wikimedia.org/T89414#1038180 (10Halfak) Confirmed that scipy and numpy can be compiled successfully. Thanks all! [00:04:16] (03PS1) 10BryanDavis: Revert "beta: switch logstash transport from redis to syslog" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190602 [00:04:57] awight: One more config change if that's ok [00:05:28] bd808: I'm idle! [00:05:37] (03CR) 10BryanDavis: [C: 032] Revert "beta: switch logstash transport from redis to syslog" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190602 (owner: 10BryanDavis) [00:05:58] (03Merged) 10jenkins-bot: Revert "beta: switch logstash transport from redis to syslog" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190602 (owner: 10BryanDavis) [00:06:33] !log bd808 Synchronized wmf-config/logging-labs.php: Revert: Switch beta to syslog logging (d9dcccb) (duration: 00m 06s) [00:06:38] Logged the message, Master [00:07:37] (03CR) 10BryanDavis: "Revert was because we still have monolog 1.11.0 in mediawiki/vendor and it has a broken syslog handler. Otherwise the patch worked perfect" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190246 (https://phabricator.wikimedia.org/T88870) (owner: 10BryanDavis) [00:10:20] awight: {{done}} (and back into the code to fix the problem this exposed) [00:11:20] 3operations, Wikimedia-Logstash, Incident-20150205-SiteOutage, MediaWiki-Core-Team: Prototype Monolog and rsyslog configuration to ship log events from MediaWiki to Logstash - https://phabricator.wikimedia.org/T88870#1038205 (10bd808) [00:11:26] bd808: rad, thx [00:15:34] (03PS3) 10Dzahn: static-bz: rewrite /show_bug.cgi to static HTML [puppet] - 10https://gerrit.wikimedia.org/r/190132 (https://phabricator.wikimedia.org/T85140) [00:18:27] (03CR) 10Dzahn: [C: 032] static-bz: rewrite /show_bug.cgi to static HTML [puppet] - 10https://gerrit.wikimedia.org/r/190132 (https://phabricator.wikimedia.org/T85140) (owner: 10Dzahn) [00:24:42] 3operations, Wikimedia-Bugzilla, Phabricator: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#1038215 (10Dzahn) these URLs with "show_bug.cgi" work now on static-bugzilla. so users can just insert the "static" part into original URLs example: http://static-bugzilla.wikimedia... [00:30:37] (03PS1) 10BBlack: varnish+jessie filesystem stuff [puppet] - 10https://gerrit.wikimedia.org/r/190610 [00:33:40] greg-g: sorry to use you as a hub. I'm stealing a little more daylight savings deployment time [00:33:51] (03PS1) 10Dzahn: static-bz: also rewrite activity URLs [puppet] - 10https://gerrit.wikimedia.org/r/190613 (https://phabricator.wikimedia.org/T85140) [00:34:01] awight: still on that banner stuffs? [00:34:59] yeeeah more debug logging [00:35:37] :) [00:38:53] 3operations, Wikimedia-Bugzilla, Phabricator: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#1038257 (10jayvdb) No doubt you are aware, but just noting: * the header on static-bugzilla pages is / will be wrong ``` In order to access the Phabricator task corresponding to a... [00:39:04] (03PS2) 10Dzahn: static-bz: also rewrite activity URLs [puppet] - 10https://gerrit.wikimedia.org/r/190613 (https://phabricator.wikimedia.org/T85140) [00:41:03] (03CR) 10Dzahn: [C: 032] static-bz: also rewrite activity URLs [puppet] - 10https://gerrit.wikimedia.org/r/190613 (https://phabricator.wikimedia.org/T85140) (owner: 10Dzahn) [00:42:49] 3operations, Wikimedia-Bugzilla, Phabricator: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#1038270 (10Dzahn) >>! In T85140#1038257, @jayvdb wrote: > * need rewrite rule for /show_activity.cgi?id=1 -> activity1.html done just now. https://old-bugzilla.wikimedia.org/show_ac... [00:47:17] AndyRussG: buckle up! here goes the last logging patch [00:47:32] !log awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s) [00:47:45] !log awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s) [00:47:47] Logged the message, Master [00:48:47] awight: logs coming in.... nowhere near the expceted volume, tho [00:49:32] really? That seems about right to me [00:49:43] 30k requests/day, but how many fatal errors? [00:52:40] 3operations, Wikimedia-Bugzilla, Phabricator: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#1038283 (10Dzahn) >>! In T85140#1038257, @jayvdb wrote: > * the header on static-bugzilla pages is / will be wrong sed 's/remove \"old-\"/remove \"static-\"/' bug1.html in a loop on... [00:53:37] * awight runs out of gas and ditches the stolen car in a culvert < greg-g [00:55:45] "Suspect is on the run. Repeat. Suspect is on the run." [00:56:23] 3operations: wikitech-static is out of date - https://phabricator.wikimedia.org/T87202#1038289 (10Dzahn) >>! In T87202#985864, @Reedy wrote: > It's also on 1.25wmf2 (a bit out of date) Now on 1.25wmf16. Should be in dsh/scap now. [00:57:30] 3operations: wikitech-static is out of date - https://phabricator.wikimedia.org/T87202#1038291 (10Dzahn) duplicate task: T83596 add monitoring to detect this: T89323 [00:58:11] awight: AndyRussG you guys rock, btw. [00:58:13] 3operations: wikitech-static is out of date - https://phabricator.wikimedia.org/T87202#1038294 (10Dzahn) [00:58:28] * awight gets the git log drunk and it starts singing loudly [00:58:48] greg-g: awight gets all the credit here :) [00:59:34] oh stop :p I just pushed your patches into the street [01:00:13] plus, I'm not sure there's much credit available :) it was mostly cleaning up after all the vomiting I did in the code on Tuesday [01:01:04] 3operations: Decommission svn.wikimedia.org server (import SVN into Phabricator) - https://phabricator.wikimedia.org/T86655#1038300 (10Dzahn) [01:01:06] 3operations, Wikimedia-General-or-Unknown: svn.wikimedia.org security certificate expired - https://phabricator.wikimedia.org/T88731#1038299 (10Dzahn) [01:01:40] awight: it was fine, who'd have known about these requests from the netherworld [01:02:15] :) [01:02:18] have a good weekend [01:02:21] awight: definitely many congratulations! [01:02:29] greg-g: thanks also so much for the help! [01:02:43] greg-g: cya [01:02:49] "Go team!" /me awkward high fives [01:03:51] * AndyRussG hi fives with snow gloves [01:03:59] 3operations: Decommission svn.wikimedia.org server (import SVN into Phabricator) - https://phabricator.wikimedia.org/T86655#1038324 (10Dzahn) what Faidon said, just pay for it. but let's not keep ignoring the icinga check (or remove it if we don't consider it critical) [01:04:10] * awight chuckles and gives greg-g a stinger [01:04:10] mutante, hi [01:05:19] Krenair: hi, you got me a minute before i was about to leave [01:05:31] https://phabricator.wikimedia.org/T87202 - Dzahn closed this task as a duplicate of Restricted Task. [01:05:55] the link in the email shows its WMF-NDA only [01:06:01] Krenair: oh, i didn't notice. it's because the older one was imported from RT [01:06:21] 3operations: fix wikitech-static updates - https://phabricator.wikimedia.org/T83596#1038334 (10Dzahn) [01:06:26] Krenair: fixed [01:06:38] I don't know if I should be in that project [01:06:55] fixed [01:06:57] I do have deployment and security access [01:06:58] oh, to slow [01:07:09] but the task edit works, thanks mutante [01:07:11] Krenair: you probably should [01:07:11] 3operations: fix wikitech-static updates - https://phabricator.wikimedia.org/T83596#916108 (10hoo) [01:07:33] Krenair: try out the NDA volunteer process :) [01:07:43] it needs a test anyways [01:08:32] I don't really have an actual need to be in that group at the moment. [01:08:55] i wish "merge" which also merge all the comments [01:09:28] yea, ok, keep asking me if this happens. it's because RT imports are NDA by default but many/most of them can just be released to public [01:10:14] 3operations, Wikimedia-Labs-wikitech-interface: fix wikitech-static updates - https://phabricator.wikimedia.org/T83596#1038380 (10Dzahn) [01:10:34] have a nice weekend. cya [01:10:36] how many are these? Maybe someone can take a few hours and go through all them? [01:11:21] hoo: around 8000 [01:11:31] Ok, that's more than a few hours :S [01:11:33] just estimate [01:12:22] the plan is to check them, but it will take some time, yea [01:12:25] bye, gotta run [01:17:21] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep [01:17:35] not prod, just a new error for me [01:29:56] too deep! [01:30:21] https://www.youtube.com/watch?v=Ipp8v56Jy5Q [01:39:25] (03PS2) 10Ori.livneh: xenon: drop minwidth arg from flamegraph.pl invocation [puppet] - 10https://gerrit.wikimedia.org/r/190513 [01:39:31] (03CR) 10Ori.livneh: [C: 032 V: 032] xenon: drop minwidth arg from flamegraph.pl invocation [puppet] - 10https://gerrit.wikimedia.org/r/190513 (owner: 10Ori.livneh) [01:42:00] PROBLEM - Ori committing changes on the weekend on palladium is CRITICAL: CRITICAL: Ori committed a change on a weekend [01:44:11] 3operations, Wikimedia-Logstash, Incident-20150205-SiteOutage, MediaWiki-Core-Team: Prototype Monolog and rsyslog configuration to ship log events from MediaWiki to Logstash - https://phabricator.wikimedia.org/T88870#1038473 (10Legoktm) [01:46:33] (03PS1) 10Ori.livneh: vbench: re-introduce warmup runs; run 30 times by default [puppet] - 10https://gerrit.wikimedia.org/r/190624 [01:46:44] (03PS2) 10Ori.livneh: vbench: re-introduce warmup runs; run 30 times by default [puppet] - 10https://gerrit.wikimedia.org/r/190624 [01:46:49] (03CR) 10Ori.livneh: [C: 032 V: 032] vbench: re-introduce warmup runs; run 30 times by default [puppet] - 10https://gerrit.wikimedia.org/r/190624 (owner: 10Ori.livneh) [02:00:18] 3operations, Wikimedia-Logstash, Incident-20150205-SiteOutage, MediaWiki-Core-Team: Prototype Monolog and rsyslog configuration to ship log events from MediaWiki to Logstash - https://phabricator.wikimedia.org/T88870#1038523 (10Legoktm) [02:06:31] (03PS1) 10BryanDavis: Revert "Revert "beta: switch logstash transport from redis to syslog"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190629 [02:07:53] (03PS2) 10BryanDavis: Revert "Revert "beta: switch logstash transport from redis to syslog"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190629 (https://phabricator.wikimedia.org/T88870) [02:16:43] !log l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s) [02:16:47] Logged the message, Master [02:17:50] !log LocalisationUpdate completed (1.25wmf16) at 2015-02-14 02:16:47+00:00 [02:17:53] Logged the message, Master [02:30:58] !log l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s) [02:31:03] Logged the message, Master [02:32:06] !log LocalisationUpdate completed (1.25wmf17) at 2015-02-14 02:31:02+00:00 [02:32:09] Logged the message, Master [02:47:31] RECOVERY - Ori committing changes on the weekend on palladium is OK: OK: Ori is behaving himself [03:21:48] !dn foo [03:21:55] !dn act files $* into dev/null as requested. [03:21:55] Key was added [03:22:00] ;D [03:31:09] 3operations, Beta-Cluster: Make www-data the web-serving user (is currently apache) - https://phabricator.wikimedia.org/T78076#1038715 (10bd808) [03:32:08] 3operations, CirrusSearch: Poke ops about an elasticsearch cluster restart - https://phabricator.wikimedia.org/T88354#1038717 (10bd808) [04:02:04] 3operations, MediaWiki-extensions-GWToolset, Multimedia: Can Commons support a mass upload of 14 million files (1.5 TB)? - https://phabricator.wikimedia.org/T88758#1038752 (10dan-nl) @Harej, > Would a 14 million entry XML file be too big? yes, as @Reedy already mentioned. i believe the largest xml file i used d... [04:31:06] 3operations, Wikimedia-General-or-Unknown: Add a wiki on wikitech is out of date, incomplete - https://phabricator.wikimedia.org/T87588#1038783 (10yuvipanda) [04:48:40] 3operations: jessie kernel vm subsystem issues for upload caches - https://phabricator.wikimedia.org/T88996#1038801 (10BBlack) Status update: moving to a 3.18-based kernel from debian-experimental (on top of VM tuning and the ext4 switch, etc) has gotten this issue under control on cp1064. The vm subsystem chan... [04:49:15] (03PS1) 10Hoo man: Set $wgCentralAuthPreventUnattached = true; on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190634 [04:50:26] (03PS2) 10Hoo man: Set $wgCentralAuthPreventUnattached = true; on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190634 (https://phabricator.wikimedia.org/T70069) [04:51:46] (03CR) 10Keegan: [C: 031] "It's time" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190634 (https://phabricator.wikimedia.org/T70069) (owner: 10Hoo man) [04:54:06] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 14 04:53:02 UTC 2015 (duration 53m 1s) [04:54:12] Logged the message, Master [05:18:27] (03CR) 10Legoktm: [C: 031] Set $wgCentralAuthPreventUnattached = true; on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190634 (https://phabricator.wikimedia.org/T70069) (owner: 10Hoo man) [06:28:10] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:51] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:01] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:01] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:01] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:31] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:11] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:46:01] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:46:01] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:46:20] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:10] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:47:11] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:56:41] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: puppet fail [07:15:53] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:46:41] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 1 failures [08:04:51] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [11:55:24] 3operations, Scrum-of-Scrums, Services, RESTBase: RESTbase deployment - https://phabricator.wikimedia.org/T1228#1039015 (10mobrovac) [11:57:07] 3operations, Scrum-of-Scrums, Services, RESTBase: RESTbase deployment - https://phabricator.wikimedia.org/T1228#787155 (10mobrovac) [13:22:28] (03Abandoned) 10KartikMistry: cxserver: Enable English to Russian MT [puppet] - 10https://gerrit.wikimedia.org/r/188517 (owner: 10KartikMistry) [13:38:18] (03PS15) 10KartikMistry: cxserver: Add Yandex support [puppet] - 10https://gerrit.wikimedia.org/r/186538 (https://phabricator.wikimedia.org/T88512) [14:03:14] 3operations, Wikimedia-General-or-Unknown: Add a wiki on wikitech is out of date, incomplete - https://phabricator.wikimedia.org/T87588#1039093 (10Aklapper) [14:35:27] What is way to get notification if service is down. Obviously, my interest is cxserver/apertium here. [14:35:38] akosiaris: ^^ [14:35:54] (Beta is being taken care of) [14:36:17] kart_: Set up an icinga alert, make it cry to you [14:36:45] hoo: done via puppet? [14:37:06] yeah [14:39:25] hoo: thanks! [14:45:25] 3operations, Wikimedia-General-or-Unknown: Add a wiki on wikitech is out of date, incomplete - https://phabricator.wikimedia.org/T87588#1039118 (10yuvipanda) [18:03:01] 3operations, Scrum-of-Scrums, Services, RESTBase: RESTbase deployment - https://phabricator.wikimedia.org/T1228#1039274 (10GWicke) [18:15:43] 3operations, Scrum-of-Scrums, Services, RESTBase: RESTbase deployment - https://phabricator.wikimedia.org/T1228#1039295 (10mobrovac) [18:15:46] 3operations, Scrum-of-Scrums, Services, RESTBase: RESTbase deployment - https://phabricator.wikimedia.org/T1228#1039297 (10GWicke) [18:40:59] (03PS2) 10Tim Landscheidt: Use apt::repository instead of file resources [puppet] - 10https://gerrit.wikimedia.org/r/123903 [18:42:42] (03PS3) 10Tim Landscheidt: Use apt::repository instead of file resources [puppet] - 10https://gerrit.wikimedia.org/r/123903 [19:44:45] 3operations, Services: Create a standard service template / init / logging / package setup - https://phabricator.wikimedia.org/T88585#1039367 (10GWicke) We have started to package our existing tools around nodejs clustering, logging, statsd reporting, config loading in https://github.com/wikimedia/supervisoid. B... [19:46:15] 3operations, Services, Parsoid: Create a standard service template / init / logging / package setup - https://phabricator.wikimedia.org/T88585#1039368 (10GWicke) [20:07:56] (03PS1) 10Matanya: localssl: fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/190666 [20:10:04] (03PS1) 10Matanya: webserver: fix string containing only a variable [puppet] - 10https://gerrit.wikimedia.org/r/190667 [20:11:07] (03PS1) 10Matanya: fix 5 string containing only a variable [puppet] - 10https://gerrit.wikimedia.org/r/190668 [20:13:06] (03PS1) 10Matanya: offload: fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/190669 [20:13:52] (03CR) 10jenkins-bot: [V: 04-1] offload: fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/190669 (owner: 10Matanya) [20:15:36] (03PS2) 10Matanya: interface: fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/190669 [20:16:25] (03CR) 10jenkins-bot: [V: 04-1] interface: fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/190669 (owner: 10Matanya) [20:17:42] (03PS3) 10Matanya: interface: fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/190669 [20:19:09] (03PS1) 10Matanya: eventlogging: fix file mode [puppet] - 10https://gerrit.wikimedia.org/r/190671 [20:41:01] (03CR) 10Andrew Bogott: [C: 032] "Many thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/190668 (owner: 10Matanya) [22:01:43] 3operations, Scrum-of-Scrums, Services, RESTBase: RESTbase deployment - https://phabricator.wikimedia.org/T1228#1039469 (10GWicke) [22:09:47] (03PS1) 10Hoo man: dataset: Add link to other/wikidata/ from other/ [puppet] - 10https://gerrit.wikimedia.org/r/190683 (https://phabricator.wikimedia.org/T87328) [22:11:59] (03PS2) 10Tim Landscheidt: Tools: Only update ssh configuration when necessary [puppet] - 10https://gerrit.wikimedia.org/r/145441 [22:35:41] 3operations, RESTBase: Detailed cassandra monitoring - https://phabricator.wikimedia.org/T78514#1039550 (10GWicke) [23:06:57] (03PS1) 10Gerardduenas: Set $wgBabelCategoryNames true at outreachwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190686 (https://phabricator.wikimedia.org/T89484) [23:08:24] (03PS1) 10Ori.livneh: RESTBase: add Icinga check_procs check for Cassandra [puppet] - 10https://gerrit.wikimedia.org/r/190688 [23:13:32] (03PS1) 10Matanya: lvs: init.pp lint [puppet] - 10https://gerrit.wikimedia.org/r/190689 [23:14:20] (03CR) 10jenkins-bot: [V: 04-1] lvs: init.pp lint [puppet] - 10https://gerrit.wikimedia.org/r/190689 (owner: 10Matanya) [23:15:28] (03PS2) 10Matanya: lvs: init.pp lint [puppet] - 10https://gerrit.wikimedia.org/r/190689 [23:17:43] (03PS2) 10Gerardduenas: Set $wgBabelCategoryNames true at outreachwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190686 (https://phabricator.wikimedia.org/T89484) [23:35:23] (03PS1) 10Legoktm: Add GlobalUserPageWikis hook handler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190691