[00:00:43] paravoid: i wouldn't have guessed that you listened to neko case, but i'm not sure what about it is surprising [00:01:14] (03CR) 10Nemo bis: "Thanks. How about "Thanks" too? And BetaFeatures, CentralNotice, Echo?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105622 (owner: 10Reedy) [00:01:22] heh [00:01:27] Nemo_bis: I was doing the obvious ones [00:01:46] Wikimedians are never satisfied. [00:01:52] Reedy: and I was just dropping some only-slightly-less-obvious in the comments :) [00:01:59] Gloria: I prepended a "Thanks" [00:02:07] You're saintly. [00:02:19] consider that the standard nowadays is just clicking a button and letting a robot/program do the job for you [00:02:39] $wgCategoryTreeDynamicTag = true; [00:02:39] require( $IP . '/extensions/CategoryTree/CategoryTree.php' ); [00:02:45] How did anyone think that was going to work? [00:02:47] Actually writing or pronouncing the word is probably an habit going to extintion [00:02:50] +c [00:05:47] wtf [00:09:17] !log reedy updated /a/common to {{Gerrit|I624133239}}: Disable more stuff on loginwiki and votewiki [00:09:35] Logged the message, Master [00:10:23] (03PS1) 10Reedy: Fixup CategoryTree [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105623 [00:11:45] greg-g: did you enjoy the video? [00:12:37] (03PS2) 10Reedy: Fixup CategoryTree config, disable where necessary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105623 [00:13:39] (03CR) 10Reedy: [C: 032] Fixup CategoryTree config, disable where necessary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105623 (owner: 10Reedy) [00:13:48] (03Merged) 10jenkins-bot: Fixup CategoryTree config, disable where necessary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105623 (owner: 10Reedy) [00:15:44] !log reedy synchronized wmf-config/ [00:16:02] Logged the message, Master [00:18:38] !log reedy updated /a/common to {{Gerrit|Ibdd8671a1}}: Fixup CategoryTree config, disable where necessary [00:18:44] (03PS1) 10Reedy: Disable BetaFeatures on votewiki and loginwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105624 [00:18:55] Logged the message, Master [00:19:46] (03CR) 10Reedy: [C: 032] Disable BetaFeatures on votewiki and loginwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105624 (owner: 10Reedy) [00:19:56] (03Merged) 10jenkins-bot: Disable BetaFeatures on votewiki and loginwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105624 (owner: 10Reedy) [00:20:08] Reedy: An if branch + require seems weird. [00:20:52] why [00:20:53] ? [00:20:54] !log reedy synchronized wmf-config/InitialiseSettings.php [00:21:10] Dunno. [00:21:12] Logged the message, Master [00:21:23] Because the others near it use include. [00:21:34] we're pretty inconsistent throughout the file [00:28:41] (03Abandoned) 10Andrew Bogott: Add support for miscellaneous proxy-specific settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/105513 (owner: 10Andrew Bogott) [00:35:24] !log reedy updated /a/common to {{Gerrit|Ie45a06c38}}: Disable BetaFeatures on votewiki and loginwiki [00:35:31] (03PS1) 10Reedy: Disable a handful more extensions form loginwiki and votewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105627 [00:35:43] Logged the message, Master [00:36:30] (03CR) 10Reedy: [C: 032] Disable a handful more extensions form loginwiki and votewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105627 (owner: 10Reedy) [00:36:52] (03Merged) 10jenkins-bot: Disable a handful more extensions form loginwiki and votewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105627 (owner: 10Reedy) [00:37:17] !log reedy synchronized wmf-config/InitialiseSettings.php [00:37:33] Logged the message, Master [00:44:52] !log reedy updated /a/common to {{Gerrit|I854ea451b}}: Disable a handful more extensions form loginwiki and votewiki [00:44:56] (03PS1) 10Reedy: Disable a few more extensions from VoteWiki and LoginWiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105628 [00:45:10] Logged the message, Master [00:45:37] (03CR) 10Reedy: [C: 032] Disable a few more extensions from VoteWiki and LoginWiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105628 (owner: 10Reedy) [00:45:45] (03Merged) 10jenkins-bot: Disable a few more extensions from VoteWiki and LoginWiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105628 (owner: 10Reedy) [00:47:33] !log reedy synchronized wmf-config/ [00:47:51] Logged the message, Master [00:48:40] !log reedy updated /a/common to {{Gerrit|I23469a49a}}: Disable a few more extensions from VoteWiki and LoginWiki [00:48:58] Logged the message, Master [00:49:20] !log reedy synchronized wmf-config/CommonSettings.php 'Fix notice' [00:49:33] (03PS1) 10Reedy: Fix NOTICE/typo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105630 [00:49:35] Logged the message, Master [00:52:51] !log reedy synchronized wmf-config/ '$wgUseAjax = true;' [00:52:57] 'wgUseAjax' => array( [00:52:57] 'default' => false, [00:52:58] ), [00:53:08] Logged the message, Master [00:53:08] Reedy: You found that in InitialiseSettings.php? [00:53:11] Krenair: ^ [00:53:19] Gah [00:53:22] lol [00:53:28] I don't even. [00:54:09] (03PS1) 10Reedy: [x] Yes to AJAX [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105632 [00:54:46] Reedy: Give me a second to review that. [00:55:07] It's fine, Gloria... [00:55:18] (03CR) 10MZMcBride: "Looks good to me." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105632 (owner: 10Reedy) [00:55:34] Krenair: I was double-checking that InitialiseSettings.php got updated. [00:55:55] https://noc.wikimedia.org/conf/CommonSettings.php.txt seems outdated. [00:55:57] (03CR) 10Reedy: [C: 032] [x] Yes to AJAX [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105632 (owner: 10Reedy) [00:55:58] But might be local cache. [00:56:19] I see "$wgUseAjax = true;" in there. [00:57:06] https://git.wikimedia.org/raw/operations%2Fmediawiki-config/7047848725cc099d76094c16ebf1e4b8ce5299d5/wmf-config%2FCommonSettings.php [00:57:11] Doesn't have the same line. [00:57:13] (03CR) 10Reedy: [C: 032] Fix NOTICE/typo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105630 (owner: 10Reedy) [00:57:23] (03Merged) 10jenkins-bot: Fix NOTICE/typo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105630 (owner: 10Reedy) [00:57:25] (03Merged) 10jenkins-bot: [x] Yes to AJAX [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105632 (owner: 10Reedy) [00:59:14] !log reedy synchronized wmf-config/ '[x] Yes to AJAX' [00:59:30] Logged the message, Master [00:59:42] reedy@tin:/a/common$ mwscript eval.php enwiki [00:59:43] > echo $wgUseAjax; [00:59:43] 1 [00:59:43] WFM [01:00:28] Remind me what /a is? [01:00:46] 1st extra storage partition [01:01:07] Or not even [01:01:14] It's a folder [01:36:56] Reedy: Yeah, I think the conf/ files at noc can be slightly oudated. [01:37:02] Outdated, even. [01:37:05] Possibly [01:37:11] https://noc.wikimedia.org/conf/CommonSettings.php.txt looks fine now. [01:37:13] I think there's still on fenari... [01:37:25] So syncing should hit there too [01:37:40] Probably a caching layer or two. [01:40:58] curl says no [01:40:59] Server: Apache/2.2.22 (Ubuntu) [01:41:13] Probably should be behind the misc proxy... [01:58:06] (03CR) 10MZMcBride: "Related: bug 59704" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105476 (owner: 10Reedy) [02:00:36] (03CR) 10MZMcBride: "Related: bug 59702" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105622 (owner: 10Reedy) [02:00:45] (03CR) 10MZMcBride: "Related: bug 59702" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105623 (owner: 10Reedy) [02:00:50] (03CR) 10MZMcBride: "Related: bug 59702" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105624 (owner: 10Reedy) [02:01:08] (03CR) 10MZMcBride: "Related: bug 59702" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105627 (owner: 10Reedy) [02:01:22] (03CR) 10MZMcBride: "Related: bug 59702" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105628 (owner: 10Reedy) [02:01:36] (03CR) 10MZMcBride: "Related: bug 59702" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105630 (owner: 10Reedy) [02:01:45] (03CR) 10MZMcBride: "Related: bug 59702" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105632 (owner: 10Reedy) [02:01:51] (Finished.) [02:18:30] !log LocalisationUpdate completed (1.23wmf8) at Mon Jan 6 02:18:30 UTC 2014 [02:18:54] Logged the message, Master [02:35:14] !log LocalisationUpdate completed (1.23wmf9) at Mon Jan 6 02:35:14 UTC 2014 [02:35:31] Logged the message, Master [02:44:44] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [02:45:44] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [03:00:27] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 6 03:00:27 UTC 2014 [03:00:44] Logged the message, Master [05:18:49] Aaron|home: huh, apparently salt uses msgpack for serialization [05:19:03] maybe that php code will have other uses [05:25:13] * Aaron|home detects evil grinning [05:33:34] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:35:34] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [05:36:23] !log aaron started scap: timing test (beta) [05:36:40] Logged the message, Master [05:45:40] ugh searchidx1001 again [06:10:09] Aaron|home: that system needs ops attention [06:10:25] it's not something we should hack around in scap or by renicing [06:10:47] dmesg is filled w/: [06:10:50] [11604026.902923] INFO: task xfsbufd/sda6:731 blocked for more than 120 seconds. [06:10:50] [11604026.910081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [06:10:59] except it's any number of processes [06:12:01] * Aaron|home killed the resync proc after a while [06:12:24] my main terminal seemed to have disconnected by then [06:12:55] but yeah that box is fcked [06:14:23] just running 'ls' hangs [06:16:28] could be an XFS deadlock or a bad disk [06:17:52] apergo.s should be up in a bit & could possibly investigate. probably best to file an RT ticket tho [06:25:34] RECOVERY - udp2log log age for oxygen on oxygen is OK: OK: all log files active [06:26:35] first relevant dmesg entry is "[9997623.601184] INFO: task kswapd0:115 blocked for more than 120 seconds." tc [06:29:54] PROBLEM - MySQL Slave Running on db68 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error You have an error in your SQL syntax: check the manual that co [06:30:32] I would just reboot the box and see how it is after that [06:30:38] ori: Aaron|home ^^ [06:31:54] RECOVERY - MySQL Slave Running on db68 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [06:38:58] !log rebooting searchidx1001, hanging on various things, first sign of trouble was 'task kswapd0:115 blocked for more than 120 seconds.' [06:39:16] Logged the message, Master [06:40:04] PROBLEM - Disk space on searchidx1001 is CRITICAL: Connection refused by host [06:40:15] PROBLEM - DPKG on searchidx1001 is CRITICAL: Connection refused by host [06:40:15] PROBLEM - SSH on searchidx1001 is CRITICAL: Connection refused [06:40:24] PROBLEM - puppet disabled on searchidx1001 is CRITICAL: Connection refused by host [06:40:24] PROBLEM - RAID on searchidx1001 is CRITICAL: Connection refused by host [06:42:54] PROBLEM - Host searchidx1001 is DOWN: PING CRITICAL - Packet loss = 100% [06:44:14] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [06:44:24] RECOVERY - puppet disabled on searchidx1001 is OK: OK [06:44:24] RECOVERY - Host searchidx1001 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [06:44:24] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [06:45:04] RECOVERY - Disk space on searchidx1001 is OK: DISK OK [06:45:14] RECOVERY - DPKG on searchidx1001 is OK: All packages OK [06:58:44] PROBLEM - NTP on searchidx1001 is CRITICAL: NTP CRITICAL: Offset unknown [07:04:44] RECOVERY - NTP on searchidx1001 is OK: NTP OK: Offset -0.0007642507553 secs [07:27:34] PROBLEM - udp2log log age for oxygen on oxygen is CRITICAL: CRITICAL: log files /var/log/udp2log/packet-loss.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [07:56:52] apergos: looks better. thanks! [07:57:04] sure [09:04:56] (03CR) 10DrTrigon: [C: 031] "Merge please! :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105192 (owner: 10Tim Landscheidt) [10:06:46] (03CR) 10Dzahn: [C: 032] Add Frank Schulenburg's blog to the English Planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/105460 (owner: 10Odder) [10:13:25] WEE [10:15:00] !log jenkins: refreshing jslint files for mediawiki extensions, making sure they are in sync [10:15:17] Logged the message, Master [10:16:02] (03CR) 10Dzahn: [C: 032] install graphviz on bugzilla role [operations/puppet] - 10https://gerrit.wikimedia.org/r/103525 (owner: 10Dzahn) [10:18:50] (03CR) 10Dzahn: "Checking for GraphViz (any) ok" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103525 (owner: 10Dzahn) [10:35:12] (03PS1) 10TTO: Add Wikimania wikis to global abuse filters [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105655 [10:36:50] (03CR) 10TTO: "Yay for cleanup!! Thanks Reedy." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105632 (owner: 10Reedy) [10:40:42] (03CR) 10Dzahn: [C: 031] Add Wikimania wikis to global abuse filters [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105655 (owner: 10TTO) [10:41:18] (03CR) 10Odder: "https://gerrit.wikimedia.org/r/#/c/104726/ closes the wikimania2013wiki, so it probably isn't worth the hassle to have to remove the setti" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105655 (owner: 10TTO) [10:50:53] !log jenkins: made mediawiki-core-phpunit-parser job executable concurrently, might cause race conditions. {{gerrit|102152}} [10:51:09] Logged the message, Master [10:59:21] (03CR) 10TTO: "Good point, didn't know it was happening so soon." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105655 (owner: 10TTO) [10:59:30] (03PS2) 10TTO: Add Wikimania 2014 wiki to global abuse filters [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105655 [11:16:49] (03CR) 10Odder: [C: 031] Add Wikimania 2014 wiki to global abuse filters [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105655 (owner: 10TTO) [11:17:18] !log jenkins: uninstalling phpcs on gallium (was installed with pear, no deployed using our deploy system) {{bug|57064}} [11:17:35] Logged the message, Master [11:50:30] PROBLEM - Disk space on wtp1007 is CRITICAL: DISK CRITICAL - free space: / 200 MB (2% inode=72%): [11:54:34] PROBLEM - Disk space on wtp1024 is CRITICAL: DISK CRITICAL - free space: / 184 MB (2% inode=72%): [11:54:44] PROBLEM - Parsoid on wtp1007 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:45] PROBLEM - Disk space on wtp1010 is CRITICAL: DISK CRITICAL - free space: / 144 MB (1% inode=72%): [11:56:44] PROBLEM - Disk space on wtp1014 is CRITICAL: DISK CRITICAL - free space: / 257 MB (2% inode=72%): [11:58:44] PROBLEM - Parsoid on wtp1010 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:45] RECOVERY - Disk space on wtp1014 is OK: DISK OK [11:59:04] PROBLEM - Parsoid on wtp1024 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:02] (03PS1) 10Dzahn: hackish fix for wikisource portal special case [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/105668 [12:01:04] PROBLEM - Parsoid on wtp1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:34] PROBLEM - Disk space on wtp1019 is CRITICAL: DISK CRITICAL - free space: / 320 MB (3% inode=72%): [12:06:30] re: socket timeout on wtp1014 - processes are running, can login, disk space available .. [12:07:34] does this cause an actual problem on the site or not [12:09:42] suppose not because no paging or other reports [12:09:43] (03CR) 10Addshore: Start wikidata puppet module for builder (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 (owner: 10Addshore) [12:10:14] PROBLEM - Parsoid on wtp1019 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:34] RECOVERY - Disk space on wtp1024 is OK: DISK OK [12:11:47] (03PS10) 10Addshore: Start wikidata puppet module for builder [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 [12:13:24] RECOVERY - Disk space on wtp1007 is OK: DISK OK [12:13:35] (03PS11) 10Addshore: Start wikidata puppet module for builder [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 [12:15:30] (03PS2) 10Parent5446: Undeploy AssertEdit (merged into core) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96931 (owner: 10Legoktm) [12:24:44] RECOVERY - Disk space on wtp1010 is OK: DISK OK [12:32:34] RECOVERY - Disk space on wtp1019 is OK: DISK OK [12:34:44] PROBLEM - Disk space on wtp1014 is CRITICAL: DISK CRITICAL - free space: / 263 MB (2% inode=72%): [12:49:34] PROBLEM - Disk space on wtp1024 is CRITICAL: DISK CRITICAL - free space: / 186 MB (2% inode=72%): [12:50:24] PROBLEM - Disk space on wtp1007 is CRITICAL: DISK CRITICAL - free space: / 211 MB (2% inode=72%): [13:01:44] PROBLEM - Disk space on wtp1010 is CRITICAL: DISK CRITICAL - free space: / 148 MB (1% inode=72%): [13:06:37] (03PS1) 10Dzahn: let metapedias table have language columns [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/105675 [13:07:38] (03CR) 10Dzahn: [C: 032] hackish fix for wikisource portal special case [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/105668 (owner: 10Dzahn) [13:08:23] (03CR) 10Dzahn: [C: 032] let metapedias table have language columns [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/105675 (owner: 10Dzahn) [13:10:34] PROBLEM - Disk space on wtp1019 is CRITICAL: DISK CRITICAL - free space: / 264 MB (2% inode=72%): [13:11:23] (03CR) 10Dzahn: "reported by Robert Hanke, thx, fixed" [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/105675 (owner: 10Dzahn) [13:45:06] (03PS1) 10Odder: Enable UploadWizard on Romanian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105679 [13:48:42] mutante: interested in importing some 2k independent mediawikis into wikistats? [13:49:10] they're at https://wikiteam.googlecode.com/svn/trunk/listsofwikis/mediawikis_2013_checked.txt , already added to https://wikiapiary.com/wiki/Category:January_2014_Import [13:50:01] Nemo_bis: yes, i am, thanks. can make bug ?:) [13:50:10] it has a component [13:51:16] sure [13:56:13] mutante: btw it's always such a pain to navigate the *three* _pruducts_ for labs/tools [13:56:40] !log jenkins: migrated mediawiki extension loader from /tools/extensions-loader.php to mediawiki/conf.d/50_mw_ext_loader.php {{gerrit|105680}} [13:56:46] Nemo_bis: :P i know heh:) thank you [13:56:56] Logged the message, Master [13:57:00] Nemo_bis: Bugzilla main navigation has a new link since today :) [13:57:23] mutante: what link ? [13:57:50] My Requests ? [13:59:17] hashar: https://gerrit.wikimedia.org/r/#/c/104342/ [13:59:23] Browse projects [13:59:28] ahhh [14:10:09] I know how to find projects, it's just frustrating to always be presented with all that confusion of similar products [14:12:37] not supposed to change any products or components without tickets to andre because of confusion in the past and multi-admins :p [14:13:53] (03PS1) 10Aude: Enable Wikibase Client on testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105682 [14:15:14] sure [14:25:33] (03CR) 10Reedy: [C: 032] Enable Wikibase Client on testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105682 (owner: 10Aude) [14:25:47] (03Merged) 10jenkins-bot: Enable Wikibase Client on testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105682 (owner: 10Aude) [14:26:12] Bugzilla products too confusing? Add your opinion and grand plan to https://bugzilla.wikimedia.org/show_bug.cgi?id=38990 ! :P [14:26:37] !log reedy synchronized wmf-config/InitialiseSettings.php [14:26:55] Logged the message, Master [14:27:05] aude: Any other prep we need? [14:29:09] don't think so [14:29:21] I can't find your config change commit again... [14:29:32] * aude looks [14:29:39] https://github.com/wikimedia/operations-mediawiki-config/commits/master?page=2 [14:30:12] https://github.com/wikimedia/operations-mediawiki-config/commit/e6fc8d9db38947776523bfdcc330bdecbaadd034 [14:30:15] Need to be looking for the right thing ;) [14:30:20] I7cf7af5525f3223dbd044e7676b5c0255b45928c [14:30:25] shhh, don't let them catch you using github links [14:30:47] too late [14:30:51] Reedy, do you have a script or something that reports fatal errors in the production logs? [14:31:01] Sort of [14:31:21] There's one that do some grouping and combining of the last 1000 lines of the apache syslogs [14:31:27] Useful as an overview of what's going on [14:31:38] Then can grep through the full exception/fatal logs as necessary [14:31:42] Reedy: so the goal is to have beta run the Wikidata git repo (everything is inside there) [14:31:45] on beta [14:31:50] and keep production as-is [14:31:54] Reedy: is it checked in somewhere? [14:31:55] YuviPanda: can i ask re: commons app ?: [14:32:05] mutante: sure! switch to -mobile? [14:32:12] YuviPanda: kthx [14:32:24] for localisation cache, i tried to tell beta to use one set of stuff for wikibase and production the other [14:32:31] Reedy: I'm looking for something that will monitor fatal.log on beta, and wondering if I should just write something myself. [14:32:50] Uhh [14:33:08] chrismcmahon: we don't have such thing? [14:33:20] watch "tail -n 1000 /home/wikipedia/syslog/apache.log | grep -v 'Search backend error' | grep -v -i 'swift' | grep 'PHP\|Segmentation fault' | grep -v 'filemtime\|failed to mkdir\|GC cache entry\|cache slam averted\|SHA-1 metadata' | sed -r 's/\[notice\] child pid [0-9]+ exit signal //g' | sed 's/, referer.*$//g' | cut -d ' ' -f 7- | sort | uniq -c | sort -rn" [14:33:23] aude: not on beta, not right now. It's time. [14:33:27] is the summary script [14:33:28] oh [14:34:52] Reedy: hmm, I was thinking of something simpler. [14:35:09] (03PS1) 10Reedy: Enable Wikidata build on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105684 [14:35:22] chrismcmahon: The fatal logs are even more complex than the syslogs... [14:35:40] (03PS2) 10Reedy: Enable Wikidata build on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105684 [14:36:11] Reedy: on beta, I think we just want to know if fatal.log just changes at all. And check at intervals. [14:36:39] We have graphs for those on ganglia [14:38:31] Reedy: we have http://ganglia.wmflabs.org/latest/?c=deployment-prep [14:38:59] Nemo_bis: Where are those graphs? :P [14:40:08] so Reedy, that script above just runs watch() in a screen then? does it do notifications? [14:40:57] !log reedy updated /a/common to {{Gerrit|I349589a49}}: Enable Wikibase Client on testwiki [14:40:58] chrismcmahon: Nope [14:40:58] Logged the message, Master [14:40:58] Not even run in a screen [14:41:00] logmsgbot: LIES [14:41:19] Reedy: so it's basically sync-common mw1017 [14:41:32] That's exactly what I've just done ;) [14:41:35] k [14:41:37] * aude reading https://wikitech.wikimedia.org/wiki/Test.wikipedia.org [14:41:44] No l10n stuff done yet [14:41:49] Which is where things should break again [14:42:06] ottomata: I just realized that oxygen's udp2log is not producing data. [14:42:24] ottomata: Its produced tsv since the last rotations are empty :-( [14:42:51] ottomata: e.g.: /a/log/webrequest/mobile-sampled-100.tsv.log on oxygen. [14:43:39] so testwiki has no sites table [14:43:48] shouldn't break anything though [14:43:55] might aswell fix it [14:43:57] easily enough done [14:44:02] ok [14:44:17] Reedy: while I have you on the line then, if had a script checking beta for fatals, and I wanted it to send email, what MTA would it send through? [14:44:32] OOoooo [14:44:33] wa [14:44:33] ottomata: Seems it's broken since around 2014-01-05 03:39:25 UTC [14:44:36] Not sure if you can from labs [14:44:50] Reedy: yes, this would be a deployment-foo host. [14:45:00] ottomata: (At least those are the last time stamps in mobile and zero tsvs. Typically rotation is around 06:30) [14:45:03] qchris, yeah i see some alerts in my email [14:45:03] yeah [14:45:06] checking into it now [14:45:10] ottomata: Thanks. [14:46:40] Query OK, 798 row(s) affected [14:46:44] Query OK, 1588 row(s) affected [14:47:00] sounds right [14:47:12] what's the command to flush the caches for those? [14:47:26] it should be automatic [14:47:38] Reedy: these graphs linked in the /topic you mean? http://ur1.ca/edq1f [14:47:48] chrismcmahon: ^ [14:47:50] aude: ok... [14:47:53] Reedy: looks good [14:47:58] https://test.wikipedia.org/wiki/New_York_City [14:48:24] PROBLEM - Disk space on wtp1004 is CRITICAL: DISK CRITICAL - free space: / 1333 MB (3% inode=92%): [14:48:34] RECOVERY - udp2log log age for oxygen on oxygen is OK: OK: all log files active [14:48:43] (03CR) 10Faidon Liambotis: [C: 032] Fix dependency libtiff5-alt-dev -> libtiff4-dev [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 (owner: 10coren) [14:49:08] I think we've gotta update the l10ncache on tin now [14:49:10] Got a few hours [14:49:18] I'll make a backup of it before changing it though ;) [14:49:23] ok [14:50:45] still no updates @ https://wikitech.wikimedia.org/wiki/RT_Triage_Duty#Duty_desk_rotation_-_who_is_next.3F [14:51:11] jeremyb: still waiting on the apache stuff? [14:51:25] aude: yah [14:51:27] * aude can't do anything about it [14:53:14] i'm still getting used to the new redirects.dat workflow [14:53:18] Reedy: speaking of which... wikisource needs sites table [14:53:22] before deployment next week [14:53:39] (03PS1) 10Yuvipanda: dynamicproxy: Enable XFF based on a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/105687 [14:53:43] oh gadolinum is down!? [14:53:45] eef [14:54:01] Oooh :-) [14:54:15] apergos: Gadolinium died Jan 5, 3AM and is now no longer on ganglia. will it resurrect from the deaths? [14:54:59] dunno, can't seem to log into mgmt either [14:55:06] hmmmmMMM [14:55:16] damn you file permissions [14:55:18] cmjohnson1: got helping ability? [14:57:16] hedonil: 20:24 apergos: gadolinium down/unreachable, can't get to mgmt console, no ping even [14:57:29] jeremyb: 6572 [14:57:44] mutante: haha [14:58:02] mutante: danke :-) [14:58:33] mutante: few minutes ago gadolinium was at least /dead/ on ganglia, but now disappeared completely [14:58:51] yeah guess it expired there, its dead in icinga [14:58:59] doo dee doo, well not much I can do atm :/ [14:59:07] ottomata: what's up? [14:59:09] doesnt sound good [14:59:14] gadolinum is dead! [14:59:15] when apergos said not even mgmt [14:59:18] its the udp2log multicast relay [14:59:22] then i dont think it will come back [14:59:26] any time soon [14:59:27] well, i said that i think [14:59:59] hedonil: i think that part was just the ganglia update [15:00:04] PROBLEM - Parsoid on wtp1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:10] and it was dead since that SAL entry [15:00:12] i can't seem to get anything out of mgmt [15:00:19] could be doing something wrong, but ja [15:00:23] yeah...i can't either [15:00:56] confirmed, no reply [15:02:06] yea, at this points it needs smart hands [15:02:27] and connecting monitor cable [15:03:02] ottomata mutante: I will be there within the hour...need to wrap up what I am doing [15:03:05] who wants to make ticket? seems we dont have one yet [15:03:32] cmjohnson1: cool, thanks, i think otto knows better than I do how urgent it really is [15:04:49] i can do it [15:05:23] ottomata: 6573 [15:05:27] but please add some details? [15:05:32] if it makes sense [15:05:35] ok thanks [15:06:07] ottomata: i am heading there now [15:08:54] assigned to chris , we'll see later [15:11:24] RECOVERY - Disk space on wtp1004 is OK: DISK OK [15:15:28] (03CR) 10Manybubbles: [C: 031] "Will deploy with the config change today." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105479 (owner: 10Chad) [15:17:58] moving to the library, back in a few [15:21:35] (03PS3) 10Aude: Enable Wikidata build on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105684 (owner: 10Reedy) [15:41:06] (03PS1) 10Jgreen: increase max message size for spam scanning on OTRS server [operations/puppet] - 10https://gerrit.wikimedia.org/r/105692 [15:44:29] (03CR) 10Jgreen: [C: 032 V: 031] increase max message size for spam scanning on OTRS server [operations/puppet] - 10https://gerrit.wikimedia.org/r/105692 (owner: 10Jgreen) [15:44:56] (03CR) 10Reedy: [C: 032] "Tested and confirmed working on testwiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105684 (owner: 10Reedy) [15:45:05] (03Merged) 10jenkins-bot: Enable Wikidata build on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105684 (owner: 10Reedy) [15:46:49] !log rebooting gadolinium [15:47:07] Logged the message, Master [15:47:12] (03PS1) 10Hashar: multiversion: replace die() with print; exit(1); [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105693 [15:50:08] !log reedy started scap: Rebuild lolcalisation cache after https://gerrit.wikimedia.org/r/#/c/105684 [15:50:26] Logged the message, Master [15:53:22] ottomata: definitely a h/w issue...getting an idrac not responding error. this requires a lengthy call to dell...will keep you posted [15:53:32] oof ok [15:53:33] hm [15:53:38] i wonder if we should move the relay [15:53:41] hmmmmmMMMMM [15:55:05] cmjohnson1: so this will probably take more than a few hours to get fixed? [15:55:12] probably [15:55:26] could be just a firmware update [15:55:31] Eeeeeee [15:55:34] hmph [15:56:18] hmmmm [15:56:25] i think i could move the relay to an26 temporarily [15:57:29] 2014-01-06 15:56:57 mw1080 enwiki: [eb8c5447] /?search=Hhvvvvv Exception from line 468 of /usr/local/apache/common-local/php-1.23wmf8/includes/cache/LocalisationCache.php: No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php. [15:57:32] eh? [15:57:38] English? [15:57:50] it isn't common, but it is weird [15:58:20] OOF, this also means eventlogging is down i think [15:58:35] huh [15:58:37]