[00:00:04] RoanKattouw, ^d, marktraceur: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150106T0000). Please do the needful. [00:01:24] !log ori Synchronized wmf-config/StartProfiler.php: I341a50bef: 'Re-enable xhprof for single-request profiling' (duration: 00m 06s) [00:01:28] Logged the message, Master [00:01:34] (03PS1) 10Jalexander: Allow CollabWiki crat to remove crat/sysop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182961 [00:04:53] so whose doing SWAT :D [00:06:33] who's [00:06:56] Also, s/roll/role/ earlier. ;-) [00:07:30] I make no apologies or excuses for my lack of grammar :) [00:08:03] *nor [00:08:04] * Fiona hides. [00:08:32] jamesofur: Let's actually get together sometime between January 25 and 28. [00:08:42] I'm available anytime. [00:08:50] are superm401 and kaldari here for their patches? [00:08:57] Definitely! [00:08:59] here [00:09:09] aude, yep. :) Please ping me if I can clarify anything. [00:09:34] if no one is doing swat, suppose i could :) [00:09:40] * aude waits a couple more minutes [00:10:16] well if you do aude I added one a couple minutes ago ;) (small config patch, it's the same one that grrrit-wm just announced I submitted) [00:10:23] jamesofur: ok [00:11:05] alright. [00:11:21] * aude starts with moodbar [00:11:38] MoodBar is back from the dead? o_O [00:11:50] apparently so [00:12:49] thanks aude [00:12:51] sure [00:14:36] * aude wonders why jenkins is not doing gate & submit [00:14:54] tired [00:15:58] let's do the config patch [00:16:25] (03CR) 10Aude: [C: 032] Allow CollabWiki crat to remove crat/sysop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182961 (owner: 10Jalexander) [00:17:19] aude: since you're here, any ideas on why we get a surge of requests for dewiki Deutschland (coming from google, mostly image search clicking on the flag) [00:17:25] also for Carl Friedrich Gauss [00:17:32] paravoid: no idea [00:17:50] maybe a google doodle? [00:18:15] I don't see one, do you? [00:18:29] * aude not in germany right now :) [00:18:34] oh :) [00:18:34] do they do country specific ones? [00:18:38] ryeah [00:18:39] Yes, regularly. [00:18:39] -r [00:19:09] ok, zuul ... not funny [00:19:21] http://www.google.com/doodles doesn't have it [00:19:23] does anyone know why zuul is not running? [00:19:25] it was also too short [00:19:34] 30 April 1777 – 23 February 1855 [00:19:51] no apparent reason for a gauss doodle [00:20:09] a doodle-originating surge would probably last for more than that [00:21:27] aude, I think it's failing because of a Tidy error and needs to be forced. [00:21:37] i don't think [00:21:43] it's also not submitting the config patch [00:21:47] Unrelated to my change, and apparently only on one branch, which signals an improvement. :) [00:21:50] aude, https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/755/consoleFull [00:21:51] yeah, config should work [00:21:56] https://twitter.com/search?q=Carl%20Friedrich%20Gauss has "ThemenTrends ‏@thementrends 3h3 hours ago [00:21:56] See https://gerrit.wikimedia.org/r/#/c/182878/ [00:21:57] ping Krinkle ^ [00:21:59] "Carl Friedrich Gauß" ist momentan der meistgelesene Artikel auf Wikipedia http://de.wikipedia.org/wiki/Carl_Friedrich_Gau%C3%9F …" [00:22:05] Config could be a separate issue. [00:22:07] superm401: aware of that [00:22:16] Oh, okay. [00:22:22] calling it a day... cu [00:23:08] * aude would prefer not to force submit everything [00:23:27] * greg-g tries https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Known_issues [00:23:40] probably just needs restart [00:25:06] !log restarting jenkins, hope that kicks it enough [00:25:10] Logged the message, Master [00:25:32] thanks [00:25:47] * aude bookmarks the page [00:25:59] also https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Issue.3F [00:26:34] annnnd, now jenkins won't come back [00:26:48] suppose it takes a minute :/ [00:26:53] it's back [00:27:04] (03CR) 10Aude: Allow CollabWiki crat to remove crat/sysop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182961 (owner: 10Jalexander) [00:27:12] whew [00:27:46] aude: next kick is to do the gearman one from the first link [00:28:16] ok [00:29:27] it's not totally back yet, but coming back it seems [00:29:30] yeah [00:29:33] it's thinking [00:30:14] aude: I have to run, unfortunately, I'll try to get online on the bus, otherwise, ping people if things don't get better soon [00:30:33] ok [00:32:19] (03CR) 10Aude: [C: 032] Allow CollabWiki crat to remove crat/sysop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182961 (owner: 10Jalexander) [00:34:41] think i have to try the gearman thing [00:35:14] no permission for that [00:35:16] :/ [00:35:28] Krinkle: around? [00:36:24] paravoid: do you have permission for https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Gearman_deadlock ? [00:37:25] * aude can overrule this time but not nice to leave it broken [00:40:48] anyone else can kick zuul? [00:41:06] jamesofur: still around [00:41:12] yup [00:41:17] ok, shall proceed [00:41:40] (03CR) 10Aude: [V: 032] Allow CollabWiki crat to remove crat/sysop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182961 (owner: 10Jalexander) [00:44:02] !log aude Synchronized wmf-config/InitialiseSettings.php: Update collabwiki bureaucrat permissions (duration: 00m 06s) [00:44:06] Logged the message, Master [00:44:07] jamesofur: ^ [00:44:14] superm401: still here? [00:44:28] aude: verified, tahnks [00:44:31] thanks too [00:44:31] great [00:44:38] aude, yep. [00:44:41] ok [00:44:54] will proceed, have to overrule jenkins on wmf12 anyway [00:45:55] Yeah [00:46:39] hmm, seems i'm pulling in ori's patch for tidy also [00:46:43] not a problem though [00:46:53] i forgot to sync that [00:46:53] thanks [00:46:56] k [00:47:04] ori: can you kick zuul? [00:47:53] i'll try [00:49:01] thanks [00:49:05] done [00:49:09] logged it in -qa [00:49:15] great [00:49:48] !log aude Synchronized php-1.25wmf13/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 06s) [00:49:52] Logged the message, Master [00:49:57] superm401: please check [00:50:00] then i can do wmf12 [00:50:44] aude, checking now. [00:50:47] k [00:51:32] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 307 seconds [00:51:38] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 310 seconds [00:52:37] aude, looks good. [00:52:42] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:52:45] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -1 seconds [00:52:58] ok [00:55:09] hmm, zuul still doesn't seem to be doing anything :( [00:57:05] !log aude Synchronized php-1.25wmf12/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 09s) [00:57:06] there ^ [00:57:09] Logged the message, Master [00:57:11] kaldari: around? [00:57:19] aude: yep [00:57:38] you are next [00:57:42] yay! [00:57:56] aude: Thanks for SWATting [01:01:59] Thanks, aude. [01:06:40] updating submodule for mobile now [01:09:00] wonders if jenkins is doing anything now [01:14:01] sent a mail to poke about jenkins / zuul [01:16:52] !log aude Synchronized php-1.25wmf13/extensions/MobileFrontend: Fix MobileFrontend bugs (duration: 00m 06s) [01:16:55] kaldari: ^ [01:17:01] Logged the message, Master [01:17:14] next is wikidata :) [01:17:30] aude: thanks. checking... [01:18:51] aude: did you get the Jenkins/Zuul thing working? I just saw the email message... [01:19:30] chrismcmahon: no [01:19:48] i don't know what else to try, nor do i have permissions to do so [01:20:07] aude: it is almost always the gearman problem :-/ [01:20:39] ori tried that [01:20:46] worth to try again [01:20:47] aude: ah, it isn't fixed after that gearman kick? :/ [01:20:53] apparently not [01:21:04] and jenkins is not running at all (e.g. new patches) [01:21:04] chrismcmahon: I'm on a bus, so cell connection, defer to you [01:21:32] i can deal with it for swat, but not a nice situation for everyone [01:21:51] I may not be able to do anything. [01:22:18] :/ [01:23:44] !log aude Synchronized php-1.25wmf13/extensions/WikimediaMessages: Add Wikidata other projects message (duration: 00m 06s) [01:23:48] Logged the message, Master [01:24:10] this will require scap actually [01:25:02] I think I just unstuck Jenkins, but it was scary. I told it to prepare to shut down, then I cancelled shutdown. After that the beta-scap-eqiad job started up running again. [01:26:03] stupid gearman/jenkins lockup bug [01:26:09] !log aude Started scap: Add Wikidata other projects message [01:26:14] Logged the message, Master [01:26:34] chrismcmahon: hope it worked this time [01:29:46] and I just disabled/enabled gearman for luck [01:29:54] ok [01:30:19] at one point I thought I could restart Jenkins completely from the configure menu, but I can't find that control now. [01:30:43] It's incredibly difficult for me to see gearman and not replace it with german and start getting confused [01:31:25] heh [01:34:59] * aude will +2 a Wikibase patch in a minute [01:37:18] I made a harmless commit to a patch in gerrit and I do not see Zuul picking up the change. :( [01:38:56] :( [01:46:45] I don't seem to have the privileges to restart Jenkins completely or I would. I can't tell what's wrong either. [01:47:25] hopefully krinkle or hashar will see the mail soon enough [01:47:41] idk who else would know or have privileges [01:49:17] (03PS1) 10Rush: phab reset sprint app to 0.6.1.4 [puppet] - 10https://gerrit.wikimedia.org/r/182985 [01:49:19] (03PS1) 10Rush: phab disable the security event listener temp [puppet] - 10https://gerrit.wikimedia.org/r/182986 [01:50:03] (03CR) 10Rush: [C: 032 V: 032] "somewhere in the hoopla there was a regression here" [puppet] - 10https://gerrit.wikimedia.org/r/182985 (owner: 10Rush) [02:03:28] !log aude Finished scap: Add Wikidata other projects message (duration: 37m 18s) [02:03:37] Logged the message, Master [02:04:18] (03CR) 10Rush: [C: 032 V: 032] phab disable the security event listener temp [puppet] - 10https://gerrit.wikimedia.org/r/182986 (owner: 10Rush) [02:04:21] looks good :) [02:24:12] (03PS1) 10Rush: phab remove security load-libraries entry [puppet] - 10https://gerrit.wikimedia.org/r/182990 [02:24:46] (03CR) 10Rush: [C: 032 V: 032] phab remove security load-libraries entry [puppet] - 10https://gerrit.wikimedia.org/r/182990 (owner: 10Rush) [02:46:34] (03PS2) 10Ori.livneh: Update my (=ori) dotfiles [puppet] - 10https://gerrit.wikimedia.org/r/182898 [02:46:41] (03CR) 10Ori.livneh: [C: 032 V: 032] Update my (=ori) dotfiles [puppet] - 10https://gerrit.wikimedia.org/r/182898 (owner: 10Ori.livneh) [02:50:32] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 1 failures [02:54:24] (03PS1) 10Ori.livneh: "Un-disable" xhprof [puppet] - 10https://gerrit.wikimedia.org/r/182992 [03:08:32] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:27:33] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [03:45:32] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:45:32] PROBLEM - MySQL Processlist on db1056 is CRITICAL: CRIT 116 unauthenticated, 0 locked, 0 copy to table, 0 statistics [04:48:52] RECOVERY - MySQL Processlist on db1056 is OK: OK 1 unauthenticated, 0 locked, 0 copy to table, 0 statistics [04:59:14] (03CR) 10Ori.livneh: "I'm not sure how it's less confusing to interleave erb and rsyslog if-statements, which are interpreted at different times." [puppet] - 10https://gerrit.wikimedia.org/r/177432 (owner: 10BryanDavis) [06:16:19] (03CR) 10Yuvipanda: [C: 032 V: 032] Add base table name to indexed views [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182799 (owner: 10Yuvipanda) [06:16:30] (03CR) 10Yuvipanda: [C: 032 V: 032] Make table columns dicts instead of lists [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182801 (owner: 10Yuvipanda) [06:16:42] (03CR) 10Yuvipanda: [C: 032 V: 032] Fix whitelist value being ignored when reading greylisted.yaml [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182847 (owner: 10Yuvipanda) [06:21:12] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:41] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:01] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:11] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:22] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:21] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:32] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:41] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:42] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:22] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:22] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:22] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 1 failures [06:37:51] (03PS6) 10KartikMistry: Content Translation configuration for Production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 [06:39:12] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:45:42] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:45:51] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:46:02] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:46:21] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:46:22] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:46:22] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:48:16] (03PS1) 10KartikMistry: Beta: Fix language code [puppet] - 10https://gerrit.wikimedia.org/r/183001 [06:49:12] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.026 second response time [06:50:12] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.027 second response time [06:54:16] (03PS5) 10Yuvipanda: Add report that verifies view definitions [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 [06:55:10] kart_: ^^ since the commit is in the cxserver module, it should probably be prefixed with ‘cxserver’ rather than beta :) [07:13:12] (03PS6) 10Yuvipanda: Add report that verifies view definitions [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 [07:22:12] YuviPanda|zzz: ah :) [07:23:39] (03PS2) 10KartikMistry: cxserver: Fix language code [puppet] - 10https://gerrit.wikimedia.org/r/183001 [07:30:02] (03PS1) 10Springle: repool db1061 db1062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183002 [07:33:55] (03CR) 10Springle: [C: 032 V: 032] repool db1061 db1062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183002 (owner: 10Springle) [07:35:35] !log springle Synchronized wmf-config/db-eqiad.php: repool db1061 db1062, warm up (duration: 00m 06s) [07:50:51] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet has 1 failures [07:51:30] (03CR) 10Nikerabbit: [C: 031] "See also https://gerrit.wikimedia.org/r/#/c/183000" [puppet] - 10https://gerrit.wikimedia.org/r/183001 (owner: 10KartikMistry) [07:59:02] (03PS1) 10Andrew Bogott: Move ldap-codfw to nembus. [dns] - 10https://gerrit.wikimedia.org/r/183003 [08:01:47] (03CR) 10Andrew Bogott: [C: 032 V: 032] Move ldap-codfw to nembus. [dns] - 10https://gerrit.wikimedia.org/r/183003 (owner: 10Andrew Bogott) [08:02:09] !log moved codfw ldap service to nembus [08:02:41] !log restarted pdns on virt1000 and labcontrol2001 to handle the change to nembus (just in case pdns is upset by change!) [08:05:46] good morning [08:06:00] 'morning! [08:07:31] morebots, how's it going? [08:07:31] I am a logbot running on tools-exec-13. [08:07:31] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [08:07:31] To log a message, type !log . [08:07:39] !log moved codfw ldap service to nembus [08:07:47] Logged the message, Master [08:07:48] !log restarted pdns on virt1000 and labcontrol2001 to handle the change to nembus (just in case pdns is upset by change!) [08:07:52] Logged the message, Master [08:08:09] hm, much better [08:08:27] (03PS2) 10Hashar: Unbreak Parsoid URL config in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182935 (owner: 10Catrope) [08:08:41] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [08:09:58] (03CR) 10Hashar: [C: 032] "Chris wrote:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182935 (owner: 10Catrope) [08:11:14] hm [08:11:16] sn: The death wish2 [08:11:17] homeDirectory: /home/zz11zz [08:11:18] mail: fucking@1234yahoo.com [08:11:19] cn: The death wish2 [08:11:32] !log Zuul stalled for some reason :( [08:11:35] Logged the message, Master [08:11:54] I guess that's quite possibly a valid user account [08:12:12] hashar: yeah, see ops@ [08:12:19] paravoid: lacking a bouncer :( [08:12:32] no, ops@ mailing list [08:13:24] (03PS1) 10Andrew Bogott: Move the codfw ldap server off of labcontrol2001. [puppet] - 10https://gerrit.wikimedia.org/r/183005 [08:14:19] (03CR) 10Andrew Bogott: [C: 04-2] "I just moved the dns entry for ldap-codfw, so let's give this a few hours before merging while everything catches up with the dns change." [puppet] - 10https://gerrit.wikimedia.org/r/183005 (owner: 10Andrew Bogott) [08:16:16] (03PS2) 10Andrew Bogott: Move the codfw ldap server off of labcontrol2001. [puppet] - 10https://gerrit.wikimedia.org/r/183005 [08:23:19] I might have find the Zuul issue :D [08:24:02] hashar: hugs :) [08:32:23] so Zuul is hanging reporting back to Gerrit [08:32:30] some ssh connection is apparently deadlocked :( [08:33:26] andrewbogott: paravoid: would you mind restarting the Gerrit service on ytterbium.wikimedia.org please? :-] [08:33:54] I am hoping that will RST the tcp connection and unblock zuul [08:34:04] puppet already restart Gerrit whenever we do a config change to it [08:34:09] so should be safe [08:34:12] (03CR) 10Nikerabbit: [C: 04-1] "The part of labs code which is identical to this can be removed. Specifically, in CommonSettings-labs we would only need to set parsoid an" (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 (owner: 10KartikMistry) [08:37:19] hashar: ok, doing [08:38:11] !log restarted gerrit service on ytterbium [08:38:15] Logged the message, Master [08:38:53] hashar: done [08:39:03] andrewbogott: thanks, will see what happens on Zuul side :] [08:40:00] restart everything until it works? [08:40:03] not very scientific :) [08:40:34] I lack the skills to RST a single TCP connection :-D [08:40:58] (03CR) 10Hashar: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/183005 (owner: 10Andrew Bogott) [08:41:34] apergos: If you are still out sick, get well soon! If you are around… can you tell me about the 'publicdata' project? It's using 12Tb of space on /data/project -- I'm wondering if that's by design or if something has run amok. [08:42:33] !log Zuul scheduler was stuck while reporting a change back to Gerrit waiting for data to be received. For some reason none came back and Zuul halted entirely. Restarting Gerrit killed the stalled connection and made Zuul to drop all events and resume operations. [08:42:37] Logged the message, Master [08:42:56] paravoid: would it be ok if I help tgr|away set up debian packages? or you prefer a more experianced DD to do it ? [08:43:45] hashar: ss is your friend? :) [08:44:08] matanya: I'm suspecting he wants more than a debian package [08:44:16] so, let's have the conversation on the list first [08:44:27] matanya: I found out tcpkill which listen for traffic on a port and manage to spoof a RST tcp packet, but since the connection was stalled .. [08:44:28] seems so from the request [08:44:30] andrewbogott: fixed it; thank you! [08:45:15] (03PS4) 10Faidon Liambotis: Remove update-notifier-common from labs instances. [puppet] - 10https://gerrit.wikimedia.org/r/181539 (owner: 10Andrew Bogott) [08:45:21] (03CR) 10Faidon Liambotis: [C: 032] Remove update-notifier-common from labs instances. [puppet] - 10https://gerrit.wikimedia.org/r/181539 (owner: 10Andrew Bogott) [08:49:28] (03CR) 10KartikMistry: "wmgContentTranslationCluster is set false on Beta though." (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 (owner: 10KartikMistry) [08:50:02] (03CR) 10Faidon Liambotis: [C: 04-1] "For all of these, if you notice, there's a preceding File['/etc/init/foo'] stanza. Removing provider => upstart won't hurt but it won't ma" [puppet] - 10https://gerrit.wikimedia.org/r/182764 (owner: 10KartikMistry) [08:50:29] (03PS7) 10KartikMistry: Content Translation configuration for Production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 [09:07:31] (03PS8) 10KartikMistry: Content Translation configuration for Production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 [09:13:02] (03CR) 10Hashar: [C: 032] "+2 again, Zuul got stuck." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182935 (owner: 10Catrope) [09:13:08] (03Merged) 10jenkins-bot: Unbreak Parsoid URL config in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182935 (owner: 10Catrope) [09:13:41] !log hashar Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 05s) [09:13:47] Logged the message, Master [09:16:52] !log upgraded python version on zirconium [09:16:55] Logged the message, Master [09:18:51] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 1 failures [09:19:07] greetings [09:20:40] hello [09:36:52] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:37:17] (03CR) 10Faidon Liambotis: "You're right, I'm pretty sure I swapped the diff. So what the original change did was to set up pam_access, which we apparently use to do " [puppet] - 10https://gerrit.wikimedia.org/r/181789 (owner: 10Tim Landscheidt) [09:41:20] !log starting hhvm-profiler-to-carbon on tungsten T85641 [09:41:26] Logged the message, Master [09:41:40] !log started an analytics ETL run on dbstore1002. to disable: set global event_scheduler=0; [09:41:42] Logged the message, Master [09:43:38] (03PS1) 10Dzahn: planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 [09:44:32] (03CR) 10jenkins-bot: [V: 04-1] planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 (owner: 10Dzahn) [09:45:40] wow, pep8 is voting [09:45:48] i didnt write the .py file :p [09:46:21] just 1177 violations, come on [09:48:13] :( [09:48:24] i just want to import the file ... [09:48:33] did we really want it to be voting? [09:48:35] mutante: it can be ignored by adding a .pep8 file with proper content in modules/planet/files/ [09:48:41] ah! [09:48:47] (03PS2) 10Faidon Liambotis: Use motd::script instead of File across the tree [puppet] - 10https://gerrit.wikimedia.org/r/182376 [09:48:49] (03PS2) 10Faidon Liambotis: motd: manage /etc/motd, support Debian [puppet] - 10https://gerrit.wikimedia.org/r/182377 [09:48:51] (03PS2) 10Faidon Liambotis: base: use dash instead of bash for last-puppet-run [puppet] - 10https://gerrit.wikimedia.org/r/182372 [09:48:53] (03PS2) 10Faidon Liambotis: motd: misc fixes [puppet] - 10https://gerrit.wikimedia.org/r/182373 [09:48:53] ok,thx [09:48:55] (03PS2) 10Faidon Liambotis: motd: add motd::defaults class [puppet] - 10https://gerrit.wikimedia.org/r/182374 [09:48:57] (03PS2) 10Faidon Liambotis: base: include the motd class by default [puppet] - 10https://gerrit.wikimedia.org/r/182375 [09:49:02] mutante: there are a few examples around (find . -type f -name .pep8 ) [09:49:20] [pep8] [09:49:21] exclude=* [09:49:24] would do apparently :] [09:49:32] i see them, alright thanks [09:49:55] mutante: isn't that feedparser.py provided by a Debian package? [09:50:04] 2 # This is a third-party file, so I'm just turning off everything... 3 ignore = W191,E225,E501,W239,E303,W209,E302,W293,E101,E128,E121,E703,W291 [09:50:33] (03CR) 10Faidon Liambotis: [C: 032] base: use dash instead of bash for last-puppet-run [puppet] - 10https://gerrit.wikimedia.org/r/182372 (owner: 10Faidon Liambotis) [09:50:34] mutante: if we have a Debian package you could create a wmf version of it which monkey patch the code :] [09:50:35] hashar: yes, but we made a modification to it [09:50:43] i wanted to avoid that [09:50:43] (03CR) 10Faidon Liambotis: [C: 032] motd: misc fixes [puppet] - 10https://gerrit.wikimedia.org/r/182373 (owner: 10Faidon Liambotis) [09:51:01] Hi, does anybody know whom I should ask for a LDAP/Wikitech/Gerrit/Phabricator account renaming? [09:51:11] (03CR) 10Faidon Liambotis: [C: 032] motd: add motd::defaults class [puppet] - 10https://gerrit.wikimedia.org/r/182374 (owner: 10Faidon Liambotis) [09:51:12] maintain a package to comment out one line? [09:51:31] (03CR) 10Faidon Liambotis: [C: 032] base: include the motd class by default [puppet] - 10https://gerrit.wikimedia.org/r/182375 (owner: 10Faidon Liambotis) [09:51:51] (03CR) 10Faidon Liambotis: [C: 032] Use motd::script instead of File across the tree [puppet] - 10https://gerrit.wikimedia.org/r/182376 (owner: 10Faidon Liambotis) [09:51:57] (There is https://wikitech.wikimedia.org/wiki/Renaming_users, which helder claimed is out of date) [09:53:07] (03PS3) 10Faidon Liambotis: motd: manage /etc/motd, support Debian [puppet] - 10https://gerrit.wikimedia.org/r/182377 [09:54:03] (03CR) 10Filippo Giunchedi: Use monitoring::graphite_threshold for varnishkafka delivery error check (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182860 (owner: 10Ottomata) [09:55:18] (03PS2) 10Dzahn: planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 [09:55:59] (03CR) 10jenkins-bot: [V: 04-1] planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 (owner: 10Dzahn) [09:59:12] (03PS3) 10Dzahn: planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 [09:59:40] (03CR) 10Faidon Liambotis: [C: 031] "I think I'd prefer the c-relay to be in its own class as it's not shipped by Graphite proper and included from either the ::graphite class" [puppet] - 10https://gerrit.wikimedia.org/r/181080 (owner: 10Filippo Giunchedi) [10:00:07] (03CR) 10jenkins-bot: [V: 04-1] planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 (owner: 10Dzahn) [10:00:44] (03CR) 10Faidon Liambotis: [C: 031] "Change LGTM. Is this is a deb? Will local patching just work or does it need a debian/patches change instead?" [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/180786 (owner: 10Filippo Giunchedi) [10:04:07] hashar: without .pep8 file: 1177 violations. | with ignore = W191,E225,E501,W239,E303,W209,E302,W293,E101,E128,E121,E703,W291: 590 violations | with ignore = *: 1243 violations [10:04:24] mutante: you should probably exclude the file instead [10:04:28] ignore = * got me _more_ violations? [10:04:35] ok [10:04:39] mutante: i am sure some .pep8 file does it already :] [10:04:48] or package the hack :] [10:05:19] (03CR) 10Faidon Liambotis: [C: 032] motd: manage /etc/motd, support Debian [puppet] - 10https://gerrit.wikimedia.org/r/182377 (owner: 10Faidon Liambotis) [10:07:47] duh! ignore != exclude of course [10:10:38] (03PS4) 10Dzahn: planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 [10:14:38] (03CR) 10Dzahn: [C: 032] monitoring: fix a bunch of indentation warnings [puppet] - 10https://gerrit.wikimedia.org/r/182800 (owner: 10Dzahn) [10:18:38] oh [10:18:44] Zuul / Gerrit stalled again bah [10:18:51] PROBLEM - Corp OIT LDAP Mirror on plutonium is CRITICAL: Could not bind to the LDAP server [10:19:16] at the exact moment I had the Gerrit web UI spurting out an error reaching the database [10:20:40] (03CR) 10Dzahn: "what's wrong with using logrotate? also, what Yuvi said. i think find .. -delete is better than parsing output of ls" [puppet] - 10https://gerrit.wikimedia.org/r/182173 (owner: 10Hoo man) [10:20:53] RECOVERY - Corp OIT LDAP Mirror on plutonium is OK: LDAP OK - 0.031 seconds response time [10:21:08] !log Restarted Zuul. Gerrit transiently died out just like ~10 hours ago which locked Zuul entirely [10:21:12] Logged the message, Master [10:23:12] ah, first time i saw this one [10:23:24] Guice provision errors: 1) Cannot open ReviewDb [10:24:19] (03CR) 10Dzahn: [C: 032] ganglia_new: fix indentation warnings [puppet] - 10https://gerrit.wikimedia.org/r/182802 (owner: 10Dzahn) [10:26:03] (03CR) 10Dzahn: "looks good. all is missing is manager approval" [puppet] - 10https://gerrit.wikimedia.org/r/181556 (owner: 10Filippo Giunchedi) [10:30:54] mutante: yeah that is what caused Zuul to stall :-( [10:35:11] (03PS1) 10Dzahn: bugzilla: let varnish also handle bug-attachment [puppet] - 10https://gerrit.wikimedia.org/r/183012 [10:35:36] (03CR) 10Dzahn: "https://gerrit.wikimedia.org/r/#/c/183012/" [dns] - 10https://gerrit.wikimedia.org/r/182782 (owner: 10Dzahn) [10:36:43] (03PS5) 10Dzahn: planet: puppetize feedparser.py bug workaround [puppet] - 10https://gerrit.wikimedia.org/r/183007 [10:36:48] !log The Zuul issues have their root cause in Gerrit which cannot open ReviewDB. Filled as https://phabricator.wikimedia.org/T85916 [10:36:52] mutante: ^^ :-] [10:36:53] Logged the message, Master [10:37:50] gotcha [10:50:22] (03CR) 10Hoo man: "The problem with logrotate is that it usually rotates while the process is running, thus we end up with incomplete logs." [puppet] - 10https://gerrit.wikimedia.org/r/182173 (owner: 10Hoo man) [10:59:33] (03CR) 10Filippo Giunchedi: "what about syslog instead? if you really want your own logging for example cronolog will DTRT" [puppet] - 10https://gerrit.wikimedia.org/r/182173 (owner: 10Hoo man) [11:05:28] (03CR) 10Dzahn: "this looks like it will start enforcing https while before etherpad did not do that. afair there was a discussion about that back when new" [puppet] - 10https://gerrit.wikimedia.org/r/181413 (owner: 10John F. Lewis) [11:10:50] (03PS2) 10Dzahn: bugzilla: let varnish also handle bug-attachment [puppet] - 10https://gerrit.wikimedia.org/r/183012 [11:11:28] (03PS11) 10Hashar: contint: provision hhvm on CI slaves [puppet] - 10https://gerrit.wikimedia.org/r/178806 [11:11:47] (03CR) 10Dzahn: [C: 032] bugzilla: let varnish also handle bug-attachment [puppet] - 10https://gerrit.wikimedia.org/r/183012 (owner: 10Dzahn) [11:29:02] hashar: https://bugs.debian.org/774143 [11:29:15] (03CR) 10Dzahn: [C: 032] bugzilla: remove bug-attachment 443 virtual host [puppet] - 10https://gerrit.wikimedia.org/r/182781 (owner: 10Dzahn) [11:29:18] just came across this [11:29:23] are we using the http responder? [11:31:45] (03CR) 10Dzahn: [C: 032] move bug-attachments to misc-web [dns] - 10https://gerrit.wikimedia.org/r/182782 (owner: 10Dzahn) [11:32:36] (03PS1) 10Hashar: contint: hourly auto update of wikimedia packages [puppet] - 10https://gerrit.wikimedia.org/r/183019 [11:33:09] (03PS12) 10Hashar: contint: provision hhvm on CI slaves [puppet] - 10https://gerrit.wikimedia.org/r/178806 [11:33:44] (03CR) 10Hashar: "I have removed the package_ensure / latest in favor of using unattended upgrade (which is done in parent commit https://gerrit.wikimedia.o" [puppet] - 10https://gerrit.wikimedia.org/r/178806 (owner: 10Hashar) [11:38:14] (03PS2) 10Hashar: contint: hourly auto update of wikimedia packages [puppet] - 10https://gerrit.wikimedia.org/r/183019 [11:45:03] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [11:45:24] (03PS3) 10Hashar: contint: hourly auto update of wikimedia packages [puppet] - 10https://gerrit.wikimedia.org/r/183019 [11:48:02] (03CR) 10Hashar: "PS2 fix typos in the cron paths" [puppet] - 10https://gerrit.wikimedia.org/r/183019 (owner: 10Hashar) [11:53:36] (03CR) 10Hoo man: "I don't see the benefit of using cronolog over here. I just want one log file for each run. Nothing sophisticated, keep it simple." [puppet] - 10https://gerrit.wikimedia.org/r/182173 (owner: 10Hoo man) [11:53:43] (03PS4) 10Hashar: contint: hourly auto update of wikimedia packages [puppet] - 10https://gerrit.wikimedia.org/r/183019 [11:54:14] (03CR) 10Hashar: "Fixed syntax for APT::Periodic::RandomSleep" [puppet] - 10https://gerrit.wikimedia.org/r/183019 (owner: 10Hashar) [12:04:37] puppet sync issues between the 2 masters? [12:04:58] puppet reverts something i already merged to be removed [12:05:36] and now it worked... [12:08:42] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 91.198.174.247, interfaces up: 36, down: 1, dormant: 0, excluded: 1, unused: 0BRge-0/0/0: down - Core: msw-oe12-esamsBR [12:08:57] (03PS5) 10Hashar: contint: hourly auto update of wikimedia packages [puppet] - 10https://gerrit.wikimedia.org/r/183019 [12:11:20] (03CR) 10Hashar: "Replaced ${distro_id} with 'Wikimedia' to match our packages properly or they were not matched. unattended-upgrade was doing the followin" [puppet] - 10https://gerrit.wikimedia.org/r/183019 (owner: 10Hashar) [12:17:05] (03Abandoned) 10KartikMistry: Remove provider => upstart [puppet] - 10https://gerrit.wikimedia.org/r/182763 (owner: 10KartikMistry) [12:17:26] (03CR) 10Dzahn: "..but is writing your own bash script vs. using existing tools really 'keep it simple'?" [puppet] - 10https://gerrit.wikimedia.org/r/182173 (owner: 10Hoo man) [12:17:42] (03CR) 10KartikMistry: "Agree!" [puppet] - 10https://gerrit.wikimedia.org/r/182764 (owner: 10KartikMistry) [12:17:59] (03Abandoned) 10KartikMistry: Don't specify provider => upstart [puppet] - 10https://gerrit.wikimedia.org/r/182764 (owner: 10KartikMistry) [12:18:32] (03Abandoned) 10KartikMistry: Map wikishared DB to x1 master server [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182148 (owner: 10KartikMistry) [12:21:40] Need akosiaris now :) [12:25:51] kart_: he is not there today :D [12:26:25] (03CR) 10Hoo man: "In this case, I think so, yes... cronolog is not really meant for something like this (although it would work)." [puppet] - 10https://gerrit.wikimedia.org/r/182173 (owner: 10Hoo man) [12:26:42] mutante: How do you even tell cronolog to keep the last n logs around only? [12:27:11] sigh [12:32:27] seems like cronolog can't even do that [12:38:08] hoo: but you said you just want one log file anyways? i didnt suggest cronolog, but i think using find would be better than parsing the output of ls [12:41:08] Well, I guess I can make it use find, if you people like it [12:41:21] although it not exactly has the semantics I want [12:41:45] but that's more of a theoretical thing [12:43:12] hoo: it's just the first thought i have when i see something `ls .. | tail`. because of http://mywiki.wooledge.org/ParsingLs [12:43:40] but you dont even have to see it as blocking it [12:44:44] bbl ..food [13:04:24] mutante: Interesting read, yeah... I'll amend it to use find later on. [13:35:26] (03CR) 10QChris: "ottomata moved the analytics machines in the analytics vlan." [puppet] - 10https://gerrit.wikimedia.org/r/170668 (owner: 10QChris) [14:08:28] https://noc.wikimedia.org/pybal/eqiad/apaches is giving me a 403 error. Is that intended, or accidental when stuff was moved from fenari to new places in eqiad? [14:16:46] anomie: you want https://config-master.wikimedia.org/pybal/eqiad/apaches [14:16:53] (it was moved) [14:16:58] thanks [14:44:28] (03CR) 10Ottomata: Use monitoring::graphite_threshold for varnishkafka delivery error check (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182860 (owner: 10Ottomata) [14:45:30] (03CR) 10Ottomata: "Hm, I think we need to think about how to make sure maven on stat1002 only uses archiva." [puppet] - 10https://gerrit.wikimedia.org/r/170668 (owner: 10QChris) [14:47:19] I see lots of failures logged in memcached-serious.log, which seems to be partially behind T85428. Is that level of flakiness normal? [15:16:14] (03CR) 10Mark Bergsma: [C: 031] Make parsoid roots actually root on the parsoid boxes [puppet] - 10https://gerrit.wikimedia.org/r/182585 (owner: 10GWicke) [15:19:35] (03PS2) 10Rush: Make parsoid roots actually root on the parsoid boxes [puppet] - 10https://gerrit.wikimedia.org/r/182585 (owner: 10GWicke) [15:19:56] (03CR) 10Rush: [C: 032 V: 032] Make parsoid roots actually root on the parsoid boxes [puppet] - 10https://gerrit.wikimedia.org/r/182585 (owner: 10GWicke) [15:21:31] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [15:30:03] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/182860 (owner: 10Ottomata) [15:32:44] (03CR) 10Rush: "Hoo, my apologies it seems this did not get discussed in the Ops meeting. I am making a note to ensure it is brought up. AFAIK this woul" [puppet] - 10https://gerrit.wikimedia.org/r/152724 (owner: 10Hoo man) [15:37:25] (03CR) 10Rush: "could you add T85644 to the commit message well :)" [puppet] - 10https://gerrit.wikimedia.org/r/182347 (owner: 10Ori.livneh) [15:42:06] (03CR) 10coren: [C: 032] contint: tmpfs is now root:root and world writable [puppet] - 10https://gerrit.wikimedia.org/r/173511 (owner: 10Krinkle) [15:42:28] (03CR) 10coren: [C: 032] contint: Add tmpfs mount in jenkins-deploy homedir for labs slaves [puppet] - 10https://gerrit.wikimedia.org/r/173512 (https://bugzilla.wikimedia.org/72063) (owner: 10Krinkle) [15:50:34] marktraceur, manybubbles, ^d: Who wants to SWAT this morning? [15:50:39] Oof. [15:50:39] * anomie can if no one else does [15:50:51] I was about to say that but anomie can have it. [15:51:10] RoanKattouw, Glaisher: Ping for SWAT in about 9 minutes [15:51:19] pong [15:51:20] anomie: Pong [15:53:01] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 4 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [15:53:03] RoanKattouw: Is there already a core patch to bump the extension version for your VE fix? [15:53:56] anomie: No, there isn't. I can build one if you like [15:54:04] RoanKattouw: Please do [16:00:04] manybubbles, anomie, ^d, marktraceur, RoanKattouw: Dear anthropoid, the time has come. Please deploy Morning SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150106T1600). [16:00:56] Glaisher: I'll do yours first [16:01:08] (03PS4) 10Anomie: FlaggedRevs configuration on de.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180813 (owner: 10Glaisher) [16:01:17] (03CR) 10Anomie: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180813 (owner: 10Glaisher) [16:01:22] (03Merged) 10jenkins-bot: FlaggedRevs configuration on de.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180813 (owner: 10Glaisher) [16:01:45] !log anomie Synchronized wmf-config/flaggedrevs.php: SWAT: Add Flexion to $wgFlaggedRevsNamespaces for dewiktionary [[gerrit:180813]] (duration: 00m 08s) [16:01:47] Glaisher: ^ Test please [16:01:49] Logged the message, Master [16:01:53] thanks :) [16:01:54] looking [16:02:17] it's there :) [16:02:25] anomie: Sorry, got distracted, will do that now [16:02:29] RoanKattouw: Ok [16:02:37] * anomie waits for RoanKattouw [16:02:43] * RoanKattouw waits for git [16:14:13] anomie: https://gerrit.wikimedia.org/r/183042 [16:14:19] anomie: Sorry that took so long [16:14:25] * anomie looks [16:14:40] Reedy: online? I seem to have copy/pasted the wrong time for the tuesday deploy window (wiki says 19:00 UTC and we want to switch to 13:00) [16:15:03] orly? [16:15:04] Reedy: so, just saying "oops" and "kick me if I make the same mistake next week" [16:15:10] I thought we switched, and switched back? [16:15:12] i r confused [16:15:32] * anomie waits for Jenkins now [16:15:33] we switched back temporarily, but I think the germans (and euro ops) still want an earlier one [16:16:06] right aude ? :) ( you all still prefer the early deploy on Tuesdays?) [16:22:40] !log anomie Synchronized php-1.25wmf13/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.MobileViewTarget.js: SWAT: Update setupToolbar signature in mobile target [[gerrit:182993]] (duration: 00m 06s) [16:22:40] RoanKattouw: ^ Test please [16:22:42] Logged the message, Master [16:26:30] anomie: It's good enough for now, but the back arrow is still broken. I'll complain to the mobile people [16:26:46] * anomie is done with SWAT [16:26:56] (03PS1) 10Ottomata: Install virtualenv on stat* boxes now that they are in the Analytics VLAN [puppet] - 10https://gerrit.wikimedia.org/r/183044 [16:27:58] RoanKattouw: Do you mean https://phabricator.wikimedia.org/T85481 ? [16:28:24] Oh, yeah [16:28:28] I'll add the console error to that bug [16:28:33] Ta. [16:28:36] It's related to something Jon did recently, trying to find that too [16:28:41] * James_F nods. [16:30:43] (03PS1) 10Ottomata: Add ananthrk to statistics-users group [puppet] - 10https://gerrit.wikimedia.org/r/183047 [16:31:14] (03CR) 10Ottomata: [C: 032] Install virtualenv on stat* boxes now that they are in the Analytics VLAN [puppet] - 10https://gerrit.wikimedia.org/r/183044 (owner: 10Ottomata) [16:32:11] (03CR) 10Filippo Giunchedi: "what I'm trying to say is that it isn't clear to me why doing dumps should be" (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/182173 (owner: 10Hoo man) [16:32:32] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [16:47:31] Double posting from wikimedia-dev. Is there a configuration setting that enforce a valid email address and/or to ban some known abuse spam providers (e.g. trash-mail.com). I don’t see anything about it in https://www.mediawiki.org/wiki/Manual:Configuration_settings [16:49:30] Reedy: When's the wmf12->wmf13 change going out for group 1 wikis? [16:50:09] James_F: 1900 UTC today [16:50:26] Reedy: Aha, it's scheduled for … yeah. Weren't we doing those at 16:00 from now on so that Wikidata had more time, or something? [16:50:40] Yeah, we were, then reverted, then went back [16:50:47] Except greg copy paste failed for this week [16:50:50] Ha. :-) [16:53:35] I thought 16:00 was before 19:00. [16:53:52] TIL. [16:54:45] James_F: mea culpa [17:06:02] manybubbles ^d ottomata meeting today? [17:06:18] godog: in an hour and half, yeah! [17:06:54] manybubbles: yep! ack [17:08:47] (03CR) 10Filippo Giunchedi: "it is our own deb, local patching is fine, I'm not even sure it is upstream-worthy" [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/180786 (owner: 10Filippo Giunchedi) [17:08:56] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] txstatsd: add support for graphite line-protocol [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/180786 (owner: 10Filippo Giunchedi) [17:20:14] (03CR) 10Filippo Giunchedi: "I stand corrected, debian/patches review coming up" [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/180786 (owner: 10Filippo Giunchedi) [17:24:07] (03PS1) 10Glaisher: Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 [17:24:11] (03CR) 10jenkins-bot: [V: 04-1] Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 (owner: 10Glaisher) [17:25:01] (03PS2) 10Glaisher: Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 [17:29:33] (03PS1) 10Glaisher: Create 'interface_editor' user group on cawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183056 [17:31:03] PROBLEM - ElasticSearch health check for shards on logstash1003 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.136:9200/_cluster/health error while fetching: Request timed out. [17:33:21] RECOVERY - ElasticSearch health check for shards on logstash1003 is OK: OK - elasticsearch status production-logstash-eqiad: status: green, number_of_nodes: 3, unassigned_shards: 0, timed_out: False, active_primary_shards: 62, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 186, initializing_shards: 0, number_of_data_nodes: 3 [17:34:32] cmjohnson1: still on ops duty? [17:35:53] No, but might be able to help [17:41:37] cmjohnson1: mind looking at https://gerrit.wikimedia.org/r/#/c/181415/ [17:41:52] * cmjohnson1 goes to look [17:46:03] JohnFLewis: I want to wait and see how paravoid responds to mutante https://phabricator.wikimedia.org/T85789 [17:46:30] cmjohnson1: ah missed that comment. Alright [17:47:33] (03CR) 10Cmjohnson: [C: 031] "Looks okay but not going to +2 because of a question on Bug T85789" [puppet] - 10https://gerrit.wikimedia.org/r/181415 (owner: 10Dzahn) [17:49:52] cmjohnson1: another patch I have is https://gerrit.wikimedia.org/r/#/c/181987/ - just an IPv6 map on a server so not really a big patch [17:50:43] (03CR) 10Cmjohnson: [C: 032] map ipv6 on dataset1001 [puppet] - 10https://gerrit.wikimedia.org/r/181987 (owner: 10John F. Lewis) [17:51:50] (03PS2) 10Cmjohnson: map ipv6 on dataset1001 [puppet] - 10https://gerrit.wikimedia.org/r/181987 (owner: 10John F. Lewis) [17:52:03] cmjohnson1: sweet, once it is applied to dataset1001 - could you source the IP and I'll add a DNS patch for it [17:52:41] (03CR) 10Cmjohnson: [C: 032] map ipv6 on dataset1001 [puppet] - 10https://gerrit.wikimedia.org/r/181987 (owner: 10John F. Lewis) [17:58:11] (03CR) 10Nikerabbit: [C: 04-1] Content Translation configuration for Production (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 (owner: 10KartikMistry) [18:00:04] maxsem, kaldari: Respected human, time to deploy Mobile Web (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150106T1800). Please do the needful. [18:02:58] (03PS1) 10Ori.livneh: redis: Get rid of unused (and unhelpful) module parameters [puppet] - 10https://gerrit.wikimedia.org/r/183060 [18:03:12] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [18:03:56] (03PS1) 10Cmjohnson: Revert "map ipv6 on dataset1001" [puppet] - 10https://gerrit.wikimedia.org/r/183061 [18:06:04] (03PS9) 10KartikMistry: Content Translation configuration for Production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 [18:06:54] (03PS1) 10John F. Lewis: add releng staff to contint-admins (gallium) [puppet] - 10https://gerrit.wikimedia.org/r/183062 [18:07:42] (03PS3) 10Ori.livneh: Revoke Brett Simmer's key [puppet] - 10https://gerrit.wikimedia.org/r/182347 [18:07:53] @chasemp ^ done [18:09:07] (03PS4) 10Rush: Revoke Brett Simmer's key [puppet] - 10https://gerrit.wikimedia.org/r/182347 (owner: 10Ori.livneh) [18:11:43] greg-g: i think evening / later deploys on tuesday are generally better, since hoo can't be around earlier [18:12:00] and works much better when i am working from the US :) [18:12:30] but the change to wednesday (instead of thursday) has been helpful [18:15:09] (03CR) 10Rush: [C: 032] Revoke Brett Simmer's key [puppet] - 10https://gerrit.wikimedia.org/r/182347 (owner: 10Ori.livneh) [18:16:32] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [18:22:57] chasemp: thanks! [18:23:43] np could you run that on a theoretically affected host to make sure [18:23:43] it wipes that out [18:25:20] sure [18:26:29] chasemp: confirmed [18:26:47] thanks man [18:29:12] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: puppet fail [18:30:40] !log upload txstatsd 1.0.0-2 to trusty-wikimedia [18:30:45] Logged the message, Master [18:34:12] PROBLEM - Disk space on dataset1001 is CRITICAL: Timeout while attempting connection [18:35:32] PROBLEM - Host dataset1001 is DOWN: CRITICAL - Host Unreachable (208.80.154.11) [18:38:52] RECOVERY - Host dataset1001 is UP: PING OK - Packet loss = 0%, RTA = 10.50 ms [18:46:12] (03Draft1) 10Filippo Giunchedi: Revert "txstatsd: add support for graphite line-protocol" [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/183057 [18:46:18] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Revert "txstatsd: add support for graphite line-protocol" [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/183057 (owner: 10Filippo Giunchedi) [18:46:24] (03Draft1) 10Filippo Giunchedi: txstatsd: add patch to debian/patches and changelog entry [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/183058 [18:46:32] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] txstatsd: add patch to debian/patches and changelog entry [debs/txstatsd] - 10https://gerrit.wikimedia.org/r/183058 (owner: 10Filippo Giunchedi) [18:48:14] (03PS1) 10Reedy: Non wikipedias to 1.25wmf13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183068 [18:48:45] (03PS1) 10Chad: Remove MWSearch from remaining wikis, uninstall [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 [18:49:07] <^d> manybubbles, godog: Wheee ^^ [18:50:01] ^d \o/ \o/ \o/ [18:50:12] did I do that right? [18:50:14] first go round [18:50:14] ^d: \o/ [18:50:20] chasemp: yes [18:50:28] k thanks [18:51:03] we shall have celebrations in SF [18:51:34] (03CR) 10Manybubbles: [C: 031] "Looks like it'll work." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 (owner: 10Chad) [18:52:11] (03Abandoned) 10Chad: Remove lsearchd pools 2,4,5 from DNS [dns] - 10https://gerrit.wikimedia.org/r/181092 (owner: 10Chad) [18:52:19] (03Abandoned) 10Chad: Kill search pools 2,4,5 from LVS [puppet] - 10https://gerrit.wikimedia.org/r/181091 (owner: 10Chad) [18:52:45] (03CR) 10Reedy: [C: 04-1] "You want to delete docroot/noc/conf/lucene-common.php.txt and docroot/noc/conf/lucene-production.php.txt too" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 (owner: 10Chad) [18:53:03] <^d> thx [18:53:53] (03PS2) 10Chad: Remove MWSearch from remaining wikis, uninstall [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 [18:54:00] <^d> Reedy: {{done}} [18:54:06] (03CR) 10Nemo bis: [C: 031] "\o/ How about docroot/mediawiki/xml/lsearch-0.1.xsd and docroot/mediawiki/xml/lsearch-0.1/ ?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 (owner: 10Chad) [18:54:21] PROBLEM - Varnishkafka Delivery Errors on cp3016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 88.091667 [18:54:42] (03CR) 10Reedy: [C: 031] "diediedie :D" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 (owner: 10Chad) [18:55:35] <^d> Nemo_bis: Harmless enough to keep for now. [18:55:37] Nemo_bis: I think those want to stay (indefinitely?) [18:55:39] <^d> Can always come back later. [18:55:52] If other people are using it, and linking and such [18:56:08] https://phabricator.wikimedia.org/T44084 [18:56:21] ^d: did you do what you were saying the other day after all? api backend selector etc. [18:56:23] lol [18:56:36] <^d> paravoid: it fails gracefully, i'm pleased. [18:56:47] <^d> it just warns about an unused parameter and spits out results anyway [18:56:51] <^d> So nobody should break hard. [18:57:22] RECOVERY - Varnishkafka Delivery Errors on cp3016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:58:42] ^d: when are you deploying it? :) [18:58:51] <^d> jouncebot_: next [18:58:51] In 0 hour(s) and 1 minute(s): MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150106T1900) [18:58:57] <^d> Well crud. [18:59:48] <^d> Was gonna say right now ;-) [19:00:04] Reedy, greg-g: Respected human, time to deploy MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150106T1900). Please do the needful. [19:00:33] (03CR) 10Reedy: [C: 032] Non wikipedias to 1.25wmf13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183068 (owner: 10Reedy) [19:00:38] (03Merged) 10jenkins-bot: Non wikipedias to 1.25wmf13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183068 (owner: 10Reedy) [19:01:47] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf13 [19:01:49] Logged the message, Master [19:02:24] 19:01:44 sudo -u mwdeploy -n -- /usr/bin/rsync -l tin.eqiad.wmnet::common/wikiversions*.{json,cdb} /srv/mediawiki on mw1203 returned [255]: Error reading response length from authentication socket. [19:02:24] Permission denied (publickey). [19:03:15] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: nooop [19:03:18] Logged the message, Master [19:03:18] transient I guess [19:03:57] ^d: you can go now if you want, I'll do some config change stuff in a little bit [19:04:23] (03CR) 10Chad: [C: 032] Remove MWSearch from remaining wikis, uninstall [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 (owner: 10Chad) [19:04:27] (03Merged) 10jenkins-bot: Remove MWSearch from remaining wikis, uninstall [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183069 (owner: 10Chad) [19:05:11] :D [19:06:35] !log demon Started scap: mwsearch is no more [19:06:39] Logged the message, Master [19:10:11] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [19:15:03] PROBLEM - Graphite Carbon on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:15:12] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 1.462 second response time [19:15:12] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [19:15:32] PROBLEM - DPKG on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:16:52] PROBLEM - configured eth on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:17:01] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:17:02] PROBLEM - SSH on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:12] PROBLEM - dhclient process on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:17:12] PROBLEM - puppet last run on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:17:21] PROBLEM - MediaWiki profile collector on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:17:21] PROBLEM - uWSGI web apps on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:17:52] RECOVERY - DPKG on tungsten is OK: All packages OK [19:17:52] RECOVERY - configured eth on tungsten is OK: NRPE: Unable to read output [19:18:02] RECOVERY - SSH on tungsten is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [19:18:12] RECOVERY - dhclient process on tungsten is OK: PROCS OK: 0 processes with command name dhclient [19:18:12] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [19:18:22] RECOVERY - MediaWiki profile collector on tungsten is OK: OK: All defined mwprof jobs are runnning. [19:18:22] RECOVERY - uWSGI web apps on tungsten is OK: OK: All defined uWSGI apps are runnning. [19:18:31] RECOVERY - Graphite Carbon on tungsten is OK: OK: All defined Carbon jobs are runnning. [19:19:12] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [19:21:02] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 2.977 second response time [19:25:43] (03PS1) 10Faidon Liambotis: Revert "map ipv6 on dataset1001" [puppet] - 10https://gerrit.wikimedia.org/r/183074 [19:25:54] (03PS2) 10Faidon Liambotis: Revert "map ipv6 on dataset1001" [puppet] - 10https://gerrit.wikimedia.org/r/183074 [19:26:10] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Revert "map ipv6 on dataset1001" [puppet] - 10https://gerrit.wikimedia.org/r/183074 (owner: 10Faidon Liambotis) [19:29:11] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:29:31] (03PS1) 10Reedy: Add freepats for Score usage [puppet] - 10https://gerrit.wikimedia.org/r/183075 [19:29:33] Any ops around? [19:29:41] ^ seems score/timidity is broken on 14.04 [19:30:27] chasemp: ^^ [19:30:50] looking now [19:31:21] <^d> Reedy: 3/4 done with my scap. [19:31:22] Should be trivial. Addition of a package with no dependencies [19:31:37] I don't really understand the extent of the problem yet so looking at the ticket [19:31:54] It seems timidity depends on freepats [19:32:36] lol, it's a reccomends [19:32:39] http://packages.ubuntu.com/lucid/timidity [19:32:44] http://packages.ubuntu.com/lucid/freepats [19:34:10] /etc/timidity/timidity.cfg [19:34:16] # By default, try to use the instrument patches from freepats: [19:34:17] source /etc/timidity/freepats.cfg [19:34:19] "Guice provision errors" gerrit db error again? [19:35:05] based on https://launchpad.net/ubuntu/trusty/armhf/timidity/2.13.2-40.2 [19:35:09] and teh report yeah I get it [19:35:11] weird tho [19:35:30] <^d> Krinkle: That's the only way gerrit knows how to respond when it can't reach the database. [19:35:32] (03CR) 10Faidon Liambotis: [C: 031] "Generally looks okay, so +1. I left a few comments for (optional) further cleanups, if you're planning to continue those." (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/183060 (owner: 10Ori.livneh) [19:35:42] ^d: Yeah, you told me in the past. [19:35:51] But that seems to happen way too oftne. [19:35:59] It's like every few hours I get that error [19:36:07] usually refres fixes it but that's seriously broken. [19:36:24] <^d> Nothing's changed on the gerrit side. [19:36:29] <^d> Dunno about m2. [19:36:40] (03CR) 10Rush: [C: 032] "Based on https://launchpad.net/ubuntu/trusty/armhf/timidity/2.13.2-40.2" [puppet] - 10https://gerrit.wikimedia.org/r/183075 (owner: 10Reedy) [19:37:22] chasemp: sweet. thanks :) [19:42:15] !log demon Finished scap: mwsearch is no more (duration: 35m 40s) [19:42:21] Logged the message, Master [19:42:45] ding dong [19:43:41] <^d> paravoid: T85009 is totally in ops' hands now. Rip apart as you see fit. [19:43:46] (03PS2) 10Reedy: Undeploy OpenSearchXml [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182288 (owner: 10Legoktm) [19:43:52] (03CR) 10Reedy: [C: 032] Undeploy OpenSearchXml [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182288 (owner: 10Legoktm) [19:43:56] (03Merged) 10jenkins-bot: Undeploy OpenSearchXml [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182288 (owner: 10Legoktm) [19:44:16] ^d: you're awesome [19:44:27] assigned to godog already I see, so I won't interfere :) [19:44:56] (03PS2) 10Reedy: Update noc's index.html [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182193 (owner: 10Glaisher) [19:45:00] (03CR) 10Reedy: [C: 032] Update noc's index.html [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182193 (owner: 10Glaisher) [19:45:04] (03Merged) 10jenkins-bot: Update noc's index.html [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182193 (owner: 10Glaisher) [19:45:53] <^d> happy times [19:47:15] aude: well then.... :) [19:47:24] (03PS3) 10Reedy: Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 (owner: 10Glaisher) [19:48:51] (03PS2) 10Reedy: Move composer.json into repository root [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180589 (owner: 10Legoktm) [19:48:58] (03CR) 10Greg Grossmeier: [C: 031] ""Manager approval comment" :)" [puppet] - 10https://gerrit.wikimedia.org/r/183062 (owner: 10John F. Lewis) [19:49:00] (03CR) 10Faidon Liambotis: [C: 04-1] contint: hourly auto update of wikimedia packages (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/183019 (owner: 10Hashar) [19:50:20] (03CR) 10Legoktm: "Hmm, I filed T85946 for techblog.wm.o being broken." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182193 (owner: 10Glaisher) [19:50:30] (03CR) 10Reedy: [C: 032] Move composer.json into repository root [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180589 (owner: 10Legoktm) [19:50:31] godog: ms-be2003 isn't very happy... [19:50:35] (03Merged) 10jenkins-bot: Move composer.json into repository root [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180589 (owner: 10Legoktm) [19:50:43] can't deal with it now though [19:51:49] (03PS4) 10Reedy: Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 (owner: 10Glaisher) [19:53:32] (03PS2) 10Reedy: mediawikiwiki: Add Api: and Skin: namespace to default searched namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182818 (owner: 10Florianschmidtwelzow) [19:53:37] (03CR) 10Reedy: [C: 032] mediawikiwiki: Add Api: and Skin: namespace to default searched namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182818 (owner: 10Florianschmidtwelzow) [19:53:41] (03Merged) 10jenkins-bot: mediawikiwiki: Add Api: and Skin: namespace to default searched namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182818 (owner: 10Florianschmidtwelzow) [19:54:52] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 4 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/). [19:55:09] (03PS3) 10Reedy: Add en.wikisource to global abuse filters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179864 (owner: 10TTO) [19:55:14] (03CR) 10Reedy: [C: 032] Add en.wikisource to global abuse filters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179864 (owner: 10TTO) [19:55:18] (03Merged) 10jenkins-bot: Add en.wikisource to global abuse filters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179864 (owner: 10TTO) [19:57:25] (03PS2) 10Reedy: Fix typo in dvwiktionary project talk namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182364 (owner: 10Glaisher) [19:57:30] (03CR) 10Reedy: [C: 032] Fix typo in dvwiktionary project talk namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182364 (owner: 10Glaisher) [19:57:34] (03Merged) 10jenkins-bot: Fix typo in dvwiktionary project talk namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182364 (owner: 10Glaisher) [19:57:59] (03PS5) 10Reedy: Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 (owner: 10Glaisher) [19:58:04] (03CR) 10Reedy: Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 (owner: 10Glaisher) [19:58:11] (03CR) 10Reedy: [C: 032] Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 (owner: 10Glaisher) [19:58:17] (03Merged) 10jenkins-bot: Change $wgUploadNavigationUrl on rowiki and tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183055 (owner: 10Glaisher) [19:58:27] finally [19:59:12] (03PS3) 10Reedy: Removed 'OTRS-member' user group on commons. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180560 (owner: 10Dereckson) [19:59:17] (03CR) 10Reedy: [C: 032] Removed 'OTRS-member' user group on commons. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180560 (owner: 10Dereckson) [19:59:22] (03Merged) 10jenkins-bot: Removed 'OTRS-member' user group on commons. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180560 (owner: 10Dereckson) [20:00:20] !log reedy Synchronized docroot and w: update noc index (duration: 00m 06s) [20:00:26] Logged the message, Master [20:01:11] (03PS3) 10Reedy: Apply workaround for chrome font bug in 404 error page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181869 (owner: 10Glaisher) [20:01:14] (03CR) 10Reedy: [C: 032] Apply workaround for chrome font bug in 404 error page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181869 (owner: 10Glaisher) [20:01:18] (03Merged) 10jenkins-bot: Apply workaround for chrome font bug in 404 error page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181869 (owner: 10Glaisher) [20:02:29] (03PS2) 10Reedy: Add task references to wgImportSources block [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180543 (owner: 10Dereckson) [20:02:35] (03CR) 10Reedy: [C: 032] Add task references to wgImportSources block [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180543 (owner: 10Dereckson) [20:02:41] (03Merged) 10jenkins-bot: Add task references to wgImportSources block [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180543 (owner: 10Dereckson) [20:03:56] (03PS3) 10Reedy: Import sources configuration on el.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180546 (owner: 10Dereckson) [20:04:03] (03CR) 10Reedy: [C: 032] Import sources configuration on el.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180546 (owner: 10Dereckson) [20:04:08] (03Merged) 10jenkins-bot: Import sources configuration on el.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180546 (owner: 10Dereckson) [20:05:25] (03PS2) 10Reedy: Remove unused mobile CentralNotice URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182132 (owner: 10AndyRussG) [20:05:31] (03CR) 10Reedy: [C: 032] Remove unused mobile CentralNotice URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182132 (owner: 10AndyRussG) [20:05:35] (03Merged) 10jenkins-bot: Remove unused mobile CentralNotice URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182132 (owner: 10AndyRussG) [20:07:43] (03PS4) 10Reedy: Namespaces configuration on kab.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179433 (owner: 10Dereckson) [20:07:52] (03CR) 10Reedy: [C: 032] Namespaces configuration on kab.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179433 (owner: 10Dereckson) [20:07:57] (03Merged) 10jenkins-bot: Namespaces configuration on kab.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179433 (owner: 10Dereckson) [20:08:36] (03PS2) 10Reedy: Enabled NewUserMessage on fa.wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179596 (owner: 10Dereckson) [20:08:40] (03CR) 10Reedy: [C: 032] Enabled NewUserMessage on fa.wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179596 (owner: 10Dereckson) [20:08:45] (03Merged) 10jenkins-bot: Enabled NewUserMessage on fa.wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179596 (owner: 10Dereckson) [20:10:26] (03CR) 10Reedy: [C: 04-1] "If you're adding spaces after the require/includes, you should remove the brackets :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178772 (owner: 10KartikMistry) [20:10:46] (03PS3) 10Reedy: Namespace configuration on el.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180518 (owner: 10Dereckson) [20:10:51] (03CR) 10Reedy: [C: 032] Namespace configuration on el.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180518 (owner: 10Dereckson) [20:10:56] (03Merged) 10jenkins-bot: Namespace configuration on el.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180518 (owner: 10Dereckson) [20:11:04] (03PS2) 10Reedy: Namespace configuration on sa.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180684 (owner: 10Dereckson) [20:11:12] (03CR) 10Reedy: [C: 032] Namespace configuration on sa.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180684 (owner: 10Dereckson) [20:11:16] (03Merged) 10jenkins-bot: Namespace configuration on sa.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180684 (owner: 10Dereckson) [20:12:01] (03PS2) 10Reedy: Create 'interface_editor' user group on cawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183056 (owner: 10Glaisher) [20:12:05] (03CR) 10Reedy: [C: 032] Create 'interface_editor' user group on cawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183056 (owner: 10Glaisher) [20:12:11] (03Merged) 10jenkins-bot: Create 'interface_editor' user group on cawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183056 (owner: 10Glaisher) [20:13:57] (03PS2) 10Reedy: Reduce global function count. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172559 [20:14:20] (03CR) 10Reedy: [C: 032] Reduce global function count. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172559 (owner: 10Reedy) [20:14:24] (03Merged) 10jenkins-bot: Reduce global function count. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172559 (owner: 10Reedy) [20:14:47] (03PS2) 10Reedy: Add Anhang and Zitaten namespaces for ndswiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182342 (owner: 10TTO) [20:14:58] (03CR) 10Reedy: [C: 032] Add Anhang and Zitaten namespaces for ndswiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182342 (owner: 10TTO) [20:15:02] (03Merged) 10jenkins-bot: Add Anhang and Zitaten namespaces for ndswiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182342 (owner: 10TTO) [20:16:11] !log reedy Synchronized wmf-config/: Config updates (duration: 00m 06s) [20:16:12] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [20:16:17] Logged the message, Master [20:16:46] !log Ran namespaceDupes.php on ndswiktionary [20:16:50] Logged the message, Master [20:17:57] (03PS3) 10Reedy: Hygiene: Move wgMFAnonymousEditing to InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182175 (owner: 10Florianschmidtwelzow) [20:18:10] (03CR) 10Reedy: [C: 032] Hygiene: Move wgMFAnonymousEditing to InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182175 (owner: 10Florianschmidtwelzow) [20:18:14] (03Merged) 10jenkins-bot: Hygiene: Move wgMFAnonymousEditing to InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182175 (owner: 10Florianschmidtwelzow) [20:18:17] (03CR) 10Vogone: "IMHO, the name 'interface_editor' is crap as it does not follow the naming conventions other user groups seem to follow to (due to the und" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183056 (owner: 10Glaisher) [20:19:42] (03CR) 10Reedy: "Damn, yeah... Will revert" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183056 (owner: 10Glaisher) [20:19:49] (03PS1) 10Reedy: Revert "Create 'interface_editor' user group on cawikibooks" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183085 [20:19:54] (03PS2) 10Reedy: Revert "Create 'interface_editor' user group on cawikibooks" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183085 [20:19:59] (03CR) 10Reedy: [C: 032] Revert "Create 'interface_editor' user group on cawikibooks" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183085 (owner: 10Reedy) [20:20:04] (03Merged) 10jenkins-bot: Revert "Create 'interface_editor' user group on cawikibooks" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183085 (owner: 10Reedy) [20:21:46] (03PS3) 10Reedy: Change tboverride to tboverride-account for enwiki accountcreators [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182197 (owner: 10Jackmcbarn) [20:21:52] (03CR) 10Reedy: [C: 032] Change tboverride to tboverride-account for enwiki accountcreators [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182197 (owner: 10Jackmcbarn) [20:21:56] (03Merged) 10jenkins-bot: Change tboverride to tboverride-account for enwiki accountcreators [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182197 (owner: 10Jackmcbarn) [20:24:05] !log reedy Synchronized wmf-config/: (no message) (duration: 00m 06s) [20:24:11] Logged the message, Master [20:30:48] (03PS1) 10Legoktm: Add Renameuser debug log group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183087 [20:32:24] Go Reedy go. [21:21:38] (03PS2) 10Ori.livneh: redis: Get rid of unused (and unhelpful) module parameters [puppet] - 10https://gerrit.wikimedia.org/r/183060 [21:24:20] (03PS3) 10Ori.livneh: redis: Get rid of unused (and unhelpful) module parameters [puppet] - 10https://gerrit.wikimedia.org/r/183060 [21:26:55] (03CR) 10Gergő Tisza: [C: 04-1] "$wgSentryEndpoint was changed to $wgSentryDsn, needs update." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181439 (owner: 10Gergő Tisza) [21:29:29] greg-g: ping [21:29:29] aude: You sent me a contentless ping. This is a contentless pong. Please provide a bit of information about what you want and I will respond when I am around. [21:29:33] aaaah [21:29:57] greg-g: want to schedule a deployment slot on thursday (since hoo and i are not around tomorrow) [21:30:22] (03PS4) 10Ori.livneh: redis: Get rid of unused (and unhelpful) module parameters [puppet] - 10https://gerrit.wikimedia.org/r/183060 [21:36:01] PROBLEM - Varnishkafka Delivery Errors on cp3016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1733.75 [21:36:45] greg-g: https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=139990&oldid=139951 [21:39:11] RECOVERY - Varnishkafka Delivery Errors on cp3016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:41:30] (03PS13) 10Ori.livneh: contint: provision hhvm on CI slaves [puppet] - 10https://gerrit.wikimedia.org/r/178806 (owner: 10Hashar) [21:41:46] (03CR) 10Faidon Liambotis: [C: 031] "Oh wow, thanks :)" [puppet] - 10https://gerrit.wikimedia.org/r/183060 (owner: 10Ori.livneh) [21:42:59] paravoid: thanks! i tested it using the puppet compiler, and now i'm testing it on labs too. if it's smooth and i have the time i may merge it. [21:43:35] awesome [21:43:50] that hhvm/CI change is becoming simpler and simpler isn't it [21:44:26] yep [21:45:06] are the HHVM settings otherwise sane? [21:46:22] i should check them over again, i think. 'use_syslog' is false by default, for example [21:46:51] oh, but we override it in hhvm/init.pp [21:50:39] (03CR) 10Ori.livneh: contint: provision hhvm on CI slaves (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/178806 (owner: 10Hashar) [21:52:43] (03CR) 10Ori.livneh: "Tested using the puppet compiler and on labs. I think I'm too distracted to merge this myself today, so if you feel like doing it tomorrow" [puppet] - 10https://gerrit.wikimedia.org/r/183060 (owner: 10Ori.livneh) [21:56:48] is hashar around? jenkins hates jslint today. https://integration.wikimedia.org/ci/job/parsoidsvc-jslint/3153/console [21:58:49] cscott: jshint must not like your code ! [21:58:59] cscott: I think jshint got upgraded this week [22:00:31] cscott: ah perm denied bah [22:01:02] jshint got "upgraded" [22:01:16] the file belong to root:root :-/ [22:01:32] * cscott sings root, root, root for the home team... [22:01:36] how is that even possible [22:04:32] cscott: too late for me. Can you fill a bug stating that on integration-slave1007 the directory /srv/deployment/integration/slave-scripts is borked ? [22:05:04] cscott: it is a git clone of integration/jenkins.git maintained by puppet. For some reason some files are not group/other writables [22:05:33] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: puppet fail [22:19:35] (03CR) 10QChris: "> Hm, I think we need to think about how to make sure maven" [puppet] - 10https://gerrit.wikimedia.org/r/170668 (owner: 10QChris) [22:24:42] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:47:20] (03PS1) 10Tim Starling: Fix timeout comment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183160 [23:12:00] (03PS1) 10Ori.livneh: Password-protect `forceprofile` magic URL parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183165 [23:15:33] (03Abandoned) 10Ori.livneh: Password-protect `forceprofile` magic URL parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183165 (owner: 10Ori.livneh) [23:18:09] zuul is stuck again? [23:41:11] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail