[00:00:50] !log unattended rolling restart of Elasticsearch cluster is going just fine - adding the 30 minute sleep between servers and turning down the replication rate makes it pretty boring. [00:00:55] Logged the message, Master [00:05:20] !log if anyone is reading the SAL for fun or sees an error in Elasticsearch cluster in the next 24 hours - we're performing an elasticsearch upgrade. We've set it up this time so its super slow and boring. So boring I'm going to sleep through it. If you see more then transient complaining from icinga about elasticsearch you can call me/have someone with access to the contact list call me. I expect icinga to complain about a [00:05:20] single node going down but I expect the cluster to stay "yellow" during the process- no alerts. [00:05:27] Logged the message, Master [00:06:08] * bd808 was crushed under a wall of SAL announcements [00:07:10] manybubbles: I guess this means I need to plan a logstash cluster update soon [00:07:42] !log bd808 needs to plan a logstash upgrade soon - let it be logged [00:07:48] Logged the message, Master [00:08:32] !log (manybubbles contd.) …a single node going down but I expect the cluster to stay "yellow" during the process- no alerts. [00:08:38] Logged the message, Master [00:36:16] PROBLEM - ElasticSearch health check on elastic1006 is CRITICAL: CRITICAL - Could not connect to server 10.64.0.113 [00:49:56] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [01:07:37] PROBLEM - Disk space on elastic1005 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 20086 MB (3% inode=99%): [01:59:56] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 23:59:40 UTC [02:00:16] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Thu Aug 21 02:00:10 UTC 2014 [02:10:17] PROBLEM - mailman_qrunner on sodium is CRITICAL: PROCS CRITICAL: 0 processes with UID = 38 (list), regex args /mailman/bin/qrunner [02:10:47] PROBLEM - mailman_ctl on sodium is CRITICAL: PROCS CRITICAL: 0 processes with UID = 38 (list), regex args /mailman/bin/mailmanctl [02:11:46] RECOVERY - mailman_ctl on sodium is OK: PROCS OK: 1 process with UID = 38 (list), regex args /mailman/bin/mailmanctl [02:12:16] RECOVERY - mailman_qrunner on sodium is OK: PROCS OK: 8 processes with UID = 38 (list), regex args /mailman/bin/qrunner [02:20:30] !log LocalisationUpdate completed (1.24wmf16) at 2014-08-21 02:19:26+00:00 [02:20:39] Logged the message, Master [02:22:40] (03PS1) 10Yurik: Zero: to unified 401-01, new unified 631-02 [operations/puppet] - 10https://gerrit.wikimedia.org/r/155492 [02:23:07] bblack, when you have a sec - https://gerrit.wikimedia.org/r/155492 [02:28:07] enwiki db is locked [02:29:50] fixed now [02:31:17] PROBLEM - mailman_qrunner on sodium is CRITICAL: PROCS CRITICAL: 0 processes with UID = 38 (list), regex args /mailman/bin/qrunner [02:31:47] PROBLEM - mailman_ctl on sodium is CRITICAL: PROCS CRITICAL: 0 processes with UID = 38 (list), regex args /mailman/bin/mailmanctl [02:32:31] (03CR) 10BBlack: [C: 032] Zero: to unified 401-01, new unified 631-02 [operations/puppet] - 10https://gerrit.wikimedia.org/r/155492 (owner: 10Yurik) [02:33:46] RECOVERY - mailman_ctl on sodium is OK: PROCS OK: 1 process with UID = 38 (list), regex args /mailman/bin/mailmanctl [02:34:17] RECOVERY - mailman_qrunner on sodium is OK: PROCS OK: 8 processes with UID = 38 (list), regex args /mailman/bin/qrunner [02:35:25] thx bblack ! [02:35:59] !log LocalisationUpdate completed (1.24wmf17) at 2014-08-21 02:34:56+00:00 [02:36:04] Logged the message, Master [02:37:01] part of the problem above (sodium) seems to be that the amanda backup job kills i/o perf [02:37:04] maybe we could ionice that? [02:38:17] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:07] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.003 second response time [02:40:37] PROBLEM - Number of mediawiki jobs running on tungsten is CRITICAL: CRITICAL: Anomaly detected: 35 data above and 0 below the confidence bounds [02:40:37] PROBLEM - Number of mediawiki jobs queued on tungsten is CRITICAL: CRITICAL: Anomaly detected: 35 data above and 0 below the confidence bounds [02:50:56] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [02:55:16] RECOVERY - ElasticSearch health check on elastic1006 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2017: active_shards: 6050: relocating_shards: 3: initializing_shards: 0: unassigned_shards: 0 [03:21:53] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 21 03:20:47 UTC 2014 (duration 20m 46s) [03:21:59] Logged the message, Master [04:00:56] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Thu 21 Aug 2014 02:00:10 UTC [04:51:56] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [05:19:37] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Thu Aug 21 05:19:32 UTC 2014 [06:28:07] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 2 failures [06:28:37] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:07] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:26] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 3 failures [06:29:36] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:36] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 3 failures [06:45:06] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:45:07] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:45:26] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:45:37] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:46:36] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:47:07] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:36] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:52:56] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [06:52:56] PROBLEM - Disk space on ms1004 is CRITICAL: DISK CRITICAL - free space: / 685 MB (3% inode=94%): /var/lib/ureadahead/debugfs 685 MB (3% inode=94%): [07:05:07] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:15:36] good morning [08:41:26] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Epic puppet fail [08:43:46] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Epic puppet fail [08:53:56] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [09:00:26] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [09:02:46] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:54:54] (03CR) 10Filippo Giunchedi: "yep that's correct!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/155244 (owner: 10Filippo Giunchedi) [10:00:51] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift-drive-audit: import icehouse version [operations/puppet] - 10https://gerrit.wikimedia.org/r/155244 (owner: 10Filippo Giunchedi) [10:01:05] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] drive-audit: clear up exit status [operations/puppet] - 10https://gerrit.wikimedia.org/r/155245 (owner: 10Filippo Giunchedi) [10:02:46] !log Jenkins installed plugin [https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin Throttle Concurrent Builds]. [10:02:51] Logged the message, Master [10:13:34] lunch break [10:13:47] if Zuul / Jenkins is broken, just restart Jenkins on gallium.wikimedia.org [10:13:50] but it should be fine [10:19:56] (03CR) 10Filippo Giunchedi: "btw this is now https://bugs.launchpad.net/swift/+bug/1359664" [operations/puppet] - 10https://gerrit.wikimedia.org/r/155245 (owner: 10Filippo Giunchedi) [10:20:47] ack hexmode [10:20:50] nope, hashar [10:29:09] just curious.. are mediawiki devs and/or wmf sysadmins able to view an individual user's notification logs (Special:Notifications)? [10:32:10] (03PS1) 10Filippo Giunchedi: filippo: use bashrc, fix prompt [operations/puppet] - 10https://gerrit.wikimedia.org/r/155518 [10:32:23] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] filippo: use bashrc, fix prompt [operations/puppet] - 10https://gerrit.wikimedia.org/r/155518 (owner: 10Filippo Giunchedi) [10:45:04] (03PS1) 10Filippo Giunchedi: swift: raise container availability thresholds [operations/puppet] - 10https://gerrit.wikimedia.org/r/155522 [10:46:42] (03CR) 10Nikerabbit: Crreot another file reference from I11b85c87a (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154999 (owner: 10Ori.livneh) [10:51:25] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: raise container availability thresholds [operations/puppet] - 10https://gerrit.wikimedia.org/r/155522 (owner: 10Filippo Giunchedi) [10:54:56] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [11:06:07] (03CR) 10Matanya: [C: 031] Add ferm::service rule for zookeeper admin port [operations/puppet] - 10https://gerrit.wikimedia.org/r/153801 (owner: 10Ottomata) [11:10:29] (03CR) 10Matanya: [C: 04-1] "Please remove templates/misc/email-blog-pageviews.erb as well." [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 (owner: 10Dzahn) [11:10:58] (03CR) 10Matanya: [C: 031] puppetmaster Apache template - retab [operations/puppet] - 10https://gerrit.wikimedia.org/r/153987 (owner: 10Dzahn) [11:11:11] (03PS2) 10Mark Bergsma: Allocate codfw public/private subnets for rows A-D [operations/dns] - 10https://gerrit.wikimedia.org/r/155266 [11:11:56] (03CR) 10Mark Bergsma: [C: 032] Allocate codfw public/private subnets for rows A-D [operations/dns] - 10https://gerrit.wikimedia.org/r/155266 (owner: 10Mark Bergsma) [11:15:37] (03PS1) 10Mark Bergsma: Add management IPs for asw-[a-d]-codfw [operations/dns] - 10https://gerrit.wikimedia.org/r/155524 [11:16:11] (03CR) 10Mark Bergsma: [C: 032] Add management IPs for asw-[a-d]-codfw [operations/dns] - 10https://gerrit.wikimedia.org/r/155524 (owner: 10Mark Bergsma) [11:16:26] (03CR) 10Matanya: [C: 031] delete blog SSL certificates [operations/puppet] - 10https://gerrit.wikimedia.org/r/153228 (owner: 10Dzahn) [11:16:52] godog: hi, do you plan to take another look at https://bugzilla.wikimedia.org/show_bug.cgi?id=69760 at some point? [11:17:20] (03CR) 10Matanya: [C: 031] salt - minion.erb - fix compiler warnings [operations/puppet] - 10https://gerrit.wikimedia.org/r/154347 (owner: 10Dzahn) [11:20:36] godog/andre__: Can GWT cause such problems? [11:20:43] (03CR) 10Matanya: [C: 031] sudoers.erb - deprecated variable access [operations/puppet] - 10https://gerrit.wikimedia.org/r/154372 (owner: 10Dzahn) [11:21:31] (03CR) 10Matanya: [C: 031] "just a comment, but oh well." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154373 (owner: 10Dzahn) [11:30:15] PROBLEM - nutcracker port on mw1178 is CRITICAL: CRITICAL - Socket timeout after 2 seconds [11:37:44] PROBLEM - check if dhclient is running on mw1178 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:37:45] PROBLEM - nutcracker process on mw1178 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:38:05] PROBLEM - SSH on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:35] PROBLEM - puppet last run on mw1178 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:38:35] PROBLEM - RAID on mw1178 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:39:44] PROBLEM - Apache HTTP on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:05] PROBLEM - NTP on mw1178 is CRITICAL: NTP CRITICAL: No response from NTP server [11:48:24] PROBLEM - Host mw1178 is DOWN: PING CRITICAL - Packet loss = 100% [11:58:34] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [12:53:18] (03PS1) 10Mark Bergsma: Typo [operations/dns] - 10https://gerrit.wikimedia.org/r/155531 [12:53:43] (03CR) 10Mark Bergsma: [C: 032] Typo [operations/dns] - 10https://gerrit.wikimedia.org/r/155531 (owner: 10Mark Bergsma) [12:55:24] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [13:07:10] andre__: AFAICT it is working as expected after resolving the load issue, there's indeed still followup on how to detect if it happens again, I've pushed some nagios changes today to the swift thresholds [13:08:16] godog: ah, thanks! I'm tempted to copy that line to the bug report for the sake of communicating "yes, this is still on the screen", if you don't plan to comment there yourself :) [13:09:21] andre__: yep let me update the bug [13:09:27] thanks [13:37:45] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.0633333333333 [13:51:44] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 1 failures [14:01:30] looks like the Swift bug is back :) [14:03:04] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Epic puppet fail [14:04:04] PROBLEM - Disk space on ms1004 is CRITICAL: DISK CRITICAL - free space: / 712 MB (3% inode=94%): /var/lib/ureadahead/debugfs 712 MB (3% inode=94%): [14:08:03] PierreSelim: do you have deleted via ajax quick delete - (at) api error msg? [14:08:34] yes [14:08:44] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:08:46] I can try directly but I guess it will be the same [14:10:03] yes the same. [14:12:53] godog: it is not a new problem, one year ago there was teh same but (but not affected a lot of files. i can't find the bug, was closed as resolved :/ [14:13:28] (03PS1) 10Filippo Giunchedi: elasticsearch: increase ganglia timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/155541 [14:14:06] ^d: ^ [14:14:39] (03CR) 10Chad: [C: 031] "lgtm, especially when we want to know what graphs actually say :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/155541 (owner: 10Filippo Giunchedi) [14:15:05] Steinsplitter: ack, sorry I'm in the middle of something now, anyways then it is a different bug than what was causing the problems yesterday [14:15:16] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] elasticsearch: increase ganglia timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/155541 (owner: 10Filippo Giunchedi) [14:15:56] for sure it's not new [14:17:44] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [14:22:04] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:24:03] (03PS1) 10Ottomata: kafkatee should consume from all partitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/155543 [14:24:14] (03PS2) 10Ottomata: kafkatee should consume from all partitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/155543 [14:24:27] (03CR) 10Ottomata: [C: 032 V: 032] kafkatee should consume from all partitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/155543 (owner: 10Ottomata) [14:25:15] PROBLEM - Unmerged changes on repository puppet on virt0 is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [14:30:13] (03PS16) 10Andrew Bogott: mailman: use a new default theme (prettier mailman) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154964 (https://bugzilla.wikimedia.org/61283) (owner: 10John F. Lewis) [14:30:41] (03PS1) 10Hashar: contint: phpcs mw standard on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/155544 (https://bugzilla.wikimedia.org/64858) [14:31:38] (03CR) 10Andrew Bogott: [C: 032] mailman: use a new default theme (prettier mailman) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154964 (https://bugzilla.wikimedia.org/61283) (owner: 10John F. Lewis) [14:35:11] (03CR) 10Hashar: [C: 031] "Cherry picked on contint puppetmaster." [operations/puppet] - 10https://gerrit.wikimedia.org/r/155544 (https://bugzilla.wikimedia.org/64858) (owner: 10Hashar) [14:35:31] andrewbogott: thanks for the merge, how long do you think puppet would take to run? [14:36:08] Thehelpfulone: everything should be updated after 30 mins [14:36:38] !log Jenkins: updating mediawiki code sniffer repo bf82117..bc4e590 [14:36:45] Logged the message, Master [14:38:16] andrewbogott: Thanks! :) [14:50:14] PROBLEM - mailman list info on sodium is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string About Wikimedia-l not found on https://lists.wikimedia.org:443/mailman/listinfo/wikimedia-l - 15083 bytes in 0.107 second response time [14:53:11] (03PS1) 10Ottomata: Add cron job to drop old data in HDFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/155549 [14:53:28] (03PS2) 10Ottomata: Add cron job to drop old data in HDFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/155549 [14:53:55] !log Jenkins: updated PHP CodeSniffer MediaWiki standard on all slaves. [14:54:01] Logged the message, Master [14:56:24] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [14:58:20] hm, mailman HTTP cirital? [14:58:24] Thehelpfulone ^^ [14:58:38] hmm? [14:58:47] I'm looking at icinga now [15:03:05] RECOVERY - Unmerged changes on repository puppet on virt0 is OK: No changes to merge. [15:03:07] andrewbogott: check Sodium's logs please (or anyone which access) [15:03:41] Mailman hit a bug and I don't understand why with regards to the patch deployed above [15:06:18] Still no jouncing. But nothing requested for SWAT this morning. [15:08:00] JohnLewis: someone merged a change to prettify mailman [15:08:12] seems like it broke the check [15:08:13] matanya: I know; it's my patch [15:08:24] but the site is ok [15:08:26] Mailman admin interfaces are no longer working [15:08:55] Thehelpfulone: can you log in ? [15:09:23] where I'm already logged in that's ok [15:09:24] https://lists.wikimedia.org/mailman/admin/wikimedia-l [15:09:28] Bug in Mailman version 2.1.13 [15:09:28] We're sorry, we hit a bug! [15:09:29] Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs. [15:09:31] :S [15:09:52] * JohnLewis gets to test that patch on labs [15:11:32] Thehelpfulone: let's hope puppet works on labs this time :/ [15:13:41] JohnLewis: I'm back, looking... [15:13:58] andrewbogott: i see why the alert is raised [15:14:07] no "About Wikimedia-l" test on the page [15:14:21] i can adjust the check, would that be useful ? [15:14:41] matanya: not really because that won't fix the admin interface :) [15:14:53] but it will fix the nagios alert [15:15:02] matanya: feel free then I guess [15:15:09] Except the nagios alert is correct, isn't it? [15:15:13] Or are they somehow unrelated? [15:15:15] it is not [15:15:18] JohnLewis: what should I look for on sodium? [15:15:24] Ah, ok, then feel free to fix :) [15:15:33] the nagios alert checks to see the web ui is there [15:15:53] by searching for a string on the web page [15:15:59] JohnLewis: IOError: [Errno 2] No template file found: 'verify.txt' [15:15:59] which was removed [15:16:12] Hm? [15:16:44] andrewbogott: can you grep that in the templates directory and see where it exists? [15:16:59] I think it might be an issue it doesn't exist in some directories and I think I might know which ones [15:17:14] JohnLewis: which templates dir? [15:17:32] And, ok, just so we're on the same page... [15:17:33] /var/lib/mailman/ and there should be a 'templates' directory [15:17:40] grep there :) [15:17:44] we don't want wikimedia-l to be removed, right? so the fact that it's missing, that's the problem? [15:18:11] root@sodium:/var/lib/mailman/templates# grep -ir wikimedia-l * [15:18:12] root@sodium:/var/lib/mailman/templates# [15:18:33] (03PS1) 10Matanya: mailman: search for a new string on new web ui [operations/puppet] - 10https://gerrit.wikimedia.org/r/155563 [15:18:35] oh, sorry, verify.txt [15:18:58] no reference to that either. [15:20:03] andrewbogott: that patch should fix the nagios alert [15:20:19] andrewbogott: no where at all? [15:20:24] * JohnLewis checks old labs instance [15:20:42] root@sodium:/var/lib/mailman/templates# grep -ir verify.txt * [15:20:43] root@sodium:/var/lib/mailman/templates# find . -name verify.txt [15:20:43] root@sodium:/var/lib/mailman/templates# [15:23:18] verify.txt does not exist on labs either [15:23:48] Shall I just revert the theming patch? Or do you have some idea as to what's happening? [15:24:50] JohnLewis: the failures are all over. Like, most recently No template file found: 'postheld.txt' [15:25:06] No template file found: 'article.html' [15:25:09] I think I know what is wrong [15:25:10] etc [15:25:55] yeah I know what is wrong [15:26:08] andrewbogott: is sodium backed up anywhere? [15:26:15] ... [15:26:29] I would prefer you not ask things like that! [15:26:29] actually; any mailman thing will do actually [15:26:37] I don't know. [15:26:45] Um… mchenry used to host mailman I think. [15:26:52] I dont' know if it still exists. [15:27:08] this is why you guys need to check puppet stuff [15:27:38] I think the bit I added to puppet overwrites the whole directory removing all files.. [15:28:29] uh ohhh, what's happenin? [15:28:56] recurse remote, you mean? [15:28:56] andrewbogott: look at the puppet part I wrote and see if it does that, I don't think it did? [15:30:07] andrewbogott: listinfo.html needed to be added to the directories overwriting the listinfo, what happened here is it added the file and delete *everything* else in the directory [15:30:36] docs say "Allows copying of a few files into a directory containing many unmanaged files without scanning all the local files. " which sounds like what you wanted. [15:30:40] Not that it seems to have done that... [15:30:51] Yes [15:30:58] and it didn't apparently. Hm. [15:31:16] At least we've identified the issue and what casued it. Now we need to resolve it manually and automatically. [15:31:59] lemme verify that that's happening... [15:32:05] please [15:32:16] JohnLewis: this is why you guys need to check puppet stuff <-- please don't blame others for your mistake :) [15:32:43] andrewbogott: i think this is because templates was previsouly a symlink [15:32:46] i think we're ok [15:32:56] ottomata: oh? What was it a link to? [15:32:58] matanya: :) [15:33:05] /etc/mailman [15:33:10] ls /etc/mailman/en [15:33:16] so, the puppet change did what it was suppsoed to do [15:33:25] but it changed /var/lib/mailmain/templates to a directory [15:33:30] and then recurse copied the stuff from puppet there [15:33:36] all the old templates are still in /etc/mailman [15:33:47] ok, so a proper fix is to point that patch at /etc/mailman instead of /var/lib/mailman/templates [15:33:50] matanya: wasn't my mistake though; something was undocumented that I didn't know abvout or missed ;) [15:34:01] And switch things back to a symlink, presumably by hand [15:34:10] or via puppet, wouldn't hurt [15:34:13] ensure => link [15:34:14] andrewbogott / ottomata: Want to make a patch reversing it now? [15:34:20] yep, hang on [15:34:28] andrewbogott: i would use a link by puppet [15:34:51] verified: A local file in that dir is /not/ clobbered by recurse. [15:36:15] RECOVERY - mailman list info on sodium is OK: HTTP OK: HTTP/1.1 200 OK - 10511 bytes in 0.116 second response time [15:37:48] (03PS1) 10Andrew Bogott: /var/lib/mailman/templates is a link to /etc/mailman [operations/puppet] - 10https://gerrit.wikimedia.org/r/155564 [15:38:11] matanya, ottomata, happy with that? [15:38:58] (03CR) 10Matanya: [C: 031] /var/lib/mailman/templates is a link to /etc/mailman [operations/puppet] - 10https://gerrit.wikimedia.org/r/155564 (owner: 10Andrew Bogott) [15:39:12] (03CR) 10John F. Lewis: [C: 031] "Perfect fix to my stupidity :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/155564 (owner: 10Andrew Bogott) [15:39:59] ottomata: out of interest; was this symlink documented anywhere? :) [15:40:14] i guess in ls -la [15:40:18] Yeah, given that that symlink wasn't puppetized, I don't think there was any way for you to know. [15:40:19] i think that's fine, andrewbogott, i'd add a comment explicitly saying what the intention here is. the recurse => remote isn't very obvious [15:40:28] JohnLewis: dunno, doubtful [15:40:44] the only reason I knew it was a symilnk was because you had me look into this the other day [15:40:56] and puppet logs :) [15:40:56] Aug 21 14:48:01 sodium puppet-agent[3598]: (/Stage[main]/Mailman::Web-ui/File[/var/lib/mailman/templates]/ensure) ensure changed 'link' to 'directory' [15:41:07] :p [15:41:09] ideally someone should have manually run puppet and watched the logs whne they merged that change [15:41:22] ottomata: recommented [15:41:24] (03PS2) 10Andrew Bogott: /var/lib/mailman/templates is a link to /etc/mailman [operations/puppet] - 10https://gerrit.wikimedia.org/r/155564 [15:41:33] ottomata: yeah, that would've been me. [15:41:59] s'ok ! looks good cept: [15:42:00] Recurse => remove [15:42:29] running in the puppet compiler would reveal that too [15:42:53] (03PS1) 10Reedy: Add/update symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155566 [15:42:55] (03PS1) 10Reedy: testwiki to 1.24wmf18 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155567 [15:42:57] (03PS1) 10Reedy: Wikipedias to 1.24wmf17 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155568 [15:42:59] (03PS1) 10Reedy: group0 to 1.24wmf18 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155569 [15:43:06] eek [15:43:12] andrewbogott: it is recurse -> remote [15:43:17] not remove [15:43:18] (03CR) 10Reedy: [C: 032] Add/update symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155566 (owner: 10Reedy) [15:43:22] (03Merged) 10jenkins-bot: Add/update symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155566 (owner: 10Reedy) [15:43:29] (03PS3) 10Andrew Bogott: /var/lib/mailman/templates is a link to /etc/mailman [operations/puppet] - 10https://gerrit.wikimedia.org/r/155564 [15:46:12] (03CR) 10Ottomata: [C: 032] /var/lib/mailman/templates is a link to /etc/mailman [operations/puppet] - 10https://gerrit.wikimedia.org/r/155564 (owner: 10Andrew Bogott) [15:46:33] andrewbogott: want to merge and run puppet? (i made a copy of /etc/mailman in /tmp/ just in case!) [15:46:43] heh,I copied it too :) [15:46:52] I'll merge [15:48:46] (03PS3) 10Ottomata: Add cron job to drop old data in HDFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/155549 [15:50:15] PROBLEM - mailman list info on sodium is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string About Wikimedia-l not found on https://lists.wikimedia.org:443/mailman/listinfo/wikimedia-l - 15083 bytes in 0.192 second response time [15:50:46] andrewbogott: if you can please merge my change to fix this ^ [15:51:11] matanya: which one? [15:51:20] mailman worked and now has gone back to the bug issue [15:51:26] https://gerrit.wikimedia.org/r/155563 [15:52:21] ottomata: ^ [15:53:04] PROBLEM - mailman_ctl on sodium is CRITICAL: PROCS CRITICAL: 0 processes with UID = 38 (list), regex args /mailman/bin/mailmanctl [15:53:43] so… JohnLewis, Thehelpfulone, happy with how things look now? [15:54:24] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Puppet has 1 failures [15:54:37] (03CR) 10Andrew Bogott: [C: 032] mailman: search for a new string on new web ui [operations/puppet] - 10https://gerrit.wikimedia.org/r/155563 (owner: 10Matanya) [15:54:39] andrewbogott: yeah :) [15:54:48] andrewbogott: we only had to kill mailman to get it ;) [15:55:04] RECOVERY - mailman_ctl on sodium is OK: PROCS OK: 1 process with UID = 38 (list), regex args /mailman/bin/mailmanctl [15:55:24] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:00:18] RECOVERY - mailman list info on sodium is OK: HTTP OK: HTTP/1.1 200 OK - 15083 bytes in 0.099 second response time [16:03:57] thanks andrewbogott [16:04:10] thank you, and ottomata. [16:13:40] (03PS1) 10Mark Bergsma: Add access switch stack asw-d-codfw to RANCID [operations/puppet] - 10https://gerrit.wikimedia.org/r/155574 [16:15:05] JohnLewis: want to try again with the announcement email? ;) [16:15:18] Thehelpfulone: was it not queued? [16:15:26] nope didn't come through [16:15:27] Thehelpfulone: are you also going to privately email the list owners please? [16:15:39] Nemo_bis: yes, to every list [16:15:49] that's what the announcement email is [16:16:08] Thehelpfulone: every list members? :o [16:16:17] every list -owner email address [16:16:24] ok; owners is enough [16:16:56] sent [16:17:28] 'pending moderator approval' Thehelpfulone: just give me the password to the list :p [16:17:29] (03CR) 10Mark Bergsma: [C: 032] Add access switch stack asw-d-codfw to RANCID [operations/puppet] - 10https://gerrit.wikimedia.org/r/155574 (owner: 10Mark Bergsma) [16:17:51] you forgot to cc me ;) [16:17:59] on you did nvm :p [16:18:35] sent [16:19:05] JohnLewis: so now that it's updated, the list info pages should be able to be updated relatively easily right (I'm thinking of move of the translate link to the sidebar) [16:19:30] hmm, This is a closed translation request. [16:19:40] ok so maybe we remove that link altogether then? Nemo_bis [16:19:47] RECOVERY - Disk space on elastic1005 is OK: DISK OK [16:21:37] Thehelpfulone: dunno; it's your only attribution to translators isn't it [16:21:46] yeah [16:23:31] why did we close it Nemo_bis? are we actually done? [16:23:39] I thought we were going to do another sync in a few weeks? [16:23:44] ohnoes this discussion again [16:23:59] if you are, say so on talk and re-enable :) [16:24:42] heh [16:26:01] hmm which languages were they again [16:27:08] JohnLewis: got 2 of them [16:27:16] Nemo_bis: is there a list of the lang codes for the ones Vogone restricted to? Arabic, Catalan, Czech, Danish, German, English, Spanish, Estonian, Basque, Persian, Finnish, French, Hindi, Croatian, Hungarian, Interlingua, Indonesian, Italian, Japanese, Korean, Lithuanian, Norwegian Bokmål, Dutch, Polish, Portuguese, Brazilian Portuguese, Romanian, Russian, Slovenian, Serbian, Swedish, Tamil, Turkish, Ukrainian, Vietnamese, [16:27:16] Chinese (China), Simplified Chinese, Traditional Chinese and Chinese (Taiwan) [16:30:38] Thehelpfulone: probably in the json or db representation of the log [16:37:07] ganglia question: recently the elasticsearch cluster has seen choppy ganglia stats, like no stats at all http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Elasticsearch%20cluster%20eqiad&m=disk_free&r=4hr&s=by%20name&hc=3&mc=2&st=1408638854&g=load_report&z=large [16:37:44] it looks like some sort of packet loss where [16:37:57] where multicast data wouldn't make it to the aggregator from the nodes [16:38:12] did we see this before? [16:43:34] hm, godog, i've not seen that happen on the base ganglia metrics before, [16:43:47] i've had problems with custom stats disappearing from time to time [16:45:26] andrewbogott: so the one thing that's come up so far is that the templates aren't saved as UTF-8 https://lists.wikimedia.org/mailman/listinfo/wikimedia-ru [16:45:53] which is causing problems with Russian for example [16:46:00] Yeah, that seems serious. [16:46:07] do you know how we can re-save them as UTF-8? [16:46:12] ottomata: ye there's custom stats involved, it could be gmond too, I'll keep diggin thanks ! [16:46:31] Thehelpfulone: When you say 'the templates' you mean the new ones that were just added? [16:46:40] I think so yes [16:47:00] Most likely the files are correct and mailman just doesn't know how to interpret them [16:47:27] might be that you just need a flag or mark at the top of the files... [16:48:04] that template file has up top [16:49:06] Thehelpfulone: could it be that the new theme uses a new font? [16:49:36] yeah I added that andrewbogott but apparently just because it has that it doesn't mean the html documents /are/ actually UTF-8 [16:50:14] indeed it doesn't [16:56:47] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [17:13:42] (03CR) 10Ottomata: "Ok, seems fine. Why does the CNAME not work though?" [operations/dns] - 10https://gerrit.wikimedia.org/r/154222 (https://bugzilla.wikimedia.org/44731) (owner: 10Jeremyb) [17:17:05] godog: you are assigned on this ticket [17:17:06] https://rt.wikimedia.org/Ticket/Display.html?id=8071 [17:17:18] are you planning on doing something with it? [17:18:22] ottomata: yep, likely next week tho [17:22:27] ok cool [17:22:28] no probs [17:24:06] (03PS1) 10Cmjohnson: Removing mw1130 from dsh files to replace disk and re-install [operations/puppet] - 10https://gerrit.wikimedia.org/r/155584 [17:28:55] !log removing mw1130 from pybal [17:29:01] Logged the message, Master [17:30:24] (03CR) 10Cmjohnson: [C: 032] Removing mw1130 from dsh files to replace disk and re-install [operations/puppet] - 10https://gerrit.wikimedia.org/r/155584 (owner: 10Cmjohnson) [17:39:07] PROBLEM - Disk space on ms1004 is CRITICAL: DISK CRITICAL - free space: / 712 MB (3% inode=94%): /var/lib/ureadahead/debugfs 712 MB (3% inode=94%): [18:13:09] (03PS2) 10Dzahn: remove blog.wikmedia.org related things [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 [18:15:35] (03PS3) 10Dzahn: remove blog.wikmedia.org related things [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 [18:17:22] no deploy? [18:19:52] (03CR) 10Ottomata: "Cool, cronjob will need removed from stat1003 too. I assume Tilman knows about this? I think he was the one who originally wanted those " [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 (owner: 10Dzahn) [18:21:53] (03CR) 10Dzahn: "well, i did not speak to Tilman about this specific patch, but i'm quite sure Tilman knows we don't host the blog anymore and either he is" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 (owner: 10Dzahn) [18:22:44] (03CR) 10Dzahn: [C: 031] "maybe RobH can review this too being involved in a lot of blog things in the past" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 (owner: 10Dzahn) [18:23:49] (03CR) 10Dzahn: "also see Change-Id: I682cb816e08d8" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 (owner: 10Dzahn) [18:27:57] !log trying to recover from weird Elasticsearch upgrade failure by redoing the upgrade on one node while also blowing away the data directory during the upgrade. elastic1005, you are my first victem. [18:28:03] Logged the message, Master [18:28:11] !log *victim* [18:28:17] Logged the message, Master [18:32:47] (03CR) 10Anomie: [C: 04-1] "I see no evidence of support for this change at this time. See https://bugzilla.wikimedia.org/show_bug.cgi?id=67709#c8 for details." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154408 (https://bugzilla.wikimedia.org/67709) (owner: 10Jforrester) [18:43:06] (03PS1) 10Aude: Use new Wikibase serialization format on test wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155596 [18:45:11] robh: let's delete the old paging script for the usb device? https://gerrit.wikimedia.org/r/#/c/153227/ [18:46:30] (03CR) 10Dzahn: "Ori, should class { 'apache': be completely removed in this one?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153843 (owner: 10Dzahn) [18:57:47] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed 20 Aug 2014 22:49:01 UTC [19:02:25] Reedy: ^d no deploy? [19:03:02] <^d> I wasn't asked about today. [19:03:03] <^d> Hmm [19:03:05] jouncebot: !next [19:03:12] ok [19:03:22] no hurry [19:03:38] jouncebot is broken for three days now [19:03:40] or four [19:04:20] schedule a meta deployment of jouncebot [19:04:49] <^d> !jouncebot schedule your own upgrade [19:05:40] Guessing based on the SAL no one has attempted my deploy ;) [19:06:05] RoanKattouw: You should really stop going in via fenari [19:06:09] Shouldn't you be using iron? :) [19:06:35] Yes, I should be :) [19:06:41]