[00:05:50] PROBLEM - Puppet freshness on copper is CRITICAL: Puppet has not run in the last 10 hours [00:05:50] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [01:15:33] New patchset: Lcarr; "Adding in sun JDK to contint servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1591 [01:15:44] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/1591 [01:15:50] can I get a what, what? (maybe maplebed ? ) https://gerrit.wikimedia.org/r/1591 [01:18:51] New patchset: Lcarr; "Adding in sun JDK to contint servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1591 [01:19:03] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1591 [01:20:36] fyi it's for rt.wikimedia.org/Ticket/Display.html?id=2147 [01:30:32] New patchset: Lcarr; "Adding in sun JDK to contint servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1591 [01:37:08] New patchset: Lcarr; "Adding in sun JDK to contint servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1591 [01:37:18] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/1591 [01:39:38] New patchset: Lcarr; "Adding in sun JDK to contint servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1591 [01:39:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1591 [01:40:41] LeslieCarr: we need libswt-gtk-3.5-java too I think [01:41:15] http://rt.wikimedia.org/Ticket/Display.html?id=2147 [01:42:08] New patchset: Lcarr; "Adding in sun JDK to contint servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1591 [01:42:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1591 [01:44:39] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1591 [01:44:46] LeslieCarr: we might need ia32-libs as well [01:45:02] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1591 [01:45:02] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1591 [01:45:44] LeslieCarr: we also need to export ANDROID_SWT= [01:46:05] preilly: just make sure to put it all in the ticket :) [01:57:48] preilly: ping? [01:58:24] 15 02:39:02 < jeremyb> someone have a public link to info about last night's mobile deploy? [01:58:24] he was here just a second ago. [01:58:27] 15 02:39:21 < jeremyb> specifically the breakage with desktop logins going to m. [01:59:26] jeremyb: ? [01:59:36] preilly: see my paste ^^ ? [01:59:45] jeremyb: from two night ago? [01:59:48] (2 nights ago now) [01:59:49] yes [02:00:09] jeremyb: it was just an issue the a redirect hook firing on non mobile logins [02:00:25] jeremyb: it was fixed by adding a check for the mobile only header [02:01:08] preilly: well, was there a mail in pipermail or a !log or a bug (or even RT) or some other thing i'm not thinking of that mentioned either the breakage or the fact that it had been fixed? [02:01:33] jeremyb: there was an email [02:01:49] jeremyb: I'm leaving the office now [02:01:52] preilly: well, there was the internal one, but one i can link to for the public? [02:01:56] sure [03:11:23] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [03:12:14] PROBLEM - SSH on maerlant is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:27] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [03:32:26] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [04:44:04] RECOVERY - SSH on maerlant is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [04:54:51] PROBLEM - Recursive DNS on 91.198.174.6 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:37:55] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours [07:33:37] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: Puppet has not run in the last 10 hours [07:33:37] PROBLEM - Puppet freshness on ssl2 is CRITICAL: Puppet has not run in the last 10 hours [07:33:37] PROBLEM - Puppet freshness on ssl3004 is CRITICAL: Puppet has not run in the last 10 hours [08:02:14] sigh, spence come on [08:02:30] have you notived the ganglia graphs for the past day? [08:02:47] i am looking at top right now [08:03:01] have you noticed the ganglia graphs for the cluser as a whole for the last day? [08:03:09] *cluster [08:03:13] oh [08:03:55] trying to open it, getting refused right now [08:04:01] yeah look slike it [08:04:09] well much of the last half of the day is missing data [08:04:26] when I checked a few minutes ago it was back to graphing, so I couldn't fix anything [08:06:35] would you think "ganglia_parser" is part of the problem? [08:07:52] no idea, can't get on the box [08:09:17] it's rsyncing stuff: /usr/bin/rsync -a /mnt/ganglia_tmp/rrds.pmtpa/ /var/lib/ganglia/rrds.pmtpa [08:09:37] and quite a bit of /usr/bin/python -tt /usr/sbin/ganglia_parser [08:09:57] that rsync needs to happen [08:10:01] once an hour or something doesn't it? [08:10:13] otherwise if ganglia restarts we lose the data [08:10:23] are there many of those? [08:10:44] yea, just wondering what makes it so slow [08:11:33] i'm checking, but it takes patience ... [08:12:35] 20 procs with string "ganglia" [08:13:27] 17 of them ganglia_parser, user nagios runs it [08:18:58] !log spence slow again, side-note: tried to use "sar" to investigate but "Please check if data collecting is enabled in /etc/default/sysstat" (want to?) [08:19:08] Logged the message, Master [08:23:46] load's dropping some but [08:23:59] still a ton of swap used [08:24:15] notifications.cgi and history.cgi use a lot of memory [08:24:23] yes indeed [08:24:27] makes me want to restart nagios again [08:24:30] k? [08:24:34] yes, please do [08:24:42] I would stop it [08:24:49] leave it stopped for 5 min, watch swap [08:24:53] restart [08:26:03] !log spence - almost unusable, Nagios notifications.cgi and history.cgi use a lot of memory, stopping Nagios, watching swap [08:26:12] Logged the message, Master [08:26:39] Warning - nagios did not exit in a timely manner [08:26:47] that's life [08:27:19] finally I am starting to see swap decrease, if slowly [08:29:33] killed 2 more procs, notifications.cgi [08:29:47] oh it was still going? heh [08:29:48] hrmm, let me also kill history.cgi [08:29:57] thank you [08:30:18] load avg going down [08:30:33] there were a couple of "noficiations", you got em both? [08:30:40] now "rsync" is the top CPU user [08:30:56] shouldn't take too long to finish [08:30:58] yes, got all notifications.cgi [08:31:22] no "cgi" at all anymore [08:32:47] ganglia_parser, run by user nagios, still doing stuff and didnt touch it [08:33:06] mmm [08:33:39] fine [08:37:32] so what I don't know is whether these other jobs owned by nagios ought to be shot before it gets restarted [08:37:34] mutante: hi :) so spence is dieting out because Nagios eats all memory? :-) [08:37:49] I don't know about the nsca stuff either [08:37:53] yep [08:38:09] so we're forcing it to take a little break and give back all the swap it ate [08:38:19] i wouldnt blame nsca, it has been running, didnt kill it, not using a lot [08:38:24] you probably have to many checks by now [08:38:26] and spence is much faster now [08:38:30] no, I don't blame those processes [08:38:53] I just wonder if they will be "left over " processes and therefore a problem when nagios is restarted [08:39:22] or you already restarted it? [08:39:26] or.... puppet restarted it? [08:39:37] NSCA is a demon which let you receive "passive checks" [08:39:44] hashar: hi, yeah, notifications.cgi [08:39:51] I see [08:39:58] apergos: 5 minutes over.. starting again now..hm [08:39:58] you can get a script feeding nagios with check monitoring [08:40:01] ok [08:40:10] for example giving it radius accounting or SNMP traps [08:40:17] well I see some puppet freshness checks in there already [08:40:34] I think puppet send SNMP traps [08:40:41] ok [08:40:47] they are then probably send back in NSCA for nagios processing [08:40:54] (all of that is pure supposition from me) [08:41:00] heh [08:41:03] (btw: nagios sucks) [08:41:05] ;-D [08:41:28] !log spence - had killed additional notifications.cgi and history.cgi procs, waited 5 minutes, load went down a lot, restarting nagios [08:41:37] Logged the message, Master [08:41:55] hashar: NSCA receives UDP packages [08:42:05] if we see it climb a huge amount again (shortly) this is going to prolly mean restarting the host [08:42:07] hashar: and they include the check results [08:44:17] the daemon just sits there and waits for incoming packages which include host names and service check results forwards them to Nagios, which compares that to existing host and service definitions. and if it never receives any result it also turns it into a CRIT. [08:44:18] RECOVERY - Recursive DNS on 91.198.174.6 is OK: DNS OK: 0.228 seconds response time. www.wikipedia.org returns 91.198.174.225 [08:44:48] (yea, it's encrypted) [08:46:10] you can use "send_nsca" on a monitored host to send the results of any plugin over to the nsca host [08:47:19] I have been using NRPE for that [08:47:31] it's "the other way around" [08:47:45] but we did use NSCA to receive informations from other software [08:47:57] NRPE = Nagios master executes stuff on other hosts , NSCA = hosts send results to the master, which just needs to wait for them [08:47:58] well actually we had a trap and syslog listener [08:48:14] thats why they are "passive" checks [08:48:14] then two perl daemons reading them and formatting those messages to then send then to NSCA [08:48:34] got you [08:48:40] :-)) [08:48:48] yep, the last step is send_nsca (its in puppet;) [08:49:13] then eventually everything was dropped in favor of nagios doing only active checks and sending alarms to a collector [08:49:25] all sys logs / accounting etc were also send to a collector [08:49:40] then we had NetCool on top of that to display nice alarms :D [08:49:59] I love monitoring, that is all funny [08:49:59] some people setup 2 layers of Nagios, one master which only receives passive results, but it receives that from multiple other Nagios (sub-)masters, who do active checks on hosts [08:50:39] that is a good way to scale [08:50:40] to scale up to more than a couple hundred hosts [08:50:44] * mutante nods [08:50:52] I think that is what the team did after I left [08:51:26] several "slaves" doing checks. A controller to generates their configuration and every checks result send to two master servers [08:51:35] not sure how they manage all of that though [08:52:05] anyway, two freelancers I have met there told me to have a look at http://www.opennms.org/ [08:52:23] hrmm, it almost feels like there should be a warning "dont use history.cgi" [08:52:23] seems to be way easier to setup and it could discovers services as I understood it [08:52:48] when you make nagios look through history, that seems to be reaallly hard work [08:53:04] by default it looks up for all hosts / all services [08:53:13] plus it uses flat files iirc [08:53:14] maybe we are keeping too much of it? blind guess [08:53:42] so it probably need to load the whole history file in memory then process it and display all the nice HMTL [08:53:49] yea [08:54:20] if it really matter to anyone, they should be export to a SQL database of some sort [08:54:25] I can't remember the plugin name though [08:54:45] and then disable all the histogram / history cgi [08:55:05] check out "Icinga" [08:55:44] https://www.icinga.org/nagios/ [08:56:32] oh well, but Nagios itself also is listed as offering MySQL support [08:56:55] just remembered that Icinga has it for sure https://www.icinga.org/nagios/feature-comparison/ [08:57:20] I have not followed nagios for the last 3 years or so :D [08:57:28] so I am a bit outdated with all theses forks [08:57:40] Icinga probably worth a look [08:57:43] also noteworthy keywords on that page: LDAP, IRC [08:58:18] looks good for now [08:58:20] as I understand it icinga is a community fork. Much like X.org forked from xf86 (or something similar) [08:58:34] I dunno if nagios is doing what it needs to do but ganglia is happy, load is happy, mem use is happy [08:58:45] apergos: cool:) [08:59:10] apergos: well, i tried to use history.cgi, saw it use a lot of CPU, killed it one more time, and not hitting that button anymore for now:) [08:59:14] * apergos just used the hell out of adblock on a site [08:59:38] mutante: makesure to disable history.cgi [08:59:41] maybe you should have to give explicit permission before a site can send you swf stuff [08:59:51] the best way is probably to deny it with an Apache directive [09:00:17] hashar: but i want it to work .. and not even sure if it really is the culprit yet [09:00:45] well it is probably better to have something disabled for now than having it crash the monitoring box :-D [09:00:52] hashar: then we cant check anymore when Nagios alarms have started and stopped after an event [09:01:08] we couldn't check anyhow :-P [09:01:12] yea [09:01:23] ok [09:05:08] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [09:07:08] PROBLEM - Puppet freshness on aluminium is CRITICAL: Puppet has not run in the last 10 hours [09:07:08] PROBLEM - Puppet freshness on db1001 is CRITICAL: Puppet has not run in the last 10 hours [09:07:08] PROBLEM - Puppet freshness on db1003 is CRITICAL: Puppet has not run in the last 10 hours [09:07:08] PROBLEM - Puppet freshness on db1005 is CRITICAL: Puppet has not run in the last 10 hours [09:07:08] PROBLEM - Puppet freshness on db1002 is CRITICAL: Puppet has not run in the last 10 hours [09:09:30] mutante: on another subject. TestSwarm is running on labs now :-D [09:09:32] https://testswarm.wmflabs.org/testswarm/ [09:09:48] pushing straight to test branch without review made it easier [09:09:54] err faster [09:11:27] hey, nice to hear it is running! yay [09:12:18] test branches ftw [09:12:28] I have been running the dumps out of one now for over a year :-D [09:15:27] ideally I would need a dedicated branch [09:15:33] and have my VM puppet daemon run from it [09:15:44] IIRC it is a work in progress [09:21:42] I should look at your stuff next time I have some spare time [09:21:49] also I should get a labs project set up [09:21:56] I have an account but no project [09:22:11] (I was one of the first test accounts, you couldn't create projects then) [09:29:15] i can setup security groups and instances, but i'm still not sure where you configure which group is used on which instance [09:29:27] next step i need on my labs vm is php5-cli [09:29:31] bbiaw, short break [09:32:05] well I am going back to bed :-) have a good day! [09:35:10] sleep well [09:35:28] I'm going to get groceries while it's sunny (it was pouring buckets earlier and is supposed to again, for the weekend) [09:42:48] PROBLEM - Puppet freshness on db1025 is CRITICAL: Puppet has not run in the last 10 hours [09:42:48] PROBLEM - Puppet freshness on db18 is CRITICAL: Puppet has not run in the last 10 hours [09:42:48] PROBLEM - Puppet freshness on db25 is CRITICAL: Puppet has not run in the last 10 hours [09:42:48] PROBLEM - Puppet freshness on mw1004 is CRITICAL: Puppet has not run in the last 10 hours [09:42:48] PROBLEM - Puppet freshness on erzurumi is CRITICAL: Puppet has not run in the last 10 hours [09:42:48] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [09:42:48] PROBLEM - Puppet freshness on mw1030 is CRITICAL: Puppet has not run in the last 10 hours [09:42:49] PROBLEM - Puppet freshness on mw1045 is CRITICAL: Puppet has not run in the last 10 hours [09:42:49] PROBLEM - Puppet freshness on mw1055 is CRITICAL: Puppet has not run in the last 10 hours [09:42:50] PROBLEM - Puppet freshness on mw1100 is CRITICAL: Puppet has not run in the last 10 hours [09:42:50] PROBLEM - Puppet freshness on mw5 is CRITICAL: Puppet has not run in the last 10 hours [09:42:51] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [09:42:51] PROBLEM - Puppet freshness on search15 is CRITICAL: Puppet has not run in the last 10 hours [09:42:52] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [09:42:52] PROBLEM - Puppet freshness on srv269 is CRITICAL: Puppet has not run in the last 10 hours [09:42:53] PROBLEM - Puppet freshness on srv276 is CRITICAL: Puppet has not run in the last 10 hours [09:54:47] PROBLEM - Puppet freshness on mw1068 is CRITICAL: Puppet has not run in the last 10 hours [09:54:47] PROBLEM - Puppet freshness on mw1072 is CRITICAL: Puppet has not run in the last 10 hours [09:54:47] PROBLEM - Puppet freshness on mw1076 is CRITICAL: Puppet has not run in the last 10 hours [09:54:47] PROBLEM - Puppet freshness on mw1138 is CRITICAL: Puppet has not run in the last 10 hours [09:54:47] PROBLEM - Puppet freshness on mw36 is CRITICAL: Puppet has not run in the last 10 hours [09:54:48] PROBLEM - Puppet freshness on mw54 is CRITICAL: Puppet has not run in the last 10 hours [09:54:48] PROBLEM - Puppet freshness on search9 is CRITICAL: Puppet has not run in the last 10 hours [09:54:49] PROBLEM - Puppet freshness on srv211 is CRITICAL: Puppet has not run in the last 10 hours [09:54:49] PROBLEM - Puppet freshness on srv238 is CRITICAL: Puppet has not run in the last 10 hours [09:54:50] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [09:58:58] PROBLEM - Puppet freshness on db1026 is CRITICAL: Puppet has not run in the last 10 hours [09:58:58] PROBLEM - Puppet freshness on db1012 is CRITICAL: Puppet has not run in the last 10 hours [09:58:58] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [09:58:58] PROBLEM - Puppet freshness on db1033 is CRITICAL: Puppet has not run in the last 10 hours [09:58:58] PROBLEM - Puppet freshness on mw1046 is CRITICAL: Puppet has not run in the last 10 hours [09:58:59] PROBLEM - Puppet freshness on mw11 is CRITICAL: Puppet has not run in the last 10 hours [09:58:59] PROBLEM - Puppet freshness on mw1113 is CRITICAL: Puppet has not run in the last 10 hours [09:59:00] PROBLEM - Puppet freshness on mw1127 is CRITICAL: Puppet has not run in the last 10 hours [09:59:00] PROBLEM - Puppet freshness on srv236 is CRITICAL: Puppet has not run in the last 10 hours [09:59:01] PROBLEM - Puppet freshness on mw1149 is CRITICAL: Puppet has not run in the last 10 hours [09:59:01] PROBLEM - Puppet freshness on mw42 is CRITICAL: Puppet has not run in the last 10 hours [09:59:02] PROBLEM - Puppet freshness on mw1133 is CRITICAL: Puppet has not run in the last 10 hours [09:59:02] PROBLEM - Puppet freshness on srv263 is CRITICAL: Puppet has not run in the last 10 hours [09:59:03] PROBLEM - Puppet freshness on srv256 is CRITICAL: Puppet has not run in the last 10 hours [10:00:57] PROBLEM - Puppet freshness on mw1064 is CRITICAL: Puppet has not run in the last 10 hours [10:00:57] PROBLEM - Puppet freshness on mw1151 is CRITICAL: Puppet has not run in the last 10 hours [10:00:57] PROBLEM - Puppet freshness on mw32 is CRITICAL: Puppet has not run in the last 10 hours [10:00:57] PROBLEM - Puppet freshness on sockpuppet is CRITICAL: Puppet has not run in the last 10 hours [10:04:27] PROBLEM - Puppet freshness on db1004 is CRITICAL: Puppet has not run in the last 10 hours [10:04:27] PROBLEM - Puppet freshness on db1031 is CRITICAL: Puppet has not run in the last 10 hours [10:04:27] PROBLEM - Puppet freshness on db10 is CRITICAL: Puppet has not run in the last 10 hours [10:04:27] PROBLEM - Puppet freshness on db1039 is CRITICAL: Puppet has not run in the last 10 hours [10:04:27] PROBLEM - Puppet freshness on db44 is CRITICAL: Puppet has not run in the last 10 hours [10:04:28] PROBLEM - Puppet freshness on db33 is CRITICAL: Puppet has not run in the last 10 hours [10:04:28] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [10:21:38] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:30:27] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [10:43:20] New patchset: Dzahn; "tweak nagios to relieve spence - external command_check_interval - event_broker_options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1592 [10:47:29] New review: Dzahn; "external command check_interval was set to "-1" which is "as often as possible". (this would influen..." [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/1592 [10:49:21] New review: Dzahn; "external command check_interval was set to "-1" which is "as often as possible". (this would influen..." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1592 [10:49:21] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1592 [10:56:22] aha! "test/mediawiki/core" stuff in gerrit! [11:21:13] New patchset: Dzahn; "add techblog apache_site configs - rebased - move to /misc/blogs.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1433 [11:21:28] PROBLEM - Puppet freshness on copper is CRITICAL: Puppet has not run in the last 10 hours [11:21:28] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [11:22:24] New patchset: Dzahn; "add techblog apache_site configs - rebased - move to /misc/blogs.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1433 [11:23:21] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1433 [11:23:22] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1433 [11:28:57] New patchset: Dzahn; "move blog.wm to blogs.pp, together with techblog" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1593 [11:29:58] New patchset: Dzahn; "move blog.wm to blogs.pp, together with techblog" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1593 [11:32:06] New patchset: Dzahn; "move blog.wm to blogs.pp, together with techblog - remove duplicate apache_site in techblog after merging configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1593 [11:32:42] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1593 [11:32:42] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1593 [11:34:46] New patchset: Dzahn; "fix apache_site name of techblog" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1594 [11:35:05] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1594 [11:35:06] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1594 [11:39:28] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [11:42:54] New patchset: Dzahn; "add ocs.wikimania2009 apache_site configs - and basic puppet class - move to wikimanias.pp (wikimania special sites)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1439 [11:45:20] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1439 [11:45:21] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1439 [11:54:48] New patchset: Dzahn; "add wm09schols, wm10schols, wm10reg apache_site configs & basic puppet classes, rebase, move to wikimanias.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1435 [12:07:36] New patchset: Dzahn; "wm09schols, wm10schols, wm10reg apache_sites,rebase, move to wikimanias.pp, nicer class name structure, fix admin email" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1435 [12:09:09] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1435 [12:09:09] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1435 [12:14:13] New patchset: Dzahn; "consistent ServerAdmin email, somebody said software@ -> noc@, really for all though?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1595 [12:15:27] New patchset: Dzahn; "consistent ServerAdmin email, somebody said software@ -> noc@, really for all though?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1595 [12:28:10] New patchset: Mark Bergsma; "First step in cleaning up site.pp: folding node entries" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1596 [12:34:36] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1596 [12:39:29] New patchset: Dzahn; "merge contacts.wm into a single apache config as well, move from misc-servers to outreach.pp, fix tabs, admin email" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1597 [12:41:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1595 [12:41:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1595 [12:41:18] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1597 [12:41:19] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1597 [12:43:23] New review: Mark Bergsma; "Please fix the class name in the system_role definition" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1597 [12:44:38] New review: Mark Bergsma; "Can you rename misc/wikimanis.pp to misc/wikimania.pp, consistent with the class naming?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1435 [12:47:47] New patchset: Dzahn; "sure, fix class name in system_role and rename wikimanias.pp to wikimania.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1598 [12:50:45] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1598 [12:50:45] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1598 [12:51:08] New review: Mark Bergsma; "Perhaps I'm missing something, but how is this supposed to work then? Is SSL enabled on that vhost, ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1575 [12:56:13] mark, it seems we might be having upload speed issues again [12:56:38] oh? [12:56:52] There's a bug in BZ saying https is slow, but secure.wm.o is ok [12:57:08] Uploading from Hume was getting 40 KB/s [12:57:14] New patchset: Dzahn; "add outreachcivi apache_site configs, merge into one file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1432 [12:57:18] only https? [12:57:26] I was using HTTP [12:58:05] https://bugzilla.wikimedia.org/show_bug.cgi?id=31607 "Uploads at https://commons are slow/low-speed (0-20-40 KB/s), but fine with secure.wikimedia.org" [12:58:17] ok i'll look [12:58:39] New patchset: Dzahn; "add outreachcivi apache_site configs, merge into one file, fix tabs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1432 [12:59:16] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1432 [12:59:17] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1432 [12:59:32] Thanks [13:01:47] hmmmz indeed [13:01:48] generic-receive-offload: on [13:03:04] It's not the same issue as last time, is it? [13:03:08] it is [13:03:11] but I wonder why [13:03:15] since we do set it to off [13:03:16] still it was on [13:03:30] hmm on most servers it's off [13:04:09] New patchset: Dzahn; "add outreachcivi to outreach.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1599 [13:04:47] !log Found lvs5 and lvs6 with offload-gro enabled, even though it's set disabled in /etc/network/interfaces... corrected [13:04:52] weird [13:04:55] Logged the message, Master [13:04:58] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1599 [13:04:58] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1599 [13:12:34] mark, that looks a lot better, the bot is uploading a 3 meg file in under a minute now [13:12:34] Thanks [13:12:44] That's from TS, not hume [13:13:01] ok [13:13:05] but this is worrysome ;) [13:13:11] Puppet? [13:13:22] yeah but it sucks a bit [13:14:00] apergos, ^ [13:15:07] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1596 [13:15:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1596 [13:15:37] New patchset: Dzahn; "add url.wm apache_site config, fix tabs, email (do we still use url.wm?)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1440 [13:21:33] how is it from hume now? [13:23:03] Not sure, could stop Rotatebot and try it [13:23:31] might as well see if it's any better [13:29:13] New patchset: Mark Bergsma; "Enforce interface gro offloading in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1600 [13:29:22] Feck [13:29:27] ? [13:29:31] 1 second to upload 4 meg :D [13:29:41] get to it :-D [13:29:52] what's today's backlog? [13:29:56] https://commons.wikimedia.org/w/index.php?limit=50&tagFilter=&title=Special%3AContributions&contribs=user&target=Reedy+RotateBot&namespace=&tagfilter=&year=&month=-1 [13:29:57] (how many files) [13:29:58] That's insane [13:30:01] 550 or so [13:30:27] ok. could you sleep 2 secs between each or so... give the scalers a little teeny breather? it will still finish fast [13:30:35] New patchset: Mark Bergsma; "Enforce interface gro offloading in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1600 [13:30:41] Yeah [13:30:43] Well... [13:30:51] https://gerrit.wikimedia.org/r/#change,1600 [13:30:53] how's that? [13:30:54] On TS, it runs every 5 minutes via cron, if it's not already running [13:31:08] huh [13:31:43] ok well however you work it out, just so we don't pound on it too continuously [13:32:30] Reedy: however - we apparently cannot tag fast enough. I already placed notices to start mass tagging a day ago on the relevant sections. [13:32:32] hah [13:32:41] uh huh [13:33:05] New patchset: Dzahn; "add secure.wm apache site_config + basic puppet class, add proxy and proxy_http modules, fix tabs, move to secure.pp instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1436 [13:33:53] So it's whether we keep it on cluster, or let rotatebot do it, as that's significantly faster too [13:33:56] noone wants to review my commits :'-( [13:35:31] it was loading [13:35:38] heh, there is an extra whitespace in generic-definitions :) [13:35:46] took a while, sorry, my machine was swapping and when it does that new ff tabs take *forever* to open [13:36:10] well if you want to clear out the backlog fast running it from hume is th way to go [13:36:25] once every 5 mins seems fine to me [13:36:44] hah ff didn't even load the page [13:36:46] * apergos tries again [13:39:01] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1600 [13:40:33] New patchset: Mark Bergsma; "Enforce interface gro offloading in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1600 [13:41:19] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1600 [13:41:19] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1600 [13:43:51] New patchset: Mark Bergsma; "Fix syntax error" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1601 [13:44:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1601 [13:44:03] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1601 [13:44:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1601 [13:44:37] why is lvs1003 still down? [13:44:47] when are the changes in /files/etc/network/interfaces applied? only at network startup right? [13:44:51] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1436 [13:44:57] ie typically at boot time ? [13:45:03] theoretically always when the network interface is brought up [13:45:38] so if the change went out and the hosts haven't had the interface cycled, they'd keep the old setting [13:47:43] New patchset: Dzahn; "add secure.wm apache site_config + basic puppet class, add proxy and proxy_http modules, fix tabs, move to secure.pp instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1436 [13:47:54]