[00:21:02] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [00:41:01] gwicke: Are you in the office? [00:41:30] Krinkle: no, at home [00:41:41] Someone in ops needs to be pinged to fix Gerrit [00:41:49] It has been down for an hour now [00:44:58] Don't we have nagios check for this so ops knows? [00:45:00] argh [00:47:21] hi, what's up with gallium ? [00:47:26] it's like.. extremely unresponsive [00:47:33] I'm trying to submit a patch to gerrit [00:47:42] average: gallium runs Jenkins [00:47:43] and git-review is running .. forever [00:47:46] average: Gerrit is down [00:47:59] I'm trying to reach someone in ops [00:48:03] Krinkle: ok, thanks for confirming [00:48:21] Krinkle: if you can reach any ops, please ask when gerrit will be up [00:48:36] !log as of an hour ago Gerrit is completely unresponsive. Filed details at https://bugzilla.wikimedia.org/show_bug.cgi?id=48061 [00:48:44] Logged the message, Master [00:49:24] average: Gerrit isn't intentionally down. there is a server problem. When ops are reached they will try to fix it. Hopefully an ETA will not be needed. [00:49:40] marktraceur: Are you in the office? [00:49:52] Nope [00:50:10] Krinkle: do you have access to gallium ? [00:50:24] I'll go down on 3 and see if anybody's down there... [00:51:03] ok [00:51:14] average: I do, I created a lot of the continuous integration infrastructure [00:51:25] average: what do you need from gallium? [00:51:37] Krinkle: I need my gerrit back, and my jenkins please :) [00:51:45] Gerrit doesn't run on gallium [00:52:07] Jenkins and Zuul are running fine [00:52:17] https://integration.wikimedia.org/zuul/ [00:52:18] https://integration.wikimedia.org/ci/ [00:52:20] https://integration.wikimedia.org/ [00:52:21] well.. then.. gerrit then, I would like to have it back [00:52:33] They're just waiting for stuff to do :) [00:52:34] but I understand.. I'll wait.. [00:52:41] As a loyal Jenkins does [00:52:56] yeah, except they're hooked to gerrit so... [00:53:06] They're pretty isolated, but I know what you mean. [00:53:10] if gerrit doesn't run, then neither will jenkins.. [00:53:12] yeah.. [00:53:36] There's actually quite a few jobs on Jenkins that don't interact with Gerrit. [00:53:49] wish I was using one of those jobs... [00:54:10] but we're having code reviews(there are reasons behind that, won't go into them now) [00:54:24] Anyhow, that's why we have local topic branches (which you should have anyway) [00:54:50] Krinkle: you mean regular un-gerrited-regular-git-branches ? [00:54:52] I work offline a lot, no issues. Pushing to gerrit is only a minor part of the workflow. Can be deferred until later. [00:55:12] You commit locally, those branches are there, you can continue working, not? [00:55:24] you have a full clone of the entire repository locally [00:55:29] sure... but I want to see jenkins run the testsuite... [00:55:33] Sure [00:55:41] and zuul makes it a big pain for me to set up a manual run of the tests [00:55:48] You can also run them locally if you're doing something test-sensitive [00:56:01] zuul is like a big barrier for me, I had much more flexibility before it was introduced [00:56:03] php tests/phpunit.php [00:56:17] but.. hopefuly gerrit will be back and all will be solved [00:56:18] I fail to see how Zuul is relevant? [00:56:34] Zuul communicated to Jenkins what builds to trigger based on Gerrit events [00:56:45] Krinkle: let's suppose I wanna set up a manual run right ? zuul has this huge parameter list thing, with all sorts of stuff I have no idea what they all mean [00:56:56] it isn't at all related to running tests themselves, whatever you used to do manaully is no different with or without zuul. [00:57:18] average: You mean trigger a build inside jenkins? [00:57:22] exactly [00:57:27] no ops people in the office [00:58:06] average: 1) why would you want to do that? 2) do you mean for a specific commit or just latest master? 3) You can re-trigger any previous build from jenkins, noneed to know parameters. [00:58:25] Jenkins doesn't know about Zuul in the sense that re-triggering a build is a simple click, just as before [00:58:53] yeah, except when I want to re-trigger I get this big form with all these fields [00:58:53] average: With manually, I thought you meant locally [00:59:20] average: They are all prefilled, that is the point of retriggering (instead of building manually) [00:59:20] no, I mean on integration.wikimedia.org , sorry for the confusion [00:59:37] average: What is the point of retriggering a previous build? The results are going to be the same [00:59:58] hence I thought you wanted to test something you are working on locally [01:00:23] no, locally stuff runs fine(on my machine) [01:00:57] And now Wikipedia is in read-only mode :( [01:01:00] average: So you an run tests on the code you are currently unable to push to gerrit. Awesome :) We've abstracted most tests to a simple bash command, that's quite nice indeed. [01:01:10] kaldari: what? [01:01:12] wy [01:01:29] no maintenance message [01:01:39] someone enabled read-only without a message [01:02:07] YAY ! [01:02:09] Gerrit back up [01:02:48] might have been due to the maintanence script. I stopped it just to be safe [01:02:53] "The administrator who locked it offered this explanation: The database has been automatically locked while the slave database servers catch up to the master" [01:03:04] from a random edit form [01:03:09] I don't see that [01:04:24] The Wikipedia database is temporarily in read-only mode. This is probably due to routine maintenance; if so, you will be able to edit again within a few minutes. We apologize for any inconvenience this might have caused. You can continue editing in the text box below, but you may not be able to save. Before attempting to save, you may wish to copy-and-paste the text into a text file so you will [01:04:24] have it to attempt to make the edit again. For further information, you can visit the #wikipedia channel on the freenode IRC network. If you do not have an IRC client, you can connect using a web-based client. [01:04:26] that's what I see. [01:04:39] I got this message on Special:upload, "The database has been automatically locked while the slave database servers catch up to the master" [01:04:45] This was actually https://en.wikipedia.org/w/index.php?title=Richard_Hill_(athlete)&action=edit&editintro=Template:BLP_editintro [01:05:40] all the slaves are seriously lagged: http://noc.wikimedia.org/dbtree/ [01:06:35] Lock has been lifted [01:07:52] @replag [01:07:53] Krinkle: No replag currently. See also "replag all". [01:16:56] !log binasher restarted gerrit [01:17:05] Logged the message, Master [01:21:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [01:30:43] dbbot-wm is suffering from an unexpected username change in labs. Fixing it now. [01:31:04] @externals update [01:31:07] @externals [01:31:12] @replag [01:31:14] @replag all [01:31:41] Krinkle: Successfully updated externals! [01:31:42] Krinkle: [operations/mediawiki-config.git] Checked out HEAD: fc015966a3a9de71c7bcb5d7a05cb50fbd8661ef - https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=commit;h=fc015966a3a9de71c7bcb5d7a05cb50fbd8661ef [01:31:44] Krinkle: No replag currently. See also "replag all". [01:31:45] Krinkle: [s1] db1056: 0s, db1049: 0s, db1050: 0s, db1051: 0s, db1052: 0s; [s2] db1034: 0s, db1002: 0s, db1009: 0s, db1018: 0s; [s3] db1019: 0s, db1003: 0s, db1010: 0s, db1035: 0s [01:31:47] Krinkle: [s4] db1038: 0s, db1004: 0s, db1011: 0s; [s5] db1058: 0s, db1005: 0s, db1026: 0s, db1021: 0s; [s6] db1006: 0s, db1022: 0s, db1027: 0s, db1040: 0s; [s7] db1041: 0s, db1007: 0s, db1024: 0s, db1028: 0s [01:31:59] great [01:51:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.155 second response time [02:12:01] !log LocalisationUpdate completed (1.22wmf3) at Sat May 4 02:12:01 UTC 2013 [02:12:10] Logged the message, Master [02:20:30] !log LocalisationUpdate completed (1.22wmf2) at Sat May 4 02:20:30 UTC 2013 [02:20:37] Logged the message, Master [03:47:54] New patchset: Ori.livneh; "tcpircbot: subscribe to (rather than require) config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62214 [03:50:01] New review: Ori.livneh; "Could the person merging this please also 'service tcpircbot-logmsgbot restart' on neon, please? Alt..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62214 [03:53:16] New patchset: Krinkle; "contint: Add rewrite rules for favicon.ico to favicon.php" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62125 [03:53:29] New patchset: Krinkle; "contint: Move apache logs to readable place for localhost testing" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61997 [03:54:30] Krinkle: how does one end up with an unexpected username change? [03:54:53] jeremyb_: Ask Coren|Away [03:55:08] I don't know the details [03:55:26] Something in labs, tools or osm got changed [03:55:29] hemm [03:55:40] triggering a linux user name change [03:57:33] * jeremyb_ wonders if numeric uid changed [03:58:38] New review: Hydriz; "Is there any relevant change for apache-config to reflect this new change?" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53885 [04:16:01] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [04:36:09] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat May 4 04:36:09 UTC 2013 [04:36:16] Logged the message, Master [04:43:11] jeremyb_: Tool accounts used to be project-local; moving to the NFS server needed them to be managed by OpenStack. Openstack folded case without warning. [04:44:46] aha [04:49:51] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [05:39:21] New patchset: Jeremyb; "Update wgServer and wgCanonicalServer for multi subdomain wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53885 [05:40:11] New patchset: Jeremyb; "Update wgServer, wgCanonicalServer for sub.subdomain wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53885 [05:42:20] New review: Jeremyb; "rebased and then tweaked commit msg" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53885 [06:35:17] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:04] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [07:11:44] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 1.27 ms [07:39:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [08:19:21] New patchset: Aklapper; "Weekly Bugzilla Report mail: Fix wrong SQL query on bug resolutions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62220 [08:24:00] New patchset: Aklapper; "Weekly Bugzilla Report mail: Fix wrong SQL query on bug resolutions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62220 [08:34:07] New review: Ori.livneh; "Hey Physikerwelt," [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61767 [09:41:57] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:41:57] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:41:57] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:41:57] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:21:05] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [10:31:30] hello Operations [10:31:36] can someone help me sort something out please ? [10:31:47] gallium access would be necessary [10:31:58] there's a directory that just doesn't seem to be there [10:32:09] and some of my tests fail because of that.. [10:32:10] :( [11:05:58] average: hashar and krinkle would be the ones to talk to [11:08:22] p858snake|l_: I know.. [11:08:34] p858snake|l_: except hashar's not here [11:08:38] and nor is Krinkle. [12:53:53] average: It's Saturday. [13:06:58] average: if you file a bug, and they will do it when they are around [13:08:49] Susan: yeah, and tommorow's Sunday [13:08:56] Susan: what's new ? [13:21:58] average, who are you sorry? [13:23:44] average: people don't tend to work over weekends [13:26:45] * legoktm pokes Reedy about https://gerrit.wikimedia.org/r/#/c/62012/ [13:35:26] Thehelpfulone: I'm busy sorry.. ttyl [13:35:39] heh [14:04:24] is icinga-wm cheating or is the cluster really so quiet? did the servers have enough fun yesterday? [14:16:44] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [14:40:50] average: what directory? [14:42:19] Krinkle: /var/lib/jenkins/jobs/analytics-wikistats/workspace/squids/testdata/merge-australia-into-oceania [14:42:22] Krinkle: hi [14:42:37] average: what is trying to access the directory? [14:42:47] e.g. where does the error show [14:43:13] a build link [14:43:49] Krinkle: this is trying to access it cd /var/lib/jenkins/jobs/analytics-wikistats/workspace/squids; perl -Iperl t/11*.t [14:44:08] Krinkle: the directory I mentioned above is Git versioned so it should be checked out but it's not [14:44:11] dunno why [14:44:18] average: unfortunately that doesn't mean much to me. [14:44:21] Krinkle: can you maybe wipe out /var/lib/jenkins/jobs/analytics-wikistats ? [14:44:30] average: jobs are auto-wiped on build [14:44:44] Krinkle: the .git of /var/lib/jenkins/jobs/analytics-wikistats/.git also > [14:44:47] ? [14:45:00] workspaces were moved to ssf [14:45:02] ssd* [14:45:16] /var/lib/jenkins/jobs/*/workspace is no longer used [14:45:27] Krinkle: if I had access to gallium, this would be the first thing I'd try.. [14:45:53] average: access to gallium shouldn't be needed. All jenkins configuration is in version control [14:46:08] can you link to a build failure? [14:46:39] https://integration.wikimedia.org/ci/job/analytics-wikistats/95/consoleText [14:46:44] this is the build failure [14:46:57] Krinkle: I made the same build on Travis and it worked out fine [14:47:01] 11:52:08 Wiping out workspace first. [14:47:07] :) [14:47:22] pwd: 11:52:09 /srv/ssd/jenkins/workspace/analytics-wikistats [14:47:40] Krinkle: so erm.. I should use /srv.... instead of /var/lib/jenkins/... ? [14:47:43] Are you hardcoding paths to varlibjenkins somewhere in your repository? [14:47:50] Krinkle: yes sir, I am [14:47:54] You shouldn't use either path. It can vary by a lot of factors. [14:48:07] for example, in concurrent workers it will use @1 or @2 subdirectories [14:48:11] Krinkle: then maybe I can use an ENV variable for this ? [14:48:15] Yes [14:48:17] $WORKSPACE [14:48:23] Krinkle: thank you sir [14:48:26] I'll use $WORKSPACE [14:48:43] Let me know how it works out :) [14:48:54] Krinkle: i'm trying it out right now [14:50:46] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [14:52:03] Krinkle: works great sir [14:52:06] Krinkle: https://integration.wikimedia.org/ci/job/analytics-wikistats/96/consoleText [14:52:11] Krinkle: Thank you very much ! :) [14:52:17] average: great! [14:52:30] average: do you why this is stuck? http://www.infodisiac.com/Wikipedia/ScanMail/index.html [14:54:02] average: Try to avoid trailing whitespace :) [14:54:11] (noticed it randomly in one of the files0 [14:54:21] https://gerrit.wikimedia.org/r/#/c/62206/3/.travis.yml [14:55:44] Nemo_bis: Mr. Nemo_bis sir, I have no power over the domain infodisiac.com . If you're interested in that table, please contact Erik Zachte [14:56:15] Nemo_bis: if you believe this is a bug that the Analytics team should take care of, please contact the Analytics team during regular work hours Monday=>Friday on #wikimedia-analytics [14:56:50] Nemo_bis: As I do not know who you are, I'd like to relay you to our team leader Diederik van Liere (drdee on IRC) [14:57:24] Nemo_bis: he is the person in charge of organize/prioritze all our development activity [14:59:19] Krinkle: yes, will do, thanks [15:01:01] average: sorry, yes, I know all those things, I thought that maybe you happened to know some relevant change in the wikistats repo where the code for those stats is :) [15:01:06] thanks [15:09:37] Nemo_bis: we have many bugs which need attention, and they are prioritized because we're using SCRUM so we're commiting to solving a chunk of them each week, and showing them in the showcase that we have on Wednesday [15:10:27] average: I'm not challenging your prioritisation, I just asked if you happened to know something. You don't, fine, it's all I need. :) [15:10:35] Nemo_bis: the report you mentioned seems to be produced by https://github.com/wikimedia/analytics-wikistats/blob/master/mail-lists/ScanMail.pl [15:11:16] that's all I know [15:12:28] thanks [15:12:31] np [15:13:04] last change is by me ^^ [15:13:49] Nemo_bis: just saw :) [15:53:09] New patchset: Reedy; "(bug 48013) Creating a flood user group for wikidatawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62012 [15:53:24] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62012 [15:53:45] New patchset: Reedy; "(bug 45696) Add a msg to $wgForceUIMsgAsContentMsg for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62175 [15:53:54] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62175 [15:56:14] New patchset: Reedy; "(bug 47820) Change $wgSitename and NS_PROJECT_TALK for udmwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62197 [15:56:28] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62197 [15:57:13] New patchset: Reedy; "(bug 47620) Exclude user and talk pages from wikidata features in clients" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60978 [15:57:48] New patchset: Reedy; "cswiktionary: Set display title restriction" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61789 [15:57:55] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61789 [15:58:09] New patchset: Reedy; "Update FirefoxOS app for manifest validation issue" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62153 [15:58:16] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62153 [15:58:38] New patchset: Reedy; "(bug 48043) Activate Flagged Revisions for modules namespace on Polish Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62147 [15:58:47] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62147 [15:59:51] New patchset: Reedy; "(bug 47933) Rename Module talk namespace for Japanese Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61782 [16:00:00] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61782 [16:02:17] !log reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/ [16:02:26] Logged the message, Master [16:02:51] !log reedy synchronized wmf-config/ [16:02:59] Logged the message, Master [16:05:38] Reedy: thanks :D [16:31:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [16:37:04] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [17:10:51] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [17:12:17] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 2.47 ms [18:02:24] Thehelpfulone: he's an analytics contractor [18:02:34] stefan petrea [18:24:18] thanks jeremyb_, he's not even on the staff page though [18:25:31] looks like he was announced in september too - do you happen to know which department he works in (so I can poke someone to get him added?) [18:26:57] oh I found it from scrollback [18:29:16] Thehelpfulone: i would say "analytics contractor" means analytics dept? :) [18:29:23] assuming that exists [18:29:29] must be under robla [18:29:32] it doesn't :p [18:30:20] well it might be a team, but there are analysts in grants and engineering [18:30:57] Stefan is in Analytics, which is currently part of Platform, but is being spun out into another group [18:31:44] (hence the advertisement for the Director of Analytics position) [18:32:03] ah robla so you would be the person to ask for now, can you request that he's added to the staff page? [18:32:39] sure [18:32:42] I think the rule for contractors is that they're usually only added if their manager asks for it, but given that he's been around since September, he's a longer term contractor so should probably go on that page [18:32:50] thanks [18:33:13] well, actually, that's not the way that it works [18:34:00] * jeremyb_ does in fact wonder how it works... [18:34:15] had this misunderstanding in #-fundraising not too long ago [18:34:39] robla, oh, could you explain then? [18:35:06] giving robla work on teh weekend! [18:35:35] I personally think all staff/contractors should go on that page (the {{staff count}} template works using some lua magic based on the information in that page) [18:35:43] jeremyb_, hey, you're the one who pinged him :P [18:35:51] hah [18:35:58] oh, it's converted to lua, huh [18:36:17] Thehelpfulone: i agree. but see #-fundraising some days ago [18:37:51] Thehelpfulone: was on 2013-04-29 [18:38:43] it boils down to the role that they're filling based on internal accounting [18:41:18] why not just have everyone? [18:41:45] http://dpaste.com/1104299/plain/ fwiw [18:54:28] * jeremyb_ spies Thehelpfulone making fixes [18:55:46] heh [18:56:00] greg-g, what's your mediawiki.org username? [18:57:32] looks like Greg G && Greg_(WMF) [18:58:11] hrmmm, https://toolserver.org/~quentinv57/sulinfo/Greg%20%28WMF%29 [18:58:46] something's wrong with the interface at https://meta.wikimedia.org/wiki/Special:CentralAuth/Greg_%28WMF%29 [19:05:48] thanks jeremy, updated a few links :) [19:08:13] jeremyb_: yeah Nemo pointed that our earlier [19:08:30] which? [19:08:46] the CentralAuth interface being broken [19:08:51] idk if a bug was filed [19:08:54] ahh [19:08:58] idk either [19:09:15] i just suddenly want a "before" pic [19:09:20] have to run, bblllll [19:15:08] Be back later, let's lick lily-livered landlubbers [19:15:26] Hm, too many [19:42:31] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [19:42:31] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [19:42:31] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [19:42:31] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [19:46:53] jeremyb_: so are you going to file that? or is legoktm :) [20:01:27] New patchset: Odder; "(bug 47820) Localise $wgMetaNamespace for udmwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62244 [20:21:07] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [20:39:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [21:03:48] aww why didn't Krenair point out the existing bug before :) [21:06:21] Sorry, I didn't notice [21:42:40] !log reedy synchronized php-1.22wmf3/extensions/TorBlock/ [21:42:48] Logged the message, Master [21:43:59] !log Manually force ran TorBlock/loadExitNodes.php [21:44:08] Logged the message, Master [21:44:50] Reedy: how old was the list? [21:45:00] Doesn't really matter [21:45:03] It was a broken list [21:45:12] https://bugzilla.wikimedia.org/show_bug.cgi?id=47626 [22:40:55] New patchset: Reedy; "Fixup CDB listing" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62307 [22:41:18] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62307 [22:41:51] !log reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache' [22:41:59] Logged the message, Master [22:42:28] New patchset: Reedy; "Update interwiki.cdb" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62308 [22:42:38] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62308 [22:42:59] New patchset: Reedy; "Commit wikimania2014 entries" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62309 [22:43:10] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62309 [22:44:41] New patchset: Reedy; "Fixup autocreation for cdb entries" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62310 [22:45:06] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62310 [22:46:39] New patchset: Reedy; "trusted-xff.cdb" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62311 [22:46:59] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62311 [22:48:48] New patchset: Reedy; "wikimedia.conf doesn't exist there" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62312 [22:49:04] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62312 [22:57:10] New patchset: Reedy; "Fix git repo path for wmf-config files" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62314 [22:58:03] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62314 [23:27:58] Nemo_bis: not me, i'm on the train between 2 events [23:43:30] jeremyb_: no need, thanks :)