[00:02:19] RoanKattouw: Reedy needs your help. :-) [00:02:28] OK [00:02:35] You have until :15-ish [00:02:40] RoanKattouw: so, we added 2 new wikis, login and iegcom [00:02:59] Apache knows about them, sees them as virtual hosts [00:03:07] DNS has them , point to lb [00:03:15] yet, we get "No wiki found" [00:03:37] Which lb [00:03:44] https://iegcom.wikimedia.org/ https://login.wikimedia.org/ [00:03:44] wikimedia-lb [00:03:47] New patchset: Dr0ptp4kt; "Add extra IP block to Varnish for Wikipedia Zero partner TATA." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62763 [00:03:49] OK [00:03:55] wikimedia.org:iegcom 1H IN CNAME wikimedia-lb [00:04:00] wikimedia.org:login 1H IN CNAME wikimedia-lb [00:04:14] port * namevhost iegcom.wikimedia.org (/etc/apache2/wmf/remnant.conf:1715) [00:04:21] port * namevhost login.wikimedia.org (/etc/apache2/wmf/wikimedia.conf:257) [00:04:32] Oh, wait [00:04:39] Reedy: What are the db names? [00:04:48] iegcomwiki and .. [00:04:53] loginwiki [00:05:04] OK [00:05:08] Let me check something real quick [00:05:15] mediawiki is fine with them [00:05:25] Ryan_Lane: Could you enable $wgVectorUseIconWatch = true; on wikitech? (and btw, where is its configuration?) [00:05:34] * RoanKattouw clones mediawiki-config onto his loaner machine [00:05:42] actually, i should probably move iegcom to wikimedia.conf , but unrelated [00:05:45] configuration is only on virt0 [00:05:48] I need to puppetize it [00:05:56] I think andrewbogott may be working on that, actually [00:06:32] done [00:06:39] what does that setting do? [00:06:42] Reedy: adding to s3.dblist and special.dblist, that was already done by ./refresh-dblist [00:06:53] ah [00:06:54] I see [00:07:21] Ryan_Lane: lol, you ask me now? [00:07:30] ) [00:07:33] err [00:07:33] ;) [00:07:39] mutante: Need committing ;) [00:07:47] Ryan_Lane: It turns the "Watch" item in the action menu into a primary item on the toolbar with the star icon [00:07:51] yeah [00:08:02] I looked it up right after I asked [00:08:04] Ryan_Lane: It's a nice change to have. :-) [00:08:09] indeed [00:08:34] PROBLEM - DPKG on analytics1008 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:08:40] that is me [00:08:55] Reedy: slowness when git pulling [00:08:58] trying to fix these machines up ! [00:09:01] done [00:09:14] PROBLEM - DPKG on analytics1025 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:09:14] PROBLEM - DPKG on analytics1014 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:09:34] PROBLEM - DPKG on analytics1026 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:09:34] RECOVERY - DPKG on analytics1008 is OK: All packages OK [00:09:34] PROBLEM - DPKG on analytics1016 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:09:44] PROBLEM - DPKG on analytics1022 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:10:14] PROBLEM - DPKG on analytics1015 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:10:14] RECOVERY - DPKG on analytics1025 is OK: All packages OK [00:10:14] PROBLEM - DPKG on analytics1012 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:10:34] RECOVERY - DPKG on analytics1026 is OK: All packages OK [00:10:44] RECOVERY - DPKG on analytics1022 is OK: All packages OK [00:10:56] Oh OK [00:11:05] Multiversion does some weird stuff based on the docroot [00:11:07] Checking Apache config [00:11:29] i'm moving something from remnants.conf to wikimedia.conf but not making changes to the actual config [00:12:00] very weird stuff [00:12:14] RECOVERY - DPKG on analytics1015 is OK: All packages OK [00:12:14] RECOVERY - DPKG on analytics1014 is OK: All packages OK [00:12:34] RECOVERY - DPKG on analytics1016 is OK: All packages OK [00:12:43] New patchset: Dzahn; "move newer private wikis (iegcom, transitionteam) out of remants.conf use wikimedia.conf instead" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62764 [00:13:14] RECOVERY - DPKG on analytics1012 is OK: All packages OK [00:13:46] New review: Dzahn; "no changes to the actual config, just move to other file" [operations/apache-config] (master) C: 2; - https://gerrit.wikimedia.org/r/62764 [00:13:46] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62764 [00:14:12] Hmm this totally looks like it should work [00:14:18] The docroot looks right [00:14:38] mutante: I do note, this is exactly what we've both done for other wikis in the not so distant past and it worked fine ;) [00:14:49] acknowledged, yea [00:15:19] OK I gotta run [00:15:21] lemme graceful one more time after having merged that [00:15:27] Sorry I couldn't figure it out [00:15:29] * RoanKattouw disappears to midterm [00:15:38] RoanKattouw: thanks for confirming it's weird , cya, heh [00:16:08] RoanKattouw: Have fun. [00:16:54] And at the same time, othe wikis haven't broken.. [00:17:21] yea, did not break existing wikis [00:17:27] apaches dont complain about it [00:17:54] and .. affects both new wikis created by separate people ..hrrm [00:18:29] * James_F blames cosmic rays. [00:20:04] Maybe we should add index.html with "It works!" [00:20:30] Reedy: can also confirm them both on http://noc.wikimedia.org/dbtree/ on s3 [00:20:58] Mediawiki seems ok run via eval.php [00:21:04] !log upgrading a bunch of tampa squids [00:22:05] Reedy: --> http://login.wikimedia.org/Foo and wait a couple seconds [00:22:12] gets us to wikimediafoundation.org [00:23:01] iegcom is hitting missing.php [00:25:03] Reedy: so Apache wikimedia.conf, line 170 ServerAlias *.wikimedia.org that is before the others [00:25:24] hits the wildcard first? [00:25:30] mutante: The missing.php seems to suggest it's not correctly in wikiversions.cdb [00:25:44] PROBLEM - DPKG on ms-be1 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:26:05] Reedy: hrmm, i did run sync-wikiversions though [00:26:14] PROBLEM - DPKG on mc12 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:26:33] reedy@fenari:/home/wikipedia/common$ grep iegcom wikiversions.* [00:26:33] Binary file wikiversions.cdb matches [00:26:33] wikiversions.dat:iegcomwiki php-1.22wmf3 * [00:26:34] PROBLEM - DPKG on mc1013 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:26:34] PROBLEM - DPKG on mc1011 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:26:44] PROBLEM - DPKG on ms-fe4 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:26:44] PROBLEM - DPKG on ms-be4 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:27:14] RECOVERY - DPKG on mc12 is OK: All packages OK [00:27:21] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [00:27:34] RECOVERY - DPKG on mc1013 is OK: All packages OK [00:27:34] RECOVERY - DPKG on mc1011 is OK: All packages OK [00:27:44] RECOVERY - DPKG on ms-fe4 is OK: All packages OK [00:27:59] !log reedy synchronized wmf-config/ [00:28:09] mutante: Aha. Cached things. Touching helped [00:28:18] It's now at the same screen as loginwiki [00:31:40] https://login.wikimedia.org/ [00:31:50] ^ Anyone know what landing screen that is? [00:31:56] wikimedia.org Reedy [00:32:12] controlled via Meta [00:32:15] Right, thanks [00:32:23] mutante: Which suggests it is hitting the wildcard one first [00:32:34] mutante: Does it do something weird like order of virtualhost definitions? [00:32:44] RECOVERY - DPKG on ms-be1 is OK: All packages OK [00:32:45] i just tested that on mw1044 [00:32:48] (which you said above) [00:32:49] putting them first [00:32:56] but i don't see a difference yet [00:33:09] http://iegcom.wikimedia.org/wiki/ * 301 Moved Permanently https://iegcom.wikimedia.org/wiki/ [00:33:14] http://iegcom.wikimedia.org/wiki/Foo * 301 Moved Permanently https://iegcom.wikimedia.org/wiki/Foo [00:33:19] http://login.wikimedia.org/wiki/ * 301 Moved Permanently http://wikimediafoundation.org/wiki/ [00:33:23] https://login.wikimedia.org * 200 OK 6251 [00:33:44] RECOVERY - DPKG on ms-be4 is OK: All packages OK [00:33:46] mutante: [00:33:47] Name-based virtual hosts for the best-matching set of s are processed in the order they appear in the configuration. The first matching ServerName or ServerAlias is used, with no different precedence for wildcards (nor for ServerName vs. ServerAlias). [00:33:56] So order does matter, yes [00:33:58] yes, it does [00:34:05] just had the same thing when moving secure [00:34:14] heh [00:34:18] but..it doesnt appear to fix it anyways.. wth [00:34:50] Wouldn't hurt to move them and push them out as is [00:35:07] yea, doing [00:36:43] New review: Reedy; "(1 comment)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566 [00:37:02] ^ Same problem would happen with my wikimania docroot moves. Comment left so I remember to "fix" it [00:38:28] that would mean transitionteam would have broken by moving it [00:38:56] and it did! [00:39:02] :D [00:40:56] New patchset: Dzahn; "move 3 private wikis before wildcard ServerName" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62767 [00:41:34] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:36] New review: Dzahn; "this should work better :p" [operations/apache-config] (master) C: 2; - https://gerrit.wikimedia.org/r/62767 [00:41:36] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62767 [00:43:33] i hate it when my ssh agent dies on graceful [00:43:55] Yeah,not helpful [00:45:41] so, transitionteam is back [00:46:04] login redirects to foundation Main_Page [00:46:13] mutante: Wohooo [00:46:17] and iegcom is not found [00:46:18] I just purged login [00:46:20] and logged in :) [00:46:25] and i was about to purge..heh [00:46:25] nice [00:46:43] Hmm, iegcom is back to not found :/ [00:48:25] mutante: To answer our previous discussion. It was something stupidly simple [00:49:01] which one?:) but yea, it usually is [00:50:27] !log reedy synchronized wmf-config/InitialiseSettings.php 'loginwiki logo' [00:51:45] !log reedy synchronized wmf-config/InitialiseSettings.php 'favico' [00:51:58] New patchset: Reedy; "Set favico and logo for loginwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62769 [00:52:36] hmm what's login wiki going to be used for? [00:53:12] Thehelpfulone: oauth/openid stuffs [00:53:16] Thehelpfulone: Logins. :-) [00:53:51] Thehelpfulone: https://login.wikimedia.org/wiki/Special:ListUsers :-) [00:53:54] oh it's a SUL wiki [00:54:05] Thehelpfulone: Yeah. [00:54:13] I didn't see a create account link so was wondering :) [00:54:35] https://login.wikimedia.org/wiki/Special:ListGroupRights is amusingly less populated [00:54:59] We should probably drop a few of those groups too [00:54:59] Thehelpfulone: No local accounts, only global. [00:55:37] ok [00:55:49] Reedy: Yeah, drop importers at least. [00:55:53] I guess people should be able to create accounts from there. But wouldn't be the common starting point for most [00:55:54] And bots [00:56:02] Hmm, no.. [00:56:04] Reedy: And edituserJS should be false for sysops. [00:56:07] Bots might be used for notifications [00:56:13] yeah, bots should stay. [00:56:14] oh you can't even edit James_F? [00:56:24] At least until OAuth is working well. ;-) [00:56:50] Thehelpfulone: Yeah, this is a very locked-down wiki; essentially no user-editable parts, at least for now. [00:57:28] https://login.wikimedia.org/w/index.php?title=Main_Page&diff=2&oldid=1 [00:57:33] Hmm, why can I edit the main page? [00:57:37] show off :p [00:57:43] Reedy, global sysadmin/staff? [00:57:44] Reedy: It's unprotected. [00:57:45] Global groups? [00:57:46] Yeah [00:58:14] Reedy: Now protected. [00:58:20] My vandalism!! [00:58:53] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62769 [00:59:19] Reedy: The Sister Projects section forgets Commons and Wikidata. [00:59:53] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/WikimediaMaintenance.git;a=blob;f=addWiki.php;h=dab7d255f68d68bb3ab7091891dc905cba522f78;hb=HEAD [01:00:18] Easily added, not sure in what order they should go [01:00:50] * James_F nods. [01:01:06] For stuff like this, it'd be really nice if gerrit had a web interface for creating changesets [01:01:13] I added Wikidata then Commons, which probably suffices. [01:01:15] Yeah, I'd kill for that. [01:01:24] I'm mostly make language and setting changes. [01:01:48] Exactly [01:01:54] Or fixing small typos in larger commits [01:02:07] rather than shell, login to server... [01:02:28] Indeed. [01:02:35] You can fix commit summaries. Why not commits? [01:02:42] Indeed [01:02:43] * James_F grumbles with Reedy. [01:02:50] magical button "fix typos" [01:02:50] That's a newer feature though [01:03:04] Sure. Maybe we should throw the ideas into the gerrit upstream tracker? [01:03:17] Where's ^Demon to tell me where that is when I need him? ;-) [01:03:54] http://code.google.com/p/gerrit/issues/list [01:04:35] so what else besides dblists can cause "No wiki found" [01:04:56] it's wikiversions not dblists [01:05:06] I'm slightly confused why that regressed when you fixed the apache rules [01:05:08] eh, yea, both, both have it [01:05:21] wikversions.cdb and .dat and the dblists [01:05:34] dont see a diff to loginwiki [01:05:51] mutante: Thanks! [01:07:40] * Jasper_Deng draws attention to http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28proposals%29#Developer.27s_Noticeboard .... doubting anyone on this channel would follow that noticeboard [01:09:01] what are they asking for? another mailing list? IRC channel number 5? [01:09:19] mutante: A wiki page for us to all follow. [01:09:21] SAL transclusion ? [01:09:28] !log reedy synchronized wmf-config/CommonSettings.php [01:10:05] James_F: sounds like they want SAL [01:10:18] plus gerrit pending changes [01:11:25] !log reedy synchronized wmf-config/CommonSettings.php [01:12:22] doesn't follow the "put you are putting it where there is high-traffic, that's why we don't see it" [01:13:05] New patchset: Reedy; "More misc config for loginwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62771 [01:13:35] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62771 [01:14:01] " You know, from bitter experience, that the enwp community isn't going to follow you to mediawiki.org and ferret out what changes you're planning on making to code that affects enwp." [01:14:11] !log reedy synchronized wmf-config/ [01:14:18] <-- doesn't work, that would mean every single language wiki has to be told separately [01:14:31] Echo! [01:14:43] We should leave every user on every wiki a talk page message [01:14:56] mutante: Indeed. [01:14:57] ah, yes, can you turn SAL into that:) [01:15:07] Reedy: Well, actually, that is going to be a part of Notifications. [01:15:11] Reedy: But it's not built yet [01:15:30] Reedy: Partially because it's only deployed to one wiki, so it's no better than now. [01:16:41] Reedy: loginwiki sitename is Wikipeida [01:16:48] " a place for Wikimedia developer announcements, not general tech discussion" <- as soon as a developer makes an announcement it will likely turn into tech discussion.. until people say to do that elsewhere [01:17:12] mutante: We could hard-protect the page in the DB? [01:17:13] What should it be called? Loginwiki? Login Wiki? Login? [01:17:19] mutante: Page only editable via SQL? [01:17:25] Reedy: Wikimedia Login Wiki. [01:17:35] mutante: Special:HearMeMortals? [01:17:38] What is this used for exactly? [01:17:42] Nothing yet [01:17:44] Reedy: James_F: Wikimedia Login? [01:17:53] i don't see how en.wp expects to be treated other than non-en wikis though [01:17:53] Krinkle: Loginwiki? It's the new replacement login system for CA. [01:17:57] Reedy: I gathered that much [01:18:05] Need to name the Project/Wikipedia namespace something sensible too [01:18:21] i'd recommend Project as Project namespace [01:18:21] Reedy: "Login" is fine. [01:18:28] like on mediawiki.org [01:18:28] And yeah, Project works fine. [01:18:32] It's not going to have much. [01:20:27] !log reedy synchronized wmf-config/InitialiseSettings.php [01:20:35] "We, sitting here on enwp, would really like to know about upcoming deployments. " <-- i can't help it, i'd have to say "soo.. subscribe to lists, join IRC, read SAL, have one of the bots talk to you, use Twitter or Identi.ca or whatnot.. ..." [01:21:05] New patchset: Reedy; "Set sitename and meta namespace for loginwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62772 [01:21:11] or open up the deployment calendar? [01:21:18] mutante: Right, and for anything we need to push (instead of having them poll) we use wikitech-ambasadors [01:21:23] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62772 [01:21:27] Another channel is certainly not the solution [01:23:22] wikitech-ambasadors: major change announcements [01:23:33] mutante: don't forget technology category on blog.wikimedia.org [01:23:44] Krinkle: heh, and the monthly reports [01:23:48] those are both on a high level, ideal for the community I think [01:23:52] we have too many channels [01:24:03] for more details, SAL, IRC, wikitech-l, .. [01:24:11] Definitely. [01:24:40] then people are going to say they don't understand the technical discussion on wikitech [01:24:51] after they said they want to know more about the technical side of things :p [01:25:04] been there [01:25:36] If wikitech is too detailed, then that means you shouldn't subscribe to it and instead using something on higher level (such as techblog.wikimedia.org) [01:25:45] Yeah [01:25:47] :) [01:25:51] Same same, but differenc [01:25:51] simple.sal.wikimedia.org :) [01:25:59] !log fixed [01:26:48] btw, no logbot:) [01:27:27] hrmm, if we could just see iegcom now that would be nice [01:27:44] Thehelpfulone: it's like practically done, just that it's not found:) [01:27:48] but it's there [01:29:21] Reedy: need swift containers for login? i guess not [01:29:35] need search index for login? [01:30:15] mutante: NOOOOOO not multiple subdomains!!!! [01:30:44] Reedy: haha, i wasn't that serious about simple.sal [01:31:01] I think search indexing is needed [01:31:13] If we're going to have some documentation there, we might aswell [01:31:30] can do tomorrow? it takes soo long :p [01:31:50] Reedy: What would be documented there that doesn't belong on metawiki, mw.org or wikitech? [01:31:52] and last time i added one.. oh wait.. did that ever work [01:32:21] I think loginwiki would ideally only have special pages, and no content :P [01:32:21] I've no idea [01:33:04] it did not.. wikimania2014 wiki search .. The search backend returned an error: [01:33:14] so i wouldn't know how to fix that :( [01:33:30] I was only really asked to create the wiki at this point ;) [01:33:58] kk [01:34:47] That was to Krinkle :p [01:34:55] http://iegcom.wikimedia.org * 301 Moved Permanently https://iegcom.wikimedia.org/ [01:34:58] https://iegcom.wikimedia.org * 404 Not Found [01:36:09] so that tells me the Apache part is ok [01:36:10] because: [01:36:18] RewriteCond %{HTTP:X-Forwarded-Proto} !https [01:36:29] RewriteRule ^/(.*)$ https://iegcom.wikimedia.org/$1 [R=301,L] [01:36:39] Yup, that's wanted behavior [01:37:36] I wonder what made it go back to 404ing [01:39:42] https://login.wikimedia.org/wiki/Special:Version?printable=yes [01:39:47] Special pages should have printable links [01:40:52] Special:Random?printable=yes works:) [01:41:32] Exactly [01:41:49] Not going to be the most common thing to do, but if you wanted to print it, you really dont want the border stuff [01:43:15] !date-au [01:43:44] Reedy: 2:43 AM .. you wanted earlier than 4:30 [01:43:54] i'll have to look at iegcom again tomorrow [01:44:00] Yup, I'm in the process of going to bed ;) [01:44:06] Guinea pigs need bringing in and feeding next [01:44:10] i don't see it and too tired too keep staring at it [01:44:19] guinea pigs :) [01:45:19] alright, cya later [02:11:52] quit [02:13:34] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:00] !log LocalisationUpdate completed (1.22wmf3) at Wed May 8 02:15:00 UTC 2013 [02:17:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [02:24:54] !log LocalisationUpdate completed (1.22wmf2) at Wed May 8 02:24:54 UTC 2013 [02:27:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:03:14] PROBLEM - Host mw72 is DOWN: PING CRITICAL - Packet loss = 100% [03:03:44] RECOVERY - Host mw72 is UP: PING OK - Packet loss = 0%, RTA = 26.58 ms [03:04:34] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [03:06:44] PROBLEM - Apache HTTP on mw72 is CRITICAL: Connection refused [03:27:54] !log morebots is down? [03:28:30] legoktm: were you still going to work on it? [03:28:45] Hi [03:28:52] Yeah, I just have one final left on Thursday [03:29:03] ok, that comes first :) [03:29:23] So I'll work on it when I get home on Saturday :) [03:44:11] i don't see how en.wp expects to be treated other than non-en wikis though # The Wikimedia Foundation seems to have no trouble treating the English Wikipedia differently. ;-) [03:45:14] What kind of logo is being used on login.wikimedia.org... [03:45:50] Susan, it's a wikimedia foundation logo, what else could it be? [03:45:56] the logo [03:46:25] It's not rendering proproperly. [03:46:27] Properly. [03:46:31] It has some weird in it. [03:46:34] Can you see it? [03:46:42] It's fine for me [03:46:48] There's a green line connecting the two green pieces. [03:46:51] On my screen. [03:46:55] I see it [03:47:02] very faint [03:47:34] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [03:47:34] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:47:34] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [03:47:34] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:52:04] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed May 8 03:52:04 UTC 2013 [03:56:08] https://bugzilla.wikimedia.org/show_bug.cgi?id=47228 [05:10:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [05:22:20] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62581 [05:29:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:32:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [05:57:46] New review: ArielGlenn; "I'd prefer we choose one and stick to it (string OR boolean but not both)." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62583 [05:58:42] New review: ArielGlenn; "will +2 once dependencies are straightened out" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62603 [06:03:59] AaronSchulz: https://bugzilla.wikimedia.org/show_bug.cgi?id=48164 is tracking the bug btw [06:04:35] PROBLEM - SSH on searchidx2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:24] RECOVERY - SSH on searchidx2 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [06:06:45] morebots is dead! <-- I believe this more effective than linking a bug few will click [06:07:10] So is StewardBot in -stewards [06:07:21] 02.20 -!- morebots [~morebots@wikitech-static.wikimedia.org] has quit [Ping timeout: 264 seconds] [06:08:15] completely unrelated: other code, other cluster and 06.35 -!- StewardBot [stewardbot@wikimedia/bot/StewardBot] has quit [Excess Flood] [06:08:45] you will need to talk to someone with access to wikitech-static to reboot moreboots (eg: TimStarling for one) [06:09:36] talk to ori-l, he was the one who specifically asked me to not merge the fix for its periodic failures [06:10:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:38] i dutifully harassed the person who committed to doing it earlier and the latter said they'd update the patch on saturday after their last exam [06:11:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:11:46] TimStarling: having the logs logged, even sporadically, would be better in this case, no? [06:13:36] legoktm: would you mind if i touched up your patch? i wouldn't ask if morebots's current broken state was not so disruptive [06:13:50] go for it [06:13:56] !log restarted adminbot (morebot) on wikitech-static [06:14:05] Logged the message, Master [06:14:37] thanks. i think the code is kind of crappy so there will be plenty left to do. [06:14:58] :P [06:15:32] the bot's code, i mean. your patch is fine other than the issues i pointed it. [06:15:38] * out [06:15:47] right :) [06:22:28] I guess I killed the wrong one [06:23:37] apergos: maybe next time you restart it, you should kill the one that is running already first [06:24:51] I did stop it (but it claimed the process was not running) [06:25:50] and when I looked to see what was running, after starting it, there was only the one... [06:27:14] well, I suppose it's possible that we both stopped it simultaneously [06:29:46] guess so [06:45:22] !fakelog hrm [06:45:23] Logged the message, Python nerd [06:45:36] :>>> [06:49:54] New patchset: Ori.livneh; "Convert logbot to use ircbot.SingleServerIRCBot" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60240 [06:50:59]