[14:01:07] Hello, something is wring with phabricator, I think. Do you see this too? [14:03:17] yes, it's discussed at #wikimedia-operations for whatever reason [14:03:35] oh, ok, thanks [14:28:23] More users reporting issues which sound like DNS or networking, like yesterday's wikisource-l https://it.wikipedia.org/wiki/Wikipedia:Bar/Discussioni/Accesso_agli_altri_progetti_bloccato [15:13:24] phab is loading stuff from wikivoyage.org?? [15:15:24] * legoktm sees -operations [18:19:20] I can't reach any Wikimedia sites right now. My access provider is Century Link in Saint Paul, MN. Other sites work fine. According to http://www.isitdownrightnow.com/wikipedia.org.html only Century Link has the problem. [18:19:36] Traceroute stops at a *.gtt.net address. Where's the best place to report this? [18:23:22] According to one report at http://www.isitdownrightnow.com/wikipedia.org.html this has been going on for 45 hours. [18:25:20] Here's the complete tracert output: $ tracert en.wikipedia.org Tracing route to en.wikipedia.org [208.80.154.224] over a maximum of 30 hops: 1 2 ms 1 ms 2 ms 192.168.0.1 2 44 ms 41 ms 44 ms 207.225.140.237 3 44 ms 48 ms 56 ms mpls-agw1.inet.qwest.net [75.168.229.97] 4 54 ms 54 ms 59 ms chp-brdr-03.inet.qwest.net [67.14.8.194] 5 55 ms 51 ms 55 ms ae5.chi12.ip4.g [18:26:16] ms 78 ms xe-0-3-1.was10.ip4.gtt.net [89.149.128.141] 7 * * * Request timed out. 8 * * 299 ms text-lb.eqiad.wikimedia.org [208.80.154.224] Trace complete. [18:27:07] Is there some email address to report issues like these? [18:27:39] AxelBoldt: best to file a bug. [18:27:46] if you can't access phabricator, i think you can file by email [18:28:51] AxelBoldt: from https://www.mediawiki.org/wiki/Phabricator/Help#Using_e-mail : You can also create tasks by sending e-mail to task@phabricator.wikimedia.org. The subject will be used as task title, the body will be used directly as is, and attachments will be included on the task. To select a project, use its hashtag somewhere in the body, e.g., #mediawiki-core. [18:28:54] Note: if your e-mail signature is not formatted following de facto standards (including "-- " to be separated from the body text) it will be posted as well. You are encouraged to remove private information from your signature in your first attempt, just to be sure. [18:29:27] (this probably requires that you have a Phabricator account associated with your email) [18:31:28] Thanks MatmaRex. The sites just came up again, so I'll wait to see if it happens again. [18:42:46] MatmaRex: the sites are down again, so I will file now. Could you tell me the hashtag of the relevant project to use in the bug report? [18:43:06] probably #operations is fine [20:51:17] Hi on uploading I get a chunk-too-small error with the message ”Minimum chunk size is 1024 bytes” [20:51:31] Which is weird as I can upload the first chunk with only 512 bytes [20:56:35] xzise: i believe the last chunk you upload can be smaller, and only the last chunk. [20:56:43] ^^ [20:56:48] but it fails on the second chunk [20:56:56] if the first one you uploaded was smaller than the minimum, well, then you used up your get-out-of-jail-free card [20:57:07] anomie: ^ [20:57:35] xzise: out of curiosity, what are you using to upload? UploadWizard uses 5 MB chunks [20:57:38] It has to be added recently (I'm testing on test.wikipedia.org) and a few days ago it works [20:57:43] Pywikibot [20:57:53] yes, it was changed recently [20:58:06] And I'm testing uploading and so I want small chunks [20:58:09] and choose 512 bytes [20:58:53] Afaik the file is large enough that I could bump it up to 1024… so it's mainly about the inconsitency [21:00:15] i think it's just what i said, the last chunk (or just one of the chunks?) is allowed to be smaller, since otherwise you'd have no way to upload a 1025 byte file with chunked uploads. [21:00:29] anomie would know for sure (or know if it's a bug), but it seems he's not online right now. [21:00:42] xzise: want to file a bug? [21:00:47] Ah … maybe it's one of the chunks [21:01:07] I first wanted to determine if I have overlooked something [21:01:25] So I could test if it's one of the chunks [21:01:37] (as opposed t the last only) [21:05:16] Mah that file is barely larger than 1024 so I can't test if 1024+rest+1024 works [21:08:03] Hmm no the second chunk cannot be < 1024 even if the first was >= 1024 byts [21:30:14] MatmaRex, done: https://phabricator.wikimedia.org/T112405 [21:52:12] okay great the next problem: I can't finish the upload as the server thinks that the filekey is not valid while query+stashimageinfo returns data for that filekey ?1 [21:58:41] jynus emailed me the other day making it an RfC-written goal to turn on SQL strict mode. Does anyone know if he publicly documented/discussed that desire anywhere? [22:01:57] Okay I get the "stashfailed" error with the message "No chunked upload session with this key" [22:02:00] what does that mean? [22:05:05] robla, there's a ticket somewhere [22:05:31] robla, https://phabricator.wikimedia.org/T108255 [22:05:54] * robla looks [22:06:29] Krenair: thanks! that's pretty much what I'm looking for [22:06:32] Doesn't mention RfC goals though [22:06:34] great [22:07:50] I'm happy to tweak it into an RfC (I may just slap the RfC tag on it and call it good) :-) [22:21:54] thanks for your addition Nemo_bis! I used it in my update to https://phabricator.wikimedia.org/T108255 [22:22:41] Sorry for any edit conflict. :) [22:23:53] robla what do you refer as RFC at wikimedia? [22:26:16] I found out [22:26:19] Toordog: depends on who you ask :-) [22:26:35] RFC for me is related to the RFC entity specifying engineering standard ;) [22:26:49] i saw you using it in a more in house way. Request For Comment [22:26:53] which is new to me ;) [22:27:15] Toordog: yeah, our naming causes confusion for me too, especially since I've been involved in the IETF RFC process [22:27:53] renaming is something I'd like to do, but it's not at the top of my priority list [22:27:53] with the context, it is less confusing indeed :) [22:28:26] it only make people ask questions which isn't bad in the end [22:28:48] and it gives me the opportunity to say hi [22:29:07] o/ [22:29:47] I'm trying to get familiar with the IT infrastructure at wikimedia, technology in used, challenge you face, [22:30:26] Toordog: cool! what are you working on (more broadly)? [22:31:09] I'm looking forward to maybe join the team if my interview goes well with Faidon. I try to prepare myself at best for it. [22:31:57] Currently, i'm an IT consultant in Montreal, working as IT architecte and senior sysadmin Unix/LInux. But mostly in more traditional company like banking, aerospatial and in the past in big web company [22:34:36] great! good luck with your interview! lurking and asking clarifying questions here is a great strategy [22:35:22] Thanks! yes, i found out quite randomly about the channel while i was on ##linux [22:35:43] just make sense in the end to coordinate wikimedia engineering effort from here [22:38:11] robla what is the pseudo of mark? [22:39:50] Toordog: "mark" :-) He's over on #wikimedia-operations . We have lots of channels. [22:40:35] there's a wiki page somewhere that describes them all [22:40:47] it's lonnnng [22:40:52] thanks, his description doesn't make it obviuos which mark he is ;) [22:41:45] Toordog: he probably prefers it that way :-) [22:42:29] he doesn't like fans to poke him all the time? :P [22:43:16] Toordog: it's more about the the very real fear of becoming tech support [22:43:53] "Wikipedia isn't working! What's wrong?" [22:44:04] i'm actually surprised to see the technical back channel open to public [22:44:05] like here [22:44:30] I understand that, been there, and it only slow down resolution of outages [22:45:15] used to own an online company and we used IRC extensively for support and clients contacts. When outage happened, customer was taking over the channel. [22:46:10] it seems the custom here has been to move the party to another channel when too many people learn of the previous one ;) [22:48:38] MatmaRex: yeah :-) I'm starting to reclaim #wikimedia-tech for Architecture stuff because it's become pretty quiet, it's appropriately named, and historically, it *was* the architecture channel [22:50:31] at least, I think it was....now I'm trying to figure out if I remember that right :-) [22:50:31] robla are you involved with the IT architecture let say related to the monitoring infrastructure? [22:52:05] Toordog: answer is longer than I have time for :-) short answer is I'm involved in the overall Wikimedia tech architecture [22:54:20] I should have asked more directly then, Do you know what is the scaling constraint of Icinga? I saw a post of Mark that he is looking for another monitoring solution that would scale better than Icinga. [22:55:17] I was the project lead last year for one of the major bank in Canada, we were auditing Icinga to see if it could realise the monitoring objective of the bank. For me, it looks like Icinga was highly scalable if you architect it from start correctly. I'm curious to know what I might have missed. [22:59:44] there's a wiki page somewhere that describes them all [22:59:45] heh... if only [23:00:14] I've found several new channels simply by watching the repo that people edit to make wikibugs sent things to their channel [23:00:24] Who knows what else might be on Freenode [23:00:34] send* [23:01:54] krenair one could always do a channel list | grep wikimedia [23:01:56] Toordog: your question may be a better one to ask in #wikimedia-operations , since the people that might know the answer are lurking there (I don't) [23:02:13] alright, thanks for the redirection :) [23:02:38] re: grep wikimedia....even that won't work :-) [23:04:28] that wikimedia using IRC extensively remind me of my 90s :P [23:04:36] feel like I'm young again [23:04:54] there's also prefixes for every content project wikimedia runs [23:05:03] Krenair and Toordog: here's the page I mentioned earlier: https://meta.wikimedia.org/wiki/IRC/Channels [23:05:08] now i have to fight with security department to just be able to open some forums *not even talking about IRC ... way above the credential* [23:06:03] there is a channel for vagrant :) [23:06:11] "describes them all" might have been an overstatement :-) [23:06:40] let say most of them [23:06:49] maybe... [23:06:55] it is the kind of wiki page impossible to keep up to date [23:06:55] define "most" [23:07:11] IRC is too alive [23:07:16] morphing, changing [23:07:36] you explained quite well before as channel change and move around