. [18:53:29] Dispenser, I haven't, but your example doesn't tell me much [18:53:43] like are you just trying to match div#id_field1 ? [18:53:44] m_foobar is hungarian [18:53:55] oh [18:54:03] I see what you mean [18:58:50] He’s right about it being very grep-able [19:43:33] Hi there. Did today's deployment came with database issues? [19:44:36] Strainu, at this early stage it sounds like the deployment that was in progress was interrupted by and not responsible for it, but we'll see [19:44:37] No [19:44:49] there may have been other deployments today [19:45:29] Hmm, not sure I understand: are the database issues known? [19:45:38] yes [19:45:46] Ok. [19:46:00] but you were asking about whether it was caused by a deployment [19:46:23] it seems not caused by a deployment [19:46:27] And the error page contains a search engine that leads to Google? [19:46:28] for reference, it's https://phabricator.wikimedia.org/T195293 [19:46:35] we won't know about the cause for sure until we see recovery [19:46:39] and when people figure out the details, they will be noted there [19:46:56] I don't remember that in past outages [19:47:14] It also has encoding issues [19:47:18] hm, wait, no [19:47:19] different types of outages will probably lead to different error pages [19:47:28] aaaaand suddenly there was silence [19:47:32] hey Krenair :D [19:47:38] Here's mine. (Cannot access the database: Cannot access the database: No working replica DB server: Unknown error (10.64.16.84:3318)) [19:47:44] It looked like a phishing page to me [19:47:45] we'll hold that for now [19:47:53] I noticed a complete lack of edits on Huggle, then saw this [19:48:05] this one is MediaWiki breaking at an early stage [19:48:14] folks are shootig connections; we will see if that's all that's needed after the potential fix went around [19:48:19] hi ToBeCloud [19:48:32] https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q182471 <-- Internal Server Error [19:48:34] lol -- can this be in any way be related to the software update pushed strongly by the enwiki admins? [19:48:47] edward, see topic [19:48:51] that would be a wonderful irony (sorry, my humor) [19:48:55] Krenair: thanks [19:49:00] we are in an outage; people are working on it; [19:49:02] task for the outage is https://phabricator.wikimedia.org/T195520 [19:49:11] ToBeCloud, the cause is not entirely known yet [19:49:16] subscribe to it, someone will post the details when we know them :) [19:49:19] a potential fix has been deployed; people are killing existing connections to see if the site recovers [19:49:20] DB down? :o [19:49:26] ok [19:49:27] more information will be available soon [19:49:58] there will almost certainly be documentation on the outage available later [19:51:23] I get IP test edits on Huggle again [19:51:24] :) [19:53:29] So, relating to the error page, how can I find the source for it? I want to investigate the encoding issues to see if they are software related or not [19:53:57] Searching for Google in the source is too generic [19:54:56] Strainu, possible MWExceptionRenderer::reportOutageHTML [19:55:00] possibly* [19:55:12] Seems like nowiki is back, but It reported database problems [19:56:11] Just for the record; Cannot access the database: No working replica DB server: Unknown error (10.64.16.84:3318) [19:57:15] …and the error page is buggy… :D [19:57:27] yes it was almost everywhere https://phabricator.wikimedia.org/T195520 [19:58:47] jeblad: hm, how so? [19:58:50] jeblad: Can you try bypassing your cache? That changed the message for me [19:59:00] investigation going on [19:59:34] ...temporarily. [19:59:40] It is an encoding error. I'll make a screenshot and write a bug report. [19:59:48] i noticed that too. thanks jeblad [19:59:53] @Krenair spot on, thanks [20:00:08] if you take the text and git grep through MW core... that often does the trick [20:00:10] Apparently it was supposed to have a Google search [20:01:10] I know, too bad it doesn't work as well on github [20:01:27] hmm, yeah [20:01:38] solution: don't use github [20:02:01] looks like we don't output or anything like that [20:02:13] so your browser guesses some legacy encoding [20:02:49] doesn't even output a tag, nevermind [20:02:52] the issue might be related to wikidata property suggester / reindex [20:02:52] we still have an outage; it's probably wikidata-related at the root; we're still looking for the source of one of the queries to disable [20:02:55] and is being worked on [20:03:00] more information will be available soon [20:03:16] Oh yeah, it's on the html path only [20:04:00] that's right about encoding issues on the error page. looks like missing charset [20:04:34] For the encoding problem https://phabricator.wikimedia.org/T195525 [20:04:46] thanks [20:04:53] jeblad: can you copy the HTML source there too? i don't get the errors anymore [20:05:02] so i can't [20:05:19] saw that same issue on German error page earlier [20:05:27] still had it incash! [20:06:20] +e [20:10:30] thanks [20:11:35] the call to the query that seems to be the problem, is being hacked temporarily to not isse the query and to return empty results, [20:11:44] so we can have some breathing room to investigate [20:16:41] this fix is still being implemented yet [20:18:04] and it's done, now to see the effects [20:19:14] MatmaRex: Anything else you need before I reload the page? [20:20:45] jeblad: no, thank you, i think it's clear now why that happens [20:21:20] should be fairly easy to reproduce the error locally with the right language + bad DB config, right? [20:21:22] i'm not working on a fix for it right now, but i can do it later/tomorrow if no one else does [20:22:01] yeah, should be. this could probably be an #easy task ;) but the exception reporting stuff is a bit ugly [20:23:06] alright, for those of you who were interested, docs have begun being written at https://wikitech.wikimedia.org/wiki/Incident_documentation/20180524-wikidata [20:27:25] thank you very much :) [20:32:54] anyone still seeing errors? [20:32:56] ToBeFree, strainu edward jeblad [20:38:40] nope, all OK now [20:51:42] reloaded the diff that previously had errors, and it loads correctly [21:09:35] Oh. Everything alright, thank you very much :) [23:08:07] Hi everyone, after today's outage my bot now gets the message "You have made too many recent login attempts. Please wait 2 days before trying again". It does several tasks every few minutes in eswiki... any idea for a solution? [23:11:20] Hi? [23:11:47] jem: make a ticket like https://phabricator.wikimedia.org/T194160 [23:12:08] or actually wait the 2 days, but sounds like that isnt an option [23:12:32] i dont think you were throttled because of the outage specifically.. afaict [23:13:55] (and in the future, change your bot to wait for some time if the login fails, or to save cookies permanently so it doesn't have to log in repeatedly) [23:15:04] Ok, those fails were unexpected and other outages didn't cause that [23:15:31] And no, 2 days are not an option :) [23:16:20] And... that ticket is still open after 16 days... ? [23:17:18] yea, but also the throttle was lifted [23:17:27] Ah, I see [23:17:29] not sure why it's open despite that [23:17:53] and that happened within a day or so [23:18:04] i guess the bot owner needs to confirm [23:18:06] The point is I was going to bed in a few minutes and I hoped to solve it before [23:19:11] there are few things that can happen within a few minutes on IRC only [23:19:41] I know :) But I also know it can depend on finding the right person [23:19:43] and the time of day means even fewer people online [23:19:53] Yes [23:20:10] well, you see who did it on that ticket [23:21:24] I see [23:21:31] But it's not here now [23:21:35] he's* [23:23:46] Ok, I'm going to create the ticket and I hope not too many people will have complained by tomorrow morning :) [23:33:51] Done: https://phabricator.wikimedia.org/T195541