[00:02:52] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:04] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:08] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:32] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:36] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:38] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:40] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:16] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:24] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:40] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:04] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [00:11:30] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [00:11:38] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:12:16] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [00:12:25] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:41] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:51] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:05] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:31] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:41] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:21:55] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpDDv [00:21:56] [02miraheze/puppet] 07Southparkfan 031f8e43e - Remove my mail alias [00:22:05] \o/ SPF|Cloud [00:53:15] Hello LakesideMiners! If you have any questions feel free to ask and someone should answer soon. [02:02:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpDQZ [02:02:43] [02miraheze/mw-config] 07paladox 0325bcb21 - Set wgCommentTableSchemaMigrationStage to MIGRATION_NEW [08:33:14] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6627 [10:59:15] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:09:16] !log dropping db centriswiki the wiki was removed a year ago under the DP [17:09:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:11:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpyQT [17:11:58] [02miraheze/mw-config] 07paladox 03e487fdc - Update LocalSettings.php [17:18:08] !log dropping db metaautonomywiki (wiki was deleted a while ago) [17:18:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:20:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpyQj [17:20:29] [02miraheze/mw-config] 07paladox 0316298cf - Fix wgMediaViewerEnableByDefault default value [17:21:31] !log dropping db rezeroswiki (wiki was deleted a while ago) [17:21:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:31:15] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6572 [18:56:07] i managed to create the XML dump of my old wiki now.. after heavily upgrading old stuff [18:56:24] can i also get my images imprted though? those surely arent in the XML? [18:56:48] do i need to reupload , maybe by bot? [18:58:04] basically, there's some tips on https://meta.miraheze.org/wiki/Moving_a_wiki_to_Miraheze#Caveats [18:58:05] Title: [ Moving a wiki to Miraheze - Miraheze Meta ] - meta.miraheze.org [18:58:17] however, if the files are publicly visible, you can upload them by URL [19:00:45] Voidwalker: thanks! [19:02:06] it answered other questions i had already [19:02:29] yea, the user part kind of sucks but of course you cant import those, cant expect that [19:03:06] so either phab ticket or try the wikiteam tool. gotcha [19:03:37] yeah, though we've had issues with the wikiteam tool's dumps before [19:14:51] paladox: can you give me admin powers on my wiki? [19:14:53] You do not have permission to import pages from another wiki, for the following reason: [19:14:57] The action you have requested is limited to users in the group: Administrators. [19:14:59] mutante you should have admin perms [19:15:00] i am logged in [19:15:01] oh, sorry [19:15:10] i am wrong, i should not try this on meta [19:15:12] all users who request the wiki are the admin of that wiki :) [19:15:16] ah yeh [19:15:39] ok, i hae my XML file and the right URL [19:15:42] it is 235M [19:15:55] so it depends how much i can upload before the timeout?:) [19:15:58] the xml dump is 235M ? [19:16:02] yea [19:16:09] yeh we will need to import that one through the command line :) [19:16:11] is that small or large? [19:16:32] well, i think i may be able to push it within 30 seconds [19:16:32] that will most definitely hit the timeout we have set if done from the UI [19:16:35] mutante large i think [19:16:37] if i use ethernet here [19:17:11] i notice the "import from another wiki" but already have the dump [19:17:22] and my other wiki is not reachable via https right now [19:17:29] but i could fix that part [19:17:55] that feature won't work for you [19:17:56] should i just try it anyways or does that cause more issues if it hits timeout [19:18:02] only allows importing from meta / templatewiki [19:18:03] i have pretty good internet [19:18:06] currently [19:18:13] oh, ok [19:18:16] mutante yeh, but it's how fast the server can process the xml dump [19:18:21] not internet connection :) [19:18:45] ok, do you have some place i can scp a file to > [19:18:48] you could upload a 1GB file in 1s, but the server could can from hours to days [19:18:52] gzipped it is much smaller [19:18:58] mutante phabricator [19:19:07] via https? [19:19:09] https://phabricator.miraheze.org/file/ [19:19:11] Title: [ ⇪ Query: All ] - phabricator.miraheze.org [19:19:13] yup [19:19:13] or you mean the actual ssh server [19:19:18] ok [19:19:22] via https [19:19:29] on the command time it will work as it has no timeout [19:21:24] paladox: i am not sure the xml file should be public though, contains user table? [19:21:39] mutante you can set it so only certain people can view it i think [19:21:53] via the Visable to option [19:21:59] tries [19:22:29] paladox: where will you actually need the file in the end [19:22:43] i am thinking if you do this for all imports.. and you keep dumps for all wikis in phab [19:22:49] that will make phab install really large [19:22:56] we keep the file on misc4 [19:22:58] but it's ok [19:23:02] you can delete it i think? [19:23:05] maybe better to allow to upload with scp? [19:23:16] in some unprivileged incomoing folder [19:23:20] with "scponly" shell [19:23:30] and then you guys can delete it after import [19:23:42] just thinking out loud about the general process and not just my case [19:23:45] hmm [19:23:57] though in phab you can delete the file [19:24:23] not sure , but most applications dont actually delete stuff and only hide it because db consistency and stuff [19:24:26] we have 49G left on misc4 [19:24:32] and deleteing from db breaks stuff. but shrug [19:24:55] ok [19:25:01] gzips it though [19:25:15] :) [19:31:26] eh.. this is weird, i was just logged in on meta and it was fine [19:31:38] and then i tried to use the same password on phab [19:31:52] and it didnt' work, then i logged out of meta and back in and that fails too [19:32:01] No active login attempt is in progress for your session. [19:32:08] clear your cookies [19:32:12] ok [19:32:13] for miraheze [19:34:30] on other browser i never used for this before: login on meta works, login on phab doesn't work but just the normal "incorrect pass" [19:34:38] expectation is though that it works on both [19:35:05] tried "Username or password are incorrect. [19:35:10] i can reset your pass on phab i think [19:35:13] * paladox looks [19:35:15] tried both user name and email address [19:35:39] i can click forgot pass too [19:35:40] paladox: import from another wiki can be changed per wiki FYI [19:35:45] just reporting because you said LDAP [19:35:45] (In ls) [19:35:53] ok [19:35:59] mutante we use Oauth [19:36:02] not ldap :) [19:36:07] mutante ah yeh click forgot pass [19:36:13] it will send it to your email i think [19:37:00] paladox: i got 2 emails from phab. one says "welcome to phab" and the other says "password reset" . 6 minutes in between, heh [19:37:06] ah [19:37:07] ok [19:37:11] "welcome to phab" [19:37:12] was me [19:37:14] mutante: are you clicking the login with MediaWiki button? [19:37:15] ok [19:37:48] MacFan4000: no:) [19:37:58] yea, it makes sense [19:38:27] mutante: you have to click that button the login to phab [19:38:30] allows Oauth, now it asks for the email verification step [19:38:34] yea, did [19:39:29] i wonder what happens if i click the password reset for phabricator now, but i wont [19:39:47] waits for verification mail [19:40:10] clicks "send another email" button [19:40:22] now has email [19:40:48] logged in on phab with oauth now. thx [19:41:05] Or you can also attach a github account and login with it [19:43:52] ok, thanks [19:44:11] paladox: so regarding "files too large for using Special:Import should be on Phabricator"... [19:44:17] yup [19:44:18] Uploaded file is too large: current limit is 10M. To adjust this limit change 'upload_max_filesize' in php.ini. [19:44:20] :p [19:44:22] oh [19:44:31] how big is it? [19:44:34] after being gzipped? [19:44:43] 34 [19:44:47] oh [19:44:57] 7zip? [19:44:59] rar ?:) [19:45:02] yeh try 7zip [19:45:04] scp to your desktop :) [19:45:15] heh, you can send me the file through irc if you want :) [19:45:31] that is fun, i will actually try right now [19:45:41] heh [19:45:43] but i will be surrprised if your side receives it [19:45:48] due to NAT and firewall [19:45:48] why? [19:45:49] :) [19:45:55] i doin't block firewalls [19:45:56] because it's peer2peer [19:46:03] and yoi have to allow my IP to connect to you [19:46:06] at some port [19:46:11] ok, hold on :) [19:46:32] scp'ing it to where my IRC client runs [19:46:37] ok [19:48:06] lol, you tried to send a screenshot [19:48:08] i see it now [19:48:15] heh [19:48:40] 14:48 DCC SEND request sent to paladox: s23wiki_20181207.xml.gz [19:48:47] mutante try again [19:48:56] i clicked the notification [19:49:01] done [19:49:20] 14:48 Medium risk threat [#miraheze] - mutante - using the DC.C SE.ND exploit; ping reception123, paladox, macfan4000, spf|cloud, and puppykun !att-#miraheze-medium https://bfnt.io/u2Qr5jD [19:49:22] Uh [19:49:27] wow,it's actually sending [19:49:40] lol, no, i am just sending him the file [19:49:57] mutante downloaded now [19:49:59] :P\ [19:50:07] i am impressed :) [19:50:26] it was 100% legit use of dcc send [19:50:36] * paladox uses scp now [19:50:37] it's just rare that people still use it [19:51:03] [33MB] for paladox in 00:00:25 [1392.14kB/s] [19:51:16] :) [19:51:48] it's uploaded to mw1 now! [19:54:02] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=s23wiki --report=1 --uploads s23wiki_20181207.xml on mw1 [19:54:04] mutante ^^ [19:54:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:54:18] [235f86487b2e1d444e16276f] [no req] CannotCreateActorException from line 2456 of /srv/mediawiki/w/includes/user/User.php: Cannot create an actor for a usable name that is not an existing user [19:54:19] Backtrace: [19:55:03] paladox: that's cool and all.. but [19:55:12] since we just talked about the user table NOT being in this dump [19:55:23] it surprises me to see such an error [19:55:27] talking about usable names [19:55:51] i filed https://phabricator.wikimedia.org/T211450 [19:55:52] Title: [ ⚓ T211450 importDump does not work with actor if the user does not exist ] - phabricator.wikimedia.org [19:56:01] mutante it does not contain a user. [19:56:09] It contains the user that created the page / rev [19:56:24] not any users details, just the name that created the rev / page [19:57:29] yea, like the i referecnced [19:57:41] that has an IP / dial-up hostname [19:57:47] when i edited without logging in [19:58:14] back in the days it was default that any page can be edited without logging in at all [19:58:25] that used to be normal for a wiki [19:58:59] would specifying --username-prefix have an impact? [19:59:06] * paladox looks [19:59:26] Voidwalker nope [19:59:28] still fails [19:59:56] apparently it's being fixed [19:59:57] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/476508/ [19:59:58] Title: [ Gerrit Code Review ] - gerrit.wikimedia.org [19:59:59] Voidwalker mutante ^^ [20:04:11] paladox: ariel says this is "really really weird" [20:04:14] for mediawiki [20:04:19] yeh [20:04:21] the part in my dump [20:04:23] it's caused by actor [20:04:31] it would normally never do DNS reverse lookips [20:04:34] for [20:04:37] but i have these: [20:04:41] dial-213-168-92-80.netcologne.de [20:04:54] that was my IP in 2003 or whatever [20:05:00] i mean, the hostname [20:05:05] of my dial-up internet connection [20:05:07] it's not doing reverse lookup [20:06:39] omg, you linked to an anomie patch [20:06:42] that means mw core [20:06:48] and rabbit hole [20:06:55] yep [20:07:00] more "this is weird" comments from people who really know their stuff [20:07:02] ough [20:07:55] i also have these: [20:07:56] (logged).netcologne.de [20:08:07] where it hides the actual IP [20:09:05] paladox: do we want to try and patch User.php ?:o [20:09:08] with https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/476508/2/includes/user/User.php [20:09:09] Title: [ Gerrit Code Review ] - gerrit.wikimedia.org [20:09:14] yep [20:09:18] im trying that now [20:09:22] but make sure reverting is easy [20:09:24] cool [20:22:15] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpSJD [20:22:16] [02miraheze/mediawiki] 07Brad Jorsch 031673ac8 - User: Bypass repeatable-read when creating an actor_id When MySQL is using repeatable-read transaction isolation (which is the default), the following sequence of events can occur: 1. Request A: Begin a transaction. 2. Request A: Try to select the actor ID for a user. Find no rows. 3. Request B: Insert an actor ID for that user. 4. Request A: Try to insert an [20:22:16] actor ID for the user. Fails because one exists. 5. Request A: Try to select the actor ID that must exist. Fail because of the snapshot created at step 2. In MySQL we can avoid this issue at step #5 by using a locking select (FOR UPDATE or LOCK IN SHARE MODE), so let's do that. Bug: T210621 Change-Id: I6c1d255fdd14c6f49d2ea9790e7bd7d101e98ee4 [20:30:11] paladox: i tried with Special:Import. fyi it did NOT actually fail due to size or timeout [20:30:22] it failed with teh same error you got .. after uploading [20:30:22] oh [20:30:28] Import failed: Cannot create an actor for a usable name that is not an existing user [20:31:24] paladox: 14:35 <+MacFan4000> paladox: import from another wiki can be changed per wiki FYI [20:31:30] is that the last chance? [20:31:39] or does it also just copy the same data [20:31:49] and will have the same issue [20:32:14] it will fail with the same error [20:32:51] It would also require adding an Interwiki prefix [20:32:57] i wonder how far back in mediawiki versions this still worked [20:33:05] at some point it must have [20:34:06] and also if this happens to ANY xml import currently or just because the one i have that is old [20:34:22] This is broken becuase we use actor now [20:34:25] one would mean mw upstream is broken to restore from dumps [20:34:32] that would seem pretty important [20:35:18] paladox: do you know when it started? [20:35:28] when we migrated to actor [20:35:33] a month ago [20:36:31] hmm. is that recommened from upstream ? [20:36:40] then it seems like a major bug [20:36:42] not sure :) [20:36:44] it [20:36:50] *it should be affecting the wmf too [20:36:54] what is "actor" [20:37:00] i see the tables with that name [20:37:12] https://www.mediawiki.org/wiki/Manual:Actor_table [20:37:13] Title: [ Manual:actor table - MediaWiki ] - www.mediawiki.org [20:37:19] yea, that's what i saw too [20:37:24] but what does it do [20:37:38] when you say "migrate to actor" did you mean "schema change" [20:38:14] ok, so it's all just mw core changes then? [20:38:14] well it copied data from a table into the actor thing [20:38:21] yeh [20:38:26] ok, i see , yea [20:38:51] i think it needs a wmf phab ticket [20:39:34] https://phabricator.wikimedia.org/T210621 [20:39:35] Title: [ ⚓ T210621 Internal api error: CannotCreateActorException ] - phabricator.wikimedia.org [20:39:52] mutante https://phabricator.wikimedia.org/T211450 [20:39:54] Title: [ ⚓ T211450 importDump does not work with actor if the user does not exist ] - phabricator.wikimedia.org [20:40:24] i guess it must be because of "dial-xxx-x-xxx-xx.netcologne.de" [20:40:43] since it worked for other wikis [20:46:14] paladox: i commented and then deleted it again :p [20:46:20] ok [20:46:21] heh [20:48:51] paladox: in other wiki xml dump files.. do they never have an IP address in there [20:48:58] yeh [20:49:07] i think it's choking on the checks [20:49:11] actually it's not an IP address in my case [20:49:15] it's a hostname containing an IP [20:49:31] and that is the strange part [20:49:43] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://git.io/fpSkt [20:49:45] [02miraheze/ssl] 07MacFan4000 03e5482ca - Create gu.gyaanipedia.co.in.crt [20:49:47] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://git.io/vxP9L [20:50:15] MacFan4000 maybe we should do a wildcard cert for gyaanipedia.co.in [20:50:30] maybe [20:51:55] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://git.io/fpSkZ [20:51:56] [02miraheze/ssl] 07MacFan4000 030b7072a - Update certs.yaml [20:52:16] [02ssl] 07MacFan4000 opened pull request 03#111: add a cert - 13https://git.io/fpSkc [20:52:22] [02ssl] 07MacFan4000 closed pull request 03#111: add a cert - 13https://git.io/fpSkc [20:52:23] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fpSkC [20:52:25] [02miraheze/ssl] 07MacFan4000 0323c621e - add a cert (#111) * Create gu.gyaanipedia.co.in.crt * Update certs.yaml [20:52:58] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://git.io/vxP9L [20:52:59] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [20:56:56] paladox: ok, i cleaned up the XML file and replaced all that with just "netcologne" [20:57:09] paladox: can we try that. new file transfer incoming [20:58:19] ok [20:58:29] mutante you may want to make it a ip :) [20:58:38] though that's better! [20:58:55] paladox: but there are lots of normal user names in that field too [20:59:06] and clearly they cant all exist on mirahez [20:59:15] so it should not matter as long as no special cahracters [20:59:17] i was hoping [20:59:44] oh you mean if that user has 0 edits in mediawiki. yea [21:00:27] yea, netcologne does not exist in user table... makes you wonder how thiw wiki was ever running though :) [21:00:30] and i know it was [21:00:38] still fails mutante [21:00:43] same error? [21:01:28] !log depool mw1 [21:01:30] mutante yes [21:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:02:00] ok, sigh. i blame either Miley Spears or Illuminat [21:02:10] jk [21:03:13] paladox: oh, wait, i see i missed some of those weird usernames [21:03:25] ip67-153-93-135.z93-153-67.customer.algx.net [21:03:36] man.. how to clean this [21:06:28] there is also stuff like 82.83.226.xxx with the literal x.x.x in the XML .. sigh sigh [21:06:42] how do i know what is invalid and what is not [21:06:48] and why did ancient mediawiki do this :) [21:13:05] mutante found it [21:13:07] i think [21:13:07] > var_dump( User::isUsableName( 'Mutante' ) ); [21:13:07] bool(true) [21:13:55] paladox: oh :) cool [21:15:58] mutante lowercase everything [21:16:06] anything that starts with a uppercase [21:16:08] lowecase it :) [21:17:01] wait, really. usernames in MW have always started Uppercase [21:17:04] like page names [21:17:20] you just said Mutante = true [21:17:27] what is also uppercase [21:17:39] that [21:18:00] heh [21:18:04] that's what the check is doing [21:18:08] dosen't like uppercase [21:23:48] "isUsable" = true sounds good, not bad [21:24:19] try that with one of the netcologne strings from the original [21:24:50] or var_dump( User::isUsableName( '82.83.226.xxx' ) ); [21:24:51] ? [21:25:24] var_dump( User::isUsableName( 'ip67-153-93-135.z93-153-67.customer.algx.net' ) ); [21:26:32] i think those work, im not sure why it's getting caught in actor.... [21:27:10] so it says "bool(true)" for all that, just like for "Mutante" ? [21:27:19] then we followed a red herring [21:27:31] i guess.. uhm. be back after lunch [21:27:37] thanks for trying so far [21:28:08] can you find something that does NOT return "bool(true)" ? [21:29:21] mutante yeh the ips [21:29:27] and anything lowercased. [21:53:56] mutante it's importing! [21:54:01] i found where the issue is [22:36:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [22:37:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb [22:39:06] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [22:40:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:41:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:41:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:01:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [23:03:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:12:48] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [23:13:22] paladox: omg, thank you [23:13:31] i hope that ^ wasnt related, heh [23:13:34] your welcome :) [23:13:35] paladox: how did you do it [23:13:47] mutante i changed https://github.com/wikimedia/mediawiki/blob/REL1_31/includes/user/ExternalUserNames.php#L77 [23:13:48] Title: [ mediawiki/ExternalUserNames.php at REL1_31 · wikimedia/mediawiki · GitHub ] - github.com [23:13:50] to use a prefix [23:14:09] you are the best, paladox [23:14:16]