[01:04:24] there is no rc channel for wikimania sites? [01:04:44] There is [01:05:07] I don't remember what it is [01:05:22] i tried #wikimania2013.wikimedia and it did not work [01:06:30] There used to be, anyways. There was a freenode mirror for RC patrol [01:09:16] gosh, it is under #wikimania2013.wikiPedia [02:23:27] !log LocalisationUpdate completed (1.20wmf7) at Fri Jul 20 02:23:27 UTC 2012 [02:23:38] Logged the message, Master [02:23:39] !log LocalisationUpdate completed (1.20wmf6) at Fri Jul 20 02:23:39 UTC 2012 [02:23:47] Logged the message, Master [18:33:34] I just made an edit on mediawiki.org and was asked "Re-review this revision Accuracy: [ ] Sighted ". In the land of the blind, can I ask a one-eyed man what "Sighted" means here? [18:33:59] spagewmf: reviewed [18:35:56] hoo, I'm pretty sure that's an incorrect use of http://www.merriam-webster.com/dictionary/sighted [18:36:44] spagewmf: mhm, probably... maybe a German did that, as "gesichtet" in German means seen [18:38:08] * jeremyb wonders why spagewmf doesn't lose the wmf on the end of his nick? [18:38:15] also, hi! willkommen [18:38:40] tervetuloa [20:00:11] evening guys, just a quick check - are you doing anything atm which affects uploads at all? [20:04:06] BarkingFish: hwat are you seeing? [20:05:30] just issues when trying to upload a pic to enwp - i get a message back from firefox "connection to the server timed out while attempting to contact upload.wikimedia.org" [20:05:51] yet the net is fine, i can nav wp perfectly well, I have other windows and tabs open, plus irc and decent throughput [20:08:43] hrmm, well i have to run. maybe someone else will try to repro or investigate [20:24:04] BarkingFish: where are you? [20:24:30] UK [20:36:10] BarkingFish: I did manage to pload a file, how big is yours? [20:36:35] 88.2kb [20:36:41] hold that thought, i have to reboot [20:36:45] back in a few moments [20:40:47] oh, it was en.wiki [20:40:56] * Nemo_bis doesn't even consider it as an option for uploads [20:47:06] sorry about that, had some stuff to fix this end [20:47:35] so you were saying you managed to get a file to upload, yes, Nemo_bis? [20:48:34] I'm mystified then, cause as I say, everything net related this end is working like a charm. I have 2 browsers open, numerous tabs in both, and widgets with net connections, AND xchat, and I'm still not seeing any lag. Why I can't upload is a mystery [20:49:13] and not forgetting kmail doing an auto-mail collection every 5 minutes too :) [21:11:32] BarkingFish: sorry, Nemo_bis> oh, it was en.wiki / Nemo_bis doesn't even consider it as an option for uploads [21:12:00] but if it u.wm.o maybe it doesn't matter? [21:12:25] ah, well what I'm doing at the moment is clearing a backlog of images which have watermarks - by removing the watermarks and reuploading them [21:12:38] it matters, because I can't get anything to upload [21:21:47] BarkingFish: I mean that uploading on en.wiki or Commons as a test should be the same [21:22:49] i can't upload it to commons, it's a non free image [21:23:18] so by sending it to enwp, it should be the same. I assume you tested by uploading to commons [21:42:25] Hi all. May I ask a question? Well, maybe a few :) How and who actions on these alerts? Are they automatically actioned or manually done? I'm quite new to the place and was wondering if I could help as a volunteer. Thanks. [21:53:02] adminxor, which alerts? [21:54:12] Disk usage, service being in maintenance mode, etc [21:55:03] that's handled by ops [21:55:25] you could help out there [21:55:35] take a look at the configs: https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git [21:56:20] Thank you. [22:02:21] I keep being disconnected :( [22:02:29] > there are a few other operations/ repos, but that's imho the most important one [22:02:32] > you can provide patches with a gerrit account [22:05:46] Platonides: How do I start, I mean first of all I need to have a very good idea about the server roles and the network. Where can I get that information? [22:07:21] there is https://wikitech.wikimedia.org/view/Server_roles but that ptobably won't explain you what each role does [22:07:37] there are three basic roles [22:07:43] the db servers [22:07:49] the apache workers [22:08:18] and the caching (squid/varnish) avoiding that every page view reaches the apaches [22:09:01] those are the basics [22:09:09] Okay. [22:09:14] but there are many other less roles, such as servers providing memcached instances [22:09:23] (many being apaches at the same time) [22:09:30] serving mobile views [22:09:35] https frontends [22:09:44] search backends [22:09:48] generation of dumps... [22:10:24] oh, you will also see mentions to External Storage or ES [22:10:42] Okay. [22:10:48] the content of the pages is not directly stored in the text table of the db, but in a separate cluster [22:10:57] that's the External Storage [22:11:27] I can only provide a thin layer of everything ops manages, but feel free to ask :) [22:11:28] Okay. How about the Solaris servers? I read somewhere that they are used for serving images. [22:11:33] Yeah [22:11:40] At least one of them was moved to Linux I believe [22:11:50] I'm not sure [22:12:10] toolserver has been recently moving a couple of servers from Solaris to Linux [22:13:17] Thank you for being so kind. It's been a couple of weeks and I was still lost. So thought of asking here. [22:13:43] > I'm not sure [22:13:45] > toolserver has been recently moving a couple of servers from Solaris to Linux [22:13:48] > but for dumps... [22:14:16] okay [22:14:17] I understand everything is managed by puppet [22:14:21] yes [22:14:24] Theoretically [22:16:37] > the cool feature of Solaris for storage, is the usage of ZFS and its ZFS snapshots [22:17:35] Absolutely. ZFS is my interest too. [22:17:45] What is the propect for a person who is trying to learn puppet? I have worked mostly in commercial environments but never in a project like this one. [22:20:29] well, you would be providing patches, and ops team would review them [22:20:36] either approving or pointing out issues [22:20:51] about how to learn puppet, I don't really have good tips for you [22:21:00] I'm not a pupper speaker :P [22:22:55] notpeter, maybe you can provide some guidance to adminxor? [22:29:04] Thanks again. So for an example, you see a low disk space alert or a critical service is in maintenance or offline mode, whaat do you do then? Isn't it actioned immediately? Maybe through puppet? [22:30:55] usually, critical things are "the wikis are giving out a php error" or "apache servers can't accept new connections" [22:32:09] Okay. How do you go about it? I'm sorry if I'm asking too many questions. [22:32:21] well, I don't go about it [22:32:32] I wasn't given a shell account [22:32:54] but so far, I think they connect to the server in an interactive session, and try to troubleshoot it [22:33:09] the solution sometimes involve restarting a problematic server [22:33:28] or in case of a db disk-full conditions, it needs to switch the db master [22:33:40] ah, take a look at http://ganglia.wikimedia.org/ [22:34:04] Thank you! [22:34:19] for requesting a gerrit account: https://www.mediawiki.org/wiki/Developer_access [22:34:53] there's also a testing cluster, called labs [22:34:54] https://labsconsole.wikimedia.org/wiki/Main_Page [22:35:22] it's intended for testing things before productions, with more liberal access [22:35:43] although that's not how it works so far [22:37:55] the files of mediawiki and its configuration are spreade by rsync [22:38:02] I have an account for wiki labs as suggested by Ryne_Lane. But, I have no idea how to use it. I was expecting some kind of a shell account. [22:38:08] Okay [22:38:27] a labs account gives you access to gerrit at the same time [22:38:46] yes, you should have shell access to bastion.wmflabs.org [22:38:56] which is not any interesting [22:39:19] it's just a bastion host before jumping to your project instances [22:39:55] labs contain several projects [22:40:11] then you can create/delete virtual machines there [22:40:25] and have shell access, including sudo [22:40:35] Oh really? Awesome! [22:43:21] bastion.wmflabs.org - does it authenticate only through ssh keys? [22:43:49] yuo [22:43:50] yup [22:44:17] you add it through labsconsole [22:44:29] How do I plant the public key there? [22:45:33] https://labsconsole.wikimedia.org/wiki/Special:NovaKey [22:51:53] Thank you, Reedy. I have uploaded my public key there. [22:52:13] But I get this message and it does not connect [22:52:14] $ ssh adminxor@bastion.wmflabs.org [22:52:14] If you are having access problems, please see: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [22:52:14] Connection closed by 208.80.153.207 [22:53:54] Because you aren't/weren't a member of the bastion project [22:53:55] just added you [22:54:03] need to wait for stuff to run and generate you a home dir [22:54:20] Thank you so much! [22:54:21] IIRC the labs bot will announce it's been done [22:54:41] Okay. I'll wait for it then. [23:03:58] Reedy: I'm still facing problem. I enabled ssh-agent, added the private key by ssh-add. Is it not finished yet? [23:06:26] RoanKattouw: ping [23:06:43] should be.. bot has announced it [23:09:48] Still getting the same message and getting disconnected. [23:11:17] try connecting -v [23:14:57] does not it support 2048 bit RSA key? [23:15:02] I'm getting this [23:15:03] $ ssh -Av adminxor@bastion.wmflabs.org [23:15:03] OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011 [23:15:03] debug1: Connecting to bastion.wmflabs.org [208.80.153.207] port 22. [23:15:03] debug1: Connection established. [23:15:03] debug1: identity file /home/adminxor/.ssh/id_rsa type 1 [23:15:04] debug1: identity file /home/adminxor/.ssh/id_rsa-cert type -1 [23:15:06] debug1: identity file /home/adminxor/.ssh/id_dsa type 2 [23:15:08] debug1: identity file /home/adminxor/.ssh/id_dsa-cert type -1 [23:15:10] debug1: identity file /home/adminxor/.ssh/id_ecdsa type -1 [23:15:13] debug1: identity file /home/adminxor/.ssh/id_ecdsa-cert type -1 [23:15:14] debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3ubuntu7 [23:15:16] debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH* [23:15:20] debug1: Enabling compatibility mode for protocol 2.0 [23:15:22] debug1: Local version string SSH-2.0-OpenSSH_5.9 [23:15:24] debug1: SSH2_MSG_KEXINIT sent [23:15:26] debug1: SSH2_MSG_KEXINIT received [23:15:28] debug1: kex: server->client aes128-ctr hmac-md5 none [23:15:30] debug1: kex: client->server aes128-ctr hmac-md5 none [23:15:32] debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent [23:15:35] debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP [23:15:36] debug1: SSH2_MSG_KEX_DH_GEX_INIT sent [23:15:38] debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY [23:15:40] debug1: Server host key: RSA 26:02:b9:20:f7:6c:5c:c8:2d:58:a2:4c:27:f8:f5:a7 [23:15:42] debug1: Host 'bastion.wmflabs.org' is known and matches the RSA host key. [23:15:44] debug1: Found key in /home/adminxor/.ssh/known_hosts:3 [23:15:46] debug1: ssh_rsa_verify: signature correct [23:15:50] debug1: SSH2_MSG_NEWKEYS sent [23:15:52] debug1: expecting SSH2_MSG_NEWKEYS [23:15:54] debug1: SSH2_MSG_NEWKEYS received [23:15:56] debug1: Roaming not allowed by server [23:15:58] debug1: SSH2_MSG_SERVICE_REQUEST sent [23:16:00] debug1: SSH2_MSG_SERVICE_ACCEPT received [23:16:02] If you are having access problems, please see: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [23:16:03] um [23:16:05] debug1: Authentications that can continue: publickey [23:16:06] flood [23:16:07] debug1: Next authentication method: publickey [23:16:09] debug1: Offering RSA public key: /home/adminxor/.ssh/id_rsa [23:16:11] debug1: Server accepts key: pkalg ssh-rsa blen 279 [23:16:13] Connection closed by 208.80.153.207 [23:16:14] Dude, put it on pastebn [23:16:22] Pastebin is ur friend [23:17:16] Was probably just about done at that point anyways [23:17:53] adminxor: Please use a pastebin next time [23:18:06] oops sorry! [23:25:01] is there anyone who could look at https://gerrit.wikimedia.org/r/#/c/16194/ ? it's a sync to master of a deployed extension; it needs to be sync'd to testwiki [23:35:21] ori-l: done [23:35:32] Reedy: yessss! thank you [23:35:36] Thanks Reedy. Especially on a Friday. :) [23:36:52] It's Saturday ;)