[08:18:03] 06Labs, 10Tool-Labs: puppet failed on tools-bastion-03.tools.eqiad.wmflabs - https://phabricator.wikimedia.org/T138742#2408726 (10valhallasw) [08:20:51] 06Labs, 10Tool-Labs: puppet failed on tools-bastion-03.tools.eqiad.wmflabs - https://phabricator.wikimedia.org/T138742#2408739 (10valhallasw) a:03yuvipanda ``` /var/log/auth.log.1:Jun 23 09:43:50 tools-bastion-03 sudo: yuvipanda : TTY=pts/25 ; PWD=/home/yuvipanda ; USER=root ; COMMAND=/usr/bin/puppet agent -... [13:48:31] 06Labs, 10Tool-Labs: puppet failed on tools-bastion-03.tools.eqiad.wmflabs - https://phabricator.wikimedia.org/T138742#2409211 (10chasemp) 05Open>03Resolved I recall talking w/ yuvi last week that this was to stage a higher tc threshold for testing virtualenv which has been unmercifully slow as of late. T... [14:10:30] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409233 (10chasemp) a:03yuvipanda can you poke about and drop some notes here? current state https://phabricator.wikimedia.org/T137924#2406092 [14:11:11] 06Labs, 10Tool-Labs: Virtualenvs slow on tool labs NFS - https://phabricator.wikimedia.org/T136712#2409236 (10valhallasw) >>! In T138742#2409211, @chasemp wrote: > I recall talking w/ yuvi last week that this was to stage a higher tc threshold for testing virtualenv which has been unmercifully slow as of late.... [14:28:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/UkrFace was created, changed by UkrFace link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/UkrFace edit summary: Created page with "{{Tools Access Request |Justification=Hello. I am created wikipedia rating for Ukrainian Wikipedia http://wrating.ukrface.org Now, I use wikipedia API with limit 500 items p..." [14:33:04] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409263 (10Cmjohnson) [14:34:14] 06Labs, 10wikitech.wikimedia.org: wikitech: Tell people to log out and in again? - https://phabricator.wikimedia.org/T138373#2398480 (10chasemp) >>! In T138373#2398792, @Peachey88 wrote: > Is anything being doing towards fixing the cause issue? having a looking in #wikitech.wikimedia.org the only relevant tas... [14:36:28] Hello. Excuse me, I applied a test wiki : https://phabricator.wikimedia.org/T138679 . Want to know where is the wiki ? How do I set it or use it? [14:37:39] MGdesigner: so you want to setup a testwiki at that project? [14:38:15] Luke081515: yes [14:38:29] MGdesigner: did you created already a instance? [14:39:07] MGdesigner: I think you can use mw-vagrant for a testwiki, here is a manual: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs [14:39:34] shows how to setup a single wiki. if ou need a cluster, that is a bit more difficult, than you can ask me ;) [14:40:10] Luke081515: I mean I saw "yuvipanda :I've created it - it is mw-extension-ids, and added Niharika as admin. Feel free to add others." So it is created? [14:40:41] MGdesigner: yuvi created a project, which allows you to create instance so run a testwiki [14:40:53] the project which was created is only a container [14:41:12] Luke081515: Oh! it's a "container" [14:41:15] for example Project:Tools is a container for all toollabs related, but for running something there, you need instances [14:41:40] Luke081515: Ok now I go to read the manual. [14:43:37] Luke081515: About the container, do I need to login to shoichi@login.tools.wmflabs.org to control it? [14:43:58] Luke081515: sorry , through login.tools.wmflabs.org [14:44:21] MGdesigner: nope [14:44:27] login at wikitech [14:44:30] create an instance [14:44:36] apply the puppet rule [14:44:49] then login as shoichi@bastion.wmflabs.org [14:45:01] and ssh to you instance: "ssh " [14:45:37] Luke081515: Thank you. I go to handle these steps. [14:45:49] login.tools.wmflabs.org is just for toollabs needed, so you don't need it at the whole process ;) [14:46:53] Luke081515: thanks for helping out man, MGdesigner it may be visually easier to parse https://horizon.wikimedia.org which is the system we are moving to [14:47:48] chasemp: are we actually able to apply puppet things from horizon too? Normally all my instances are working normaly, so I don'T need to visiting horizon ;) [14:47:54] Luke081515: About "login at wikitech" Does it mean https://wikitech.wikimedia.org/ ? [14:48:02] MGdesigner: yep [14:48:03] puppet not yet, hopefuly soon, instance creation etc is there [14:49:03] MGdesigner: if wou are at that "create instance" page, I would recommend to you a medium instance [14:49:11] that's ideal for vagrant IIRC [14:49:46] 06Labs, 10MediaWiki-extensions-Newsletter: newsletter-test.wmflabs.org is broken - https://phabricator.wikimedia.org/T138686#2409302 (10chasemp) p:05Triage>03Normal [14:50:17] 06Labs: tools-worker-1011 has no working user accounts?! - https://phabricator.wikimedia.org/T138447#2409303 (10chasemp) p:05Triage>03Normal [14:50:31] Luke081515: About "create an instance" ,sorry, is it a link,or something, I didn't see it. [14:50:56] MGdesigner: wait a bit, I need to search my phone first [14:50:58] for 2FA [14:51:01] 06Labs: Initial puppet run failing for new instances in privpol-captcha - https://phabricator.wikimedia.org/T138438#2409305 (10chasemp) p:05Triage>03Normal @tom29739 whatever came of this? I somewhat recall you got it working. [14:51:10] 06Labs, 10wikitech.wikimedia.org: wikitech: Tell people to log out and in again? - https://phabricator.wikimedia.org/T138373#2409307 (10chasemp) p:05Triage>03Normal [14:51:28] 06Labs, 10MediaWiki-extensions-Newsletter: newsletter-test.wmflabs.org is broken - https://phabricator.wikimedia.org/T138686#2407308 (10chasemp) I'm not sure who owns this to assign [14:52:33] MGdesigner: ok, go to horizon (https://horizon.wikimedia.org/project/), then login [14:52:40] do you have already 2FA enabled? [14:53:06] Luke081515: Ok I turn to https://horizon.wikimedia.org/project/ [14:53:55] MGdesigner: then go to https://horizon.wikimedia.org/project/instances/ [14:54:00] and click "start instance" [14:54:40] 06Labs: Initial puppet run failing for new instances in privpol-captcha - https://phabricator.wikimedia.org/T138438#2409312 (10tom29739) @chasemp it just happens randomly, not every time but still often enough to get annoying. I don't think it's anything that I'm doing because Niharika was having the exact same... [14:54:43] then choose a short instance-name, instance size should be m1.medium, and as boot-source you need to select ubuntu-14.04 trusty [14:58:46] Luke081515: Ok I logged into https://horizon.wikimedia.org/project/ . I go to next step. [14:59:17] MGdesigner: you are at thep two: you need to create a instance [14:59:57] MGdesigner: otherwise: If you need only once a testwiki, I can do that setup stuff for you if you want, to this may not help you if you need more often instances with testwikis [15:00:13] Luke081515: I am on https://horizon.wikimedia.org/project/instances/ now. [15:00:32] ok, then: [15:00:36] 16:54 < Luke081515> then choose a short instance-name, instance size should be m1.medium, and as boot-source you need to select ubuntu-14.04 trusty [15:01:58] Luke081515: I only need need a testwiki with my extension installed. [15:02:23] MGdesigner: ok, so it is your decision, because if you want me to setup, I need projectadmin access [15:03:02] (projectmembers can adjusted here: https://wikitech.wikimedia.org/wiki/Special:NovaProject) [15:03:30] so if you want me to do this, you need to add me as member first, then as projectadmin. or you follow that manuel. your decision [15:04:23] Luke081515: Oh.. I think I didn't get the admin from Niharika . [15:04:45] Luke081515: So I need go to contact her... [15:05:18] hm [15:05:24] there are already two instances at that project [15:05:42] Luke081515: oh! [15:06:07] MGdesigner: maybe she already created a wiki, but I can't help you actually, cause I don't have access to DNS/proxy stuff and the instances [15:07:01] Luke081515: I see. So there are 2 instances? Where can I see them? [15:07:41] MGdesigner: https://wikitech.wikimedia.org/wiki/Nova_Resource:Mw-extension-ids [15:07:52] that's the page for that project [15:08:29] and looks like something is already running at that instances, at least there is space, CPU and ram used, more than at an idle instance [15:08:48] Luke081515: So can I ssh into it? [15:09:10] looks like actually you are not a member of that project [15:09:16] so you don't have access to it :-/ [15:09:26] means you can't ssh into the instances too [15:09:46] Luke081515: ok, so I need to wait her to add me. [15:09:52] yep [15:11:34] Luke081515: But I heard from my companion. She just went to a vacation for a week. [15:12:31] labs admins can add people too, but I'm not sure, if they will do that in that case [15:12:39] but since you authored the task it looks possible [15:17:12] Luke081515: ok. [15:21:36] chasemp: do you know if there is a puppet role for shinken actually? [15:22:18] for the main labs shinken I think modules/role/manifests/labs/shinken.pp but to do a project specific instnace [15:22:22] I'm not sure what massaging it would take [15:24:47] chasemp: so role::shinken, or something different? [15:25:08] that would translate to probably role::labs::shinken [15:25:24] which may be applied as 'role labs::shinken' if you go looking [15:25:37] ok thx :) I will try it out [15:25:39] the role keyword is a hiera convention mainly but looks like that [15:28:44] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2409386 (10Shoichi) [15:28:45] chasemp: btw, what are the conditions to get higher quotas? Actually I don't need higher, and mostly are fine, but I'm actually using 18/20 CPUs [15:28:46] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2409382 (10Shoichi) 05Resolved>03Invalid p:05Triage>03High a:05yuvipanda>03None >>! In T138679#2407650, @yuvipanda wrote: > I've created it - it is mw-extension-ids, and added Niharika as admin. Feel free to add ot... [15:29:37] well, it's kind of an informal ask and evaluate situation but we are really sitting at a high tide right now waiting on more servers to be racked [15:29:52] ok [15:30:54] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2409393 (10Luke081515) [15:30:56] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2409391 (10Luke081515) 05Invalid>03Open From IRC: The actually problem of @Shoichi is that (according to him) @Niharika is currently at vacation, and he has actually no access to that project to try out his extensions. So... [15:31:34] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2409401 (10Shoichi) [15:31:36] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2409398 (10Shoichi) 05Open>03Invalid a:03yuvipanda Sorry I made a mistake. I resend again. Now I want to go to set up a testwiki. But I didn't got admin. I have heard from Shangkuanlc sayed that Niharika is on a vacati... [15:32:21] Luke081515: thank you. [15:32:52] MGdesigner: but please don't close the task as invalid, because then the task is *closed* [15:32:52] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2409405 (10Luke081515) [15:32:54] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2409403 (10Luke081515) 05Invalid>03Open (You want this task open, so that someone can correct the things here. If you close that task as invalid, no one will change your membership) [15:33:24] which would mean that the task is invalid,, but you want to change the actual situation, not the original task [15:33:24] Luke081515: sorry my mistake. [15:33:33] np ;) just let it oopem atm [15:33:34] *open [15:33:56] Luke081515: I just remove comment. Is it ok? [15:34:13] I think it is always ok, if we don'T start an edit-war ;) [15:35:06] Luke081515: thank you. I am too rookie. XD [15:37:20] chasemp: something is seriously wrong with my cyberbot-exec-iabot-01 instance [15:37:53] CP678|Laptop: what is going on at that instance? [15:37:56] maybe I can help you [15:37:57] I cannot access it, it died a few days ago, Wikitech reports it is in the SHUTOFF state, and I cannot reboot it [15:38:13] I'm totally cutoff. [15:40:03] Luke081515: any ideas? [15:41:33] give me a few minutes [15:42:18] hm, ok, that instance reports nothing to graphite actually. CP678|Laptop: what happends, if you try to reboot? [15:42:34] and is there actually output at the console-log? [15:43:08] Luke081515: I just said I can't reboot [15:43:22] Failed to reboot instance cyberbot-exec-iabot-01. [15:43:36] CP678|Laptop: did you tried the hard reboot from horizon, or which one? [15:43:47] Wikitech [15:44:02] I can't even login to the instance [15:44:32] CP678|Laptop: I think in more cases, you can'T login, but reboot works ;). Try otu a horizon reboot? maybe it could help [15:44:42] otherwise we need a labsadmin with salt [15:44:59] ssh: connect to host cyberbot-exec-iabot-01 port 22: No route to host [15:45:07] CP678|Laptop: it may be coming back now [15:45:20] here is the tldr guys, we have an oversubscription on resources issue in labs atm [15:45:26] not making new instances is the best idea [15:45:41] I didn't make a new instance though. [15:45:42] we seem to have a bit of breathing room but this has happened a few times where a node is squeezed out [15:45:51] yes I understand that, it was general commentary [15:45:55] Ah [15:46:11] see if it comes back but fyi we are running very thin here until new nodes which are coming [15:46:22] Is there anyway to keep this specific instance from being squeezed out? [15:46:42] It's running a rather important bot. [15:46:51] maybe migrate it to anther labvirt if one is less taxed [15:46:56] for now let's see if it holds on [15:47:05] you said a few days ago and things are a bit better since [15:47:08] so I'm hoping it will be ok [15:47:14] Ok. [15:47:23] * CP678|Laptop watches out. [15:48:21] sorry for the issues, there was a surge to threshold and things got overly packed fast [15:49:10] I'll keep on a lookout. [15:50:50] * CP678|Laptop leaves the channel with a big thank you. [15:52:50] chasemp: ok, apllying that shinken puppet role works, but do you know if I have to do addiotional steps? Creating a proxy to that instance throws 502. [15:53:51] Luke081515: I really don't at all, I haven't setup shinken [15:54:17] chasemp: ok. But thanks for the role name :) [16:08:09] 06Labs, 10Tool-Labs: Install debootstrap and fakechroot on tools - https://phabricator.wikimedia.org/T138138#2409475 (10chasemp) ran this by moritz and we are on the same page that this is sane. Thanks for explaining your reasoning @tom29739. I will merge this today or tomorrow when I can. [16:41:31] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2409636 (10yuvipanda) I've added @Shoichi to be a projectadmin too! [16:54:26] Luke081515: Hi I got admins. I found I can't ssh shoichi@Ids.mw-extension-ids.eqiad.wmflabs or do I need to ssh from another link? [16:54:57] MGdesigner: you sshed from bastion-01? [16:55:09] and what's the error message? [16:55:27] Luke081515: ssh: Could not resolve hostname ids.mw-extension-ids.eqiad.wmflabs: Name or service not known [16:55:55] Luke081515: So I need to ssh shoichi@bastion-01 ? [16:56:08] you need to ssh *from* that instance [16:56:12] because it is a bastion host [16:56:29] otherwise inside labs is not reachable for you (except tools via tools-bastions) [16:57:26] MGdesigner: so connect as shoichi@primary.bastion.wmflabs.org first [16:57:28] then ssh [16:57:36] MGdesigner: and make sure, that agent forwarding is enabled [17:00:00] Luke081515: got it [17:00:45] nice [17:01:18] MGdesigner: if you want to add me add least tempoaray as member (admin atm not needed) I can look up, if there is already a wiki [17:08:12] Luke081515: OK . I am researching,so I ssh Ids.mw-extension-ids.eqiad.wmflabs again? [17:10:07] MGdesigner: nope, you need to lookup https://horizon.wikimedia.org/project/proxy/ [17:10:14] are there actually any entrys? [17:10:59] Luke081515: There is nothing in proxy. [17:11:08] hm, ok [17:11:44] MGdesigner: can you add me as member, and later as admin here? https://wikitech.wikimedia.org/wiki/Special:NovaProject would help me to lookup, how the instances are actually configured [17:12:06] because I think we should not change the work, niharika did [17:12:22] Luke081515: How do I add you? by useradd? [17:12:54] frist useradd, at the right, than, after that finished, add to the role "projectadmin" [17:13:02] *first [17:13:08] I'm Luke081515 at wikitech [17:14:05] Luke081515: If I use "adduser Luke081515" then will you user setting come from wikitech? [17:14:27] MGdesigner: no, you are at the wrong page [17:14:35] you need to add me as user here: https://wikitech.wikimedia.org/wiki/Special:NovaProject [17:14:38] at that wikitech page [17:14:49] ok I go to there [17:14:53] select your project, and then there is a link to add users [17:16:43] Luke081515: webpage? [17:16:53] yep, at that webpage [17:17:12] MGdesigner: I will take a look at you project later in the evening, currently I have to go now. See you later :) [17:17:21] Luke081515: ok. [17:42:22] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409766 (10yuvipanda) So I re-enabled puppet and that killed the line @chasemp added, and that exhibits the following behavior: 1. https://graphite.wmflabs.org/ works (over HTTP) on a browser 2. http://grap... [17:48:26] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409777 (10yuvipanda) ok, so that was on firefox, and on Chrome HTTPS doesn't work for me (there's a 'ERR_INVALID_RESPONSE' error) while the http hit downloads the exact same strange file as on firefox. [17:49:53] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409778 (10yuvipanda) A render URL, like https://graphite.wmflabs.org/render/?width=588&height=310&_salt=1467049750.544&target=tools.tools-elastic-02.cpu.total.user works for me both on HTTPs and HTTP on bot... [17:50:19] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2383843 (10tom29739) @yuvipanda neither 1. or 2. work for me, chrome tries to download a file called 'download.gz'. Both work with curl. [17:51:30] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409781 (10yuvipanda) I'm on Iceweasel 38.5, what are you on, @tom29739? and can you upload this download.gz somewhere? [17:57:13] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409782 (10yuvipanda) So the 'strange file' is gzip encoded, and I can repro this in curl with: ``` curl -H 'Accept-Encoding: gzip' https://graphite.wmflabs.org ``` No idea why setting DocumentRoot fixes t... [17:58:45] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409785 (10tom29739) @yuvipanda Chrome Version 51.0.2704.103 m (32 bit) on Windows 7. download.gz is 707 bytes from the https:// version. WinRAR can open it, there's a file named 'download' inside which is 6... [17:59:34] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409786 (10yuvipanda) I've re-enabled setting DocumentRoot, which does fix this. I'll disable puppet again and re-investigate later. [18:04:41] !log Bots Restarted wm-bot to fix file permission error [18:04:42] Bots is not a valid project. [18:06:56] Luke|away: OK I have add you as projectadmin. [18:07:50] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2409802 (10tom29739) It looks like it's spewing out incorrect gzipped data, which the browser can't decode so it downloads it as a file instead. Maybe the default DocumentRoot is in the wrong place so that t... [18:08:05] that's slow.. [18:13:34] anyone know how to select files for adding to a .7z or .zip based off file name prefix? [18:20:35] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2409808 (10Shoichi) >>! In T138679#2409636, @yuvipanda wrote: > I've added @Shoichi to be a projectadmin too! Thank you. I have further question. when I am on server,primary.bastion.wmflabs.org ,I found "ssh -A shoichi@ids.m... [18:27:56] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2407149 (10Krenair) You shouldn't use agent forwarding, you need to "ssh shoichi@ids.mw-extension-ids.eqiad.wmflabs" on your own computer and have it set up to proxy via bastion.wmflabs.org in your ~/.ssh/config file, see http... [18:53:39] Betacommand, if you were using Windows, then I think you can add a prefix to search, and just select all the results and drag + drop/right click to archive. [18:53:59] I think Mac + Linux file managers have search functions too. [18:55:09] tom29739: doesnt work ofr cases where you dont have a good UI (Labs) [18:55:27] tom29739: but Ive found a work around [18:55:41] You could use something like find. [18:58:32] tom29739: was having difficulties [18:59:08] What was the workaround? [19:06:44] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2409891 (10Shoichi) By using the new method, I found I sucessfully login to Ids-1.mw-extension-ids.eqiad.wmflabs ,but still failed on logining to Ids.mw-extension-ids.eqiad.wmflabs. >>! In T138679#2409830, @Krenair wrote: >... [19:13:30] MGdesigner: FYI, I'm back [19:13:43] Luke081515: WB [19:15:55] Luke081515: I have added you. [19:16:19] thanks :) [19:16:22] I will take a look now [19:16:49] Luke081515: m(_ _)m [19:20:08] MGdesigner: looks like niharika didn't installed a testwiki at any of these instances, anyway we should not delete this instances without asking here. Wanna me to create a third instance, and install a testwiki on it? [19:21:05] Luke081515: ok . Please. [19:22:24] MGdesigner: ok. what name should that instance have? Normally the URL that this instance has, contains the same name [19:23:46] Luke081515: mw-extension-ids-testing <-- Is it OK? [19:24:29] Luke081515: oh sorry I see wrong. Just the same name ok. [19:25:03] maybe something short [19:25:18] but if you only want a single wiki, you can choose a different domain name [19:25:23] but for that you need a name too [19:25:39] so in my opinion it is more easy to choose same url name as instance [19:25:57] Luke081515: Ye,I only need a single testwiki site. [19:26:41] MGdesigner: ok, but we still need a name ;) [19:27:05] I mean a long URL is technically not a problem, but maybe for you to remember [19:27:08] Luke081515: You mean name of the test wiki? [19:27:23] MGdesigner: the url would be .wmflabs.org [19:27:27] so you need a domainname [19:27:32] and ideally the instance has the same name [19:28:01] Luke081515: oh!!! I see. "ids-testing " is it ok? [19:28:13] yep [19:28:55] MGdesigner: I will ping you, when the installation finished. this may take some time [19:29:23] Luke081515: All right. Thank you first. :) [19:30:37] hi MGdesigner [19:30:48] doctaxon: Hi [19:30:59] the task white on black is sill open [19:31:27] doctaxon: "white on black"? [19:31:40] weiß auf schwarz [19:32:19] you remember? [19:32:43] doctaxon: Do you mean change status? [19:34:14] oh, i thought, you were another person, sorry [19:35:17] doctaxon: Maybe we have chated before. Here? Or in real world? [19:38:15] no, I confused you with another user, sorry [19:39:28] can someone delete 7513178 from SGE ? [19:51:23] MGdesigner: heh, your wiki is created now. I will send you the required data via PM now [19:51:54] Luke081515: Thank you so much! [20:27:19] 06Labs: Measure capacity and utilization of labvirt**** servers - https://phabricator.wikimedia.org/T107067#1485740 (10hashar) There is a Grafana dashboard for labvirt* CPU, RAM, Disk https://grafana-admin.wikimedia.org/dashboard/db/labs-capacity-planning Needs polishing though. [20:37:26] Any tools roots around? Need a job killed that I cannot kill [20:45:32] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2410147 (10Shoichi) [20:45:34] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2410144 (10Shoichi) 05Open>03Resolved p:05High>03Lowest Thx to Luke081515 help me setting up ids-testing instance,and setup the testing wiki . Now working : http://ids-testing.wmflabs.org/wiki/ [20:57:23] 06Labs, 10Tool-Labs: SGE zombie needs killed - https://phabricator.wikimedia.org/T138787#2410217 (10Betacommand) [21:02:34] 06Labs, 10Tool-Labs: tools-exec-1216 down - https://phabricator.wikimedia.org/T138787#2410241 (10valhallasw) [21:12:19] 06Labs, 10Tool-Labs: tools-exec-1216 down - https://phabricator.wikimedia.org/T138787#2410287 (10valhallasw) ``` [2249041.585992] INFO: task nscd:19454 blocked for more than 120 seconds. [2249041.586769] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [2249041.587741] INFO: task nscd:... [21:13:04] 06Labs, 10Tool-Labs: tools-exec-1216 down - https://phabricator.wikimedia.org/T138787#2410293 (10valhallasw) 05Open>03Resolved a:03valhallasw [21:13:26] 06Labs, 10Tool-Labs: tools-exec-1216 down - https://phabricator.wikimedia.org/T138787#2410217 (10valhallasw) I have rebooted the host and it should be back up shortly. [21:14:35] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, and 2 others: Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2410303 (10greg) [21:25:21] PROBLEM - Host tools-exec-1216 is DOWN: CRITICAL - Host Unreachable (10.68.17.255) [21:30:59] 06Labs, 10Tool-Labs: tools-exec-1216 down - https://phabricator.wikimedia.org/T138787#2410394 (10tom29739) 05Resolved>03Open tom29739@tools-bastion-03:~$ ssh tools-exec-1216 ssh: connect to host tools-exec-1216 port 22: No route to host @valhallasw looks like it's down again.. [21:41:46] 06Labs, 10Tool-Labs: tools-exec-1216 down - https://phabricator.wikimedia.org/T138787#2410414 (10valhallasw) @tom29739: Thanks for reopening the bug! The host is now in ERROR state. @Andrew, is that related to the shortage of resources on labs? [22:32:45] !log deployment-prep deployed gerrit.wikimedia.org/r/296279 to puppetmaster to test kibana4 role [22:32:45] Please !log in #wikimedia-releng for beta cluster SAL [22:32:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [22:43:05] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2410597 (10JAufrecht) [22:43:07] 06Labs, 10Phlogiston (Interrupt), 15User-bd808: Phlogiston-1 unresponsive - https://phabricator.wikimedia.org/T137736#2410595 (10JAufrecht) 05Resolved>03Open Same thing happened today. Do you want me to open a fresh ticket? [22:55:51] 06Labs, 10Tool-Labs: Virtualenvs slow on tool labs NFS - https://phabricator.wikimedia.org/T136712#2345235 (10tom29739) ``` (newsopel)tools.piagetbot@tools-bastion-03:~$ time pip -V pip 1.5.4 from /data/project/piagetbot/.virtualenvs/newsopel/local/lib/python2.7/site-packages (python 2.7) real 0m23.821s us... [23:12:25] FASFD [23:12:45] Anyone here play runescape? [23:14:03] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:11] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:14] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:15] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:16] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:16] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:16] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:18] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:18] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:19] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:19] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:19] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:19] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:20] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:20] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:20] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:21] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:21] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:32] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:32] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:33] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:33] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:34] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:35] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:35] v [23:14:35] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:36] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:36] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:37] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:14:37] LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOLOLLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [23:59:22] Why don't we have any bot to kick spammers [23:59:40] maybe wm-bot has that enabled [23:59:48] !kick yesyesyes [23:59:57] did that work? [23:59:58] 01:14 -!- yesyesyes [ae5c2e83@gateway/web/freenode/ip.174.92.46.131] has left #wikimedia-labs []