[01:41:36] hello! [02:12:54] !log deployment-prep Regenerated captcha images for T164047 [02:13:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [02:13:05] T164047: Captchas sent with wrong mime type on beta - https://phabricator.wikimedia.org/T164047 [07:57:06] zhuyifei1999_: Any chance you're able to explain to me how to setup a cron in PAWs? [08:03:00] rileyh: https://phabricator.wikimedia.org/T124972 [08:03:10] not doable right now [08:03:37] 2 years later lol [09:04:43] what’s wrong with https://tools.wmflabs.org/replag/ ? [13:27:44] revi: is there/was there something wrong? [13:28:56] at that moment s1-3 and s7 was lagged behind few secs iirc [13:28:59] but seems fine now [13:29:22] (not sure which was it, but 3 or 4 of them were showing red background) [13:30:51] Not sure what it was, if it happens again we can try to investigate more [13:31:25] k, will report back if I see such wow again [14:22:54] Hello, outreachdashboard.wmflabs.org seems to be down [14:25:02] Indeed, ragesoss ^ [14:31:29] Maybe some funky iops https://graphite-labs.wikimedia.org/render/?width=586&height=308&_salt=1520173766.252&target=globaleducation.outreachdashboard.iostat.vda.iops [14:32:49] Very low free memory as well globaleducation.outreachdashboard.memory.MemFree [14:33:09] https://graphite-labs.wikimedia.org/render/?width=586&height=308&_salt=1520173917.717&from=-24hours&target=globaleducation.outreachdashboard.memory.MemFree [14:34:23] Urbanecm: is there a lot of expected Edit-a-thons today? [14:36:05] chicocvenancio, I don't know, I'm maintaining a gadget that's highlighting users signed up in one of four defined campaigns on-wiki and I was wondering why the stylesheet is almost empty :) [14:36:58] Urbanecm: I don't have access to poke arround and try to find the problem (probably would not be the right person to do that quickly even if I had) [14:37:46] I understand, I just noticed that dashboard is down and wrote on -cloud because the dashboard runs in cloud :) [14:38:11] It does worry me a bit because of the massive number of Edit-a-thons this month relying in it [14:38:44] chicocvenancio, I personally don't like the dashboard from tech point of view but...what I can do with it :) [14:39:02] The maintainers might be able to deal with the situation https://tools.wmflabs.org/openstack-browser/project/globaleducation [14:39:18] Urbanecm: did you prefer the education extension? [14:40:26] chicocvenancio, yes, mostly because I could access the raw data directly from Toolforge replicas AND there was some basic API that allowed me to edit the data programatically. [14:42:09] Have you discussed that with the maintainers? Speaking as a user, the outreachdashboard is much better than the extension [14:43:15] No. In our chapter (Wikimedia Czech republic) almost everybody who use dashboard think that it is very user-unfriendly software [14:43:58] Huh, did they use the extension before [14:44:00] ? [14:44:06] Yes [14:44:17] For example, if you want to delete a facilitator, you must press PLUS button. [14:44:52] In the extension, there's "delete" button next to each person [14:47:13] If you want some metrics for a campaign, you can a) test with the "Generic course/Edit-a-thon/Article scoped" thing b) scrap usernames from the dashboard and calculate it on your own [14:47:43] With the extension, you can get it programatically without pretending a browser ;) [14:48:10] And the last thing for now: extension was more stable because it was in prod env, not in cloud/labs [15:04:20] musikanimal: I see you as a maintainer of the outreachdashboard as well [15:23:53] Urbanecm: could you create a phabricator task for that outage? [15:24:23] chicocvenancio, yes, of course. What tags it should have? [15:28:14] I think its "programs and events dashboard sprint 2" or something like that [15:29:26] Urbanecm: ^ [15:37:49] chicocvenancio, thx [15:40:44] It exists T188864 [15:40:44] T188864: Outreachdashboard is down - https://phabricator.wikimedia.org/T188864 [15:54:22] thanks Urbanecm [15:54:28] yw [15:56:45] It's back up for the moment. [15:57:08] It's supposed to have 160 GB on that cloud instance, but somehow it only has 20. [15:57:15] it went down because it ran out of storage. [15:57:49] Ahhj [15:58:08] You need to mount te extra quota [15:58:19] how? [15:58:39] There is a puppet class for it, or manually [15:59:21] I can help more in a little while [15:59:38] * chicocvenancio is driving [16:04:55] chicocvenancio: don't use your phone while driving, it's dangerous :) [16:10:32] chicocvenancio ragesoss it's lvs [16:10:43] lvs:srv [16:11:13] uh [16:11:15] wrong class [16:11:33] chicocvenancio ragesoss role::labs::lvm::srv [16:11:45] (remember to move everything out of srv before applying that) [16:11:53] otherwise you will have data loss in /srv [16:21:31] * chicocvenancio sees there is nothing he can add [16:21:53] ragesoss: if you need any help with that feel free to ping me [16:22:24] Hauskatze: indeed it is. I try not to. [16:22:40] chicocvenancio: heavy fines too :-) [16:23:02] Hauskatze depends on the country's laws [16:23:59] 200 € and minus 3 points in the permission [16:27:46] Here it was arround 30$ when I took the driving exam, but has recently been raised to arround 90$ [16:28:20] I have no idea how much it is in the uk and £ :) [16:28:41] well i found https://www.gov.uk/speeding-penalties [16:28:55] chicocvenancio: i would love some help with that. I don't really know it it means. [16:29:39] Do you have anything under `/srv` ? [16:30:41] for outreachdashboard? I don't think so. `/srv` is empty. [16:30:53] Good [16:31:09] Then you can apply the puppet class [16:31:24] For that you can go to horizon [16:31:30] do I need to take the instance down first, or just do it? [16:31:55] You'll probably have to reboot it after [16:32:13] No need to shut it down before [16:33:00] I recomend applying the class on the instance itself, other instances might have data on /srv [16:33:41] okay. will that be an option if i click 'apply class'? [16:34:37] You have two menus, one for project-wide or prefix puppet configuration [16:34:41] I'm at https://horizon.wikimedia.org/project/puppet/ but I don't see any instance info there. [16:35:03] The other, inside compute allows a per instance configuration [16:35:25] Go compute, instances choose the instance [16:35:40] found it. [16:36:15] done [16:36:55] chicocvenancio: will this add a new storage device, or just expand what the instance sees as available, or what? [16:37:21] It'll add a new device and mount it in /srv [16:38:00] okay. so I will still need to figure out how to make sure it's using that and not running out of storage on the device it's currently using. [16:38:12] Yes [16:38:23] damn. okay. [16:39:43] Do check, with `mount` or `df ` that it has already mounted that before moving data [16:40:15] I'm unsure if it is actually needed, but probably you need to reboot after puppet has run to make it take effect [16:40:24] df -h [16:43:20] it's not showing up with `df -h` before a reboot. [16:43:44] I'll work on this next week, so not going to reboot now. [16:54:17] Thanks much! [16:54:23] You're welcome