[00:00:11] hummm... okay, I'm going to check my [00:00:15] code [00:01:19] Is there any way to be informed of this bug? I mean if there is any site to receive feedback when it will be solved :) [00:01:55] Ivanhercaz: yes, https://phabricator.wikimedia.org/T136118 is the bug :) [00:02:16] Ivanhercaz: if you login and click 'subscribe' you will be notified when it is fixed [00:02:49] Perfect! Thank you so much YuviPanda! :D [00:02:58] 06Labs, 10Labs-Infrastructure: Creating new instance failed - https://phabricator.wikimedia.org/T136656#2343333 (10yuvipanda) [00:02:59] Ivanhercaz: yw! [00:03:56] I discovered PAWS a few days ago and I think that it is an interesting project. [00:05:01] Ivanhercaz: nice! where did you discover it from? [00:05:06] * YuviPanda is the primary person who works on it so far [00:06:03] Searching how to create a bot with Pywikibot, in the MediaWiki page Manual:Pywikibot/Installation [00:06:11] ah [00:06:13] nice [00:06:23] did you see https://www.mediawiki.org/wiki/Manual:Pywikibot/PAWS_walk-through yet [00:06:40] Ivanhercaz: there's also a #pywikibot channel with more people who use pywikibot [00:06:41] yes! :D [00:08:31] Oh! I will keep in mind #pywikibot channel to consult about it [00:08:37] :D [00:09:17] It's my first time developing a bot for Wikipedia (Spanish wiki in this case) and PAWS is very intuitive [00:09:27] yay! [00:09:29] that's great to hear, Ivanhercaz [00:10:09] btw, what happens if you copy/paste charactes with the accents? [00:11:02] We have to report the bugs and keep in mind the good features YuviPanda :) [00:11:14] Hmmm... give me a second mutante [00:12:04] Well, I have the problem that I use IceWeasel and in the PAWS Manual specify that the function of Copy&Paste is only available to chromium :/ [00:13:37] ah, i see [00:14:01] It's possible that with chromium it works. [00:14:33] Yeah the terminal emulator there could use some improvements [00:16:32] There is not much documentation of PAWS in Spanish [00:20:35] Someday I will try to translate something, it could bring closer to Spanish users. [00:22:09] Ivanhercaz: yeah paws is very new [00:22:30] Is there an API to get status if a user is currently blocked or globally blocked? (eg: true or false or 1 or 2)? [00:23:00] Might have asked in the wrong channel. sorry. [00:44:26] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [00:56:58] Well, I'm going to rest. Thank you so much for your help YuviPanda and for your idea mutante. [00:57:03] Good night! :) [02:14:29] !log deployment-prep Started redis-server on deployment-rcstream to stop MW hhvm.log spam [02:14:30] Please !log in #wikimedia-releng for beta cluster SAL [02:14:32] no [02:14:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [02:15:01] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:19] rebel Krenair [02:15:36] Krenair: btw, I added kubernetes credentials to the lolrrit-wm service group, so users can now restart bot, but not push new versions [02:15:41] I'll comment + amend docs in a few hours [02:17:06] yay [02:17:21] does this give lolrrit-wm service group members any bad permissions that we wouldn't give to other people? [02:17:35] also why can't we push new versions? [02:17:43] because it is still its own docker image [02:17:47] and only admins can pus new docker images [02:18:00] Krenair: nope, these permissions are coming to everyone shortly, once I write up a credential generator [02:18:40] so when can we push docker images? I'd like to get that bug closed [02:20:33] probably not ever [02:20:41] will need to find an alternate solution [02:20:50] So when can we change the code? [02:21:02] but my first focus is on webservices, and I want to get that out in this month [02:21:05] so I'll take a look after that [02:21:30] I don't have any firm ETAs :) [02:21:47] the ability to restart came due to work on making webservices work, and they needed this to happen [02:22:01] perhaps when I work on the nodejs webservice stuff there'll be an easy way out for this too and I'll make it happen [02:22:31] brb [02:22:51] I guess we'd need the registry server to implement authorization for pushing to a namespace [02:23:11] so that user X can't upload a new base image that roots all the containers [02:23:53] 06Labs, 10DBA: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2341449 (10MZMcBride) ``` MariaDB [enwiki_p]> SELECT page_id, page_namespace, page_title FROM enwiki_p.page where page_id IN (50274778,1272531,976991,50274777) ORDER by page_namespace; +------... [02:27:11] This looks neat -- https://github.com/SUSE/Portus -- authn/authz for a docker registry [02:53:43] bd808 yeah but that causes a whole other host of problems though [02:54:14] Such as needing to have a docker registry that can have arbitrary containers that we can't rebuild to do package deploys [02:54:34] And bring back the source availability problem too [02:54:42] Not that it ever left [02:55:02] Hopefully the PaaS will help with such things [02:55:16] you should add some notes about those concerns [02:55:23] If you look at deis build packs they have an elegant solution for it [02:55:26] Yeah [02:55:48] I plan on doing a mega dump on the paas task soon [02:55:53] nice [02:57:33] * YuviPanda is afk again [02:58:03] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [03:32:39] 06Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#2343708 (10Beetstra) Thanks, vallhallasw. [03:38:14] ok, with my bandwidth issue -- looks like between the web proxy and my instance I get a nice 200 Mbps or so, but between the proxy and me i sometimes see a mere 2-5 Mbps on my download [03:39:48] interestingly, a straight file download streams the entire file at full speed to the proxy, which then sends it down to me mmuucchh sslloowweerr :) [03:40:27] let me try from my server instead of from my home comcast [03:43:06] ok i can dl one stream at 80 Mbits from my linode server in dallas [03:43:14] while seeing only 3 Mbits on my Comcast in Portland [03:43:16] sighhhhh [03:49:03] but it's not always this slow; other times i get a nice ~75 Mbits [03:49:29] either something in between is flapping routes and one is awful, or something is intermittently congested [03:49:47] (I've confirmed i can pull from, say, upload.wikimedia.org very fast at the same time, which goes via sfo caches) [04:06:18] downloads can spike high at start, but don't always. meh [04:06:20] * brion wanders off [04:07:23] Brion can you file a bug [04:07:26] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [04:08:01] sure [04:08:44] Thanks [04:09:56] Sneakernet! [04:20:11] YuviPanda: ok I see same slowdown on download.wikimedia.org [04:20:28] which is also on eqiad, two ip addresses off ;) [04:24:37] wait no those are different [04:25:23] ugh tired :) [04:26:00] ah but they both go through zayo before reaching comcast [04:50:36] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2343755 (10brion) [07:08:06] (03PS3) 10Lokal Profil: Add test to validate entries in monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/290557 [07:35:47] 06Labs, 10DBA: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2341449 (10valhallasw) The enwiki database has drifted from production (see {T133715}, {T134203}). [07:36:32] (03CR) 10Jean-Frédéric: [C: 032] Add test to validate entries in monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/290557 (owner: 10Lokal Profil) [07:37:26] (03Merged) 10jenkins-bot: Add test to validate entries in monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/290557 (owner: 10Lokal Profil) [07:57:49] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2344085 (10jcrespo) I'm sorry, what? When did I say it was going to take 12 hours? My last estimation was: > Revision table is ongoing now, but it has 700 M rows and it takes almost half a day to import and f... [08:00:51] RECOVERY - Puppet run on tools-pastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:09:35] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:38:53] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#2344289 (10hashar) [09:39:55] 06Labs, 10Labs-Infrastructure, 10Monitoring, 06Operations: Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656#2344291 (10hashar) [09:54:03] (03PS1) 10Jcrespo: Invite wikibugs to #wikimedia-databases [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/292105 (https://phabricator.wikimedia.org/T101937) [09:54:39] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:39] (03CR) 10Merlijn van Deen: [C: 032] Invite wikibugs to #wikimedia-databases [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/292105 (https://phabricator.wikimedia.org/T101937) (owner: 10Jcrespo) [09:55:20] (03Merged) 10jenkins-bot: Invite wikibugs to #wikimedia-databases [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/292105 (https://phabricator.wikimedia.org/T101937) (owner: 10Jcrespo) [09:55:55] !log tools.wikibugs Updated channels.yaml to: 9da4fc07d28f33fb74207a78b96fe57068ef6d46 Invite wikibugs to #wikimedia-databases [09:55:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [09:55:58] wow, that was fast [09:56:09] I was still searching for an owner [09:56:32] for wikibugs? that's legoktm and me, I think. [09:56:52] yes, I saw that :-) [10:11:35] 06Labs, 10Tool-Labs: Trusty instances do not show the motd banners - https://phabricator.wikimedia.org/T85307#943684 (10hashar) The PAM fixed is explained on https://lists.wikimedia.org/pipermail/labs-l/2015-December/004158.html which namely involves running on the instance `/usr/local/sbin/cleanup-pam-config`... [10:14:11] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:08] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2344534 (10elukey) @chasemp yes it is :( [12:02:36] 10PAWS: I can not write some special characters in PAWS - https://phabricator.wikimedia.org/T136118#2344546 (10Ivanhercaz) Hi @yuvipanda, yesterday you inform me about this error in #wikimedia-labs channel IRC. Today I decide to open PAWS with another browser, concretely with Chromium, and I had a surprise: I ca... [12:22:37] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2343755 (10faidon) Since you get bad //download// speeds, the opposite traceroute (from eqiad to you) is the more interesting one. I didn't have your IP,... [12:33:41] 10PAWS: I can not write some special characters in PAWS - https://phabricator.wikimedia.org/T136118#2344609 (10Dvorapa) @Ivanhercaz I use Google Chrome v51, so I think you are wrong. Rather it could be caused some basic module in Chrome and Firefox, which is not present in Chromium [14:05:27] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2344776 (10brion) Thanks, I'll keep an eye out tonight and see if it gets congested again (currently seeing a cool 80 Mbits download rate at 7:04am pacifi... [14:27:40] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2344830 (10chasemp) a:05chasemp>03None thanks @krenair that makes sense [15:06:57] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2344900 (10valhallasw) Restarting the bot is now also possible from tools-login directly: https://wikitech.wikimedia.org/w/index.php?title=Grrrit-wm&type=revision&diff=600999&oldi... [15:42:28] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345045 (10RobH) [15:44:19] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345064 (10RobH) [15:54:03] 06Labs: Puppet stale on labtestweb2001 - https://phabricator.wikimedia.org/T136611#2345128 (10chasemp) a:05chasemp>03Andrew [15:58:54] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345159 (10RobH) [16:03:27] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345179 (10RobH) [16:15:04] 06Labs: Puppet stale on labtestweb2001 - https://phabricator.wikimedia.org/T136611#2345216 (10Andrew) I'm still developing on labtestweb2001 but I just now refreshed puppet. [16:24:19] 06Labs, 10Tool-Labs: Virtualenvs slow on tool labs NFS - https://phabricator.wikimedia.org/T136712#2345235 (10valhallasw) [16:30:04] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345269 (10Cmjohnson) [16:55:26] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345378 (10RobH) [17:00:42] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345398 (10RobH) [17:42:04] 06Labs, 07LDAP: Restore ldaplist -l passwd - https://phabricator.wikimedia.org/T122595#2345569 (10MoritzMuehlenhoff) My old patch from https://gerrit.wikimedia.org/r/#/c/262745/ was wrong, it still needs more work to actually request the followup pages, [17:54:27] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345595 (10Cmjohnson) [17:58:28] 06Labs, 06Operations: labnet100[12].eqiad.wmnet need to be reimaged with RAID] - https://phabricator.wikimedia.org/T136718#2345604 (10chasemp) [18:18:24] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: connect usb external disk to labmon1001 - https://phabricator.wikimedia.org/T136242#2345681 (10RobH) a:05RobH>03Cmjohnson Tried to mount and format this, and it only shows up as 800GB disk in the OS. Can you plug this into your laptop and see i... [18:33:02] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#2345822 (10yuvipanda) What this would concretely solve (for me) is projects like tools, where now I have to manually set hiera config on each host of a part... [18:33:58] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#2345829 (10yuvipanda) We're also trying to kill all LDAP variables, see T101447 [18:49:42] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2345885 (10brion) As of 11:44 am pacific time I'm seeing 24Mbps on the new route through Chicago, down from 80Mbps earlier this morning. [18:54:33] 06Labs, 10Tool-Labs: wikiviewstats is using 232G on Tools - https://phabricator.wikimedia.org/T136198#2345929 (10chasemp) @technical13 are you actively involved in this project? [18:54:57] uhh, somehow I just disabled 2 factor auth on labs [18:55:04] i thought i was logging in... [18:55:11] not sure how to turn it back on [18:55:36] oh in prefs...i should click around more... [18:55:40] before asking ... [18:59:04] nm carry on! :) [19:22:47] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: connect usb external disk to labmon1001 - https://phabricator.wikimedia.org/T136242#2346061 (10Cmjohnson) Checked the 3TB on my macbook and it came up w/ 800GB as well. Swapped to a 2TB and shows normal. Plugged into labmon1001 and it appears as we... [19:23:08] 06Labs, 10Labs-Infrastructure: Copy graphite data from labmon1001 to an external HDD - https://phabricator.wikimedia.org/T136226#2346063 (10RobH) [19:23:10] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: connect usb external disk to labmon1001 - https://phabricator.wikimedia.org/T136242#2346062 (10RobH) 05Open>03Resolved [20:24:46] 06Labs, 10Labs-Infrastructure: Copy graphite data from labmon1001 to an external HDD - https://phabricator.wikimedia.org/T136226#2346249 (10RobH) a:03yuvipanda Chris had to setup the partition via his laptop, then I was able to mkfs.ext4 the disk and mount it as /media/backup. You have a 1.8TB usable space... [20:48:33] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2346387 (10russblau) Sorry; it appears that I must have stopped reading before the end of the sentence. So, if importing revision will take roughly a month, that means that pagelinks will take another //thre... [20:50:43] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2346420 (10RobH) [21:48:19] (03PS1) 10Alexandros Kosiaris: Fix ores redis password lookup [labs/private] - 10https://gerrit.wikimedia.org/r/292277 [21:48:36] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Fix ores redis password lookup [labs/private] - 10https://gerrit.wikimedia.org/r/292277 (owner: 10Alexandros Kosiaris) [21:51:03] If anyone is familiar with Grafana, my project completely dissappeared. [22:00:40] YuviPanda, would you happen to know who's in charge of Grafana? [22:06:27] chasemp, do you know who manages Grafana? [22:07:02] my entire cyberbot project disappeared on Grafana. [22:07:20] I cannot monitor it from Grafana now. [22:07:24] CP678: yuvi doing maint on labmon and depending what grafana you mean the grafana.wikimedia.org I would imagine it's a result of that maint and not permanent [22:07:41] if you mean grafana.wmflabs.org I don't know where that comes from and it's not supported afaict [22:08:31] CP678: see the topic graphite is the grafana source of data [22:10:51] chasemp, ^ [22:11:10] wait... [22:11:12] ...why would you set the topic to that [22:11:23] That wasn't intentional [22:11:39] That was supposed to be my response to you. [22:11:51] * CP678 leaves [22:11:59] no worries [22:23:43] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: I/O on labmon1001 is very slow - https://phabricator.wikimedia.org/T127957#2347393 (10yuvipanda) a:03yuvipanda [22:54:49] 06Labs: Flaky tools-checker pages - https://phabricator.wikimedia.org/T136775#2347505 (10yuvipanda) [22:55:25] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 06Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#527721 (10AlexMonk-WMF) [22:55:44] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 06Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#527731 (10AlexMonk-WMF) [22:59:10] !log ogvjs-integration increase floating ip quota by 1 to help brion do tests [22:59:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ogvjs-integration/SAL, Master [23:22:17] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2347595 (10brion) Currently seeing my baseline 80 Mbps; floating IP 208.80.155.243 has been assigned for now to test without the proxy, just to double-con... [23:37:47] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 06Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#2347667 (10Dzahn) Its puppetized now, should not be hard anymore. We already use it in prod. [23:49:18] 06Labs, 10Labs-Infrastructure: Copy graphite data from labmon1001 to an external HDD - https://phabricator.wikimedia.org/T136226#2347690 (10yuvipanda) It's copying now, but is pretty slow (7MB/s). I'm going to leave it running overnight (puppet is disabled too), and we'll swap this to internal disk tomorrow an... [23:50:47] 06Labs, 10Labs-Infrastructure: Copy graphite data from labmon1001 to an external HDD - https://phabricator.wikimedia.org/T136226#2347695 (10RobH) [23:50:49] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: connect usb external disk to labmon1001 - https://phabricator.wikimedia.org/T136242#2347693 (10RobH) 05Resolved>03Open So the data is copying, but very slowly. At this rate, its over 1.8 days of time. While it will remain in rsync overnight, o...