[00:02:01] 10Cloud-VPS, 10MediaWiki-Vagrant: MediaWiki/Apache port is chosen weirdly on Vagrant - https://phabricator.wikimedia.org/T175100#3582782 (10Tgr) [00:05:33] !log quarry backup quarry main database to /data/project/dump-2017-09-05.sql because I fear it die :(. Executing `sudo mysqldump quarry | sudo tee /data/project/dump-2017-09-05.sql > /dev/null` [00:05:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [00:28:10] (03Draft2) 10Krinkle: wikimedia-visualeditor: Add #Editing-team [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/376156 [00:28:12] (03CR) 10Krinkle: [C: 032] wikimedia-visualeditor: Add #Editing-team [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/376156 (owner: 10Krinkle) [00:28:35] (03Merged) 10jenkins-bot: wikimedia-visualeditor: Add #Editing-team [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/376156 (owner: 10Krinkle) [00:28:42] (03CR) 10jenkins-bot: wikimedia-visualeditor: Add #Editing-team [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/376156 (owner: 10Krinkle) [00:29:41] !log tools.wikibugs Updated channels.yaml to: 21a2b6712b4777afa7636d09b8461fff7cf82308 wikimedia-visualeditor: Add #Editing-team [00:39:32] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:54:30] 10Cloud-VPS (Quota-requests), 10cloud-services-team (Kanban), 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS testing setup platform sizing - https://phabricator.wikimedia.org/T169133#3582917 (10Smalyshev) I see it but cannot use it because it goes over the RAM quota. Can we have it increased a l... [01:01:35] 10Cloud-VPS (Quota-requests), 10cloud-services-team (Kanban), 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS testing setup platform sizing - https://phabricator.wikimedia.org/T169133#3582929 (10bd808) @Smalyshev please make a separate task for the RAM quota bump so that we can keep track of it a... [01:10:22] 10Cloud-VPS, 10MediaWiki-Vagrant: MediaWiki/Apache port is chosen weirdly on Vagrant - https://phabricator.wikimedia.org/T175100#3582782 (10bd808) > I would expect Vagrant to redirect host: to guest:80 but it redirects it to guest: instead This was a conscious choice that was made by Gil... [01:16:36] 10Cloud-VPS, 10MediaWiki-Vagrant: MediaWiki/Apache port is chosen weirdly on Vagrant - https://phabricator.wikimedia.org/T175100#3582955 (10bd808) > The only way to get internal requests work correctly would be http://localhost:<%= scope['::forwarded_port'] %>/w/api.php but nothing does that (and it's semantic... [01:19:32] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [01:19:34] 10Cloud-VPS, 10MediaWiki-Vagrant: MediaWiki/Apache port is chosen weirdly on Vagrant - https://phabricator.wikimedia.org/T175100#3582958 (10bd808) >>! In T175100#3582939, @bd808 wrote: >> I would expect Vagrant to redirect host: to guest:80 but it redirects it to guest: instead > > This... [02:17:44] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:24:01] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:58:59] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [03:39:44] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [03:42:53] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:47:47] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [04:07:51] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [04:54:44] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [06:45:41] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:12:17] PROBLEM - High iowait on tools-exec-1416 is CRITICAL: CRITICAL: tools.tools-exec-1416.cpu.total.iowait (>11.11%) [07:22:19] RECOVERY - High iowait on tools-exec-1416 is OK: OK: All targets OK [07:25:39] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [08:23:39] (03CR) 10Jean-Frédéric: [C: 032] Construct registrar_url in country table rather than monuments_all [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/376005 (owner: 10Lokal Profil) [08:23:56] (03CR) 10Jean-Frédéric: [C: 032] "Yes, definitely!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/376005 (owner: 10Lokal Profil) [08:24:28] (03Merged) 10jenkins-bot: Construct registrar_url in country table rather than monuments_all [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/376005 (owner: 10Lokal Profil) [08:25:18] (03CR) 10jenkins-bot: Construct registrar_url in country table rather than monuments_all [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/376005 (owner: 10Lokal Profil) [08:34:09] 10Tool-Attribution-Generator, 10TCB-Team: Corrections in Help Text - https://phabricator.wikimedia.org/T175121#3583359 (10Katja_Ullrich_WMDE) [08:47:29] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, 10Release-Engineering-Team (Watching / External): rabbitmq: Consume and log messages sent to notifications.error - https://phabricator.wikimedia.org/T175029#3583399 (10hashar) OpenStack SearchLight looks lik... [09:14:08] 10Cloud-VPS (Quota-requests), 10Recommendation-API: Request custom instance for recommendation-api labs project - https://phabricator.wikimedia.org/T169766#3583458 (10schana) @bd808 I'm moving this to paused for now, as there aren't available resources to drive the task. For the future, if we still want to loa... [12:49:49] Hey guys, I am trying to use PAWS to run a pywikibot based script [12:50:06] But I am not able to edit user-config file for language and family [12:50:31] As it's mentioned here: https://www.mediawiki.org/wiki/Manual:Pywikibot/PAWS [12:50:37] Any helps? [12:51:26] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/376238 [12:51:47] (03CR) 10Hashar: "check experimental" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/376238 (owner: 10Hashar) [14:53:17] Technical Advice IRC meeting starting at 3 pm UTC/5 pm CEST in channel #wikimedia-tech, hosts: @addshore & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:08:23] 10Cloud-Services, 10Tracking, 10User-bd808: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#3584623 (10bd808) [15:08:25] 10Cloud-VPS (Quota-requests), 10Recommendation-API: Request custom instance for recommendation-api labs project - https://phabricator.wikimedia.org/T169766#3584621 (10bd808) 05Open>03stalled [15:16:46] amisha: I'm trying to start a PAWS container myself so I can walk through testing, but for now, what did you try and how did it fail? [15:17:38] I *think* you should be able to create a $HOME/user-config.py file with the mylang and family settings using a text editor from the PAWS shell. [15:18:07] bd808, Actually was trying to edit 'user-config.py` file in /home dir but was not able to [15:18:23] But the error is resolved. Was just a stupid 'langugae [15:18:28] Code issue` [15:18:42] ah. glad you found a solution [15:18:48] But still wondering why I wasn't able to edit user-config file [15:19:10] do you remember what error message (if any) you got? [15:19:38] * bd808 is not getting his paws container to start [15:21:07] pasting [15:21:46] https://pastebin.com/8i7nzvc0 [15:21:49] bd808, [15:21:50] ^ [15:22:14] * bd808 looks [15:22:34] First I got above, but I went in edit mode, and was trying to write but it wasn't able to [15:22:37] okie [15:24:21] amisha: ok. This error looks scarier than it really is. What it is saying is that at some point you either had the file open in two copies of vim at the same time *or* that you opened it with vim and then vim crashed or was closed without cleaning up its temp file [15:25:11] Usually I would quit vim and then delete the swap file before trying the edit again [15:25:11] ah [15:25:20] I see [15:25:22] Let me try [15:26:16] the "process ID: 209 (still running)" sounds like you had the file open in vim in 2 different windows somehow [15:27:06] Now I get this [15:27:08] "user-config.py" [15:27:09] "user-config.py" E212: Can't open file for writing [15:27:13] madhuvishy: no rush, but if you get a chance could you look for and kill stuck PAWS pods for my "BDavis (WMF)" user? [15:27:59] amisha: if you exit vim and then run "ls -l ~/user-config.py" what does it tell you? [15:28:17] sounds like the file ownership is messed up somehow [15:28:57] `ls: cannot access '/home/paws/user-config.py': No such file or directory` [15:29:12] bd808, ^ [15:29:34] ok, so the file does not exist but the directory permissions are keeping you from creating it. [15:29:55] (I wish I could get a PAWS shell running to poke around) [15:30:19] Oops, anything I can do to change permissions [15:31:05] amisha: I'd like to get inside PAWS myself to help figure that out. I don't want to tell you the wrong things to do and make things worse. [15:31:12] * bd808 tries another account [15:32:12] No worries bd808 , thanks a lot for helping me out with this [15:32:14] : ) [15:32:31] _will wait_ [15:34:52] PAWS has been known to have funny issues with my staff account. Hopefully my volunteer account will work better... [15:36:24] Hmm I see [15:36:46] Also am wondering if Wikidata integrator is better than pywikibot? [15:37:06] I have been pywikibot so far and found it to be reliable [15:37:30] But mostly setting it up is a big pain [15:37:41] Otherwise it works amazingly [15:37:55] heh. it does have a lot of options that can be changed [15:38:03] Wondering how reliable is Wikidata intergrator [15:38:17] I haven't used that one [15:39:00] Ah okay [15:39:34] And any comparison between paws and tool forge ? [15:39:55] There may be people in the #wikidata channel with opinions on pywikibot vs other things [15:41:10] Cool I can ping there [15:41:16] PAWS can do pretty much anything that Toolforge can. Sometimes it is easier to get help/advice for Toolforge just because it has been around longer and more people are familiar with it. [15:41:20] bd808: uhhh I have no idea how [15:41:41] madhuvishy: ok. I've been poking around for PAWS admin docs and not found any yet. :/ [15:41:47] I know that new paws master is at tools-paws-master-01 [15:41:50] https://www.irccloud.com/pastebin/px1e2K3J/ [15:42:07] there isn't any, and everything is different now [15:42:13] that looks not so good :/ [15:42:23] time to try the yuvi-signal :) [15:42:38] yup yup [15:42:50] need to make him write admin docs! [15:44:57] amisha: This may take a while unfortunately. Would you mind opening a Phabricator task so that we can get you answers once we have a way to do some testing? [15:45:35] Yes sure, no urgency with this though. Creating a Phabricator task [15:45:44] thanks! [15:45:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:45:48] My first Phabricator task probably :p [15:45:50] : ) [15:46:14] Its a useful tool for getting technical support :) [15:46:55] Great [15:55:14] 10PAWS, 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3584797 (10Amishas157) [15:56:06] 10PAWS, 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3584811 (10Amishas157) [15:56:53] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3584815 (10bd808) [16:15:48] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:42] 10Cloud-Services, 10Operations: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3585446 (10madhuvishy) @Robh @Cmjohnson I'm able to log in to both machines with their .wikimedia.org hostnames and run puppet fine. However, when I hop into the serial console, they bo... [17:38:35] 10Cloud-Services, 10cloud-services-team (Kanban), 10Wikimedia-Mailing-lists: Create cloud-admin and archive labs-admin mailing list - https://phabricator.wikimedia.org/T167155#3585525 (10bd808) [17:40:24] 10Cloud-Services, 10Operations: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3585536 (10Cmjohnson) Probably could use a bios update. [17:42:48] 10Cloud-Services, 10cloud-services-team (Kanban), 10Wikimedia-Mailing-lists: Create cloud-admin and archive labs-admin mailing list - https://phabricator.wikimedia.org/T167155#3585544 (10bd808) Task description has been changed to describe a create and archive change rather than a rename. [17:45:08] 10cloud-services-team (FY2017-18), 10Goal, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3585580 (10bd808) [17:48:43] 10PAWS: Enable downloading notebooks as PDF - https://phabricator.wikimedia.org/T159617#3585598 (10Tbayer) Tried it out on PAWS and it works great, thanks! (This will be especially useful for archiving notebooks on our own sites - Commons - apart from/instead of third-party ones like GitHub.) Per the task descr... [17:52:02] 10Cloud-Services, 10Wikimedia-Mailing-lists: Create cloud mailman list and archive labs-l list - https://phabricator.wikimedia.org/T175190#3585613 (10bd808) [17:55:10] 10cloud-services-team (FY2017-18), 10Goal, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3585648 (10bd808) [18:16:31] 10Cloud-VPS (Quota-requests), 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Request increased quota for wikidata-query labs project - https://phabricator.wikimedia.org/T175196#3585759 (10Smalyshev) [19:06:22] 10cloud-services-team, 10Community-Tech, 10DBA, 10Security: create production ip_changes table for RangeContributions - https://phabricator.wikimedia.org/T173891#3585986 (10Umherirrender) [19:09:50] 10Cloud-Services, 10wikitech.wikimedia.org, 10Operations, 10HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#1278203 (10Jdforrester-WMF) >>! In T98813#3365586, @bd808 wrote: >>>! In T98813#3135116, @greg wrote: >> Added T161553 as a subtask per above comments. > > I removed... [19:16:36] 10PAWS: Cull Idle does not work on pAWS - https://phabricator.wikimedia.org/T175202#3586010 (10yuvipanda) [19:18:37] 10PAWS: Wikivoyage is missing from PAWS system user-config.py - https://phabricator.wikimedia.org/T174867#3575394 (10yuvipanda) whoops. It would be great if someone could make a PR on https://github.com/yuvipanda/paws fixing it :) If not I'll try to fix this next week. [19:19:06] 10PAWS: 504 Gateway Time-Out when loading PAWS - https://phabricator.wikimedia.org/T173983#3547077 (10yuvipanda) is this still the case? [19:29:54] 10cloud-services-team (FY2017-18), 10Goal, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3586073 (10Quiddity) [19:52:02] 10PAWS: Cull Idle does not work on pAWS - https://phabricator.wikimedia.org/T175202#3586010 (10bd808) @yuvipanda had a hunch that dns was not working as hoped inside the containers, but: ``` bdavis_(wmf)@PAWS:~$ dig paws.wmflabs.org ; <<>> DiG 9.10.3-P4-Ubuntu <<>> paws.wmflabs.org ;; global options: +cmd ;; Go... [20:00:31] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3586236 (10bd808) From inside a terminal in my PAWS container: ``` bdavis_(wmf)@PAWS:~$ ls -ld . drwxr-sr-x 6 tools.paws tools.paws 4096 Sep 6 19:55 . bdavis_(wmf)@P... [20:13:44] !log tools.heritage Deploy latest from Git master: 7874ca3 (T173929), c53b5dc, 2b37db6 (T174901), 7206e0d [20:13:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [20:13:50] T173929: [co-es] Base class for import of Colombia - https://phabricator.wikimedia.org/T173929 [20:13:50] T174901: [nl-aw] Class for Netherlands / Aruba - https://phabricator.wikimedia.org/T174901 [20:15:58] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3586370 (10bd808) I checked the file permissions for `/data/project/paws/userhomes/48602252` which is @Amishas157's home directory on the NFS server side. They look j... [20:19:08] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3586378 (10bd808) @Amishas157, there were some issues with the PAWS cluster that were fixed between when you were having problems creating the config file and when I... [20:19:20] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3586381 (10bd808) 05Open>03stalled [20:45:41] 10Cloud-Services, 10Operations: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3586455 (10RobH) IIRC the bios is already the newest version. I flashed the bios and the ilom when I installed them. [20:56:25] !log rcm Neon: Updating Software [20:56:25] !log rcm Xenon: Updating Software [20:56:25] !log rcm CAC: Running vagrant git-update [20:56:25] !log rcm Tin: Updating Software [20:56:25] !log rcm Oxygen: Updating Software [20:56:26] !log rcm Neon: Restarting webservice [20:56:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [20:56:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [20:56:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [20:56:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [20:56:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [20:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:11:05] 10Cloud-Services, 10Design: Update Cloud Services logo+wordmark - https://phabricator.wikimedia.org/T174094#3550694 (10BFlores) Hi everyone! I'm Blanca, the designer on the communications team. I will help you update this logo. The project brief only mentions updating the text in the logo. Do you want to use t... [21:42:20] 10Cloud-Services, 10Design: Update Cloud Services logo+wordmark - https://phabricator.wikimedia.org/T174094#3586698 (10Quiddity) Yes, the logo mark remains the same (plain version at https://commons.wikimedia.org/wiki/File:Wikimedia_labs_logo.svg ), we just need the text updated, and properly styled. Thanks! [21:55:00] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#3586729 (10Yarl) [22:37:52] 10cloud-services-team (FY2017-18), 10Goal, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3586867 (10Quiddity) [22:45:38] quiddity: I actually made new banners but didn't swap them in place -- https://tools-static.wmflabs.org/toolforge/ [22:46:15] It wouldn't be hard to change the ones that people are using from https://tools-static.wmflabs.org/static/logos/ though [22:46:56] * bd808 wants to cleanup https://tools-static.wmflabs.org/static/ at some point [23:00:30] I looked for the code in https://phabricator.wikimedia.org/source/tool-keystone-browser/ but couldn't find it, hence just added details to that task. :) [23:00:51] (now offline for a bit) [23:03:36] bd808: +1 [23:15:26] 10Cloud-Services, 10Toolforge: Still issues with node.js webservice - https://phabricator.wikimedia.org/T160353#3586969 (10bd808) 05Open>03declined part duplicate and part unactionable [23:18:31] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3586971 (10Dispenser) From arwiki: [[https://ar.wikipedia.org/wiki/File:%D8%A7%D9%84%D9%84%D9%87_%D8%B9%D8%B2_%D9%88%D8%AC%D9%84.png|File:الله عز وجل.png]]. Deleted in Nov 2015. ```lang=bash $ sql arwiki_p...