[00:09:29] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:42:41] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:44:29] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:20] PROBLEM - Free space - all mounts on tools-logs-02 is CRITICAL: CRITICAL: tools.tools-logs-02.diskspace._srv.byte_percentfree (<100.00%) [01:17:42] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [01:32:54] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:36:29] !log tools Depooling tools-worker-1020 [01:36:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:43:36] !log tools Uncordoned tools-worker-1020 after it deleted pods with local storage that were filling the entire disk [01:43:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:50:03] RECOVERY - Free space - all mounts on tools-worker-1020 is OK: OK: All targets OK [01:50:37] RECOVERY - Puppet staleness on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:58:20] RECOVERY - Free space - all mounts on tools-logs-02 is OK: OK: All targets OK [02:07:32] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:07:54] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [02:22:30] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [02:24:32] 10Tool-Labs-tools-XTools: Convert bash to use IRC quotes rather than Bugzilla quotes - https://phabricator.wikimedia.org/T170333#3442465 (10Matthewrbowker) 05Open>03declined [02:32:27] 10Tool-Labs-tools-XTools: Figure out XTools Git Repositories - https://phabricator.wikimedia.org/T170367#3442467 (10Matthewrbowker) 05Open>03Resolved Okay, repositories have been moved. GitHub should put redirects in place. [03:21:58] 10Cloud-Services: 500 error when trying to create an account on meta (/register/done) - https://phabricator.wikimedia.org/T170783#3442514 (10Niedzielski) [03:25:34] 10Cloud-Services, 10MediaWiki-extensions-OAuth: 500 error when trying to create an account on meta (/register/done) - https://phabricator.wikimedia.org/T170783#3442528 (10Niedzielski) [03:26:22] 10Cloud-Services, 10MediaWiki-extensions-OAuth: 500 error when trying to create an LDAP account with meta unified account (/register/done) - https://phabricator.wikimedia.org/T170783#3442514 (10Niedzielski) [03:26:36] 10Tool-Labs-tools-XTools: Figure out XTools Git Repositories - https://phabricator.wikimedia.org/T170367#3442530 (10Samwilson) URLs updated in the code. [03:33:58] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10LDAP, 10User-bd808: Cannot create account on Wikitech wiki - https://phabricator.wikimedia.org/T170774#3442535 (10bd808) [03:34:00] 10Cloud-Services, 10MediaWiki-extensions-OAuth: 500 error when trying to create an LDAP account with meta unified account (/register/done) - https://phabricator.wikimedia.org/T170783#3442537 (10bd808) [03:34:39] 10Cloud-Services, 10MediaWiki-extensions-OAuth: 500 error when trying to create an LDAP account with meta unified account (/register/done) - https://phabricator.wikimedia.org/T170783#3442514 (10bd808) This has the same root cause as {T170774} [03:35:59] 10Cloud-Services, 10MediaWiki-extensions-OAuth: 500 error when trying to create an LDAP account with meta unified account (/register/done) - https://phabricator.wikimedia.org/T170783#3442540 (10Niedzielski) Thanks! What's the recommended workaround for this issue? (Sorry, I'm not as familiar with the account i... [04:24:47] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Remove automated edits interface in Edit Counter - https://phabricator.wikimedia.org/T170185#3442563 (10Samwilson) 05Open>03Resolved Looks good! Merged and deployed. [04:37:28] 10Tool-Labs-tools-XTools, 10Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3442575 (10Samwilson) [04:37:31] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3442572 (10Samwilson) 05Open>03Resolved The beta-testing notice has been added to old XTools: > A new version of XTools is being beta-tested. Please try it out at xtools.wmflabs.org (Th... [05:15:18] 10cloud-services-team, 10DBA: labsdb1009 crashed while doing an alter table on templatelinks - https://phabricator.wikimedia.org/T170657#3442593 (10Marostegui) 05Open>03Resolved a:03Marostegui The alters finished correctly, I have also stopped the server and started it without any issues, so I am going t... [07:37:24] 10Tool-Labs-tools-XTools, 10translatewiki.net: Add translatewiki.net support for XTools - https://phabricator.wikimedia.org/T170789#3442751 (10Matthewrbowker) [07:37:49] 10Tool-Labs-tools-XTools, 10translatewiki.net: Add translatewiki.net support for XTools - https://phabricator.wikimedia.org/T170789#3442764 (10Matthewrbowker) [07:37:51] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Convert xtools intuition to its own repository - https://phabricator.wikimedia.org/T165708#3442765 (10Matthewrbowker) [08:26:36] 10Cloud-VPS (Project-requests): Request creation of openipmap labs project - https://phabricator.wikimedia.org/T166671#3442816 (10akosiaris) >>! In T166671#3439670, @Pintoch wrote: > Then I will kindly ask @akosiaris if I could have a database on his postgres instance. hehe, not `my` instance. It's a shared po... [10:10:37] 10Cloud-Services, 10Continuous-Integration-Infrastructure, 10Beta-Cluster-reproducible, 10Puppet: New instances attached to a role::puppetmaster::standalone Puppetmaster need manual changes after switching from the default Puppetmaster - https://phabricator.wikimedia.org/T148929#3443054 (10hashar) Thank yo... [11:16:11] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:21:04] PROBLEM - High iowait on tools-exec-1431 is CRITICAL: CRITICAL: tools.tools-exec-1431.cpu.total.iowait (>11.11%) [11:27:19] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:36:04] RECOVERY - High iowait on tools-exec-1431 is OK: OK: All targets OK [11:56:11] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:20] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:04] 10Tool-Labs-tools-XTools: 500: Internal Server Error - https://phabricator.wikimedia.org/T170808#3443346 (10Sphilbrick) [13:12:23] 10Tool-Labs-tools-XTools: 500: Internal Server Error - https://phabricator.wikimedia.org/T170808#3443346 (10Samtar) ```You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 's_Basketball_Tournament' AND log_timestamp > 1 AND log_ty... [13:13:40] 10Tool-Labs-tools-XTools: Section links broken in Edit Counter Latest Global Edits - https://phabricator.wikimedia.org/T170809#3443374 (10Izno) [13:14:03] 10Tool-Labs-tools-XTools: Section links broken in Edit Counter Latest Global Edits - https://phabricator.wikimedia.org/T170809#3443386 (10Izno) [13:32:13] 10Data-Services, 10Toolforge, 10cloud-services-team (Kanban), 10Wikimedia-Incident: Cleanup: 2017-07-02 Toolforge data loss for permissive data - https://phabricator.wikimedia.org/T169774#3443511 (10chasemp) p:05High>03Lowest [13:32:31] 10Tool-Labs-tools-XTools: 500: Internal Server Error - https://phabricator.wikimedia.org/T170808#3443512 (10Sphilbrick) Thanks for the incredibly timely response (maybe we could use you at OTRS). I like to make sure I'm interpreting your response correctly. The first and it sounds like I'm supposed to look up s... [13:34:09] 10cloud-services-team (FY2017-18), 10Goal: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3443551 (10chasemp) [13:34:12] 10Cloud-Services, 10Cloud-VPS, 10Epic: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3443550 (10chasemp) [13:35:25] 10Cloud-Services, 10Cloud-VPS, 10Epic: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3323568 (10chasemp) [13:35:27] 10Cloud-Services, 10cloud-services-team (Kanban), 10Operations: Initial OpenStack Neutron PoC deployment in Labtest - https://phabricator.wikimedia.org/T153099#3443553 (10chasemp) 05Open>03Resolved Resolving in favor of T167559 which will have more details and hopefully some authoritative plan. I don't... [13:37:46] 10cloud-services-team (FY2017-18), 10Goal: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3443560 (10chasemp) [13:38:03] 10cloud-services-team (FY2017-18), 10Goal: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3336801 (10chasemp) [13:39:11] 10cloud-services-team (FY2017-18), 10Goal: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3336801 (10chasemp) [14:24:46] 10Data-Services, 10DBA, 10WMF-Legal, 10Patch-For-Review: Expose ar_content_format and ar_content_model columns of archive table on Labs replicas - https://phabricator.wikimedia.org/T89741#3443770 (10Umherirrender) Adding wmf-legal as suggested on the patch set [14:26:30] 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Patch-For-Review: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3443790 (10Andrew) Since https://gerrit.wikimedia.org/r/364901 there haven't been any obvious timeouts in contintclou... [14:30:58] 10Cloud-Services, 10MediaWiki-extensions-OAuth: 500 error when trying to create an LDAP account with meta unified account (/register/done) - https://phabricator.wikimedia.org/T170783#3443809 (10bd808) >>! In T170783#3442540, @Niedzielski wrote: > Thanks! What's the recommended workaround for this issue? (Sorry... [14:37:54] 10Cloud-Services, 10DBA, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3443849 (10Marostegui) s6 is now available on the new labs servers. I have created the views too. s2 is being imported now [14:37:59] 10Cloud-Services, 10DBA, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3443850 (10Marostegui) [14:39:59] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:49:20] 10cloud-services-team (FY2017-18), 10Goal: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3443876 (10chasemp) = Dual stacks and migration and options As part of the larger T167293 we know we need to move instances ultimately to... [14:50:18] 10Cloud-Services, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3443883 (10Marostegui) As per T168782#3443867 the first user was created. I can see it in production but correctly filtered on labs. I will wait for a few more to be created... [14:51:40] 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3433128 (10hashar) [14:52:37] PROBLEM - Puppet errors on tools-prometheus-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:07:57] PROBLEM - Puppet errors on tools-prometheus-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:08:29] 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3443947 (10hashar) For the record, changes to `nodepool.yaml` are... [15:09:27] andrewbogott: good morning :) Thank you for the nodepool/rabbitmq fix up last week, I was in vacations :/ [15:09:55] hashar: I'm still not 100% sure that it was the cause of the problem… if it was it took a while to be very noticeable [15:09:58] I found out you wrote a diamond collector for rabbitmq metrics, though there is apparently a single one reported ( statistics_db_event_queue )https://graphite.wikimedia.org/render/?width=800&height=600&_salt=1500303965.525&from=-10days&target=servers.labcontrol1001.rabbitmq.statistics_db_event_queue [15:10:38] well if Rabbitmq is on the edge of saturation and nodepool becomes super busy, that would surely be a good suspect to cause an issue [15:12:51] hm, that graph changed pretty dramatically when I slowed down the rate and then sped it up again [15:13:15] I really feel like rabbit should be able to handle this much traffic, but [15:13:21] there is a bunch of other metrics under servers.labcontrol1001.rabbitmq.* but they are apparently all empty [15:13:30] maybe rabbitmq exposes other stats ? [15:15:29] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:15:29] ohhh [15:16:18] hashar: I'm mired in other things but I'm happy for you to mess with that collector if you want. [15:16:22] i was looking for metrics on graphite.wikimedia.org but it does not show all of them [15:16:33] andrewbogott: I will try to craft some dashboard in Grafana [15:19:08] 10Cloud-VPS: Build VPS base images - https://phabricator.wikimedia.org/T170828#3443986 (10Andrew) [15:20:02] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:46] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10LDAP, 10User-bd808: Cannot create account on Wikitech wiki - https://phabricator.wikimedia.org/T170774#3444042 (10bd808) Fixed the numeric uid values that were placed in the wrong range: ``` name=T170774-fix-uid.ldif dn: uid=jberkley,ou=people,dc... [15:33:30] 10Cloud-Services, 10Wikimania-Hackathon-2017: History & Purpose of Wikimedia Cloud Services @ Wikimania Hackathon - https://phabricator.wikimedia.org/T170833#3444078 (10madhuvishy) [15:33:47] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10LDAP, 10User-bd808: Cannot create account on Wikitech wiki - https://phabricator.wikimedia.org/T170774#3444092 (10bd808) 05Open>03Resolved Verified fix by creating: * https://wikitech.wikimedia.org/wiki/User:T170774_Wikitech * https://wikitech... [15:35:53] 10Cloud-Services, 10Wikimania-Hackathon-2017: History & Purpose of Wikimedia Cloud Services @ Wikimania Hackathon - https://phabricator.wikimedia.org/T170833#3444101 (10madhuvishy) [15:42:43] !log ores deleting ores-web-04 (intermittent puppet issues and wasn't pooled anyway) [15:42:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [15:45:29] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:38] (03Draft1) 10Paladox: Remove ores-web-04 [labs/icinga2] - 10https://gerrit.wikimedia.org/r/365620 [15:47:41] (03PS2) 10Paladox: Remove ores-web-04 [labs/icinga2] - 10https://gerrit.wikimedia.org/r/365620 [15:47:50] (03CR) 10Paladox: [V: 032 C: 032] Remove ores-web-04 [labs/icinga2] - 10https://gerrit.wikimedia.org/r/365620 (owner: 10Paladox) [15:52:00] 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3444210 (10hashar) There is too many metrics under `servers.labco... [15:52:58] andrewbogott: https://grafana.wikimedia.org/dashboard/db/labs-queue-server-rabbitmq :] [15:54:56] but that does not show much [15:57:19] that 'total objects' graph looks like there's a leak? [15:57:31] Maybe it doesn't mean what I think it means. [16:01:06] 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3444291 (10hashar) And on the [[ https://grafana-admin.wikimedia.... [16:01:34] andrewbogott: and apparently the swap was full https://phabricator.wikimedia.org/T170492#3444291 at some point [16:01:55] andrewbogott: the total objects, I can not tell :( [16:03:05] I know nothing about rabbitmq [16:04:41] team meeting time && dinner [16:19:25] (03Draft1) 10Paladox: Update a few things [labs/icinga2] - 10https://gerrit.wikimedia.org/r/365623 [16:19:27] (03PS2) 10Paladox: Update a few things [labs/icinga2] - 10https://gerrit.wikimedia.org/r/365623 [16:24:43] (03CR) 10Paladox: [V: 032 C: 032] Update a few things [labs/icinga2] - 10https://gerrit.wikimedia.org/r/365623 (owner: 10Paladox) [16:56:46] how can i make it so a git repo my tool has cloned auto pulls? [16:59:23] (03Draft2) 10Zppix: Update About.php with new info [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/365649 [16:59:27] (03CR) 10Zppix: [V: 032 C: 032] Update About.php with new info [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/365649 (owner: 10Zppix) [17:05:45] 10cloud-services-team (FY2017-18), 10Goal: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3444726 (10chasemp) [17:07:30] 10cloud-services-team (FY2017-18), 10Goal: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3336801 (10chasemp) [17:38:01] 10wikitech.wikimedia.org, 10Operations: Update mediawiki on wikitech-static - https://phabricator.wikimedia.org/T170854#3444900 (10Andrew) [17:38:09] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:42:12] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10LDAP, 10User-bd808: Cannot create account on Wikitech wiki - https://phabricator.wikimedia.org/T170774#3444951 (10Niedzielski) Yay! Thanks @bd808! [18:08:09] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [18:17:24] 10Tool-Labs-tools-Matthewrbowker's-tools: Create a bash script or a script that can be run from the cron daily to download new i18n messages - https://phabricator.wikimedia.org/T170859#3445086 (10Cyberpower678) [18:19:07] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10LDAP, 10User-bd808: Cannot create account on Wikitech wiki - https://phabricator.wikimedia.org/T170774#3445105 (10BethNaught) Indeed, thank you! [18:24:20] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:41:16] PROBLEM - High iowait on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1416.cpu.total.iowait (>11.11%) [18:46:30] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:56:15] RECOVERY - High iowait on tools-webgrid-lighttpd-1416 is OK: OK: All targets OK [18:59:19] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:07] 10Cloud-VPS (Project-requests): Request creation of openipmap labs project - https://phabricator.wikimedia.org/T166671#3445609 (10Pintoch) @akosiaris It will all be much clearer when the app is hosted somewhere. I am using Leaflet.js as a means to display a large image (not a geographical map) by means of tiles... [19:11:29] 10Tool-Labs-tools-XTools: Section links broken in Edit Counter Latest Global Edits - https://phabricator.wikimedia.org/T170809#3445644 (10MusikAnimal) p:05Triage>03Low a:03MusikAnimal [19:28:05] 10Tool-Labs-tools-XTools, 10Security, 10Vuln-Inject: 500: Internal Server Error with ArticleInfo when using an apostriphe in article title - https://phabricator.wikimedia.org/T170808#3445715 (10Aklapper) [19:28:17] 10Tool-Labs-tools-XTools, 10Security, 10Vuln-Inject: 500: Internal Server Error with ArticleInfo when using an apostriphe in article title - https://phabricator.wikimedia.org/T170808#3443346 (10Aklapper) [Task made public as patch was deployed] [20:08:11] i never got an answer so ill ask again, how can i make it so on toolforge i can have a repo automatically run git pull so i dont have to do it manually [20:15:27] Zppix: it depends on what you mean by 'automatic' [20:15:44] as far as I know we don't have any specific service in place for that, but you could have a cron update things every hour or something like that. [20:16:30] andre__ how would i do that? [20:16:45] oops sorry andre__ [20:16:51] andrewbogott how would i do that [20:17:53] There are some docs here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Scheduling_jobs_at_regular_intervals_with_cron [20:17:57] Which I think are current [20:18:33] I haven't done it, but it's basically just [20:19:21] You'll probably want to have one long command that pulls and then restarts the service [20:19:23] do i have to do cd git-repo-dir [20:20:16] andrewbogott ^ [20:20:40] Probably. Or maybe you can specify the repo dir in the git command. [20:21:00] so i could just do git pull dir ? [20:21:10] im not a expert on git [20:21:15] I don't know, but I bet the git man page knows :) [20:21:41] Not that anyone should have to read the git manpage :( [20:23:05] looks like -C might do it [20:23:05] https://stackoverflow.com/questions/5083224/git-pull-while-not-in-a-git-directory [20:23:19] git pull -C [20:23:43] what to pull = branch i assume [20:24:49] You'll have to experiment, I don't know how -C works exactly [20:24:58] you might have to specify the url entirely, don't know [20:25:20] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:25:39] andrewbogott couldnt i just run cd dir && git pull? [20:25:45] sure [20:25:50] k [20:27:12] well hope that works [21:00:20] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [21:17:34] !log ores-staging deploying 3990160cd50166338ad0d0f77f04c2cfe836cf5d into staging [21:17:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores-staging/SAL [21:28:14] 10Toolforge, 10InternetArchiveBot, 10User-Zppix: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3446295 (10Cyberpower678) p:05Triage>03Normal I'm going to ping @bd808 and @Andrew for this one. [21:30:03] 10Tool-Labs-tools-XTools: Automated edits always uses mainspace - https://phabricator.wikimedia.org/T170888#3446307 (10Train2104) [21:32:44] 10Tool-Labs-tools-XTools: Automated edits always uses mainspace - https://phabricator.wikimedia.org/T170888#3446330 (10MusikAnimal) a:03MusikAnimal [21:44:41] 10Tool-Labs-tools-XTools, 10Documentation: Document algorithm for AdminScore - https://phabricator.wikimedia.org/T170892#3446412 (10Matthewrbowker) [21:50:20] 10Toolforge, 10InternetArchiveBot, 10User-Zppix: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3446470 (10bd808) > Unfortunately it’s tool labs that’s killing the sessions. Are they being stored on disk or in my... [21:54:10] 10Toolforge, 10InternetArchiveBot, 10User-Zppix: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3446479 (10Cyberpower678) >>! In T170849#3446470, @bd808 wrote: >> Unfortunately it’s tool labs that’s killing the se... [21:56:15] 10Tool-Labs-tools-XTools: Add time zone label to edit counter time card - https://phabricator.wikimedia.org/T170894#3446489 (10Ks0stm) [21:56:23] 10Tool-Labs-tools-XTools: wikidata.org should be considered as www.wikidata.org - https://phabricator.wikimedia.org/T170763#3446502 (10Matthewrbowker) a:03Samwilson > An exception occurred while executing 'SELECT user_id FROM `_p`.`user` WHERE user_name = :username LIMIT 1' with params ["Matthewrbowker"]: SQLS... [21:56:28] 10Tool-Labs-tools-XTools: Add time zone label to edit counter time card - https://phabricator.wikimedia.org/T170894#3446504 (10MusikAnimal) a:03MusikAnimal [22:03:18] 10Toolforge, 10InternetArchiveBot, 10User-Zppix: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3446528 (10bd808) Is this running on Grid Engine or on Kubernetes? Are you expecting local disk storage in either env... [22:16:28] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [22:19:30] 10Toolforge, 10InternetArchiveBot, 10User-Zppix: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3446585 (10Cyberpower678) >>! In T170849#3446528, @bd808 wrote: > Is this running on Grid Engine or on Kubernetes? Ar... [22:32:45] !log ores-staging deploying 68326bad3eb96fb81649cb5d0e18b869648c31bb into staging [22:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores-staging/SAL [22:58:58] 10Cloud-Services, 10Phabricator, 10Project-Admins: Replace tracking bug T76375 by new project "Labs Project Requests" - https://phabricator.wikimedia.org/T136910#2352020 (10Liuxinyu970226) @mmodell This tracking task is probably replaced by #cloud-vps-project-requests ? [23:05:39] 10Cloud-Services, 10Phabricator: Create custom Phab form for requesting a new Labs project - https://phabricator.wikimedia.org/T128300#3446722 (10bd808) [23:05:44] 10Cloud-Services, 10Phabricator, 10Project-Admins, 10User-bd808: Replace tracking bug T76375 by new project "Labs Project Requests" - https://phabricator.wikimedia.org/T136910#3446719 (10bd808) 05Open>03Resolved a:05Danny_B>03bd808 >>! In T136910#3446666, @Liuxinyu970226 wrote: > @mmodell This trac... [23:05:48] 10VPS-Projects, 10Discovery, 10Maps, 10Tracking: [DO NOT USE] [tracking] OSM on Labs [superseded by #Maps] - https://phabricator.wikimedia.org/T60797#3446724 (10Liuxinyu970226) [23:10:43] 10Cloud-Services, 10Tracking, 10User-bd808: [DO NOT USE] New Labs project requests (tracking) [superseded by #cloud-vps-project-requests] - https://phabricator.wikimedia.org/T76375#3446731 (10Liuxinyu970226) [23:49:00] 10cloud-services-team (FY2017-18), 10Goal: Program 10 Outcome 3: Outreach - https://phabricator.wikimedia.org/T166406#3446811 (10bd808) [23:49:02] 10cloud-services-team (Kanban), 10User-bd808: Design and submit Wikimania poster art for "FLOSS best practices for bots and tools" - https://phabricator.wikimedia.org/T169919#3446809 (10bd808) 05Open>03Resolved PDF sent to printer [23:50:13] 10cloud-services-team (Kanban), 10User-bd808: Design and submit Wikimania poster art for "What is Cloud Services?" - https://phabricator.wikimedia.org/T169927#3446816 (10bd808) https://github.com/bd808/what-is-cloud-services-poster PDF sent to printer [23:50:51] 10cloud-services-team (FY2017-18), 10Goal: Program 10 Outcome 3: Outreach - https://phabricator.wikimedia.org/T166406#3295299 (10bd808) [23:50:53] 10cloud-services-team (Kanban), 10User-bd808: Design and submit Wikimania poster art for "FLOSS best practices for bots and tools" - https://phabricator.wikimedia.org/T169919#3446818 (10bd808) 05Resolved>03Open Reopening until I get the poster up on commons too.