[00:29:23] 10Cloud-Services, 10wikitech.wikimedia.org, 10Operations, 10HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#3587052 (10bd808) >>! In T98813#3585990, @Jdforrester-WMF wrote: > Does that mean we can Just Do It™ now? See T168470. Cloud Services has 2 new physical servers that... [00:30:22] 10Cloud-Services, 10wikitech.wikimedia.org, 10Operations, 10HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#3587066 (10Jdforrester-WMF) Brilliant. :-) [00:31:21] 10Cloud-Services, 10wikitech.wikimedia.org, 10Operations: Determine whether wikitech should really depend on production search cluster - https://phabricator.wikimedia.org/T110987#1591503 (10bd808) >>! In T110987#1610756, @chasemp wrote: > We could run a local instance of elasticsearch? Could we, probably. S... [00:31:35] bd808: As well you know already, you and your team are awesome. [00:31:58] we like to make plans anyway :) [00:33:22] bd808: Plan to get me to buy y'all a beer when I can stop silencing silver from my scouring prod logs for MW errors. :-) [00:33:40] :) [00:35:53] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10MusikAnimal) ```lang=sql SELECT COUNT(rev_id) FROM enwiki_p.revision WHERE rev_page = 48357647 ``` returns 4 when it should be 0, as the [[ https://en.wikipedia.org/wiki/Draft:Sohail_Khan | page... [00:37:10] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:57:30] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3587107 (10bd808) >>! In T138967#3586971, @Dispenser wrote: > From arwiki: [[https://ar.wikipedia.org/wiki/File:%D8%A7%D9%84%D9%84%D9%87_%D8%B9%D8%B2_%D9%88%D8%AC%D9%84.png|File:الله عز وجل.png]]. Deleted... [00:59:31] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3587109 (10bd808) [01:03:41] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:05:09] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3587122 (10bd808) @jcrespo & @Marostegui: do either of you have a reasoned objection to us using the `.(web|analytics).db.svc.eqiad.... [01:12:10] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [01:39:36] 10Cloud-Services, 10wikitech.wikimedia.org, 10Operations: Determine whether wikitech should really depend on production search cluster - https://phabricator.wikimedia.org/T110987#3587170 (10Dzahn) The reasons to keep it self-contained (information available if other stuff is down) are probably also mitigate... [01:48:41] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [03:52:55] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:12:56] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [04:44:55] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Promote beta test of new Wiki Replica servers - https://phabricator.wikimedia.org/T172704#3587253 (10Samwilson) I've switched [[ http://tools.wmflabs.org/ws-cat-browser/ | ws-cat-browser ]] to use `wikireplica-analytics.eqiad.wmnet` and all seems... [05:24:00] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/376466 (owner: 10L10n-bot) [05:24:32] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3587275 (10Marostegui) >>! In T174860#3587122, @bd808 wrote: > @jcrespo & @Marostegui: do either of you have a reasoned objection to us us... [06:16:07] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3587310 (10Amishas157) @bd808 , I am still facing the issue. Have left the terminal open. Thanks [06:57:06] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:37:05] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [08:12:31] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3587433 (10jcrespo) Looks ok to me. I was worried if underscores would be allowed on dns entries (which some wikis sadly have, which are a... [08:20:16] 10Cloud-VPS: wikistream.wmflabs.org down - unable to ssh to ws-web - https://phabricator.wikimedia.org/T174850#3587435 (10edsu) 05Open>03Resolved a:03edsu Yes, I'm able to log in now. Thanks so much! [08:25:00] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:00:01] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [09:56:42] 10Tool-Attribution-Generator, 10TCB-Team: Differentiate events between languages in Piwik - https://phabricator.wikimedia.org/T175246#3587689 (10Katja_Ullrich_WMDE) [11:21:09] 10Tool-Attribution-Generator, 10TCB-Team: Swith between languages - https://phabricator.wikimedia.org/T175251#3587833 (10Katja_Ullrich_WMDE) [11:21:18] 10Tool-Attribution-Generator, 10TCB-Team: Switch between languages - https://phabricator.wikimedia.org/T175251#3587847 (10Katja_Ullrich_WMDE) [11:45:32] 10Tool-Pageviews: Userviews halts if PageViews API fails - https://phabricator.wikimedia.org/T175254#3587873 (10Shyamal) [12:37:59] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:42:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:44:03] andrewbogott chasemp: I cannot login anymore due to login {result Failed reason {You have made too many recent login attempts. Please wait 5 minutes before trying again.}} by API login for user:Doc_Taxon for more than 24 hours now. Can anyone give free the login for this user, please? (But a login onwiki is possible (shrug)) [13:08:33] can anyone help me anyhow? ^ [13:17:47] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [13:44:19] 10Data-Services, 10DBA, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Blocked-on-schema-change: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3588328 (10jcrespo) @bd808 @Anomie I think this could "break" some tools, maybe it would be nice to give to "announce... [14:18:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:40:18] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3588514 (10chasemp) >>! In T174860#3587433, @jcrespo wrote: > Looks ok to me. I was worried if underscores would be allowed on dns entries... [14:42:58] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [14:44:43] 10Data-Services, 10DBA, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Blocked-on-schema-change: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3588548 (10Anomie) This change itself shouldn't break any tools. I can't promise that it won't because tools might be... [14:47:23] 10Tools: Tool "montage-beta" loads fonts from google - https://phabricator.wikimedia.org/T172741#3588560 (10Yarl) a:03Yarl [14:47:48] 10Tools, 10Montage: Tool "montage-beta" loads fonts from google - https://phabricator.wikimedia.org/T172741#3507869 (10Yarl) [14:48:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [14:55:19] doctaxon: api.php is locking you out for failed attempts? [14:56:21] yes [14:56:45] login onwiki is possible only by using captcha [14:57:02] but there is no chance to login by api [14:57:35] that throttle is a rolling 24 hour count. I don't know of any way to clear it for a user [14:57:55] 24 hour? [14:58:11] * bd808 looks for the prod config [14:58:35] 10Data-Services, 10DBA, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Blocked-on-schema-change: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3588577 (10jcrespo) Thank you @Anomie, that is exactly why I was confused and why I said I didn't know all the steps... [14:59:09] if i try it now again, will the countdown of this 24 hour count start again by 0 [15:01:20] the defaults from DefaultSettings are to check for >5 bad attempts in a 5 minute window and for >150 bad attempts in a 48 hour window it looks like [15:01:35] 10Data-Services, 10DBA, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Blocked-on-schema-change: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3588585 (10jcrespo) @Anomie one last question, related to the process, this task (T174569) is blocking the process, w... [15:01:40] I don't see a config override for Wikimedia wikis [15:03:40] bd808: okay, it was >150 bad because of an error [15:03:51] so I have to wait 48 hours? [15:05:36] doctaxon: messing with the production wiki state isn't really something the Cloud Services team does. You could try asking for help from someone like Reedy or no_justification (Chad/^demon) in the operations channel, but I don't know if they can do anything about it either. [15:06:15] bd808: okay, I will wait 48 hours [15:06:53] or how long I have to wait [15:08:58] It will keep blocking you until there are 150 or fewer failures in the cache. It tracks by IP + account if I am reading the code and config correctly. [15:11:14] bd808: the local cache? [15:11:29] its a global cache across all Wikimedia wikis [15:12:37] bd808: so I only have to empty/delete the cache? [15:12:42] but how [15:12:55] doctaxon: no, its *our* cache on the server side [15:13:13] cool, please push the button! [15:13:28] the point is to protect accounts from brute force password attacks [15:13:36] right [15:13:50] I can not mess with it [15:14:06] okay, how will the cache loose my misattempts? [15:14:20] *when, not how [15:15:42] each attempt isa timestamp. it checks each time to for timestamps in the various windows that are configured. Similar to SELECT COUNT(1) FROM login_failures WHERE ip=X and user=Y and time <= NOW() - 48 hours. [15:17:06] okay, thank you. great help [15:18:47] bd808: Table 'dewiki_p.login_failures' doesn't exist [15:19:43] its not actually stored in the database and even if it was that would be sensitive data that is not replicated [15:19:49] you have to wait [15:20:10] and you should probably figure out how to fix your bot so that it doesn't do this to you again in the future [15:21:01] okay, thank you [15:28:41] 10Tool-Pageviews: Userviews halts if PageViews API fails - https://phabricator.wikimedia.org/T175254#3588683 (10MusikAnimal) @Shyamal I just ran it at http://tools.wmflabs.org/userviews/?project=en.wikipedia.org&platform=all-access&agent=user&namespace=0&redirects=0&range=latest-20&sort=views&direction=1&view=li... [15:30:03] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3588686 (10bd808) >>! In T174860#3587433, @jcrespo wrote: > Are you (cloud) going to take care of changing the dns every time a wiki is ad... [15:36:14] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3588719 (10jcrespo) There is: puppet:modules/toollabs/files/sql And puppet:modules/role/manifests/labs/dnsrecursor.pp I do not dare to... [15:46:15] 10Data-Services, 10DBA, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Blocked-on-schema-change: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3588736 (10Anomie) No problem at all, I'm happy to answer good questions. I should be on IRC as usual, 1300 UTC to 21... [15:49:09] 10Data-Services, 10DBA, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Blocked-on-schema-change: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3588742 (10jcrespo) p:05Low>03Normal Setting it to normal, I will adjust it to High if more time passes with no f... [16:00:08] 10Tool-Pageviews: Userviews halts if PageViews API fails - https://phabricator.wikimedia.org/T175254#3588775 (10Shyamal) @MusikAnimal Yes, your query works. Perhaps as you say the issue is with bigger queries. I was using a 1 year time period which I have retried and it seems to get stuck presumably due to memor... [16:09:20] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588797 (10zhuyifei1999) [16:10:22] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3588811 (10jcrespo) As an advice you have been hearing from me many times- all refactoring is cool and I am more than ok with it, but plea... [16:15:01] 10Tool-Pageviews: Userviews halts if PageViews API fails - https://phabricator.wikimedia.org/T175254#3588847 (10MusikAnimal) It still worked for me when querying for the last 365 days http://tools.wmflabs.org/userviews/?project=en.wikipedia.org&platform=all-access&agent=user&namespace=0&redirects=0&range=latest-... [16:15:43] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3588865 (10bd808) I'll put up a patch to generate the full mappings as CNAMEs to the two dbproxy servers. I agree that we can iterate on t... [16:16:23] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588868 (10zhuyifei1999) (Originally reported by @IKhitron on https://www.mediawiki.org/wiki/Topic:Txnc2f9ndpwke6w4) [16:24:40] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588895 (10zhuyifei1999) https://quarry.wmflabs.org/query/21425, 2048 characters plain text can be downloaded. [16:29:29] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588907 (10zhuyifei1999) https://quarry.wmflabs.org/query/21426, 151 characters url can be but 279 cannot. [16:43:33] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588921 (10zhuyifei1999) The same happens with xlsxwriter 0.9.9 instead of 0.5.2. [16:44:36] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588922 (10zhuyifei1999) ``` Sep 7 16:42:41 jessie python[26945]: /srv/venv/local/lib/python2.7/site-packages/xlsxwriter/worksheet.py:831: UserWarning: Ignoring URL 'http://www.example.com/aaaaaaaaaaaaaaaaaaa... [16:45:53] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588924 (10IKhitron) Wait a moment, it's excel's bug??? [16:51:28] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588928 (10zhuyifei1999) This is raised in https://github.com/jmcnamara/XlsxWriter/blob/b4c4b499ffb3db8e0fa1b306880bcbcb3675fd4d/xlsxwriter/worksheet.py#L828 What do you think of forcing string instead of url... [16:52:14] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588931 (10zhuyifei1999) >>! In T175285#3588924, @IKhitron wrote: > Wait a moment, it's excel's bug??? Yes, you can't possible add urls longer than 255 chars apparently. [16:52:19] 10Quarry: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3588932 (10IKhitron) Excellent for me. [16:56:19] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3588937 (10bd808) >>! In T175167#3587310, @Amishas157 wrote: > @bd808 , I am still facing the issue. Have left the terminal open. Thanks I connected to the running p... [17:09:14] (03PS1) 10Andrew Bogott: Remove presumed typo from hiera def of profile::openstack::main::monitor::spread_check_password [labs/private] - 10https://gerrit.wikimedia.org/r/376547 [17:10:06] (03CR) 10Andrew Bogott: [V: 032 C: 032] Remove presumed typo from hiera def of profile::openstack::main::monitor::spread_check_password [labs/private] - 10https://gerrit.wikimedia.org/r/376547 (owner: 10Andrew Bogott) [17:24:03] 10Data-Services, 10cloud-services-team, 10DBA: Identify tools hosting databases on labsdb100[13] and notify maintainers - https://phabricator.wikimedia.org/T175096#3589046 (10bd808) Initial list of accounts: {P5960} [17:24:27] (03PS1) 10Andrew Bogott: openstack: add a few more stub passwords to labs-private [labs/private] - 10https://gerrit.wikimedia.org/r/376548 [17:24:39] (03CR) 10Andrew Bogott: [V: 032 C: 032] openstack: add a few more stub passwords to labs-private [labs/private] - 10https://gerrit.wikimedia.org/r/376548 (owner: 10Andrew Bogott) [17:29:27] 10Cloud-Services, 10cloud-services-team (Kanban), 10Wikimedia-Mailing-lists: Create cloud-admin and archive labs-admin mailing list - https://phabricator.wikimedia.org/T167155#3589073 (10RobH) a:03RobH Chatted with BD about this via irc, will attempt to get to this later today/tomorrow. [17:31:35] 10Cloud-VPS (Project-requests), 10cloud-services-team (Kanban), 10User-bd808: Request creation of deep-learning-services VPS project - https://phabricator.wikimedia.org/T172421#3589083 (10bd808) 05Open>03Resolved https://wikitech.wikimedia.org/wiki/Nova_Resource:Deep-learning-services https://tools.wmfla... [17:33:56] 10Cloud-VPS (Project-requests), 10cloud-services-team (Kanban), 10User-bd808: Request creation of project-smtp VPS project - https://phabricator.wikimedia.org/T174618#3589086 (10bd808) 05Open>03Resolved https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-smtp https://tools.wmflabs.org/openstack-bro... [17:34:28] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3589088 (10bd808) [18:08:58] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:44:00] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:51] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:14:07] 10Cloud-VPS (Project-requests), 10cloud-services-team (Kanban), 10User-bd808: Request creation of project-smtp VPS project - https://phabricator.wikimedia.org/T174618#3589553 (10herron) Sweet! Is there a vps project getting started type doc I could follow to get up and running? [19:18:09] 10Cloud-VPS (Project-requests), 10cloud-services-team (Kanban), 10User-bd808: Request creation of project-smtp VPS project - https://phabricator.wikimedia.org/T174618#3589565 (10bd808) @herron https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS is probably the best place to start. https://wikitech.wikimedia... [19:37:53] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:00] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#3589736 (10bd808) @Vacio, your LDAP account has been attached on Wikitech, so you should now be able to go to https... [20:27:37] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:32:48] 10Striker, 10wikitech.wikimedia.org: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#3589763 (10bd808) p:05High>03Normal a:05bd808>03None [20:33:26] 10Striker, 10wikitech.wikimedia.org: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#3563019 (10bd808) Task description updated to show remaining tasks. [21:02:36] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [21:04:59] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:20:08] !log git created accounts for the Wikimedia Scoring Platform team due to recreation of auth database [21:20:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [21:20:12] paladox ^ [21:20:26] thanks [21:20:29] np [21:45:00] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0]