[01:06:58] (03PS154) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [01:13:00] (03CR) 10Ricordisamoa: [C: 04-2] "PS154 is free of mixins" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [01:16:20] (03PS155) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [01:18:55] (03CR) 10Ricordisamoa: [C: 04-2] "PS155 uses super in a class" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [01:20:13] (03PS156) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [01:24:57] (03CR) 10Ricordisamoa: [C: 04-2] "PS156 changes some vars into consts" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [03:59:22] 10Cloud-Services, 10Outreachy (Round-15): Proposal: Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T177603#3686299 (10bd808) > In my school, the winter vacation commences before Dec 5, 2017 and the next semester starts in the 3rd week of January. But I'll be able to giv... [04:21:31] 10Cloud-Services: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#3686314 (10bd808) Actually since we already expose the user_registration date, we could substitute that date and not leak any new information other than the confirmed status: ``` (wikiadmin@db1080) [... [04:47:57] 10Cloud-Services: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#3686321 (10bd808) >>! In T70876#3686314, @bd808 wrote: > Actually since we already expose the user_registration date, we could substitute that date and not leak any new information other than the con... [04:48:26] 10Data-Services: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#3686323 (10bd808) [04:51:44] 10Data-Services: Cloud services analytics replicas problem - https://phabricator.wikimedia.org/T178155#3682550 (10bd808) I can't recreate this problem. Was it transient for you @magnus? [04:54:10] 10Data-Services: Cloud services analytics replicas problem - https://phabricator.wikimedia.org/T178155#3686329 (10bd808) There seem to be quite a large number of causes for https://dev.mysql.com/doc/refman/5.7/en/gone-away.html [04:55:48] 10Data-Services: ores_classification table corrupt on enwiki labs replica labsdb1001 - https://phabricator.wikimedia.org/T178135#3686332 (10bd808) [04:55:52] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3686331 (10bd808) [04:59:09] 10Data-Services, 10DBA: ores_classification table corrupt on enwiki labs replica labsdb1001 - https://phabricator.wikimedia.org/T178135#3686334 (10bd808) Adding the #dba tag, but the best answer is probably to use the new replicas instead of labsdb1001/1003 ({T142807}). [05:31:22] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3686350 (10Marostegui) [05:31:24] 10Data-Services, 10DBA: ores_classification table corrupt on enwiki labs replica labsdb1001 - https://phabricator.wikimedia.org/T178135#3686347 (10Marostegui) 05Open>03Resolved a:03Marostegui Hi, I have fixed this and also migrated the table to InnoDB which will hopefully prevent this corruption from h... [06:24:29] 10Data-Services, 10DBA: ores_classification table corrupt on enwiki labs replica labsdb1001 - https://phabricator.wikimedia.org/T178135#3686398 (10Bawolff) > But yes, the best answer is to start using the new servers :-) /me as a cloud services newb usually just uses whatever server that `$ sql enwiki_p` gets... [06:38:07] 10Cloud-Services, 10Outreachy (Round-15): Proposal: Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T177603#3686416 (10Sowjanyavemuri) Hi @bd808 , According to the eligibility criteria, I do have 7 weeks free from full-time school(Dec5 - Jan18). Here's my answer for el... [06:40:25] 10Cloud-Services, 10Outreachy (Round-15): Proposal: Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T177603#3686417 (10Sowjanyavemuri) [08:03:49] (03PS6) 10Lokal Profil: [WIP] Add Eslint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378011 (https://phabricator.wikimedia.org/T175907) (owner: 10Jean-Frédéric) [08:05:04] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add Eslint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378011 (https://phabricator.wikimedia.org/T175907) (owner: 10Jean-Frédéric) [08:09:26] (03CR) 10Lokal Profil: [C: 04-1] "The last patch removed two js files (one unused and one now fetched via the wiki cdnjs." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378011 (https://phabricator.wikimedia.org/T175907) (owner: 10Jean-Frédéric) [08:53:42] 10Data-Services: Cloud services analytics replicas problem - https://phabricator.wikimedia.org/T178155#3686619 (10Magnus) 05Open>03Resolved a:03Magnus The problem appears to have resolved itself. I have switched everything back to normal. It it was just a one-off fluke, fine. The problem was that that the... [09:01:51] 10Data-Services, 10DBA: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686654 (10jcrespo) [09:09:43] 10Data-Services, 10DBA: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686680 (10jcrespo) There is not much to do, this is memory exhaustion/load coming from templatetiger/external links monitor. [09:11:32] 10Data-Services, 10DBA: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686683 (10jcrespo) The high load, however, continues: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=19&fullscreen&orgId=1&var-server=labsdb1005&var-network=bond0&from=now-7d&to=now [09:17:54] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Remove 'importers' (note the ending 's') group from wikitech - https://phabricator.wikimedia.org/T171682#3686699 (10Dereckson) p:05Triage>03Low [09:36:12] 10Cloud-Services, 10Toolforge, 10wikitech.wikimedia.org, 10Collaboration-Team-Triage, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#3686728 (10Dereckson) a:05Dereckson>03None [ Deassigning as not a curr... [13:01:16] bd808: fwiw I saw weirdness on android mobile too [13:01:57] ^ me [13:37:01] 10cloud-services-team (Kanban), 10DBA, 10Operations, 10Ops-Access-Requests: Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3681627 (10jcrespo) This is a problem created by cloud people themselves. I told them to create the view user with localhost... [13:42:57] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Remove 'importers' (note the ending 's') group from wikitech - https://phabricator.wikimedia.org/T171682#3687304 (10MarcoAurelio) This needs someone with access to silver I guess. @bd808 do you have access there and can run the SQL query? [14:01:36] bd808: the DB is down again [14:01:49] https://tools.wmflabs.org/iabot/index.php [14:02:04] And now it's up [14:02:15] Now it's down again [14:02:33] Now it's up again. [14:02:45] This is very disruptive. [14:03:02] It appears the tools-db host is very flaky [14:45:34] 10Tool-Database-Queries, 10DBA, 10InternetArchiveBot: Not Working - https://phabricator.wikimedia.org/T178294#3687596 (10Cyberpower678) This appears to be a DB problem. This is making the tool and the bot notoriously unreliable. IABot workers are constantly crashing on different wikis due to DB failures. [15:14:17] 10Toolforge: Elasticsearch credential request for strephit - https://phabricator.wikimedia.org/T178310#3687790 (10Kiailandi) [15:17:45] 10Tool-Database-Queries, 10DBA, 10InternetArchiveBot: Not Working - https://phabricator.wikimedia.org/T178294#3687804 (10Marostegui) Can you give some more details about this? Where is this happening? Where are you trying to connect to? Which user are you using to connect to the DB? [15:18:07] 10Data-Services, 10DBA, 10Security-Team, 10Patch-For-Review: pagetranslation log_type missing on replicas - https://phabricator.wikimedia.org/T178052#3687806 (10Andrew) 05Open>03Resolved a:03Andrew All set. [15:19:34] 10Toolforge, 10cloud-services-team (Kanban): Elasticsearch credential request for strephit - https://phabricator.wikimedia.org/T178310#3687822 (10bd808) [15:20:53] 10Tool-Database-Queries, 10DBA, 10InternetArchiveBot: Not Working - https://phabricator.wikimedia.org/T178294#3687826 (10bd808) Likely related to {T178272} [15:21:22] 10Data-Services, 10DBA, 10InternetArchiveBot: Not Working - https://phabricator.wikimedia.org/T178294#3687829 (10bd808) [15:22:34] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687379 (10bd808) [15:22:46] 10Data-Services, 10DBA: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686654 (10Marostegui) The high load looks gone (or at least as it used to be before): https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=19&fullscreen&orgId=1&var-server=labsdb1005&var... [15:25:13] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687843 (10Cyberpower678) >>! In T178294#3687804, @Marostegui wrote: > Can you give some more details about this? > Where is this h... [15:27:11] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687861 (10Marostegui) >>! In T178294#3687843, @Cyberpower678 wrote: >>>! In T178294#3687804, @Marostegui wrote: >> Can you give so... [15:29:42] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687866 (10Cyberpower678) Yes it is still happening. It usually happens when it can't connect to the DB. [15:32:49] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687873 (10Marostegui) >>! In T178294#3687866, @Cyberpower678 wrote: > Yes it is still happening. It usually happens when it can't... [15:33:04] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687379 (10jcrespo) Could it mean someone is abusing the bot usage? Does it depend on recentchanges or user requests? Is there a li... [15:34:53] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687893 (10Cyberpower678) >>! In T178294#3687873, @Marostegui wrote: >>>! In T178294#3687866, @Cyberpower678 wrote: >> Yes it is st... [15:36:08] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687903 (10Marostegui) >>! In T178294#3687893, @Cyberpower678 wrote: > > You'll find those connections aren't constantly sleeping.... [15:39:32] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687923 (10Cyberpower678) Hmm... IABot doesn't leave connections to idle. Where are they coming from. I do have Workbench open o... [15:40:49] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687924 (10jcrespo) "Usage of connection pools (maintaining open connections without them being in use), persistent connections, or... [15:42:04] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687925 (10Marostegui) It started to work after I removed the limit of 10 connections, so it was clearly trying to create more tha... [15:42:19] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687926 (10Cyberpower678) And I thought it was harder on the server to constantly reconnect when making a query. Where are those t... [15:45:43] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687937 (10jcrespo) > And I thought it was harder on the server to constantly reconnect If this was a dedicated server, pooling wo... [15:46:37] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687940 (10Cyberpower678) >>! In T178294#3687925, @Marostegui wrote: > > It started to work after I removed the limit of 10 connec... [15:51:37] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687951 (10Cyberpower678) >>! In T178294#3687937, @jcrespo wrote: >> And I thought it was harder on the server to constantly reconn... [16:02:02] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687981 (10Cyberpower678) BTW, I'm still getting the errors. [16:03:02] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687983 (10Marostegui) >>! In T178294#3687981, @Cyberpower678 wrote: > BTW, I'm still getting the errors. It works for me everytim... [16:06:02] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687994 (10Cyberpower678) >>! In T178294#3687983, @Marostegui wrote: >>>! In T178294#3687981, @Cyberpower678 wrote: >> BTW, I'm sti... [16:25:49] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688043 (10jcrespo) We need the mysql error... [17:00:13] fyi [17:00:14] https://easylist-downloads.adblockplus.org/easyprivacy.txt [17:00:22] lists "/pageviews/*$domain=~tools.wmflabs.org" [17:01:36] bah. that name [17:01:58] musikanimal: ^ pageviews got on the easyprivacy block list [17:02:12] yeah we had to beg them to do it [17:02:35] if I remember correctly we tried to whitelist all of tools.wmflabs.org but they wouldn't [17:02:58] ah. that's an exception? [17:03:12] yeah [17:03:58] https://hg.adblockplus.org/easylist/rev/a45b2b79abc8 [17:04:19] we asked at https://forums.lanik.us/ [17:04:33] paravoid: ^ its actually a good thing :) [17:05:21] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688145 (10Cyberpower678) 2017-10-16 17:01:25: (mod_fastcgi.c.2673) FastCGI-stderr: PHP Warning: mysqli_connect(): (HY000/1226): U... [17:06:11] oh so $domain=~ means "except"? [17:06:29] guess so! [17:06:48] https://forums.lanik.us/viewtopic.php?f=64&t=29643&p=89053&hilit=tools.wmflabs.org#p89053 [17:07:01] cool, so that explains why /api/rest_v1/metrics/pageviews is blocked :) [17:07:10] that's a pretty stupid block [17:07:26] yeah [17:07:30] ah kaldari mentioned that too in the thread [17:07:41] yeah, we asked about whitelisting it [17:07:54] I was looking at wikistats vs2 [17:07:58] I'm guessing they just overlooked it. If you asked again they might do it [17:07:59] *ask [17:08:18] easyprivacy also blocks piwik.wikimedia.org, because there's a /piwik rule :( [17:08:18] millimetric reported /api/rest_v1/metrics/pageviews buthen didn't respond to the follow up? [17:08:34] piwik should be blocked ;) [17:08:59] heh I guess so [17:10:58] someone should probably try to get /api/rest_v1/metrics/pageviews whitelisted I guess. The variable domain for that may make it difficult to do though [17:11:30] those broad path component blocks are kind of nasty [17:21:34] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688185 (10Marostegui) >>! In T178294#3688145, @Cyberpower678 wrote: > 2017-10-16 17:01:25: (mod_fastcgi.c.2673) FastCGI-stderr: PH... [17:24:48] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688190 (10Cyberpower678) >>! In T178294#3688185, @Marostegui wrote: >>>! In T178294#3688145, @Cyberpower678 wrote: >> 2017-10-16 1... [17:31:40] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688213 (10jcrespo) Cyberpower678 file a ticket to wmcs Data services, and CC me. Do not expect it to be quick. For the time being,... [17:33:43] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688215 (10bd808) >>! In T178294#3688190, @Cyberpower678 wrote: > Thank you. What do I do to request a dedicated DB instance for C... [17:36:38] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688218 (10Cyberpower678) >>! In T178294#3688213, @jcrespo wrote: > Cyberpower678 file a ticket to wmcs Data services, and CC me. D... [17:43:08] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688247 (10jcrespo) > with that and confirmed himself that the load was negligible https://grafana.wikimedia.org/dashboard/file/se... [18:07:14] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688319 (10Cyberpower678) I do find that somewhat difficult to believe. When the new DB was tested, that was with 28 concurrent co... [18:24:36] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688366 (10Marostegui) Let's see how the graph goes after disabling the limit as we did an hour ago... [18:31:48] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688386 (10bd808) tools.cyberbot (s51059) is currently the number 5 user of disk space on tools.labsdb ([[https://tools.wmflabs.org... [18:33:46] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Investigate and implement alternative for showmount based check at instance boot time - https://phabricator.wikimedia.org/T171508#3688393 (10madhuvishy) I've decided to look at alternatives because the /etc/nologin mechanism seems to be fl... [18:44:49] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688455 (10jcrespo) We were commenting this morning the binary_log usage: ```lines=10 SELECT * FROM user_statistics ORDER BY BINL... [18:54:12] 10Data-Services, 10Tools: s52421__commonsdelinquent_p.event needs index on done column? - https://phabricator.wikimedia.org/T178327#3688507 (10bd808) [19:06:32] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688553 (10russblau) I have found an apparent replication issue on one of the new servers: > tools.dplbot@tools-bastion-03:~$ mysql -h enwiki.analytics.db.svc.eqiad.wmflabs enwiki_p > Reading table info... [19:07:50] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688556 (10Cyberpower678) >>! In T178294#3688386, @bd808 wrote: > tools.cyberbot (s51059) is currently the number 5 user of disk sp... [19:12:11] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688577 (10jcrespo) ``` root@labsdb1009[enwiki]> select page_title, cl_to from page, categorylinks where page_id = cl_from and cl_from=335652; +------------+--------------------------------+ | page_title |... [19:13:04] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688580 (10Marostegui) On a core/production server it has the same result: ``` root@PRODUCTION s1 slave[enwiki]> select page_title, cl_to from page, categorylinks where page_id = cl_from and cl_from=33565... [19:14:53] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688581 (10bd808) >>! In T178294#3688556, @Cyberpower678 wrote: > Hmm...That was the major table I spent the most time optimizing.... [19:17:14] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688601 (10Cyberpower678) >>! In T178294#3688581, @bd808 wrote: >>>! In T178294#3688556, @Cyberpower678 wrote: >> Hmm...That was th... [19:45:01] (03PS157) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [19:45:18] bd808: Can we use the currently open ticket as the request for a new VM for the cyberbot project? [19:45:38] Unless there are ways to further optimize the queries. [19:45:55] 10Data-Services, 10Toolforge, 10cloud-services-team (Kanban), 10Wikimedia-Incident: Cleanup: 2017-07-02 Toolforge data loss for permissive data - https://phabricator.wikimedia.org/T169774#3688683 (10chasemp) >>! In T169774#3426773, @chasemp wrote: > marked stalled so we remember to remove the temp restore... [19:47:46] Cyberpower678: What you probably need is just a quota increase for the existing project? [19:47:52] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688688 (10russblau) User error :-) please disregard. [19:48:02] bd808: yep. [19:48:15] But the DB would need to be on it's own VM? [19:48:28] (03CR) 10Ricordisamoa: [C: 04-2] "PS157 replaces a function with an arrow function" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [19:48:38] The IABot exec nodes needs the CPU and RAM for expansion. [19:49:13] *nod* so at https://phabricator.wikimedia.org/project/view/2880/ there are instructions for requesting more cup/ram/disk quota for an existing project. [19:50:01] bd808: can I merge that request with the one requesting a dedicated public IP? [19:50:19] 10Data-Services, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688692 (10jcrespo) We should update https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database/Replica_drift and close this ticket, as it is unlikely that there will be drift again (due to ROW + GTID), u... [19:50:20] Or should I file a seperate one? [19:50:52] Cyberpower678: I think separate. they are distinct issues [19:51:11] Ok [19:51:38] and it sounds a bit like you need to find some additional help to figure out how to maintain your own database server [19:51:50] not sure where that help will come from for you [19:58:01] 10Cloud-VPS (Quota-requests): Request increased quota for cyberbot Cloud VPS project - https://phabricator.wikimedia.org/T178332#3688709 (10Cyberpower678) [19:59:31] bd808: or at least some teaching to do it myself. I'm happy to learn the essentials. [20:00:37] bd808: I filed the ticket btw. [20:00:49] 10Cloud-VPS, 10Patch-For-Review: nova compute hosts disk space alert does not page - https://phabricator.wikimedia.org/T175077#3688741 (10chasemp) ```Resources modified Class[Profile::Base] Parameters differences: --- Class[Profile::Base].orig +++ Class[Profile::Base] @@ - check_disk_critical => False +... [20:01:06] Cyberpower678: thanks. we will look at it tomorrow in the WMCS team meeting [20:01:21] Great. :-) [20:02:42] In the meantime, I'll try not to crash the DB. :p [20:02:50] 10Cloud-VPS, 10Patch-For-Review: nova compute hosts disk space alert does not page - https://phabricator.wikimedia.org/T175077#3688757 (10chasemp) 05Open>03Resolved [20:03:45] 10Data-Services, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3688758 (10MusikAnimal) No errors since October 12. I think we can close this. Thank you again for your assistance! :) [20:04:32] 10Data-Services, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3688761 (10jcrespo) 05Open>03Resolved a:03Marostegui [20:09:19] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3688787 (10greg) 05Open>03Invalid being bold and closing. The context of this is probably long gone. [20:22:24] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10User-bd808: Create and announce timeline for shutting down labsdb100[13] - https://phabricator.wikimedia.org/T175086#3688820 (10bd808) Drafted on wikitech -- https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown We need to pick dates/... [20:22:47] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3688821 (10Dzahn) >Upgrade phab-01.wmflabs.org to the same version like the production one, so new features can be tested there. Actually i don't think this issue has been resolved. Unless phab-01 (or another) lab... [20:27:08] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3688828 (10Paladox) I will rename the domain to phab and redirect hopefully phab-01 to it. so phab.wmflabs.org -> backend is phabricator [20:31:20] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3688840 (10Dzahn) Talked to paladox. Situation was: - instance named phab-01 isn't used anymore (a project admin can delete it please, labs admins would be happy) - URL https://phab-01.wmflabs.org pointed to inst... [20:32:56] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3688848 (10Dzahn) Just one thing to do maybe: Update all wiki references to "https://phab-01" [20:57:14] (03PS1) 10Andrew Bogott: add dummy star.tools.wmflabs.org.key [labs/private] - 10https://gerrit.wikimedia.org/r/384609 [20:57:26] (03CR) 10Andrew Bogott: [V: 032 C: 032] add dummy star.tools.wmflabs.org.key [labs/private] - 10https://gerrit.wikimedia.org/r/384609 (owner: 10Andrew Bogott) [21:02:44] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3688929 (10Luke081515) The old phab instance is now gone. [21:04:25] !log phabricator switch domain from phab-01.wmflabs.org to phab.wmflabs.org [21:04:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [21:07:55] !log phabricator deleted phab-01 at 23:01 CEST, was unused, cleaning up [21:07:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [21:08:12] !log rcm Xenon: Updating Phabricator [21:08:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:10:49] (03Draft1) 10Paladox: Update url for phabricator to phab.wmflabs.org [labs/icinga2] - 10https://gerrit.wikimedia.org/r/384612 [21:10:51] (03PS2) 10Paladox: Update url for phabricator to phab.wmflabs.org [labs/icinga2] - 10https://gerrit.wikimedia.org/r/384612 [21:11:20] (03CR) 10Paladox: [V: 032 C: 032] Update url for phabricator to phab.wmflabs.org [labs/icinga2] - 10https://gerrit.wikimedia.org/r/384612 (owner: 10Paladox) [21:13:17] (03PS1) 10Andrew Bogott: update fake star.tools.wmflabs.org.key to be a bit more convincing [labs/private] - 10https://gerrit.wikimedia.org/r/384613 [21:13:25] (03CR) 10Andrew Bogott: [V: 032 C: 032] update fake star.tools.wmflabs.org.key to be a bit more convincing [labs/private] - 10https://gerrit.wikimedia.org/r/384613 (owner: 10Andrew Bogott) [21:13:38] !log git deployed https://gerrit.wikimedia.org/r/#/c/384612/ [21:13:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [21:25:31] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Investigate and implement alternative for showmount based check at instance boot time - https://phabricator.wikimedia.org/T171508#3467337 (10chasemp) Replied on CR :) [21:36:04] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3689040 (10Dzahn) thanks @Luke081515 ! It turns out we actually still don't have any instance that is using the prod role and can also be used for testing. The instance that does use the prod role redirects to p... [21:36:54] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3689041 (10Dzahn) Additional work is needed to make modules/phabricator/templates/ flexible if we want to be able to actually test prod changes in labs/beta. [21:42:27] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3689047 (10Dzahn) This is T139475 and T144112 and T131899 . Somehow they are all closed yet none of them are really resolved. [21:53:55] 10VPS-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#3689068 (10Paladox) The role works, see https://phab.wmflabs.org/ . Puppet passes locally too. Also phab-01.wmflabs.org is now phab.wmflabs.org. [23:10:43] (03PS158) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [23:16:44] (03CR) 10Ricordisamoa: [C: 04-2] "PS158 changes some vars into lets and consts" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [23:23:39] (03PS159) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [23:28:11] 10Data-Services, 10DBA, 10Tracking: Wikireplica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#2795855 (10bd808) @jcrespo Would you feel ok about closing this tracking bug and instead pointing people to the #data-services tag? If we need... [23:30:00] (03CR) 10Ricordisamoa: [C: 04-2] "PS159 replaces a function with an arrow function" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [23:32:15] 10Cloud-VPS, 10Data-Services, 10DBA: Add pp_propname/pp_value index to Labs replica - https://phabricator.wikimedia.org/T140609#3689293 (10bd808) [23:32:17] 10Data-Services, 10Toolforge: Document all tables missing in replicas - https://phabricator.wikimedia.org/T139289#3689294 (10bd808) [23:32:21] 10Cloud-VPS, 10Data-Services, 10DBA: rev_len should be available also for deleted revisions in database replicas - https://phabricator.wikimedia.org/T101631#3689296 (10bd808) [23:32:23] 10Data-Services, 10MediaWiki-extension-requests, 10Wikimedia-Labs-General: Create web version of metainformation table on replica servers - https://phabricator.wikimedia.org/T56703#3689298 (10bd808) [23:32:25] 10Cloud-VPS, 10Data-Services, 10DBA: Queries of commonswiki_p.filearchive for fa_sha1 are slow - https://phabricator.wikimedia.org/T71088#3689297 (10bd808) [23:53:54] (03PS160) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [23:59:20] (03CR) 10Ricordisamoa: [C: 04-2] "PS160 enables the ESLint dot-notation rule and fixes errors" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)