[02:09:45] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:41:51] 10Cloud-VPS (Project-requests): Request creation of vpsmail VPS project - https://phabricator.wikimedia.org/T174618#3568042 (10Legoktm) Would "vps-mail" also work? My brain first split the two words into "VP smail". [03:19:44] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [03:27:29] 10Cloud-VPS (Project-requests): Request creation of vpsmail VPS project - https://phabricator.wikimedia.org/T174618#3568839 (10bd808) project-smtp? That sort of matches up with the existing project-proxy project. Maybe @herron has a good idea for the project name. [05:02:22] neat ^ [05:58:32] 10Data-Services, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10Marostegui) Hi, Just to make sure I am getting this right, centralnotice database means metawiki databas... [06:02:37] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374925 (owner: 10L10n-bot) [06:28:19] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:32:59] (03PS1) 10Legoktm: Implement single upgrade mode [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374942 [06:33:01] (03PS1) 10Legoktm: Only include necessary things inside the container [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374943 [06:35:32] (03CR) 10Legoktm: [C: 032] Implement single upgrade mode [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374942 (owner: 10Legoktm) [06:35:41] (03CR) 10Legoktm: [C: 032] Only include necessary things inside the container [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374943 (owner: 10Legoktm) [06:35:50] (03Merged) 10jenkins-bot: Implement single upgrade mode [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374942 (owner: 10Legoktm) [06:36:00] (03Merged) 10jenkins-bot: Only include necessary things inside the container [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374943 (owner: 10Legoktm) [06:37:45] 10VPS-project-libraryupgrader: Create list of canary repositories for libraryupgrader - https://phabricator.wikimedia.org/T174646#3568944 (10Legoktm) [06:46:04] PROBLEM - Puppet errors on tools-exec-1421 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:50:46] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:55:03] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:57:16] !log wikilabels ff82aa3 is going to staging [06:57:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [07:00:09] !log wikilabels:ff82aa3 is going to prod [07:00:09] Amir1: Unknown project "wikilabels:ff82aa3" [07:00:17] !log wikilabels wikilabel:ff82aa3 is going to prod [07:00:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [07:08:17] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [07:21:05] RECOVERY - Puppet errors on tools-exec-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:16] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:30:04] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:45] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:21] 10Data-Services, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10Reedy) >>! In T135405#3568908, @Marostegui wrote: > Hi, > > Just to make sure I am getting this right, c... [07:48:27] 10Data-Services, 10cloud-services-team, 10DBA, 10Fundraising-Backlog, and 2 others: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#3569045 (10Marostegui) >>! In T135405#3569043, @Reedy wrote: >>>! In T135405#3568908, @Marostegui wrote: >> Hi, >> >> Just to make sure I... [07:51:31] 10Data-Services, 10cloud-services-team, 10DBA, 10Fundraising-Backlog, and 2 others: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#3569059 (10Reedy) >>! In T135405#2553819, @AndyRussG wrote: >>>! In T135405#2374114, @Base wrote: >> The tables in the desc looks like indee... [08:04:16] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [11:26:36] 10wikitech.wikimedia.org, 10Wikimedia-Video: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3569601 (10Aklapper) {T93041}, for past records. [11:34:17] 10wikitech.wikimedia.org, 10Wikimedia-Video: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3563246 (10Reedy) TimedMediaHandler isn't enabled on the wiki... I guess that might explain to some extent MW on wikitech not knowing how to handle it properly... With it being a re... [13:04:03] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3569772 (10jcrespo) I do not think we should postpone the reboots too much, my proposal would be to: 0) document access to the new hosts (bare essentials) 1)... [13:47:47] 10Cloud-Services, 10DBA: Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#3569870 (10jcrespo) [13:49:54] 10Cloud-Services, 10DBA: Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#3569878 (10jcrespo) [13:50:34] 10Cloud-Services, 10DBA: Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#2546917 (10jcrespo) [13:52:20] !log wikilabels 5cc380f is going to staging [13:52:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [13:52:27] and prod [13:55:08] 10Cloud-Services, 10Cloud-VPS, 10DBA, 10Epic, 10Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#3569906 (10jcrespo) [14:20:24] 10Cloud-VPS (Project-requests): Request creation of vpsmail VPS project - https://phabricator.wikimedia.org/T174618#3570071 (10herron) Sure, project-smtp or project-mail sound good to me :) [14:31:18] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:54:26] 10Cloud-VPS (Project-requests): Request creation of project-smtp VPS project - https://phabricator.wikimedia.org/T174618#3570148 (10bd808) [14:58:32] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3570152 (10bd808) @jcrespo's plan sounds like a good one. Working on the announce of the new cluster was already on my todo list for today, so I'll add foresh... [15:03:47] 10cloud-services-team (Kanban), 10Patch-For-Review: Add redirect and pagelinks tables for partition repair in sqoop job for mediawiki history - https://phabricator.wikimedia.org/T174484#3570156 (10JAllemandou) [15:06:16] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:14] 10cloud-services-team, 10Analytics: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3570499 (10Nuria) @Andrew : you guys own the script that logs this data (i think it was created by Yuvi a while back) . It doesn't seem li... [16:55:35] 10wikitech.wikimedia.org, 10TimedMediaHandler: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3570645 (10Quiddity) [17:00:39] 10wikitech.wikimedia.org, 10TimedMediaHandler: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3570663 (10Quiddity) >>! In T174476#3569611, @Reedy wrote: > TimedMediaHandler isn't enabled on the wiki... Ha! Ok then. I will [[https://wikitech.wikimedia.org/w/index.php?t... [17:09:16] 10wikitech.wikimedia.org, 10TimedMediaHandler: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3570700 (10Reedy) Question for maybe @brion if we can just enable it, knowing it won't do any transcodes etc? Purely for the viewing/viewer capabilities? [17:55:42] 10cloud-services-team, 10Analytics: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3570865 (10Krenair) modules/toollabs/files/log-command-invocation in puppet [18:19:03] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:33:14] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:51:36] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3571127 (10Cmjohnson) @madhuvishy I reseated the controller card, removed all cabling, and started over again. This time around, the controller c... [18:54:03] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:08:14] RECOVERY - Puppet errors on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:56] 10VPS-project-libraryupgrader: Create list of canary repositories for libraryupgrader - https://phabricator.wikimedia.org/T174646#3571222 (10Legoktm) [19:25:51] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3571225 (10hashar) I have noticed: ``` # Security groups for big projects (e.g. tools) are... [19:27:22] (03PS1) 10Legoktm: Split docker functions into separate file [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375061 [19:27:24] (03PS1) 10Legoktm: Don't hardcode extension assumption in thing.py [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375062 [19:27:26] (03PS1) 10Legoktm: Use a prefetch generator to make getting the extension list faster [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375063 [19:27:28] (03PS1) 10Legoktm: Move functions related to building extension lists into mw.py [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375064 [19:28:55] (03CR) 10Legoktm: [C: 032] Split docker functions into separate file [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375061 (owner: 10Legoktm) [19:29:03] (03CR) 10Legoktm: [C: 032] Don't hardcode extension assumption in thing.py [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375062 (owner: 10Legoktm) [19:29:09] (03CR) 10Legoktm: [C: 032] Use a prefetch generator to make getting the extension list faster [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375063 (owner: 10Legoktm) [19:29:12] (03CR) 10Legoktm: [C: 032] Move functions related to building extension lists into mw.py [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375064 (owner: 10Legoktm) [19:29:14] (03Merged) 10jenkins-bot: Split docker functions into separate file [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375061 (owner: 10Legoktm) [19:29:22] (03Merged) 10jenkins-bot: Don't hardcode extension assumption in thing.py [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375062 (owner: 10Legoktm) [19:29:26] (03Merged) 10jenkins-bot: Use a prefetch generator to make getting the extension list faster [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375063 (owner: 10Legoktm) [19:29:29] (03Merged) 10jenkins-bot: Move functions related to building extension lists into mw.py [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375064 (owner: 10Legoktm) [19:32:04] (03PS1) 10Jean-Frédéric: Skip Wikidata harvesting during daily harvesting [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375065 [19:32:31] (03CR) 10Jean-Frédéric: "> Whenever this is deployed we probably want to make sure the" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [19:33:10] bd808: toolsadmin is a breath of fresh air ! [19:33:34] (03PS2) 10Jean-Frédéric: Skip Wikidata harvesting during daily harvesting [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375065 [19:34:22] (03CR) 10Jean-Frédéric: "> Uploaded patch set 2: Patch Set 1 was rebased." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375065 (owner: 10Jean-Frédéric) [19:37:15] (03CR) 10Jean-Frédéric: [C: 032] Make handling of wikidata urls more compatible with other handlers [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374901 (owner: 10Lokal Profil) [19:42:44] toolsadmin is awesome ++ [19:45:37] thedj: glad you like it :) [19:46:31] there are about 100 more things I want to make it do, but its a good start I think [19:49:11] bd808: this should make it a lot more accessible to beginners I think. [19:49:19] thank you for working on it [19:49:45] :) [19:51:00] (03CR) 10Jean-Frédéric: "> Marked this as [WIP] for three reasons" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [19:51:30] madhuvishy: so labstores seem ready for us to setup hardware raid and then install [19:51:43] robh: yup! [19:51:45] we'll have to pick out a recipe or write one [19:52:00] robh: dumpsdata100x should work [19:52:20] we talked about that, but had the sdd sde volume label issue [19:52:41] oh yes [19:52:42] i recall now [19:52:54] well, now that its owrking, ill setup the hw raid and see what the actual installer sees and does [19:53:17] sorry, ive basically been working on really odd problems for partitioning and firmware for the past 2 weeks, its all bleed together in my brain ;D [19:53:42] let me give the first of the two a shot on imaging and ill let you know what happens shortly =] [19:53:45] robh: no worries :) [19:53:48] okay [19:53:51] thank you! [19:54:20] (03CR) 10Jean-Frédéric: [C: 031] [WIP]Merge Wikidata branch into master [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [20:23:07] !log tools Disabling puppet on tools-proxy-* and tools-static-* for star.wmflabs.org SSL cert update [20:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:24:05] !log tools Disabling puppet on tools-proxy-* and tools-static-* for star.wmflabs.org SSL cert update (T174611) [20:24:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:24:10] T174611: update *.wmflabs.org by 2017-10-16 - https://phabricator.wikimedia.org/T174611 [20:25:10] !log tools Merging new cert https://gerrit.wikimedia.org/r/#/c/374873/ (T174611) [20:25:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:33:28] !log tools Updated certs and ran puppet, restarted nginx on tools-proxy-* and tools-static-* (T174611) [20:33:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:33:33] T174611: update *.wmflabs.org by 2017-10-16 - https://phabricator.wikimedia.org/T174611 [20:37:57] 10wikitech.wikimedia.org, 10TimedMediaHandler: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3571423 (10brion) Offhand I don't know of any reason it couldn't be turned on there. Note if you enable TimedMediaHandler, *and* upload video files locally, they'll be transcoded... [20:39:53] madhuvishy: so the raid is presenting the disks iwth the odd labels [20:39:56] sdd, sde, etc... [20:40:08] do you recall the task paravoid linked that has an example of godog handling something like it [20:40:13] i thought i flagged it, but seems not =[ [20:40:31] robh: no... I lost it too [20:41:33] im now comparing 1006 and 1007 to see if its the same issue between them at least [20:42:08] chris set the raid1 1tb array to be bootable though [20:42:15] so its odd, since i recall that being the issue on what faidon linked [20:43:54] robh: right, yeah may be we give up and fork the recipe :| [20:44:21] also the P840 is the internal raid controller, and its on slot 3 versus the external controller being on slot 1 [20:44:34] i dunno if that can be swapped easily due to cable routing within the chassis though [20:44:50] also not sure if htat would change detection and possiblly labeling [20:48:00] so yeah, its the same on 1006 and 1007, they have the raid1 as sde, the raid10 in chassis as sdf, and the external shelf presenting as sdd.... [20:48:15] madhuvishy: it seems very strange, we can fork the recipe but it bugs me! [20:50:41] ugh yeah that is so weird [20:53:46] ill likely fork if i dont figure out how to relabel properly in the next 30 minutes of messing with it. [21:10:36] robh: okay! (sorry for delayed replies, I'm doing the *.wmflabs cert update) [21:10:51] no worries [21:12:37] !log project-proxy Updated star.wmflabs.org cert, ran puppet and restarted nginx in novaproxy-01 and 02 (Copied the private key manually to /etc/ssl/private) T174611 [21:12:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [21:12:39] T174611: update *.wmflabs.org by 2017-10-16 - https://phabricator.wikimedia.org/T174611 [21:20:10] 10cloud-services-team (Kanban), 10Operations: update *.wmflabs.org by 2017-10-16 - https://phabricator.wikimedia.org/T174611#3571544 (10madhuvishy) 05Open>03Resolved a:03madhuvishy This is all done, new private key committed in ops/private. New certs are showing up okay! [21:26:11] bleh, have to disable the 10gbit card on new labstores, they have both and only stretch can detect the difference and assign eth0 correctly in the installer (to whichever is being used live) [21:26:24] so if/when these move to 10gb, it'll rquire reboot into bios to enable. [21:26:40] these are precise, and precise isnt smart enough to figure it out. [21:26:48] (just fyi) [21:28:12] robh: precise? [21:28:41] i thgouth thats goign on these? [21:28:51] madhuvishy: jessie or precise? [21:28:53] other labstores had nothing in pxe file for trusty [21:29:00] sorry, not precise [21:29:01] jessie [21:29:03] ok [21:29:05] yes [21:29:07] :) [21:29:12] precise is ubuntu, im having a brain fart. [21:29:20] no worries! [21:29:29] but yeah, jessie all ubuntu arent smart about nic ordering [21:29:32] only stretch is [21:30:52] robh: right, okay [21:36:26] hrmm, no root filesystem, it doesnt like the new partman with the /dev/sde and sdf, booting back into isntaller without a recipe defined to see what it sees the disks as. [21:39:58] haha [21:40:03] so it detects as sde sdf in raid controller [21:40:04] damn [21:40:14] but it presents to jessie differently [21:40:24] witht he sdb being the tiny raid1 [21:40:50] that is so confusing! [21:56:07] Ok, so first it presends the dists in the installer as sda, sdb, and sdc, but sda is the disk shelf [21:56:10] while b and c are internal [21:56:21] due to the exernal controler being slot 1 while the internal is slot 3 [21:57:25] also the disk shelf made a raid6 not raid10, so its beinf fixed now by chris onsite. my remote commands via the raid bios dont seem to be accepted to delete virtual disks. [21:57:43] (or if they do it times out and kills my connection at minimum) so labstores still not cooperating 100% ;] [22:01:03] (03CR) 10Lokal Profil: [C: 032] Skip Wikidata harvesting during daily harvesting [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375065 (owner: 10Jean-Frédéric) [22:03:13] robh: okay, thanks for dealing with this! [22:04:23] 10Data-Services, 10cloud-services-team, 10DBA, 10Fundraising-Backlog, and 2 others: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10Ejegg) Yes, everything in the CentralNotice schema can be visible to the public. Most of it is already visible via https://meta.w... [22:11:37] 10Data-Services: `pr_index`to be replicated to Labs public databases - https://phabricator.wikimedia.org/T113842#3571635 (10Reedy) [22:21:18] 10Data-Services, 10DBA: `pr_index`to be replicated to Labs public databases - https://phabricator.wikimedia.org/T113842#3571666 (10Reedy) [22:22:01] 10Data-Services, 10DBA: `pr_index`to be replicated to Labs public databases - https://phabricator.wikimedia.org/T113842#1677380 (10Reedy) Can a DBA check why pr_index isn't being replicated? Thanks! [22:23:46] 10Cloud-Services, 10Cloud-VPS: Create top-level domain for a project on project creation - https://phabricator.wikimedia.org/T162977#3571685 (10Krenair) I think we'll have to do the makedomain-style subdomain creation dance with wmflabsdotorg Actually let's just import that code and call it? [22:24:16] !log rcm Neon: Updating Software [22:24:17] !log rcm Xenon: Updating Software [22:24:17] !log rcm CAC: Running vagrant git-update [22:24:17] !log rcm Tin: Updating Software [22:24:17] !log rcm Neon: Restarting webservice [22:24:18] !log rcm Oxygen: Updating Software [22:24:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [22:24:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [22:24:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [22:24:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [22:24:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [22:24:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [22:24:55] (03PS2) 10Lokal Profil: [WIP]Merge Wikidata branch into master [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) [22:26:42] (03CR) 10Lokal Profil: "> > Marked this as [WIP] for three reasons" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [22:27:15] (03PS3) 10Lokal Profil: Merge Wikidata branch into master [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) [22:28:06] (03CR) 10Lokal Profil: [C: 031] Merge Wikidata branch into master [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [22:34:00] 10Data-Services, 10cloud-services-team, 10DBA, 10Fundraising-Backlog, and 2 others: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10madhuvishy) @Marostegui @Reedy Merged and ran maintain-views in all the labs replicas (1001/3/9/10/11) [22:34:53] 10Cloud-Services, 10Patch-For-Review: Create top-level domain for a project on project creation - https://phabricator.wikimedia.org/T162977#3571734 (10Krenair) a:03Krenair [22:52:45] 10Toolforge: Openstack-browser server pages fail to load information with 'Unknown server' error - https://phabricator.wikimedia.org/T174740#3571811 (10madhuvishy) [22:53:17] 10Toolforge: Openstack-browser server pages fail to load information with 'Unknown server' error - https://phabricator.wikimedia.org/T174740#3571825 (10madhuvishy) Looks like it is talking to the old puppetmaster url and failing ``` [2017-08-31 22:41:19,008] ERROR in app: Error collecting information for server... [23:10:07] where does striker take the toolinfo from? [23:10:24] it does not seem to recognize the existing toolinfo.json file [23:14:07] bd808: ^ [23:14:45] !log openstack-browser Restarted webservice after fixing T174740 [23:14:46] tgr: it is stored locally in Striker. It is not trying to find data randomly on the internet [23:14:46] madhuvishy: Unknown project "openstack-browser" [23:14:46] madhuvishy: Did you mean to say "tools.openstack-browser" instead? [23:14:47] T174740: Openstack-browser server pages fail to load information with 'Unknown server' error - https://phabricator.wikimedia.org/T174740 [23:14:57] !log tools.openstack-browser Restarted webservice after fixing T174740 [23:15:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [23:15:25] 10Toolforge: Openstack-browser server pages fail to load information with 'Unknown server' error - https://phabricator.wikimedia.org/T174740#3571922 (10madhuvishy) 05Open>03Resolved a:03madhuvishy [23:15:37] tgr: I've been thinking about trying to import the existing data from Hay's db into it [23:15:43] to seed things [23:15:53] but I haven't worked on it really yet [23:22:34] 10Cloud-Services, 10Cloud-VPS, 10DBA, 10Tracking: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#3571954 (10madhuvishy) [23:22:38] 10Data-Services, 10cloud-services-team, 10DBA, 10Fundraising-Backlog, and 2 others: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#3571951 (10madhuvishy) 05Open>03Resolved a:03madhuvishy I'm closing this as resolved since running the maintain-views script for the n... [23:53:40] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3572019 (10RobH) Ok, so these detect with the internal raid1 as sdb, the internal raid10 array as sdc, and the external disk array as sda. d-i grub-i...