[02:28:37] (03CR) 10BryanDavis: [V: 032 C: 032] Mark repository as read only [labs/invisible-unicorn] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/387026 (https://phabricator.wikimedia.org/T154099) (owner: 10MarcoAurelio) [07:11:10] !log tools Clear nscd cache across all projects post labsdb dns switchover [07:11:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:11:26] !log tools Clear nscd cache across all projects post labsdb dns switchover T179464 [07:11:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:11:30] T179464: labsdb1001 crashed - storage issue - https://phabricator.wikimedia.org/T179464 [13:10:35] https://tools.wmflabs.org/replag/ is broken :-( [13:29:53] heh, 500 [13:59:22] tgr + BDavis: https://wikitech.wikimedia.org/wiki/User:DatGuy/sandbox [14:23:16] replag being broken is probably from c1.labsdb begin offline. I bet there is no error handling in the code I wrote for that [14:27:36] poor labsdb1001 [14:28:55] * DatGuy starts a labsdb1001 gofundme [14:29:11] * DatGuy contacts The Support Fund for Servers [14:29:36] we have the new servers up and running [14:30:19] * DatGuy creates a labsdb1001 memorial [14:30:58] Use *.{analytics,web}.db.svc.eqiad.wmflabs instead and all should be well -- except if your tool needs to write back the the db. Then you need to change some code to split writes to tools.db.svc.eqiad.wmflabs [14:31:56] bd808: so not enwiki.labsdb? [14:32:49] !log wikilabels staging wikilabels-wmflabs-deploy:fbfee0d [14:32:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [14:33:09] until we do the final switch to the new servers the *.labsdb service names will point to the dying servers. More info at https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown [14:33:49] !log wikilabels deploying wikilabels-wmflabs-deploy:fbfee0d [14:33:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [14:33:59] alright, and then I'm guessing it's changed to *.web.blablabla ? [14:34:41] web for queries that need quick responses, analytics for things that can run slower [14:41:30] !log wikilabels staging wikilabels-wmflabs-deploy:d5fafbe [14:41:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [14:42:47] rip labsdb1001 [14:43:14] Dispenser: https://tools.wmflabs.org/replag/ works again. I made it show PHP_MAX_INT seconds of replag for servers that are offline [14:43:45] !log wikilabels ran patches/patch-campaign-add-info-column.sql on wikilabels-staging-01 [14:43:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [14:45:19] We can try to force depool the one disk that we can tell is failing and see if MySQL on labsdb1001 will start up. If that doesn't work it's done though. :/ [14:47:26] !log wikilabels ran patches/patch-campaign-add-info-column.sql on wikilabels-01 [14:47:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [14:47:36] !log wikilabels deploying wikilabels-wmflabs-deploy:d5fafbe [14:47:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [15:01:56] (03PS187) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [15:02:18] !log tools.replag Updated to fbf405a (Add really basic error handling) [15:02:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.replag/SAL [15:03:23] !log tools.tool-db-usage Updated to d521816 (Add basic error handling for MySQL failures) [15:03:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tool-db-usage/SAL [15:11:51] (03CR) 10Ricordisamoa: [C: 04-2] "PS187 changes some strings into template strings" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [15:59:12] Technical Advice IRC meeting starting now in channel #wikimedia-tech, hosts: @addshore & @Christoph_Jauera_(WMDE) - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [16:22:20] (03CR) 10Hashar: "check experimental" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/387171 (owner: 10L10n-bot) [16:23:46] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/387171 (owner: 10L10n-bot) [16:35:22] (03CR) 10Hashar: "check experimental" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/387171 (owner: 10L10n-bot) [16:36:01] (03CR) 10Hashar: "check experimental" [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/386768 (owner: 10L10n-bot) [16:37:04] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/386768 (owner: 10L10n-bot) [16:37:41] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/387171 (owner: 10L10n-bot) [18:07:48] hi, any idea how/why some tool DB was vanished from host enwiki.labsdb ? [18:24:10] eranroz: probably https://lists.wikimedia.org/pipermail/cloud-announce/2017-November/000007.html [18:27:02] legoktm: 10x [20:57:35] (03PS1) 10Ayounsi: Add fake keys for Netbox deployment [labs/private] - 10https://gerrit.wikimedia.org/r/387878