[00:00:08] I'm starting to collect all the bits needed to set up a betalabs node in https://wikitech.wikimedia.org/wiki/User:GWicke/betalabs_node_setup [00:00:55] now I'm getting "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass trusted_group to Class[Keyholder] on node i-0000010b.eqiad.wmflabs" [00:01:44] although nm, that was on the wrong node [00:02:09] (03PS1) 10Ori.livneh: Auto-link EventLogging SCIDs in Gerrit [puppet] - 10https://gerrit.wikimedia.org/r/175156 [00:03:13] (03CR) 10Ori.livneh: [C: 032] Auto-link EventLogging SCIDs in Gerrit [puppet] - 10https://gerrit.wikimedia.org/r/175156 (owner: 10Ori.livneh) [00:04:03] still not working though [00:04:15] what's the error? [00:04:29] no error, but still no submodule checkout on the nodes [00:04:47] can you just ignore trebuchet and use git? [00:05:44] I was hoping that I could make our deployment system work at least in betalabs [00:05:58] but yeah, maybe that's a bit too ambitious [00:06:13] sadly, yes [00:08:18] the good news is that things seem to finally be working after sidestepping trebuchet [00:08:28] on one node [00:08:36] now to repeat on the others / a new node.. [00:09:01] (03PS1) 10Ori.livneh: ensure ::keyholder is applied before ::keyholder::monitoring [puppet] - 10https://gerrit.wikimedia.org/r/175157 [00:10:15] gwicke: trebuchet in its entirety is not a lot of code. when i run into breakage, i try to improve things a little, and i'd love to see you to do the same. moaning about how much it sucks is ineffective, sadly. [00:11:35] (03CR) 10Ori.livneh: [C: 032] ensure ::keyholder is applied before ::keyholder::monitoring [puppet] - 10https://gerrit.wikimedia.org/r/175157 (owner: 10Ori.livneh) [00:12:11] ori: thanks for the advice [00:12:16] That cehckout-submodules stanza should come from $role::deployment::config::repo_config [00:12:55] bd808: how does that work with the package provider? [00:12:58] gwicke: fwiw, I'm not criticizing you -- I've done my share of moaning about it myself. And I have the same opinion as you about the overall quality of the implementation. [00:13:30] gwicke: AFAIK the package provider still needs all that config to setup the deploy master [00:14:07] The package bit just simplifies the target host parts [00:18:03] (03CR) 10Dzahn: [C: 04-1] "arg, no, it would't. it would fail to find the SSL cert" [puppet] - 10https://gerrit.wikimedia.org/r/175144 (owner: 10Dzahn) [00:18:50] (03PS1) 10Ori.livneh: Force Class['::keyholder'] to apply before Keyholder::Private_key because WTF labs [puppet] - 10https://gerrit.wikimedia.org/r/175158 [00:20:04] bd808: is labs using a different version of puppet or ruby or something? [00:20:14] why on earth would that error occur on lab but not production? [00:20:26] "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass trusted_group to Class[Keyholder] on node i-0000010b.eqiad.wmflabs" [00:21:10] deplloyment-bastion should basically match tin [00:21:15] gerrit 503ing for anyone else? [00:21:24] yup, me too [00:21:32] working now... [00:21:42] yup [00:21:57] (03PS1) 10GWicke: Enable submodule checkout for restbase [puppet] - 10https://gerrit.wikimedia.org/r/175159 [00:22:35] so it looks like this was my fault [00:23:33] bd808: ^^ [00:24:00] gwicke: *nod* easy to miss [00:24:34] I'd like to see that hash moved into hiera and broken out into per module files [00:24:36] PROBLEM - CI: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: integration.integration-slave1004.puppetagent.failed_events.value (25.00%) [00:24:55] That would make it easier to review I think [00:28:44] (03PS4) 10Dzahn: bugzilla: switch svc_name to old-bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/175144 [00:35:08] (03CR) 10GWicke: [C: 031] Enable submodule checkout for restbase [puppet] - 10https://gerrit.wikimedia.org/r/175159 (owner: 10GWicke) [00:35:25] bd808: would you mind merging ^^ ? [00:35:39] gwicke: I'm not a root :) [00:35:43] oh, okay [00:35:51] sorry for bugging you then ;) [00:35:55] ori: ^^ [00:36:01] No worries [00:36:12] <^demon|away> I can escalate my privs in gerrit and merge it, but that might upset ops :) [00:36:18] <^demon|away> (also I couldn't merge on puppetmaster) [00:38:08] (03PS2) 10Ori.livneh: Enable submodule checkout for restbase [puppet] - 10https://gerrit.wikimedia.org/r/175159 (owner: 10GWicke) [00:38:16] (03CR) 10Ori.livneh: [C: 032 V: 032] Enable submodule checkout for restbase [puppet] - 10https://gerrit.wikimedia.org/r/175159 (owner: 10GWicke) [00:40:02] ori: thanks! [00:40:07] np [00:40:12] (03PS2) 10Ori.livneh: Force Class['::keyholder'] to apply before Keyholder::Private_key because WTF labs [puppet] - 10https://gerrit.wikimedia.org/r/175158 [00:41:33] (03PS3) 10Ori.livneh: Force Class['::keyholder'] to apply before Keyholder::Private_key because WTF labs [puppet] - 10https://gerrit.wikimedia.org/r/175158 [00:41:48] (03CR) 10Ori.livneh: [C: 032 V: 032] Force Class['::keyholder'] to apply before Keyholder::Private_key because WTF labs [puppet] - 10https://gerrit.wikimedia.org/r/175158 (owner: 10Ori.livneh) [00:43:28] (03PS1) 10Dzahn: bugzilla: install old-bugzilla SSL cert [puppet] - 10https://gerrit.wikimedia.org/r/175162 [00:44:55] !log Disabled login for dewiki accounts "W" and "H" [00:44:59] Logged the message, Master [00:45:52] (03PS3) 10Ori.livneh: Move *.dblist to dblists/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175007 [00:46:09] (03CR) 10jenkins-bot: [V: 04-1] Move *.dblist to dblists/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175007 (owner: 10Ori.livneh) [00:47:12] <^demon|away> ori: There's probably a ton of jobs and ton of puppet stuff that depend on those dblists being where they are at the moment. [00:47:25] <^demon|away> (not that I disagree about moving them out of the root :)) [00:47:26] And MZMcBride. [00:47:27] ^demon|away: yeah, i'm not merging that anytime soon [00:47:41] ^demon|away: i was just experimenting with EventLogging SCID autolinking in the commit message [00:48:04] * ^demon|away nods [00:48:10] <^demon|away> Carmela: Who's that? [00:48:31] A tireless crank, that's who. [00:48:44] ^demon|away: it's like the Dread Pirate Roberts -- a moniker adopted by various people across time [00:48:53] or Zorro, for that matter [00:49:12] * ^demon|away imagines MZMcBride in a zorro mask, giggles [00:49:16] RECOVERY - CI: Puppet failure events on labmon1001 is OK: OK: All targets OK [00:51:27] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 316 seconds [00:51:46] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 338 seconds [00:52:45] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds [00:53:05] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:58:55] PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [90.0] [00:59:15] hoo, did these accounts do something terribly bad to you? :P [00:59:24] MaxSem: Not to me [00:59:36] If you're on security@ you'll get a heads up in a second [01:06:06] RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [60.0] [01:39:26] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: puppet fail [01:59:56] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:20:33] !log l10nupdate Synchronized php-1.25wmf8/cache/l10n: (no message) (duration: 00m 01s) [02:20:36] !log LocalisationUpdate completed (1.25wmf8) at 2014-11-22 02:20:36+00:00 [02:20:40] Logged the message, Master [02:20:42] Logged the message, Master [02:29:41] (03PS13) 1020after4: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 [02:33:32] !log l10nupdate Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) [02:33:36] Logged the message, Master [02:33:36] !log LocalisationUpdate completed (1.25wmf9) at 2014-11-22 02:33:36+00:00 [02:33:40] Logged the message, Master [04:30:54] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 22 04:30:53 UTC 2014 (duration 30m 51s) [04:30:58] Logged the message, Master [05:10:18] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: puppet fail [05:29:57] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [05:43:20] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [06:02:48] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:28:08] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: puppet fail [06:28:38] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 3 failures [06:29:18] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:47] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:58] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:58] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:37:48] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:39:58] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 3 failures [06:45:42] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [06:46:08] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:46:17] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:46:58] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:47:38] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:51:43] (03PS2) 10TTO: Clean up indents, comments, spacing in InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) [06:54:58] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:57:22] (03CR) 10TTO: "@Reedy: I know you love these cleanup patches (after all you did file bug 29902). I'd appreciate if you could review this before it rots a" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) (owner: 10TTO) [06:57:48] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:12:38] PROBLEM - SSH on lvs4001 is CRITICAL: Server answer: [07:13:47] RECOVERY - SSH on lvs4001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [09:09:58] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: puppet fail [09:28:18] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [12:37:21] (03CR) 10MZMcBride: Clean up indents, comments, spacing in InitialiseSettings (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) (owner: 10TTO) [12:39:50] (03CR) 10MZMcBride: "ProTip: when viewing a diff in Gerrit, click "Preferences" near the top of the page, "Ignore Whitespace: All" from the drop-down menu, cli" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) (owner: 10TTO) [13:29:38] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: puppet fail [13:44:58] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [500.0] [13:48:18] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [14:01:38] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [16:44:47] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 1 failures [17:07:28] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:10:28] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: puppet fail [17:24:08] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 1 failures [17:32:08] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:07:38] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:42:13] (03CR) 10Rush: [C: 032 V: 032] add old-bugzilla.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/175133 (owner: 10Dzahn) [18:44:47] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 1 failures [18:59:34] This server could not prove that it is old-bugzilla.wikimedia.org; its security certificate is from bugzilla.wikimedia.org [19:00:24] Krenair: old-bugzilla currently does not have an SSL cert [19:01:12] why the change over has taken place despite this missing it rather, annoying. andre__? [19:03:52] chasemp actually ^^ [19:04:19] yes we know, it's in teh works but will have to be crappy for a bit [19:04:23] (not my call just relaying) [19:04:23] we know. [19:04:30] working on it. [19:06:38] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:11:20] Krenair: stole my words [19:30:59] quick heads up, getting an "Due to high database server lag, changes newer than 51 seconds may not appear in this list." error on en.wp right now. [19:33:03] looks good again(?) [19:35:27] yeah [19:38:19] (03CR) 10Ori.livneh: [C: 032] Clean up indents, comments, spacing in InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) (owner: 10TTO) [19:38:31] (03Merged) 10jenkins-bot: Clean up indents, comments, spacing in InitialiseSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) (owner: 10TTO) [19:39:53] !log ori Synchronized wmf-config/InitialiseSettings.php: Ifae6e0ab6: Clean up indents, comments, spacing in InitialiseSettings (duration: 00m 05s) [19:39:59] Logged the message, Master [19:40:15] (03CR) 10Ori.livneh: "Thanks for this, TTO." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) (owner: 10TTO) [19:48:06] He also changed a // bug comment!! liar!!111!!! [19:48:33] * ori reverts [20:03:57] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:33] oh noes, gerrit is dead :( [20:17:46] Guice provision errors: 1) Cannot open ReviewDb ..... [20:17:49] OTRS is dead as well, "Lost connection to MySQL server at 'reading initial communication packet', syste[..] [20:17:49] " [20:17:58] :( [20:18:11] ori: can you fix gerrit? or someone? [20:18:22] Backend ERROR: OTRS-CGI-10 Perl: 5.14.2 OS: linux Time: Sat Nov 22 20:17:08 2014 Message: Lost connection to MySQL server at 'reading initial communication packet', system error: 0 RemoteAddress: 194.230.155.107 RequestURI: /otrs/index.pl?Action=AgentZoom&TicketID=7973910 [20:18:39] * aude time to eat, can't work [20:18:48] i'll look, but we sorta need opsen for that [20:18:53] yeah [20:19:15] springle: around? [20:19:25] ori: yes saw it [20:19:36] springle: cool, thanks. let me know if i can help. [20:24:28] PROBLEM - Check status of defined EventLogging jobs on vanadium is CRITICAL: CRITICAL: Stopped EventLogging jobs: consumer/mysql-m2-master [20:27:53] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:28:02] RECOVERY - haproxy failover on dbproxy1002 is OK: OK check_failover servers up 0 down 0 [20:29:00] RECOVERY - Check status of defined EventLogging jobs on vanadium is OK: OK: All defined EventLogging jobs are runnning. [20:29:05] !log db1046 m2-master threadpool lockup, restarted mysqld, investigating [20:29:09] Logged the message, Master [20:30:09] was back for a moment, but just got a new error: "Error Message: The MariaDB server is running with the --read-only option so it cannot execute tthis statement: [...]" on ticket.wikimedia.org. [20:31:10] PROBLEM - haproxy failover on dbproxy1002 is CRITICAL: CRITICAL check_failover servers up 1 down 1 [20:41:36] springle: it's not related to the EventLogging batching writes change, is it? [20:44:56] ori: i really don't know yet. have gdb backtrace to review, which may help with an upstream bug search, but little else useful yet [20:45:21] nod [21:02:54] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [21:03:10] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [21:03:49] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: puppet fail [21:04:10] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 1 failures [21:16:19] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [21:19:29] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [21:20:00] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [21:20:52] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [21:21:00] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [21:21:29] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 1 failures [21:21:52] !log Jenkins: disconnected/reconnected gallium slave. All executors were being busy / deadlocked [21:21:55] Logged the message, Master [21:22:30] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [21:23:01] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:24:30] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 1 failures [21:34:01] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [21:37:30] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [21:38:00] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [21:38:40] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [21:47:20] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:48:12] (03PS14) 1020after4: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 [21:54:34] (03PS15) 10Rush: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 (owner: 1020after4) [21:58:52] (03PS16) 1020after4: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 [22:00:45] (03PS17) 10Rush: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 (owner: 1020after4) [22:01:48] (03CR) 10Rush: [C: 032] Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 (owner: 1020after4) [22:06:11] (03PS1) 10Rush: phab handle cherry-pick for T1343 [puppet] - 10https://gerrit.wikimedia.org/r/175297 [22:06:29] (03PS2) 10Rush: phab handle cherry-pick for T1343 [puppet] - 10https://gerrit.wikimedia.org/r/175297 [22:07:58] (03CR) 10Rush: [C: 032] phab handle cherry-pick for T1343 [puppet] - 10https://gerrit.wikimedia.org/r/175297 (owner: 10Rush) [22:09:52] RECOVERY - mysqld processes on db1020 is OK: PROCS OK: 1 process with command name mysqld [22:30:47] (03Abandoned) 10Rush: phab during migration raise upload limit [puppet] - 10https://gerrit.wikimedia.org/r/174343 (owner: 10Rush) [22:32:51] (03PS1) 1020after4: * Correction for maniphest vs manifest in variable name * add a catch all redirect for bugzilla.wikimedia.org/ [puppet] - 10https://gerrit.wikimedia.org/r/175298 [22:34:21] (03CR) 10Rush: [C: 032] * Correction for maniphest vs manifest in variable name * add a catch all redirect for bugzilla.wikimedia.org/ [puppet] - 10https://gerrit.wikimedia.org/r/175298 (owner: 1020after4) [22:34:41] (03CR) 10Dzahn: kill facilities.pp, move to nagios_common (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/173999 (owner: 10Dzahn) [22:36:31] (03CR) 10Dzahn: [C: 031] Phab: Change user visible strings "Execute Query" and "Real Name" [puppet] - 10https://gerrit.wikimedia.org/r/174583 (owner: 10Aklapper) [22:44:00] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 1 failures [22:44:45] (03PS1) 10Springle: switch MariaDB ::misc (only used by m2; still ::coredb on m1) to /srv [puppet] - 10https://gerrit.wikimedia.org/r/175301 [22:46:16] (03CR) 10Springle: [C: 032] switch MariaDB ::misc (only used by m2; still ::coredb on m1) to /srv [puppet] - 10https://gerrit.wikimedia.org/r/175301 (owner: 10Springle) [22:51:32] (03PS1) 1020after4: * don't use mysqlnd * show an error if the cross-reference for bug# is not found in database [puppet] - 10https://gerrit.wikimedia.org/r/175302 [22:52:55] (03CR) 10Rush: [C: 032] * don't use mysqlnd * show an error if the cross-reference for bug# is not found in database [puppet] - 10https://gerrit.wikimedia.org/r/175302 (owner: 1020after4) [23:02:07] !log upgrade db1020 trusty, xtrabackup clone db1046 to db1020 [23:02:09] Logged the message, Master [23:12:45] (03PS1) 10Rush: phab bz user metadata update crons [puppet] - 10https://gerrit.wikimedia.org/r/175307 [23:21:11] (03PS1) 10Dzahn: bugzilla: disable cron jobs [puppet] - 10https://gerrit.wikimedia.org/r/175308 [23:23:47] (03CR) 10Aklapper: "As far as I can cluelessly tell, this looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/175308 (owner: 10Dzahn) [23:47:10] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:54:02] (03PS2) 10Rush: phab bz user metadata update crons [puppet] - 10https://gerrit.wikimedia.org/r/175307 [23:54:50] (03CR) 10Dzahn: [C: 032] "as requested by the bz2phab migration team" [dns] - 10https://gerrit.wikimedia.org/r/172469 (owner: 10Dzahn) [23:55:22] (03CR) 10Rush: [C: 032] phab bz user metadata update crons [puppet] - 10https://gerrit.wikimedia.org/r/175307 (owner: 10Rush) [23:57:24] (03PS5) 10Rush: bugzilla: switch svc_name to old-bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/175144 (owner: 10Dzahn) [23:57:29] (03CR) 10Rush: [C: 031] bugzilla: switch svc_name to old-bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/175144 (owner: 10Dzahn)