[00:15:48] (03PS1) 10Gerrit Patch Uploader: Changes to patrol settings for shwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112306 [00:15:59] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112306 (owner: 10Gerrit Patch Uploader) [00:18:39] (03CR) 10John F. Lewis: [C: 04-1] "Change introduces a new file which is not supposed to be present." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112306 (owner: 10Gerrit Patch Uploader) [00:23:13] (03CR) 10PiRSquared17: [C: 04-1] "Didn't mean to add that file. This was the diff http://tools.wmflabs.org/pirsquared/diff.txt" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112306 (owner: 10Gerrit Patch Uploader) [00:25:15] (03PS2) 10Bartosz Dziewoński: Changes to patrol settings for shwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112306 (owner: 10Gerrit Patch Uploader) [00:30:08] (03CR) 10John F. Lewis: [C: 031] "Look fine now" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112306 (owner: 10Gerrit Patch Uploader) [00:51:03] (03CR) 10PiRSquared17: [C: 031] "Fine now. Thank you for fixing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112306 (owner: 10Gerrit Patch Uploader) [01:51:46] (03PS1) 10Ori.livneh: reprepro: import HHVM from Facebook's repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/112314 [01:58:59] (03PS1) 10Ryan Lane: Simplify trebuchet developer environment creation [operations/puppet] - 10https://gerrit.wikimedia.org/r/112315 [02:08:41] (03PS2) 10Ryan Lane: Deployment module changes for trebuchet-trigger [operations/puppet] - 10https://gerrit.wikimedia.org/r/110239 [02:09:58] !log LocalisationUpdate completed (1.23wmf12) at 2014-02-09 02:09:57+00:00 [02:10:11] Logged the message, Master [02:17:57] !log LocalisationUpdate completed (1.23wmf13) at 2014-02-09 02:17:56+00:00 [02:18:03] Logged the message, Master [02:34:16] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-02-09 02:34:15+00:00 [02:34:23] Logged the message, Master [02:39:57] (03PS1) 10Ryan Lane: Remove old parsoid deploy repos and remove parsoid salt module [operations/puppet] - 10https://gerrit.wikimedia.org/r/112316 [03:01:51] (03PS1) 10Ryan Lane: Continue on single repo failures for deployment server init [operations/puppet] - 10https://gerrit.wikimedia.org/r/112317 [03:06:42] (03CR) 10Zhuyifei1999: "Adding reedy to reviewer" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110876 (owner: 10Ebe123) [03:13:50] (03CR) 10Ebe123: "Added Reedy as reviewer" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110822 (owner: 10Ebe123) [03:14:13] (03CR) 10Ori.livneh: "Looks good, but I wonder if we could do a bit more to ensure this doesn't bite again in the future. A simple change would be to make each " [operations/puppet] - 10https://gerrit.wikimedia.org/r/112316 (owner: 10Ryan Lane) [03:14:42] ori: so, for that specific change... [03:14:53] we need to change how we handle modules/returners/etc anyway [03:14:59] to properly handle multi-master salt [03:15:11] the salt master needs to have the same state when the sync occurs [03:15:28] we always run a salt master with a puppet master [03:15:38] hmmmmm, it's raining a fair bit and the lights have started flickering [03:15:43] that's not a good sign [03:15:57] so, the salt master can actually just change its modules/returners/etc to use the puppet git repo directly [03:16:05] hm. need to disconnect? [03:16:25] maybe it's just the bulb [03:16:28] nah, ok for now [03:16:30] * Ryan_Lane nods [03:17:01] so, we sync the puppet repo during a puppet-merge, which would make the salt masters fully sync'd before a puppet run [03:17:20] it also means we don't need to list all the resources [03:18:02] similarly, we should move the deployment configuration hash into a yaml file that salt reads directly, rather than generating it [03:18:15] which is a good idea for the future anyway (hiera) [03:18:57] OK, that's fine then. Is there some way I'm not thinking of that this change would have some effect on Parsoid? (Seems implausible since it's using parsoid/deploy, but...) [03:19:10] nope [03:19:22] in fact, it won't even touch the old repos [03:19:51] (03CR) 10Ori.livneh: [C: 032] "Per Ryan, a more sophisticated approach would be to have the salt master be aware of changes to the repository." [operations/puppet] - 10https://gerrit.wikimedia.org/r/112316 (owner: 10Ryan Lane) [03:20:17] ugh. I just did q! rather than wq! in vim [03:20:24] well, there goes that change [03:20:29] vim -r filename? [03:21:01] that works after a q!? [03:21:19] heh. no swap file [03:21:30] oh well, it wasn't a huge change [03:23:09] should i force a puppet run on the salt master? [03:23:34] to ensure puppet runs properly, yeah [03:24:00] it'll trigger a pillar refresh everywhere [03:24:22] which will also trigger a check on the deployment server for any repos that need to be added [03:24:42] (03CR) 10Alex Monk: [C: 04-1] Revoking my access [operations/puppet] - 10https://gerrit.wikimedia.org/r/111960 (owner: 10Ryan Lane) [03:27:02] (03CR) 10Peachey88: "@Ebe123, You don't need to put a comment when you add a review (they get a email anyway)." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110822 (owner: 10Ebe123) [03:27:27] not a popular changeset, that [03:27:50] heh [03:32:08] (03PS1) 10Ryan Lane: Ensure submodules are checked out on the deployment server [operations/puppet] - 10https://gerrit.wikimedia.org/r/112319 [03:34:07] ah. this change actually depends on another. [03:34:08] * Ryan_Lane rebases [03:36:09] -_- stupid gerrit. [03:36:49] meh, I'll just put the dependency info in the comments. gerrit won't let me rebase it easily [03:37:15] (03CR) 10Ryan Lane: "Depends on https://gerrit.wikimedia.org/r/#/c/112317" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112319 (owner: 10Ryan Lane) [03:51:08] (03PS2) 10Ryan Lane: Ensure submodules are checked out on the deployment server [operations/puppet] - 10https://gerrit.wikimedia.org/r/112319 [04:22:01] (03PS3) 10Ryan Lane: Deployment module changes for trebuchet-trigger [operations/puppet] - 10https://gerrit.wikimedia.org/r/110239 [04:22:03] (03PS2) 10Ryan Lane: Simplify trebuchet developer environment creation [operations/puppet] - 10https://gerrit.wikimedia.org/r/112315 [04:28:29] (03Abandoned) 10Ryan Lane: Revoking my access [operations/puppet] - 10https://gerrit.wikimedia.org/r/111960 (owner: 10Ryan Lane) [09:42:09] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (205549) [09:51:09] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [11:13:09] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (202868) [11:32:51] orly [11:39:09] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [11:43:09] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (201539) [11:46:09] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [12:59:19] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:09] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 127450 bytes in 7.288 second response time [13:27:11] (03PS1) 10Odder: Let admins add users to 'accountcreator' on itwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112337 [13:34:37] (03PS1) 10Odder: Remove redundant GroupOverride entries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112339 [13:40:19] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:09] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 127807 bytes in 6.957 second response time [15:18:58] Some op knowledgable with Labs around? [15:19:54] Any idea what's wrong with Labs? [15:36:01] anyone contacted coren? [15:36:13] I emailed him and posted on his enwiki talk page [15:36:38] incognitus: someone here should have is phone number [15:36:48] * incognitus doesn't [15:36:56] it isn't so critical [15:37:05] it is sunday, let the guy rest :) [15:37:19] matanya: complete outage is critical [15:37:29] Betacommand: only tools [15:37:40] not the site [15:37:56] matanya: http on tools seems to be down for me [15:38:24] yes, tools is down, for what i see. site = wikis [15:38:44] the bastion is working [15:38:53] incognitus: ssh, http https, ssh, and sftp are all down [15:39:09] and ping too [15:39:57] matanya: I'm getting 0 packet loss [15:40:41] yeah, depends from where [15:40:53] ok, i'm out. good luck with this [15:41:01] bye matanya [16:58:29] PROBLEM - Host labstore4 is DOWN: PING CRITICAL - Packet loss = 100% [16:58:49] RECOVERY - Host labstore4 is UP: PING OK - Packet loss = 0%, RTA = 35.37 ms [17:12:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 132.46666 [17:13:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:16:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 144.933334 [17:17:40] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 21.066668 [17:18:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 743.533325 [17:18:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:19:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:21:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 329.799988 [17:22:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:25:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 128.199997 [17:26:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 727.299988 [17:28:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 660.733337 [17:36:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:39:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:40:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 452.866669 [17:42:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 784.06665 [17:44:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:45:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:49:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:49:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 649.266663 [17:52:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 279.066681 [17:55:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 73.699997 [17:56:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:00:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:03:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 330.066681 [18:05:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:06:29] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [18:07:09] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 36.09 ms [18:08:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:11:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 246.266663 [18:12:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 310.833344 [18:14:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 285.399994 [18:15:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:16:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:18:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 488.933319 [18:21:49] PROBLEM - NTP on mw31 is CRITICAL: NTP CRITICAL: Offset unknown [18:23:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:24:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 213.46666 [18:25:49] RECOVERY - NTP on mw31 is OK: NTP OK: Offset 0.002154350281 secs [18:26:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 372.399994 [18:26:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:27:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:27:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:27:56] so, do we care about those? [18:29:06] we care about mw31 [18:29:54] the varnishkafka delivery errors represent errors on the bits varnishes sending the request log to the analytics cluster. the setup is new and nothing depends on it yet. so we care, but it's not an alarm. [18:30:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 328.833344 [18:30:47] mw31 is part of a cluster of hosts with a fair bit of redundancy, so that's not a crisis either, but still a bit disconcerting [18:31:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:32:28] ori: thanks man [18:32:49] :) [18:33:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:34:08] (also, I didn't see the mw31 one, was drowned out by the kafka) [18:34:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 426.399994 [18:34:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 169.199997 [18:37:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 220.333328 [18:41:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 88.73333 [18:42:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:42:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:43:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:44:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:44:49] (03PS6) 10JanZerebecki: turn planet into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 (owner: 10Dzahn) [18:50:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 328.266663 [18:51:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:54:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 143.866669 [18:55:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 117.599998 [18:55:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [18:56:33] (03PS7) 10JanZerebecki: turn planet into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 (owner: 10Dzahn) [18:57:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:00:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 363.333344 [19:01:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 280.933319 [19:06:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:08:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:08:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 220.96666 [19:12:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 61.066666 [19:12:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 292.399994 [19:12:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 248.566666 [19:17:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:20:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 672.400024 [19:20:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:24:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:26:20] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:28:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:31:11] (03CR) 10JanZerebecki: "See the patch 3 inline comments for the errors I fixed. (Puppet is so bloated it gets oom killed when it does an infinite recursion ;) .) " [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 (owner: 10Dzahn) [19:31:43] (03CR) 10JanZerebecki: turn planet into a module (036 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 (owner: 10Dzahn) [19:36:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 68.0 [19:37:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 122.633331 [19:39:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:41:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:44:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 299.399994 [19:44:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 468.133331 [19:45:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 450.266663 [19:45:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 310.733337 [19:47:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:47:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:48:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:51:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 99.466667 [19:53:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:53:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:56:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 68.866669 [20:00:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 175.199997 [20:00:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 380.066681 [20:00:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 111.73333 [20:04:10] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:04:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:04:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:07:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 257.466675 [20:07:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 324.266663 [20:08:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 594.799988 [20:12:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:13:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:13:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:17:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 209.03334 [20:17:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:20:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 102.73333 [20:22:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:22:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 25.233334 [20:22:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:23:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:29:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 232.199997 [20:29:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 448.233337 [20:29:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 139.0 [20:29:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 108.066666 [20:31:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:33:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:34:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:37:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:40:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 482.600006 [20:41:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 332.533325 [20:41:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 456.833344 [20:44:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 51.566666 [20:45:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:48:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 584.233337 [20:53:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:56:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 272.166656 [20:58:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:01:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 164.53334 [21:13:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:16:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.266666 [21:19:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:20:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:20:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:22:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 741.333313 [21:23:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 415.066681 [21:28:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 69.466667 [21:31:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:35:39] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 17.0 [21:37:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:41:59] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 297.600006 [21:44:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:45:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:47:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 493.733337 [21:47:39] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:48:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 846.200012 [21:50:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [21:53:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 47.466667 [21:59:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [22:04:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 28.4 [22:10:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [22:13:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [22:17:09] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.4 [22:18:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 12.533334 [22:20:20] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [22:23:19] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 17.866667 [22:25:09] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [22:25:19] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [22:28:59] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0