[00:12:44] (03PS1) 10Ori.livneh: Change metric prefix from 'mw' back to 'MediaWiki', for back-compat [mediawiki-config] - 10https://gerrit.wikimedia.org/r/193524 [00:15:08] (03CR) 10Ori.livneh: [C: 032] Change metric prefix from 'mw' back to 'MediaWiki', for back-compat [mediawiki-config] - 10https://gerrit.wikimedia.org/r/193524 (owner: 10Ori.livneh) [00:15:13] (03Merged) 10jenkins-bot: Change metric prefix from 'mw' back to 'MediaWiki', for back-compat [mediawiki-config] - 10https://gerrit.wikimedia.org/r/193524 (owner: 10Ori.livneh) [00:16:29] !log ori Synchronized wmf-config/CommonSettings.php: If2704b0f7: Change metric prefix from 'mw' back to 'MediaWiki', for back-compat (duration: 00m 06s) [00:16:34] Logged the message, Master [00:40:01] (03CR) 10BBlack: [C: 031] 10.in-addr.arpa - indentation fixes [dns] - 10https://gerrit.wikimedia.org/r/193160 (owner: 10Dzahn) [00:59:22] Hey robh, what are eqord and eqdfw? [01:01:21] ah the linked tickets show Equnix in Dallas and Chicago. interesting [01:01:32] yep, peering exchanges in those places [01:01:38] site naming is vendor+airport [01:01:41] equinix [01:02:12] 6operations, 10ops-codfw: prepare equipment list for eqdfw - https://phabricator.wikimedia.org/T91077#1075151 (10Krenair) [01:02:20] 6operations, 10ops-codfw: prepare equipment list for eqord - https://phabricator.wikimedia.org/T91079#1075152 (10Krenair) [01:04:33] 6operations, 10ops-codfw: prepare equipment list for eqdfw - https://phabricator.wikimedia.org/T91077#1075154 (10RobH) [01:04:42] i had called them the wrong thing, i went with closest airport [01:04:46] but we now go with major airport [01:05:43] 6operations, 10ops-codfw: prepare equipment list for eqord - https://phabricator.wikimedia.org/T91079#1075161 (10RobH) [01:10:00] 6operations: order onsite tools for eqdfw/eqord - https://phabricator.wikimedia.org/T91095#1075172 (10RobH) [01:10:01] 6operations, 10ops-codfw: prepare equipment list for eqdfw - https://phabricator.wikimedia.org/T91077#1075171 (10RobH) [01:10:10] 6operations, 10ops-codfw: prepare equipment list for eqord - https://phabricator.wikimedia.org/T91079#1075175 (10RobH) [01:10:11] 6operations: order onsite tools for eqdfw/eqord - https://phabricator.wikimedia.org/T91095#1074192 (10RobH) [01:10:17] (03PS2) 10BBlack: Add $template_name param to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193288 [01:10:19] (03PS1) 10BBlack: cache late_command: update firmware before kernel [puppet] - 10https://gerrit.wikimedia.org/r/193541 [01:10:21] (03PS1) 10BBlack: convert txstatsd to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193542 [01:10:23] (03PS1) 10BBlack: convert varnish to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193543 [01:11:07] (03CR) 10BBlack: [C: 032 V: 032] cache late_command: update firmware before kernel [puppet] - 10https://gerrit.wikimedia.org/r/193541 (owner: 10BBlack) [01:15:28] (03PS2) 10BBlack: convert varnish to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193543 [01:18:34] (03PS3) 10BBlack: convert varnish to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193543 [01:18:36] (03PS3) 10BBlack: Add $template_name param to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193288 [01:18:37] 6operations: Give Google webmaster tools access to jon katz (Read only is fine) - https://phabricator.wikimedia.org/T90980#1075177 (10Jalexander) >>! In T90980#1074697, @Dzahn wrote: > > Yeah, which is why in the past I think that for anyone who actually needed them all we gave the password (at t... [01:18:38] (03PS2) 10BBlack: convert txstatsd to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193542 [01:20:09] 6operations: Give Google webmaster tools access to jon katz (Read only is fine) - https://phabricator.wikimedia.org/T90980#1075180 (10Jalexander) I'm not against giving Jon that access (or or that matter having one or multiple of us sit there and share a bunch of the wikipedias) I just think we need to ensure th... [01:21:20] 6operations: Give Google webmaster tools access to jon katz (Read only is fine) - https://phabricator.wikimedia.org/T90980#1075183 (10Krenair) >>! In T90980#1075177, @Jalexander wrote: > at this point we have more then 1000 sites listed, so many in fact that yesterday we got an email saying they changed their po... [01:21:31] (03PS1) 10Ori.livneh: statsdlb / txstatsd: add txstatsdctl and monitoring [puppet] - 10https://gerrit.wikimedia.org/r/193544 [01:22:34] (03PS2) 10Ori.livneh: statsdlb / txstatsd: add txstatsdctl and monitoring [puppet] - 10https://gerrit.wikimedia.org/r/193544 [01:22:42] (03CR) 10Ori.livneh: [C: 032 V: 032] statsdlb / txstatsd: add txstatsdctl and monitoring [puppet] - 10https://gerrit.wikimedia.org/r/193544 (owner: 10Ori.livneh) [01:27:26] PROBLEM - Ori committing changes on the weekend on palladium is CRITICAL: CRITICAL: Ori committed a change on a weekend [01:27:57] 6operations: Give Google webmaster tools access to jon katz (Read only is fine) - https://phabricator.wikimedia.org/T90980#1075186 (10Jalexander) >>! In T90980#1075183, @Krenair wrote: >>>! In T90980#1075177, @Jalexander wrote: >> at this point we have more then 1000 sites listed, so many in fact that yesterday... [01:29:06] ok, ok [01:29:08] i'm done [01:30:46] ori: cf also https://gerrit.wikimedia.org/r/#/c/193542/ [01:31:09] (we need to convert to that for upstart-only services like txstatsd, to support them on jessie w/ a separate systemd unit file) [01:31:23] don't ask me how txstatsdctl plays in that world, but it conflicts [01:31:43] bblack: I saw. There won't be a conflict; the txstatsd role is not currently applied on graphite1001 [01:31:57] I'm just writing up an explanation of the current state of the world in phabricator, I'll ping you with a link in a second [01:32:11] txstatsd gets used on varnishes, which are already moving to jessie [01:32:12] (03PS1) 10Thcipriani: Add hiera yaml for staging, refactor salt class [puppet] - 10https://gerrit.wikimedia.org/r/193545 (https://phabricator.wikimedia.org/T88304) [01:32:27] (hence my packaging it for jessie and mucking the init stuff) [01:32:27] bblack: right, which is why I didn't touch the txstatsd module [01:32:34] so your change is fine. [01:32:54] the multi-instance setup on graphite1001 is a hack, managed discretely from the statsdlb role. it is only meant to carry us over until filippo gets back. [01:33:00] oh jeez [01:33:27] I saw files/txstatsd/init/init.conf, my brain assumed it was the one in the module. I didn't know we had two! [01:33:31] it's just very slow. i'm now up to 10 instances and they are at 80-90% CPU [01:33:43] we didn't have two until an hour or two ago [01:33:47] ok :) [01:33:57] i could either: [01:34:37] long-term: we can do multiple instances with parameter templating for e.g. ports and pidfiles with service_unit, once my template_name patch is merged up [01:34:38] a) refactor the whole txstatsd module to be multi-instance-capable, which would mean having radically divergent and complicated upstart and systemd configs [01:34:51] and would needlessly complicate things for the varnishes [01:35:11] O [01:35:17] err, I'm ok with (a) [01:35:27] no, a) is awful [01:35:36] :) [01:35:50] or (b) have a single virtual service that manages N instances? [01:35:58] b) is accept the fact that we're going to be migrating away from txstatsd (at least on graphite1001) to something more scalable, and just be content with a slightly kludgy bandaid solution until then [01:36:05] ok [01:36:20] seems fair! [01:36:58] godo.g had statsite (the txstatsd replacement) packaged and ready to go, but its behavior is slightly different and it emits a different set of aggregate metrics than txstatsd, so it's a migration that needs some care to pull off, and he didn't want to flip the switch right before going on vacation [01:37:02] which is sensible [01:39:04] anyways, i'm rambling. this is slightly ugly puppet-wise, but it's safe and it works. [01:39:21] are there puppet things that are not ugly? :) [01:40:16] no, there really aren't [01:43:49] 6operations, 7Graphite: scale statsd reporting/aggregation (plan) - https://phabricator.wikimedia.org/T89857#1075206 (10ori) OK, we're out of the woods. statsdlb is provisioned on graphite1001. It is listening on port 8125 and forwarding to ten txstatsd instances on ports 8126 to 8135. Making the txstatsd pu... [01:47:30] PROBLEM - salt-minion processes on ms-be1009 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [01:48:42] 6operations, 7Graphite, 5Patch-For-Review: replace txstatsd - https://phabricator.wikimedia.org/T90111#1075208 (10ori) [01:48:43] 6operations, 7Graphite: scale statsd reporting/aggregation (plan) - https://phabricator.wikimedia.org/T89857#1075209 (10ori) [01:50:44] 6operations, 10RESTBase: Investigate apparent restbase request rate under-reporting in graphite: statsd issue? - https://phabricator.wikimedia.org/T89846#1075211 (10ori) >>! In T89846#1047443, @GWicke wrote: > @ori, re https://gerrit.wikimedia.org/r/191372: Are requests to port 8135 handled by statsd rather th... [01:50:46] 6operations, 10RESTBase: Investigate apparent restbase request rate under-reporting in graphite: statsd issue? - https://phabricator.wikimedia.org/T89846#1075212 (10GWicke) {F49373} [01:50:49] RECOVERY - salt-minion processes on ms-be1009 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [01:51:34] 6operations, 10RESTBase: Investigate apparent restbase request rate under-reporting in graphite: statsd issue? - https://phabricator.wikimedia.org/T89846#1075213 (10GWicke) 5Open>3Resolved a:3GWicke This is now resolved thanks to Ori's statsdlb work. Thank you, @ori! [02:04:10] !log l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s) [02:04:15] Logged the message, Master [02:05:17] !log LocalisationUpdate completed (1.25wmf18) at 2015-02-28 02:04:14+00:00 [02:05:21] Logged the message, Master [02:05:48] !log l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s) [02:05:51] Logged the message, Master [02:06:55] !log LocalisationUpdate completed (1.25wmf19) at 2015-02-28 02:05:51+00:00 [02:06:59] Logged the message, Master [02:08:56] (03PS4) 10BBlack: convert varnish to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193543 [02:08:58] (03PS4) 10BBlack: Add $template_name param to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193288 [02:09:00] (03PS1) 10BBlack: starting point for varnish storage sanitization T90583 [puppet] - 10https://gerrit.wikimedia.org/r/193548 [02:18:51] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 28 02:17:47 UTC 2015 (duration 17m 46s) [02:18:54] Logged the message, Master [02:22:50] RECOVERY - Ori committing changes on the weekend on palladium is OK: OK: Ori is behaving himself [02:54:10] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 1 failures [03:11:40] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:57:29] (03CR) 10Alex Monk: AbuseFilter config change for ukwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/192246 (https://phabricator.wikimedia.org/T89379) (owner: 10Base) [05:38:10] RECOVERY - Graphite Carbon on graphite2001 is OK: OK: All defined Carbon jobs are runnning. [05:41:40] PROBLEM - Graphite Carbon on graphite2001 is CRITICAL: CRITICAL: Not all configured Carbon instances are running. [05:58:39] RECOVERY - uWSGI web apps on graphite2001 is OK: OK: All defined uWSGI apps are runnning. [06:02:00] PROBLEM - uWSGI web apps on graphite2001 is CRITICAL: CRITICAL: Not all configured uWSGI apps are running. [06:28:50] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: puppet fail [06:29:20] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:29] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:29] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:40] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:49] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:50] PROBLEM - puppet last run on db2040 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:00] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:00] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:46:30] RECOVERY - puppet last run on db2040 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:47:00] PROBLEM - puppet last run on mc2010 is CRITICAL: CRITICAL: puppet fail [06:47:10] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:47:21] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:47:29] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:49:11] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:50:10] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:50:50] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:51:21] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:04:40] RECOVERY - puppet last run on mc2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:21] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:49:24] (03PS1) 10Yuvipanda: tools: Move uwsgi services also to generic nodes [puppet] - 10https://gerrit.wikimedia.org/r/193558 (https://phabricator.wikimedia.org/T91065) [07:49:34] (03CR) 10jenkins-bot: [V: 04-1] tools: Move uwsgi services also to generic nodes [puppet] - 10https://gerrit.wikimedia.org/r/193558 (https://phabricator.wikimedia.org/T91065) (owner: 10Yuvipanda) [07:49:41] (03PS2) 10Yuvipanda: tools: Move uwsgi services also to generic nodes [puppet] - 10https://gerrit.wikimedia.org/r/193558 (https://phabricator.wikimedia.org/T91065) [08:35:38] 7Puppet, 10Beta-Cluster, 5Patch-For-Review: Puppet failures on deployment-bastion - https://phabricator.wikimedia.org/T75520#1075468 (10hashar) 5Open>3Resolved It seems so [09:06:28] (03PS1) 10Yuvipanda: tools: Remove tomcat node definitions from puppet [puppet] - 10https://gerrit.wikimedia.org/r/193561 (https://phabricator.wikimedia.org/T91066) [09:06:36] (03CR) 10jenkins-bot: [V: 04-1] tools: Remove tomcat node definitions from puppet [puppet] - 10https://gerrit.wikimedia.org/r/193561 (https://phabricator.wikimedia.org/T91066) (owner: 10Yuvipanda) [09:07:11] <_joe_> YuviPanda: it's saturday... [09:07:57] _joe_: you’re here too :P [09:08:17] <_joe_> YuviPanda: but I'm not working [09:08:27] <_joe_> well, I'm coding, but not for work :P [09:08:35] _joe_: I’m filling in a US Visa form. it’s slow enough that I can write small patches between. [09:08:57] _joe_: toollabs is a lot better today than it was two days ago. will survive a virt* node outage. [09:09:07] but yeah [09:09:10] I’ll stop working soon [09:09:11] :D [09:09:29] EVERYTHING. I DID EVERYTHING. I EVEN WASHED DISHES IN THE OFFICE ONCE. THEY WOULDN'T LET ME SLEEP THERE, THO. SAD. [09:09:29] PLZ2GIVE VISA? FOR EVERYTHING ELSE, THERE IS MASTERCARD. [09:09:42] probably doesn’t count for a ‘job description’ field in the US visa form [09:09:43] <_joe_> lol [09:09:56] when I was young I thought ‘getting a visa’ meant you wouldn’t have foreign currency so needed a debit card to go abroad [09:10:01] since otherwise you would have no money [09:10:04] if only things were that simple [09:11:14] <_joe_> YuviPanda: :/ [09:11:22] <_joe_> I thought you already had a visa [09:13:08] _joe_: work visa. I need to get it stamped. It’s been approved. [09:13:47] <_joe_> lol [09:40:20] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 1 failures [09:56:39] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [10:13:55] (03CR) 10Springle: [C: 04-1] puppetize dbtree config file to connect to tendril (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/193505 (https://phabricator.wikimedia.org/T90837) (owner: 10Dzahn) [10:14:59] (03CR) 10Springle: [C: 031] let puppet clone dbtree into noc/docroot/dbtree [puppet] - 10https://gerrit.wikimedia.org/r/193492 (https://phabricator.wikimedia.org/T90837) (owner: 10Dzahn) [10:15:45] (03CR) 10Springle: [C: 031] move dbtree into its own subrepository, rm here [software] - 10https://gerrit.wikimedia.org/r/193491 (https://phabricator.wikimedia.org/T90837) (owner: 10Dzahn) [10:15:57] (03CR) 10Springle: [C: 031] noc/dbtree: remove the entire dbtree directory [mediawiki-config] - 10https://gerrit.wikimedia.org/r/193143 (https://phabricator.wikimedia.org/T90837) (owner: 10Dzahn) [10:20:29] PROBLEM - HTTP on dataset1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:40] RECOVERY - HTTP on dataset1001 is OK: HTTP OK: HTTP/1.1 200 OK - 5503 bytes in 5.347 second response time [10:37:49] RECOVERY - Graphite Carbon on graphite2001 is OK: OK: All defined Carbon jobs are runnning. [10:41:11] PROBLEM - Graphite Carbon on graphite2001 is CRITICAL: CRITICAL: Not all configured Carbon instances are running. [11:38:30] RECOVERY - Graphite Carbon on graphite2001 is OK: OK: All defined Carbon jobs are runnning. [11:41:59] PROBLEM - Graphite Carbon on graphite2001 is CRITICAL: CRITICAL: Not all configured Carbon instances are running. [12:49:31] 6operations, 6Multimedia, 7HHVM: Convert Imagescalers to HHVM, Trusty - https://phabricator.wikimedia.org/T84842#1075613 (10Kelson) [12:54:27] 6operations, 6Multimedia, 7HHVM: Convert Imagescalers to HHVM, Trusty - https://phabricator.wikimedia.org/T84842#1075615 (10Joe) @Bawolff first of all, thanks for the list of urls to test, this is a great starting point. Then, I apologize for lagging behind on this, but I have been busy with other priorities... [14:09:36] (03CR) 10Hashar: "Yup that is the command used by the Jenkins Debian Glue set of scripts which let us "easily" build Debian packages on patchset proposals :" [puppet] - 10https://gerrit.wikimedia.org/r/191677 (owner: 10Hashar) [14:10:03] (03CR) 10Hashar: [V: 031] "V+1 since I have applied this patch on the integration puppetmaster." [puppet] - 10https://gerrit.wikimedia.org/r/191677 (owner: 10Hashar) [14:21:32] (03CR) 10Yuvipanda: "Why is this ensure present when when other packages are ensure latest?" [puppet] - 10https://gerrit.wikimedia.org/r/191677 (owner: 10Hashar) [15:00:29] (03PS3) 10Thcipriani: Remove regex from labs hiera config [puppet] - 10https://gerrit.wikimedia.org/r/193165 (https://phabricator.wikimedia.org/T90466) [17:00:10] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [17:05:19] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [17:46:07] (03CR) 10Steinsplitter: [C: 031] Set $wgBabelCategoryNames true at outreachwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190686 (https://phabricator.wikimedia.org/T89484) (owner: 10Gerardduenas) [18:05:30] (03CR) 10BBlack: [C: 032] Add $template_name param to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193288 (owner: 10BBlack) [18:08:28] (03CR) 10BBlack: [C: 032] convert varnish to service_unit [puppet] - 10https://gerrit.wikimedia.org/r/193543 (owner: 10BBlack) [18:16:27] _joe_: service_unit is awesome, it DTRT the first time for me on both systemd and upstart-with-sysvinitfile for varnish :) [18:51:34] (03CR) 10Krinkle: "Awesome! I didn't know this existed in puppet. I don't know how tidy works internally, but as trivial as it may seem, there's actually ent" [puppet] - 10https://gerrit.wikimedia.org/r/168999 (owner: 10Ori.livneh) [18:58:00] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: puppet fail [19:17:59] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:49:39] hi - there's some scrutiny in progress for the steward elections, which is based on a labs tool. They've run some queries and now the tool takes a lot to show the results. I wonder if you can do some magic and de-stress it? [19:50:17] your queries have got noticeably slower? [19:50:27] yes Krenair [19:50:47] the tool takes a lot to load and show the results [19:52:30] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [19:52:30] PROBLEM - HTTP 5xx req/min on graphite2001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [19:59:17] mafk: which tool is it? [19:59:23] and what queries is it running? [19:59:32] wmflabs/meta [19:59:42] they're checking votes for steward elections [19:59:45] the electcom [19:59:53] final check before publishing results [20:00:14] https://tools.wmflabs.org/meta/accounteligibility/ [20:01:03] thanks for having a look legoktm [20:01:22] perhaps too much queries per minute? [20:05:40] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [20:05:50] RECOVERY - HTTP 5xx req/min on graphite2001 is OK: OK: Less than 1.00% above the threshold [250.0] [22:00:20] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [22:05:29] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [22:33:07] (03PS1) 10BBlack: dysprosium -> new jessie install test on real hw [puppet] - 10https://gerrit.wikimedia.org/r/193655 [22:33:39] (03CR) 10BBlack: [C: 032 V: 032] dysprosium -> new jessie install test on real hw [puppet] - 10https://gerrit.wikimedia.org/r/193655 (owner: 10BBlack) [22:45:15] (03CR) 10Hashar: "because the others are frequently updated on apt.wikimedia.org ? :-D" [puppet] - 10https://gerrit.wikimedia.org/r/191677 (owner: 10Hashar) [22:58:23] 6operations, 6Security: define in Puppet or remove user account - mglaser - https://phabricator.wikimedia.org/T90947#1075983 (10Mglaser) Hi @Dzahn, thanks for bringing this up. I am not aware of any of these systems. In my time as release manager, I needed access to caesium.equiad, but to none of the above men... [23:25:45] Any opsen around able to do some poking about mail delivery delays with mailman? (I know, stupid question :p) [23:43:49] (03PS3) 10Nemo bis: Enable Collection by default on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/165490 (https://phabricator.wikimedia.org/T73416) (owner: 10Reedy)