[00:00:46] (03PS2) 10Faidon Liambotis: cache: use check_sslxNN instead of NN x check_ssl [puppet] - 10https://gerrit.wikimedia.org/r/182326 [00:01:31] bblack: btw, you can also do sni_cert { $array: } [00:01:53] (or, equivalently, sni_cert { [ 'foo', 'bar', 'baz' ]: } [00:02:50] (03CR) 10Faidon Liambotis: [C: 032] nagios: introduce a new check_sslxNN check [puppet] - 10https://gerrit.wikimedia.org/r/182305 (owner: 10Faidon Liambotis) [00:03:57] (03CR) 10Faidon Liambotis: [C: 032] cache: use check_sslxNN instead of NN x check_ssl [puppet] - 10https://gerrit.wikimedia.org/r/182326 (owner: 10Faidon Liambotis) [00:07:54] (03PS1) 10Faidon Liambotis: nagios: actually ship check_sslxNN [puppet] - 10https://gerrit.wikimedia.org/r/182327 [00:08:50] (03CR) 10Faidon Liambotis: [C: 032 V: 032] nagios: actually ship check_sslxNN [puppet] - 10https://gerrit.wikimedia.org/r/182327 (owner: 10Faidon Liambotis) [00:13:05] oh crap [00:13:42] ? [00:16:09] PROBLEM - HTTPS on cp1048 is CRITICAL: Cant locate check_ssl in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at /usr/lib/nagios/plugins/check_sslxNN line 39. [00:16:09] PROBLEM - HTTPS on cp1063 is CRITICAL: Cant locate check_ssl in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at /usr/lib/nagios/plugins/check_sslxNN line 39. [00:16:12] that [00:16:31] PROBLEM - HTTPS on amssq56 is CRITICAL: Cant locate check_ssl in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at /usr/lib/nagios/plugins/check_sslxNN line 39. [00:18:21] (03PS1) 10Faidon Liambotis: nagios: fix check_sslxNN's require path [puppet] - 10https://gerrit.wikimedia.org/r/182328 [00:18:40] RECOVERY - HTTPS on cp1048 is OK: SSLXNN OK - 36 OK [00:18:40] RECOVERY - HTTPS on cp1063 is OK: SSLXNN OK - 36 OK [00:18:53] (03CR) 10Faidon Liambotis: [C: 032 V: 032] nagios: fix check_sslxNN's require path [puppet] - 10https://gerrit.wikimedia.org/r/182328 (owner: 10Faidon Liambotis) [00:19:11] RECOVERY - HTTPS on amssq56 is OK: SSLXNN OK - 36 OK [00:19:31] better :) [00:20:14] what we really need is a distributed architecture [00:20:20] where checks can be run locally in each DC [00:29:52] ok, fully deployed [00:30:04] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&h=neon.wikimedia.org&m=cpu_report&s=descending&mc=2&g=load_report&c=Miscellaneous+eqiad [00:31:20] 1-min load avg is 23 now [00:31:45] I started off with 700 so I guess 23 is better :) [00:33:15] !log restarted logstash on logstash1001; log2udp events not being recorded in elasticsearch [00:33:19] Logged the message, Master [00:35:17] (03PS1) 10GWicke: Bump cassandra module [puppet] - 10https://gerrit.wikimedia.org/r/182330 [00:36:43] (03CR) 10GWicke: "@alex: Follow-up in https://gerrit.wikimedia.org/r/#/c/182330/." [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/182129 (owner: 10GWicke) [00:42:14] !log log2udp events still not making it into logstash; possibly related to earlier elasticsearch cluster issues; I don't want to restart elasticsearch on logstash1001 while the cluster is still recovering form that. [00:42:19] Logged the message, Master [00:47:45] is that a typo, or is log2udp something different than udp2log? [00:47:53] bah. elasticsearch on logstash1001 is in gc thrash [00:48:20] jgage: they are two haves of the app [00:48:34] gotcha [00:48:36] udp2log writes in; log2udp send out [00:49:11] I think I've got to bounce 1001 even though it is master and the cluster is recovering still [00:49:23] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Logstash%20cluster%20eqiad&h=logstash1001.eqiad.wmnet&r=hour&z=small&jr=&js=&st=1419986761&v=215637&m=es_old_gcs&vl=collections%2Fsec&ti=es_old_gcs&z=large [00:49:36] ah yeah i see example on wikitech: awk -f /a/squid/minnesota.awk | log2udp -h 123.45.67.89 -p 9999 [00:52:09] !log restarted elasticsearch on logstash1001 [00:52:12] Logged the message, Master [01:39:37] bblack: I still can't figure out why load spiked when the hardware was switched back in Sept [01:40:40] sorry, just thinking aloud here :) [01:47:20] (03CR) 10Ori.livneh: [C: 032] "Alexandros approved the submodule commit, so this is fine." [puppet] - 10https://gerrit.wikimedia.org/r/182330 (owner: 10GWicke) [01:49:20] :) [01:51:00] hrm [01:51:43] 'BIOS.PowerMgmt': {'current': 'ActivePwrCtrl', [01:51:48] 'possible': ['OsCtrl', 'ActivePwrCtrl', 'Custom', 'MaxPerf']}, [01:51:55] 'BIOS.ProcPwrPerf': {'current': 'SysDbpm', [01:51:58] 'possible': ['SysDbpm', 'MaxPerf', 'MinPwr', 'OsDbpm']}, [01:52:58] it's an R310 [01:54:04] well we could take a reboot and fix those. tbh I don't recall if that matters or if the linux cpufreq stuff takes precedence at runtime anyways [01:54:33] if you don't set OsDbpm, Linux doesn't see the P-states (frequency) at all [01:54:39] cpufreq isn't loaded [01:54:43] ah! [01:55:12] ActivePwrCtrl is the Dell Active Power Controller [01:55:20] essentially, cpufreq running in the bios :) [01:56:33] http://ganglia.wikimedia.org/latest/graph.php?r=year&z=xlarge&h=neon.wikimedia.org&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Miscellaneous+eqiad [01:56:39] this is a bit ridiculous [01:56:58] either we moved to a much slower system or something else is going on [01:57:18] which I guess is possible [01:59:17] what are you investigating? the reason for the spike in late july? [01:59:30] yeah [01:59:34] not july, september [01:59:44] well, the reason is that the machine was swapped [02:00:15] bdc33a34 files/dhcpd/linux-host-entries.ttyS1-115200 (robh 2012-04-05 17:57:00 -0400 4228) hardware ethernet 78:2b:cb:08:a4:2f; [02:00:21] hm, that's the current one [02:00:33] is the big red band wait or system? i have red-green color blindness [02:00:35] so was it swapped? [02:00:52] system [02:01:20] oh wait, you're right, it's july [02:01:28] how did I not see that [02:02:27] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 0 below the confidence bounds [02:08:38] https://gerrit.wikimedia.org/r/#/c/150010/ probably [02:08:50] there's a drop on Jun 9th, which is when https://gerrit.wikimedia.org/r/#/c/138302/1 was merged [02:09:09] and then a bump with the revert, although higher [02:09:44] there are some items in https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_25#July_30 about neon [02:11:07] ah right, then maybe I was wrong about the date of the swap [02:12:35] (03PS1) 10Springle: depool db1065 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182340 [02:14:04] (03CR) 10Springle: [C: 032] depool db1065 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182340 (owner: 10Springle) [02:14:08] (03Merged) 10jenkins-bot: depool db1065 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182340 (owner: 10Springle) [02:16:08] !log springle Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 05s) [02:16:11] Logged the message, Master [02:22:43] !log upgrade db1065 trusty [02:22:50] Logged the message, Master [02:23:56] (03PS1) 10TTO: Add Anhang and Zitaten namespaces for ndswiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182342 [02:24:38] springle: since you're here... [02:24:50] db1066's puppet is disabled for the past 2 weeks [02:24:55] can I reenable? [02:26:43] paravoid: oh bugger. yes [02:27:02] forgot to reenable after upgrade [02:27:04] The last Puppet run was at Wed Dec 17 05:06:14 UTC 2014 (20000 minutes ago). [02:27:07] haha [02:27:12] lucky me! [02:27:59] sorry [02:28:05] np [02:28:11] I'm laughing at the 20.000 minutes [02:28:42] :) [02:38:41] (03PS1) 10Springle: upgrade db1065 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/182346 [02:44:36] (03CR) 10Springle: [C: 032] upgrade db1065 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/182346 (owner: 10Springle) [02:48:11] (03PS1) 10Ori.livneh: Revoke Brett Simmer's key [puppet] - 10https://gerrit.wikimedia.org/r/182347 [02:48:52] (03CR) 10Ori.livneh: "Is this sufficient, or should we delete the account?" [puppet] - 10https://gerrit.wikimedia.org/r/182347 (owner: 10Ori.livneh) [02:52:43] (03PS1) 10Faidon Liambotis: nagios: minor check_sslxNN ePN fix [puppet] - 10https://gerrit.wikimedia.org/r/182348 [02:53:33] (03CR) 10Faidon Liambotis: [C: 032] nagios: minor check_sslxNN ePN fix [puppet] - 10https://gerrit.wikimedia.org/r/182348 (owner: 10Faidon Liambotis) [02:57:42] our nagios/icinga stuff is so crappy [02:57:54] I'm restraining myself from rewriting the whole damn thing [03:12:51] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [04:38:17] (03CR) 10Faidon Liambotis: [C: 04-1] "The diff is correct but note that precise & trusty sshd pam configs are different so this should be tested against precise as well." [puppet] - 10https://gerrit.wikimedia.org/r/181789 (owner: 10Tim Landscheidt) [05:04:14] springle: I didn't get what to add for wikishared as you suggested. [05:04:34] eg kart_: see ['externalLoads']['extension1'] instead of ['sectionLoads'] [05:10:01] kart_: grep InitializeSettings.php for 'extension1' and see how those array entries are used to establish a DB connection. Echo and Flow both have to do this, so they are your best model. [05:11:08] kart_: and do get a core dev opinion :) i only -1'd your commit to get discussion happening, not because i know all the intricacies of extension configuration :) [05:31:24] springle: no worry about -1 at all :) [05:31:27] Thanks! [05:39:35] springle: https://gerrit.wikimedia.org/r/#/c/182148/ should be discarded? [06:28:41] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:11] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:20] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:42] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:12] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 3 failures [06:30:13] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 3 failures [06:45:50] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:46:01] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:46:21] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:46:41] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [08:33:48] (03PS4) 10KartikMistry: WIP: Content Translation configuration for Production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 [08:41:50] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 91.198.174.247, interfaces up: 36, down: 1, dormant: 0, excluded: 1, unused: 0BRge-0/0/0: down - Core: msw-oe12-esamsBR [09:01:20] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [09:09:20] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 91.198.174.247, interfaces up: 36, down: 1, dormant: 0, excluded: 1, unused: 0BRge-0/0/0: down - Core: msw-oe12-esamsBR [09:24:40] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [09:33:29] (03PS5) 10Filippo Giunchedi: install-server: include lsiutil in reprepro [puppet] - 10https://gerrit.wikimedia.org/r/182194 [09:33:42] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] install-server: include lsiutil in reprepro [puppet] - 10https://gerrit.wikimedia.org/r/182194 (owner: 10Filippo Giunchedi) [09:43:00] !log updating trusty-wikimedia from third-party repo (hwraid) [09:43:07] Logged the message, Master [09:45:05] !log previous reprepro update also accidentally updated elasticsearch in trusty-wikimedia to 1.3.7 [09:45:13] Logged the message, Master [09:47:30] !log updating precise-wikimedia from third-party repo (hwraid) [09:47:35] Logged the message, Master [10:20:03] trying to ack an alert in icinga and got 'Error: Could not open command file '/var/lib/nagios/rw/nagios.cmd' for update!' anyone seeing the same? [10:20:06] permissions look ok to me [10:20:08] prw-rw---- 1 icinga icinga 0 Dec 31 10:19 /var/lib/nagios/rw/nagios.cmd [10:48:51] RECOVERY - ElasticSearch health check for shards on logstash1003 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 8, timed_out: False, active_primary_shards: 41, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 111, initializing_shards: 4, number_of_data_nodes: 3 [10:49:30] RECOVERY - ElasticSearch health check for shards on logstash1002 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 8, timed_out: False, active_primary_shards: 41, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 111, initializing_shards: 4, number_of_data_nodes: 3 [10:49:31] RECOVERY - ElasticSearch health check for shards on logstash1001 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 8, timed_out: False, active_primary_shards: 41, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 111, initializing_shards: 4, number_of_data_nodes: 3 [14:00:38] (03PS1) 10Springle: repool db1065 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182360 [14:01:03] (03CR) 10Springle: [C: 032] repool db1065 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182360 (owner: 10Springle) [14:01:08] (03Merged) 10jenkins-bot: repool db1065 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182360 (owner: 10Springle) [14:02:29] !log springle Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 06s) [14:02:33] Logged the message, Master [14:19:30] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 1 failures [14:37:40] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:20:45] (03PS1) 10RobH: typo in dbstore2001/2002 entries [puppet] - 10https://gerrit.wikimedia.org/r/182363 [15:22:57] (03CR) 10RobH: [C: 032] typo in dbstore2001/2002 entries [puppet] - 10https://gerrit.wikimedia.org/r/182363 (owner: 10RobH) [15:55:44] (03PS1) 10Glaisher: Fix typo in dvwiktionary project talk namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182364 [16:34:59] (03PS1) 10Glaisher: Create 'uploader' user group on kowiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182367 [16:36:28] (03CR) 10Revi: [C: 031] Create 'uploader' user group on kowiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182367 (owner: 10Glaisher) [16:48:58] * valhallasw`cloud waves at wikibugs [16:49:10] enjoy the ops-* messages, all, and happy new year *waves* [16:49:11] <^d> yay! [17:18:35] PROBLEM - Host d-i-test is DOWN: PING CRITICAL - Packet loss = 100% [17:21:35] RECOVERY - Host d-i-test is UP: PING OK - Packet loss = 0%, RTA = 2.27 ms [17:37:56] PROBLEM - NTP on d-i-test is CRITICAL: NTP CRITICAL: Offset unknown [17:40:15] RECOVERY - NTP on d-i-test is OK: NTP OK: Offset 0.000635266304 secs [18:41:36] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 1 failures [18:44:18] 3RESTBase, operations: Public entry point for RESTBase - https://phabricator.wikimedia.org/T78194#951159 (10ssastry) How about rest.wikimedia.org? [18:49:50] 3RESTBase, operations: Public entry point for RESTBase - https://phabricator.wikimedia.org/T78194#951167 (10GWicke) `rest.wikimedia.org` would wfm too. [18:59:16] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:04:20] 3RESTBase, operations: Public entry point for RESTBase - https://phabricator.wikimedia.org/T78194#951185 (10Jdouglas) {icon thumbs-up} rest.wikimedia.org {icon thumbs-up} [19:05:57] 3RESTBase, operations: Public entry point for RESTBase - https://phabricator.wikimedia.org/T78194#951188 (10mobrovac) +1 for `rest.wikimedia.org` [19:26:42] 3RESTBase, operations: Public entry point for RESTBase - https://phabricator.wikimedia.org/T78194#951199 (10Hardikj) big +1 to rest.wikimedia.org or rest.api.wikimedia.org [19:45:55] (03PS1) 10Faidon Liambotis: base: use dash instead of bash for last-puppet-run [puppet] - 10https://gerrit.wikimedia.org/r/182372 [19:45:57] (03PS1) 10Faidon Liambotis: motd: misc fixes [puppet] - 10https://gerrit.wikimedia.org/r/182373 [19:45:59] (03PS1) 10Faidon Liambotis: motd: add motd::defaults class [puppet] - 10https://gerrit.wikimedia.org/r/182374 [19:46:01] (03PS1) 10Faidon Liambotis: base: include the motd class by default [puppet] - 10https://gerrit.wikimedia.org/r/182375 [19:46:03] (03PS1) 10Faidon Liambotis: Use motd::script instead of File across the tree [puppet] - 10https://gerrit.wikimedia.org/r/182376 [19:46:05] (03PS1) 10Faidon Liambotis: motd: manage /etc/motd, support Debian [puppet] - 10https://gerrit.wikimedia.org/r/182377 [19:46:14] and with that, I'm off [19:46:16] happy 2015 :) [19:46:20] happy 2015, paravoid [19:46:31] oh hi [19:47:02] reviews for the above welcome, ori, I saw this the module is imported from vagrant [19:47:11] yeah, i'm reviewing :) [19:47:13] so it'd be nice to keep it compatible [19:48:10] (03CR) 10Ori.livneh: [C: 031] base: use dash instead of bash for last-puppet-run [puppet] - 10https://gerrit.wikimedia.org/r/182372 (owner: 10Faidon Liambotis) [19:48:18] happy new year guys [19:49:02] you too matanya [20:00:45] (03CR) 10Ori.livneh: base: use dash instead of bash for last-puppet-run (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182372 (owner: 10Faidon Liambotis) [20:02:29] (03CR) 10Ori.livneh: "runparts takes about a tenth of a second to run on our systems, so I'd prefer to have Puppet manage state rather than manage things that m" [puppet] - 10https://gerrit.wikimedia.org/r/182373 (owner: 10Faidon Liambotis) [20:06:36] (03CR) 10Ori.livneh: "How about simply adding ignore => [ '00-header', '99-footer' ] to File['/etc/update-motd.d']?" [puppet] - 10https://gerrit.wikimedia.org/r/182374 (owner: 10Faidon Liambotis) [20:07:23] (03CR) 10Ori.livneh: [C: 031] base: include the motd class by default [puppet] - 10https://gerrit.wikimedia.org/r/182375 (owner: 10Faidon Liambotis) [20:11:36] (03CR) 10Ori.livneh: "Nice! I don't think the exact order matters to these scripts, so leaving priority unspecified is probably best." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182376 (owner: 10Faidon Liambotis) [20:12:20] 3RESTBase, operations: Public entry point for RESTBase - https://phabricator.wikimedia.org/T78194#951212 (10GWicke) [21:05:26] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [21:07:46] (03PS1) 1020after4: Add the SecurityPolicyEventListener to phabricator config [puppet] - 10https://gerrit.wikimedia.org/r/182380 [21:11:15] (03PS2) 1020after4: Add the SecurityPolicyEventListener to phabricator config [puppet] - 10https://gerrit.wikimedia.org/r/182380 [21:44:30] godog, yt? it seems that mw performance metrics aren't being recorded in graphite anymore [21:45:26] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [22:28:47] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [23:24:25] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [23:42:08] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [23:42:32] (03PS1) 10Springle: depool db1061 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182389 [23:43:20] (03CR) 10Springle: [C: 032] depool db1061 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182389 (owner: 10Springle) [23:43:28] (03Merged) 10jenkins-bot: depool db1061 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182389 (owner: 10Springle) [23:44:56] !log springle Synchronized wmf-config/db-eqiad.php: depool db1061 (duration: 00m 05s) [23:45:02] Logged the message, Master [23:51:45] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 1 failures [23:51:45] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 1 failures [23:52:16] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 1 failures