[00:00:01] awesome! [00:00:02] looking [00:01:40] looks good to me [00:02:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 41.03, 26.43, 19.03 [00:02:25] I wouldn't expect a different answer for the varnish expert :-P [00:02:31] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:03:06] :P [00:03:26] SPF|Cloud one thing i just thought of, do we what to whitelist our internal ips? [00:04:05] unless there is proof internal services send too much, but legitimate traffic, no [00:04:35] an internal service may act rogue due to any reason, and that shouldn't be able to bring mediawiki down [00:05:10] ok [00:05:15] if parsoid needs to contact the API lots of times, it should receive a higher limit by looking at the exact URL it's showing [00:05:47] e.g. if 50 requests per second are sent to /w/api.php?action=visualeditorblabla&page=blabla, the rate limit should only be increased for that URL [00:05:53] and not for everything on /w/ [00:06:19] ah ok! [00:06:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.33, 21.57, 18.84 [00:06:25] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:06:43] ^ cp7 is fine, that's me testing [00:07:25] I wonder why https://schuelerwiki.miraheze.org/wiki/Hauptseite?action=edit 500s [00:07:26] [ Bearbeiten von „Hauptseite“ – Schülerwiki ] - schuelerwiki.miraheze.org [00:08:18] 51.89.160.132 - - [03/Oct/2020:00:07:30 +0000] "POST https://schuelerwiki.miraheze.org/w/api.php" 429 305 "-" "Parsoid/0.11.0" 0.002 TLSv1.2/ECDHE-RSA-CHACHA20-POLY1305 [00:08:19] cp7 looks good [00:08:20] [ MediaWiki-API-Hilfe – Schülerwiki ] - schuelerwiki.miraheze.org [00:08:51] seems parsoid is hitting the rate limit [00:09:08] not raising the limit unless there's proof [00:09:36] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUQZe [00:09:38] [02miraheze/dns] 07Southparkfan 03d82899f - Revert "Depool cp7" This reverts commit 8ddc95087976640747145bf50e443239aabbcb4f. Testing is done :) [00:10:05] aren't there error logs for parsoid on services1/2? [00:10:09] yeh [00:10:25] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 2% [00:10:37] but looking at the access.log on cp6, it's showing parsoid is being rate limited per 429 and the user agent. [00:12:45] looks correct [00:13:06] I'll take a look to raise the limit [00:13:10] thanks! [00:14:22] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.80, 18.94, 18.80 [00:21:22] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUQZP [00:21:23] [02miraheze/puppet] 07Southparkfan 034532046 - Raise rate limit for Parsoid [00:23:39] and fixed [00:25:33] one high prio task gone :) [00:26:02] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 43% [00:26:10] :D [00:26:21] cp9: is that someone hitting the new rate limit? [00:26:26] getting curious [00:27:28] yep! [00:27:47] heh [00:27:59] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 36% [00:28:00] SPF|Cloud that means icinga will be spamming. [00:28:40] that's not an issue I need to fix by raising limits [00:28:47] ok [00:28:52] that indicates actual abuse [00:29:14] ok [00:29:23] fixing that should be asking those folks to stop spamming us, not us raising limits for them [00:29:28] that defeats the purpose of this limit [00:34:20] Oof, I see what you guys mean... > Error 429 Varnish Rate Limit Exceeded > Varnish Rate Limit Exceeded > > Guru Meditation: > XID: 7471796 > > Varnish cache server [00:34:34] Just got that on Meta [00:35:51] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 49% [00:35:53] Doug: I am able to trigger that error, but that shouldn't happen unless you open five pages at the same time [00:35:58] I can view the watchlist on Meta, but trying to view any log page on Meta, I get hit with a 429 [00:36:52] @SPF|Cloud, nope, I only have 4 Miraheze tabs open, and three haven't been refreshed in more than a day, plus they're for different wikis [00:38:03] Doug: mind sending me in PM: (a part of) your IP and the name of the cache proxy you're using [00:38:40] @SPF|Cloud, was just going to ask if it'd help if I sent you my IP in a DM [00:39:15] yes, please [00:39:30] and if you are able to figure our which cache proxy you are using, that would save me a lot of time [00:39:42] sure, should I get the HTTP headers from an online curl tool, or just use the Chrome/Vivaldi developer console to get the name of the cache proxy? [00:39:46] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 33% [00:40:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUQno [00:40:21] [02miraheze/services] 07MirahezeSSLBot 03808777e - BOT: Updating services config for wikis [00:40:26] the contents of the HTTP header X-Cache is sufficient [00:40:44] @SPF|Cloud, okay, opening a DM then [00:43:41] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 46% [00:45:38] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 32% [00:55:28] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 40% [00:59:20] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 34% [01:03:14] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 44% [01:05:14] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 35% [01:09:14] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 48% [01:11:14] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 19% [01:14:53] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUQCK [01:14:55] [02miraheze/puppet] 07Southparkfan 031fd943a - Increase fallback rate limit from 5 to 12 requests per 2s [01:43:27] !log reboot graylog1 for memory upgrade [01:43:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:06] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUQl5 [02:01:08] [02miraheze/dns] 07Southparkfan 03a35e9c9 - Add graylog CNAME to graylog1.mh.o [02:29:43] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+113/-0/±1] 13https://git.io/JUQ46 [02:29:45] [02miraheze/puppet] 07Southparkfan 03a53dc5a - add base config for graylog [02:32:35] PROBLEM - graylog1 Puppet on graylog1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [02:34:33] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:35:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUQ45 [02:35:12] [02miraheze/services] 07MirahezeSSLBot 03f732055 - BOT: Updating services config for wikis [02:35:55] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUQ4b [02:35:56] [02miraheze/puppet] 07Southparkfan 0335f6390 - Fix graylog file [02:51:14] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+39/-44/±66] 13https://git.io/JUQBo [02:51:16] [02miraheze/puppet] 07Southparkfan 03726dcf8 - Commit more fixes for graylog? [03:04:14] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:08] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:09:38] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+51/-10/±21] 13https://git.io/JUQRo [03:09:40] [02miraheze/puppet] 07Southparkfan 03d20de65 - Add graylog module, update java module, update graylog.pp [03:10:06] [02miraheze/puppet] 07dependabot[bot] pushed 031 commit to 03dependabot/bundler/modules/graylog/json-2.3.1 [+0/-0/±1] 13https://git.io/JUQRK [03:10:08] [02miraheze/puppet] 07dependabot[bot] 0346caebc - build(deps): bump json from 1.8.3 to 2.3.1 in /modules/graylog Bumps [json](https://github.com/flori/json) from 1.8.3 to 2.3.1. - [Release notes](https://github.com/flori/json/releases) - [Changelog](https://github.com/flori/json/blob/master/CHANGES.md) - [Commits](https://github.com/flori/json/compare/v1.8.3...v2.3.1) Signed-off-by: dependabot[bot] [03:10:08] [03:10:08] [ GitHub - flori/json: JSON implementation for Ruby ] - github.com [03:10:09] [ Releases · flori/json · GitHub ] - github.com [03:10:09] [02puppet] 07dependabot[bot] opened pull request 03#1502: build(deps): bump json from 1.8.3 to 2.3.1 in /modules/graylog - 13https://git.io/JUQR6 [03:10:10] [ json/CHANGES.md at master · flori/json · GitHub ] - github.com [03:10:11] [02puppet] 07dependabot[bot] created branch 03dependabot/bundler/modules/graylog/json-2.3.1 - 13https://git.io/vbiAS [03:10:12] [02puppet] 07dependabot[bot] labeled pull request 03#1502: build(deps): bump json from 1.8.3 to 2.3.1 in /modules/graylog - 13https://git.io/JUQR6 [03:10:14] [02miraheze/puppet] 07dependabot[bot] pushed 031 commit to 03dependabot/bundler/modules/graylog/rake-13.0.1 [+0/-0/±1] 13https://git.io/JUQRi [03:10:15] [02miraheze/puppet] 07dependabot[bot] 03eb309b4 - build(deps): bump rake from 11.1.2 to 13.0.1 in /modules/graylog Bumps [rake](https://github.com/ruby/rake) from 11.1.2 to 13.0.1. - [Release notes](https://github.com/ruby/rake/releases) - [Changelog](https://github.com/ruby/rake/blob/master/History.rdoc) - [Commits](https://github.com/ruby/rake/compare/v11.1.2...v13.0.1) Signed-off-by: dependabot[bot] [03:10:15] [03:10:16] [ GitHub - ruby/rake: A make-like build utility for Ruby. ] - github.com [03:10:16] [ Releases · ruby/rake · GitHub ] - github.com [03:10:17] [02puppet] 07dependabot[bot] created branch 03dependabot/bundler/modules/graylog/rake-13.0.1 - 13https://git.io/vbiAS [03:10:17] [ rake/History.rdoc at master · ruby/rake · GitHub ] - github.com [03:10:17] [ Comparing v1.8.3...v2.3.1 · flori/json · GitHub ] - github.com [03:10:18] [02puppet] 07dependabot[bot] opened pull request 03#1503: build(deps): bump rake from 11.1.2 to 13.0.1 in /modules/graylog - 13https://git.io/JUQRX [03:10:20] [02puppet] 07dependabot[bot] labeled pull request 03#1503: build(deps): bump rake from 11.1.2 to 13.0.1 in /modules/graylog - 13https://git.io/JUQRX [03:10:23] [ Comparing v11.1.2...v13.0.1 · ruby/rake · GitHub ] - github.com [03:15:05] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:16:21] !log add ufw rule on ldap1 to allow graylog1 -> ldap1 port 636 (to be puppetised) [03:16:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:15:28] PROBLEM - wiki.lbcomms.co.za - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.lbcomms.co.za All nameservers failed to answer the query. [05:48:48] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 47% [05:50:46] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 31% [06:02:28] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 40% [06:04:28] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 18% [06:07:01] RECOVERY - misc1 APT on misc1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:45:52] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 40% [06:47:49] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 31% [07:13:18] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki testwiki --r "Requested - [[phab:T6260|T6260]]" /home/reception/testwikidel.txt [07:13:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:27:09] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 46% [07:31:04] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 39% [07:36:55] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 48% [07:38:53] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 35% [07:42:45] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 45% [07:48:37] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 16% [08:24:55] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 41% [08:28:50] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [08:30:47] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 46% [08:36:43] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [08:38:41] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 53% [08:44:34] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [08:48:30] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 58% [09:00:29] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 37% [09:04:28] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 59% [09:14:28] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 39% [09:24:15] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 47% [09:34:05] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 39% [09:39:54] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 57% [09:41:53] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 15% [09:45:48] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 54% [09:49:46] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [09:51:45] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 42% [09:53:43] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 39% [09:57:35] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 44% [09:59:34] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 39% [10:05:27] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 54% [10:13:17] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [10:15:15] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 49% [10:25:08] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 36% [10:29:04] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 49% [10:40:51] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 37% [10:44:43] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 58% [10:58:30] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 36% [11:02:28] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 43% [11:38:27] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 38% [11:46:28] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 48% [11:47:20] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:54:28] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [11:56:06] !log silenced icinga check for cp9 http error rate [11:56:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:16] PROBLEM - graylog1 Puppet on graylog1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [12:07:52] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7tm [12:07:54] [02miraheze/puppet] 07Southparkfan 03750c8cc - Graylog: fix broken elasticsearch setup [12:09:15] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [12:09:23] !log disable puppet on test2 [12:09:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:02] PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: SPF; setting up syslog-ng for graylog, last run 5 minutes ago with 2 failures [12:15:17] PROBLEM - graylog1 Puppet on graylog1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [13:42:05] RECOVERY - graylog1 HTTPS on graylog1 is OK: HTTP OK: HTTP/1.1 200 OK - 1670 bytes in 0.013 second response time [13:43:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7ZG [13:43:45] [02miraheze/puppet] 07paladox 03ede1e86 - Allow graylog to access ldap ssl port [14:02:47] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.17, 20.48, 17.73 [14:04:45] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.21, 17.92, 17.12 [14:09:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JU7cs [14:09:22] [02miraheze/mw-config] 07paladox 03bd02df9 - CookieWarning: Change wgCookieWarningGeoIPLookup to js [14:09:23] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [14:09:25] [02mw-config] 07paladox opened pull request 03#3275: CookieWarning: Change wgCookieWarningGeoIPLookup to js - 13https://git.io/JU7cG [14:10:58] miraheze/mw-config/paladox-patch-1/bd02df9 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/732513564 [14:42:31] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7l8 [14:42:33] [02miraheze/puppet] 07Reception123 0345710bc - add z.moatads.com to CSP approved by second SRE John (T4914#122296) for AddThis extension [14:54:44] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.26, 23.89, 19.65 [14:55:36] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.19, 5.76, 4.08 [14:55:48] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [14:56:15] PROBLEM - test2 php-fpm on test2 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.3' [14:56:41] PROBLEM - cp3 Stunnel Http for test2 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.760 second response time [14:57:03] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.014 second response time [14:57:20] PROBLEM - cp7 Stunnel Http for test2 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.005 second response time [14:57:36] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.90, 5.53, 4.18 [14:57:48] PROBLEM - cp9 Stunnel Http for test2 on cp9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.243 second response time [14:57:49] PROBLEM - cp6 Stunnel Http for test2 on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [14:58:41] ^ expected [14:58:45] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.80, 22.33, 20.02 [14:58:48] alerts should have been silenced, my fault [15:00:46] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.25, 23.60, 20.74 [15:02:46] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.14, 23.23, 20.97 [15:03:49] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:44] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.33, 24.73, 21.74 [15:06:44] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.98, 22.67, 21.38 [15:08:44] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.00, 19.61, 20.40 [15:15:13] [02puppet] 07paladox closed pull request 03#1503: build(deps): bump rake from 11.1.2 to 13.0.1 in /modules/graylog - 13https://git.io/JUQRX [15:15:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7BR [15:15:16] [02miraheze/puppet] 07dependabot[bot] 0352d5c26 - build(deps): bump rake from 11.1.2 to 13.0.1 in /modules/graylog (#1503) Bumps [rake](https://github.com/ruby/rake) from 11.1.2 to 13.0.1. - [Release notes](https://github.com/ruby/rake/releases) - [Changelog](https://github.com/ruby/rake/blob/master/History.rdoc) - [Commits](https://github.com/ruby/rake/compare/v11.1.2...v13.0.1) Signed-off-by: dependabot[bot] [15:15:16] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [15:15:16] [ GitHub - ruby/rake: A make-like build utility for Ruby. ] - github.com [15:15:17] [ Releases · ruby/rake · GitHub ] - github.com [15:15:17] [02puppet] 07paladox deleted branch 03dependabot/bundler/modules/graylog/rake-13.0.1 - 13https://git.io/vbiAS [15:15:17] [ rake/History.rdoc at master · ruby/rake · GitHub ] - github.com [15:15:19] [02miraheze/puppet] 07paladox deleted branch 03dependabot/bundler/modules/graylog/rake-13.0.1 [15:15:22] [02puppet] 07paladox closed pull request 03#1502: build(deps): bump json from 1.8.3 to 2.3.1 in /modules/graylog - 13https://git.io/JUQR6 [15:15:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7BE [15:15:24] [ Comparing v11.1.2...v13.0.1 · ruby/rake · GitHub ] - github.com [15:15:25] [02miraheze/puppet] 07dependabot[bot] 0308a189a - build(deps): bump json from 1.8.3 to 2.3.1 in /modules/graylog (#1502) Bumps [json](https://github.com/flori/json) from 1.8.3 to 2.3.1. - [Release notes](https://github.com/flori/json/releases) - [Changelog](https://github.com/flori/json/blob/master/CHANGES.md) - [Commits](https://github.com/flori/json/compare/v1.8.3...v2.3.1) Signed-off-by: dependabot[bot] [15:15:25] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [15:15:25] [ GitHub - flori/json: JSON implementation for Ruby ] - github.com [15:15:26] [ Releases · flori/json · GitHub ] - github.com [15:15:26] [02miraheze/puppet] 07paladox deleted branch 03dependabot/bundler/modules/graylog/json-2.3.1 [15:15:26] [ json/CHANGES.md at master · flori/json · GitHub ] - github.com [15:15:28] [02puppet] 07paladox deleted branch 03dependabot/bundler/modules/graylog/json-2.3.1 - 13https://git.io/vbiAS [15:15:31] [ Comparing v1.8.3...v2.3.1 · flori/json · GitHub ] - github.com [15:16:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7Br [15:16:08] [02miraheze/puppet] 07paladox 03f0561a8 - Update Gemfile [15:26:46] !log disable puppet2's puppet agent and disable git pull for puppet repo [15:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:28:55] PROBLEM - puppet2 Puppet on puppet2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [15:33:15] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.32, 21.27, 19.47 [15:35:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.65, 20.64, 19.42 [15:37:07] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.38, 19.87, 19.30 [15:43:45] [02miraheze/puppet] 07Southparkfan pushed 032 commits to 03master [+89/-1/±6] 13https://git.io/JU7Ef [15:43:46] [02miraheze/puppet] 07Southparkfan 030bc0e8a - Introduce base support for syslog-ng [15:43:48] [02miraheze/puppet] 07Southparkfan 0350d7f5c - Merge branch 'master' of github.com:miraheze/puppet [15:45:03] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core] [15:45:49] RECOVERY - cp6 Stunnel Http for test2 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15672 bytes in 0.060 second response time [15:46:14] RECOVERY - test2 php-fpm on test2 is OK: PROCS OK: 27 processes with command name 'php-fpm7.3' [15:46:39] RECOVERY - cp3 Stunnel Http for test2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15664 bytes in 0.934 second response time [15:47:04] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 17201 bytes in 0.328 second response time [15:47:18] RECOVERY - cp7 Stunnel Http for test2 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15664 bytes in 0.005 second response time [15:47:43] RECOVERY - cp9 Stunnel Http for test2 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15664 bytes in 0.323 second response time [15:57:38] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.24, 19.97, 19.02 [15:59:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.19, 18.94, 18.79 [16:32:19] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:26] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:29] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:29] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:30] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:39] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:44] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:45] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:48] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:53] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:03] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:06] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:11] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:17] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:18] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:19] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:23] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:24] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:25] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:32] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:33] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:38] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:51] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:56] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:57] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:58] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:34:06] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:34:09] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:34:10] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:34:12] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:39:02] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:41:57] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:41:58] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:42:06] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:42:08] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:42:21] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:42:27] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:42:29] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:42:30] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:42:39] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:42:45] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:42:48] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:42:48] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:42:53] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:04] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:06] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:11] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:17] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:19] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:20] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:23] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:24] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:43:25] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:32] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:34] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:38] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:43:50] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:56] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:10] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:30] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:11] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:50:28] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7V1 [16:50:29] [02miraheze/puppet] 07Southparkfan 03e1cd105 - syslog::base: define global options, collect all internal messages and send to graylog [16:50:30] paladox ^ [16:50:47] cool eh? ;) [16:53:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.46, 21.17, 19.74 [16:54:31] @southparkfan We cannot start a search right now, because you are not allowed to access any stream. If you feel this is an error, please contact your administrator. [16:54:43] On graylog when logged in [16:54:58] good to know the ldap groups are working properly [16:55:13] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.83, 20.38, 19.61 [16:55:41] only the sre group is allowed to do about any action in graylog, although a group for mediawiki-admins must be created, yes [16:56:11] Yes a group for us would be nice so we can check exceptions we can normally read [16:56:27] of course. [16:57:08] is very nosey and actually has some time today so decided to try sign in [16:57:14] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7wB [16:57:15] [02miraheze/puppet] 07Southparkfan 034bf69f4 - syslog-ng: restrict to TLSv1.2+ [16:59:25] paladox: is there a mediawiki-admins group in ldap? [16:59:52] I can't see one, but would be useful for ldap [16:59:57] err, graylog [17:00:25] Nope [17:01:57] Should we create one? [17:02:11] that is preferred, yes [17:02:34] looks back at @paladox [17:03:31] Yup [17:03:32] SPF|Cloud: if you want to do that now, instructions are on wiki for creating the group [17:03:33] Currently movile [17:03:34] Mobile [17:03:41] alright [17:05:05] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:11:58] !log zipping individual att image directories [17:12:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:24] paladox: I think the documentation is outdated [17:19:04] according to wiki: adding a new group with objectclasses top and posixGroup, but ldapvi says all other groups have top and groupOfNames [17:21:38] and when changing that via ldapvi, I get "additional info: structural object class modification from 'posixGroup' to 'groupOfNames' not allowed", so ... [17:23:14] I love our documentation at times [17:24:44] Rhinos: anyways, I have given you extra permissions now [17:31:27] Ahhh [17:36:40] SPF|Cloud did you manage to create the group? [17:37:06] ah, you did! [17:37:16] I see it [17:37:56] I see 2 nginx streams [17:38:00] But nothing in them [17:40:43] paladox: yes, but the group is not properly created [17:40:49] Ah ok [17:40:51] It must have the same object classes as the others [17:40:55] SPF|Cloud i'll delete it and recreate it [17:41:08] Rhinos: well, nginx logging is not enabled yet [17:41:28] It was earlier today though, so you need to set the search limit to 12 hours ag9 [17:44:35] Ok [17:45:48] Search in last day still shows nothing in access logs [17:48:07] SPF|Cloud done [18:02:36] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JU7Xe [18:02:38] [02miraheze/ssl] 07Pix1234 03fda340c - add minecraftathomewiki per T6269 [18:25:12] RhinosF1: ah yes, the stream only works for messages that were sent after the stream was created [18:25:42] paladox: thanks for updating the docs [18:25:46] yw [18:25:59] I guess you need to tell SRE to use the write-user instead of admin as well? [18:28:00] yeh [18:30:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7Mu [18:30:21] [02miraheze/services] 07MirahezeSSLBot 0386a5626 - BOT: Updating services config for wikis [18:30:43] done [18:30:43] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:34:25] @southparkfan ack [18:39:43] I've enabled nginx streaming rhinos [18:39:46] should see events now [18:41:41] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JU7Dc [18:41:42] [02miraheze/puppet] 07Southparkfan 036139793 - nginx module: if using syslog-ng/graylog, send messages from socket to graylog [18:42:23]