[00:01:59] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for misc2 on cp8 is UNKNOWN: NRPE: Command 'check_stunnel_misc2' not defined
[00:02:22] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for misc2 on cp4 is UNKNOWN: NRPE: Command 'check_stunnel_misc2' not defined
[00:12:53] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mon1 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host
[00:12:55] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mon1 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host
[00:13:33] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mon1 on cp8 is CRITICAL: HTTP CRITICAL - No data received from host
[00:42:23] <Not-5a0a>	 [02puppet] 07The-Voidwalker opened pull request 03#1257: rm cnoc from cvt mail - 13https://git.io/JvRpm
[01:07:29] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2111 MB (10% inode=93%);
[01:40:11] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRhh
[01:40:12] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03da6a828 - matomo: Switch to db6
[01:41:29] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mon1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 29495 bytes in 0.220 second response time
[01:41:47] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mon1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 29525 bytes in 1.160 second response time
[01:42:20] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mon1 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 29495 bytes in 0.484 second response time
[01:45:48] <paladox>	 !log drop database piwik on db5
[01:46:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:07:58] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRj1
[02:08:00] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0382c31bb - misc2: Remove matomo
[02:12:28] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRjM
[02:12:29] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03dafb62e - php_fpm: Add mon1 to firewall
[02:13:15] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRjD
[02:13:17] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0379cf221 - redis_exporter: Add mon1 to firewall
[02:13:57] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRj9
[02:13:58] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03fc67d93 - nginx: Add mon1 to firewall
[02:14:56] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRjH
[02:14:57] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c5d120f - varnish_prometheus_exporter: Add mon1 to firewall
[02:27:15] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0eT
[02:27:16] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03f8cadb2 - switch rc feed to use mon1 (51.89.160.138)
[02:37:35] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0el
[02:37:36] <Not-5a0a>	 [02miraheze/dns] 07paladox 037751c84 - grafana: Switch backend to mon1 Also remove grafana-new
[02:39:19] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0eB
[02:39:20] <Not-5a0a>	 [02miraheze/puppet] 07paladox 035d59275 - matomo: Configure memory limit of 2g for cli
[02:43:47] <paladox>	 !log remove php/nginx from misc2
[02:43:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:45:02] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[02:49:23] <paladox>	 !log apt-get install puppet-agent mariadb-client-10.1 mariadb-client-core-10.1 mariadb-common - misc2
[02:49:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:51:02] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jv0eD
[02:51:03] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03ed0f6d0 - grafana: Switch to new db (db6)
[02:51:05] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS
[02:51:06] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1258: grafana: Switch to new db (db6) - 13https://git.io/Jv0ey
[02:52:07] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1258: grafana: Switch to new db (db6) - 13https://git.io/Jv0ey
[02:52:09] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0eQ
[02:52:10] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03b916b80 - grafana: Switch to new db (db6) (#1258)
[02:55:03] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0eb
[02:55:05] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0309c7ee5 - misc1: Remove grafana
[02:55:36] <paladox>	 !log drop grafana db on db4
[02:55:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:56:48] <icinga-miraheze>	 PROBLEM - misc1 grafana.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time
[02:57:10] <icinga-miraheze>	 PROBLEM - misc1 HTTPS on misc1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.005 second response time
[03:02:22] <icinga-miraheze>	 RECOVERY - misc1 HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 334 bytes in 0.009 second response time
[03:07:21] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0ve
[03:07:22] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03a2a8543 - mw[4567] increase childs
[03:07:24] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://git.io/vbiAS
[03:07:25] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1259: mw[4567] increase childs - 13https://git.io/Jv0vv
[03:10:58] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0vT
[03:10:59] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03ee17868 - Update mw4.yaml
[03:11:01] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1259: mw[4567] increase childs - 13https://git.io/Jv0vv
[03:11:08] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0vk
[03:11:10] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03e8786b1 - Update mw5.yaml
[03:11:11] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1259: mw[4567] increase childs - 13https://git.io/Jv0vv
[03:11:22] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0vL
[03:11:23] <Not-5a0a>	 [02miraheze/puppet] 07paladox 031b88b42 - Update mw6.yaml
[03:11:25] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1259: mw[4567] increase childs - 13https://git.io/Jv0vv
[03:11:33] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0vt
[03:11:35] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03b5c9c92 - Update mw7.yaml
[03:11:36] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1259: mw[4567] increase childs - 13https://git.io/Jv0vv
[03:11:46] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1259: mw[4567] increase childs - 13https://git.io/Jv0vv
[03:11:47] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jv0vm
[03:11:49] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03de856bc - mw[4567] increase childs (#1259) * mw[4567] increase childs * Update mw4.yaml * Update mw5.yaml * Update mw6.yaml * Update mw7.yaml
[03:15:53] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://git.io/vbiAS
[03:15:55] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9
[03:16:59] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0vC
[03:17:01] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0390d8c45 - Fix childs
[03:17:02] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://git.io/vbiAS
[03:17:04] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1260: Fix childs - 13https://git.io/Jv0vW
[03:17:20] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0vl
[03:17:22] <Not-5a0a>	 [02miraheze/puppet] 07paladox 034264740 - Update mw5.yaml
[03:17:23] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1260: Fix childs - 13https://git.io/Jv0vW
[03:17:34] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0v8
[03:17:35] <Not-5a0a>	 [02miraheze/puppet] 07paladox 034593bf3 - Update mw6.yaml
[03:17:37] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1260: Fix childs - 13https://git.io/Jv0vW
[03:17:46] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/Jv0v4
[03:17:48] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03d247ac7 - Update mw7.yaml
[03:17:49] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1260: Fix childs - 13https://git.io/Jv0vW
[03:18:01] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1260: Fix childs - 13https://git.io/Jv0vW
[03:18:02] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jv0vB
[03:18:04] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c628458 - Fix childs (#1260) * Fix childs * Update mw5.yaml * Update mw6.yaml * Update mw7.yaml
[03:18:05] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://git.io/vbiAS
[03:18:07] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9
[03:18:30] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS
[03:18:32] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8
[03:20:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0vR
[03:20:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 037cfb530 - BOT: Updating services config for wikis
[03:21:48] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±8] 13https://git.io/Jv0v0
[03:21:50] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0328249bb - Revert "Fix childs (#1260)" This reverts commit c628458278f34b9660c80a8ce848275d0ca14176.
[03:21:51] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03983938f - Revert "mw[4567] increase childs (#1259)" This reverts commit de856bc745dcb8bfaaa2a1bb4f552380577304a7.
[03:44:00] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jv0vx
[03:44:01] <Not-5a0a>	 [02miraheze/puppet] 07paladox 037478e11 - switch cp[248] to use mw[4567] This is a experiment to see how well mw will handle things. Note that reliability may improve (e.g pages load, rather then slow itermittently, but performance may drop a bit due to db4 latency).
[03:44:03] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS
[03:44:04] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1261: switch cp[248] to use mw[4567] - 13https://git.io/Jv0vp
[03:44:29] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1261: switch cp[248] to use mw[4567] - 13https://git.io/Jv0vp
[03:44:31] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jv0vj
[03:44:32] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03e895b7c - Update default.vcl
[05:01:06] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2752 MB (14% inode=93%);
[06:27:31] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3084 MB (12% inode=94%);
[07:00:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0UN
[07:00:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 033dd9aca - BOT: Updating services config for wikis
[08:04:17] <Not-5a0a>	 [02puppet] 07Reception123 closed pull request 03#1257: rm cnoc from cvt mail - 13https://git.io/JvRpm
[08:04:19] <Not-5a0a>	 [02miraheze/puppet] 07Reception123 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jv0Tl
[08:04:20] <Not-5a0a>	 [02miraheze/puppet] 07The-Voidwalker 03c66bdfe - rm cnoc from cvt mail
[08:04:22] <Not-5a0a>	 [02miraheze/puppet] 07Reception123 03acb6e7e - Merge pull request #1257 from The-Voidwalker/patch-8 rm cnoc from cvt mail
[08:06:08] <RhinosF1>	 Hi Reception123
[08:10:06] <Reception123>	 hi
[09:42:19] <Not-5a0a>	 [02puppet] 07RhinosF1 opened pull request 03#1262: Remove “temp” redirect to my personal email - 13https://git.io/Jv0In
[09:43:08] <RhinosF1>	 Reception123: ^
[09:43:58] <Not-5a0a>	 [02puppet] 07Reception123 closed pull request 03#1262: Remove “temp” redirect to my personal email - 13https://git.io/Jv0In
[09:43:59] <Not-5a0a>	 [02miraheze/puppet] 07Reception123 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jv0Ic
[09:44:01] <Not-5a0a>	 [02miraheze/puppet] 07RhinosF1 03da351a6 - Remove “temp” redirect to my personal email That was still there? Long month! https://bash.toolforge.org/quip/AU7VTzhg6snAnmqnK_pc
[09:44:02] <ZppixBot>	 [ quip - Quips ] - bash.toolforge.org
[09:44:02] <Not-5a0a>	 [02miraheze/puppet] 07Reception123 03f16f493 - Merge pull request #1262 from RhinosF1/patch-4 Remove “temp” redirect to my personal email
[09:50:00] <RhinosF1>	 Thx Reception123
[10:00:21] <ZppixBot>	 Hello Carnival! If you have any questions, feel free to ask and someone should answer soon.
[10:00:45] <Carnival>	 It's Carnival season!! Almost forgot so time for some Spanish tuneees RhinosF1 Reception123 Voidwalker Zppix 
[10:00:48] <Carnival>	 Dale a tu cuerpo alegria Macarena
[10:00:49] <Carnival>	 Que tu cuerpo es pa' darle alegria y cosa buena
[10:00:54] <RhinosF1>	 grumble:
[10:00:55] <Carnival>	 Dale a tu cuerpo alegria, Macarena
[10:00:55] <Carnival>	 Hey Macarena
[10:01:03] <Carnival>	 Macarena tiene un novio que se llama
[10:01:03] <Carnival>	 Que se llama de apellido Vitorino
[10:01:09] <Carnival>	 Que en la jura de bandera el muchacho
[10:01:09] <Carnival>	 Se la dio con dos amigos
[10:01:10] <RhinosF1>	 Reception123:
[10:01:20] <Carnival>	 Macarena tiene un novio que se llama
[10:01:20] <Carnival>	 Que se llama de apellido Vitorino
[10:01:20] <Carnival>	 Y en la jura de bandera el muchacho
[10:01:20] <Carnival>	 Se la dio con dos amigos
[10:31:22] <ZppixBot>	 Hello totoafrica! If you have any questions, feel free to ask and someone should answer soon.
[10:31:24] <totoafrica>	 It's gonna take a lot to drag me away from you
[10:31:29] <totoafrica>	 There's nothing that a hundred men or more could ever do
[10:31:36] <totoafrica>	 I bless the rains down in Africa
[10:31:39] <totoafrica>	 Gonna take some time to do the things we never had (ooh, ooh)
[10:31:47] <aldnonymous>	 spam again
[10:32:13] <totoafrica>	 She's coming in, 12:30 flight
[10:32:18] <totoafrica>	 The moonlit wings reflect the stars that guide me towards salvation
[10:32:22] <totoafrica>	 I stopped an old man along the way
[10:32:31] <totoafrica>	 Hoping to find some old forgotten words or ancient melodies
[10:32:47] <totoafrica>	 He turned to me as if to say, "Hurry boy, it's waiting there for you"
[10:32:59] <totoafrica>	 It's gonna take a lot to drag me away from you
[10:33:00] <RhinosF1>	 aldnonymous: im on it
[10:33:03] <totoafrica>	 There's nothing that a hundred men or more could ever do
[10:33:07] <totoafrica>	 I bless the rains down in Africa
[10:33:09] <totoafrica>	 Gonna take some time to do the things we never had (ooh, ooh)
[10:34:15] <RhinosF1>	 aldnonymous: reported to James to hopefully get him banned
[10:37:56] <Not-5a0a>	 [02mw-config] 07holdtheline11 opened pull request 03#2897: Create HOLD THE LINE - 13https://git.io/Jv0Lz
[10:38:17] <Not-5a0a>	 [02mw-config] 07holdtheline11 opened pull request 03#2898: It's not in the way that you hold me It's not in the way you say you … - 13https://git.io/Jv0La
[10:38:28] <RhinosF1>	 Reception123:
[10:38:42] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2899: It's not in the way that you hold me you're mine, ooh It's not in the way that you came back to me It's not in the way that your love set me free It's not in the way you look or the things that you say that you'll do  Hold the line, love isn't always on time, oh oh oh Hold the line, love isn't always on time (Love isn't always on time)  Hold the
[10:38:42] <Not-5a0a>	 line, love isn't always on time (love isn't always, love isn't always on time) Hold the line, love isn't always on time Love isn't always on time Love isn't always on time Love isn't always on time, oh oh oh - 13https://git.io/Jv0LV
[10:38:43] <Reception123>	 :(
[10:39:03] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2900: hold hte lineeee - 13https://git.io/Jv0Lw
[10:39:31] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2901: htl - 13https://git.io/Jv0Lr
[10:39:46] <Not-5a0a>	 [02mw-config] 07Reception123 closed pull request 03#2897: Create HOLD THE LINE - 13https://git.io/Jv0Lz
[10:39:48] <Not-5a0a>	 [02mw-config] 07Reception123 closed pull request 03#2898: It's not in the way that you hold me It's not in the way you say you … - 13https://git.io/Jv0La
[10:40:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0LK
[10:40:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03fd7b40d - BOT: Updating services config for wikis
[11:36:34] <RhinosF1>	 Reception123: I fixed translatable templates!
[11:39:15] <Reception123>	 RhinosF1: nice :_
[11:39:16] <Reception123>	 *:)
[11:50:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0qP
[11:50:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03302e777 - BOT: Updating services config for wikis
[13:08:25] <icinga-miraheze>	 PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Mon 09 Mar 2020 01:04:49 PM GMT +0000).
[13:08:39] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0Yi
[13:08:41] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 0382a2494 - Bot: Update SSL cert for wiki.mxlinuxusers.de
[13:10:17] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2616 MB (10% inode=94%);
[13:14:25] <icinga-miraheze>	 RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Fri 22 May 2020 12:08:34 PM GMT +0000.
[13:35:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv0Ow
[13:35:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 033b63528 - BOT: Updating services config for wikis
[14:50:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv03d
[14:50:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03a172876 - BOT: Updating services config for wikis
[15:12:42] <Reception123>	 paladox: https://phabricator.miraheze.org/T5242#99859
[15:12:44] <ZppixBot>	 [ ⚓ T5242 Rename endeavourosde to archlinuxfamily ] - phabricator.miraheze.org
[16:03:23] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:04:06] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:04:13] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:04:19] <RhinosF1>	 okay icinga-miraheze 
[16:10:51] <Reception123>	 RhinosF1: icinga-miraheze can spam more than our music troll sometimes not going to lie 
[16:11:06] <RhinosF1>	 stop making me laugh so much
[16:11:21] <Reception123>	 it's true, it's a competition between them
[16:11:35] <Reception123>	 and icinga-miraheze has the advantage of not being checked by Sigyn so it has free reign to spam
[16:12:17] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:21] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:22] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:22] <icinga-miraheze>	 PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:44] <Reception123>	 RhinosF1: ^ see I told you
[16:12:46] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:51] <Reception123>	 icinga-miraheze: if you continue, I'll have to ban your range
[16:12:54] <RhinosF1>	 weeeeeee puppet spa,
[16:12:55] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:57] <RhinosF1>	 spam*
[16:13:04] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:13:15] <Reception123>	 we might complain about music troll, but the real spammer is clearly icinga-miraheze 
[16:13:27] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:13:28] <icinga-miraheze>	 PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:14:06] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:14:15] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:15:27] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[16:16:04] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[16:16:04] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[16:16:15] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[16:16:21] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[16:16:22] <icinga-miraheze>	 RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[16:16:47] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:17:04] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:17:29] <icinga-miraheze>	 RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:17:29] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:18:10] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[16:18:14] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:18:55] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:22:22] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:24:19] <Reception123>	 RhinosF1: ^ wow bring the music troll back, sounds better than icinga-miraheze moaning about inexistent issues :P
[16:25:25] <RhinosF1>	 Reception123: keep it where he cant see
[16:26:01] <paladox>	 That was me
[16:27:25] <Reception123>	 RhinosF1: meh, I doubt they'll go looking through icinga spam
[16:49:25] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3
[16:49:26] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb
[16:50:13] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[16:50:22] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3
[16:51:22] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[16:51:24] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[16:52:09] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[16:52:20] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 11 backends are healthy
[17:07:58] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb
[17:08:07] <RhinosF1>	 hi flu-pm 
[17:09:52] <flu-pm>	 Hi, RhinosF1!
[17:09:55] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:10:14] <RhinosF1>	 how can we help flu-pm 
[17:12:52] <flu-pm>	 I'm trying to upload a shell script to a wiki, but scripts are not valid file types for uploads.  I've found discussion of $wgFileExtensions, but I have no idea where I could manage that variable.  Any advice?
[17:13:59] <RhinosF1>	 flu-pm: file extensions need to be done by sysadmins but I don't think they'd allow shell scripts. We've blocked executables before.
[17:17:01] <flu-pm>	 I sort of expected that.  Do you know any "best practice" for dealing with such a cases?  Putting the script contents into to wiki as text would be a workaround, but not very user-friendly.  Hm.
[17:18:57] <RhinosF1>	 flu-pm: put it as wiki-text or host it somewhere else and link to the file on-wiki
[17:22:37] <flu-pm>	 Have to think about what works best for me.  Having the file on wiki has the advantage of getting notification whenever changes happen.  Thanks for sharing your thoughts!
[17:39:18] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[17:39:34] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[17:40:22] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. lizardfs6
[17:42:22] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy
[17:43:15] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[17:49:27] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[18:56:57] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2607:5300:205:200::17f6/cpweb
[18:58:17] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3
[18:58:56] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[18:59:19] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw3
[18:59:39] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2
[19:01:50] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[19:01:51] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:02:27] <icinga-miraheze>	 PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:03:08] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[19:03:50] <icinga-miraheze>	 PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:03:54] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18688 bytes in 1.448 second response time
[19:04:23] <paladox>	 hmm
[19:04:32] <icinga-miraheze>	 RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18688 bytes in 4.634 second response time
[19:05:15] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy
[19:05:46] <icinga-miraheze>	 RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18688 bytes in 1.473 second response time
[19:05:50] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[19:07:48] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[19:08:35] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 11 backends are healthy
[19:08:58] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[19:28:10] <icinga-miraheze>	 PROBLEM - www.bushcraftpedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.bushcraftpedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[19:28:34] <RhinosF1>	 Hi Voidwalker
[19:28:42] <Voidwalker>	 hi
[19:29:29] <RhinosF1>	 Voidwalker: can you look into the account i sent you after hispano’s message I copied to #sre discord chan
[19:29:44] <RhinosF1>	 And close the 2 really old RfCs
[19:40:57] <Voidwalker>	 I'm not sure how to close the harassment RFC, but since the only actual proposal got opposed, I don't think there is any outcome?
[19:42:09] <icinga-miraheze>	 RECOVERY - www.bushcraftpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bushcraftpedia.org' will expire on Fri 22 May 2020 06:41:58 PM GMT +0000.
[19:52:03] <RhinosF1>	 Voidwalker: no consensus i suppose
[19:53:10] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[19:55:09] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[22:10:01] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.17, 1.83, 1.05
[22:12:05] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.80, 1.47, 1.02
[22:14:50] <RhinosF1>	 Voidwalker: can u look at my cu request in your pms yesterday
[22:17:45] <paladox>	 RhinosF1 did you ask on that on a public page? CU requests should not be handled in private.
[22:21:01] <RhinosF1>	 paladox: no because the evidence is private
[22:21:37] <paladox>	 You can link to the evidence though?
[22:21:49] <RhinosF1>	 And drawing attention to certain users is not ideal
[22:22:09] <paladox>	 Yes, but asking someone to do a CU in private is scary.
[22:22:13] <RhinosF1>	 paladox: I’m sure void can handle it.
[22:22:31] <paladox>	 Yes i'm sure, but still you must follow the process like everyone else.
[22:22:31] <RhinosF1>	 Why is it scary? It’s a suspected well known LTa
[22:22:46] <RhinosF1>	 paladox: you can request cu by contacting a steward
[22:22:49] <paladox>	 Because anyone could ask for a CU for a user?
[22:23:03] <RhinosF1>	 And any steward can ignore nonsense
[22:24:11] <paladox>	 Yup, the same process apply's to you, you request it on wiki like everyone else has to.
[22:24:42] <paladox>	 Everyone has a right to know if they've been CU'd
[22:25:41] <RhinosF1>	 Everyone doesn’t have to request it on wiki
[22:26:50] <RhinosF1>	 And technically there’s no policy saying you have to be told. We’re dealing with users known to be abusive. Void has evidence from another party and I’m looking at more. If he needs to do a CU, the user will find themselves locked if true.
[22:27:21] <paladox>	 Lock != they know they've been CU'd.
[22:28:50] <RhinosF1>	 If they are innocent, we’ll discuss action after.
[22:30:33] <paladox>	 I think i'm going to bring this up, since CU requests must (and i emphasis) be reported on-wiki, for logging, so that users know if they have a CU request on them and if they've been CU'd.
[22:33:11] <RhinosF1>	 We’re dealing with a user who shouldn’t have a miraheze account and have harrased people previously. There’s never anywhere been a requirement that requests must be public. I’d rather not know every time a troll gets annoyed at me and requests a CU. Letting a user like this know they are under scrutiny will make them 100x worse.
[22:36:43] <paladox>	 I don't care who you're dealing with, i think it's gotta be public. Things should not be done in private end of, unless it's an emergency
[22:39:10] <paladox>	 We have to be transparent regardless if the socks will abuse that, because in that case, everything should be hidden including logs just so socks/spammers doin't see it. 
[22:43:47] <RhinosF1>	 To notify after, I see no issue. To notify before, you’re potentially and likely going to put victims of harrasment at risk.
[23:15:59] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2114 MB (10% inode=93%);
[23:44:55] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins]
[23:45:09] <Not-5a0a>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jv0uJ
[23:45:11] <Not-5a0a>	 [02miraheze/ManageWiki] 07paladox 037efabe0 - Force group name to be lowercased Bug: T5262
[23:45:12] <Not-5a0a>	 [02ManageWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpSns
[23:45:14] <Not-5a0a>	 [02ManageWiki] 07paladox opened pull request 03#135: Force group name to be lowercased - 13https://git.io/Jv0uU
[23:52:55] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures