[00:00:12] no need to be sorry Voltan :) [00:00:14] Since English isn't your first language, I can see why it may have been confusing. [00:00:15] Thanks for the help anyway, greatly appreciated [00:02:15] [02puppet] 07Amanda-Catherine opened pull request 03#1015: Add m.media-amazon.com to whitelist T4423 - 13https://git.io/fjELQ [00:02:18] ah xD [00:02:36] [02puppet] 07Amanda-Catherine opened pull request 03#1016: Add m.media-amazon.com to whitelist T4423 - 13https://git.io/fjEL7 [00:02:50] [02puppet] 07paladox closed pull request 03#1015: Add m.media-amazon.com to whitelist T4423 - 13https://git.io/fjELQ [00:02:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEL5 [00:02:53] [02miraheze/puppet] 07Amanda-Catherine 03d0f000a - Add m.media-amazon.com to whitelist T4423 (#1015) [00:03:37] [02puppet] 07Amanda-Catherine commented on pull request 03#1016: Add m.media-amazon.com to whitelist T4423 - 13https://git.io/fjELd [00:03:38] [02puppet] 07Amanda-Catherine closed pull request 03#1016: Add m.media-amazon.com to whitelist T4423 - 13https://git.io/fjEL7 [00:14:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-1/±1] 13https://git.io/fjELp [00:14:34] [02miraheze/puppet] 07paladox 0314b5fc6 - Update vpncloud to 1.0 [00:23:41] !log upgraded vpncloud to 1.0 accross all hosts [00:23:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:26:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjEtU [00:26:11] [02miraheze/puppet] 07paladox 03d53426a - Install vpncloud on misc4 [00:26:12] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [00:26:14] [02puppet] 07paladox opened pull request 03#1017: Install vpncloud on misc4 - 13https://git.io/fjEtT [00:27:18] [02puppet] 07paladox closed pull request 03#1017: Install vpncloud on misc4 - 13https://git.io/fjEtT [00:27:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEtI [00:27:21] [02miraheze/puppet] 07paladox 0338c4505 - Install vpncloud on misc4 (#1017) [00:27:43] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [00:27:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [00:42:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEtm [00:42:19] [02miraheze/puppet] 07paladox 030c040d7 - Base: Install net-tools Bug: T4127 [00:49:52] what does stalled mean on this request https://phabricator.miraheze.org/T4419 [00:49:54] [ ⚓ T4419 Request Extension TreeAndMenu ] - phabricator.miraheze.org [00:51:33] It means it's stalled pending a security review. [00:53:01] Ah i see it's all one item [00:57:06] About how long does that step normally take? [00:57:22] it can take a while (needs someone to look at it) [00:58:24] I suppose so. Thank you. [01:14:30] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 5060 MB (20% inode=94%); [05:06:32] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:06:41] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.009 second response time [05:06:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [05:07:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [05:07:21] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [05:07:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:07:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:07:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [05:08:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [05:08:44] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 21% [05:11:41] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 49% [05:11:49] everything is broken! Can't connect to any wiki or phabricator. Looks like the DB is offline. [05:12:20] PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:12:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [05:14:33] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4077 bytes in 0.026 second response time [05:15:17] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4077 bytes in 0.060 second response time [05:16:11] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4077 bytes in 0.023 second response time [05:16:28] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19248 bytes in 0.032 second response time [05:16:56] @System Administrators [05:17:17] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19248 bytes in 0.027 second response time [05:18:11] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19249 bytes in 0.022 second response time [05:20:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 57% [05:22:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [05:23:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 21% [05:30:04] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5806 bytes in 0.105 second response time [05:30:50] RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 348 bytes in 0.021 second response time [05:31:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:31:05] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41136 bytes in 0.134 second response time [05:31:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:31:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [05:31:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [05:32:17] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 3% [05:32:32] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [05:32:44] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 4% [05:33:39] !log removed /tmp on db4 [05:33:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:35:19] !log purge binary logs before '2019-05-31 12:00:00'; [05:35:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:24:58] PROBLEM - db4 Puppet on db4 is WARNING: WARNING: Puppet last ran 1 hour ago [06:26:30] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5431 MB (22% inode=94%); [07:46:59] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 48 seconds ago with 2 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server] [07:48:54] ^ paladox [08:31:13] [02miraheze/landing] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/fjEY8 [08:31:15] [02miraheze/landing] 07Reception123 03157c9b6 - french translation [08:31:16] [02landing] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/fxEQA [08:31:18] [02landing] 07Reception123 opened pull request 03#5: french translation - 13https://git.io/fjEY4 [08:35:55] [02landing] 07Reception123 synchronize pull request 03#5: french translation - 13https://git.io/fjEY4 [08:35:56] [02miraheze/landing] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/fjEY0 [08:35:57] [02miraheze/landing] 07Reception123 036fd1c7b - Update translations.php [08:42:04] [02landing] 07Reception123 closed pull request 03#5: french translation - 13https://git.io/fjEY4 [08:42:05] [02miraheze/landing] 07Reception123 pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/fjEYg [08:42:06] [02miraheze/landing] 07Reception123 03b622277 - Merge pull request #5 from miraheze/Reception123-patch-1 french translation [08:58:21] [02miraheze/landing] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEYi [08:58:22] [02miraheze/landing] 07Reception123 03aca3bf4 - add french to translate options [12:51:59] Everyone, so i found out the host for the relay limits the bot after so much usage so if the bot does go down like it did last time, it will be back at the first of the next month, its a crappy inconvenience i know but thats what you get for using a free host [12:54:09] hmm ok [12:55:44] Zppix: What host [12:59:28] [02landing] 07paladox deleted branch 03Reception123-patch-1 - 13https://git.io/fxEQA [12:59:29] [02miraheze/landing] 07paladox deleted branch 03Reception123-patch-1 [14:40:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEZm [14:40:11] [02miraheze/puppet] 07paladox 03890e8d3 - vpncloud: Add ipv6 address for mw* and cp* [14:42:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEZG [14:42:36] [02miraheze/puppet] 07paladox 034041cbe - vpncloud: Add ipv6 address for misc[14] [14:46:59] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:51:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEZW [14:51:20] [02miraheze/puppet] 07paladox 0384bdbe2 - Update miraheze-internal.net.erb [14:54:14] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [14:54:40] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [14:55:35] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [14:55:42] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [14:56:18] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [14:57:34] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:58:40] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:59:42] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:00:14] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:00:17] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:00:33] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:00:34] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:00:59] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:01:39] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:01:40] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:01:52] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:02:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEZ4 [15:02:20] [02miraheze/puppet] 07paladox 03e23849d - vpncloud: Fix restart command [15:02:27] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:02:33] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [15:02:34] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [15:02:55] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:02:59] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:03:37] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:03:40] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:03:51] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:23] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:30] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:04:54] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [15:11:55] !log mkdir /tmp on db4 at 14:46 [15:11:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:13:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:20:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEnN [16:20:45] [02miraheze/mw-config] 07paladox 03e19e67e - Blacklist generate-dump from * Change-Id: I244790af206a7b1f4a9b65b87db647382c69956c [16:33:45] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:14] !log disabled OATHAuth for Eduaddad (php disableOATHAuthForUser.php "Eduaddad" --wiki metawiki) [16:36:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:16:20] Do Miraheze offer anyone cloaks? [17:16:25] RhinosF1: Yes! [17:16:53] Reception123: Where do I apply? [17:16:57] RhinosF1: https://meta.miraheze.org/wiki/IRC/Group [17:16:58] [ IRC/Group - Miraheze Meta ] - meta.miraheze.org [17:17:30] RhinosF1: I see that you currently have a Wikipedia cloack [17:17:33] *cloak [17:18:20] Reception123: I don't edit WikiMedia Projects as of a few hours ago. [17:18:42] RhinosF1: well in that case if you don't want that cloak you would have to get that removed first [17:19:43] Will do that now [17:22:13] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2449 [17:29:25] Reception123, PuppyKun: https://login.miraheze.org/w/index.php?title=User:RhinosF1&diff=335&oldid=326 - glguy has said as soon as you ask for my cloak he'll drop the current one. Thanks! [17:29:26] [ Difference between revisions of "User:RhinosF1" - Miraheze Login Wiki ] - login.miraheze.org [17:30:52] He's around all day so just PM him when you're doing it. [17:31:23] RhinosF1: Thank you, I am not an IRC GC so I can't do it, you will have to wait for PuppyKun or JohnLewis [17:32:03] Yeah, That's why I pinged puppyKun. John isn't online [17:35:10] RhinosF1: what cloak would you like? Just @miraheze/RhinosF1? [17:35:27] Reception123: Yes, that's fine [17:35:45] ok :) [17:37:28] RhinosF1: :) [17:37:47] All done [18:17:19] PROBLEM - test1 php-fpm on test1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.3' [18:23:19] RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 4 processes with command name 'php-fpm7.3' [18:36:40] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [18:50:13] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEWo [18:50:15] [02miraheze/landing] 07paladox 03e2dc012 - Add status url to landing page [19:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjEWS [19:05:13] [02miraheze/services] 07MirahezeSSLBot 037d3823a - BOT: Updating services config for wikis [19:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjElA [19:55:13] [02miraheze/services] 07MirahezeSSLBot 03d36d365 - BOT: Updating services config for wikis [20:06:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjE8k [20:06:20] [02miraheze/puppet] 07paladox 038261b4b - vpncloud: Add --keepalive to send a message to keep the connection alive [20:11:42] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:13:37] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:14:14] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:14:21] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:14:33] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:14:34] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:14:40] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:14:59] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:15:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:15:35] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:15:40] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:15:41] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:15:42] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:15:46] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:15:52] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:16:18] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:16:30] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:16:55] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:18:01] Paladox: wtf [20:18:06] ? [20:18:32] Paladox: That's one big list of issues. [20:18:37] Look above [20:18:49] well it's not an issue, it will resolve on the next puppet run. [20:19:36] Paladox: Good [20:20:08] It makes itself look worse then. [20:21:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjE8O [20:21:26] [02miraheze/puppet] 07paladox 039b0964f - Update miraheze-internal.net.erb [20:22:14] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:22:20] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:22:33] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:22:34] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:22:40] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:22:58] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:34] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:40] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:41] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:42] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:45] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:02] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:18] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:54] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:25:26] That sounds good: What did your commit do Paladox? [20:25:35] And what is puppet? [20:26:14] it didn't fix anything :) also we are building our own virtual private network which encrypts everything internally. [20:26:33] puppet is https://puppet.com [20:26:33] [ Powerful infrastructure automation and delivery | Puppet ] - puppet.com [20:27:41] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:29:41] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:31:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.88, 6.85, 4.76 [20:34:14] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:34:18] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:34:22] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:34:27] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:34:33] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:34:34] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:34:41] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:34:59] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:35:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:35:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.92, 7.10, 5.35 [20:35:35] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:35:40] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:35:42] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:35:46] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:36:11] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:36:30] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:36:30] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 4999 MB (20% inode=94%); [20:36:38] Wouldn't that be best in an operations channel? [20:36:55] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [20:37:25] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.65, 6.26, 5.26 [20:40:30] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5145 MB (21% inode=94%); [20:42:14] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:42:18] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [20:42:20] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:42:33] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:42:34] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:42:40] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [20:42:54] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [20:42:59] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:43:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:43:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.79, 7.76, 6.05 [20:43:34] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:43:40] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:43:42] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:43:45] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:44:20] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:44:21] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:44:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:45:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.52, 7.49, 6.15 [20:47:25] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.30, 6.46, 5.95 [20:49:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjE8K [20:49:40] [02miraheze/puppet] 07paladox 03ebd85c4 - vpncloud: Change reload to restart This service does not support "reload". [22:30:12] RhinosF1 if your talking about the github notifications, theres been multiple ideas about moving it to a separate channel but each time theres been a “consensus” that it doesnt seem to disrupt #miraheze as it stands. [22:30:58]

Zppix: I was talking about incignia [22:31:09]

I don't mind GitHub ones [22:31:40] Honestly i think having icinga alerts in here make miraheze seem more transparent imho [22:32:13] Also rhinosf1 you dont have to switch between irc and discord to chat in here 😃 [22:32:55]

I know but I felt like it Zppix, it does add transparency but we do get quite a flood at times [22:34:35] Usually the floods are usually due to a sysadmin cough paladox cough forgets to downtime an host/service when rebooting/maint/etc otherwise its usually due to legit issues. RhinosF1 [22:35:00]

This was something with Puppet [22:35:46] RhinosF1 which was caused by maint/editing of puppet 😃 [22:36:22]

Didn't look at why. I might need someone for an import in a few minutes [22:37:16] If its a big one only sysadmins i think can [22:37:29] Unless you split it up into smaller parts [22:38:07]

Doesn't matter - I can't even get Special:Export to work - sever error on enwp [22:38:34]

I'll give it 5 [22:39:12] What error code [22:39:26]

Standard Wikimedia error page [22:39:38] It should still give an error code? [22:40:24]

https://cdn.discordapp.com/attachments/435711390544560128/584511587906355244/AGDFBTMgERmBAAAAAElFTkSuQmCC.png [22:40:39]

Nothing else [22:53:19]

Yeah, timed out enwiki [23:07:33] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.78, 1.19, 0.61 [23:09:33] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.45, 0.91, 0.58 [23:19:04] !log delete backups on bacula (for them to have a full backup in ~40mins) [23:19:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:20:25] !log upgrade a few minor packages on bacula1 [23:20:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:20:33] RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 470936 MB (98% inode=99%); [23:21:04]

Paladox: Could I use phab to track my import so I can pass it on to you as soon as it's ready [23:21:41] If you want us to import something, sure. [23:22:10] paladox: Will assign to me until I get the final list/files ready [23:22:16] ok [23:26:59]

Paladox: See https://phabricator.miraheze.org/T4436 [23:27:00] [ ⚓ T4436 Import of Twinkle and Userlinks templates to QuIRC wiki ] - phabricator.miraheze.org [23:27:20] ok