[00:00:15] <Zppix>	 Voidwalker:  kubermetes is a pain in my ass
[00:00:22] <Voidwalker>	 anyway, try changing docker-registry.tools.wmflabs.org/toollabs-python-base:latest to docker-registry.tools.wmflabs.org/toollabs-python35-base:latest
[00:07:23] <Zppix>	 now im getting line11 did not find expected key when trying to apply the change using kubectl apply -f sopelbot.yaml
[00:08:42] <Zppix>	 Voidwalker: 
[00:10:22] <Voidwalker>	 weird, that shouldn't be the line giving any issues
[00:13:14] <Zppix>	 Voidwalker:  so now what
[00:14:26] <Voidwalker>	 I dunno, maybe just revert and try installing python3 at the top of the .sh script
[00:14:40] <Zppix>	 Voidwalker:  you cant run sudo or apt 
[00:14:45] <Zppix>	 Voidwalker:  and i dont know how to revert
[00:15:20] <Voidwalker>	 well, all that you should need to do to revert is change python35 to python
[00:15:31] <Zppix>	 Voidwalker:  tried that still same error
[00:15:46] <Voidwalker>	 then you changed something you weren't supposed to :P
[00:15:55] <Zppix>	 Voidwalker:  all i changed was the image line
[00:16:23] <Voidwalker>	 well if the issue is from line 11, that's not the image line
[00:17:00] <Zppix>	 Voidwalker:  i didnt touch that line
[00:17:16] <Voidwalker>	 can I see the file?
[00:19:37] <Zppix>	 Voidwalker: https://www.irccloud.com/pastebin/8xy3ga9K/
[00:19:38] <ZppixBot>	 [ Snippet | IRCCloud ] - www.irccloud.com
[00:20:13] <Voidwalker>	 oh, line ten needs to be indented two more spaces
[00:20:39] <Zppix>	 Voidwalker:  okay i did that yaml: line 10: found character that cannot start any token
[00:20:44] <Voidwalker>	 or it was previously, and that the only difference
[00:22:42] <Voidwalker>	 try to standardize the file, but I don't know what's wrong
[00:27:25] <Zppix>	 .pyver
[00:27:25] <ZppixBot>	 I am running Python 2.7.9.
[00:27:37] <Zppix>	 okay just making sure it didnt fix itself while i dick with the .yaml
[00:39:41] <Zppix>	 Voidwalker:  i dont get it!!!
[00:40:42] <Voidwalker>	 what's wrong now?
[00:40:57] <Zppix>	 yaml: line 4: mapping values are not allowed in this context
[00:41:11] <Zppix>	 https://www.irccloud.com/pastebin/FrpQtwpY/
[00:42:36] <Zppix>	 Voidwalker:  have i mentioned i hate kubermetes
[00:47:47] <Zppix>	 Voidwalker:  pls help
[00:48:21] <Voidwalker>	 uhh, give me a bit so I can figure out how the yaml needs to be structured
[00:48:50] <Zppix>	 k
[01:05:17] <Zppix>	 lmk what you come up with Voidwalker 
[01:05:25] <Voidwalker>	 sure, currently working
[01:12:09] <Voidwalker>	 Zppix, try this: https://pastebin.com/R5PN3HTb
[01:12:10] <Voidwalker>	 disclaimer, I am not responsible for exploding the server
[01:18:20] <Zppix>	 Voidwalker:  permission denied running the starter.sh
[01:18:28] <Voidwalker>	 rip
[01:18:40] <Voidwalker>	 do you have perms set on the file properly
[01:18:59] <Zppix>	 Voidwalker:  Idk
[01:19:06] <Zppix>	 i dont know what they should be set to
[01:19:10] <Voidwalker>	 try that
[01:19:14] <Voidwalker>	 try playing with it
[01:24:18] <MH-Discord>	 <Rubyjunk> Zppix; the yaml file that doesn't work doesn't have an indent on 4 and has too much of an indent on 5, maybe you should try a yaml validator before setting up the deployments in general
[01:24:46] <Voidwalker>	 Zppix, if you want to be unsecure about it, chmod 777
[01:32:29] <Zppix>	 voidwalker: https://www.irccloud.com/pastebin/Nhi3wJgp/
[01:32:45] <Zppix>	 but it does exist
[01:35:56] <Voidwalker>	 so, python3 exists?
[01:38:14] <Zppix>	 Voidwalker:  it did it doesnt now
[01:38:45] <Voidwalker>	 weird, what does ls -alh | grep python show you?
[01:39:21] <Zppix>	 nothing
[01:39:50] <Voidwalker>	 huh
[01:40:42] <Voidwalker>	 you are doing the ls in the project /bin right?
[01:42:42] <Zppix>	 Voidwalker:  im recreating the env
[01:43:36] <Zppix>	 Voidwalker:  okay so its not the env its idk
[01:43:45] <Zppix>	 Voidwalker:  cause everything is there
[01:44:18] <Voidwalker>	 whack
[01:52:37] <Zppix>	 .help
[01:52:41] <ZppixBot>	 Hang on, I'm creating a list.
[01:52:41] <ZppixBot>	 I've posted a list of my commands at https://clbin.com/TxUec - You can see more info about any of these commands by doing .help <command> (e.g. .help time)
[01:52:50] <Zppix>	 Voidwalker:  holy shit
[01:52:52] <Zppix>	 I fixed it
[01:52:56] <Zppix>	 .pyver
[01:52:56] <ZppixBot>	 I am running Python 3.5.3.
[01:53:17] <Zppix>	 ZppixBot:  your a pain my ass
[01:54:15] <Zppix>	 .weather Chicago
[01:54:16] <ZppixBot>	 Chicago, US: 23°C (73°F), Clouds, Humidity: 83%, Light air 1.7m/s (↙)
[01:54:32] <Zppix>	 .version
[01:54:32] <ZppixBot>	 Zppix: Sopel v. 6.6.9
[01:55:11] <Zppix>	 I'm never touching kubermetes config ever again
[01:55:38] <Voidwalker>	 lol
[01:56:04] <Zppix>	 i dont care if python 3 gets depreicated its staying
[02:07:57] <Zppix>	 Voidwalker:  lmk when/if you get toolforge shell access and ill add you as a maintainer
[02:08:10] <Voidwalker>	 ok
[02:08:30] <Voidwalker>	 should get an email, right?
[02:08:46] <Zppix>	 Voidwalker:  you should since you get a user right change on wikitech
[02:09:06] <Zppix>	 Voidwalker:  you know how to apply right?
[02:09:19] <Voidwalker>	 yeah, already sent in
[02:09:24] <Zppix>	 ok
[02:09:29] <Voidwalker>	 on tooladmin
[02:09:49] <Zppix>	 Voidwalker:  ah yes i can see it
[02:11:00] <Zppix>	 Voidwalker:  ill talk to -cloud tomorrow to see if i can expedite it
[02:11:33] <Voidwalker>	 cool, I've added that to my autojoin list, do you recommend any others?
[02:11:57] <Zppix>	 Voidwalker:  for irc, no, they do have a mailing list, its cloud-l iirc
[02:12:01] <Zppix>	 or cloud-announce
[02:12:24] <Voidwalker>	 mhm, maybe should join more mailing lists
[02:14:35] * Voidwalker adds that to a todo
[02:14:53] <Zppix>	 once added to maintainer also you will be forwarded tools.zppixbot[at]tools.wmflabs.org
[02:15:15] <Voidwalker>	 cool
[02:15:29] <Voidwalker>	 don't imagine that gets much traffic though :)
[02:15:45] <Zppix>	 Voidwalker:  mostly cron errors when they do maint on toolforge
[02:16:05] <Voidwalker>	 ah, well, mail filters will solve that ;)
[02:16:23] <Zppix>	 Voidwalker:  thats rare though usually once every couple months
[02:16:29] <Zppix>	 Voidwalker:  so a filter may just be more work
[02:16:45] <Voidwalker>	 yeah, but a one time commitment for the first time it happens
[02:17:17] <Zppix>	 Voidwalker:  all the cron **supposed** to do is auto git pull the github repo 
[02:17:43] <Zppix>	 and the bot's website
[02:17:52] <Zppix>	 tools.wmflabs.org/zppixbot
[02:21:54] <Zppix>	 Voidwalker:  also #ZppixBot is its irc chan
[02:22:50] <Voidwalker>	 building up more autojoins :P
[03:00:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOUm
[03:00:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03d83d7d8 - BOT: Updating services config for wikis
[03:19:07] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[03:19:09] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2
[03:21:07] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[03:21:09] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[04:04:41] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown)
[04:05:21] <icinga-miraheze>	 PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown)
[04:05:29] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 247 failures. Last run 2 minutes ago with 247 failures. Failed resources (up to 3 shown)
[04:05:44] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 25 failures. Last run 3 minutes ago with 25 failures. Failed resources (up to 3 shown)
[04:06:06] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown)
[04:06:07] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 235 failures. Last run 3 minutes ago with 235 failures. Failed resources (up to 3 shown)
[04:06:14] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 230 failures. Last run 3 minutes ago with 230 failures. Failed resources (up to 3 shown)
[04:06:49] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 249 failures. Last run 3 minutes ago with 249 failures. Failed resources (up to 3 shown)
[04:07:07] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 284 failures. Last run 3 minutes ago with 284 failures. Failed resources (up to 3 shown): File[wiki.jacksonheights.nyc],File[wiki.jacksonheights.nyc_private],File[wiki.besuccess.com],File[wiki.besuccess.com_private]
[04:07:32] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 186 failures. Last run 3 minutes ago with 186 failures. Failed resources (up to 3 shown): File[wiki.drones4nature.info_private],File[savage-wiki.com],File[savage-wiki.com_private],File[wikipuk.cl]
[04:13:21] <icinga-miraheze>	 RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[04:13:29] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[04:13:43] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[04:14:04] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[04:14:06] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:14:13] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[04:14:40] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:14:54] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:15:05] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[04:15:35] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[05:20:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOkP
[05:20:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 030b2546b - BOT: Updating services config for wikis
[06:25:34] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3091 MB (12% inode=94%);
[09:15:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOqm
[09:15:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 031df80ea - BOT: Updating services config for wikis
[10:00:54] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.78, 1.77, 1.52
[10:02:55] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.57, 1.69, 1.52
[10:06:54] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.77, 1.81, 1.60
[10:10:54] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.27, 2.04, 1.74
[10:12:54] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.98, 1.95, 1.74
[10:18:54] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.43, 1.65, 1.66
[12:19:02] <ZppixBot>	 Hello milvagox! If you have any questions feel free to ask and someone should answer soon.
[16:05:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOZs
[16:05:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 0302e32cb - BOT: Updating services config for wikis
[16:51:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.77, 6.74, 5.36
[16:52:03] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.98, 1.68, 1.55
[16:53:57] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.31, 1.59, 1.54
[17:00:41] <Not-f817>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOny
[17:00:42] <Not-f817>	 [02miraheze/ManageWiki] 07translatewiki 034b05dc3 - Localisation updates from https://translatewiki.net.
[17:00:43] <ZppixBot>	 [ Main page - translatewiki.net ] - translatewiki.net.
[17:03:27] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.93, 1.83, 1.65
[17:04:46] <RhinosF1>	 paladox, JohnLewis: any issue with me updating with that change? ^
[17:05:07] <paladox>	 Nope.
[17:05:17] * RhinosF1 will do then in a mo
[17:07:29] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.84, 7.44, 7.08
[17:09:10] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.10, 1.85, 1.70
[17:11:04] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.82, 1.81, 1.69
[17:12:15] <Not-f817>	 [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JeOnF
[17:12:17] <Not-f817>	 [02miraheze/mediawiki] 07RhinosF1 0310bf48c - Updating ManageWiki
[17:12:58] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.38, 1.66, 1.65
[17:13:08] <RhinosF1>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki (mw*)
[17:13:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:13:39] <RhinosF1>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki (mw*)
[17:13:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:18:15] <RhinosF1>	 !log update managewiki on test1 due to puppet being off
[17:18:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:18:29] <RhinosF1>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki (test1 and mw* (again))
[17:18:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:18:48] <RhinosF1>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki (test1 and mw* (again))
[17:18:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:18:54] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.28, 2.00, 1.77
[17:19:31] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.40, 5.92, 6.53
[17:20:54] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.62, 1.83, 1.73
[17:22:55] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.50, 1.69, 1.69
[17:31:30] <Not-f817>	 [02ssl] 07RhinosF1 opened pull request 03#218: Create wiki.tallfreeguys.com.cert - 13https://git.io/JeOce
[17:31:41] <Not-f817>	 [02ssl] 07RhinosF1 commented on pull request 03#218: Create wiki.tallfreeguys.com.cert - 13https://git.io/JeOcv
[17:33:03] <RhinosF1>	 !log generated ssl cert for wiki.tallfreeguys.com on mw1
[17:33:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:33:20] <Not-f817>	 [02ssl] 07Reception123 commented on pull request 03#218: Create wiki.tallfreeguys.com.cert - 13https://git.io/JeOcf
[17:35:19] <Not-f817>	 [02ssl] 07RhinosF1 synchronize pull request 03#218: Create wiki.tallfreeguys.com.cert - 13https://git.io/JeOce
[17:35:49] <Not-f817>	 [02ssl] 07RhinosF1 synchronize pull request 03#218: Create wiki.tallfreeguys.com.cert - 13https://git.io/JeOce
[17:36:06] <Not-f817>	 [02ssl] 07RhinosF1 commented on pull request 03#218: Create wiki.tallfreeguys.com.cert - 13https://git.io/JeOcU
[17:36:40] <RhinosF1>	 Reception123: ^
[17:40:00] <Not-f817>	 [02ssl] 07Reception123 closed pull request 03#218: Create wiki.tallfreeguys.com.cert - 13https://git.io/JeOce
[17:40:01] <Not-f817>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeOct
[17:40:03] <Not-f817>	 [02miraheze/ssl] 07RhinosF1 03eec2a8a - Create wiki.tallfreeguys.com.cert (#218) * Create wiki.tallfreeguys.com.cert * Update certs.yaml * Rename wiki.tallfreeguys.com.cert to wiki.tallfreeguys.com.crt
[17:41:34] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%);
[17:42:14] <RhinosF1>	 .in 10 mins puppet
[17:42:15] <ZppixBot>	 RhinosF1: Okay, I will set the reminder for: 2019-09-16 - 18:52:15BST
[17:52:15] <ZppixBot>	 RhinosF1: puppet
[17:54:06] <icinga-miraheze>	 PROBLEM - wiki.tallfreeguys.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[17:54:20] <RhinosF1>	 Reception123: huh ^
[17:54:39] <RhinosF1>	 Reception123: I made a mistake
[17:55:29] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.tallfreeguys.com_private]
[17:55:42] <Not-f817>	 [02ssl] 07RhinosF1 opened pull request 03#219: Rename wiki.tallfreeguys.com.crt to wiki.tallguysfree.com.crt - 13https://git.io/JeOcw
[17:56:10] <Reception123>	 RhinosF1: my fault
[17:56:11] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.tallfreeguys.com_private]
[17:56:16] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.tallfreeguys.com_private]
[17:56:24] <Not-f817>	 [02ssl] 07RhinosF1 synchronize pull request 03#219: Rename wiki.tallfreeguys.com.crt to wiki.tallguysfree.com.crt - 13https://git.io/JeOcw
[17:56:39] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.tallfreeguys.com_private]
[17:56:42] <RhinosF1>	 Reception123: ^
[17:57:04] <RhinosF1>	 It's fine, I kept getting the site name confused, ssl cert is right just not the repo
[17:57:04] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.tallfreeguys.com_private]
[17:57:30] <Reception123>	 RhinosF1: it seems the key wasn't added
[17:57:46] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.tallfreeguys.com_private]
[17:57:48] <Not-f817>	 [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS
[17:57:50] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JeOcr
[17:57:51] <Not-f817>	 [02miraheze/puppet] 07paladox 03b20fef9 - prometheus: Set --storage.tsdb.retention to 90d
[17:57:53] <Not-f817>	 [02puppet] 07paladox opened pull request 03#1085: prometheus: Set --storage.tsdb.retention to 90d - 13https://git.io/JeOco
[17:57:57] <Not-f817>	 [02ssl] 07Reception123 closed pull request 03#219: Rename wiki.tallfreeguys.com.crt to wiki.tallguysfree.com.crt - 13https://git.io/JeOcw
[17:57:59] <Not-f817>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JeOcK
[17:58:00] <Not-f817>	 [02miraheze/ssl] 07RhinosF1 03167a2d6 - Rename wiki.tallfreeguys.com.crt to wiki.tallguysfree.com.crt (#219) * Rename wiki.tallfreeguys.com.crt to wiki.tallguysfree.com.crt * Update certs.yaml
[17:58:01] <Reception123>	 RhinosF1: ah so both our fault
[17:58:27] <RhinosF1>	 Reception123: at least icinga-miraheze shouted at me
[17:58:32] <RhinosF1>	 And I spotted it
[17:58:37] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/JeOci
[17:58:38] <Not-f817>	 [02miraheze/puppet] 07paladox 03ae066be - Create prometheus.default.conf
[17:58:40] <Not-f817>	 [02puppet] 07paladox synchronize pull request 03#1085: prometheus: Set --storage.tsdb.retention to 90d - 13https://git.io/JeOco
[17:59:01] <Not-f817>	 [02puppet] 07paladox closed pull request 03#1085: prometheus: Set --storage.tsdb.retention to 90d - 13https://git.io/JeOco
[17:59:02] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeOcP
[17:59:04] <Not-f817>	 [02miraheze/puppet] 07paladox 03fe1f609 - prometheus: Set --storage.tsdb.retention to 90d (#1085) * prometheus: Set --storage.tsdb.retention to 90d * Create prometheus.default.conf
[17:59:18] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS
[17:59:19] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3
[18:02:52] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[18:03:36] <Not-f817>	 [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS
[18:03:38] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JeOc1
[18:03:39] <Not-f817>	 [02miraheze/puppet] 07paladox 03618588d - Prometheus: Set query.max-samples to 10000000
[18:03:41] <Not-f817>	 [02puppet] 07paladox opened pull request 03#1086: Prometheus: Set query.max-samples to 10000000 - 13https://git.io/JeOcM
[18:03:58] <Not-f817>	 [02puppet] 07paladox commented on pull request 03#1086: Prometheus: Set query.max-samples to 10000000 - 13https://git.io/JeOcD
[18:04:08] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:04:10] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:04:37] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:04:45] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:05:04] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOcy
[18:05:06] <Not-f817>	 [02miraheze/puppet] 07paladox 036531b84 - Prometheus: Set storage.tsdb.max-block-duration to 24h
[18:05:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOcS
[18:05:13] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 0323e44ad - BOT: Updating services config for wikis
[18:05:42] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:15:08] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeOcF
[18:15:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03938e77b - BOT: Updating services config for wikis
[18:16:20] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.77, 7.17, 5.97
[18:16:52] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.93, 1.65, 1.46
[18:18:20] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.09, 6.77, 5.97
[18:18:47] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.26, 1.52, 1.44
[19:00:44] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.10, 5.89, 3.87
[19:02:44] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.98, 6.01, 4.17
[19:09:29] <MH-Discord>	 <RhinosF1> Paladox: The wiki you've just fixed is having issues with file uploads
[19:09:41] <Voidwalker>	 see also: https://meta.miraheze.org/wiki/Stewards%27_noticeboard#Not_allowing_me_to_upload_files_on_https:.2F.2Fnomanssky.miraheze.org
[19:09:42] <ZppixBot>	 [ Stewards' noticeboard - Miraheze Meta ] - meta.miraheze.org
[19:09:46] <MH-Discord>	 <RhinosF1> getting the error "Could not open lock file for "mwstore://local-backend/local-public/8/8a/TGF_wiki_logo.svg". Make sure your upload directory is configured correctly and your web server has permission to write to that directory." when trying to upload a logo
[19:09:47] <MH-Discord>	 <Goliath Giant> (it had this issue before the domain fix)
[19:10:39] <paladox>	 !log root@mw1:/mnt/mediawiki-static# mkdir -p tallguysfreewiki
[19:10:40] <paladox>	 RhinosF1 fixed
[19:10:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:12:18] <RhinosF1>	 paladox: why's it keep happening and see voids link
[19:12:42] <paladox>	 Did this wiki have creation error's when it was created?
[19:12:52] <paladox>	 The problems we've been having for a while?
[19:13:21] <RhinosF1>	 paladox: afaik yes, it's always been an issue. And yes I've reported multiple to you
[19:13:36] <RhinosF1>	 See voids link as well that's another wiki with the same issue
[19:13:43] <paladox>	 yup
[19:13:50] <RhinosF1>	 Fix?
[19:13:57] <MH-Discord>	 <Goliath Giant> Still experiencing the issue.
[19:14:05] <paladox>	 !log root@mw1:/mnt/mediawiki-static# mkdir -p nomansskywiki
[19:14:12] <paladox>	 RhinosF1 you can also fix it :)
[19:14:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:14:23] <paladox>	 you have perms to create folders in that dir
[19:14:32] <MH-Discord>	 <RhinosF1> @Goliath Giant give it a few minutes
[19:14:39] <MH-Discord>	 <Goliath Giant> alright. I'll give it 10 🙂
[19:14:50] <MH-Discord>	 <RhinosF1> Paladox: I'm mobile atm
[19:15:00] <paladox>	 ok
[19:15:14] <RhinosF1>	 paladox: is there anyway to ensure it's fixed for all wikis?
[19:15:15] <paladox>	 @Goliath Giant your still having issues?
[19:15:41] <paladox>	 You could write a script and have it check for all dir's (and match it against a list from all.dblist).
[19:15:46] <MH-Discord>	 <Goliath Giant> yes Paladox still experiencing the problem but going to wait a few mins to see if it's fixed
[19:16:04] <paladox>	 @Goliath Giant it should have resolved it's self instantly
[19:16:15] <paladox>	 oh!
[19:16:17] <paladox>	 damn
[19:16:19] <paladox>	 need to chown it
[19:16:40] <MH-Discord>	 <Goliath Giant> ¯_(ツ)_/¯
[19:16:53] <paladox>	 @Goliath Giant try now!
[19:17:19] <MH-Discord>	 <Goliath Giant> works!!!
[19:17:30] <RhinosF1>	 paladox: why does it even keep happening?
[19:17:31] <MH-Discord>	 <Goliath Giant> thank you so much Paladox
[19:17:36] <MH-Discord>	 <Goliath Giant> and Rhinos
[19:19:24] <paladox>	 RhinosF1 because of the job queue.
[19:19:38] <paladox>	 It's the same issue that affects the Main Page generation
[19:19:42] <RhinosF1>	 paladox: that's a bloody curse
[20:46:49] <wm-bot5>	 http://wm-bot.wmflabs.org/dump/%23miraheze.htm
[20:46:49] <RhinosF1>	 @info
[20:46:51] <ZppixBot>	 [ #miraheze ] - wm-bot.wmflabs.org
[21:06:54] <paladox>	 !log ban req.http.Host == tallguysfree.miraheze.org - cp4
[21:06:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:07:40] <paladox>	 !log ban req.http.Host == tallguysfree.miraheze.org - cp2/cp3
[21:07:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:34:18] <Not-f817>	 [02ssl] 07RhinosF1 opened pull request 03#220: Add redirect for tallguysfree wiki - 13https://git.io/JeO8l
[21:35:16] <Not-f817>	 [02ssl] 07paladox closed pull request 03#220: Add redirect for tallguysfree wiki - 13https://git.io/JeO8l
[21:35:18] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeO88
[21:35:19] <Not-f817>	 [02miraheze/ssl] 07RhinosF1 0362e183e - Add redirect for tallguysfree wiki (#220) I think this should work
[21:41:46] <RhinosF1>	 !log rhinos@mw1:/mnt/mediawiki-static$ sudo -u www-data mkdir -p milvagoxwiki
[21:41:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:44:41] <Not-f817>	 [02ssl] 07RhinosF1 opened pull request 03#221: .org to .com for tallguysfree - 13https://git.io/JeO82
[21:45:06] <Not-f817>	 [02ssl] 07RhinosF1 synchronize pull request 03#221: .org to .com for tallguysfree - 13https://git.io/JeO82
[21:45:21] <RhinosF1>	 paladox: ^
[21:45:36] <Not-f817>	 [02ssl] 07paladox closed pull request 03#221: .org to .com for tallguysfree - 13https://git.io/JeO82
[21:45:38] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeO8a
[21:45:39] <Not-f817>	 [02miraheze/ssl] 07RhinosF1 0325339a9 - .org to .com for tallguysfree (#221) * Add redirect for tallguysfree wiki I think this should work * .org to .com
[21:45:49] <RhinosF1>	 thx
[21:54:27] <RhinosF1>	 .at 06:30 -staff
[21:54:27] <ZppixBot>	 RhinosF1: Okay, I will set the reminder for: 2019-09-17 - 05:29:59UTC
[22:07:32] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.18, 1.72, 1.50
[22:09:31] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.56, 1.60, 1.48
[22:24:27] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.80, 1.63, 1.51
[22:34:24] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.18, 1.63, 1.58
[22:42:22] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.78, 1.68, 1.59
[22:44:21] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.20, 1.55, 1.55
[22:47:59] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.08, 5.51, 3.82
[22:49:26] <Zppix>	 paladox: i think they want attention(re icinga-miraheze )
[22:49:40] <Zppix>	 xD
[22:49:42] <paladox>	 Heh
[22:49:58] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.15, 6.67, 4.46
[22:50:04] <Zppix>	 paladox: i feel like mw* are really just a bunch of whiny baby
[22:50:28] <paladox>	 Lol
[23:00:06] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.05, 7.45, 6.19
[23:00:29] <Zppix>	 Its okay icinga-miraheze the world isnt ending (yet)
[23:02:04] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.49, 6.04, 5.81
[23:05:15] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 2.00, 1.58, 1.45
[23:07:15] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.38, 1.50, 1.44
[23:10:00] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.95, 7.37, 6.32
[23:27:16] <Zppix>	 paladox:  https://meta.miraheze.org/w/index.php?title=User_talk:SA_13_Bro&curid=14357&diff=83246&oldid=82900
[23:27:18] <ZppixBot>	 [ Difference between revisions of "User talk:SA 13 Bro" - Miraheze Meta ] - meta.miraheze.org
[23:28:14] <paladox>	 There is, I’m mobile atm
[23:28:26] <paladox>	 JohnLewis: maybe you can take a look please ^?
[23:29:12] <JohnLewis>	 don't have time to do a database rena,e
[23:30:42] <Zppix>	 JohnLewis:  thats when you delegate it to the next sysadmin so it can be delegated again
[23:30:53] <JohnLewis>	 Phabricator
[23:31:00] <JohnLewis>	 simples, point them there :)
[23:31:08] <Zppix>	 but thats not as fun :(
[23:33:59] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 2.56, 5.60, 7.47
[23:34:03] <Voidwalker>	 wonder what's up with that profanity
[23:34:25] <Zppix>	 Voidwalker:  because its not just a site okay :P
[23:34:46] <Zppix>	 I actually didnt even notice the profanity until you said that Voidwalker 
[23:36:15] <Voidwalker>	 I was actually kinda looking for it once I realized who posted it
[23:36:23] <Voidwalker>	 I see it a lot in -feed
[23:36:59] <Zppix>	 lol
[23:37:59] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 2.46, 3.93, 6.35
[23:38:02] <Voidwalker>	 I kinda want to grep over my -feed logs and see how many of his edits include profanity in the summary
[23:38:11] <Voidwalker>	 I'd be willing to bet a decent percent
[23:38:34] <Zppix>	 Voidwalker: s/decent/indecent 
[23:38:34] <ZppixBot>	 Zppix thinks Voidwalker meant to say: I'd be willing to bet a indecent  percent
[23:38:35] <Zppix>	 xD
[23:39:14] <Voidwalker>	 :)