[00:01:18] !log ./upload.sh /srv/mediawiki/dblist/all.dblist on test1 (this will upload /mnt/mediawiki-static wiki containers to the container created. [00:01:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:02:56] im going to need john help, it's only created 2542 containers [00:03:01] PROBLEM - Current Load on test1 is CRITICAL: CRITICAL - load average: 6.65, 3.05, 1.44 [00:03:03] where as there's 2,700+ wiki's [00:11:01] PROBLEM - Current Load on test1 is WARNING: WARNING - load average: 1.35, 1.96, 1.57 [00:13:01] RECOVERY - Current Load on test1 is OK: OK - load average: 0.68, 1.48, 1.44 [00:19:52] PROBLEM - Puppet on swift1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [00:21:32] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±7] 13https://git.io/fVEFn [00:21:34] [02miraheze/rings] 07paladox 036f0ecd6 - Update ring files [00:23:50] RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:26:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fVaUe [00:26:14] [02miraheze/puppet] 07paladox 03cdf5b78 - Update init.pp [00:38:41] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:46:41] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5361 [00:48:12] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5304 [01:26:42] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:39:51] PROBLEM - Host bacula1 is DOWN: PING CRITICAL - Packet loss = 100% [01:42:11] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [01:42:41] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [01:44:16] uh [01:44:18] bacula? [01:44:36] i have no access to bacula [02:08:41] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): [02:25:12] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [05:03:41] !log opened support ticket with backupsy (bacula1.mh.o host) [05:03:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:06:38] WSL is pretty cool [05:09:16] Backupsy already replied saying they're investigating [05:30:31] RECOVERY - Host bacula1 is UP: PING OK - Packet loss = 0%, RTA = 99.50 ms [05:32:56] !log closed ticket with bacula after acknowledging response and confirming i could ping and ssh the host [05:32:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:33:22] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [05:41:20] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:02:05] 01-Jul 03:41 bacula1-dir JobId 27: Error: bsock.c:189 gethostbyname() for host "nfs1.miraheze.org" failed: ERR=Temporary failure in name resolution [06:10:12] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5323 [06:10:42] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5296 [06:20:13] PROBLEM - JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [06:22:11] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 4732 [07:06:12] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [07:06:42] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [11:54:14] Hello khalella! If you have any questions feel free to ask and someone should answer soon. [11:57:41] can miraheze support a wiki serving 20,000 hits per day ? [11:59:36] khalella: Yes, I would think so. We have good cache proxies, and are considering upgrading one of them [11:59:39] (or even getting a new one) [11:59:45] So I wouldn't see an issue with that [11:59:54] can miraheze support Extension:SimpleMathJax ? [12:02:08] khalella: We don't currently have it, but if it passes review we could install it on your wiki [12:02:16] Or we also have https://www.mediawiki.org/wiki/Extension:Math [12:02:17] Title: [ Extension:Math - MediaWiki ] - www.mediawiki.org [12:02:30] i want to serve daily full XML exports of the wiki, which are over 500mb. is that possible ? [12:03:24] khalella: Yes, I think it would be possible as well [12:05:17] will i have shell access to upload dumps and run /maintenance scripts [12:09:47] i also have external images allowed, and a ton of weird file formats allowed in the uploads, including mp3, wav, xls [12:13:53] khalella: shell access is only for sysadmins, but we are quite a few so if you make a request on Phabricator (our request tracking system) it is done pretty fast [12:14:00] khalella: yeah, file formats work too [12:24:42] cool [12:26:23] khalella: If you have any more questions or need help setting up your wiki, feel free to ask me at any time :) [12:58:55] today's random protip: https://phabricator.wikimedia.org/T188029#4003012 "Pro tip: setting mediawiki on read only actually doesn't set it as read only. Go and complain the extension support" [12:58:56] Title: [ ⚓ T188029 Move labswiki database to m5 ] - phabricator.wikimedia.org [13:54:42] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5324 [13:55:12] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5304 [14:42:11] PROBLEM - karagash.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate karagash.info expires in 15 day(s) (Tue 17 Jul 2018 02:39:10 PM GMT +0000). [14:42:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fwlB5 [14:42:23] [02miraheze/ssl] 07MirahezeSSLBot 03af55165 - Bot: Update SSL cert for karagash.info [14:49:11] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [14:49:41] PROBLEM - saveta.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate saveta.org expires in 15 day(s) (Tue 17 Jul 2018 02:46:36 PM GMT +0000). [14:49:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fwlRU [14:49:54] [02miraheze/ssl] 07MirahezeSSLBot 0392cc4e4 - Bot: Update SSL cert for saveta.org [14:50:43] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [14:51:41] RECOVERY - saveta.org - LetsEncrypt on sslhost is OK: OK - Certificate saveta.org will expire on Sat 29 Sep 2018 01:49:47 PM GMT +0000. [14:52:11] RECOVERY - karagash.info - LetsEncrypt on sslhost is OK: OK - Certificate karagash.info will expire on Sat 29 Sep 2018 01:42:17 PM GMT +0000. [14:52:42] PROBLEM - www.saveta.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate www.saveta.org expires in 15 day(s) (Tue 17 Jul 2018 02:48:54 PM GMT +0000). [14:52:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fwlRt [14:52:54] [02miraheze/ssl] 07MirahezeSSLBot 035545c31 - Bot: Update SSL cert for www.saveta.org [14:58:04] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate www.lab612.at expires in 15 day(s) (Tue 17 Jul 2018 02:54:02 PM GMT +0000). [14:58:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fwlRC [14:58:17] [02miraheze/ssl] 07MirahezeSSLBot 03b564c99 - Bot: Update SSL cert for www.lab612.at [15:01:56] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate www.lab612.at will expire on Sat 29 Sep 2018 01:58:11 PM GMT +0000. [15:02:36] RECOVERY - www.saveta.org - LetsEncrypt on sslhost is OK: OK - Certificate www.saveta.org will expire on Sat 29 Sep 2018 01:52:48 PM GMT +0000. [15:24:10] khalella: btw something that was missed [15:24:33] if you want daily dumps (or any automated dumps) and your wiki is public, we can just configure it to automatically upload regular dumps to static.miraheze.org/dumps [15:25:08] although we would have to check the load the mw* servers go on while doing the dump. we have wikis with dumps much larger than 500mb i think but daily dumps would be a lot. [15:25:42] dumps will be interesting as we'll be losing NFS soon [15:47:23] oh [15:47:30] doesnt swift basically work like that though? [15:48:28] also [15:48:34] MirahezeSSLBot is like the best thing since sliced bread [15:48:59] :P [16:47:09] i don't mind hosting the dumps [16:47:37] its not hosting that's the problem (yet) [16:48:10] I think the server load from re-creating them and the temporary transfer to another server would be the more likely issues. [16:48:14] most of the dump is files. the pages are around 50mb [16:48:28] do you expect the content (especially files) to change so much that you need daily dumps? [16:48:42] most of the wikis with automatic backups have weekly, bi-weekly, or monthly backups [16:48:57] (although most of them likely don't get 20k hits per day. i think only two of our wikis do) [17:13:03] no probably not [18:11:51] .weather [18:11:53] Yahoo! Weather - Portland, ME, US: Partly Cloudy, 28°C (82°F), Humidity: 67%, Light breeze 3.1m/s (←) [18:22:07] Voidwalker: fancy being awesome and doing some mw-config tasks please? :P [18:24:55] sure, I'm handling some local security stuff rn, but that shouldn't take much longer [18:25:12] :D [18:25:38] also "Deleting 5378 emails..." is very satisfying [18:32:10] [02mw-config] 07The-Voidwalker opened pull request 03#2276: NS_DATA for kirarafantasiawiki fixes T3298 - 13https://git.io/fwdKx [18:32:59] [02mw-config] 07JohnFLewis closed pull request 03#2276: NS_DATA for kirarafantasiawiki fixes T3298 - 13https://git.io/fwdKx [18:33:00] [02miraheze/mw-config] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fwF0l [18:33:02] [02miraheze/mw-config] 07The-Voidwalker 038705d7d - NS_DATA for kirarafantasiawiki fixes T3298 [18:33:03] [02miraheze/mw-config] 07JohnFLewis 03c36a557 - Merge pull request #2276 from The-Voidwalker/patch-35 NS_DATA for kirarafantasiawiki fixes T3298 [18:33:14] [02services] 07The-Voidwalker opened pull request 03#141: xxcircle to services T3296 - 13https://git.io/fwFzo [18:33:51] [02services] 07JohnFLewis closed pull request 03#141: xxcircle to services T3296 - 13https://git.io/fwFzo [18:33:52] [02miraheze/services] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fwbfN [18:33:54] [02miraheze/services] 07The-Voidwalker 037282df2 - xxcircle to services T3296 [18:33:55] [02miraheze/services] 07JohnFLewis 0328a2926 - Merge pull request #141 from The-Voidwalker/patch-7 xxcircle to services T3296 [18:36:36] changed the settings (ext-visualeditor, setting-wmgWikiLicense) for "xxcirclewiki"; odd I didn't touch WikiLicense [18:36:59] it adds it to the database [18:37:49] need to change populateWikiSettings to add defaults to prevent it [18:47:16] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [18:51:16] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fwAxx [18:51:17] [02miraheze/mw-config] 07JohnFLewis 03c9d7d12 - update AB config [18:51:56] Voidwalker: the task you just resolved is resolved by the above [18:52:17] (well 8 minutes ago) [18:52:26] oh, alright [18:52:55] wasn't too sure what was going where [18:54:13] stuff like https://phabricator.miraheze.org/T3138 annoys me [18:54:14] Title: [ ⚓ T3138 Request Forum Administrator privileges for Credco.miraheze.org ] - phabricator.miraheze.org [18:54:24] that's been open like 5 weeks now and is a 1 minute job [18:56:12] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/freTT [18:56:14] [02miraheze/mw-config] 07JohnFLewis 039a814fa - addgroups for wikiforum [19:00:08] https://phabricator.miraheze.org/T2867 4 months nearly and not even a reply [19:00:10] Title: [ ⚓ T2867 Inserting an uploaded image in visual editor hangs indefinitely when clicking 'Insert' ] - phabricator.miraheze.org [19:11:57] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.86, 1.57, 1.12 [19:13:55] RECOVERY - Current Load on misc1 is OK: OK - load average: 1.68, 1.58, 1.18 [19:16:04] [02mw-config] 07The-Voidwalker opened pull request 03#2277: kyivstarwiki groups fixes T3289 - 13https://git.io/frYYN [19:18:03] [02mw-config] 07JohnFLewis closed pull request 03#2277: kyivstarwiki groups fixes T3289 - 13https://git.io/frYYN [19:18:04] [02miraheze/mw-config] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/frYOe [19:18:06] [02miraheze/mw-config] 07The-Voidwalker 037b42d90 - kyivstarwiki groups fixes T3289 [19:18:07] [02miraheze/mw-config] 07JohnFLewis 03b86f006 - Merge pull request #2277 from The-Voidwalker/patch-36 kyivstarwiki groups fixes T3289 [19:54:41] [02mw-config] 07The-Voidwalker opened pull request 03#2278: olegcinemawiki config fixes T3263 - 13https://git.io/frzqC [19:57:20] [02mw-config] 07JohnFLewis closed pull request 03#2278: olegcinemawiki config fixes T3263 - 13https://git.io/frzqC [19:57:22] [02miraheze/mw-config] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±4] 13https://git.io/frzq8 [19:57:23] [02miraheze/mw-config] 07The-Voidwalker 03b7a8fc3 - olegcinemawiki config fixes T3263 [19:57:25] [02miraheze/mw-config] 07JohnFLewis 03e1e7ce6 - Merge pull request #2278 from The-Voidwalker/patch-37 olegcinemawiki config fixes T3263 [19:57:36] Voidwalker: there you go :P [19:57:52] :) [20:44:28] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/frxwG [20:44:29] [02miraheze/puppet] 07JohnFLewis 038cc78a3 - drop mw2 as a jobrunner [20:46:56] PROBLEM - JobRunner Service on mw2 is CRITICAL: PROCS CRITICAL: 0 processes with args redisJobRunnerService [20:47:06] JohnLewis ^^ remember to remove from icinga! [20:47:16] meh [20:48:23] done [20:48:35] JohnLewis thanks! [21:00:38] so, 44 open tasks. 18 are review tasks. interesting [21:01:49] 4 are upstream [21:15:15] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [21:29:58] !log depooling mw1 to try and reduce load when running load intensive script [21:30:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:37:33] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.76, 2.73, 1.46 [21:39:33] RECOVERY - Current Load on swift1 is OK: OK - load average: 2.94, 2.76, 1.62 [22:06:33] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.66, 3.50, 2.84 [23:11:05] PROBLEM - Current Load on nfs1 is CRITICAL: CRITICAL - load average: 7.88, 4.21, 2.48 [23:19:04] RECOVERY - Current Load on mw1 is OK: OK - load average: 0.11, 0.03, 0.01 [23:19:06] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [23:19:14] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [23:21:06] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [23:21:16] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy