[00:00:00] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[00:00:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:17:07] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 70980MiB (15% inode=99%);
[03:41:18] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 23704MiB (10% inode=94%);
[06:09:18] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 10864MiB (4% inode=95%);
[09:05:09] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_44 Femiwiki]
[12:14:15] <MacFanBot>	 [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/77bd56cc3c6f786604ed1dd3490f368b914a5ec9
[12:14:16] <MacFanBot>	 02CreateWiki/03main 07translatewiki.net 0377bd56c Localisation updates from https://translatewiki.net.
[12:14:16] <MacFanBot>	 [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/704eb119c2db2b504661d34accf9b6208266b252
[12:14:16] <MacFanBot>	 02ImportDump/03main 07translatewiki.net 03704eb11 Localisation updates from https://translatewiki.net.
[12:14:17] <MacFanBot>	 [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/de37c24ad4b8226500359f76cefbc33991ce368a
[12:14:19] <MacFanBot>	 02landing/03main 07translatewiki.net 03de37c24 Localisation updates from https://translatewiki.net.
[12:14:21] <MacFanBot>	 [02DataDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/2c76f1e2c4c54ed71ad0a5766cee55864b3ea9ec
[12:14:23] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/828b34a536a6e8042d84283a884bb8498e79cc07
[12:14:24] <MacFanBot>	 02DataDump/03main 07translatewiki.net 032c76f1e Localisation updates from https://translatewiki.net.
[12:14:26] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 03828b34a Localisation updates from https://translatewiki.net.
[12:15:08] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:19:25] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:20:03] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:22:56] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:26:26] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[13:41:34] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/9b4a34be8e9fba69a28e2f4da011369f1f9fe1aa
[13:41:34] <MacFanBot>	 02ssl/03main 07Skye 039b4a34b T14607: Add redirect for superblox64wiki
[13:43:07] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49776MiB (10% inode=99%);
[15:39:46] <MacFanBot>	 [02WikiTideDebug] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 (+1 new commit) 13https://github.com/miraheze/WikiTideDebug/commit/a2f1ed95f15b
[15:39:46] <MacFanBot>	 02WikiTideDebug/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 07dependabot[bot] 03a2f1ed9 Bump eslint-config-wikimedia from 0.32.2 to 0.32.3…
[15:39:48] <MacFanBot>	 [02WikiTideDebug] 07dependabot[bot] added the label 'dependencies' to pull request #29 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/WikiTideDebug/pull/29
[15:39:50] <MacFanBot>	 [02WikiTideDebug] 07dependabot[bot] added the label 'javascript' to pull request #29 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/WikiTideDebug/pull/29
[15:39:52] <MacFanBot>	 [02WikiTideDebug] 07dependabot[bot] opened pull request #29: Bump eslint-config-wikimedia from 0.32.2 to 0.32.3 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3) 13https://github.com/miraheze/WikiTideDebug/pull/29
[15:39:54] <MacFanBot>	 [02WikiTideDebug] 07dependabot[bot] added the label 'dependencies' to pull request #29 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/WikiTideDebug/pull/29
[15:39:56] <MacFanBot>	 [02WikiTideDebug] 07dependabot[bot] added the label 'javascript' to pull request #29 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/WikiTideDebug/pull/29
[15:39:59] <MacFanBot>	 [02WikiTideDebug] 07coderabbitai[bot] commented on pull request #29: --- […] 13https://github.com/miraheze/WikiTideDebug/pull/29#issuecomment-3597261187
[16:21:00] <MacFanBot>	 [02ImportDump] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 (+1 new commit) 13https://github.com/miraheze/ImportDump/commit/73bb810f04b9
[16:21:00] <MacFanBot>	 02ImportDump/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 07dependabot[bot] 0373bb810 Bump eslint-config-wikimedia from 0.32.2 to 0.32.3…
[16:21:01] <MacFanBot>	 [02ImportDump] 07dependabot[bot] added the label 'javascript' to pull request #160 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ImportDump/pull/160
[16:21:03] <MacFanBot>	 [02ImportDump] 07dependabot[bot] added the label 'dependencies' to pull request #160 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ImportDump/pull/160
[16:21:05] <MacFanBot>	 [02ImportDump] 07dependabot[bot] opened pull request #160: Bump eslint-config-wikimedia from 0.32.2 to 0.32.3 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3) 13https://github.com/miraheze/ImportDump/pull/160
[16:21:07] <MacFanBot>	 [02ImportDump] 07dependabot[bot] added the label 'dependencies' to pull request #160 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ImportDump/pull/160
[16:21:09] <MacFanBot>	 [02ImportDump] 07dependabot[bot] added the label 'javascript' to pull request #160 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ImportDump/pull/160
[16:21:18] <MacFanBot>	 [02ImportDump] 07coderabbitai[bot] commented on pull request #160: --- […] 13https://github.com/miraheze/ImportDump/pull/160#issuecomment-3597481501
[16:26:26] <miraheze-github>	 miraheze/ImportDump - dependabot[bot] the build passed.
[16:33:01] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49858MiB (10% inode=99%);
[16:48:11] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49804MiB (10% inode=99%);
[17:40:01] <MacFanBot>	 [02MediaWikiDebugJS] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 (+1 new commit) 13https://github.com/miraheze/MediaWikiDebugJS/commit/3fa0601b2f13
[17:40:02] <MacFanBot>	 02MediaWikiDebugJS/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 07dependabot[bot] 033fa0601 Bump eslint-config-wikimedia from 0.32.2 to 0.32.3…
[17:40:02] <MacFanBot>	 [02MediaWikiDebugJS] 07dependabot[bot] added the label 'dependencies' to pull request #31 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/MediaWikiDebugJS/pull/31
[17:40:04] <MacFanBot>	 [02MediaWikiDebugJS] 07dependabot[bot] added the label 'javascript' to pull request #31 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/MediaWikiDebugJS/pull/31
[17:40:06] <MacFanBot>	 [02MediaWikiDebugJS] 07dependabot[bot] opened pull request #31: Bump eslint-config-wikimedia from 0.32.2 to 0.32.3 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3) 13https://github.com/miraheze/MediaWikiDebugJS/pull/31
[17:40:08] <MacFanBot>	 [02MediaWikiDebugJS] 07dependabot[bot] added the label 'javascript' to pull request #31 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/MediaWikiDebugJS/pull/31
[17:40:10] <MacFanBot>	 [02MediaWikiDebugJS] 07dependabot[bot] added the label 'dependencies' to pull request #31 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/MediaWikiDebugJS/pull/31
[17:40:12] <MacFanBot>	 [02MediaWikiDebugJS] 07coderabbitai[bot] commented on pull request #31: --- […] 13https://github.com/miraheze/MediaWikiDebugJS/pull/31#issuecomment-3597954277
[18:40:04] <MacFanBot>	 [02ManageWiki] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/cdca0469223c
[18:40:04] <MacFanBot>	 02ManageWiki/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3 07dependabot[bot] 03cdca046 Bump eslint-config-wikimedia from 0.32.2 to 0.32.3…
[18:40:05] <MacFanBot>	 [02ManageWiki] 07dependabot[bot] added the label 'dependencies' to pull request #754 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ManageWiki/pull/754
[18:40:05] <MacFanBot>	 [02ManageWiki] 07dependabot[bot] added the label 'javascript' to pull request #754 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ManageWiki/pull/754
[18:40:07] <MacFanBot>	 [02ManageWiki] 07dependabot[bot] opened pull request #754: Bump eslint-config-wikimedia from 0.32.2 to 0.32.3 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.3) 13https://github.com/miraheze/ManageWiki/pull/754
[18:40:09] <MacFanBot>	 [02ManageWiki] 07dependabot[bot] added the label 'dependencies' to pull request #754 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ManageWiki/pull/754
[18:40:11] <MacFanBot>	 [02ManageWiki] 07dependabot[bot] added the label 'javascript' to pull request #754 (Bump eslint-config-wikimedia from 0.32.2 to 0.32.3) 13https://github.com/miraheze/ManageWiki/pull/754
[18:40:13] <MacFanBot>	 [02ManageWiki] 07coderabbitai[bot] commented on pull request #754: --- […] 13https://github.com/miraheze/ManageWiki/pull/754#issuecomment-3598281645
[18:46:46] <miraheze-github>	 miraheze/ManageWiki - dependabot[bot] the build passed.
[19:38:37] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[19:46:37] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[20:35:21] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/0e1bf5f2fa63853ba6a987c9b2644a5b1e052229
[20:35:22] <MacFanBot>	 02ssl/03main 07WikiTideBot 030e1bf5f Bot: Auto-update domain lists
[21:42:05] <MacFanBot>	 [02ssl] 07SomeMWDev created 03T14553 (+1 new commit) 13https://github.com/miraheze/ssl/commit/7945e32ade80
[21:42:05] <MacFanBot>	 02ssl/03T14553 07SomeRandomDeveloper 037945e32 T14553: Add redirect for kpopdemonhuntersplwiki
[21:42:19] <MacFanBot>	 [02ssl] 07SomeMWDev opened pull request #899: T14553: Add redirect for kpopdemonhuntersplwiki (03main...03T14553) 13https://github.com/miraheze/ssl/pull/899
[21:44:18] <MacFanBot>	 [02ssl] 07SomeMWDev 04force-pushed 03T14553 from 037945e32 to 0375fd514 13https://api.github.com/repos/miraheze/ssl/commits/T14553
[21:44:19] <MacFanBot>	 02ssl/03T14553 07SomeRandomDeveloper 0375fd514 T14553: Add redirect for kpopdemonhuntersplwiki
[21:44:29] <MacFanBot>	 [02ssl] 07coderabbitai[bot] commented on pull request #899: Added a new redirect configuration entry for the K-pop Demon Hunters Polish wiki, mapping the Miraheze origin domain `kpopdemonhunterspl.miraheze.org` to the custom domain `pl.kpopdemonhunters.wiki` with Miraheze SSL certificate and Cloudflare CA […] 13https://github.com/miraheze/ssl/pull/899#issuecomment-3599074675
[21:45:43] <MacFanBot>	 [02ssl] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3473789a31b12ba2e27cca478fba92ebdbe2c7e9
[21:45:43] <MacFanBot>	 02ssl/03main 07SomeRandomDeveloper 033473789 T14553: Add redirect for kpopdemonhuntersplwiki (#899)
[21:45:43] <MacFanBot>	 [02ssl] 07SomeMWDev merged pull request #899: T14553: Add redirect for kpopdemonhuntersplwiki (03main...03T14553) 13https://github.com/miraheze/ssl/pull/899
[21:45:45] <MacFanBot>	 [02ssl] 07SomeMWDev 04deleted 03T14553 at 0375fd514 13https://api.github.com/repos/miraheze/ssl/commit/75fd514
[21:48:29] <MacFanBot>	 [02ssl] 07SomeMWDev created 03T14544 (+1 new commit) 13https://github.com/miraheze/ssl/commit/3572d235f1c8
[21:48:29] <MacFanBot>	 02ssl/03T14544 07SomeRandomDeveloper 033572d23 T14544: Add redirect for midnightburgerwiki
[21:48:46] <MacFanBot>	 [02ssl] 07SomeMWDev opened pull request #900: T14544: Add redirect for midnightburgerwiki (03main...03T14544) 13https://github.com/miraheze/ssl/pull/900
[21:49:27] <MacFanBot>	 [02ssl] 07SomeMWDev merged pull request #900: T14544: Add redirect for midnightburgerwiki (03main...03T14544) 13https://github.com/miraheze/ssl/pull/900
[21:49:29] <MacFanBot>	 [02ssl] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/2d1aff0d0ea4ab114f7c3d36e83b3ff548481fec
[21:49:31] <MacFanBot>	 [02ssl] 07SomeMWDev 04deleted 03T14544 at 033572d23 13https://api.github.com/repos/miraheze/ssl/commit/3572d23
[21:49:32] <MacFanBot>	 02ssl/03main 07SomeRandomDeveloper 032d1aff0 T14544: Add redirect for midnightburgerwiki (#900)
[21:50:27] <MacFanBot>	 [02ssl] 07coderabbitai[bot] commented on pull request #900: Added a new redirect configuration entry for midnightburgerwiki to redirects.yaml, mapping midnightburger.miraheze.org to wiki.weopenatsix.com with Cloudflare SSL and strict HSTS policy. […] 13https://github.com/miraheze/ssl/pull/900#issuecomment-3599113261
[21:59:15] <MacFanBot>	 [02mw-config] 07SomeMWDev created 03T14610 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/9265121fc923
[21:59:15] <MacFanBot>	 02mw-config/03T14610 07SomeRandomDeveloper 039265121 T14610: Set default minerva theme to night mode for fwtdwiki
[21:59:35] <MacFanBot>	 [02mw-config] 07SomeMWDev opened pull request #6194: T14610: Set default minerva theme to night mode for fwtdwiki (03main...03T14610) 13https://github.com/miraheze/mw-config/pull/6194
[22:00:32] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[22:00:54] <MacFanBot>	 [02mw-config] 07SomeMWDev merged pull request #6194: T14610: Set default minerva theme to night mode for fwtdwiki (03main...03T14610) 13https://github.com/miraheze/mw-config/pull/6194
[22:00:54] <MacFanBot>	 [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/4d68b4865fbc26d336b500c308a3d3487f244d09
[22:00:54] <MacFanBot>	 02mw-config/03main 07SomeRandomDeveloper 034d68b48 T14610: Set default minerva theme to night mode for fwtdwiki (#6194)
[22:00:56] <MacFanBot>	 [02mw-config] 07SomeMWDev 04deleted 03T14610 at 039265121 13https://api.github.com/repos/miraheze/mw-config/commit/9265121
[22:01:04] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[22:01:05] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s
[22:01:08] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[22:01:09] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[22:01:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:01:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:01:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:01:21] <MirahezeRelay>	 <abaddriverlol> [1/5] ```
[22:01:22] <MirahezeRelay>	 <abaddriverlol> [2/5] [somerandomdeveloper@mwtask181:~]$ mwdeploy --config --pull=config --servers=all
[22:01:22] <MirahezeRelay>	 <abaddriverlol> [3/5] Execute: sudo -u www-data git -C /srv/mediawiki-staging/config/ pull --quiet
[22:01:22] <MirahezeRelay>	 <abaddriverlol> [4/5] fatal: unable to write loose object file: No space left on device
[22:01:23] <MirahezeRelay>	 <abaddriverlol> [5/5] ```
[22:01:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:01:50] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[22:01:58] <MirahezeRelay>	 <abaddriverlol, replying to icinga-miraheze> critical since 6:09
[22:02:00] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> Shut
[22:02:01] <MirahezeRelay>	 <rhinosf1> Shit
[22:02:07] <MirahezeRelay>	 <rhinosf1> I guess that ran out of disk
[22:02:11] <MirahezeRelay>	 <abaddriverlol> yep
[22:02:14] <MirahezeRelay>	 <rhinosf1> Oh wait you're ops now
[22:02:24] <MirahezeRelay>	 <rhinosf1> I can be lazy
[22:02:39] <MirahezeRelay>	 <rhinosf1> It's probably wiki dumps or something that's filled the disk
[22:02:47] <MirahezeRelay>	 <rhinosf1> See if you can delete or compress anything
[22:03:19] <MirahezeRelay>	 <abaddriverlol> [1/4] can't even sort for the biggest folders lmao
[22:03:19] <MirahezeRelay>	 <abaddriverlol> [2/4] ```
[22:03:19] <MirahezeRelay>	 <abaddriverlol> [3/4] sort: cannot create temporary file in '/tmp': No space left on device
[22:03:20] <MirahezeRelay>	 <abaddriverlol> [4/4] ```
[22:03:23] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> where are they stored
[22:03:40] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> Reception's homedir is often a nightmare
[22:03:48] <MirahezeRelay>	 <rhinosf1> Then look at the IA script in puppet
[22:05:17] <MirahezeRelay>	 <abaddriverlol> there are backups from september in there
[22:05:51] <MirahezeRelay>	 <abaddriverlol> if there's multiple for the same wiki from different months can I delete the older ones? @rhinosf1
[22:06:10] <MirahezeRelay>	 <pskyechology> this is the most cyan i have seen in a while
[22:06:50] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> Yes
[22:17:08] <MirahezeRelay>	 <abaddriverlol> today's backup for 1d6chanwiki is taking up 23gb, which is 10% of the disk space on mwtask181
[22:18:17] <MirahezeRelay>	 <abaddriverlol> I deleted the largest duplicates but there is still only 38mb of free space
[22:21:23] <MirahezeRelay>	 <abaddriverlol> the cause must be something else because /opt/backups is now only taking 41gb (23gb of which is still 1d6chanwiki btw)
[22:40:08] <MirahezeRelay>	 <abaddriverlol> /tmp is taking 79gb, 16gb of which belongs to 1d6chanwiki as well (images)
[22:40:56] <MirahezeRelay>	 <abaddriverlol> @paladox any idea what we can do about this? I could delete one of the large backup files but we'd have to regenerate it at some point and I'm not sure how to do that considering we don't have enough disk space
[22:41:55] <MirahezeRelay>	 <abaddriverlol> IMO we shouldn't even run jobs like iaarchive on our deployment/canary server, I don't think there is any reason it couldn't run on another mwtask
[22:42:15] <MirahezeRelay>	 <paladox> it should delete the file after upload tbh
[22:42:47] <MirahezeRelay>	 <paladox> but anyways i'm not sure if it's generating a backup and then uploading it all (it should be changed).
[22:45:16] <MirahezeRelay>	 <abaddriverlol, replying to paladox> [1/4] ```
[22:45:16] <MirahezeRelay>	 <abaddriverlol, replying to paladox> [2/4] [somerandomdeveloper@mwtask181:/opt/backups]$ sudo ls /tmp | grep -- '-images-' | wc -l
[22:45:16] <MirahezeRelay>	 <abaddriverlol, replying to paladox> [3/4] 2027
[22:45:16] <MirahezeRelay>	 <abaddriverlol, replying to paladox> [4/4] ```
[22:45:22] <MirahezeRelay>	 <abaddriverlol> it really should delete the files
[22:45:30] <MirahezeRelay>	 <abaddriverlol> any suggestion what to do now?
[22:45:34] <MirahezeRelay>	 <abaddriverlol> delete the largest one ig?
[22:45:51] <MirahezeRelay>	 <paladox> yeh you can, it's an emergency
[22:46:07] <MirahezeRelay>	 <paladox> but the script should be modified to upload then delete, then backup, rinse and repeat
[22:47:05] <MirahezeRelay>	 <abaddriverlol> !log ran "sudo -u www-data rm /tmp/1d6chanwiki-images-20251201.tar.gz" on mwtask181 to free up disk space
[22:47:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:47:16] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 15412MiB (6% inode=95%);
[22:47:38] <MirahezeRelay>	 <abaddriverlol> still not a lot of free disk space but at least it's usable now to deploy stuff
[22:47:46] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[22:47:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:48:11] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s
[22:48:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:51:13] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 11982MiB (5% inode=95%);
[22:51:41] <MirahezeRelay>	 <abaddriverlol> oh ffs the backup is continuing and filling up the disk again
[23:01:47] <MirahezeRelay>	 <paladox> kill the script if you must
[23:01:59] <MirahezeRelay>	 <paladox> ideally probably should, to change it
[23:03:42] <MirahezeRelay>	 <abaddriverlol> yeah I've stopped it a few minutes ago
[23:03:52] <MirahezeRelay>	 <abaddriverlol> I can't fix the script rn though
[23:04:22] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:04:34] <MirahezeRelay>	 <abaddriverlol> !log stopped ia backup script on mwtask181 due to low disk space
[23:04:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log