[00:00:26] 06Operations, 10Wikimedia-Mailing-lists: deactivate maint-announce - https://phabricator.wikimedia.org/T143760#2606292 (10Dzahn) p:05Triage>03Normal [00:14:16] 06Operations, 06Performance-Team, 10scap, 07Epic: During deployment old servers may populate new cache URIs - https://phabricator.wikimedia.org/T47877#2606312 (10Krinkle) [01:34:34] (03PS1) 10Dzahn: icinga: add Ladsgroup to team-ores contact group [puppet] - 10https://gerrit.wikimedia.org/r/308306 [01:36:17] (03CR) 10Dzahn: [C: 032] "just added the Icinga contact in private repo as requested by Ladsgroup" [puppet] - 10https://gerrit.wikimedia.org/r/308306 (owner: 10Dzahn) [01:36:26] (03PS2) 10Dzahn: icinga: add Ladsgroup to team-ores contact group [puppet] - 10https://gerrit.wikimedia.org/r/308306 [01:52:55] 06Operations, 10MediaWiki-extensions-CentralNotice, 10Traffic: Varnish-triggered CN campaign about browser security - https://phabricator.wikimedia.org/T144194#2591355 (10Legoktm) FWIW, MediaWiki has a built-in sitenotice (https://www.mediawiki.org/wiki/Manual:Interface/Sitenotice) functionality that is not... [02:22:11] PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [1000.0] [02:24:29] (03Draft1) 10Paladox: Fix setting phabricator logo [puppet] - 10https://gerrit.wikimedia.org/r/308308 [02:28:01] (03PS2) 10Paladox: Fix setting phabricator logo [puppet] - 10https://gerrit.wikimedia.org/r/308308 [02:30:05] (03CR) 10Paladox: "Followed up by ea6ae6b33b4d0da8a47c74ac15fd66df008db4e9" [puppet] - 10https://gerrit.wikimedia.org/r/307462 (owner: 10Paladox) [02:30:45] (03PS3) 10Paladox: Fix setting phabricator logo [puppet] - 10https://gerrit.wikimedia.org/r/308308 [02:31:30] (03PS6) 10Paladox: phabricator: Add text/x-python to files.viewable-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/307456 [02:39:50] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 35s) [02:39:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:46:12] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 3 02:46:12 UTC 2016 (duration 6m 22s) [02:46:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [04:15:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 5268 MB (9% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643967 MB (35% inode=99%) [04:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 4547 MB (8% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643982 MB (35% inode=99%) [04:25:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 3827 MB (7% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643977 MB (35% inode=99%) [04:30:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 3107 MB (5% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643972 MB (35% inode=99%) [04:35:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 2386 MB (4% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643980 MB (35% inode=99%) [04:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 1666 MB (3% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643975 MB (35% inode=99%) [04:41:33] PROBLEM - Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/dumps - 288 bytes in 1.986 second response time [04:45:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 945 MB (1% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643970 MB (35% inode=99%) [04:50:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 225 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803228 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643979 MB (35% inode=99%) [04:55:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643974 MB (35% inode=99%) [05:00:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643969 MB (35% inode=99%) [05:05:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643977 MB (35% inode=99%) [05:05:40] RECOVERY - Ulsfo HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [05:10:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643972 MB (35% inode=99%) [05:15:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643967 MB (35% inode=99%) [05:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643975 MB (35% inode=99%) [05:25:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643970 MB (35% inode=99%) [05:25:33] RECOVERY - Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 3.239 second response time [05:30:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643965 MB (35% inode=99%) [05:35:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643974 MB (35% inode=99%) [05:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643968 MB (35% inode=99%) [05:45:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643963 MB (35% inode=99%) [05:50:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643972 MB (35% inode=99%) [05:55:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643967 MB (35% inode=99%) [06:00:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643961 MB (35% inode=99%) [06:05:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643970 MB (35% inode=99%) [06:10:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643965 MB (35% inode=99%) [06:15:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643959 MB (35% inode=99%) [06:20:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643969 MB (35% inode=99%) [06:25:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643963 MB (35% inode=99%) [06:30:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643958 MB (35% inode=99%) [06:35:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643967 MB (35% inode=99%) [06:40:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643961 MB (35% inode=99%) [06:42:09] PROBLEM - puppet last run on elastic2003 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 5 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[acct],Package[htop] [06:45:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 260 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643956 MB (35% inode=99%) [06:50:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643965 MB (35% inode=99%) [06:55:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643960 MB (35% inode=99%) [06:57:37] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled] [07:00:17] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643954 MB (35% inode=99%) [07:04:53] PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] [07:05:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643963 MB (35% inode=99%) [07:07:10] RECOVERY - puppet last run on elastic2003 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:10:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643958 MB (35% inode=99%) [07:15:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643953 MB (35% inode=99%) [07:17:30] 06Operations, 10fundraising-tech-ops: barium low on disk space - https://phabricator.wikimedia.org/T144659#2606513 (10Peachey88) [07:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643962 MB (35% inode=99%) [07:20:15] 06Operations, 10fundraising-tech-ops: Ensure all disaster recover documentation is in one central location - https://phabricator.wikimedia.org/T95841#2606514 (10Peachey88) [07:22:24] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643956 MB (35% inode=99%) [07:30:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643951 MB (35% inode=99%) [07:35:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643960 MB (35% inode=99%) [07:39:26] RECOVERY - Ulsfo HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [07:40:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643954 MB (35% inode=99%) [07:40:25] come on now icinga-wm, if that was CRITICAL, it'd have gone down hours ago [07:43:51] PROBLEM - BGP status on cr2-knams is CRITICAL: BGP CRITICAL - AS13030/IPv6: OpenConfirm, AS13030/IPv4: OpenConfirm [07:45:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643949 MB (35% inode=99%) [07:50:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643958 MB (35% inode=99%) [07:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643953 MB (35% inode=99%) [08:00:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643947 MB (35% inode=99%) [08:05:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643956 MB (35% inode=99%) [08:10:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643951 MB (35% inode=99%) [08:15:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643946 MB (35% inode=99%) [08:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643954 MB (35% inode=99%) [08:21:07] RECOVERY - BGP status on cr2-knams is OK: BGP OK - up: 10, down: 0, shutdown: 0 [08:24:52] (03CR) 10Hashar: "recheck" [debs/contenttranslation/apertium-ita] - 10https://gerrit.wikimedia.org/r/308153 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:24:55] (03CR) 10Hashar: "recheck" [debs/contenttranslation/apertium-srd] - 10https://gerrit.wikimedia.org/r/308154 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:24:59] (03CR) 10Hashar: "recheck" [debs/contenttranslation/apertium-srd-ita] - 10https://gerrit.wikimedia.org/r/308179 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:25:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643949 MB (35% inode=99%) [08:25:24] (03CR) 10jenkins-bot: [V: 04-1] apertium-srd-ita: Initial Debian packaging [debs/contenttranslation/apertium-srd-ita] - 10https://gerrit.wikimedia.org/r/308179 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:26:45] (03CR) 10Hashar: "Depends: apertium-srd (>= 0.9.0~r72552) which is a virtual package and is not provided by any available package" [debs/contenttranslation/apertium-srd-ita] - 10https://gerrit.wikimedia.org/r/308179 (https://phabricator.wikimedia.org/T144174) (owner: 10KartikMistry) [08:28:38] PROBLEM - BGP status on cr2-knams is CRITICAL: BGP CRITICAL - AS13030/IPv6: Active, AS13030/IPv4: OpenConfirm [08:30:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643944 MB (35% inode=99%) [08:35:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643953 MB (35% inode=99%) [08:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643947 MB (35% inode=99%) [08:41:06] RECOVERY - BGP status on cr2-knams is OK: BGP OK - up: 10, down: 0, shutdown: 0 [08:45:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643942 MB (35% inode=99%) [08:50:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643951 MB (35% inode=99%) [08:53:23] PROBLEM - BGP status on cr2-knams is CRITICAL: BGP CRITICAL - AS13030/IPv6: OpenConfirm, AS13030/IPv4: OpenConfirm [08:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 1 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643946 MB (35% inode=99%) [09:00:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643940 MB (35% inode=99%) [09:01:00] RECOVERY - BGP status on cr2-knams is OK: BGP OK - up: 9, down: 0, shutdown: 1 [09:05:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643949 MB (35% inode=99%) [09:10:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643944 MB (35% inode=99%) [09:15:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643938 MB (35% inode=99%) [09:18:34] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [09:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643947 MB (35% inode=99%) [09:21:00] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [09:25:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643942 MB (35% inode=99%) [09:28:04] PROBLEM - BGP status on cr2-knams is CRITICAL: BGP CRITICAL - AS13030/IPv4: OpenConfirm [09:30:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643936 MB (35% inode=99%) [09:32:22] the bot has gone crazy [09:33:14] RECOVERY - BGP status on cr2-knams is OK: BGP OK - up: 10, down: 0, shutdown: 0 [09:35:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643945 MB (35% inode=99%) [09:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643939 MB (35% inode=99%) [09:45:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643934 MB (35% inode=99%) [09:50:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643943 MB (35% inode=99%) [09:53:19] (03PS1) 10Hashar: ircyall: move role to module/role [puppet] - 10https://gerrit.wikimedia.org/r/308311 [09:55:05] (03CR) 10Hashar: "The role is role::ircyall" [puppet] - 10https://gerrit.wikimedia.org/r/308311 (owner: 10Hashar) [09:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 5 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643938 MB (35% inode=99%) [10:00:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643932 MB (35% inode=99%) [10:00:21] PROBLEM - BGP status on cr2-knams is CRITICAL: BGP CRITICAL - AS13030/IPv6: OpenConfirm, AS13030/IPv4: OpenConfirm [10:02:30] 06Operations, 10DBA, 10MediaWiki-Maintenance-scripts, 06Release-Engineering-Team, 07RfC: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2606542 (10jcrespo) [10:04:45] (03PS1) 10Hashar: quarry: migrate classes to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308313 (https://phabricator.wikimedia.org/T93645) [10:05:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643942 MB (35% inode=99%) [10:05:21] RECOVERY - BGP status on cr2-knams is OK: BGP OK - up: 10, down: 0, shutdown: 0 [10:05:23] (03CR) 10Hashar: "Should be fine, I have no clue where it is being used though.." [puppet] - 10https://gerrit.wikimedia.org/r/308313 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [10:05:47] 06Operations, 07Puppet, 07Epic, 07Need-volunteer, 13Patch-For-Review: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#2606559 (10hashar) [10:10:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643936 MB (35% inode=99%) [10:12:39] (03PS1) 10Hashar: ldap: migrate role classes to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308314 (https://phabricator.wikimedia.org/T93645) [10:13:03] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [10:15:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643931 MB (35% inode=99%) [10:15:21] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [10:19:24] (03CR) 10Paladox: [C: 031] ldap: migrate role classes to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308314 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [10:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643940 MB (35% inode=99%) [10:21:07] !log deploying schema change on s7 hosts T139090 [10:21:08] T139090: Deploy I2b042685 to all databases - https://phabricator.wikimedia.org/T139090 [10:21:11] (03CR) 10Alex Monk: [C: 031] ldap: migrate role classes to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308314 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [10:21:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:24:51] PROBLEM - BGP status on cr2-knams is CRITICAL: BGP CRITICAL - AS13030/IPv4: Active [10:25:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643934 MB (35% inode=99%) [10:30:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643928 MB (35% inode=99%) [10:35:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643938 MB (35% inode=99%) [10:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643932 MB (35% inode=99%) [10:43:19] (03CR) 10Paladox: [C: 031] quarry: migrate classes to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308313 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [10:43:51] (03CR) 10Paladox: [C: 031] ircyall: move role to module/role [puppet] - 10https://gerrit.wikimedia.org/r/308311 (owner: 10Hashar) [10:44:54] RECOVERY - BGP status on cr2-knams is OK: BGP OK - up: 10, down: 0, shutdown: 0 [10:45:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643926 MB (35% inode=99%) [10:50:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643935 MB (35% inode=99%) [10:52:43] (03Draft1) 10Paladox: manifests/role: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308315 [10:52:57] 06Operations, 06Discovery, 06Maps, 06WMF-Legal, 03Maps-Sprint: Define tile usage policy - https://phabricator.wikimedia.org/T141815#2606581 (10timautin) Thanks for the input! Downloading vector tiles and rendering would be really interesting, do you have more information (which tools, which format for t... [10:53:08] (03PS2) 10Paladox: manifests/role: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308315 [10:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 2 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643929 MB (35% inode=99%) [10:57:44] (03Draft1) 10Paladox: manifests/role: Fix indentation [puppet] - 10https://gerrit.wikimedia.org/r/308316 [10:59:17] (03PS2) 10Paladox: manifests/role: Fix indentation [puppet] - 10https://gerrit.wikimedia.org/r/308316 [11:00:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643922 MB (35% inode=99%) [11:04:41] PROBLEM - BGP status on cr2-knams is CRITICAL: BGP CRITICAL - AS13030/IPv4: OpenConfirm [11:05:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643933 MB (35% inode=99%) [11:10:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643927 MB (35% inode=99%) [11:12:17] RECOVERY - BGP status on cr2-knams is OK: BGP OK - up: 10, down: 0, shutdown: 0 [11:15:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643920 MB (35% inode=99%) [11:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643931 MB (35% inode=99%) [11:25:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643924 MB (35% inode=99%) [11:30:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643918 MB (35% inode=99%) [11:35:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643929 MB (35% inode=99%) [11:39:12] (03PS1) 10Hashar: labstore: ignore autoloader layout lint error [puppet] - 10https://gerrit.wikimedia.org/r/308320 (https://phabricator.wikimedia.org/T93645) [11:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643922 MB (35% inode=99%) [11:42:30] (03PS1) 10Hashar: role: move mediawiki::install to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308322 (https://phabricator.wikimedia.org/T93645) [11:45:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643916 MB (35% inode=99%) [11:50:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803226 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643926 MB (35% inode=99%) [11:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 2 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643920 MB (35% inode=99%) [12:00:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643913 MB (35% inode=99%) [12:05:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643924 MB (35% inode=99%) [12:10:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643917 MB (35% inode=99%) [12:13:01] Hi. Is there a list of tests that jenkins bot can run? also, how to tell CI not to use the wikimedia queue for a labs tool? [12:15:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643910 MB (35% inode=99%) [12:15:15] MarcoAurelio see https://phabricator.wikimedia.org/diffusion/CICF/ please [12:15:28] I come from there :) [12:15:41] also -releng is the channel to ask about jenkins. [12:15:47] ah, k [12:15:53] https://phabricator.wikimedia.org/diffusion/CICF/browse/master/zuul/layout.yaml [12:15:57] will continue browsing the repo [12:16:01] Ok [12:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643922 MB (35% inode=99%) [12:22:25] 06Operations, 10Continuous-Integration-Config, 06Operations-Software-Development: Flake8 for python files without extension in puppet repo - https://phabricator.wikimedia.org/T144169#2606640 (10hashar) p:05Triage>03Normal [12:23:40] 06Operations, 07Puppet, 10Continuous-Integration-Config, 13Patch-For-Review: post build failures for operations/puppet on operations-puppet-doc - https://phabricator.wikimedia.org/T143233#2606643 (10hashar) p:05Triage>03High [12:25:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643915 MB (35% inode=99%) [12:27:30] (03CR) 10Luke081515: [C: 04-1] "Can you add the task for the name migration as comment too?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio) [12:28:01] MarcoAurelio: ^ ? [12:28:12] I don't think that's needed [12:28:30] I won't do it, not now, I'm busy with jenkins now [12:28:36] will have a look later [12:28:36] ok [12:29:12] MarcoAurelio: Hm, I think leaving a note at the original task would be enough too. I'm going to do that, and remove my -1 [12:30:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643908 MB (35% inode=99%) [12:31:58] (03CR) 10Luke081515: [C: 031] "I left a note at the both original tasks instead, makes it easier." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio) [12:32:09] MarcoAurelio: just FYI, solved now :) [12:33:27] okay thanks [12:35:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643919 MB (35% inode=99%) [12:38:02] (03CR) 10Luke081515: [C: 031] Remove upload rights on wikis where local uploads are disabled. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:40:05] 06Operations, 10DBA, 10MediaWiki-Maintenance-scripts, 06Release-Engineering-Team, 07RfC: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2606680 (10hashar) I am definitely a fan of having a list of operations that... [12:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643913 MB (35% inode=99%) [12:42:17] (03CR) 10Hashar: [C: 031] "Do we care about arrow alignments? I thought the consensus that was to aim for it but not enforce it since it is quite annoying to adjust" [puppet] - 10https://gerrit.wikimedia.org/r/308316 (owner: 10Paladox) [12:42:20] -1 incoming [12:43:09] (03CR) 10Paladox: "I'm not sure, but I'm using a new puppet-lint version and it complains about this." [puppet] - 10https://gerrit.wikimedia.org/r/308316 (owner: 10Paladox) [12:45:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643906 MB (35% inode=99%) [12:50:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643917 MB (35% inode=99%) [12:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 2 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643910 MB (35% inode=99%) [13:00:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643903 MB (35% inode=99%) [13:00:36] (03CR) 10Paladox: [C: 031] role: move mediawiki::install to autoloader layout [puppet] - 10https://gerrit.wikimedia.org/r/308322 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [13:05:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643915 MB (35% inode=99%) [13:10:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643908 MB (35% inode=99%) [13:11:39] (03CR) 10Paladox: [C: 031] labstore: ignore autoloader layout lint error [puppet] - 10https://gerrit.wikimedia.org/r/308320 (https://phabricator.wikimedia.org/T93645) (owner: 10Hashar) [13:13:12] (03Draft1) 10Paladox: aqs and restbase: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308328 [13:15:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643901 MB (35% inode=99%) [13:17:29] (03PS2) 10Paladox: aqs and restbase: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308328 [13:19:05] (03PS3) 10Paladox: aqs and restbase: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308328 [13:20:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643912 MB (35% inode=99%) [13:20:29] There seems to be puppet errors ^^ [13:20:42] and also other errors too [13:23:05] 06Operations, 07Puppet, 10Continuous-Integration-Config, 13Patch-For-Review: post build failures for operations/puppet on operations-puppet-doc - https://phabricator.wikimedia.org/T143233#2606790 (10Paladox) @hashar the fix you wrote in T143233#2580487 I uploaded it to https://gerrit.wikimedia.org/r/30765... [13:25:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643905 MB (35% inode=99%) [13:28:42] ^ silenced barium alert for 3h [13:41:29] 06Operations, 10MediaWiki-ResourceLoader, 06Performance-Team, 10Traffic: Expires header for load.php should be relative to request time instead of cache time - https://phabricator.wikimedia.org/T105657#2606814 (10BBlack) [14:24:30] !log depool cp4005 [14:24:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:10:17] (03Draft1) 10Paladox: cgred: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308333 [15:10:45] (03PS2) 10Paladox: cgred: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308333 [15:11:21] (03PS3) 10Paladox: cgred: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308333 (https://phabricator.wikimedia.org/T93645) [15:12:06] (03PS4) 10Paladox: aqs and restbase: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308328 (https://phabricator.wikimedia.org/T93645) [15:12:34] (03PS3) 10Paladox: manifests/role: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308316 (https://phabricator.wikimedia.org/T93645) [15:13:12] (03PS3) 10Paladox: manifests/role: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308315 (https://phabricator.wikimedia.org/T93645) [15:13:33] (03PS3) 10Paladox: openstack: lint, fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308278 [15:13:45] (03PS4) 10Paladox: openstack: lint, fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308278 (https://phabricator.wikimedia.org/T93645) [15:24:02] 06Operations, 06Discovery, 06Maps, 06WMF-Legal, 03Maps-Sprint: Define tile usage policy - https://phabricator.wikimedia.org/T141815#2513156 (10Yurik) @timautin our tiles are stored in Mapbox's vector tile format, and we use [[ https://github.com/kartotherian/osm-bright.tm2 | this style ]] to convert it t... [15:30:41] (03Draft1) 10Paladox: deployment: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308336 [15:31:25] (03PS2) 10Paladox: deployment: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308336 (https://phabricator.wikimedia.org/T93645) [15:32:57] (03Draft1) 10Paladox: docker: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308337 (https://phabricator.wikimedia.org/T93645) [15:33:38] (03PS2) 10Paladox: docker: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308337 (https://phabricator.wikimedia.org/T93645) [15:34:29] (03PS3) 10Paladox: docker: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308337 (https://phabricator.wikimedia.org/T93645) [15:37:07] (03Draft1) 10Paladox: facilities: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308338 (https://phabricator.wikimedia.org/T93645) [15:37:30] (03PS2) 10Paladox: facilities: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308338 (https://phabricator.wikimedia.org/T93645) [15:38:04] (03PS3) 10Paladox: facilities: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308338 (https://phabricator.wikimedia.org/T93645) [15:41:06] (03Draft1) 10Paladox: ganglia: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308339 (https://phabricator.wikimedia.org/T93645) [15:41:43] (03PS2) 10Paladox: ganglia: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308339 (https://phabricator.wikimedia.org/T93645) [15:43:18] (03PS3) 10Paladox: ganglia: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308339 (https://phabricator.wikimedia.org/T93645) [15:46:17] (03Draft1) 10Paladox: labs_dns: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308340 (https://phabricator.wikimedia.org/T93645) [15:46:44] (03PS2) 10Paladox: labs_dns: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308340 (https://phabricator.wikimedia.org/T93645) [15:49:43] (03Draft1) 10Paladox: lvs: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308341 (https://phabricator.wikimedia.org/T93645) [15:50:09] (03PS2) 10Paladox: lvs: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308341 (https://phabricator.wikimedia.org/T93645) [15:51:19] (03Draft1) 10Paladox: lvs: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308342 (https://phabricator.wikimedia.org/T93645) [15:51:41] (03PS2) 10Paladox: lvs: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308342 (https://phabricator.wikimedia.org/T93645) [15:54:58] (03Draft1) 10Paladox: mathoid: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308343 (https://phabricator.wikimedia.org/T93645) [15:55:20] (03PS2) 10Paladox: mathoid: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308343 (https://phabricator.wikimedia.org/T93645) [15:59:32] (03Draft1) 10Paladox: nagios_common: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308344 (https://phabricator.wikimedia.org/T93645) [16:00:01] (03PS2) 10Paladox: nagios_common: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308344 (https://phabricator.wikimedia.org/T93645) [16:02:13] (03Draft1) 10Paladox: openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) [16:02:31] (03PS2) 10Paladox: openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) [16:03:08] (03PS3) 10Paladox: openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) [16:04:21] (03PS4) 10Paladox: openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) [16:05:10] (03PS5) 10Paladox: openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) [16:06:54] (03PS6) 10Paladox: openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) [16:07:41] (03PS7) 10Paladox: openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) [16:08:13] (03Draft1) 10Paladox: openstack: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308346 [16:09:00] (03PS2) 10Paladox: openstack: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308346 (https://phabricator.wikimedia.org/T93645) [16:12:49] 06Operations, 10Traffic, 07Browser-Support-Internet-Explorer, 07HTTPS: Internet Explorer 6 can not reach https://*.wikipedia.org - https://phabricator.wikimedia.org/T143539#2606938 (10Florian) @BBlack: This, however, doesn't help people, who already use a browser, which isn't able to connect to one of our... [16:12:49] (03PS3) 10Paladox: openstack: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308346 (https://phabricator.wikimedia.org/T93645) [16:16:05] (03Draft1) 10Paladox: puppetmaster: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308347 (https://phabricator.wikimedia.org/T93645) [16:16:24] (03PS2) 10Paladox: puppetmaster: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308347 (https://phabricator.wikimedia.org/T93645) [16:19:15] (03PS3) 10Paladox: puppetmaster: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308347 (https://phabricator.wikimedia.org/T93645) [16:20:10] (03Draft1) 10Paladox: puppetmaster: Fix trailing whitespace found [puppet] - 10https://gerrit.wikimedia.org/r/308348 (https://phabricator.wikimedia.org/T93645) [16:21:18] (03PS2) 10Paladox: puppetmaster: Fix trailing whitespace found [puppet] - 10https://gerrit.wikimedia.org/r/308348 (https://phabricator.wikimedia.org/T93645) [16:24:38] (03Draft1) 10Paladox: pybal: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308349 (https://phabricator.wikimedia.org/T93645) [16:24:40] (03PS8) 10MarcoAurelio: Closing wikimania2015wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298772 (https://phabricator.wikimedia.org/T139032) [16:25:14] (03PS2) 10Paladox: pybal: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308349 (https://phabricator.wikimedia.org/T93645) [16:30:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643875 MB (35% inode=99%) [16:35:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643884 MB (35% inode=99%) [16:40:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643879 MB (35% inode=99%) [16:45:12] (03Draft1) 10Paladox: rsync: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308350 (https://phabricator.wikimedia.org/T93645) [16:45:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643873 MB (35% inode=99%) [16:45:36] (03PS2) 10Paladox: rsync: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308350 (https://phabricator.wikimedia.org/T93645) [16:50:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643882 MB (35% inode=99%) [16:55:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 6 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643877 MB (35% inode=99%) [17:00:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643872 MB (35% inode=99%) [17:05:06] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643881 MB (35% inode=99%) [17:10:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643876 MB (35% inode=99%) [17:15:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643870 MB (35% inode=99%) [17:19:16] PROBLEM - puppet last run on mw2139 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py] [17:20:05] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643879 MB (35% inode=99%) [17:25:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643874 MB (35% inode=99%) [17:30:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643870 MB (35% inode=99%) [17:35:07] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643878 MB (35% inode=99%) [17:40:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643873 MB (35% inode=99%) [17:44:39] RECOVERY - puppet last run on mw2139 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:45:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643868 MB (35% inode=99%) [17:50:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643876 MB (35% inode=99%) [17:55:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 6 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643871 MB (35% inode=99%) [18:00:07] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643866 MB (35% inode=99%) [18:05:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643874 MB (35% inode=99%) [18:10:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643869 MB (35% inode=99%) [18:15:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643865 MB (35% inode=99%) [18:20:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643873 MB (35% inode=99%) [18:25:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643869 MB (35% inode=99%) [18:29:00] (03Draft1) 10Paladox: snapshot: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308355 [18:30:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643864 MB (35% inode=99%) [18:33:17] (03PS3) 10Paladox: snapshot: Fix variable contains an uppercase letter [puppet] - 10https://gerrit.wikimedia.org/r/308355 (https://phabricator.wikimedia.org/T93645) [18:35:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643871 MB (35% inode=99%) [18:36:16] (03Draft1) 10Paladox: statsd_proxy: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308356 (https://phabricator.wikimedia.org/T93645) [18:36:48] (03PS2) 10Paladox: statsd_proxy: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308356 (https://phabricator.wikimedia.org/T93645) [18:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643867 MB (35% inode=99%) [18:41:08] 06Operations, 07Puppet, 07Epic, 07Need-volunteer, 13Patch-For-Review: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#2607026 (10Paladox) Ok, ive done most of the easy stuff ^^ which I think the easy stuff is complete just needs merging then I will re run to see if I... [18:41:44] mutante ^^ :) [18:45:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643862 MB (35% inode=99%) [18:50:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803225 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643870 MB (35% inode=99%) [18:55:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 7 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643865 MB (35% inode=99%) [19:00:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643861 MB (35% inode=99%) [19:05:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643868 MB (35% inode=99%) [19:10:07] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643864 MB (35% inode=99%) [19:15:16] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643860 MB (35% inode=99%) [19:20:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643867 MB (35% inode=99%) [19:25:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 1 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643862 MB (35% inode=99%) [19:30:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643858 MB (35% inode=99%) [19:35:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643866 MB (35% inode=99%) [19:40:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643861 MB (35% inode=99%) [19:45:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643857 MB (35% inode=99%) [19:50:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643864 MB (35% inode=99%) [19:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 8 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643860 MB (35% inode=99%) [20:00:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643855 MB (35% inode=99%) [20:01:23] (03PS1) 10BryanDavis: striker: Add custom error pages to apache vhost [puppet] - 10https://gerrit.wikimedia.org/r/308359 (https://phabricator.wikimedia.org/T144040) [20:05:07] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643863 MB (35% inode=99%) [20:10:16] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643858 MB (35% inode=99%) [20:14:46] (03PS2) 10BryanDavis: striker: Add custom error pages to apache vhost [puppet] - 10https://gerrit.wikimedia.org/r/308359 (https://phabricator.wikimedia.org/T144040) [20:15:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643854 MB (35% inode=99%) [20:20:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643861 MB (35% inode=99%) [20:25:13] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643857 MB (35% inode=99%) [20:26:46] (03CR) 10BryanDavis: "V+1 via cherry-picked to striker-deploy03" [puppet] - 10https://gerrit.wikimedia.org/r/308359 (https://phabricator.wikimedia.org/T144040) (owner: 10BryanDavis) [20:27:34] (03CR) 10BryanDavis: "See I7e54c867211fea45258dfc6bae48467b43c8bb2f for the error pages that are installed." [puppet] - 10https://gerrit.wikimedia.org/r/308359 (https://phabricator.wikimedia.org/T144040) (owner: 10BryanDavis) [20:30:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643853 MB (35% inode=99%) [20:35:09] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643860 MB (35% inode=99%) [20:40:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643856 MB (35% inode=99%) [20:45:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643852 MB (35% inode=99%) [20:50:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643859 MB (35% inode=99%) [20:55:16] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 8 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643854 MB (35% inode=99%) [21:00:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643850 MB (35% inode=99%) [21:05:16] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643857 MB (35% inode=99%) [21:10:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643853 MB (35% inode=99%) [21:15:07] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643848 MB (35% inode=99%) [21:20:17] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643855 MB (35% inode=99%) [21:25:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643851 MB (35% inode=99%) [21:30:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643847 MB (35% inode=99%) [21:35:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643854 MB (35% inode=99%) [21:36:41] (03Abandoned) 10Volans: Reimaging: add option to reboot after the reimage [puppet] - 10https://gerrit.wikimedia.org/r/307482 (https://phabricator.wikimedia.org/T143536) (owner: 10Volans) [21:37:43] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 10.192.48.44 on port 6479 [21:40:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643850 MB (35% inode=99%) [21:40:13] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5146205 keys - replication_delay is 0 [21:45:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643845 MB (35% inode=99%) [21:50:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643853 MB (35% inode=99%) [21:55:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 5 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643848 MB (35% inode=99%) [22:00:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643844 MB (35% inode=99%) [22:05:10] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643851 MB (35% inode=99%) [22:09:42] 06Operations, 10fundraising-tech-ops: barium low on disk space - https://phabricator.wikimedia.org/T144659#2607135 (10Peachey88) [22:10:12] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643847 MB (35% inode=99%) [22:15:15] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643843 MB (35% inode=99%) [22:20:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643850 MB (35% inode=99%) [22:25:17] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643845 MB (35% inode=99%) [22:30:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643841 MB (35% inode=99%) [22:35:11] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643848 MB (35% inode=99%) [22:37:26] (03CR) 10Dereckson: [C: 04-1] "Let's wait a little bit, some discussions offer to distinguish two statuses: a editinterface only, a technician / technical administrator " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio) [22:40:07] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 719 MB (1% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643844 MB (35% inode=99%) [22:45:16] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 709 MB (1% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643839 MB (35% inode=99%) [22:50:14] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 717 MB (1% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643847 MB (35% inode=99%) [22:55:08] PROBLEM - check_disk on barium is CRITICAL: DISK CRITICAL - free space: / 5047 MB (9% inode=88%): /dev 7976 MB (99% inode=99%): /run 1558 MB (97% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 7985 MB (100% inode=99%): /archive 2803224 MB (98% inode=99%): /boot 202 MB (77% inode=99%): /archive/banner_logs 1643842 MB (35% inode=99%) [22:55:15] (03CR) 10Alex Monk: [C: 031] striker: Add custom error pages to apache vhost [puppet] - 10https://gerrit.wikimedia.org/r/308359 (https://phabricator.wikimedia.org/T144040) (owner: 10BryanDavis) [23:20:37] (03CR) 10Alex Monk: [C: 031] openstack: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308346 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox) [23:21:14] (03CR) 10Alex Monk: [C: 031] openstack: Fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308345 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)