[00:01:38] !log ori Synchronized php-1.26wmf16/includes: Revert I4afaecd8: "Avoiding writing sessions for no reason", and undo several uncommitted live-hacks for debugging T102199 (duration: 00m 16s) [00:01:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:03:19] 6operations, 10RESTBase, 6Services, 10Traffic, 5Patch-For-Review: Provide an API listing at /api/ - https://phabricator.wikimedia.org/T107086#1500215 (10GWicke) [00:06:07] 6operations, 10RESTBase, 6Services, 10Traffic, 5Patch-For-Review: Provide an API listing at /api/ - https://phabricator.wikimedia.org/T107086#1500226 (10GWicke) There is now a bare-bones listing template at https://meta.wikimedia.org/wiki/API_listing_template. This page is already protected, and will be... [00:11:12] (03CR) 10Ori.livneh: [C: 032] Add an API listing template to the allowed templates in extract2.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228429 (https://phabricator.wikimedia.org/T107086) (owner: 10GWicke) [00:11:17] (03Merged) 10jenkins-bot: Add an API listing template to the allowed templates in extract2.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228429 (https://phabricator.wikimedia.org/T107086) (owner: 10GWicke) [00:12:16] !log ori Synchronized extract2.php: Ie919881a4: Add an API listing template to the allowed templates in extract2.php [00:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:20:08] !log l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s) [02:20:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:23:15] !log @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-01 02:23:15+00:00 [02:23:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:23:57] PROBLEM - puppet last run on wtp2009 is CRITICAL puppet fail [02:50:36] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 744.48200669 [02:51:58] RECOVERY - puppet last run on wtp2009 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [03:03:47] PROBLEM - puppet last run on mw1184 is CRITICAL Puppet has 1 failures [03:22:56] PROBLEM - Disk space on labcontrol1001 is CRITICAL: DISK CRITICAL - free space: / 1632 MB (3% inode=94%) [03:29:37] RECOVERY - puppet last run on mw1184 is OK Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:52:48] RECOVERY - Disk space on labcontrol1001 is OK: DISK OK [03:53:31] !log cleared out nova-conductor.log on labcontrol1001, restarted nova-conductor, graceful’d apache [03:53:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:57:34] (03Abandoned) 10Andrew Bogott: remove_unused_base_images => True [puppet] - 10https://gerrit.wikimedia.org/r/228425 (owner: 10Andrew Bogott) [04:13:47] PROBLEM - nova-compute process on labvirt1005 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/nova-compute [04:15:56] RECOVERY - nova-compute process on labvirt1005 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/nova-compute [05:06:46] !log @tin ResourceLoader cache refresh completed at Sat Aug 1 05:06:46 UTC 2015 (duration 6m 45s) [05:06:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [05:57:17] PROBLEM - Disk space on mw1114 is CRITICAL: DISK CRITICAL - free space: / 8175 MB (3% inode=93%) [06:02:31] <_joe_> uh our pet api server has a full disk [06:02:35] <_joe_> let's see [06:04:57] <_joe_> !log removing some old apache access logs from mw1114 [06:05:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [06:05:17] RECOVERY - Disk space on mw1114 is OK: DISK OK [06:32:17] PROBLEM - puppet last run on db2064 is CRITICAL Puppet has 1 failures [06:32:27] PROBLEM - puppet last run on wtp2017 is CRITICAL Puppet has 1 failures [06:32:27] PROBLEM - puppet last run on mw2016 is CRITICAL Puppet has 1 failures [06:32:28] PROBLEM - puppet last run on db1056 is CRITICAL Puppet has 1 failures [06:32:37] PROBLEM - puppet last run on lvs1003 is CRITICAL Puppet has 2 failures [06:33:06] PROBLEM - puppet last run on mw1170 is CRITICAL Puppet has 1 failures [06:33:07] PROBLEM - puppet last run on wtp2008 is CRITICAL Puppet has 1 failures [06:33:08] PROBLEM - puppet last run on mw2207 is CRITICAL Puppet has 1 failures [06:33:16] PROBLEM - puppet last run on mw2043 is CRITICAL Puppet has 1 failures [06:33:47] PROBLEM - puppet last run on mw2050 is CRITICAL Puppet has 1 failures [06:56:07] RECOVERY - puppet last run on db2064 is OK Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:56:16] RECOVERY - puppet last run on db1056 is OK Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:56:18] RECOVERY - puppet last run on lvs1003 is OK Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:56:57] RECOVERY - puppet last run on wtp2008 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:57] RECOVERY - puppet last run on mw2207 is OK Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:56:58] RECOVERY - puppet last run on mw2043 is OK Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:57:37] RECOVERY - puppet last run on mw2050 is OK Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:58:07] RECOVERY - puppet last run on wtp2017 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:07] RECOVERY - puppet last run on mw2016 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:47] RECOVERY - puppet last run on mw1170 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [08:05:18] https://commons.wikimedia.org/w/index.php?limit=100&tagfilter=&title=Special%3AContributions&contribs=user&target=Allforrous&namespace=3&tagfilter=&year=2015&month=-1 [08:05:31] Function: IndexPager::buildQueryInfo (contributions page filtered for namespace or RevisionDeleted edits) [08:05:31] Error: 2013 Lost connection to MySQL server during query (10.64.16.8) [08:07:21] rip wiki [08:07:26] should I open a bug report? ^ [08:21:49] 6operations, 6Discovery, 10Traffic, 10Wikidata, and 2 others: Set up a public interface to the wikidata query service - https://phabricator.wikimedia.org/T107602#1500457 (10Legoktm) >>! In T107602#1499998, @Smalyshev wrote: > The service does not need to access them but I'm not sure how we can avoid them b... [08:25:07] legoktm: you online? or a other sysadmin? [08:25:48] apergos? ? :) [08:26:51] Steinsplitter: hi [08:27:08] legoktm: i am going to rename a user with 60000 edits [08:27:14] User:Ykt www [08:27:16] ok? [08:27:16] uhh [08:27:29] it's 1:30am here and I was going to sleep soon :( [08:28:00] I'd rather wait until monday if that's okay? there will generally be more people online then in case something goes wrong [08:28:13] the last time i renamed useres with nerar 100000 never was a problem [08:28:14] ok :( [08:29:36] most likely nothing will go wrong, but if something does go wrong, it's a giant mess to clean up [08:29:47] and I'd like to be fully awake if I have to do that :P [08:30:41] ok [08:31:29] hoo is always around on weekend and familar with this stuff. if he is around will ask im. so you don't have to do nothing :) [08:33:51] thanls aniway, legoktm. And sleep well :) [08:34:04] :) [10:24:46] PROBLEM - puppet last run on cp3013 is CRITICAL puppet fail [10:50:56] RECOVERY - puppet last run on cp3013 is OK Puppet is currently enabled, last run 0 seconds ago with 0 failures [11:28:27] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 42.86% of data above the critical threshold [500.0] [11:40:37] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [13:23:42] (03PS1) 10Dereckson: Import sources for mr.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228475 (https://phabricator.wikimedia.org/T105116) [14:04:58] (03PS1) 10Dereckson: Logo on mr.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228477 (https://phabricator.wikimedia.org/T104132) [14:11:07] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 28.57% of data above the critical threshold [500.0] [14:19:47] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL Anomaly detected: 11 data above and 9 below the confidence bounds [14:33:17] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [14:43:22] (03PS1) 10BBlack: ipsec: use shorter conn names for readability [puppet] - 10https://gerrit.wikimedia.org/r/228483 [14:43:24] (03PS1) 10BBlack: refactor/improve strongswan icinga check [puppet] - 10https://gerrit.wikimedia.org/r/228484 [15:06:04] (03PS2) 10BBlack: refactor/improve strongswan icinga check [puppet] - 10https://gerrit.wikimedia.org/r/228484 [15:19:57] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected [15:38:39] (03CR) 10BBlack: [C: 032] ipsec: use shorter conn names for readability [puppet] - 10https://gerrit.wikimedia.org/r/228483 (owner: 10BBlack) [15:38:50] (03CR) 10BBlack: [C: 032] refactor/improve strongswan icinga check [puppet] - 10https://gerrit.wikimedia.org/r/228484 (owner: 10BBlack) [15:59:25] (03PS1) 10BBlack: check_strongswan: pipe symbol is reserved for icinga perf data [puppet] - 10https://gerrit.wikimedia.org/r/228489 [15:59:52] (03CR) 10BBlack: [C: 032 V: 032] check_strongswan: pipe symbol is reserved for icinga perf data [puppet] - 10https://gerrit.wikimedia.org/r/228489 (owner: 10BBlack) [16:16:21] (03PS1) 10BBlack: check_strongswan: sort failure lists [puppet] - 10https://gerrit.wikimedia.org/r/228490 [16:16:43] (03CR) 10BBlack: [C: 032 V: 032] check_strongswan: sort failure lists [puppet] - 10https://gerrit.wikimedia.org/r/228490 (owner: 10BBlack) [16:33:40] 6operations, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: Support VP9 in TMH (Unable to decode) - https://phabricator.wikimedia.org/T55863#1500737 (10McZusatz) @fgiunchedi Would you mind hitting the ffmpeg command once again with another video like * https://commons.wikimedia.org/wiki/File:NASA_-_... [17:18:17] PROBLEM - puppet last run on eventlog1001 is CRITICAL Puppet has 1 failures [17:42:27] RECOVERY - puppet last run on eventlog1001 is OK Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:26:01] 6operations, 6Labs, 3Labs-Sprint-107, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1500789 (10Andrew) Reboot of labvirt1009 is now scheduled and announced for Wednesday. [19:10:06] PROBLEM - Host ripe-atlas-ulsfo is DOWN: PING CRITICAL - Packet loss = 100% [19:10:36] 7Puppet, 6operations: Move otrs into a module - https://phabricator.wikimedia.org/T107670#1500826 (10scfc) 3NEW [19:13:36] 7Puppet, 6operations: Move udp2log into a module - https://phabricator.wikimedia.org/T107671#1500833 (10scfc) 3NEW [19:15:00] 7Puppet, 6operations: Move misc::maintenance into a module - https://phabricator.wikimedia.org/T107672#1500840 (10scfc) 3NEW [19:15:30] 7Puppet, 6operations: Move role::otrs into a module - https://phabricator.wikimedia.org/T107670#1500847 (10scfc) [19:15:47] 7Puppet, 6operations: Move misc::udp2log into a module - https://phabricator.wikimedia.org/T107671#1500849 (10scfc) [19:28:57] PROBLEM - dhclient process on analytics1044 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:29:06] PROBLEM - salt-minion processes on analytics1044 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:29:27] PROBLEM - Hadoop DataNode on analytics1044 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:30:16] PROBLEM - Hadoop NodeManager on analytics1044 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:50:57] PROBLEM - HHVM rendering on mw1157 is CRITICAL - Socket timeout after 10 seconds [19:52:46] RECOVERY - HHVM rendering on mw1157 is OK: HTTP OK: HTTP/1.1 200 OK - 66706 bytes in 0.115 second response time [19:56:40] (03PS1) 10BBlack: tlsproxy: let nginx use keepalives to varnish [puppet] - 10https://gerrit.wikimedia.org/r/228564 [21:15:47] 6operations, 6Parsing-Team, 10Parsoid-Nowiki: Cleanup redundant - https://phabricator.wikimedia.org/T107675#1500893 (10eranroz) 3NEW [22:31:27] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.69% of data above the critical threshold [500.0] [22:51:27] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [23:13:27] PROBLEM - puppet last run on cp3033 is CRITICAL puppet fail [23:19:36] PROBLEM - puppet last run on ruthenium is CRITICAL puppet fail [23:30:50] andrewbogott: https://etherpad.wikimedia.org/p/phab_roles gives the basic gist on the phab's [23:31:22] I can clean up a few when the phab-pup patch completes [23:39:36] RECOVERY - puppet last run on cp3033 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:17] so I am unsure if this is right place to ask this but, whats the procedure for closed wikis? [23:45:36] RECOVERY - puppet last run on ruthenium is OK Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:54:41] ToAruShiroiNeko: in most cases they are just closed for editing [23:55:03] ToAruShiroiNeko: I found https://wikitech.wikimedia.org/wiki/Close_a_wiki -- that has the technical changes needed. I'm not sure what consensus is needed to initiate the shell request