[00:00:15] related: if i submit a patch, jenkins starts running the tests. if i +2 it, *don't wait for the commit tests to finish, abort them* [00:00:23] rather than wait for the test suite to finish just so you can run it again [00:00:56] hmmm, interesting, yeah [00:02:24] so, the one I skipped/didn't record ("make certain tests only run if a particular file is changed") I'm not sure how to record, maybe I'll just do the last suggestion (re composer.json) and then work from there [00:03:21] is there an RT ticket for better hardware? [00:03:36] I think so yeah, one sec [00:03:41] (03PS5) 10Dzahn: move racktables from misc to module [puppet] - 10https://gerrit.wikimedia.org/r/167903 [00:05:34] I don't think there are any tests specifically of the code in mw/vendor. Adding mw/vendor meant that it needed to be present for all tests so that the PSR-3 patch can land. [00:05:43] And there is something not quite optimal about the cloning for that as I recall. Like the old system cloned with hardlinks and now it doesn't. [00:06:13] (03CR) 10Ori.livneh: move racktables from misc to module (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/167903 (owner: 10Dzahn) [00:08:16] That being said, on a few random tests I'm looking at in Jenkins, the clone step was <5 seconds of a 6+ minute test run [00:09:04] ori: I can't find a bug nor rt re jenkins ram/hardware, though I remember there being movement on it (or I'm just being hopeful) [00:09:18] I mean, there's the general "add more jenkins slaves" one [00:09:36] ... [00:09:38] (03CR) 10Dzahn: move racktables from misc to module (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/167903 (owner: 10Dzahn) [00:09:42] https://bugzilla.wikimedia.org/show_bug.cgi?id=70049 [00:10:01] (03CR) 10Ori.livneh: [C: 031] move racktables from misc to module (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/167903 (owner: 10Dzahn) [00:10:04] whatever happened to the run all extension tests under HHVm together idea? [00:10:20] it wouldn't have worked, but it was a start in the right direction [00:26:46] greg-g: what's your wikitech name? [00:26:46] Greg Grossmeier [00:26:46] ^ [00:26:46] done [00:26:46] thanks muchly [00:26:47] (03CR) 10Dzahn: [C: 032] "yep, all these classes do is include webserver::base (already getting that from below), and installing the packages" [puppet] - 10https://gerrit.wikimedia.org/r/167999 (owner: 10Dzahn) [00:27:08] wtf, why do I still see nothing on that page? [00:28:14] you probably need to log out and log back in [00:28:14] heh /me sighs [00:28:15] you're right :/ [00:28:15] greg-g: an e-mail to engineering@ and wikitech@ saying "labs is oversubscribed and we badly need to have more jenkins slaves, could you take a look at your instances and see if there are any you don't need anymore" could help [00:28:16] idspispopd [00:28:24] idklev! [00:28:34] Coren: did you already send a similar email to labs-l like that ^ ? [00:29:06] not everyone reads labs-l [00:29:10] greg-g: We did before the migration. It helped a little. Then we lost one of the virt hosts. [00:29:15] * greg-g nods [00:29:17] right right [00:30:04] ori: will do tonight/tomorrow morning, I have to run now, other things added as tasks for antoine to see tonight, will hopefully get movement on those [00:30:13] thanks very much [00:30:16] thank you [00:30:24] hope i wasn't too big a jerk [00:30:36] nothing I can't handle ;) [00:30:53] * greg-g goes [00:35:37] (03PS1) 10Dzahn: move 'noc' from misc to module [puppet] - 10https://gerrit.wikimedia.org/r/168006 [00:39:17] Error: Could not find data item mediawiki_memcached_servers in any Hiera data file and no default supplied at /opt/wmf/software/compare-puppet-catalogs/external/change/168006/puppet/manifests/role/mediawiki.pp:26 on node terbium.eqiad.wmnet [00:39:29] seems so unrelated to the change i made, but happens to be on the same node [00:39:34] in compiler that is [00:40:54] yep, it already fails in "production" http://puppet-compiler.wmflabs.org/436/change/168006/compiled/puppet_catalogs_3_production/terbium.eqiad.wmnet.warnings [00:43:24] (03CR) 10Dzahn: "can't really run this one in compiler because it fails in the "production" run before this change. see here http://puppet-compiler.wmflabs" [puppet] - 10https://gerrit.wikimedia.org/r/168006 (owner: 10Dzahn) [00:44:56] (03CR) 10Dzahn: "unrelated Error: Could not find data item mediawiki_memcached_servers in any Hiera data file" [puppet] - 10https://gerrit.wikimedia.org/r/168006 (owner: 10Dzahn) [00:51:30] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 311 seconds [00:52:01] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 345 seconds [00:53:00] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -1 seconds [00:53:31] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [01:10:11] (03PS2) 10Dzahn: apply role::tor on radium [puppet] - 10https://gerrit.wikimedia.org/r/167161 [01:10:39] (03PS3) 10Dzahn: apply role::tor on radium [puppet] - 10https://gerrit.wikimedia.org/r/167161 [01:11:32] (03CR) 10Dzahn: [C: 032] apply role::tor on radium [puppet] - 10https://gerrit.wikimedia.org/r/167161 (owner: 10Dzahn) [01:12:33] \o/ [01:15:03] paravoid: :) almost, almost.. it applied all the stuff [01:15:05] Unknown option 'RunAsDemon'. Failing [01:15:59] typo, bah [01:16:56] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Puppet has 1 failures [01:18:35] (03PS1) 10Dzahn: tor - fix typo 'demon' -> 'daemon' [puppet] - 10https://gerrit.wikimedia.org/r/168012 [01:19:32] (03PS2) 10Dzahn: tor - fix typo 'demon' -> 'daemon' [puppet] - 10https://gerrit.wikimedia.org/r/168012 [01:20:29] "< mutante> typo, bah" +1 for linting/non-voting ;) [01:21:29] it wouldn't catch those though [01:21:43] it was inside an app config [01:22:25] (03CR) 10Dzahn: [C: 032] tor - fix typo 'demon' -> 'daemon' [puppet] - 10https://gerrit.wikimedia.org/r/168012 (owner: 10Dzahn) [01:23:10] mutante: :) [01:23:44] Nickname 'wikimedia-eqiad-1' is wrong length or contains illegal characters :p [01:23:56] hah [01:31:42] umm.. supposed to be allowed < opal> [A-Za-z_`[]{}\|][-0-9A-Za-z_`[]{}\|]{29} [01:39:31] (03PS1) 10Dzahn: tor - ensure /srv/tor exists/owned by debian-tor [puppet] - 10https://gerrit.wikimedia.org/r/168017 [01:45:46] (03PS2) 10Dzahn: tor - ensure /srv/tor exists/owned by debian-tor [puppet] - 10https://gerrit.wikimedia.org/r/168017 [01:45:53] (03PS1) 10Dzahn: tor - adjust nickname [puppet] - 10https://gerrit.wikimedia.org/r/168020 [01:46:17] (03PS2) 10Dzahn: tor - adjust nickname [puppet] - 10https://gerrit.wikimedia.org/r/168020 [01:51:41] (03CR) 10Dzahn: [C: 032] tor - ensure /srv/tor exists/owned by debian-tor [puppet] - 10https://gerrit.wikimedia.org/r/168017 (owner: 10Dzahn) [01:52:16] (03CR) 10Dzahn: [C: 032] tor - adjust nickname [puppet] - 10https://gerrit.wikimedia.org/r/168020 (owner: 10Dzahn) [01:58:27] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:48] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [02:01:36] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [02:07:12] (03PS1) 10Dzahn: tor - run as debian-tor user [puppet] - 10https://gerrit.wikimedia.org/r/168022 [02:07:17] (03CR) 10jenkins-bot: [V: 04-1] tor - run as debian-tor user [puppet] - 10https://gerrit.wikimedia.org/r/168022 (owner: 10Dzahn) [02:07:35] (03PS2) 10Dzahn: tor - run as debian-tor user [puppet] - 10https://gerrit.wikimedia.org/r/168022 [02:11:41] (03CR) 10Dzahn: [C: 032] tor - run as debian-tor user [puppet] - 10https://gerrit.wikimedia.org/r/168022 (owner: 10Dzahn) [02:11:49] (03PS1) 10Dzahn: tor - include passwords for control pass [puppet] - 10https://gerrit.wikimedia.org/r/168023 [02:12:06] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [02:12:24] (03PS2) 10Dzahn: tor - include passwords for control pass [puppet] - 10https://gerrit.wikimedia.org/r/168023 [02:13:31] (03CR) 10Dzahn: [C: 032] tor - include passwords for control pass [puppet] - 10https://gerrit.wikimedia.org/r/168023 (owner: 10Dzahn) [02:17:49] !log LocalisationUpdate completed (1.25wmf3) at 2014-10-22 02:17:49+00:00 [02:18:01] Logged the message, Master [02:18:27] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:21:46] PROBLEM - Disk space on ocg1002 is CRITICAL: DISK CRITICAL - free space: / 349 MB (3% inode=73%): [02:24:13] (03PS1) 10Dzahn: tor - remove obsolete config option 'Group' [puppet] - 10https://gerrit.wikimedia.org/r/168025 [02:24:19] (03CR) 10jenkins-bot: [V: 04-1] tor - remove obsolete config option 'Group' [puppet] - 10https://gerrit.wikimedia.org/r/168025 (owner: 10Dzahn) [02:24:24] (03PS2) 10Dzahn: tor - remove obsolete config option 'Group' [puppet] - 10https://gerrit.wikimedia.org/r/168025 [02:26:46] (03CR) 10Dzahn: [C: 032] "like https://bugzilla.redhat.com/show_bug.cgi?id=515822" [puppet] - 10https://gerrit.wikimedia.org/r/168025 (owner: 10Dzahn) [02:30:07] !log LocalisationUpdate completed (1.25wmf4) at 2014-10-22 02:30:07+00:00 [02:30:13] Logged the message, Master [02:30:45] (03PS1) 10Dzahn: tor - remove trailing / from DataDirectory [puppet] - 10https://gerrit.wikimedia.org/r/168027 [02:32:29] (03PS2) 10Dzahn: tor - remove trailing / from DataDirectory [puppet] - 10https://gerrit.wikimedia.org/r/168027 [02:37:58] (03CR) 10Dzahn: [C: 032] tor - remove trailing / from DataDirectory [puppet] - 10https://gerrit.wikimedia.org/r/168027 (owner: 10Dzahn) [02:52:24] (03PS1) 10Dzahn: tor - do not use /srv, let the package handle it [puppet] - 10https://gerrit.wikimedia.org/r/168031 [02:52:29] (03CR) 10jenkins-bot: [V: 04-1] tor - do not use /srv, let the package handle it [puppet] - 10https://gerrit.wikimedia.org/r/168031 (owner: 10Dzahn) [02:53:12] (03PS2) 10Dzahn: tor - do not use /srv, let the package handle it [puppet] - 10https://gerrit.wikimedia.org/r/168031 [02:53:57] PROBLEM - BGP status on cr2-ulsfo is CRITICAL: CRITICAL: No response from remote host 198.35.26.193 [02:54:33] (03CR) 10Dzahn: [C: 032] tor - do not use /srv, let the package handle it [puppet] - 10https://gerrit.wikimedia.org/r/168031 (owner: 10Dzahn) [02:55:47] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Puppet has 1 failures [02:56:48] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [02:56:57] RECOVERY - BGP status on cr2-ulsfo is OK: OK: host 198.35.26.193, sessions up: 42, down: 0, shutdown: 0 [03:15:54] (03Restored) 10Catrope: Clean up the mess that is SSL certificate installation [puppet] - 10https://gerrit.wikimedia.org/r/15561 (owner: 10Catrope) [03:21:16] PROBLEM - Disk space on ocg1001 is CRITICAL: DISK CRITICAL - free space: / 350 MB (3% inode=72%): [03:42:56] PROBLEM - Disk space on ocg1001 is CRITICAL: DISK CRITICAL - free space: / 350 MB (3% inode=72%): [03:52:06] PROBLEM - Disk space on ocg1003 is CRITICAL: DISK CRITICAL - free space: / 348 MB (3% inode=72%): [03:54:53] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 22 03:54:53 UTC 2014 (duration 54m 52s) [03:54:59] Logged the message, Master [03:57:16] PROBLEM - Disk space on ocg1003 is CRITICAL: DISK CRITICAL - free space: / 350 MB (3% inode=72%): [04:20:07] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [04:22:37] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: puppet fail [04:38:37] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [04:41:26] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [05:10:38] (03PS1) 10Ori.livneh: HHVM: install source to /usr/local/src/hhvm [puppet] - 10https://gerrit.wikimedia.org/r/168041 [05:13:28] (03CR) 10Ori.livneh: [C: 032] HHVM: install source to /usr/local/src/hhvm [puppet] - 10https://gerrit.wikimedia.org/r/168041 (owner: 10Ori.livneh) [05:17:47] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: puppet fail [05:18:48] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: puppet fail [05:19:39] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [05:19:48] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: puppet fail [05:19:49] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: puppet fail [05:23:32] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: puppet fail [05:23:33] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: puppet fail [05:25:24] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: puppet fail [05:28:12] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: puppet fail [05:31:45] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [05:32:14] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: puppet fail [05:33:14] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: puppet fail [05:34:15] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [05:34:24] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: puppet fail [05:34:26] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: puppet fail [05:36:14] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: puppet fail [05:39:47] (03PS1) 10Ori.livneh: hhvm: Make source file installation refreshonly [puppet] - 10https://gerrit.wikimedia.org/r/168044 [05:39:50] <_joe_> ori: why did you remove warmup.urls? [05:40:04] _joe_: it's in the mediawiki module now [05:40:12] <_joe_> oh ok :) [05:40:24] (03CR) 10Ori.livneh: [C: 032 V: 032] hhvm: Make source file installation refreshonly [puppet] - 10https://gerrit.wikimedia.org/r/168044 (owner: 10Ori.livneh) [05:40:46] <_joe_> Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter shell at /etc/puppet/modules/hhvm/manifests/init.pp:218 [05:41:35] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:42:35] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:44:15] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:46:19] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:50:55] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:51:15] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [05:51:15] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:53:25] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [05:53:26] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:55:18] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:56:05] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:56:25] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: puppet fail [05:57:07] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:57:46] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:59:05] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [05:59:15] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [06:01:10] (03CR) 10Nemo bis: [C: 031] "Looks ok on simple.wikipedia.wmflabs.org. http://pediapress.com/books/show/bdace4c4669bb7c42cfe40d818764e/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/167866 (https://bugzilla.wikimedia.org/71675) (owner: 10Cscott) [06:02:12] <_joe_> Nemo_bis: good morning [06:02:48] _joe_: mogge :) [06:14:11] PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: puppet fail [06:15:06] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:21:10] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [06:23:30] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: puppet fail [06:26:12] RECOVERY - Disk space on ocg1001 is OK: DISK OK [06:27:00] RECOVERY - Disk space on ocg1003 is OK: DISK OK [06:27:00] RECOVERY - Disk space on ocg1002 is OK: DISK OK [06:28:00] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: puppet fail [06:28:40] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: puppet fail [06:28:50] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [06:29:00] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: puppet fail [06:29:01] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [06:30:39] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:41] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:41] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:49] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:49] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:50] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:00] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:00] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:00] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:00] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:00] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:09] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:19] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:20] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:21] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:30] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:31] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:31] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:39] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:39] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:39] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:49] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:49] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:00] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:34] (03PS1) 10Yuvipanda: Add basic setup.py file [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168046 [06:32:35] legoktm: ^ CR? [06:32:38] python [06:33:22] (03CR) 10Yuvipanda: [C: 032 V: 032] Add basic setup.py file [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168046 (owner: 10Yuvipanda) [06:33:26] (03CR) 10Legoktm: Add basic setup.py file (031 comment) [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168046 (owner: 10Yuvipanda) [06:33:29] RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:33:45] (03CR) 10Legoktm: "-1" [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168046 (owner: 10Yuvipanda) [06:34:12] legoktm: bah, missed that [06:35:32] (03PS1) 10Yuvipanda: Fix URL to point to the proper git repository [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168047 [06:36:29] <_joe_> YuviPanda: what's shinkengen? [06:36:39] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: puppet fail [06:36:40] <_joe_> does it has to interact with puppet in any way? [06:37:02] (03PS1) 10Yuvipanda: Add Apache2 License [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168048 [06:37:12] _joe_: it's like naggen2, but for shinken. [06:37:46] _joe_: and no, it doesn't really interact with puppet, other than I'll probably have puppet exec it (or cron it) on the shinken host [06:37:51] it relies on the wikitech API [06:38:05] <_joe_> oh ok so it's specifc for labs [06:38:09] _joe_: yeah [06:38:11] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:38:14] <_joe_> maybe you can abstract the api part [06:38:25] <_joe_> and we'd be able to use it in prod as well :) [06:38:48] _joe_: yeah, I'm doing that next :) It expects 'instance' objects... [06:39:49] _joe_: but since we've no resource collection in labs, monitor_service, etc will need to be generated differently [06:40:12] _joe_: current thinking is that projects will define simple python functions that return Service objects based on hostname, projectname, puppet classes applied [06:40:20] <_joe_> YuviPanda: why we do not collect resources in python? [06:40:42] _joe_: we can't collect resources in labs. too many self hosted puppetmasters, plus it'll be slow, and also insecure since anyone can have root... [06:40:55] wait, what do you mean 'in python'? [06:42:10] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:43:41] There is some sort of a ui image missing : https://bits.wikimedia.org/w/images/ui-bg_highlight-soft_75_cccccc_1x100.png [06:44:15] it is called from some pages: and in the console: Failed to load resource: the server responded with a status of 404 (Not Found) https://bits.wikimedia.org/w/images/ui-bg_highlight-soft_75_cccccc_1x100.png [06:44:59] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:45:10] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:45:10] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:45:30] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:45:30] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:45:39] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:45:41] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:45:50] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:45:59] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:46:10] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:46:14] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:46:14] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:46:14] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:46:15] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:46:18] (03PS1) 10Yuvipanda: Setup generator as a script by itself [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168050 [06:46:21] matanya: did you see shinken now? [06:46:23] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:26] matanya: has host configs [06:46:39] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:46:42] I logged in on monday [06:46:48] it was empty :) [06:46:50] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:47:00] matanya: try shinken.wmflabs.org/all [06:47:09] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:47:10] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:47:10] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [06:47:20] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:47:34] deployment-soa-cache01 is down [06:47:38] that is all i see [06:47:39] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:47:50] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:47:50] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:50:46] matanya: in the 'all' tab? [06:50:50] yes [06:51:03] http://shinken.wmflabs.org/all?search=tools is all of toollabs, for example [06:51:06] hard refresh fail [06:51:59] http://shinken.wmflabs.org/user/login needs explanatory link [06:54:37] it needs a lot of love before it's useful. [06:54:45] like a MW OAuth auth provider... [07:07:10] (03PS1) 10Yuvipanda: Make sure setup.py picks up all packages [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168052 [07:07:41] (03CR) 10Yuvipanda: [C: 032 V: 032] Fix URL to point to the proper git repository [software/shinkengen] - 10https://gerrit.wikimedia.org/r/168047 (owner: 10Yuvipanda) [07:09:50] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 1 failures [07:27:02] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:31:43] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Puppet has 1 failures [07:35:46] (03PS2) 10Alexandros Kosiaris: Backup /var/lib/carbon/whisper on graphite [puppet] - 10https://gerrit.wikimedia.org/r/167817 [07:36:44] (03CR) 10Alexandros Kosiaris: [C: 032] torrus - remove subtree for eqiad-pmtpa link [puppet] - 10https://gerrit.wikimedia.org/r/167892 (owner: 10Dzahn) [07:36:57] (03CR) 10Alexandros Kosiaris: [C: 032] Backup /var/lib/carbon/whisper on graphite [puppet] - 10https://gerrit.wikimedia.org/r/167817 (owner: 10Alexandros Kosiaris) [07:37:22] (03PS3) 10Giuseppe Lavagetto: gerrit: move to module [puppet] - 10https://gerrit.wikimedia.org/r/167215 [07:39:47] (03CR) 10Giuseppe Lavagetto: "@Daniel: yes there is an issue with the puppet compiler and hiera I will take a look at today." [puppet] - 10https://gerrit.wikimedia.org/r/168006 (owner: 10Dzahn) [07:40:51] (03PS4) 10Giuseppe Lavagetto: gerrit: move to module [puppet] - 10https://gerrit.wikimedia.org/r/167215 [07:48:31] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:58:23] PROBLEM - DPKG on tungsten is CRITICAL: DPKG CRITICAL dpkg reports broken packages [08:00:40] RECOVERY - DPKG on tungsten is OK: All packages OK [08:00:40] (03PS2) 10Alexandros Kosiaris: backup ganglia data on the collector [puppet] - 10https://gerrit.wikimedia.org/r/167778 (owner: 10Filippo Giunchedi) [08:12:40] (03CR) 10Giuseppe Lavagetto: [C: 032] gerrit: move to module [puppet] - 10https://gerrit.wikimedia.org/r/167215 (owner: 10Giuseppe Lavagetto) [08:18:55] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/167310 (owner: 10Aaron Schulz) [08:23:02] (03CR) 10Filippo Giunchedi: [C: 04-1] "AFAICT x-timestamp is also used by object server during POST/DELETE/PUT to issue "409 conflict" if the timestamp on the object is more rec" [software] - 10https://gerrit.wikimedia.org/r/167828 (owner: 10Filippo Giunchedi) [08:24:55] (03CR) 10Filippo Giunchedi: [C: 031] Adding tools for banning/unbanning an ES node [puppet] - 10https://gerrit.wikimedia.org/r/164617 (owner: 10Chad) [08:26:06] (03CR) 10Filippo Giunchedi: [C: 031] Another es-tool function: restart a node the fast & easy way [puppet] - 10https://gerrit.wikimedia.org/r/164401 (owner: 10Chad) [08:27:39] (03CR) 10Filippo Giunchedi: [C: 031] sudo: create module, remove old files [puppet] - 10https://gerrit.wikimedia.org/r/167183 (owner: 10Giuseppe Lavagetto) [08:28:00] <_joe_> godog: I see admin::sudo is never used anywhere for groups [08:28:26] <_joe_> and btw, it's better to ditch the sudo::user and sudo::group defines completely [08:28:39] (03CR) 10Filippo Giunchedi: [C: 032] Introduce LLDP facts [puppet] - 10https://gerrit.wikimedia.org/r/167644 (owner: 10Alexandros Kosiaris) [08:29:26] (03CR) 10Alexandros Kosiaris: [C: 04-1] nfs.pp - remove pmtpa (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/167885 (owner: 10Dzahn) [08:33:57] _joe_: indeed doesn't seem to be used for groups [08:34:27] <_joe_> godog: we should, maybe [08:34:47] <_joe_> I'll work on it after I fix the hiera private/common.yaml fail [08:35:49] _joe_: yep, I think the missing bit is the validation, the rest looks straightforward [08:39:13] (03CR) 10Ori.livneh: "Many thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/167817 (owner: 10Alexandros Kosiaris) [08:39:41] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [08:51:56] (03PS4) 10Alexandros Kosiaris: Introduce LLDP facts [puppet] - 10https://gerrit.wikimedia.org/r/167644 [08:51:58] (03PS4) 10Alexandros Kosiaris: Introduce rack/rackrow facts based on LLDP facts [puppet] - 10https://gerrit.wikimedia.org/r/167645 [08:53:51] (03CR) 10Alexandros Kosiaris: [C: 032] Introduce LLDP facts [puppet] - 10https://gerrit.wikimedia.org/r/167644 (owner: 10Alexandros Kosiaris) [08:54:20] and lldp facts go live :-) [08:54:49] can't wait to cross refs some stuff with racktables :-) [08:56:48] (03PS1) 10Gilles: Prerender thumbnails at upload time on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168054 [08:58:13] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:58:21] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 13.33% of data above the critical threshold [500.0] [09:01:46] (03CR) 10Alexandros Kosiaris: [C: 032] backup ganglia data on the collector [puppet] - 10https://gerrit.wikimedia.org/r/167778 (owner: 10Filippo Giunchedi) [09:02:19] akosiaris: sweet, thanks! [09:02:45] :-) [09:02:59] (03PS1) 10Giuseppe Lavagetto: hiera: use the same lookup rules for private and public hiera dirs [puppet] - 10https://gerrit.wikimedia.org/r/168055 [09:06:58] !log Jenkins: upgrading gearman-plugin from 0.0.7-1-g3811bb8 to 0.1.0-1-gfa5f083 . Ie bring us to latest version + 1 commit [09:07:06] Logged the message, Master [09:07:13] (03CR) 10Alexandros Kosiaris: [C: 032] DHCP - delete unused linux-host-entry files [puppet] - 10https://gerrit.wikimedia.org/r/167865 (owner: 10Dzahn) [09:07:16] and I am going to restart jenkins [09:07:59] !log Restarting Jenkins [09:08:03] Logged the message, Master [09:12:01] PROBLEM - DPKG on gallium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [09:12:29] (03PS2) 10Alexandros Kosiaris: DHCP - remove Tampa Squid/LVS subnet [puppet] - 10https://gerrit.wikimedia.org/r/167862 (owner: 10Dzahn) [09:12:36] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] DHCP - remove Tampa Squid/LVS subnet [puppet] - 10https://gerrit.wikimedia.org/r/167862 (owner: 10Dzahn) [09:13:00] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [09:16:10] RECOVERY - DPKG on gallium is OK: All packages OK [09:22:24] (03PS2) 10Giuseppe Lavagetto: hiera: use the same lookup rules for private and public hiera dirs [puppet] - 10https://gerrit.wikimedia.org/r/168055 [09:38:56] (03PS3) 10Giuseppe Lavagetto: hiera: use the same lookup rules for private and public hiera dirs [puppet] - 10https://gerrit.wikimedia.org/r/168055 [09:39:07] <_joe_> godog: mind to take a look? ^^ [09:39:18] <_joe_> this solves the private/common fiasco [09:40:04] _joe_: yep I'll take a look [09:40:57] remembers me I am overdue with catching up with Hiera :( [09:41:50] !log Zuul/Jenkins in a deadlock [09:41:57] Logged the message, Master [09:45:46] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Please put manifests in autoload layout." [puppet] - 10https://gerrit.wikimedia.org/r/167713 (owner: 10Andrew Bogott) [10:05:58] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM, please add an example to the commit message so it is clear what the intent is" [puppet] - 10https://gerrit.wikimedia.org/r/168055 (owner: 10Giuseppe Lavagetto) [10:35:48] (03PS3) 10Giuseppe Lavagetto: sudo: create module, remove old files [puppet] - 10https://gerrit.wikimedia.org/r/167183 [10:37:07] (03PS4) 10Giuseppe Lavagetto: sudo: create module, remove old files [puppet] - 10https://gerrit.wikimedia.org/r/167183 [10:51:02] _joe_: hey [10:51:08] _joe_: https://wikitech.wikimedia.org/wiki/Puppet_coding#Puppet_Modules + https://wikitech.wikimedia.org/wiki/Puppet_Todo [10:51:16] they're a bit outdated now I think [10:51:38] but in any case, I don't think we should have a "sudo::appserver" class (for example) [10:51:57] <_joe_> me neither, honestly [10:52:02] the contents of that (that single File) fits better within the appserver module [10:52:04] <_joe_> I am deleting the whole thing [10:52:08] <_joe_> yes [10:53:07] <_joe_> so I'm going to: 1) modify admin::sudo so that it DTRT with groups too 2) convert all sudo_user and sudo_group declarations to use admin::sudo 3) move those files to where they belong [10:53:41] <_joe_> 1) is not problematic as admin::sudo has never been used with is_group :) [10:53:59] I think we've discussed this before [10:54:17] <_joe_> I missed that discussion I guess [10:54:34] and i think the result of that discussion was that "admin" should be kept for granting access (sudo too) to humans [10:54:43] <_joe_> ok [10:54:47] but dunno how it is these days, chase would know better I think [10:55:00] <_joe_> so we may remove the is_group clause [10:55:39] <_joe_> but hey, I'll check with chase as well on that, putting that on hold. [11:03:51] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: puppet fail [11:21:56] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [11:43:38] (03PS5) 10Giuseppe Lavagetto: sudo: create module, remove old files [puppet] - 10https://gerrit.wikimedia.org/r/167183 [11:48:17] (03PS1) 10Giuseppe Lavagetto: role::labs::instance: include sudo::labs_project [puppet] - 10https://gerrit.wikimedia.org/r/168062 [11:53:24] (03PS4) 10Giuseppe Lavagetto: hiera: use the same lookup rules for private and public hiera dirs [puppet] - 10https://gerrit.wikimedia.org/r/168055 [11:53:48] <_joe_> godog: ^^ [11:56:28] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet has 1 failures [11:57:20] _joe_: I'm off to lunch but will take a look afterwards! [11:57:38] <_joe_> godog: yes, not in a hurry actually [11:57:57] <_joe_> I'll be off for a couple of hours after lunch [11:58:06] <_joe_> (I'm off as well) [12:00:27] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [12:11:01] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [12:18:04] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [12:22:20] greg-g: ori: I agree with the idea with not running non-voting jobs, or even not having such a thing in the first place. Imho they should only exist when CI is debugging something or when we're looking to add a new job that isn't passing just yet while we merge commits that make it passing (e.g. enable a new style rule) It would never exist for more than a day. [12:22:26] But I know hashar doesn't agree. [12:22:54] if it's something not passing any time soon, kill it and have whoever is interested work on it themselves and let us know when it's ready. [12:23:26] Krinkle: hi :) you probably want to follow up on the related task greg-g opened https://phabricator.wikimedia.org/T794 [12:25:37] hashar: I am currently not using phabricator in my workflow. And am not really tempted to right now. [12:25:47] Does it have the concept of components inside projects? [12:26:01] Or projects inside other projects? [12:26:56] do whatever you want [12:27:28] I am not going to attempt to follow the discussion via IRC though :D [12:27:41] hashar: I see you raised the same concern (see email) [12:27:50] We need a CI grouping of sorts [12:28:09] No, projects in Phab are a flat namespace. Subprojects are not (yet) supported in Phabricator: https://secure.phabricator.com/T3670 [12:28:13] If that doesn't happen soon I'm tempted to move whatever stuff ended up in phab/releng back to bugzilla. [12:28:18] Why are there things for it on phab already? [12:28:22] Afaik we didn't move CI yet. [12:28:29] I don't want it in to places, what's the point in that? [12:28:40] Or did we import CI? That's fine by me, as long as its not in two places. [12:29:13] andre__: So what are we goign to do with mediawiki core components in bugzilla/phab? [12:29:27] I think it's an definite no-go without that kind of separation, I can't do my job otherwise. [12:29:31] Krinkle, migrate them into projects. [12:29:41] Right, that sounds good. [12:29:44] prefixes or something [12:29:56] (and you even more so of course!) [12:30:21] Project-Component, like MediaWiki-Page_editing [12:30:48] That does mean it's harder to get a grip on wider scopes for you [12:30:50] Krinkle: bring the subject to Greg. I am not sure why he filled the tasks in Phabricator instead of using Bugzilla which prompted my email this morning about how to handle CI inside Phabricator [12:31:01] andre__: unless there's a way to do MediaWiki-* [12:31:08] I guess he filled the tasks in Phabricator so we can track them in the Releng team workboard [12:31:10] which is fine to me [12:31:11] andre__: do you know of such a way? [12:31:17] andre__: I guess you'll want that for triaging and sorting [12:31:18] Krinkle, define "do". [12:31:23] not sure what the question is [12:31:31] (03CR) 10Giuseppe Lavagetto: [C: 032] hiera: use the same lookup rules for private and public hiera dirs [puppet] - 10https://gerrit.wikimedia.org/r/168055 (owner: 10Giuseppe Lavagetto) [12:31:44] andre__: If we map components to MEdiaWiki+component as phab product, how will you query all of Mediawiki product? [12:32:01] short of manually selecting them from the global project list which is gonna be huge [12:32:10] (03PS1) 10RobH: granting jkatz bastion and statistics-user access [puppet] - 10https://gerrit.wikimedia.org/r/168066 [12:32:19] Krinkle, by entering all subprojects in a query. And after doing that once you'll quickly learn to save the URL for that query. [12:32:23] I see a ticket can be in multiple projects, that's nice. [12:32:25] The other idea would be umbrella projects. [12:32:47] so we'd manually add tickets in MediaWiki-* projects to the MediaWiki umbrella project, too. [12:32:56] but there's no simple way to say "if task is in project MediaWiki-Page_editing also automatically add it to umbrella project MediaWiki" [12:33:03] ugh yeah [12:33:12] basically it's the same discussion about categories on Wikipedia or Commons [12:33:30] "Female writers" && "Male writers" vs. "Writers" with sub-categories [12:33:33] Pick your poison. [12:33:53] andre__: Have we settled on what we'll do for our phab instance? e.g. whehter to bother with umbrella tracking manually, or require interested users to do a query for all subprojects. [12:34:05] andre__: yeah, due to lack of recursive and intersection. [12:34:09] (03CR) 10RobH: [C: 032] "His three day wait started on 2014-10-15, so this is overdue." [puppet] - 10https://gerrit.wikimedia.org/r/168066 (owner: 10RobH) [12:34:12] Also Female humans and Male humans [12:34:16] and humans [12:34:24] intersect with Writers and Female humans [12:34:26] etc. [12:34:30] I guess that will depends on each team :D [12:34:41] Krinkle, "query for all subprojects" for the time being. [12:34:45] OK [12:35:17] for CI I guess we could get the major tasks / tracking tasks to be flagged with both Releng team project and contint project [12:35:35] andre__: So for Rel Eng, we'd have a CI product as well. But that does mean for workflow boards (like https://phabricator.wikimedia.org/tag/release-engineering/board/ ) we can't add them to the board of a "Parent" project, since it's not really in that project [12:35:38] and have the minor tasks / subtasks just flagged with "contint" so as to not pollute the "parent" project workboard [12:35:41] andre__: maybe wed' need umbrella for that [12:36:03]