[00:05:59] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 1 failures [00:21:34] (03PS7) 10BryanDavis: [WIP] Provision Striker via scap3 [puppet] - 10https://gerrit.wikimedia.org/r/301505 (https://phabricator.wikimedia.org/T141014) [00:31:47] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:46:07] PROBLEM - Host db2069 is DOWN: PING CRITICAL - Packet loss = 100% [01:26:28] 06Operations, 10Ops-Access-Requests: Requesting access to deployment access for Niharika - https://phabricator.wikimedia.org/T141593#2504190 (10Niharika) [01:59:24] 07Blocked-on-Operations, 06Operations, 10Kartographer, 10Wikimedia-Extension-setup, and 4 others: Enable Interactive Maps (Kartographer) on Macedonian Wikipedia - https://phabricator.wikimedia.org/T139946#2504255 (10Yurik) I thought android and iphone apps can use javascript, but i could be wrong. This is... [02:20:12] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 07m 30s) [02:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:24:18] PROBLEM - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.0.24 [02:26:08] RECOVERY - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms [02:26:09] !log l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 29 02:26:09 UTC 2016 (duration 5m 57s) [02:26:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:44:18] PROBLEM - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.0.24 [03:46:08] RECOVERY - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms [03:58:09] ffs [04:04:04] !log legoktm@tin Synchronized php-1.28.0-wmf.12/extensions/TorBlock/extension.json: Move basic torunblocked line to GrantPermissions, not GroupPermissions, see wikitech-l (duration: 00m 38s) [04:04:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [04:10:47] PROBLEM - puppet last run on mw2097 is CRITICAL: CRITICAL: Puppet has 1 failures [04:38:09] RECOVERY - puppet last run on mw2097 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [05:19:27] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 1 failures [05:43:49] (03PS1) 10MaxSem: Labs: remove duplicate $wgFlowParsoidURL assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301743 [05:43:51] (03PS1) 10MaxSem: Labs: remove MobileApp inclusion, duplicates prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301744 [05:43:53] (03PS1) 10MaxSem: Labs: remove wgCentralAuthEnableUserMerge - matches the default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301745 [05:46:38] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:50:07] (03PS1) 10MaxSem: Remove temporary wgCentralAuthEnableUserMerge override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301746 [06:08:03] (03PS1) 10MaxSem: Labs: remove $wgCentralGeoScriptURL - matches prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301747 [06:08:05] (03PS1) 10MaxSem: Labs: remove $wgCentralDBname - matches prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301748 [06:08:07] (03PS1) 10MaxSem: Labs: remove experimental $wgGadgetsCaching override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301749 [06:30:18] PROBLEM - puppet last run on ms-be2026 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:17] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:58] PROBLEM - puppet last run on cp2013 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:58] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:28] PROBLEM - puppet last run on es2013 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:09] PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:50] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:06] 06Operations, 10DBA, 10Incident-20150205-SiteOutage, 05Wikimedia-Incident: sleeper database connection surges during outage - https://phabricator.wikimedia.org/T88770#2504411 (10jcrespo) 05Open>03Resolved This is resolved: 1) All important boxes are using mariadb10 and pool of connections 2) There are... [06:49:57] 06Operations, 10ops-codfw, 10DBA: db2069 is down - https://phabricator.wikimedia.org/T141601#2504417 (10jcrespo) [06:52:32] 06Operations, 10ops-codfw, 10DBA: db2069 is down - https://phabricator.wikimedia.org/T141601#2504429 (10jcrespo) [06:56:27] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:57:08] RECOVERY - puppet last run on cp2013 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:57:08] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:57:20] 06Operations, 10ops-codfw, 10DBA: db2069 crashed due to RAID controller - https://phabricator.wikimedia.org/T141601#2504431 (10jcrespo) [06:57:38] RECOVERY - puppet last run on es2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:39] RECOVERY - puppet last run on ms-be2026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:39] 06Operations, 10ops-codfw, 10DBA: db2069 crashed due to RAID controller - https://phabricator.wikimedia.org/T141601#2504417 (10jcrespo) And I was right: ``` hpiLO-> show record5 status=0 status_tag=COMMAND COMPLETED Fri Jul 29 06:35:23 2016 /system1/log1/record5 Targets Properties... [06:58:08] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:58:27] RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:09] !log powercycling db2069 T141601 [06:59:10] T141601: db2069 crashed due to RAID controller - https://phabricator.wikimedia.org/T141601 [06:59:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:02:28] RECOVERY - Host db2069 is UP: PING OK - Packet loss = 0%, RTA = 38.21 ms [07:14:20] (03PS3) 10ArielGlenn: tiny script that retrieves config values from dump config files [dumps] - 10https://gerrit.wikimedia.org/r/301712 (https://phabricator.wikimedia.org/T141563) [07:17:56] hashar: around ? [07:18:01] good morning btw [07:18:10] <_joe_> morning [07:19:10] 06Operations, 10Ops-Access-Requests: Requesting access to stat1003.eqiad.wmnet for WMDE-jand - https://phabricator.wikimedia.org/T141339#2504441 (10Jan_Dittrich) @Abraham: Can you approve this request? [07:24:49] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-urd] - 10https://gerrit.wikimedia.org/r/296229 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [07:24:56] !log fixing s3 replication lag created by TokuDB insert problem [07:25:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:30:18] !log schema change continues for s2, s1, s4 and s5 T140108 [07:30:19] T140108: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108 [07:30:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:39:24] akosiaris: good morning yeah around [07:39:45] I went crazy yesterday night and found a hack to set BUILDRESULT on jenkins :D [07:46:38] PROBLEM - puppet last run on cp2021 is CRITICAL: CRITICAL: Puppet has 1 failures [07:47:31] hashar: I am facing a different problem [07:47:37] at least trying to right now [07:47:47] so, https://integration.wikimedia.org/ci/job/debian-glue/374/consoleFull [07:48:01] 07:25:03 ++ dpkg-parsechangelog --show-field distribution -lsource/debian/changelog [07:48:01] 07:25:03 + export distribution=trusty [07:48:16] but !! https://gerrit.wikimedia.org/r/#/c/296229/2/debian/changelog [07:48:32] seems like /usr/bin/generate-git-snapshot is doing something to the tree [07:48:44] I am trying to get SKIP_DCH to work, but it doesn't [07:49:06] http://jenkins-debian-glue.org/docs/#customization for SKIP_DCH btw [07:49:12] so I am kind of stumped... [07:49:15] that distribution=dpkg-parsechangelog is from me / in the job [07:49:19] that is to auto select the dist [07:49:30] else it default to the distribution of the host system (ie jessie) [07:49:31] well, it parses a wrong version of the file [07:49:36] ah [07:50:07] the call is correct, but source seems to be changed by generate-git-snapshot [07:50:23] ah https://gerrit.wikimedia.org/r/#/c/296229/2/debian/changelog [07:50:39] so it does not build against the proposed patchset but against the tip of the branch I guess (which is at trusty) [07:51:08] I think so too, not sure why though [07:51:20] it's clearly generate-git-snapshot doing it. I am managed to reproduce it [07:51:27] it must checkout at some point [07:51:38] I 've* [07:52:00] it also does a dch --something to change the package version and inject the date-time-git_commit [07:53:15] * hashar reads the console output [07:54:48] so I got a question. why is https://phabricator.wikimedia.org/diffusion/CICF/browse/master/jjb/operations-debs.yaml 3 different shell builders ? I was wondering [07:55:19] cause if we could move the export distribution=$(dpkg-parsechangelog --show-field distribution -lsource/debian/changelog) call before the /usr/bin/generate-git-snapshot call, my problem would be solved [07:55:31] but I doub't that's possible because they are in 2 different shell builders [07:55:41] unless I misunderstand something [07:55:53] OHHH YOU ARE SO RIGHT [07:55:56] yeah [07:56:04] so each shell builder comes with a fresh env [07:56:25] so whatever env variables that have been set by loading pbuilderrc etc is effectively gone in the piuparts and lintian ones [07:56:54] same for generate-git-snapshot vs build-and-provide-package [07:58:06] well the second builder (the one containing generate-git-snapshot does really set many env variablies. Just 2. maybe it's fine to merge it with the next one and do the move I suggested [07:58:21] does NOT really set many env variables* [07:58:41] yeah make sense [07:58:51] you can tell I have not put lot of effort in polishing that job :D [07:59:01] ok, lemme submit a patch [07:59:37] also generate-git-snapshot works on a temporary branch such as jenkins-debian-glue-buildbranch29416 (was c83ee0f). [07:59:42] and then git checkout -f master [08:00:18] there is no good indication whether that master branch has been reset to origin. But "Your branch is up-to-date with 'origin/master'." looks suspicious [08:04:18] RECOVERY - puppet last run on cp2021 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:08:51] akosiaris: I gotta land https://gerrit.wikimedia.org/r/#/c/301714/1/jjb/operations-debs.yaml [08:09:16] it crafts a new pbuilderrc based on the /etc one, and append BUILDRESULT to it then point debian glue to it [08:09:21] proven to work :] [08:09:45] I am merging it, will polish it up later [08:10:02] hashar: fine by me [08:10:08] it is a terrible hack [08:10:32] and thank you a ton to have figured out that cowbuilder --configfile parameter can have previous parameter overriden [08:10:43] yeah that was a small shock [08:10:49] I believe it is working as intended. Namely to let you tweak the build parameter via pbuilderrc [08:10:50] not that I haven't seen it before [08:11:22] so if you have a software hardcoded with cowbuilder --buildresult /tmp/foo , you can just BUILDRESULT=/proper/place [08:11:24] which is quite handy [08:13:07] akosiaris: my change merged, you will have to rebase [08:16:26] I have added a couple "git branch -vva" around generate-git-snapshot [08:17:01] 00:00:01.305 * (no branch) c83ee0f apertium-urd: New upstream release and rebuild for Jessie [08:17:02] 00:00:01.306 master e4b3ace [origin/master] Merge tag 'upstream/0.1.0_r61311' [08:17:04] so in short [08:17:20] zuul-cloner checkout the patch in an unamed branch :/ [08:17:42] leaving "master" untouched [08:18:08] PROBLEM - puppet last run on mw2163 is CRITICAL: CRITICAL: puppet fail [08:19:14] I suppose that's standard behaviour though ? [08:19:26] we use zuul-cloner everywhere, right ? [08:19:57] mostly yeah [08:20:03] well for debian glue jobs yeah everywhere [08:20:12] cause it has the support to clone from the canonical repo (gerrit) [08:20:15] 06Operations, 10DBA, 10Gerrit, 06Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2504553 (10jcrespo) @Dzahn @demon can db1042 be wiped? [08:20:17] and then auto apply the patch [08:20:24] but the checkout part should really checkout to the local branch [08:24:50] 06Operations: eqiad: Install SSD's into ganeti hosts - https://phabricator.wikimedia.org/T138414#2504571 (10akosiaris) [08:24:51] will have to fix zuul-cloner [08:25:04] 06Operations: eqiad: Install SSD's into ganeti hosts - https://phabricator.wikimedia.org/T138414#2399490 (10akosiaris) [08:25:10] meanwhile I think we can do: git checkout -f $GIT_BRANCH && git reset --hard $GIT_COMMIT [08:25:48] PROBLEM - ganeti-confd running on ganeti1002 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 111 (gnt-confd), command name ganeti-confd [08:25:56] 06Operations: eqiad: Install SSD's into ganeti hosts - https://phabricator.wikimedia.org/T138414#2399490 (10akosiaris) ganeti1004 has been fully reintegrated into the cluster. The migration of VMs back to it from the rest of the cluster took quite a long time as there was quite a lot of data copying [08:26:28] PROBLEM - ganeti-noded running on ganeti1002 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 0 (root), command name ganeti-noded [08:26:32] 06Operations: eqiad: Install SSD's into ganeti hosts - https://phabricator.wikimedia.org/T138414#2504574 (10akosiaris) @Cmjohnson ganeti1002 is empty and ready for SSD installation. I 've emptied it, downtimed it and powered it off. [08:28:27] hashar: I am wondering whether this would solve the issue though [08:28:39] well that issue yes, there might be other [08:28:41] others [08:28:48] certainly :- [08:28:49] ( [08:28:56] so git-buildpackage does have a number of assumptions [08:29:02] like for example the name of the branches [08:29:17] it expects things in master and debian branch for example [08:29:51] I 've been avoiding that in my local builds by git pulling from getting instead of git fetch and checkout [08:30:01] from gerrit* [08:34:44] and it is even more messy when cloning from the zuul-merger instance. The local repo there would miss branches :/ [08:35:49] hashar: I got this for you btw https://gerrit.wikimedia.org/r/#/c/301753/ [08:37:57] tried again this time checking out master and resetting it to the commit https://integration.wikimedia.org/ci/job/debian-glue/381/consoleFull [08:38:09] 00:00:07.222 ++ dpkg-parsechangelog --show-field distribution -lsource/debian/changelog [08:38:09] 00:00:07.386 + export distribution=jessie [08:38:25] :-) [08:38:30] 00:00:01.310 * master c83ee0f [origin/master: ahead 1] apertium-urd: New upstream release and rebuild for Jessie [08:38:49] will have to fix zuul-cloner so that it checkout the ref to the local branch instead of a detached one [08:39:46] 06Operations, 10Graphite, 05MW-1.27-release-notes, 13Patch-For-Review: udp rcvbuferrors and inerrors on graphite1001 - https://phabricator.wikimedia.org/T101141#2504593 (10fgiunchedi) >>! In T101141#2502979, @Krinkle wrote: >>>! In T101141#2502351, @Stashbot wrote: >> {nav icon=file, name=Mentioned in SAL,... [08:39:54] (03PS1) 10MarcoAurelio: Expanding throttle limits for enwiki Edit-a-thon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301761 (https://phabricator.wikimedia.org/T141421) [08:42:30] (03PS2) 10MarcoAurelio: Expanding throttle limits for enwiki Edit-a-thon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301761 (https://phabricator.wikimedia.org/T141421) [08:42:41] ok filled https://phabricator.wikimedia.org/T141607 [08:43:08] and the patch with all the debug stuff https://gerrit.wikimedia.org/r/301763 [08:44:01] akosiaris: perfect patch thx ! [08:44:41] then does generate-git-snapshot really rely on distribution being set ? :D [08:45:57] RECOVERY - puppet last run on mw2163 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:46:16] I have deployed your change and trying on https://integration.wikimedia.org/ci/job/debian-glue/382/console [08:46:19] no, it reparses changelog to set DISTRIBUTION [08:46:30] (03PS1) 10Filippo Giunchedi: Revert "statsite: flush to graphite every 30s" [puppet] - 10https://gerrit.wikimedia.org/r/301765 (https://phabricator.wikimedia.org/T101141) [08:46:37] ah no, it does have an if [ -n "${distribution:-}" ] ; then [08:46:38] hmmm [08:47:09] 00:00:02.040 Distribution variable found. Adding distribution specific version. [08:47:18] VERSION_STRING="${VERSION_STRING}+${distribution//-/_}" [08:47:22] that what it does [08:47:22] that is merely for the version yeah [08:47:31] yeah, I don't see an immediate problem [08:47:37] *** Version string set to 0.1.0~r61311-1+wmf1+0~20160729084601.382+jessie *** [08:48:33] (03PS3) 10MarcoAurelio: Expanding throttle limits for enwiki Edit-a-thon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301761 (https://phabricator.wikimedia.org/T141421) [08:48:41] ashley: so, https://integration.wikimedia.org/ci/job/debian-glue/382/console looks fine to me [08:48:45] I now wonder whether gbp build using the detached head [08:48:50] all I seem to need is change the changelog to jessie-wikimedia [08:48:50] or the master branch that is incorrect [08:49:30] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Revert "statsite: flush to graphite every 30s" [puppet] - 10https://gerrit.wikimedia.org/r/301765 (https://phabricator.wikimedia.org/T101141) (owner: 10Filippo Giunchedi) [08:49:32] I think the master branch. depends on gbp.conf really, which this project does not have so master [08:49:35] can you amend https://gerrit.wikimedia.org/r/#/c/296229/ to jessie-wikimedia [08:50:07] yeah so we have to point the local branch to the proper commit to fix zuul-cloner doing half the work :D [08:51:03] !log switch back statsite flush period to 60s T101141 [08:51:04] T101141: udp rcvbuferrors and inerrors on graphite1001 - https://phabricator.wikimedia.org/T101141 [08:51:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:51:53] akosiaris: the echo $distribution is invoked before the variable is set https://gerrit.wikimedia.org/r/#/c/301753/1/jjb/operations-debs.yaml [08:51:57] the lines got swapped [08:52:05] or am I missing something? [08:53:53] (03PS3) 10Alexandros Kosiaris: apertium-urd: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-urd] - 10https://gerrit.wikimedia.org/r/296229 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [08:55:24] hmm [08:55:31] yeah my mess [08:55:32] (03CR) 10jenkins-bot: [V: 04-1] apertium-urd: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-urd] - 10https://gerrit.wikimedia.org/r/296229 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [08:55:32] fixing [08:59:06] hashar: arg!!! parsechangelog/debian: warning: debian/changelog(l1): version '0.1.0~r61311-1+wmf1+0~20160729085527.383+jessie_wikimedia~1.gbpfe4d1a' is invalid: version number contains illegal character `_' [08:59:31] lol [08:59:53] something makes jessie-wikimedia => jessie_wikimedia [09:00:06] I should have seen that coming... [09:00:56] heheh reminds me of illegal characters in puppet modules I think? - is verboten [09:02:48] hashar: remember that line ? [09:02:53] if [ -n "${distribution:-}" ] ; then [09:02:53] echo "Distribution variable found. Adding distribution specific version." [09:02:54] VERSION_STRING="${VERSION_STRING}+${distribution//-/_}" [09:02:54] fi [09:02:56] sigh .... [09:02:57] oh no [09:07:42] that generate-git-snapshot thing is starting to get on my nerves [09:10:30] akosiaris: we could just use the original version? [09:10:34] $USE_ORIG_VERSION [09:11:33] 17c3778f (Antoine Musso 2015-04-13 21:16:02 +0200)| VERSION_STRING="${VERSION_STRING}+${distribution//-/_}" [09:12:21] https://github.com/mika/jenkins-debian-glue/commit/17c3778ffc33e557339e120b9b7cd7d2e07fd9ea [09:12:39] actuall that if line above overrides the USE_ORIG_VERSION if [09:12:49] bah [09:12:58] it happens unconditionally [09:13:00] so the reason for using underscore is that the distribution "jessie-wikimedia" is injected [09:13:23] and Debian Policy says that the versions are split by "-" [09:13:36] yeah, makes sense [09:13:41] now [09:13:47] seems that _ is no more allowed bah [09:15:59] (03PS1) 10ArielGlenn: add cron job for Content Translation dumps [puppet] - 10https://gerrit.wikimedia.org/r/301773 (https://phabricator.wikimedia.org/T127793) [09:18:18] The upstream_version may contain only alphanumerics[36] and the characters . + - : ~ [09:18:25] debian_revision may contain only alphanumerics and the characters + . ~ (plus, full stop, tilde) [09:18:52] so upstream can contain "-" [09:19:06] and the debian_revision is whatever is after the latest "-" [09:19:10] so that _ there was already wrong [09:19:20] I mean the VERSION_STRING="${VERSION_STRING}+${distribution//-/_}" [09:19:26] (03CR) 10jenkins-bot: [V: 04-1] add cron job for Content Translation dumps [puppet] - 10https://gerrit.wikimedia.org/r/301773 (https://phabricator.wikimedia.org/T127793) (owner: 10ArielGlenn) [09:19:42] on the other hand, that does not solve our problem. Even if we set jessie+wikimedia [09:19:42] maybe the debian policy has changed ? (lame attempt at blaming someone else hehe) [09:20:17] but we can do jessie+wikimedia and then change the hook [09:20:18] hmmm [09:20:25] that should be more doable [09:20:39] or invoke generate-git-snapshot with a polished distribution [09:21:29] distribution=${distribution//-/+} generate-git-snapshot [09:21:51] btw, why do that change ? [09:22:07] there is some explanation at https://github.com/mika/jenkins-debian-glue/commit/17c3778ffc33e557339e120b9b7cd7d2e07fd9ea [09:22:43] namely 0.1.0~r61311-1+wmf1+0~20160729085527.383+jessie_wikimedia~1.gbpfe4d1a [09:22:49] splitting with dash: [09:23:02] upstream version: 0.1.0~r61311 [09:23:17] debian revision: 1+wmf1+0~20160729085527.383+jessie_wikimedia~1.gbpfe4d1a [09:23:40] but if you inject "jessie-wikimedia" in the version, since it uses the chunk after the latest dash you end up with: [09:23:51] upstream version: 0.1.0~r61311-1+wmf1+0~20160729085527.383+jessie [09:23:57] debian revision: wikimedia~1.gbpfe4d1a [09:24:20] I guess that is why the debian_revision is forbidden to have a dash "-" [09:25:30] ok so the fault is probably mine. Creating illegal distribution names in package_builder namely [09:26:05] (03PS2) 10ArielGlenn: add cron job for Content Translation dumps [puppet] - 10https://gerrit.wikimedia.org/r/301773 (https://phabricator.wikimedia.org/T127793) [09:27:11] (03CR) 10jenkins-bot: [V: 04-1] add cron job for Content Translation dumps [puppet] - 10https://gerrit.wikimedia.org/r/301773 (https://phabricator.wikimedia.org/T127793) (owner: 10ArielGlenn) [09:29:29] * akosiaris sighs... [09:29:57] illegal ? [09:30:03] do you mean us having jessie-wikimedia ? [09:30:30] niah, disregard that [09:30:55] jessie-wikimedia is not illegal as a distribution name. I got carried out [09:31:06] as a version though.. it is [09:34:35] hence my patch to replace the dash with underscore [09:34:44] which apparently worked for gbp [09:35:04] but fails for parsechangelog/debian [09:35:54] no, gbp complains as well [09:36:02] dpkg-buildpackage: error: version number contains illegal character `_' [09:36:04] holy hell [09:38:57] !log Upgrading Zuul to get rid of a forced sleep(300) whenever a patch is merged T93812. zuul_2.1.0-391-gbc58ea3-wmf2precise1 [09:38:58] T93812: Change force merged cause a deadlock in Zuul gate-and-submit pipeline - https://phabricator.wikimedia.org/T93812 [09:39:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:39:57] * akosiaris goes away reading jenkins-debian-glue code [09:40:22] * godog passes akosiaris an hardhat [09:49:15] * hashar starts a RFC to switch to rpm [09:49:29] at least the hat will be red [09:50:11] {{File:Sting.ogg}} [09:56:08] 07Blocked-on-Operations, 06Operations, 10Continuous-Integration-Infrastructure, 10Zuul: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2504697 (10hashar) 05Resolved>03Open I had to rebuild the package on Precise for T93812 and I have rebuild the Je... [09:56:44] elukey: hello! Sorry I had to rebuild a zuul package to add a tiny patch on Precise and I have updated the Jessie one as well [09:57:02] if you get some free cycle, could use an upgrade on scandium and a push to apt.wm.o to keep things clear [09:58:05] sure! [09:58:13] sorry about that :( [09:58:15] maybe like yesterday around 2PM cerst? [09:58:18] *cest [09:58:20] yeah sure! [09:58:30] super :) [10:01:27] (03PS2) 10Filippo Giunchedi: prometheus: add mysqld exporter [puppet] - 10https://gerrit.wikimedia.org/r/296385 (https://phabricator.wikimedia.org/T126757) [10:02:43] (03PS1) 10Jcrespo: Restrict prometheus connections to 5 simultaneous connections [puppet] - 10https://gerrit.wikimedia.org/r/301778 (https://phabricator.wikimedia.org/T128185) [10:10:31] (03PS1) 10Alexandros Kosiaris: puppetmaster-test: Add a variety of hosts to it [puppet] - 10https://gerrit.wikimedia.org/r/301779 [10:10:50] (03CR) 10Jcrespo: [C: 032] Restrict prometheus connections to 5 simultaneous connections [puppet] - 10https://gerrit.wikimedia.org/r/301778 (https://phabricator.wikimedia.org/T128185) (owner: 10Jcrespo) [10:11:39] (03PS1) 10Elukey: Raise Cassandra auth caching to 10 minutes [puppet] - 10https://gerrit.wikimedia.org/r/301780 (https://phabricator.wikimedia.org/T140869) [10:14:07] !log applying new grants to all s1 servers [10:14:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:17:31] (03PS2) 10Elukey: Raise Cassandra auth caching to 10 minutes [puppet] - 10https://gerrit.wikimedia.org/r/301780 (https://phabricator.wikimedia.org/T140869) [10:18:27] (03PS3) 10Filippo Giunchedi: prometheus: add mysqld exporter [puppet] - 10https://gerrit.wikimedia.org/r/296385 (https://phabricator.wikimedia.org/T126757) [10:18:29] (03PS1) 10Filippo Giunchedi: hieradata: add prometheus_nodes [puppet] - 10https://gerrit.wikimedia.org/r/301781 [10:20:21] (03CR) 10Elukey: [C: 032] "https://puppet-compiler.wmflabs.org/3534/" [puppet] - 10https://gerrit.wikimedia.org/r/301780 (https://phabricator.wikimedia.org/T140869) (owner: 10Elukey) [10:20:45] (03PS2) 10Filippo Giunchedi: hieradata: add prometheus_nodes [puppet] - 10https://gerrit.wikimedia.org/r/301781 [10:20:47] (03PS4) 10Filippo Giunchedi: prometheus: add mysqld exporter [puppet] - 10https://gerrit.wikimedia.org/r/296385 (https://phabricator.wikimedia.org/T126757) [10:23:48] RECOVERY - cassandra-c CQL 10.192.48.51:9042 on restbase2006 is OK: TCP OK - 0.036 second response time on port 9042 [10:23:52] !log restarting cassandra on aqs100[123] to apply the latest config (https://gerrit.wikimedia.org/r/#/c/301780/1 - T140869) [10:23:53] T140869: Investigate why cassandra per-article-daily oozie jobs fail regularly - https://phabricator.wikimedia.org/T140869 [10:23:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:26:31] (03CR) 10Jcrespo: [C: 031] hieradata: add prometheus_nodes [puppet] - 10https://gerrit.wikimedia.org/r/301781 (owner: 10Filippo Giunchedi) [10:26:56] (03CR) 10Jcrespo: [C: 031] prometheus: add mysqld exporter [puppet] - 10https://gerrit.wikimedia.org/r/296385 (https://phabricator.wikimedia.org/T126757) (owner: 10Filippo Giunchedi) [10:27:44] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] hieradata: add prometheus_nodes [puppet] - 10https://gerrit.wikimedia.org/r/301781 (owner: 10Filippo Giunchedi) [10:27:55] (03PS3) 10Filippo Giunchedi: hieradata: add prometheus_nodes [puppet] - 10https://gerrit.wikimedia.org/r/301781 [10:28:00] (03CR) 10Filippo Giunchedi: [V: 032] hieradata: add prometheus_nodes [puppet] - 10https://gerrit.wikimedia.org/r/301781 (owner: 10Filippo Giunchedi) [10:29:45] (03PS5) 10Filippo Giunchedi: prometheus: add mysqld exporter [puppet] - 10https://gerrit.wikimedia.org/r/296385 (https://phabricator.wikimedia.org/T126757) [10:29:53] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] prometheus: add mysqld exporter [puppet] - 10https://gerrit.wikimedia.org/r/296385 (https://phabricator.wikimedia.org/T126757) (owner: 10Filippo Giunchedi) [10:30:22] jynus: ^ [10:34:38] (03PS1) 10Jcrespo: Add prometheus mysql exporter to db2069 [puppet] - 10https://gerrit.wikimedia.org/r/301782 (https://phabricator.wikimedia.org/T126757) [10:35:31] I need to clean up my site.pp ( I used to have a lot of legacy, I think now all of that is gone) [10:37:40] (03CR) 10Filippo Giunchedi: [C: 031] Add prometheus mysql exporter to db2069 [puppet] - 10https://gerrit.wikimedia.org/r/301782 (https://phabricator.wikimedia.org/T126757) (owner: 10Jcrespo) [10:51:43] jynus: PCC looks good! [10:51:51] yes [10:51:57] sorry, got distracted [10:52:25] will we need anything else on the server side? [10:52:42] or should I just deploy this, then we will see next steps? [10:53:27] I suppose it will not hurt [10:53:31] (03CR) 10Jcrespo: [C: 032] Add prometheus mysql exporter to db2069 [puppet] - 10https://gerrit.wikimedia.org/r/301782 (https://phabricator.wikimedia.org/T126757) (owner: 10Jcrespo) [10:53:40] (03PS2) 10Jcrespo: Add prometheus mysql exporter to db2069 [puppet] - 10https://gerrit.wikimedia.org/r/301782 (https://phabricator.wikimedia.org/T126757) [10:53:50] (03CR) 10Jcrespo: [V: 032] Add prometheus mysql exporter to db2069 [puppet] - 10https://gerrit.wikimedia.org/r/301782 (https://phabricator.wikimedia.org/T126757) (owner: 10Jcrespo) [11:00:29] PROBLEM - puppet last run on db2069 is CRITICAL: CRITICAL: puppet fail [11:01:30] (03PS1) 10Filippo Giunchedi: prometheus: fix arguments for mysqld_exporter [puppet] - 10https://gerrit.wikimedia.org/r/301785 [11:01:42] jynus: ^ [11:02:51] not sure that will work [11:04:52] I think it will, check interface::aggregate [11:05:39] (03PS2) 10Filippo Giunchedi: prometheus: fix arguments for mysqld_exporter [puppet] - 10https://gerrit.wikimedia.org/r/301785 [11:05:50] let's be safe [11:06:11] (I also wanted to try gerrit's online edit :-)) [11:06:24] hehehe ok [11:06:35] is that what you wanted? [11:06:51] sure that works too [11:07:10] I do not know if there could be '' and undef as to possible values [11:07:32] but I suppose that will work in both cases [11:07:40] jynus you will already be using online gerrit edit if you edit the commit msg now [11:07:52] yes, but not text [11:07:58] (03CR) 10Filippo Giunchedi: [C: 032] prometheus: fix arguments for mysqld_exporter [puppet] - 10https://gerrit.wikimedia.org/r/301785 (owner: 10Filippo Giunchedi) [11:08:15] well, I mean, souce code [11:08:22] jynus should be easy to do it, do you know how to do it? [11:08:30] jynus: merged [11:08:37] I just did^ paladox [11:08:42] Ok [11:08:44] :) [11:10:32] ther is something missing a mkdir /var/lib/prometheus/ on puppet or on the package [11:10:56] or maybe it was not intended there? [11:11:46] godog, check the puppet log on db2069 [11:12:32] (03PS3) 10ArielGlenn: add cron job for Content Translation dumps [puppet] - 10https://gerrit.wikimedia.org/r/301773 (https://phabricator.wikimedia.org/T127793) [11:12:56] jynus: ok, doing [11:13:38] (03CR) 10jenkins-bot: [V: 04-1] add cron job for Content Translation dumps [puppet] - 10https://gerrit.wikimedia.org/r/301773 (https://phabricator.wikimedia.org/T127793) (owner: 10ArielGlenn) [11:14:37] RECOVERY - puppet last run on db2069 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [11:16:07] jynus: yup it was only that missing, the rest works! [11:16:40] (03PS1) 10Jcrespo: Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) [11:16:54] ^maybe a dependency on the files for the dir? [11:17:47] (03CR) 10jenkins-bot: [V: 04-1] Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) (owner: 10Jcrespo) [11:18:07] ha ha [11:18:51] (03PS2) 10Jcrespo: Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) [11:18:56] what whitespace, nobody saw that? [11:19:26] jynus: puppet afaik would already add a dependency if it manages the parent directory [11:19:50] (03PS4) 10ArielGlenn: add cron job for Content Translation dumps [puppet] - 10https://gerrit.wikimedia.org/r/301773 (https://phabricator.wikimedia.org/T127793) [11:21:50] (03PS3) 10Jcrespo: Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) [11:21:58] oh, I didn't read that [11:22:07] should we really trust puppet? [11:23:03] (03CR) 10jenkins-bot: [V: 04-1] Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) (owner: 10Jcrespo) [11:23:29] ehhe sad_trombone.wav [11:23:40] but yeah we already trust puppet a whole lot, global root [11:24:18] I never know what is the proper indentation for arrays on puppet [11:29:21] (03PS4) 10Jcrespo: Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) [11:31:14] so I can do that with or without the require, as you wish [11:31:39] I will let you decide on this uttermost important question [11:32:16] hahaha thanks, I don't mind either way, IIRC it works without [11:32:50] jynus: change 0444 to 0550 for the directory tho [11:32:52] E_TOO_MANY_BROWSER_TABS [11:32:59] true [11:34:55] (03PS2) 10Alexandros Kosiaris: puppetmaster-test: Add a variety of hosts to it [puppet] - 10https://gerrit.wikimedia.org/r/301779 [11:35:26] are you sure prometheus doesn't need to write anything else there? [11:36:13] I'm positive, the exporters operate in memory in most cases [11:36:21] great [11:37:18] (03PS5) 10Jcrespo: Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) [11:38:18] (03PS1) 10Giuseppe Lavagetto: redis::instance: use specific aof/rdb file names by default [puppet] - 10https://gerrit.wikimedia.org/r/301789 (https://phabricator.wikimedia.org/T134400) [11:38:20] (03PS1) 10Giuseppe Lavagetto: redis: manage our redis common config with puppet [puppet] - 10https://gerrit.wikimedia.org/r/301790 (https://phabricator.wikimedia.org/T134400) [11:39:43] 06Operations, 10ops-codfw, 10DBA: db2069 crashed due to RAID controller - https://phabricator.wikimedia.org/T141601#2504844 (10jcrespo) 05Open>03Resolved a:03jcrespo Replication went back to normal and GTID reliable replication avoided corruption- resolving, keeping an eye on the data. [11:40:10] akosiaris: further hacked the jjb job to have distribution cleared before invoking generate-git-snapshot . I have refreshed the job already. https://gerrit.wikimedia.org/r/#/c/301753/1..2/jjb/operations-debs.yaml [11:41:01] (03CR) 10Jcrespo: [C: 032] Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) (owner: 10Jcrespo) [11:41:16] --> bahhhh git checkout -f ${GIT_BRANCH} # switch back to previous "branch" before removing the tmp branch [11:41:26] (03PS6) 10Jcrespo: Add /var/lib/prometheus to prometheus clients [puppet] - 10https://gerrit.wikimedia.org/r/301787 (https://phabricator.wikimedia.org/T126757) [11:42:51] (03PS3) 10Alexandros Kosiaris: puppetmaster-test: Add a variety of hosts to it [puppet] - 10https://gerrit.wikimedia.org/r/301779 [11:43:31] ηεη [11:43:33] heh [11:43:39] see https://integration.wikimedia.org/ci/job/debian-glue/385/console [11:43:43] the piuparts part fail again [11:43:48] I have pushed another fix [11:44:09] that is really monkey patching :( [11:44:32] generate-git-snapshot at the end does a git checkout -f $GIT_BRANCH [11:44:35] we is set to master [11:44:47] and zuul-cloner has not checked out the patch on that branch [11:45:09] <_joe_> akosiaris: any trouble with the puppet switches? [11:45:26] success https://integration.wikimedia.org/ci/job/debian-glue/386/console ! [11:45:35] _joe_: aside from me being incompetent ? not yet [11:45:57] <_joe_> akosiaris: what did you do wrong? :P [11:46:08] _joe_: nothing important yet [11:46:43] (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster-test: Add a variety of hosts to it [puppet] - 10https://gerrit.wikimedia.org/r/301779 (owner: 10Alexandros Kosiaris) [11:46:52] (03PS4) 10Alexandros Kosiaris: puppetmaster-test: Add a variety of hosts to it [puppet] - 10https://gerrit.wikimedia.org/r/301779 [11:46:54] (03CR) 10Alexandros Kosiaris: [V: 032] puppetmaster-test: Add a variety of hosts to it [puppet] - 10https://gerrit.wikimedia.org/r/301779 (owner: 10Alexandros Kosiaris) [11:54:33] oh [11:54:52] and I have learned about debian autotest :] or how to run a bunch of tests when building a package [11:55:27] hashar: I haven't looked yet at the success you posted earlier btw, kind of in the middle of something [11:55:31] will do though [11:55:40] no problem [12:03:35] (03CR) 10Hashar: "recheck" [debs/contenttranslation/apertium-urd] - 10https://gerrit.wikimedia.org/r/296229 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:06:38] (03PS1) 10Alexandros Kosiaris: varnish: Set mode on vtc tests directory [puppet] - 10https://gerrit.wikimedia.org/r/301792 [12:06:45] hashar: \o/!!!! [12:06:52] I see a +2 !!! [12:06:57] it is magic!!!!! [12:07:01] really [12:07:09] I am not proud of all that monkey patching :( [12:07:31] but zuul-cloner checking out the patch to a local branch instead of detached will fix it properly [12:08:58] (03CR) 10Hashar: "recheck" [debs/contenttranslation/apertium-es-it] - 10https://gerrit.wikimedia.org/r/295206 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:09:06] it is indeed magic!!! thank you [12:09:17] now to fire my script that will issue a recheck on all those [12:09:34] _joe_: so https://gerrit.wikimedia.org/r/301792 was the only issue up to now [12:09:41] which is really minor, just annoying [12:09:46] you noticing that "--configfile foorc" can override previously parameters has been the key [12:10:49] <_joe_> akosiaris: that's a known issue, right? [12:12:25] _joe_: yes. but it manifested due to that umask issue we were discussing [12:13:06] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-en-ca] - 10https://gerrit.wikimedia.org/r/294264 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:13:21] (03CR) 10jenkins-bot: [V: 04-1] apertium-en-ca: New upstream release and Jessie build [debs/contenttranslation/apertium-en-ca] - 10https://gerrit.wikimedia.org/r/294264 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:15:34] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-en-ca] - 10https://gerrit.wikimedia.org/r/294264 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:15:36] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-es-it] - 10https://gerrit.wikimedia.org/r/295206 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:15:38] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-urd] - 10https://gerrit.wikimedia.org/r/296229 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:15:40] sorry for the spam [12:15:41] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/giella-sme] - 10https://gerrit.wikimedia.org/r/294430 (https://phabricator.wikimedia.org/T120087) (owner: 10KartikMistry) [12:15:43] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-urd-hin] - 10https://gerrit.wikimedia.org/r/296368 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:15:45] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-mk-bg] - 10https://gerrit.wikimedia.org/r/296212 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:15:48] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-tat] - 10https://gerrit.wikimedia.org/r/296367 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:15:50] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-swe-nor] - 10https://gerrit.wikimedia.org/r/294245 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:15:53] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-swe-dan] - 10https://gerrit.wikimedia.org/r/294248 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:15:54] doh [12:15:55] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-swe] - 10https://gerrit.wikimedia.org/r/294244 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:15:57] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-spa-arg] - 10https://gerrit.wikimedia.org/r/295122 (https://phabricator.wikimedia.org/T124370) (owner: 10KartikMistry) [12:16:00] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-spa] - 10https://gerrit.wikimedia.org/r/294658 (https://phabricator.wikimedia.org/T124370) (owner: 10KartikMistry) [12:16:02] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-sme-nob] - 10https://gerrit.wikimedia.org/r/295185 (https://phabricator.wikimedia.org/T120087) (owner: 10KartikMistry) [12:16:04] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-pt-gl] - 10https://gerrit.wikimedia.org/r/296162 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:07] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-pt-ca] - 10https://gerrit.wikimedia.org/r/296164 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:09] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-oc-es] - 10https://gerrit.wikimedia.org/r/296209 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:09] akosiaris: most are made against "jessie" which does not have the latest apertium and will fail :( [12:16:12] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-oc-ca] - 10https://gerrit.wikimedia.org/r/296207 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:14] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-nob] - 10https://gerrit.wikimedia.org/r/269914 (https://phabricator.wikimedia.org/T124317) (owner: 10KartikMistry) [12:16:16] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-nno] - 10https://gerrit.wikimedia.org/r/269915 (https://phabricator.wikimedia.org/T124137) (owner: 10KartikMistry) [12:16:19] hashar: it does [12:16:19] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-mlt-ara] - 10https://gerrit.wikimedia.org/r/296214 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:21] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-mk-en] - 10https://gerrit.wikimedia.org/r/298250 (https://phabricator.wikimedia.org/T139918) (owner: 10KartikMistry) [12:16:24] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-isl-eng] - 10https://gerrit.wikimedia.org/r/296157 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:26] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-kaz-tat] - 10https://gerrit.wikimedia.org/r/296369 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:28] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-kaz] - 10https://gerrit.wikimedia.org/r/296366 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:30] the latest apertium package is on jessie-wikimedia [12:16:31] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-isl] - 10https://gerrit.wikimedia.org/r/296050 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:33] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-id-ms] - 10https://gerrit.wikimedia.org/r/296159 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:35] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-is-sv] - 10https://gerrit.wikimedia.org/r/296213 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:38] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-hbs-mkd] - 10https://gerrit.wikimedia.org/r/296051 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:40] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-hin] - 10https://gerrit.wikimedia.org/r/296228 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:43] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-hbs-slv] - 10https://gerrit.wikimedia.org/r/296203 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:45] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-hbs-eng] - 10https://gerrit.wikimedia.org/r/296049 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:47] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-hbs] - 10https://gerrit.wikimedia.org/r/294675 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:50] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-fra-cat] - 10https://gerrit.wikimedia.org/r/294425 (https://phabricator.wikimedia.org/T137768) (owner: 10KartikMistry) [12:16:52] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-fra] - 10https://gerrit.wikimedia.org/r/294252 (https://phabricator.wikimedia.org/T137768) (owner: 10KartikMistry) [12:16:53] I wonder how happy is jenkins/zuul gonna be [12:16:55] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-fr-es] - 10https://gerrit.wikimedia.org/r/295220 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:56] it's 56 btw [12:16:57] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-eus] - 10https://gerrit.wikimedia.org/r/294673 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:16:59] (03CR) 10jenkins-bot: [V: 04-1] giella-sme: Initial Debian packaging [debs/contenttranslation/giella-sme] - 10https://gerrit.wikimedia.org/r/294430 (https://phabricator.wikimedia.org/T120087) (owner: 10KartikMistry) [12:17:01] (03CR) 10Alexandros Kosiaris: "recheck" [debs/contenttranslation/apertium-eu-es] - 10https://gerrit.wikimedia.org/r/295697 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:17:03] (03CR) 10jenkins-bot: [V: 04-1] apertium-urd-hin: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-urd-hin] - 10https://gerrit.wikimedia.org/r/296368 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:17:18] (03CR) 10jenkins-bot: [V: 04-1] apertium-mk-bg: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-mk-bg] - 10https://gerrit.wikimedia.org/r/296212 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:17:20] (03CR) 10jenkins-bot: [V: 04-1] apertium-tat: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-tat] - 10https://gerrit.wikimedia.org/r/296367 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:17:33] (03CR) 10jenkins-bot: [V: 04-1] apertium-swe-nor: Initial Debian packaging [debs/contenttranslation/apertium-swe-nor] - 10https://gerrit.wikimedia.org/r/294245 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:17:39] up to now all these make sense [12:17:47] (03CR) 10jenkins-bot: [V: 04-1] apertium-swe-dan: Initial Debian packaging [debs/contenttranslation/apertium-swe-dan] - 10https://gerrit.wikimedia.org/r/294248 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:17:49] great [12:17:52] (03CR) 10jenkins-bot: [V: 04-1] apertium-swe: Initial Debian packaging [debs/contenttranslation/apertium-swe] - 10https://gerrit.wikimedia.org/r/294244 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:18:00] (03CR) 10jenkins-bot: [V: 04-1] apertium-spa-arg: Initial Debian packaging [debs/contenttranslation/apertium-spa-arg] - 10https://gerrit.wikimedia.org/r/295122 (https://phabricator.wikimedia.org/T124370) (owner: 10KartikMistry) [12:18:07] (03CR) 10jenkins-bot: [V: 04-1] apertium-spa: Initial Debian packaging [debs/contenttranslation/apertium-spa] - 10https://gerrit.wikimedia.org/r/294658 (https://phabricator.wikimedia.org/T124370) (owner: 10KartikMistry) [12:18:17] (03CR) 10jenkins-bot: [V: 04-1] apertium-sme-nob: Initial Debian packaging [debs/contenttranslation/apertium-sme-nob] - 10https://gerrit.wikimedia.org/r/295185 (https://phabricator.wikimedia.org/T120087) (owner: 10KartikMistry) [12:18:21] (03CR) 10jenkins-bot: [V: 04-1] apertium-pt-gl: Rebuild for Jessie, cleanup [debs/contenttranslation/apertium-pt-gl] - 10https://gerrit.wikimedia.org/r/296162 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:18:30] (03CR) 10jenkins-bot: [V: 04-1] apertium-pt-ca: Rebuild for Jessie, cleanup. [debs/contenttranslation/apertium-pt-ca] - 10https://gerrit.wikimedia.org/r/296164 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:18:36] (03CR) 10jenkins-bot: [V: 04-1] apertium-oc-es: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-oc-es] - 10https://gerrit.wikimedia.org/r/296209 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:18:41] ok and now I know what needs to be done [12:18:44] (03CR) 10jenkins-bot: [V: 04-1] apertium-oc-ca: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-oc-ca] - 10https://gerrit.wikimedia.org/r/296207 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:18:46] (03CR) 10jenkins-bot: [V: 04-1] apertium-nob: New upstream release [debs/contenttranslation/apertium-nob] - 10https://gerrit.wikimedia.org/r/269914 (https://phabricator.wikimedia.org/T124317) (owner: 10KartikMistry) [12:18:51] hashar: thanks again. I think you saved me hours [12:18:55] many many hours [12:18:59] (03CR) 10jenkins-bot: [V: 04-1] apertium-nno: New upstream release [debs/contenttranslation/apertium-nno] - 10https://gerrit.wikimedia.org/r/269915 (https://phabricator.wikimedia.org/T124137) (owner: 10KartikMistry) [12:19:07] (03CR) 10jenkins-bot: [V: 04-1] apertium-mlt-ara: Rebuild for Jessie and new upstream [debs/contenttranslation/apertium-mlt-ara] - 10https://gerrit.wikimedia.org/r/296214 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:19:10] in the days for sure [12:19:17] so in theory [12:19:19] (03CR) 10jenkins-bot: [V: 04-1] apertium-mk-en: Initial Debian packaging [debs/contenttranslation/apertium-mk-en] - 10https://gerrit.wikimedia.org/r/298250 (https://phabricator.wikimedia.org/T139918) (owner: 10KartikMistry) [12:19:24] (03CR) 10jenkins-bot: [V: 04-1] apertium-isl-eng: New upstream, rebuild for Jessie [debs/contenttranslation/apertium-isl-eng] - 10https://gerrit.wikimedia.org/r/296157 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:19:25] Hi could we have Sigyn in -labs please. [12:19:31] (03CR) 10jenkins-bot: [V: 04-1] apertium-kaz-tat: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-kaz-tat] - 10https://gerrit.wikimedia.org/r/296369 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:19:31] There was spam [12:19:33] (03CR) 10jenkins-bot: [V: 04-1] apertium-kaz: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-kaz] - 10https://gerrit.wikimedia.org/r/296366 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:19:36] and someone changed the topic [12:19:40] to some random thing [12:19:46] (03CR) 10jenkins-bot: [V: 04-1] apertium-isl: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-isl] - 10https://gerrit.wikimedia.org/r/296050 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:19:48] we could craft a job that build apertium first then build the extension with the apertium build injected [12:19:48] (03CR) 10jenkins-bot: [V: 04-1] apertium-id-ms: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-id-ms] - 10https://gerrit.wikimedia.org/r/296159 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:19:51] that would make them pass [12:20:01] (03CR) 10jenkins-bot: [V: 04-1] apertium-is-sv: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-is-sv] - 10https://gerrit.wikimedia.org/r/296213 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:08] (03CR) 10jenkins-bot: [V: 04-1] apertium-hin: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-hin] - 10https://gerrit.wikimedia.org/r/296228 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:12] (03CR) 10jenkins-bot: [V: 04-1] apertium-hbs-slv: New upstream, rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-slv] - 10https://gerrit.wikimedia.org/r/296203 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:14] (03CR) 10jenkins-bot: [V: 04-1] apertium-hbs-mkd: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-mkd] - 10https://gerrit.wikimedia.org/r/296051 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:16] (03CR) 10jenkins-bot: [V: 04-1] apertium-hbs-eng: New upstream, rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-eng] - 10https://gerrit.wikimedia.org/r/296049 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:27] (03CR) 10jenkins-bot: [V: 04-1] apertium-hbs: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-hbs] - 10https://gerrit.wikimedia.org/r/294675 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:29] (03CR) 10jenkins-bot: [V: 04-1] apertium-fra-cat: New upstream release, Rebuilt for Jessie [debs/contenttranslation/apertium-fra-cat] - 10https://gerrit.wikimedia.org/r/294425 (https://phabricator.wikimedia.org/T137768) (owner: 10KartikMistry) [12:20:31] (03CR) 10jenkins-bot: [V: 04-1] apertium-fra: Initial Debian packaging [debs/contenttranslation/apertium-fra] - 10https://gerrit.wikimedia.org/r/294252 (https://phabricator.wikimedia.org/T137768) (owner: 10KartikMistry) [12:20:35] (03CR) 10Paladox: "https://phabricator.wikimedia.org/T76459" [puppet] - 10https://gerrit.wikimedia.org/r/256663 (https://phabricator.wikimedia.org/T75997) (owner: 10Thiemo Mättig (WMDE)) [12:20:36] paladox: er what's Sigyn ? [12:20:43] (03CR) 10jenkins-bot: [V: 04-1] apertium-fr-es: New upstream and rebuild for Jessie [debs/contenttranslation/apertium-fr-es] - 10https://gerrit.wikimedia.org/r/295220 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:46] (03CR) 10jenkins-bot: [V: 04-1] apertium-eus: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-eus] - 10https://gerrit.wikimedia.org/r/294673 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:52] tom29739 i doint know what that is could you explain please ^^ [12:20:59] (03CR) 10jenkins-bot: [V: 04-1] apertium-eu-en: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-eu-es] - 10https://gerrit.wikimedia.org/r/295697 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:20:59] akosiaris: someone spammed -labs topic [12:21:12] you can ignore :] [12:21:43] Actually could we have that Sigyn please [12:21:45] hashar ^^ [12:22:03] I doint know what it is but tom29739 tryed asking here but was ignored. [12:22:03] paladox: it's a bot that k-lines spammers [12:22:10] Oh [12:22:10] paladox: not here [12:22:11] :) [12:22:11] paladox: the IRC channels are mostly managed by volunteers see #wikimedia-ops and https://meta.wikimedia.org/wiki/IRC/wikimedia-ops/Operators [12:22:17] Oh [12:22:27] paladox: I asked in -ops :D [12:22:36] Oh [12:22:42] Wait [12:22:45] -ops is for IRC [12:22:48] there are two operations channels [12:22:49] tom29739: to be fair we also use -ops to refer to #wikimedia-operations :] [12:22:52] wikimedia-ops [12:23:24] tom29739 https://github.com/alyx/sigyn [12:23:52] we could setup an instance and do that [12:23:53] ? [12:24:40] paladox: it's this: https://github.com/freenode/Sigyn [12:24:44] Ohg [12:24:45] Oh [12:24:50] It needs to be run on freenode [12:24:58] akosiaris: all fail with Depends: apertium-dev (>= 3.4) arent they? which is expecetd as I understand it [12:25:03] yes [12:25:05] paladox: I have a bot that runs on tools that could do in a pinch [12:25:14] Ok [12:25:17] yes please [12:25:17] hashar: I am fixing this as we speak [12:25:24] could you also run it on -labs please [12:25:27] tom29739 ^^ [12:25:45] akosiaris: jessie --> jessie-wikimedia right? [12:25:51] hashar: exactly [12:25:52] paladox: it isn't automatic, but it'll do [12:25:59] Ok [12:26:03] thanks [12:26:09] paladox: I'll add basic automatic stuff tonight [12:26:15] Ok thanks [12:26:18] :) [12:26:25] Your an hour ahead ah lol [12:26:29] (03CR) 10Hashar: "recheck" [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/301581 (owner: 10Hashar) [12:26:30] paladox: one problem though. It needs ops to ban people [12:26:38] Oh [12:26:41] I don't have ops in that channel [12:26:46] Nor do you [12:26:53] Oh, you could ask yuvipanda [12:26:54] (03CR) 10Hashar: "Godog wrote:" [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/301573 (owner: 10Gilles) [12:27:14] tom29739 your an hour ahead of me now lol. [12:27:35] (03CR) 10Hashar: "Bah:" [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/301581 (owner: 10Hashar) [12:27:54] paladox, 14:27 for me [12:27:57] yeh [12:28:01] (03PS2) 10KartikMistry: apertium-eu-en: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-eu-es] - 10https://gerrit.wikimedia.org/r/295697 (https://phabricator.wikimedia.org/T107306) [12:28:06] tom2973 13:28pm for me [12:28:28] Once you crossed the british border you were an hour ahead lol. [12:28:32] tom2973 ^^ [12:29:04] (03PS2) 10KartikMistry: apertium-eus: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-eus] - 10https://gerrit.wikimedia.org/r/294673 (https://phabricator.wikimedia.org/T107306) [12:29:33] paladox, Glai sher (trying not to ping) was doing stuff in #wikimedia-ops [12:29:40] A few minutes ago [12:29:45] Oh [12:29:48] (03PS2) 10KartikMistry: apertium-fra: Initial Debian packaging [debs/contenttranslation/apertium-fra] - 10https://gerrit.wikimedia.org/r/294252 (https://phabricator.wikimedia.org/T137768) [12:29:51] Doing what? [12:30:14] paladox: 2:16:22 p.m. Glaisher (Philon) set flags +AViotv on AlvaroMolina [12:30:20] (03CR) 10Jcrespo: [C: 031] "Should we deploy this or should we start with a reduced version for testing?" [puppet] - 10https://gerrit.wikimedia.org/r/299539 (https://phabricator.wikimedia.org/T126785) (owner: 10Filippo Giunchedi) [12:30:20] Oh [12:30:21] ? [12:30:24] (03CR) 10jenkins-bot: [V: 04-1] apertium-fra: Initial Debian packaging [debs/contenttranslation/apertium-fra] - 10https://gerrit.wikimedia.org/r/294252 (https://phabricator.wikimedia.org/T137768) (owner: 10KartikMistry) [12:30:26] (03PS3) 10KartikMistry: apertium-fra-cat: New upstream release, Rebuilt for Jessie [debs/contenttranslation/apertium-fra-cat] - 10https://gerrit.wikimedia.org/r/294425 (https://phabricator.wikimedia.org/T137768) [12:30:36] Glaisher: sorry for the pings [12:30:36] Glaisher the same person spamed -labs [12:30:48] (03PS2) 10KartikMistry: apertium-hbs: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-hbs] - 10https://gerrit.wikimedia.org/r/294675 (https://phabricator.wikimedia.org/T107306) [12:30:49] Changed the topic to random words [12:30:51] (03CR) 10jenkins-bot: [V: 04-1] apertium-fra-cat: New upstream release, Rebuilt for Jessie [debs/contenttranslation/apertium-fra-cat] - 10https://gerrit.wikimedia.org/r/294425 (https://phabricator.wikimedia.org/T137768) (owner: 10KartikMistry) [12:31:02] What? [12:31:08] (03PS3) 10KartikMistry: apertium-hbs-eng: New upstream, rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-eng] - 10https://gerrit.wikimedia.org/r/296049 (https://phabricator.wikimedia.org/T107306) [12:31:24] In -labs the same AlvaroMolina person or bot spammed -labs [12:31:28] with random topic [12:31:32] Oh. There's a cross channel troll who impersonates users and changes topics. [12:31:36] (03CR) 10jenkins-bot: [V: 04-1] apertium-eus: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-eus] - 10https://gerrit.wikimedia.org/r/294673 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:31:40] (03CR) 10jenkins-bot: [V: 04-1] apertium-hbs-eng: New upstream, rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-eng] - 10https://gerrit.wikimedia.org/r/296049 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:31:52] * ALVAROMOLINA| (~White@177.54.150.173) has joined [12:31:52] * ALVAROMOLINA| has changed the topic to: JEM DE M13RD4.... ERES UN PT0 DE MI3RD4 Q HACE C4C4 ENCIMA DE LOURDES Y PLATONIDES... HAHAHHAHAHAAH [12:32:00] Yeh [12:32:03] It's not the real AlvaroMolina. [12:32:11] oh. [12:32:15] (03PS2) 10KartikMistry: apertium-hbs-mkd: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-mkd] - 10https://gerrit.wikimedia.org/r/296051 (https://phabricator.wikimedia.org/T107306) [12:32:28] (03PS3) 10KartikMistry: apertium-hbs-slv: New upstream, rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-slv] - 10https://gerrit.wikimedia.org/r/296203 (https://phabricator.wikimedia.org/T107306) [12:32:39] brb lunch [12:32:40] paladox: that person had a slightly different name I think [12:32:41] (03PS3) 10KartikMistry: apertium-hin: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-hin] - 10https://gerrit.wikimedia.org/r/296228 (https://phabricator.wikimedia.org/T107306) [12:32:45] oh [12:32:54] Yeh was in capitals [12:33:00] (03PS2) 10KartikMistry: apertium-is-sv: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-is-sv] - 10https://gerrit.wikimedia.org/r/296213 (https://phabricator.wikimedia.org/T107306) [12:33:02] You can verify whether it's the real person by the cloak. [12:33:02] (03CR) 10jenkins-bot: [V: 04-1] apertium-hbs: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-hbs] - 10https://gerrit.wikimedia.org/r/294675 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:33:05] (03CR) 10jenkins-bot: [V: 04-1] apertium-hbs-slv: New upstream, rebuild for Jessie and cleanup [debs/contenttranslation/apertium-hbs-slv] - 10https://gerrit.wikimedia.org/r/296203 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:33:10] (03CR) 10jenkins-bot: [V: 04-1] apertium-hin: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-hin] - 10https://gerrit.wikimedia.org/r/296228 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:33:13] akosiaris: the fail still :( [12:33:23] (03PS2) 10KartikMistry: apertium-id-ms: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-id-ms] - 10https://gerrit.wikimedia.org/r/296159 (https://phabricator.wikimedia.org/T107306) [12:33:33] akosiaris: oh because of piuparts ! [12:33:36] (03PS2) 10KartikMistry: apertium-isl: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-isl] - 10https://gerrit.wikimedia.org/r/296050 (https://phabricator.wikimedia.org/T107306) [12:33:48] (03PS2) 10KartikMistry: apertium-kaz-tat: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-kaz-tat] - 10https://gerrit.wikimedia.org/r/296369 (https://phabricator.wikimedia.org/T107306) [12:34:22] (03PS2) 10KartikMistry: apertium-kaz: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-kaz] - 10https://gerrit.wikimedia.org/r/296366 (https://phabricator.wikimedia.org/T107306) [12:34:41] (03PS2) 10KartikMistry: apertium-isl-eng: New upstream, rebuild for Jessie [debs/contenttranslation/apertium-isl-eng] - 10https://gerrit.wikimedia.org/r/296157 (https://phabricator.wikimedia.org/T107306) [12:34:53] (03PS2) 10KartikMistry: apertium-mk-en: Initial Debian packaging [debs/contenttranslation/apertium-mk-en] - 10https://gerrit.wikimedia.org/r/298250 (https://phabricator.wikimedia.org/T139918) [12:35:09] (03PS2) 10KartikMistry: apertium-mlt-ara: Rebuild for Jessie and new upstream [debs/contenttranslation/apertium-mlt-ara] - 10https://gerrit.wikimedia.org/r/296214 (https://phabricator.wikimedia.org/T107306) [12:35:22] (03PS5) 10KartikMistry: apertium-nno: New upstream release [debs/contenttranslation/apertium-nno] - 10https://gerrit.wikimedia.org/r/269915 (https://phabricator.wikimedia.org/T124137) [12:35:34] (03PS4) 10KartikMistry: apertium-nob: New upstream release [debs/contenttranslation/apertium-nob] - 10https://gerrit.wikimedia.org/r/269914 (https://phabricator.wikimedia.org/T124317) [12:35:38] (03CR) 10jenkins-bot: [V: 04-1] apertium-kaz-tat: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-kaz-tat] - 10https://gerrit.wikimedia.org/r/296369 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:35:46] (03CR) 10jenkins-bot: [V: 04-1] apertium-kaz: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-kaz] - 10https://gerrit.wikimedia.org/r/296366 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:35:50] (03PS2) 10KartikMistry: apertium-oc-ca: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-oc-ca] - 10https://gerrit.wikimedia.org/r/296207 (https://phabricator.wikimedia.org/T107306) [12:36:04] (03PS2) 10KartikMistry: apertium-oc-es: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-oc-es] - 10https://gerrit.wikimedia.org/r/296209 (https://phabricator.wikimedia.org/T107306) [12:36:09] (03CR) 10jenkins-bot: [V: 04-1] apertium-isl-eng: New upstream, rebuild for Jessie [debs/contenttranslation/apertium-isl-eng] - 10https://gerrit.wikimedia.org/r/296157 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:36:18] (03PS2) 10KartikMistry: apertium-pt-gl: Rebuild for Jessie, cleanup [debs/contenttranslation/apertium-pt-gl] - 10https://gerrit.wikimedia.org/r/296162 (https://phabricator.wikimedia.org/T107306) [12:36:28] (03PS2) 10KartikMistry: apertium-sme-nob: Initial Debian packaging [debs/contenttranslation/apertium-sme-nob] - 10https://gerrit.wikimedia.org/r/295185 (https://phabricator.wikimedia.org/T120087) [12:36:44] (03PS2) 10KartikMistry: apertium-spa: Initial Debian packaging [debs/contenttranslation/apertium-spa] - 10https://gerrit.wikimedia.org/r/294658 (https://phabricator.wikimedia.org/T124370) [12:36:46] (03CR) 10jenkins-bot: [V: 04-1] apertium-isl: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-isl] - 10https://gerrit.wikimedia.org/r/296050 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:37:02] (03PS3) 10KartikMistry: apertium-spa-arg: Initial Debian packaging [debs/contenttranslation/apertium-spa-arg] - 10https://gerrit.wikimedia.org/r/295122 (https://phabricator.wikimedia.org/T124370) [12:37:22] (03PS2) 10KartikMistry: apertium-swe: Initial Debian packaging [debs/contenttranslation/apertium-swe] - 10https://gerrit.wikimedia.org/r/294244 (https://phabricator.wikimedia.org/T137767) [12:37:36] (03PS2) 10KartikMistry: apertium-swe-dan: Initial Debian packaging [debs/contenttranslation/apertium-swe-dan] - 10https://gerrit.wikimedia.org/r/294248 (https://phabricator.wikimedia.org/T137767) [12:37:54] (03PS2) 10KartikMistry: apertium-swe-nor: Initial Debian packaging [debs/contenttranslation/apertium-swe-nor] - 10https://gerrit.wikimedia.org/r/294245 (https://phabricator.wikimedia.org/T137767) [12:38:07] (03PS2) 10KartikMistry: apertium-tat: New upstream release and rebuild for Jessie [debs/contenttranslation/apertium-tat] - 10https://gerrit.wikimedia.org/r/296367 (https://phabricator.wikimedia.org/T107306) [12:38:20] (03PS2) 10KartikMistry: apertium-mk-bg: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-mk-bg] - 10https://gerrit.wikimedia.org/r/296212 (https://phabricator.wikimedia.org/T107306) [12:39:05] (03CR) 10jenkins-bot: [V: 04-1] apertium-nob: New upstream release [debs/contenttranslation/apertium-nob] - 10https://gerrit.wikimedia.org/r/269914 (https://phabricator.wikimedia.org/T124317) (owner: 10KartikMistry) [12:39:34] (03PS3) 10KartikMistry: apertium-urd-hin: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-urd-hin] - 10https://gerrit.wikimedia.org/r/296368 (https://phabricator.wikimedia.org/T107306) [12:41:15] hashar: aloha! do you want to upgrade zuul-merger? [12:41:20] elukey: sure [12:41:44] is it something that we should do on a Friday afternoon or on Monday morning? [12:41:49] https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb [12:41:56] I am not there next week [12:41:56] (03PS2) 10KartikMistry: giella-sme: Initial Debian packaging [debs/contenttranslation/giella-sme] - 10https://gerrit.wikimedia.org/r/294430 (https://phabricator.wikimedia.org/T120087) [12:41:58] so yeah friday :] [12:42:10] (03PS3) 10KartikMistry: apertium-es-it: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-es-it] - 10https://gerrit.wikimedia.org/r/295206 (https://phabricator.wikimedia.org/T107306) [12:42:15] (03CR) 10jenkins-bot: [V: 04-1] apertium-sme-nob: Initial Debian packaging [debs/contenttranslation/apertium-sme-nob] - 10https://gerrit.wikimedia.org/r/295185 (https://phabricator.wikimedia.org/T120087) (owner: 10KartikMistry) [12:42:22] it basically has no impact on scandium. The code being touched is not used [12:42:23] (03PS2) 10KartikMistry: apertium-en-ca: New upstream release and Jessie build [debs/contenttranslation/apertium-en-ca] - 10https://gerrit.wikimedia.org/r/294264 (https://phabricator.wikimedia.org/T107306) [12:42:34] it is merely to have the proper version installed and in sync with apt.wm.o [12:42:35] (03PS2) 10KartikMistry: apertium-eu-en: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-eu-en] - 10https://gerrit.wikimedia.org/r/295696 (https://phabricator.wikimedia.org/T107306) [12:42:48] (03PS2) 10KartikMistry: apertium-es-pt: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-es-pt] - 10https://gerrit.wikimedia.org/r/294431 (https://phabricator.wikimedia.org/T107306) [12:43:00] (03PS2) 10KartikMistry: apertium-es-gl: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-es-gl] - 10https://gerrit.wikimedia.org/r/295625 (https://phabricator.wikimedia.org/T107306) [12:43:16] (03CR) 10jenkins-bot: [V: 04-1] apertium-spa-arg: Initial Debian packaging [debs/contenttranslation/apertium-spa-arg] - 10https://gerrit.wikimedia.org/r/295122 (https://phabricator.wikimedia.org/T124370) (owner: 10KartikMistry) [12:43:25] !log upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb on scandium [12:43:25] (03PS2) 10KartikMistry: apertium-es-ca: Rebuild for Jessie and other fixes [debs/contenttranslation/apertium-es-ca] - 10https://gerrit.wikimedia.org/r/294671 (https://phabricator.wikimedia.org/T107306) [12:43:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [12:43:42] (03PS2) 10KartikMistry: apertium-es-ast: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-es-ast] - 10https://gerrit.wikimedia.org/r/295624 (https://phabricator.wikimedia.org/T107306) [12:43:55] (03PS2) 10KartikMistry: apertium-eo-fr: New upstream release and Jessie rebuild [debs/contenttranslation/apertium-eo-fr] - 10https://gerrit.wikimedia.org/r/294917 (https://phabricator.wikimedia.org/T107306) [12:44:09] (03PS2) 10KartikMistry: apertium-eo-es: Rebuild for Jessie, cleanup [debs/contenttranslation/apertium-eo-es] - 10https://gerrit.wikimedia.org/r/295611 (https://phabricator.wikimedia.org/T107306) [12:44:16] hashar: done :) [12:44:16] (03CR) 10jenkins-bot: [V: 04-1] apertium-swe-dan: Initial Debian packaging [debs/contenttranslation/apertium-swe-dan] - 10https://gerrit.wikimedia.org/r/294248 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:44:21] (03PS2) 10KartikMistry: apertium-eo-en: New upstream version and Jessie rebuild [debs/contenttranslation/apertium-eo-en] - 10https://gerrit.wikimedia.org/r/294472 (https://phabricator.wikimedia.org/T107306) [12:44:35] (03PS2) 10KartikMistry: apertium-eo-ca: Rebuild for Jessie and fixed dependencies [debs/contenttranslation/apertium-eo-ca] - 10https://gerrit.wikimedia.org/r/294432 (https://phabricator.wikimedia.org/T107306) [12:44:37] (03CR) 10jenkins-bot: [V: 04-1] apertium-swe-nor: Initial Debian packaging [debs/contenttranslation/apertium-swe-nor] - 10https://gerrit.wikimedia.org/r/294245 (https://phabricator.wikimedia.org/T137767) (owner: 10KartikMistry) [12:44:40] elukey: and the apertium spam is being handled properly on scandium :] [12:44:47] (03PS2) 10KartikMistry: apertium-en-gl: Rebuilt for Jessie and other fixes [debs/contenttranslation/apertium-en-gl] - 10https://gerrit.wikimedia.org/r/294322 (https://phabricator.wikimedia.org/T107306) [12:45:00] (03PS4) 10KartikMistry: apertium-dan-nor: New upstream release [debs/contenttranslation/apertium-dan-nor] - 10https://gerrit.wikimedia.org/r/269916 (https://phabricator.wikimedia.org/T124137) [12:45:16] (03PS4) 10KartikMistry: apertium-dan: New upstream release [debs/contenttranslation/apertium-dan] - 10https://gerrit.wikimedia.org/r/269912 (https://phabricator.wikimedia.org/T124137) [12:45:28] (03PS2) 10KartikMistry: apertium-cy-en: Rebuilt for Jessie [debs/contenttranslation/apertium-cy-en] - 10https://gerrit.wikimedia.org/r/294260 (https://phabricator.wikimedia.org/T107306) [12:45:32] elukey: could you sync up that new package to apt.wm.o please? Files are in https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/ and that should land under jessie-wikimedia/thirdparty [12:45:41] (03PS2) 10KartikMistry: apertium-cat: Initial Debian packaging [debs/contenttranslation/apertium-cat] - 10https://gerrit.wikimedia.org/r/294250 (https://phabricator.wikimedia.org/T137768) [12:45:55] (03PS3) 10KartikMistry: apertium-ca-it: Rebuild for Jessie [debs/contenttranslation/apertium-ca-it] - 10https://gerrit.wikimedia.org/r/294080 [12:46:09] (03PS2) 10KartikMistry: apertium-arg-cat: Initial Debian packaging [debs/contenttranslation/apertium-arg-cat] - 10https://gerrit.wikimedia.org/r/295121 (https://phabricator.wikimedia.org/T124369) [12:46:21] 07Blocked-on-Operations, 06Operations, 10Continuous-Integration-Infrastructure, 10Zuul: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2504931 (10hashar) [12:43:25] <@elukey> !log upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb o... [12:46:29] (03CR) 10jenkins-bot: [V: 04-1] apertium-urd-hin: Rebuild for Jessie and cleanup [debs/contenttranslation/apertium-urd-hin] - 10https://gerrit.wikimedia.org/r/296368 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:46:35] (03PS2) 10KartikMistry: apertium-arg: Initial Debian packaging [debs/contenttranslation/apertium-arg] - 10https://gerrit.wikimedia.org/r/294657 (https://phabricator.wikimedia.org/T124369) [12:47:01] (03PS2) 10KartikMistry: apertium-af-nl: New upstream version [debs/contenttranslation/apertium-af-nl] - 10https://gerrit.wikimedia.org/r/294073 (https://phabricator.wikimedia.org/T107306) [12:47:20] (03PS3) 10KartikMistry: apertium-en-es: Rebuilt for Jessie [debs/contenttranslation/apertium-en-es] - 10https://gerrit.wikimedia.org/r/294314 (https://phabricator.wikimedia.org/T107306) [12:48:18] hashar: actually quite a few now +2ed... since these are often interdependent packages it will take a couple of runs to sort them out fully but it's finally moving [12:48:23] (03CR) 10jenkins-bot: [V: 04-1] apertium-en-ca: New upstream release and Jessie build [debs/contenttranslation/apertium-en-ca] - 10https://gerrit.wikimedia.org/r/294264 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [12:49:01] hashar: sure, I'll do it on Monday first thing ok? [12:51:21] 07Blocked-on-Operations, 06Operations, 10Continuous-Integration-Infrastructure, 10Zuul: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2504946 (10hashar) 05Open>03Resolved Good. Will have to push the packages to apt.wm.o: Files are in https://peop... [12:51:51] elukey: I will not be around but then servers are up to date already. I have left a note at https://phabricator.wikimedia.org/T140894#2504946 and added a link to the Precise package (already deployed as well) [12:52:09] (03PS2) 10Alexandros Kosiaris: varnish: Set mode on vtc tests directory [puppet] - 10https://gerrit.wikimedia.org/r/301792 [12:52:14] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] varnish: Set mode on vtc tests directory [puppet] - 10https://gerrit.wikimedia.org/r/301792 (owner: 10Alexandros Kosiaris) [12:54:49] hashar: got it! [12:59:59] im back [13:11:39] (03CR) 10jenkins-bot: [V: 04-1] apertium-dan-nor: New upstream release [debs/contenttranslation/apertium-dan-nor] - 10https://gerrit.wikimedia.org/r/269916 (https://phabricator.wikimedia.org/T124137) (owner: 10KartikMistry) [13:11:46] (03CR) 10jenkins-bot: [V: 04-1] apertium-dan: New upstream release [debs/contenttranslation/apertium-dan] - 10https://gerrit.wikimedia.org/r/269912 (https://phabricator.wikimedia.org/T124137) (owner: 10KartikMistry) [13:15:20] !log Purged static resources related to mk.wiktionary (T141610) [13:15:21] T141610: Please run purgeList.php on mk.wiktionary - https://phabricator.wikimedia.org/T141610 [13:15:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [13:16:32] (03CR) 10jenkins-bot: [V: 04-1] giella-sme: Initial Debian packaging [debs/contenttranslation/giella-sme] - 10https://gerrit.wikimedia.org/r/294430 (https://phabricator.wikimedia.org/T120087) (owner: 10KartikMistry) [13:17:06] (03CR) 10jenkins-bot: [V: 04-1] apertium-arg-cat: Initial Debian packaging [debs/contenttranslation/apertium-arg-cat] - 10https://gerrit.wikimedia.org/r/295121 (https://phabricator.wikimedia.org/T124369) (owner: 10KartikMistry) [13:17:33] With [13:17:45] I don't see how it's useful to purge https://en.wikipedia.org/static/favicon/wiktionary/mk.ico ... [13:18:10] (03CR) 10jenkins-bot: [V: 04-1] apertium-en-es: Rebuilt for Jessie [debs/contenttranslation/apertium-en-es] - 10https://gerrit.wikimedia.org/r/294314 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [13:19:05] Let's do something more useful: tell IS to use the new file. [13:22:52] 06Operations, 10Graphite, 05MW-1.27-release-notes, 13Patch-For-Review: udp rcvbuferrors and inerrors on graphite1001 - https://phabricator.wikimedia.org/T101141#2505031 (10fgiunchedi) trying to track down where some metrics don't get flushed to graphite, taking `kafka.cluster.analytics-eqiad.kafka.kafka101... [13:23:34] (03PS1) 10Dereckson: Set favicon for mk.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301807 (https://phabricator.wikimedia.org/T140566) [13:23:45] (03CR) 10Hashar: "recheck" [debs/contenttranslation/hfst-ospell] - 10https://gerrit.wikimedia.org/r/296231 (https://phabricator.wikimedia.org/T107306) (owner: 10KartikMistry) [13:26:31] (03CR) 10Hashar: "recheck" [debs/pybal] - 10https://gerrit.wikimedia.org/r/90549 (owner: 10Hashar) [13:26:43] (03CR) 10Dereckson: "Follow-up: Id734dd6ffd9dd77daee502adc1a62e701d2b1ed4" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300177 (https://phabricator.wikimedia.org/T140566) (owner: 10MarcoAurelio) [13:31:37] 06Operations, 10DBA, 10Gerrit, 06Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2505059 (10demon) Ah yep, we're done. Wipe away! [13:33:40] godog: ngrep!!!!!!!! [13:33:54] thank you for the knowledge of a new tool [13:34:00] goodbye tcpdump -A and squinting [13:34:55] ottomata: hahah you are welcome! yeah ngrep is great [13:35:54] testing dologmsg on tin [13:37:08] eery logmsgbot [13:37:17] My fault. [13:38:28] Testing before filing T141619. [13:38:29] T141619: dologmsg doesn't work on terbium - https://phabricator.wikimedia.org/T141619 [13:40:29] wow I didn't know about ngrep too :O [13:40:30] 06Operations, 10Deployment-Systems: dologmsg doesn't work on terbium - https://phabricator.wikimedia.org/T141619#2505074 (10Anomie) [13:43:56] 06Operations, 10Graphite, 05MW-1.27-release-notes, 13Patch-For-Review: udp rcvbuferrors and inerrors on graphite1001 - https://phabricator.wikimedia.org/T101141#2505118 (10fgiunchedi) statsd-proxy -> statsite (8126-8131/udp) ```lines=4 root@graphite1001:~# timeout 3m ngrep -d lo -t -W byline kafka1014_eq... [13:48:29] (03PS1) 10Alexandros Kosiaris: ulsfo: switch ulsfo to use new puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/301812 [13:50:18] ^ ? [13:50:39] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "PCC OKs in https://puppet-compiler.wmflabs.org/3541/cp4001.ulsfo.wmnet/ merging" [puppet] - 10https://gerrit.wikimedia.org/r/301812 (owner: 10Alexandros Kosiaris) [13:50:53] bblack: rhodium [13:51:06] the puppetmaster.test virtualhost on palladium uses rhodium as a backend [14:13:51] (03PS1) 10Alexandros Kosiaris: puppetmaster: Enable the new puppetmaster for selected swift machines [puppet] - 10https://gerrit.wikimedia.org/r/301815 [14:16:53] (03PS8) 10BBlack: VCL backends 2/N: sort misc req_handling [puppet] - 10https://gerrit.wikimedia.org/r/300579 (https://phabricator.wikimedia.org/T110717) [14:17:49] lol@lolritt [14:17:54] ETOOMANYPATCHES? [14:18:35] (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: Enable the new puppetmaster for selected swift machines [puppet] - 10https://gerrit.wikimedia.org/r/301815 (owner: 10Alexandros Kosiaris) [14:20:00] (03CR) 10BBlack: [C: 032 V: 032] ciphersuites: add chacha20poly1305 draft support [puppet] - 10https://gerrit.wikimedia.org/r/301816 (https://phabricator.wikimedia.org/T131908) (owner: 10BBlack) [14:20:10] (03PS2) 10BBlack: ciphersuites: add chacha20poly1305 draft support [puppet] - 10https://gerrit.wikimedia.org/r/301816 (https://phabricator.wikimedia.org/T131908) [14:20:13] (03CR) 10BBlack: [V: 032] ciphersuites: add chacha20poly1305 draft support [puppet] - 10https://gerrit.wikimedia.org/r/301816 (https://phabricator.wikimedia.org/T131908) (owner: 10BBlack) [14:20:22] akosiaris: FYI if you want some more test machines also swift in esams isn't serving live traffic [14:20:39] godog: niah those should suffice [14:20:45] but all of esams is right after that [14:21:02] so be on your toes :P [14:21:59] hahah will do, though this is for the backend part correct? [14:22:08] i.e. not catalog compilation iirc/ [14:22:21] er, the backend is what does the catalog compilation [14:22:55] ah, what does the frontend do? [14:22:56] the frontend only does HTTPS termination and some selected endpoints like CA/filebuckets [14:23:33] nice, even better [14:23:50] I did find though 2 minor issues on the ms boxes [14:23:52] puppetmaster.test.ulsfo.wmnet is an alias for palladium.eqiad.wmnet. [14:23:53] patch coming up [14:24:16] bblack: yup [14:24:30] different virtualhost on palladium [14:24:49] needed in order to have different backends [14:25:36] ah [14:25:51] FWIW, it took 3 runs to apply a parser function's updated output, usually takes two [14:26:13] where ? [14:26:16] (usually first puppet run updates .rb file, second run actually changes file output based on the parser func. in this case there was an extra no-op run in the middle) [14:26:21] on cp4011.ulsfo.wmnet [14:26:38] but who knows, maybe that's always a timing race and not a real change [14:26:43] maybe we raced each other ? [14:27:01] Can I get a root to `chown -R mwdeploy:wikidev /srv/mediawiki-staging/.git/{objects,modules/portals/objects}` on tin? [14:27:24] (Also, why isn't my new alert yelling at us yet?) [14:28:24] (03CR) 10BBlack: [C: 032] ciphersuites: drop AES128(-GCM)?-SHA256 [puppet] - 10https://gerrit.wikimedia.org/r/301817 (https://phabricator.wikimedia.org/T118181) (owner: 10BBlack) [14:28:31] (03PS2) 10BBlack: ciphersuites: drop AES128(-GCM)?-SHA256 [puppet] - 10https://gerrit.wikimedia.org/r/301817 (https://phabricator.wikimedia.org/T118181) [14:28:34] (03CR) 10BBlack: [V: 032] ciphersuites: drop AES128(-GCM)?-SHA256 [puppet] - 10https://gerrit.wikimedia.org/r/301817 (https://phabricator.wikimedia.org/T118181) (owner: 10BBlack) [14:28:46] ostriches: gimme 5 mins to finish something and I 'll help [14:28:55] Okie dokie thx! [14:29:07] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 611 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 4984110 keys - replication_delay is 611 [14:31:01] yeah I think it's always a race and I tend to think about it wrong [14:31:17] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 4981537 keys - replication_delay is 0 [14:31:23] (03PS1) 10Alexandros Kosiaris: statsite: Set owner/group/mode on init directory [puppet] - 10https://gerrit.wikimedia.org/r/301819 [14:31:26] the client does sync the updated parser function down to itself for whatever reason, but parsing actually happens on the master which already had the updated file [14:31:44] so it's just a random time period after merge until catalog compilation starts applying it [14:31:57] (which often happens to be two puppet runs in when I check it quickly after merge) [14:32:05] it's usually around 5 secs [14:32:11] I 've noticed that race too [14:32:19] sometimes more [14:32:34] I tend to wait before applying a puppet run specifically for that reason [14:32:49] (03PS1) 10Addshore: Beta move $wgEchoMentionStatusNotifications to CommonSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301820 (https://phabricator.wikimedia.org/T140234) [14:32:54] godog: https://gerrit.wikimedia.org/r/301819 [14:32:59] should be a noop [14:33:10] so I 'll merge but feel free to block me [14:33:20] well not a noop, but innocuous [14:33:53] (03CR) 10Filippo Giunchedi: [C: 031] statsite: Set owner/group/mode on init directory [puppet] - 10https://gerrit.wikimedia.org/r/301819 (owner: 10Alexandros Kosiaris) [14:33:59] akosiaris: looks good, thanks! [14:34:40] (03CR) 10Alexandros Kosiaris: [C: 032] statsite: Set owner/group/mode on init directory [puppet] - 10https://gerrit.wikimedia.org/r/301819 (owner: 10Alexandros Kosiaris) [14:34:45] (03PS2) 10Alexandros Kosiaris: statsite: Set owner/group/mode on init directory [puppet] - 10https://gerrit.wikimedia.org/r/301819 [14:34:50] (03CR) 10Alexandros Kosiaris: [V: 032] statsite: Set owner/group/mode on init directory [puppet] - 10https://gerrit.wikimedia.org/r/301819 (owner: 10Alexandros Kosiaris) [14:35:02] (03PS1) 10Chad: Gerrit: Set default owners for mediawiki/* and operations/* projects [puppet] - 10https://gerrit.wikimedia.org/r/301822 [14:36:17] 06Operations, 10ops-eqiad, 10Analytics-Cluster, 06Analytics-Kanban: analytics1032 disk failure - https://phabricator.wikimedia.org/T141550#2505209 (10Ottomata) @Cmjohnson, can we look at this today? [14:38:36] (03PS1) 10Chad: Gerrit: Remove temp rsync bridge from lead [puppet] - 10https://gerrit.wikimedia.org/r/301823 [14:39:28] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 1 failures [14:39:58] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: Puppet has 1 failures [14:40:08] PROBLEM - puppet last run on mw2148 is CRITICAL: CRITICAL: Puppet has 3 failures [14:40:49] PROBLEM - puppet last run on mw2213 is CRITICAL: CRITICAL: Puppet has 1 failures [14:40:58] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 1 failures [14:41:07] test T141619 [14:41:08] T141619: dologmsg doesn't work on terbium - https://phabricator.wikimedia.org/T141619 [14:43:00] (03PS1) 10Chad: Gerrit: Rename apache logs a tad [puppet] - 10https://gerrit.wikimedia.org/r/301824 [14:43:46] 06Operations, 10Deployment-Systems: dologmsg doesn't work on terbium - https://phabricator.wikimedia.org/T141619#2505074 (10AlexMonk-WMF) From manifests/role/tcpircbot.pp: ``` ferm::rule { 'tcpircbot_allowed': # eventlog1001, tin (v4), mira (v4), localhost, tin (v6), mira (v6) rule => 'proto... [14:43:48] 06Operations, 10hardware-requests: Eqiad: procure 4 servers for kubernetes - https://phabricator.wikimedia.org/T141624#2505235 (10Joe) [14:44:19] 06Operations, 10hardware-requests: Eqiad: procure 4 servers for kubernetes - https://phabricator.wikimedia.org/T141624#2505247 (10Joe) [14:45:28] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:45:36] (03PS1) 10Alexandros Kosiaris: esams: switch esams to use new puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/301827 [14:48:08] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] esams: switch esams to use new puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/301827 (owner: 10Alexandros Kosiaris) [14:52:20] (03PS1) 10Chad: Gerrit: Redirect plain "/r" (no trailing slash) to gerrit as well [puppet] - 10https://gerrit.wikimedia.org/r/301829 [14:55:58] ostriches: chown on tin:/srv/mediawiki-staging done [14:56:20] Looks good! [14:56:44] I wonder why my new alert isn't working though. We added something like the "unmerged changes" [14:56:59] (a "house isn't on fire, but someone should probably look at this and figure out why" sort of alert) [15:04:18] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:05:08] RECOVERY - puppet last run on mw2213 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [15:05:17] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:28] RECOVERY - puppet last run on mw2148 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:37] (03PS1) 10Alexandros Kosiaris: varnishkafka.py: Set owner/group/mode [puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/301833 [15:16:42] (03CR) 10Alexandros Kosiaris: [C: 032] varnishkafka.py: Set owner/group/mode [puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/301833 (owner: 10Alexandros Kosiaris) [15:17:41] !log starting maintenance script for [[phab:T140811]] [15:17:43] T140811: Run maintenance/cleanupEmptyCategories.php - https://phabricator.wikimedia.org/T140811 [15:17:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:18:34] 06Operations, 10ops-eqiad, 10Analytics-Cluster, 06Analytics-Kanban: analytics1032 disk failure - https://phabricator.wikimedia.org/T141550#2505338 (10Cmjohnson) Cleared the foreign config from db1032...new VD needs to be setup [15:20:02] (03PS1) 10Chad: Gerrit: require openjdk-7-jdk instead of just the JRE [puppet] - 10https://gerrit.wikimedia.org/r/301835 [15:21:30] (03CR) 10Paladox: [C: 031] Gerrit: require openjdk-7-jdk instead of just the JRE [puppet] - 10https://gerrit.wikimedia.org/r/301835 (owner: 10Chad) [15:22:44] (03PS1) 10Alexandros Kosiaris: Update varnishkafka submodule [puppet] - 10https://gerrit.wikimedia.org/r/301836 [15:24:58] (03CR) 10Paladox: "Wont gerrit.config need to be updated with path change?" [puppet] - 10https://gerrit.wikimedia.org/r/301835 (owner: 10Chad) [15:25:53] (03PS6) 10Chad: Minor tweaks to 2.12.2 package [debs/gerrit] - 10https://gerrit.wikimedia.org/r/299164 [15:27:46] (03CR) 10Chad: "No, this just adds the jdk for debugging. The jre remains in the same location." [puppet] - 10https://gerrit.wikimedia.org/r/301835 (owner: 10Chad) [15:27:56] ostriches: so, the alert fires normally [15:28:04] it's just a warning, not a critical [15:28:29] shows up fine in icinga, just not icinga-wm reporting it [15:28:42] Ahhh, ok [15:28:49] (03CR) 10Paladox: "Oh ah, thanks for explaning." [puppet] - 10https://gerrit.wikimedia.org/r/301835 (owner: 10Chad) [15:28:52] I guess it should be critical then if I want it to yell at people? [15:28:59] yes [15:29:06] actually should return 2 and not 1 [15:29:13] Ah hmmm. [15:29:18] that's icinga talk for critical [15:29:26] How could I get test or something to do that? [15:30:05] The check right now is: [15:30:06] test -z "`find /srv/mediawiki-staging -uid 0 -or -gid 0`" [15:30:30] Otherwise, EXPRESSION is true or false and sets exit status [15:30:48] so can't do it with test I 'd say [15:31:09] Yeah. Problem I had was find is always gonna return a 0 [15:31:19] RECOVERY - Check size of conntrack table on analytics1032 is OK: OK: nf_conntrack is 0 % full [15:31:19] RECOVERY - Hadoop NodeManager on analytics1032 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [15:31:27] RECOVERY - Host analytics1032 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [15:31:38] RECOVERY - MegaRAID on analytics1032 is OK: OK: optimal, 13 logical, 14 physical [15:31:47] RECOVERY - Disk space on Hadoop worker on analytics1032 is OK: DISK OK [15:31:50] RECOVERY - configured eth on analytics1032 is OK: OK - interfaces up [15:31:50] RECOVERY - Disk space on analytics1032 is OK: DISK OK [15:31:50] RECOVERY - DPKG on analytics1032 is OK: All packages OK [15:31:50] RECOVERY - YARN NodeManager Node-State on analytics1032 is OK: OK: YARN NodeManager analytics1032.eqiad.wmnet:8041 Node-State: RUNNING [15:32:47] RECOVERY - salt-minion processes on analytics1032 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [15:32:47] RECOVERY - Hadoop DataNode on analytics1032 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [15:32:50] RECOVERY - dhclient process on analytics1032 is OK: PROCS OK: 0 processes with command name dhclient [15:32:50] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [15:34:02] akosiaris: I guess easiest thing to do would be a simple like 3-line bash script :) [15:34:13] Rather than trying to one-line it :) [15:34:29] actually [15:34:33] || exit 2 [15:34:47] || ? [15:35:04] find is always gonna return 0. [15:35:16] Or after the test? [15:35:51] er exit 2 is wrong [15:36:13] hmm [15:36:58] I think it can be done in about 3 lines of bash. [15:37:07] Basically the same check with -z, but in the block, return 2 [15:37:09] instead of 1 [15:37:14] yeah [15:37:25] I'll just do that rather than trying to be clever :) [15:37:28] in fact just an if in the oneline is enough [15:38:00] If I make it in a bash script though that'd make it more portable if other things wanted to reuse a similar check [15:38:09] Like we did with the git stale check [15:45:54] 06Operations, 10ops-eqiad, 10Analytics-Cluster, 06Analytics-Kanban: analytics1032 disk failure - https://phabricator.wikimedia.org/T141550#2505429 (10Ottomata) Ok, the disk is back with all its data. We don't know why it decided to go all foreign on us. Let's keep an eye on it. [15:52:41] 06Operations, 10ops-eqiad, 10Analytics-Cluster, 06Analytics-Kanban: analytics1032 disk failure - https://phabricator.wikimedia.org/T141550#2505430 (10Cmjohnson) The disk has been cleared and is back online The server booted to Raid Configuration mode because it showed a foreign disk. I performed the follo... [15:52:47] 06Operations, 10ops-eqiad, 10Analytics-Cluster, 06Analytics-Kanban: analytics1032 disk failure - https://phabricator.wikimedia.org/T141550#2505431 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson [15:56:15] (03CR) 10Alexandros Kosiaris: [C: 032] Update varnishkafka submodule [puppet] - 10https://gerrit.wikimedia.org/r/301836 (owner: 10Alexandros Kosiaris) [15:58:41] (03CR) 10Hashar: "recheck" [debs/salt] (jessie) - 10https://gerrit.wikimedia.org/r/273875 (owner: 10ArielGlenn) [15:58:44] (03CR) 10Hashar: "recheck" [debs/salt] (jessie) - 10https://gerrit.wikimedia.org/r/273876 (owner: 10ArielGlenn) [15:59:04] (03CR) 10Hashar: "recheck" [debs/salt] (jessie) - 10https://gerrit.wikimedia.org/r/273879 (owner: 10ArielGlenn) [15:59:50] 06Operations: eqiad: Install SSD's into ganeti hosts - https://phabricator.wikimedia.org/T138414#2505460 (10akosiaris) a:05akosiaris>03Cmjohnson [16:06:28] (03CR) 10Kaldari: [C: 031] "Do you want this to be SWAT deployed next week?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301550 (https://phabricator.wikimedia.org/T128806) (owner: 10Raimond Spekking) [16:07:38] (03PS1) 10Chad: Deploy masters: Improve icinga check for bad ownership [puppet] - 10https://gerrit.wikimedia.org/r/301842 [16:08:11] akosiaris: Worked that up ^ [16:08:17] I'll run it through puppet compiler [16:08:52] (03CR) 10jenkins-bot: [V: 04-1] Deploy masters: Improve icinga check for bad ownership [puppet] - 10https://gerrit.wikimedia.org/r/301842 (owner: 10Chad) [16:09:08] (03Draft2) 10Paladox: Test [debs/gerrit] - 10https://gerrit.wikimedia.org/r/301841 [16:10:44] ostriches ^^ [16:10:53] could you review that, i need a better commit msg [16:11:00] but jenkins will build your debs now [16:11:02] please [16:12:26] and a deb built [16:12:27] https://integration.wikimedia.org/ci/job/debian-glue-non-voting/29/artifact/gerrit_2.12.2+0~20160729160931.29+jessie+wikimedia~1.gbp93b455_all.deb [16:12:32] ostriches ^^ :) [16:13:32] (03PS2) 10Chad: Deploy masters: Improve icinga check for bad ownership [puppet] - 10https://gerrit.wikimedia.org/r/301842 [16:14:40] :) [16:16:26] (03PS3) 10Paladox: Add gbp.conf file for debian [debs/gerrit] - 10https://gerrit.wikimedia.org/r/301841 [16:16:48] Better commit msg now ^^ [16:17:24] (03CR) 10Chad: "Compiles file on both mira and tin. https://puppet-compiler.wmflabs.org/3542/ :)" [puppet] - 10https://gerrit.wikimedia.org/r/301842 (owner: 10Chad) [16:18:00] (03CR) 10Raimond Spekking: "@Kaldari: Would be great." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301550 (https://phabricator.wikimedia.org/T128806) (owner: 10Raimond Spekking) [16:18:28] PROBLEM - puppet last run on mw2075 is CRITICAL: CRITICAL: puppet fail [16:24:08] PROBLEM - MD RAID on ms-be1022 is CRITICAL: CRITICAL: Active: 5, Working: 5, Failed: 1, Spare: 0 [16:25:03] 06Operations, 10Ops-Access-Requests, 06Research-and-Data, 10Research-collaborations: Analytics cluster access request for ISI Foundation team - https://phabricator.wikimedia.org/T141634#2505524 (10DarTar) [16:25:29] 06Operations, 10Ops-Access-Requests, 06Research-and-Data, 10Research-collaborations: Analytics cluster access request for ISI Foundation team - https://phabricator.wikimedia.org/T141634#2505543 (10DarTar) [16:27:32] 06Operations, 10Ops-Access-Requests, 06Research-and-Data, 10Research-collaborations: Analytics cluster access request for ISI Foundation team - https://phabricator.wikimedia.org/T141634#2505549 (10DarTar) [16:29:48] PROBLEM - puppet last run on ms-be1022 is CRITICAL: CRITICAL: Puppet has 1 failures [16:30:27] 06Operations, 10Traffic, 13Patch-For-Review: Planning for phasing out non-Forward-Secret TLS ciphers - https://phabricator.wikimedia.org/T118181#2505559 (10BBlack) Recapping latest investigations, stats, and changes: 1. We're down to just `DES-CBC3-SHA` and `AES128-SHA` on the non-forward-secret list. Ever... [16:39:29] !log granted addshore admin on labs grafana [16:39:32] sigh [16:39:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:43:09] same person [16:43:15] who did it to -labs [16:43:20] (03PS1) 10Chad: Phab: Remove unused phab-deploy-key files [puppet] - 10https://gerrit.wikimedia.org/r/301847 [16:44:18] 06Operations, 10ops-eqiad, 10media-storage: diagnose failed(?) sda on ms-be1022 - https://phabricator.wikimedia.org/T140597#2505572 (10Cmjohnson) I replaced the disk on ms-be1022 but it shows up failed and does not appear to rebuild on it's own. logicaldrive 1 (186.3 GB, 0): Failed logicaldrive 2 (186.3... [16:44:21] 06Operations, 10Graphite, 05MW-1.27-release-notes, 13Patch-For-Review: udp rcvbuferrors and inerrors on graphite1001 - https://phabricator.wikimedia.org/T101141#2505573 (10fgiunchedi) capturing the traffic that goes statsite -> carbon-c-relay frontend -> local (2003 -> 1903) there's this: ```lines=4 root@... [16:44:35] Nemo_bis: Do you have the topic for here when you joined? [16:44:57] RECOVERY - puppet last run on mw2075 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:44:57] I have it, I am not an op [16:45:04] Wikimedia Platform operations, serious stuff | Status: up | Log: https://bit.ly/wikitech | Channel logs: http://ur1.ca/edq22 | Ops Clinic Duty: [16:45:14] I do not know who is on duty [16:45:23] not sure if that is right [16:45:25] that's what legoktm put last night [16:45:28] greg-g: extra spaces :P [16:45:31] er, krenair [16:45:48] let me recheck it [16:45:51] * greg-g "/last topic"'d [16:46:05] ACKNOWLEDGEMENT - MD RAID on ms-be1022 is CRITICAL: CRITICAL: Active: 5, Working: 5, Failed: 1, Spare: 0 Filippo Giunchedi disk being diagnosed, not in service, T140597 [16:46:05] ACKNOWLEDGEMENT - puppet last run on ms-be1022 is CRITICAL: CRITICAL: Puppet has 1 failures Filippo Giunchedi disk being diagnosed, not in service, T140597 [16:46:06] 06Operations, 10ops-eqiad, 10media-storage: diagnose failed disks on ms-be1027 - https://phabricator.wikimedia.org/T140374#2505576 (10Cmjohnson) Received the 2 ssds and added them to ms-be1027 [16:47:11] yes, it is m*utante [16:47:21] until Sunday [16:47:29] * greg-g nods [16:49:03] (03CR) 10Jforrester: [C: 04-1] "I'm pretty sure it is still over-ridden in code, isn't it?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301743 (owner: 10MaxSem) [16:49:32] * Disconnected (No such device or address) [16:49:32] * Now talking on #wikimedia-operations [16:49:34] * Topic for #wikimedia-operations is: Wikimedia Platform operations, serious stuff | Status: up | Log: https://bit.ly/wikitech | Channel logs: http://ur1.ca/edq22 | Ops Clinic Duty: mutante [16:49:34] * Topic for #wikimedia-operations set by elukey!~elukey@wikimedia/ltoscano-wmf (Mon Jul 25 18:19:15 2016) [16:49:37] Reedy ^^ [16:50:25] paladox: too late :) [16:50:44] Oh woops [16:50:55] greg-g: There's some extra spaces between Channel and logs :P [16:51:13] greg-g it seems to be the same White@ip person [16:51:21] targeting all wikimedia channels [16:51:28] including taking users identity [16:51:34] thanks [16:52:02] sorry, I didn't notice since it was a copy/paste in irssi and the spaces lined up with my window size's line break (plaintext4life) [16:52:23] paladox: impersonating, yes [16:52:41] Yep [16:53:09] Luckly users who use cloaks are protected since you will be able to tell who is real and who is not [16:55:25] 06Operations, 10Traffic: Age header reset to 0 after 24 hours on varnish frontends - https://phabricator.wikimedia.org/T141373#2505650 (10ema) [16:55:57] (03CR) 10Jforrester: "> Missing ">" and space (see failing unit test). I'm actually surprised this wasn't caught while it was on labs... Are we sure this experi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301129 (https://phabricator.wikimedia.org/T141349) (owner: 10Jforrester) [16:56:43] (03PS2) 10Jforrester: Change default gallery mode to 'packed' on the English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301129 (https://phabricator.wikimedia.org/T141349) [16:57:00] (03CR) 10jenkins-bot: [V: 04-1] Change default gallery mode to 'packed' on the English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301129 (https://phabricator.wikimedia.org/T141349) (owner: 10Jforrester) [17:09:48] (03PS1) 10BBlack: ciphersuites: require TLSv1.1+ for "mid" [puppet] - 10https://gerrit.wikimedia.org/r/301851 [17:10:45] (03PS2) 10BBlack: ciphersuites: require TLSv1.1+ for "mid" [puppet] - 10https://gerrit.wikimedia.org/r/301851 [17:16:23] 06Operations, 06Services, 10Wikimedia-Logstash: Kibana / logstash dashboards timing out consistently since Kibana upgrade - https://phabricator.wikimedia.org/T141384#2505717 (10Pchelolo) We've deployed a fix that should've decreased the number of keys in the logs, but restbase dashboard still times out. @E... [17:18:19] 06Operations, 10ops-eqiad: Rack/setup sodium (carbon/mirror server replacement) - https://phabricator.wikimedia.org/T139171#2505725 (10Cmjohnson) A workorder to replace the system board has been issued. Congratulations: Work Order SR933837812 was successfully submitted. [17:25:09] (03CR) 10BBlack: [C: 04-1] "My "extensive logging" might be flawed here, or it's also possible the SSL= field in our nginx output doesn't mean what I think it means (" [puppet] - 10https://gerrit.wikimedia.org/r/301851 (owner: 10BBlack) [17:26:35] (03CR) 10Hashar: "recheck" [debs/salt] (jessie) - 10https://gerrit.wikimedia.org/r/273876 (owner: 10ArielGlenn) [17:44:40] (03Abandoned) 10BBlack: ciphersuites: require TLSv1.1+ for "mid" [puppet] - 10https://gerrit.wikimedia.org/r/301851 (owner: 10BBlack) [17:53:04] 06Operations, 06Labs, 06Project-Admins: Archive old Incident-* projects - https://phabricator.wikimedia.org/T134624#2271591 (10greg) per T140202 (and the follow-up in T141493) I think we can en-masse archive these. No one was using them for their workboards and we've got all of the still open tasks now in th... [17:54:05] (03PS1) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [17:54:45] (03PS2) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [17:55:54] (03CR) 10jenkins-bot: [V: 04-1] k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) (owner: 10Yuvipanda) [17:57:28] (03PS3) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [18:00:13] (03PS4) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [18:03:26] 06Operations, 06Labs, 06Project-Admins: Archive old Incident-* projects - https://phabricator.wikimedia.org/T134624#2505879 (10Danny_B) [18:03:47] PROBLEM - Host ns2-v4 is DOWN: PING CRITICAL - Packet loss = 100% [18:04:00] 06Operations, 06Labs, 06Project-Admins: Archive old Incident-* projects - https://phabricator.wikimedia.org/T134624#2271591 (10Danny_B) 05Open>03Resolved All archived. [18:04:17] PROBLEM - Host eeden is DOWN: PING CRITICAL - Packet loss = 100% [18:05:32] 06Operations, 06Services, 10Wikimedia-Logstash: Kibana / logstash dashboards timing out consistently since Kibana upgrade - https://phabricator.wikimedia.org/T141384#2505891 (10EBernhardson) Pulled the properties list for today(2016.07.29), still has ~50k items: P3601 [18:05:36] (03PS5) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [18:07:15] (03PS6) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [18:10:27] RECOVERY - Host eeden is UP: PING OK - Packet loss = 0%, RTA = 83.81 ms [18:12:09] RECOVERY - Host ns2-v4 is UP: PING OK - Packet loss = 0%, RTA = 83.51 ms [18:14:02] ns2/eeden something known above? [18:14:52] , [18:19:16] (03CR) 10MaxSem: "Nope, ot duplicates a line from prod and both with it or without, the value is the same, http://deployment-parsoid07.deployment-prep.eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301743 (owner: 10MaxSem) [18:25:43] (03PS1) 10Eevans: Enable Cassandra instance restbase2009-c.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/301855 (https://phabricator.wikimedia.org/T134016) [18:26:28] (03CR) 10Eevans: [C: 031] "Ready for this to be merged." [puppet] - 10https://gerrit.wikimedia.org/r/301855 (https://phabricator.wikimedia.org/T134016) (owner: 10Eevans) [18:27:21] (03CR) 10Dzahn: [C: 032] Enable Cassandra instance restbase2009-c.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/301855 (https://phabricator.wikimedia.org/T134016) (owner: 10Eevans) [18:27:37] wow! [18:27:41] mutante: thanks! [18:28:48] mutante: after that one, there is only one more to do. [18:29:01] (which will come next week) [18:29:06] (03PS1) 10MaxSem: Load Elastica via extension.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301856 [18:29:41] urandom: alright, np [18:29:58] * urandom is so ready to be done [18:30:09] :) [18:31:36] (03CR) 10Jhobs: "> It's not on Beta Cluster yet (I3d61178c hasn't been merged), and the typo is why this is V-1; it's fixed in the Beta Cluster, but as the" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301129 (https://phabricator.wikimedia.org/T141349) (owner: 10Jforrester) [18:37:05] (03PS1) 10Chad: Remove furud and antimony from hiera, they don't exist anymore [puppet] - 10https://gerrit.wikimedia.org/r/301860 [18:37:15] !log T134016: Bootstrapping restbase2009-c.codfw.wmnet [18:37:16] T134016: RESTBase Cassandra cluster: Increase instance count to 3 - https://phabricator.wikimedia.org/T134016 [18:37:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:40:24] (03PS1) 10Jdlrobson: Enable new language bar on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301861 (https://phabricator.wikimedia.org/T141647) [18:41:33] (03PS1) 10Chad: Phab: Add ourselves to the list of sites to skip proxying [puppet] - 10https://gerrit.wikimedia.org/r/301862 [18:41:35] (03PS1) 10Chad: Phab: Set origin's URL to phab not gerrit [puppet] - 10https://gerrit.wikimedia.org/r/301863 [18:41:48] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.104 second response time [18:42:44] are new icinga checks enabled on some sort of stable interval? [18:46:50] (03CR) 10Chad: [C: 031] "Let's go ahead with this. Will give me a more accurate judgement when tuning cache settings later." [puppet] - 10https://gerrit.wikimedia.org/r/300446 (https://phabricator.wikimedia.org/T141064) (owner: 10Dzahn) [18:57:25] (03CR) 10EBernhardson: [C: 032] Enable new language bar on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301861 (https://phabricator.wikimedia.org/T141647) (owner: 10Jdlrobson) [18:57:51] (03Merged) 10jenkins-bot: Enable new language bar on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301861 (https://phabricator.wikimedia.org/T141647) (owner: 10Jdlrobson) [18:59:14] !log ebernhardson@tin Synchronized wmf-config/InitialiseSettings-labs.php: labs only change to enable mobile language bar (duration: 00m 27s) [18:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:04:08] 06Operations, 10ops-eqiad: Rack/setup sodium (carbon/mirror server replacement) - https://phabricator.wikimedia.org/T139171#2506019 (10Cmjohnson) A new system board has been confirmed. Dell will be sending a tech out to me next week. Your appointment has been scheduled for : 12:00 PM-05:00 PM , Wednesday, Au... [19:06:41] (03PS7) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [19:09:18] PROBLEM - cassandra-c CQL 10.192.48.56:9042 on restbase2009 is CRITICAL: Connection refused [19:10:27] ACKNOWLEDGEMENT - cassandra-c CQL 10.192.48.56:9042 on restbase2009 is CRITICAL: Connection refused eevans Bootstrapping - The acknowledgement expires at: 2016-07-30 19:10:12. [19:33:31] (03PS4) 10Reedy: Swap to using static php array for TrustedXFF usage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301016 (https://phabricator.wikimedia.org/T141120) [19:35:48] PROBLEM - puppet last run on mw2155 is CRITICAL: CRITICAL: puppet fail [19:41:39] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.104 second response time [19:47:32] (03PS4) 10Dzahn: gerrit: up heap size limit from 20GB to 28GB [puppet] - 10https://gerrit.wikimedia.org/r/300446 (https://phabricator.wikimedia.org/T141064) [19:49:11] (03PS1) 10Chad: Gerrit: Tune caches (first round) [puppet] - 10https://gerrit.wikimedia.org/r/301873 [19:55:04] (03PS1) 10Yuvipanda: labspuppetbackend: Return 404 when no roles/hiera data exist [puppet] - 10https://gerrit.wikimedia.org/r/301874 [19:55:15] (03CR) 10Dzahn: [C: 032] gerrit: up heap size limit from 20GB to 28GB [puppet] - 10https://gerrit.wikimedia.org/r/300446 (https://phabricator.wikimedia.org/T141064) (owner: 10Dzahn) [19:56:26] !log gerrit restarting to apply config change 300446 - up heap size limit [19:56:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:57:13] done [20:03:27] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:04:58] RECOVERY - puppet last run on mw2155 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:09:43] bd808 updated, https://gerrit.wikimedia.org/r/301853 and https://gerrit.wikimedia.org/r/301875 [20:10:32] mutante hey! is there somewhere to say 'after restarting gerrit, please restart the gerrit bot as well? [20:10:36] (instructions on restarting bot at https://wikitech.wikimedia.org/wiki/Grrrit-wm) [20:11:20] !log restart gerrit-wm bot [20:11:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:11:24] i know, i usually restart it every single time, thanks [20:11:35] there is another change coming ina minute [20:13:29] (03CR) 10BryanDavis: k8s: Use direct-lvm for docker storage backend (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) (owner: 10Yuvipanda) [20:13:37] ah, thanks mutante [20:14:14] bd808 wow, ctrl-c does not work for copy in Gerrit [20:14:18] actively user hostile software, I keep forgetting [20:14:37] It's a bug. [20:14:44] Fixed in next upgrade. Coming next week. [20:14:48] yeah... it's a bit frustrating right now. the keyboard bindings are all messed up [20:15:05] 06Operations, 10Analytics, 06Performance-Team, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2506628 (10ellery) @Nuria, @BBlack I need to clarify that in the example that I gave above, the experiments were not run concurrently, but in sequence. [20:15:16] Need my packaged reviewed + built + uploaded to apt ;-) [20:15:19] (03PS3) 10Dzahn: Gerrit: Tune caches (first round) [puppet] - 10https://gerrit.wikimedia.org/r/301873 (owner: 10Chad) [20:16:13] 06Operations, 10Analytics, 06Performance-Team, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2506631 (10ellery) @Nuria I'm confused about how your statement "a bucket will have control and treatment for 1 experiment". I though that a bucket represents a group of users... [20:16:17] YuviPanda: Only in Firefox and fixed in the next minor version [20:16:27] Yeh [20:16:31] Speaking of Gerrit, it seems unstable right now? [20:16:34] Deffintly fixed [20:16:34] I'm getting a lot of 503s from it [20:16:47] Actually I can't get it to serve me anything at all [20:16:53] (03CR) 10BryanDavis: k8s: Deploy worker automatically (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/301875 (owner: 10Yuvipanda) [20:16:58] Works for me [20:17:06] Hmm maybe it's just the URL I'm hitting [20:17:06] https://gerrit.wikimedia.org/r/#/c/301868/2 [20:17:06] i'm using it and seems fine [20:17:14] maybe a bit faster even [20:17:15] wfm [20:17:16] That consistently 503s but other changes consistently work [20:17:25] btw, "tune caches" change coming in [20:17:27] (03PS10) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [20:17:29] That works for me [20:17:29] (03PS3) 10Yuvipanda: k8s: Deploy worker automatically [puppet] - 10https://gerrit.wikimedia.org/r/301875 [20:17:40] WTF Chrome [20:17:51] Hard-refreshing the tab kept breaking, but opening a new tab with the same URL worked [20:18:21] Works for me on chrome [20:18:23] windows chrome [20:18:26] Yeah WFM now [20:18:31] My browser was just smoking crack it seems [20:18:36] ok then.. i'll do this now [20:18:43] (03CR) 10Dzahn: [C: 032] Gerrit: Tune caches (first round) [puppet] - 10https://gerrit.wikimedia.org/r/301873 (owner: 10Chad) [20:18:45] (03PS11) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [20:18:47] (03PS4) 10Yuvipanda: k8s: Deploy worker automatically [puppet] - 10https://gerrit.wikimedia.org/r/301875 [20:19:32] (03CR) 10Paladox: [C: 031] Gerrit: Redirect plain "/r" (no trailing slash) to gerrit as well [puppet] - 10https://gerrit.wikimedia.org/r/301829 (owner: 10Chad) [20:19:46] Should have 99% cache hit rates on most caches with that one ;-) [20:20:22] (03CR) 10Paladox: "Wont this break redirecting gerrit.wikimedia.org so you would now need to do gerrit.wikimedia.org/r to get redirect." [puppet] - 10https://gerrit.wikimedia.org/r/301829 (owner: 10Chad) [20:21:00] (03CR) 10Chad: "That's what the question mark is for...it's a regular expression..." [puppet] - 10https://gerrit.wikimedia.org/r/301829 (owner: 10Chad) [20:21:05] (03CR) 10jenkins-bot: [V: 04-1] k8s: Deploy worker automatically [puppet] - 10https://gerrit.wikimedia.org/r/301875 (owner: 10Yuvipanda) [20:22:05] (03CR) 10Paladox: [C: 031] "ah ok thanks for explaining." [puppet] - 10https://gerrit.wikimedia.org/r/301829 (owner: 10Chad) [20:22:29] !log gerrit restarting to apply config change 301873 - tuning caches [20:22:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:22:35] bd808 ok, really updated now :D [20:23:44] YuviPanda, hey, I stumbled upon Quarry the other day after needing to run some queries to get some data about 'contenttranslation' articles on en.wp - all I can say is wow and thank you <3 [20:24:02] 503ing again now for both me and mooeypoo [20:24:04] myrcx :D you're welcome! have you seen paws.wmflabs.org? ;) [20:24:12] RoanKattouw: it just got restarted :) [20:24:29] .. and there is a problem with it [20:24:38] Again? [20:24:41] Yep [20:24:50] A new patch has been applied to it [20:24:51] we'll stop after this one [20:24:55] myrcx it's generic notebook as a service + other fun stuff, including db access to replica dbs. see http://paws-public.wmflabs.org/paws-public/User:YuviPanda/replicahelper.ipynb for example [20:25:17] back [20:25:21] ostriches https://gerrit.wikimedia.org/r/#/c/301841/ [20:25:22] :) [20:25:29] It allows us to build dpkg from jenkins [20:26:05] YuviPanda, looking! :D [20:26:34] would you be able to review it ostriches please [20:26:35] :) [20:26:54] Not right now. [20:27:09] ok [20:27:47] gerrit bot needs restarting [20:27:48] ? [20:28:58] PROBLEM - puppet last run on stat1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:58] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:32:04] paladox: yep, we are on it. gerrit being fixed.. bot restarted.. [20:32:11] ok [20:32:13] thanks [20:32:38] hashar: https://gerrit.wikimedia.org/r/#/c/301499/ has a jenkins tweak for you to look at [20:33:17] YuviPanda, paws looks awesome! [20:33:39] :D [20:39:01] 06Operations, 10Analytics, 06Performance-Team, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2506770 (10ellery) Another issue that is independent of proper randomization, is that for most use cases, the data produced by the system cannot be used for statistical testing... [20:39:54] (03PS2) 10Dzahn: Gerrit: Remove temp rsync bridge from lead [puppet] - 10https://gerrit.wikimedia.org/r/301823 (owner: 10Chad) [20:41:15] (03CR) 10Dzahn: [C: 032] Gerrit: Remove temp rsync bridge from lead [puppet] - 10https://gerrit.wikimedia.org/r/301823 (owner: 10Chad) [20:42:04] (03PS12) 10Yuvipanda: k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) [20:42:10] (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Use direct-lvm for docker storage backend [puppet] - 10https://gerrit.wikimedia.org/r/301853 (https://phabricator.wikimedia.org/T141126) (owner: 10Yuvipanda) [20:42:20] (03PS6) 10Yuvipanda: k8s: Deploy worker automatically [puppet] - 10https://gerrit.wikimedia.org/r/301875 [20:42:34] (03PS2) 10Eevans: Increase permissions validity on RESTBase cluster [puppet] - 10https://gerrit.wikimedia.org/r/301878 (https://phabricator.wikimedia.org/T140869) [20:43:59] (03CR) 10Eevans: "Puppet compiler output here: http://puppet-compiler.wmflabs.org/3543/" [puppet] - 10https://gerrit.wikimedia.org/r/301878 (https://phabricator.wikimedia.org/T140869) (owner: 10Eevans) [20:47:00] (03CR) 10Paladox: [C: 031] Minor tweaks to 2.12.2 package [debs/gerrit] - 10https://gerrit.wikimedia.org/r/299164 (owner: 10Chad) [20:53:49] ^ why does it do that. that's not us [20:54:00] as opposed to the restarts earlier [20:54:47] RECOVERY - puppet last run on stat1004 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:56:14] !log restarted grrrit-wm but this time only because it died by itself [20:56:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:56:28] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.101 second response time [20:56:46] oh, well that would explain [20:56:50] if there is a kubernetes issue [20:56:57] YuviPanda: [20:57:35] (03CR) 10Hashar: "recheck" [debs/nodepool] (debian) - 10https://gerrit.wikimedia.org/r/237700 (https://phabricator.wikimedia.org/T111377) (owner: 10Hashar) [20:57:40] yeah [20:57:41] oh [20:57:42] no unrlated, mutante [20:57:43] this is just gerrit being a piece of shit [20:57:45] nevermind me [20:57:47] I'm raging elsewhere [20:57:54] well, see the k8 worker thing [20:57:56] right after [20:58:09] I know, I'm working on it [20:58:14] alright [20:58:47] (03PS3) 10Dzahn: Gerrit: require openjdk-7-jdk instead of just the JRE [puppet] - 10https://gerrit.wikimedia.org/r/301835 (owner: 10Chad) [20:59:10] mutante when it dies due to node failure it should come back again in a bit by itself, no need to explicitly restart [21:00:07] YuviPanda: gotcha [21:00:37] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.103 second response time [21:00:48] i merged both our changes [21:02:41] thanks mutante [21:02:41] !log gerrit: raised log level on sshd to ERROR from WARN. Irrelevant logspam. [21:02:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:04:06] 06Operations, 07Epic, 07Need-volunteer, 13Patch-For-Review: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#2506888 (10Dzahn) [21:05:12] 06Operations, 07Epic, 07Need-volunteer: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#1288654 (10Dzahn) [21:10:34] (03PS7) 10Yuvipanda: k8s: Deploy worker automatically [puppet] - 10https://gerrit.wikimedia.org/r/301875 [21:10:36] (03PS1) 10Yuvipanda: k8s: Bump docker version to latest of 1.11 [puppet] - 10https://gerrit.wikimedia.org/r/301889 [21:10:52] (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Bump docker version to latest of 1.11 [puppet] - 10https://gerrit.wikimedia.org/r/301889 (owner: 10Yuvipanda) [21:15:29] 06Operations, 10Monitoring, 06Release-Engineering-Team: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2506920 (10hashar) [21:22:53] (03CR) 10BryanDavis: [C: 031] k8s: Deploy worker automatically [puppet] - 10https://gerrit.wikimedia.org/r/301875 (owner: 10Yuvipanda) [21:27:52] (03PS6) 10Andrew Bogott: WIP: Horizon tab for modifying instance puppet config [puppet] - 10https://gerrit.wikimedia.org/r/294342 (https://phabricator.wikimedia.org/T91990) [21:29:06] (03CR) 10jenkins-bot: [V: 04-1] WIP: Horizon tab for modifying instance puppet config [puppet] - 10https://gerrit.wikimedia.org/r/294342 (https://phabricator.wikimedia.org/T91990) (owner: 10Andrew Bogott) [21:29:32] (03PS1) 10Hashar: 0.1.1-wmf5: debian/gbp.conf upstream-tag = %(version)s [debs/nodepool] (debian) - 10https://gerrit.wikimedia.org/r/301891 [21:33:27] (03PS1) 10Jforrester: De-deploy the CustomData extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301892 (https://phabricator.wikimedia.org/T140847) [21:36:32] ohnoes [21:45:54] (03PS2) 10Jforrester: De-deploy the MoodBar extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/280624 (https://phabricator.wikimedia.org/T131340) (owner: 10Catrope) [21:45:56] (03PS1) 10Jforrester: MoodBar: Disable on all wikis except nlwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301893 (https://phabricator.wikimedia.org/T131340) [21:50:47] (03PS1) 10Chad: Gerrit: Double size of conflicts cache [puppet] - 10https://gerrit.wikimedia.org/r/301894 [21:51:40] (03CR) 10Paladox: [C: 031] Gerrit: Double size of conflicts cache [puppet] - 10https://gerrit.wikimedia.org/r/301894 (owner: 10Chad) [21:59:22] (03CR) 10Dzahn: [C: 032] Remove furud and antimony from hiera, they don't exist anymore [puppet] - 10https://gerrit.wikimedia.org/r/301860 (owner: 10Chad) [22:00:10] (03PS2) 10Dzahn: Remove furud and antimony from hiera, they don't exist anymore [puppet] - 10https://gerrit.wikimedia.org/r/301860 (owner: 10Chad) [22:04:43] (03CR) 10Dzahn: [C: 032] "has been approved in last ops meeting" [puppet] - 10https://gerrit.wikimedia.org/r/301726 (https://phabricator.wikimedia.org/T141013) (owner: 10Dzahn) [22:06:48] (03PS2) 10Dzahn: eventbus: add new eventbus-admins group to nodes via role [puppet] - 10https://gerrit.wikimedia.org/r/301726 (https://phabricator.wikimedia.org/T141013) [22:09:06] (03PS3) 10Dzahn: eventbus: add new eventbus-admins group to nodes via role [puppet] - 10https://gerrit.wikimedia.org/r/301726 (https://phabricator.wikimedia.org/T141013) [22:09:49] (03PS1) 10Chad: Gerrit: Attempt retaining logs for 10 days [puppet] - 10https://gerrit.wikimedia.org/r/301895 [22:11:34] (03PS1) 10Chad: Gerrit: Simplify dependencies [puppet] - 10https://gerrit.wikimedia.org/r/301896 [22:12:54] (03Draft3) 10Paladox: Rely on commits name instead of branch [puppet] - 10https://gerrit.wikimedia.org/r/301849 [22:13:03] ostriches ^^ [22:13:05] 06Operations, 10Ops-Access-Requests, 10EventBus, 06Services: Allow the Services team to administer the eventbus services - https://phabricator.wikimedia.org/T141013#2507103 (10Dzahn) [kafka1001:~] $ puppet agent -tv .. Notice: /Stage[main]/Admin/Admin::Hashgroup[eventbus-admins]/Admin::Group[eventbus-admin... [22:14:35] 06Operations, 10Ops-Access-Requests, 10EventBus, 06Services: Allow the Services team to administer the eventbus services - https://phabricator.wikimedia.org/T141013#2507104 (10Dzahn) 05Open>03Resolved a:03Dzahn ``` [kafka1001:~] $ sudo cat /etc/sudoers.d/eventbus-admins # This file is managed by Pup... [22:16:14] (03Restored) 10Dzahn: admin: add shell account for Jan Dittrich [puppet] - 10https://gerrit.wikimedia.org/r/301721 (https://phabricator.wikimedia.org/T141339) (owner: 10Dzahn) [22:22:58] (03PS1) 10Chad: Gerrit: Set cache.projects.loadOnStartup = true [puppet] - 10https://gerrit.wikimedia.org/r/301898 [22:23:28] ^ That's kind of cool actually. [22:23:36] Will make post-restart behavior much faster for users. [22:25:48] (03CR) 10Paladox: [C: 031] Gerrit: Set cache.projects.loadOnStartup = true [puppet] - 10https://gerrit.wikimedia.org/r/301898 (owner: 10Chad) [22:26:47] :) [22:28:18] PROBLEM - puppet last run on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:29:07] PROBLEM - restbase endpoints health on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:29:52] (03PS1) 10Yuvipanda: prometheus: Don't include base::firewall automatically [puppet] - 10https://gerrit.wikimedia.org/r/301900 [22:30:18] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [22:31:19] (03PS1) 10Yuvipanda: prometheus: Allow applying on trusty/precise hosts as a noop [puppet] - 10https://gerrit.wikimedia.org/r/301901 [22:34:33] (03CR) 10Yuvipanda: [C: 032] prometheus: Don't include base::firewall automatically [puppet] - 10https://gerrit.wikimedia.org/r/301900 (owner: 10Yuvipanda) [22:34:51] (03PS8) 10Yuvipanda: k8s: Deploy worker automatically [puppet] - 10https://gerrit.wikimedia.org/r/301875 [22:35:27] (03PS9) 10Yuvipanda: k8s: Deploy worker automatically on first run [puppet] - 10https://gerrit.wikimedia.org/r/301875 [22:35:40] (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Deploy worker automatically on first run [puppet] - 10https://gerrit.wikimedia.org/r/301875 (owner: 10Yuvipanda) [22:35:53] (03PS2) 10Yuvipanda: prometheus: Don't include base::firewall automatically [puppet] - 10https://gerrit.wikimedia.org/r/301900 [22:36:00] (03CR) 10Yuvipanda: [V: 032] prometheus: Don't include base::firewall automatically [puppet] - 10https://gerrit.wikimedia.org/r/301900 (owner: 10Yuvipanda) [22:36:07] (03PS2) 10Yuvipanda: prometheus: Allow applying on trusty/precise hosts as a noop [puppet] - 10https://gerrit.wikimedia.org/r/301901 [22:36:15] (03CR) 10Yuvipanda: [C: 032 V: 032] prometheus: Allow applying on trusty/precise hosts as a noop [puppet] - 10https://gerrit.wikimedia.org/r/301901 (owner: 10Yuvipanda) [22:36:28] PROBLEM - MegaRAID on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:36:28] PROBLEM - puppet last run on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:37:28] PROBLEM - MD RAID on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:37:49] PROBLEM - configured eth on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:38:28] PROBLEM - cassandra-a service on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:38:59] PROBLEM - DPKG on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:39:37] PROBLEM - Restbase root url on xenon is CRITICAL: Connection refused [22:40:08] PROBLEM - Check size of conntrack table on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:40:18] (03PS1) 10Yuvipanda: prometheus: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/301902 [22:40:36] (03CR) 10Yuvipanda: [C: 032 V: 032] prometheus: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/301902 (owner: 10Yuvipanda) [22:41:29] (03PS1) 10BBlack: openssl (1.0.2h-1~wmf2) jessie-wikimedia; urgency=medium [debs/openssl] - 10https://gerrit.wikimedia.org/r/301903 [22:44:29] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [50.0] [22:46:07] RECOVERY - Check size of conntrack table on xenon is OK: OK: nf_conntrack is 0 % full [22:46:28] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [22:46:37] PROBLEM - dhclient process on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:50:37] PROBLEM - salt-minion processes on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:52:12] (03PS2) 10BBlack: openssl (1.0.2h-1~wmf2) jessie-wikimedia; urgency=medium [debs/openssl] - 10https://gerrit.wikimedia.org/r/301903 [22:52:29] RECOVERY - salt-minion processes on xenon is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [22:55:58] PROBLEM - Check size of conntrack table on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:58:28] PROBLEM - salt-minion processes on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:10:45] (03CR) 10Halfak: [WIP/POC/POS] Add python version of maintain-replicas script (031 comment) [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) (owner: 10Alex Monk) [23:13:49] !log installed openssl-1.0.2h-1~wmf2 on pinkunicorn for the weekend (not on carbon yet) - https://gerrit.wikimedia.org/r/301903 [23:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:17:18] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:08] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 378 bytes in 1.414 second response time [23:23:47] PROBLEM - SSH on xenon is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:58] RECOVERY - Check size of conntrack table on xenon is OK: OK: nf_conntrack is 0 % full [23:24:28] RECOVERY - salt-minion processes on xenon is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [23:25:38] RECOVERY - SSH on xenon is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [23:30:00] PROBLEM - Check size of conntrack table on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:30:38] PROBLEM - salt-minion processes on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:38:02] mutante: are you still around? [23:40:07] is there someone from ops around? [23:41:08] PROBLEM - Disk space on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:42:35] hi urandom [23:42:39] YuviPanda: hi [23:42:44] 'sup [23:42:48] xenon.eqiad.wmnet is... Bad [23:42:52] ah [23:42:53] it's not really production [23:42:56] needs mgmt kick? [23:42:58] RECOVERY - Disk space on xenon is OK: DISK OK [23:42:59] but it's monitored as such [23:43:08] right [23:43:16] so I can either silence it until next week [23:43:21] or try to reboot it from mgmt [23:43:25] YuviPanda: it has 10 acpi_pad kernel processes that are consuming a lot of cpu [23:43:31] load average of like 165 [23:43:34] wierd [23:43:44] all machines are monitored like that [23:43:46] that sounds like something hardware related maybe [23:43:54] have you tried rebooting it, urandom? [23:44:21] YuviPanda: no, and i can try, but wanted to consult someone in ops in case someone wanted to look at it first [23:44:28] (and in case it doesn't come back up) [23:44:36] YuviPanda: shall i try? [23:44:46] urandom yep [23:44:57] !log Rebooting xenon.eqiad.wmnet [23:45:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:46:12] W: molly-guard: SSH session detected! [23:46:12] Please type in hostname of the machine to reboot: Good thing I asked; I won't reboot xenon ... [23:46:49] did you type the full name? [23:46:52] it expects just 'xenon' [23:46:56] just do xenon I think [23:47:10] * urandom tries again [23:48:27] (03PS4) 10Paladox: Rely on commits name instead of branch [puppet] - 10https://gerrit.wikimedia.org/r/301849 [23:48:34] i think the problem was, it didn't echo the prompt until it had already timed out [23:48:39] it's... struggling [23:49:01] but i got the prompt a second time around, so maybe it'll bounce [23:49:22] * urandom gets out and pushes [23:51:57] PROBLEM - SSH on xenon is CRITICAL: Connection timed out [23:52:57] PROBLEM - Disk space on xenon is CRITICAL: Timeout while attempting connection [23:53:21] YuviPanda: I think we should stick a fork in it, it's done. [23:53:32] I don't think it's coming back up [23:53:48] PROBLEM - cassandra-a CQL 10.64.0.202:9042 on xenon is CRITICAL: Connection timed out [23:54:20] urandom ok. I'll try rebooting from mgmt [23:54:26] YuviPanda: thanks man [23:55:28] RECOVERY - MD RAID on xenon is OK: OK: Active: 9, Working: 9, Failed: 0, Spare: 0 [23:55:38] that looks promising [23:55:45] come on xenon, we need you! [23:55:47] RECOVERY - configured eth on xenon is OK: OK - interfaces up [23:55:48] RECOVERY - SSH on xenon is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [23:55:58] RECOVERY - Check size of conntrack table on xenon is OK: OK: nf_conntrack is 0 % full [23:56:06] !log hardreset xenon [23:56:09] whoops [23:56:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:56:13] well well [23:56:48] I'm on console now, it's booting up again [23:56:57] again? [23:57:48] * urandom lights a black candle [23:57:57] urandom yeah, because I didn't see that it was coming back up before rebooting... [23:58:05] oh, i see [23:58:35] YuviPanda: how long does a normal boot sequence take? [23:58:46] a few minutes [23:58:51] does it spend a lot of time in BIOS stuff? [23:58:56] (I can see it booting) [23:58:59] yeah [23:59:04] k [23:59:09] PROBLEM - Host xenon is DOWN: PING CRITICAL - Packet loss = 100% [23:59:11] it looks clean so far [23:59:33] yeah, it's pinging [23:59:54] it spent almost a minute bringing up ferm