[00:00:04] RoanKattouw, ^d, marktraceur, MaxSem, kaldari: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141119T0000). Please do the needful. [00:00:12] on it [00:00:21] kaldari|2, are you ready? :P [00:05:44] ori, I think it's safe to try repooling mw1205 - the branch that had issues with sync is not active anymore:) [00:06:01] MaxSem: did you sync-common? [00:06:06] yep [00:06:16] !log repooled mw1205 [00:06:16] and then Sam did a full scap [00:06:20] Logged the message, Master [00:06:34] thanks - let's see how it goes [00:07:22] aude, do you guys know about division by zero in WD? [00:08:41] dammit, mw1025 is still hurt [00:09:48] !log gracefulled apache on mw1205 (suspect an APC bug) [00:09:50] Logged the message, Master [00:14:37] (03Abandoned) 1020after4: Fix a bug in redirector that broke the alternate-files-domain Also, don't die when connection fails, just log the error and return so that there is no chance of this interfering with normal operation of phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174310 (owner: 1020after4) [00:15:29] (03Abandoned) 1020after4: Clean up some puppet-lint errors and warnings [puppet] - 10https://gerrit.wikimedia.org/r/174288 (owner: 1020after4) [00:19:32] (03CR) 10MaxSem: [C: 032] Adding 'types of actors' WikiGrok campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174303 (owner: 10Kaldari) [00:19:44] (03Merged) 10jenkins-bot: Adding 'types of actors' WikiGrok campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174303 (owner: 10Kaldari) [00:21:01] !log maxsem Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/174303 (duration: 00m 05s) [00:21:05] Logged the message, Master [00:22:51] (03CR) 1020after4: "@Dzahn: $passwords::mysql::phabricator::maniphest_user" [puppet] - 10https://gerrit.wikimedia.org/r/173483 (owner: 1020after4) [00:24:55] (03PS1) 10Springle: Switch m2-master to db1046 [puppet] - 10https://gerrit.wikimedia.org/r/174326 [00:24:57] (03PS1) 10Springle: switch m2-master CNAME to dbproxy1002 [dns] - 10https://gerrit.wikimedia.org/r/174327 [00:25:04] (03CR) 10jenkins-bot: [V: 04-1] switch m2-master CNAME to dbproxy1002 [dns] - 10https://gerrit.wikimedia.org/r/174327 (owner: 10Springle) [00:28:06] E: This script only supports gdnsd 2.x [00:32:38] bblack: ^ appears I caught jenkins and dns at a bad time? [00:32:51] (03PS8) 10Dzahn: (WIP) generate wikimedia wiki entries from helper [dns] - 10https://gerrit.wikimedia.org/r/171769 (https://bugzilla.wikimedia.org/38799) [00:35:47] (03PS2) 10Springle: switch m2-master CNAME to dbproxy1002.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/174327 [00:37:19] now it's happy. oh well [00:38:01] !log EventLogging: deployed 423f7dd5b2b5 & restarted. [00:38:03] Logged the message, Master [00:38:04] qchris: ^ [00:38:05] (03CR) 10Springle: [C: 032] Switch m2-master to db1046 [puppet] - 10https://gerrit.wikimedia.org/r/174326 (owner: 10Springle) [00:38:15] ori: Thanks! [00:38:40] (03CR) 10Dzahn: "re: comments from bblack above: hardcoded both. ran generating script on a labs instance. result is https://phabricator.wikimedia.org/P89" [dns] - 10https://gerrit.wikimedia.org/r/171769 (https://bugzilla.wikimedia.org/38799) (owner: 10Dzahn) [00:38:48] springle: odd [00:40:05] springle: no, that's probably my fault. I updated gallium, but I think I need to go update some other boxes that are jenkins slaves [00:40:12] (to have 2.x on all linting boxes) [00:40:56] yeah the lint failure happened on lanthanum [00:42:27] hopefully it's fixed now, those are the only two prod ci slaves anyways [00:43:56] bblack: thanks :) [00:46:14] (03CR) 10Springle: [C: 032] switch m2-master CNAME to dbproxy1002.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/174327 (owner: 10Springle) [00:48:42] !log m2-master CNAME switch to dbproxy1002, and db1046 to primary backend [00:48:45] Logged the message, Master [00:51:21] (03Abandoned) 1020after4: preamble script to read client address from HTTP_X_FORWARDED_FOR [puppet] - 10https://gerrit.wikimedia.org/r/168509 (owner: 1020after4) [00:51:30] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 315 seconds [00:51:59] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 345 seconds [00:52:59] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:53:32] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -1 seconds [00:59:06] (03CR) 10Rush: "I don't the why but I know that: $manifest_user = 'phmanifest'" [puppet] - 10https://gerrit.wikimedia.org/r/173483 (owner: 1020after4) [01:03:38] (03PS1) 1020after4: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 [01:04:25] (03CR) 10jenkins-bot: [V: 04-1] Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 (owner: 1020after4) [01:07:57] (03PS2) 1020after4: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 [01:08:44] (03CR) 10jenkins-bot: [V: 04-1] Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 (owner: 1020after4) [01:14:58] (03PS1) 10Rush: phab cleaner set native dict type files.image-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/174342 [01:15:00] (03PS1) 10Rush: phab during migration raise upload limit [puppet] - 10https://gerrit.wikimedia.org/r/174343 [01:15:17] (03PS2) 10Rush: phab cleaner set native dict type files.image-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/174342 [01:15:40] (03CR) 10Rush: [C: 032 V: 032] phab cleaner set native dict type files.image-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/174342 (owner: 10Rush) [01:20:22] !log ori Synchronized php-1.25wmf8/extensions/SyntaxHighlight_GeSHi: Ibb0f7c24: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) [01:20:30] Logged the message, Master [01:23:29] !log ori Synchronized php-1.25wmf7/extensions/SyntaxHighlight_GeSHi: I788e1beb8: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) [01:23:31] Logged the message, Master [01:28:55] !log Restarted EventLogging mysql-m2 consumer to pick up switch to dbproxy1002 [01:28:58] Logged the message, Master [01:39:50] PROBLEM - Apache HTTP on mw1017 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:20] PROBLEM - HHVM rendering on mw1017 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:37] that was me, sorry [01:40:49] RECOVERY - Apache HTTP on mw1017 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.057 second response time [01:41:09] RECOVERY - HHVM rendering on mw1017 is OK: HTTP OK: HTTP/1.1 200 OK - 68164 bytes in 0.169 second response time [02:13:39] ori, I think https://gerrit.wikimedia.org/r/#/c/167700/ is ready [02:14:12] !log LocalisationUpdate completed (1.25wmf7) at 2014-11-19 02:14:12+00:00 [02:14:17] Logged the message, Master [02:14:21] (03CR) 10Ori.livneh: [C: 032] Add cassandra role [puppet] - 10https://gerrit.wikimedia.org/r/167700 (owner: 10Ottomata) [02:15:17] ori: thanks! [02:15:23] np [02:16:14] also useful for the thing you just chatted about with James & Roan ;) [02:21:28] you have good hearing :P [02:26:12] !log LocalisationUpdate completed (1.25wmf8) at 2014-11-19 02:26:12+00:00 [02:26:19] Logged the message, Master [03:04:45] (03PS1) 10Ori.livneh: hhvm: enable perf_pid.map files w/automatic pruning [puppet] - 10https://gerrit.wikimedia.org/r/174356 [03:22:49] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:59] PROBLEM - CI: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: integration.integration-puppetmaster.diskspace._var.byte_avail.value (20.00%) [03:39:19] (03PS3) 1020after4: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 [03:39:45] (03PS4) 1020after4: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 [03:40:50] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:45:32] (03CR) 1020after4: "changes rolled into https://gerrit.wikimedia.org/r/#/c/174335/" [puppet] - 10https://gerrit.wikimedia.org/r/174310 (owner: 1020after4) [03:46:51] (03CR) 1020after4: "@rush is it manifest_user instead of maniphest_user ?" [puppet] - 10https://gerrit.wikimedia.org/r/173483 (owner: 1020after4) [03:48:09] (03CR) 1020after4: "reintroducing this change in https://gerrit.wikimedia.org/r/#/c/174335/" [puppet] - 10https://gerrit.wikimedia.org/r/173483 (owner: 1020after4) [03:53:40] PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [90.0] [04:00:50] RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [60.0] [04:15:00] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 19 04:15:00 UTC 2014 (duration 14m 59s) [04:15:02] Logged the message, Master [04:15:25] !log mw1020: disabled puppet & restarted hhvm w/hhvm.eval.perf_pid_map = true to test [04:15:27] Logged the message, Master [04:36:39] RECOVERY - CI: Low disk space on /var on labmon1001 is OK: OK: All targets OK [04:42:02] !log on mw1114: testing xhprof hack for T758 [04:42:07] Logged the message, Master [05:44:03] !log batchCAAntiSpoof finished with "34721605 user(s) done." [05:44:08] Logged the message, Master [05:55:49] (03PS1) 10Tim Starling: xhprof production profiling hack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174372 [06:28:39] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 3 failures [06:29:11] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:40] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:01] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:25] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:45:56] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:45] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:23:59] (03CR) 10Giuseppe Lavagetto: [C: 031] Move the openstack_version setting hiera. [puppet] - 10https://gerrit.wikimedia.org/r/173904 (owner: 10Andrew Bogott) [08:08:21] (03PS5) 10Qgil: Set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/174335 (owner: 1020after4) [09:42:18] PROBLEM - CI: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: integration.integration-puppetmaster.diskspace._var.byte_avail.value (11.11%) [09:44:49] !log hoo Synchronized php-1.25wmf8/extensions/Wikidata/: Fix url and commonsMedia UI editing (duration: 00m 42s) [09:44:56] Logged the message, Master [09:45:00] Lydia_WMDE: Can you verify? [09:48:13] Ok, seems to work on WD [09:59:13] (03Abandoned) 10Alexandros Kosiaris: ldap: replace iptables with ferm rule [puppet] - 10https://gerrit.wikimedia.org/r/117698 (owner: 10Matanya) [10:02:44] looking in a min :) [10:02:56] (03Abandoned) 10Alexandros Kosiaris: openstack: remove firewall access for amanda, replaced by bacula [puppet] - 10https://gerrit.wikimedia.org/r/133104 (owner: 10Matanya) [10:04:49] (03CR) 10Alexandros Kosiaris: [C: 032] git buildpackage basic configuration [debs/vips] - 10https://gerrit.wikimedia.org/r/113098 (owner: 10Hashar) [10:27:09] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [10:28:51] (03CR) 10Alexandros Kosiaris: "The git submodule also needs to be declared or the cassandra module will not be present in the puppetmaster (as it happens now)." [puppet] - 10https://gerrit.wikimedia.org/r/167700 (owner: 10Ottomata) [10:51:49] (03PS1) 10Alexandros Kosiaris: WIP: Make torrus class naming more clear [puppet] - 10https://gerrit.wikimedia.org/r/174389 [10:55:49] (03PS1) 10Giuseppe Lavagetto: varnish: remove cache separation for HHVM [puppet] - 10https://gerrit.wikimedia.org/r/174390 [11:53:11] ACKNOWLEDGEMENT - RAID on ms-be2007 is CRITICAL: CRITICAL: 1 failed LD(s) (Offline) Filippo Giunchedi RT #8902 [11:53:22] ACKNOWLEDGEMENT - puppet last run on ms-be2007 is CRITICAL: CRITICAL: Puppet has 1 failures Filippo Giunchedi RT #8902 [12:23:59] (03PS2) 10Alexandros Kosiaris: WIP: Modularize torrus [puppet] - 10https://gerrit.wikimedia.org/r/174389 [13:00:57] (03CR) 10Giuseppe Lavagetto: [C: 031] kill facilities.pp, move to nagios_common [puppet] - 10https://gerrit.wikimedia.org/r/173999 (owner: 10Dzahn) [13:23:09] (03CR) 10BBlack: "I still think we should remove Zend fallback before this step." [puppet] - 10https://gerrit.wikimedia.org/r/174390 (owner: 10Giuseppe Lavagetto) [13:24:17] (03PS1) 10ArielGlenn: audit ssh key use on production cluster [software] - 10https://gerrit.wikimedia.org/r/174408 [13:24:20] (03CR) 10jenkins-bot: [V: 04-1] audit ssh key use on production cluster [software] - 10https://gerrit.wikimedia.org/r/174408 (owner: 10ArielGlenn) [14:00:08] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [14:04:57] (03CR) 10Hashar: Allow puppetmaster to send reports to logstash (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/143788 (https://bugzilla.wikimedia.org/60690) (owner: 10BryanDavis) [14:05:00] (03Abandoned) 10Alexandros Kosiaris: torrus: move into a module [puppet] - 10https://gerrit.wikimedia.org/r/108498 (owner: 10Matanya) [14:05:06] (03CR) 10Hashar: [C: 031] Allow puppetmaster to send reports to logstash [puppet] - 10https://gerrit.wikimedia.org/r/143788 (https://bugzilla.wikimedia.org/60690) (owner: 10BryanDavis) [14:09:41] (03CR) 10Hashar: puppetmaster: Make time to keep old reports for configurable (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/174132 (https://bugzilla.wikimedia.org/73472) (owner: 10Yuvipanda) [14:09:49] RECOVERY - gdash.wikimedia.org on graphite1001 is OK: HTTP OK: HTTP/1.1 200 OK - 9447 bytes in 0.051 second response time [14:20:09] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [14:35:32] (03PS1) 10Andrew Bogott: Lay the ground for the openstack icehouse migration. [puppet] - 10https://gerrit.wikimedia.org/r/174413 [14:36:24] (03CR) 10jenkins-bot: [V: 04-1] Lay the ground for the openstack icehouse migration. [puppet] - 10https://gerrit.wikimedia.org/r/174413 (owner: 10Andrew Bogott) [14:38:35] (03PS2) 10Andrew Bogott: Lay the ground for the openstack icehouse migration. [puppet] - 10https://gerrit.wikimedia.org/r/174413 [14:43:32] Anyone know what's going on with beta enwiki [14:43:33] ? [14:44:13] I'm getting "500 hphp-invoke" [14:46:04] Nov 19 14:45:34 10.68.17.208 hhvm: #012Fatal error: unexpected N4HPHP13DataBlockFullE: Attempted to emit 23 byte(s) into a 35651584 byte DataBlock with 17 bytes available. This almost certainly means the TC is full. If this is the case, increasing Eval.JitASize, Eval.JitAColdSize, Eval.JitAFrozenSize and Eval.JitGlobalDataSize in the configuration file when running this script or application should fix this problem. [14:49:11] <_joe_> marktraceur: it needs a restart, and maybe some tuning [14:49:34] <_joe_> I am chasing a prod issue, but I'll take a look [14:49:35] /var is at 100% on deployment-mediawiki01 at least [14:49:47] <_joe_> oh man [14:49:56] So logging is killing it [14:50:07] <_joe_> 1 sec [14:52:51] <_joe_> marktraceur: it should be ok now [14:53:07] No dice. Maybe something else is wrong... [14:53:17] I'll keep digging [14:53:24] <_joe_> marktraceur: another server, proabbly [14:54:23] Now that apache can write to the logs, I'm only getting this: [14:54:23] Nov 19 14:52:39 10.68.17.96 apache2[7692]: [proxy:error] [pid 7692] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [14:54:28] But that may be unrelated [14:55:09] <_joe_> marktraceur: that's bogus [14:55:17] <_joe_> marktraceur: try now? [14:55:28] Ooh, loading [14:55:38] Winner! Thanks :) [14:55:41] <_joe_> it was deployment-mediawiki02 [14:55:51] K [15:10:04] (03PS3) 10Alexandros Kosiaris: WIP: Modularize torrus [puppet] - 10https://gerrit.wikimedia.org/r/174389 [15:15:11] (03CR) 10Andrew Bogott: [C: 032] Lay the ground for the openstack icehouse migration. [puppet] - 10https://gerrit.wikimedia.org/r/174413 (owner: 10Andrew Bogott) [15:35:49] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [15:43:18] thanks jgage [15:43:20] yt? [15:46:05] (03PS1) 10Andrew Bogott: Update hcaked libvirt driver for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174420 [15:46:07] (03PS1) 10Andrew Bogott: Update glance config for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174421 [15:48:24] log beginning upgrade of labs OpenStack from Havana to Icehouse [15:48:33] um [15:48:34] !log beginning upgrade of labs OpenStack from Havana to Icehouse [15:48:40] Logged the message, Master [15:48:48] (03PS2) 10Andrew Bogott: Update glance config for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174421 [15:48:50] (03PS2) 10Andrew Bogott: Update hacked libvirt driver for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174420 [15:49:38] !log backing up labs configs in ~andrew/osback/ [15:49:41] Logged the message, Master [15:50:19] * anomie sees nothing for SWAT (yet) [15:53:49] (03CR) 10Andrew Bogott: [C: 032] Update hacked libvirt driver for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174420 (owner: 10Andrew Bogott) [15:54:44] (03CR) 10Andrew Bogott: [C: 032] Update glance config for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174421 (owner: 10Andrew Bogott) [15:57:21] !log backed up all labs openstack databases to virt1000:~andrew/osback/havana-db-backup.sql [15:57:25] Logged the message, Master [15:58:20] I thought gi11es had something for SWAT [15:58:29] Maybe he forgot to add it [15:58:48] Oh, tgr|away is doing it tonight [15:58:50] nvm [16:00:05] manybubbles, anomie, ^d, marktraceur: Dear anthropoid, the time has come. Please deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141119T1600). [16:00:13] nope.jpg [16:02:05] (03PS1) 10Andrew Bogott: Consolidate entries for virt1001-1009. [puppet] - 10https://gerrit.wikimedia.org/r/174425 [16:02:06] (03PS1) 10Andrew Bogott: Update virt1000 (labs controller node) to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174426 [16:02:09] (03PS1) 10Andrew Bogott: Upgrade labnet1001 to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174427 [16:02:11] (03PS1) 10Andrew Bogott: Update labs compute nodes to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174428 [16:04:05] (03CR) 10Andrew Bogott: [C: 032] Consolidate entries for virt1001-1009. [puppet] - 10https://gerrit.wikimedia.org/r/174425 (owner: 10Andrew Bogott) [16:12:11] !log Jenkins: uninstalled Jenkins statsd plugin ( https://phabricator.wikimedia.org/T1278 ). It is overloading the statsd server with a bunch of metrics we don't care about ( https://phabricator.wikimedia.org/T1075 ) [16:12:19] Logged the message, Master [16:16:32] !log moved virt1000 db backup to /a/osback because it was /way/ too big to fit in my homedir [16:16:39] Logged the message, Master [16:17:45] (03PS1) 10Giuseppe Lavagetto: varnishkafka: do not reload on every puppet run [puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/174429 [16:18:01] <_joe_> bblack: ^^ [16:18:41] <_joe_> (20 lines of puppet instead of 4 lines of bash) [16:19:04] <_joe_> of course now I need to double-commit this [16:19:17] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] varnishkafka: do not reload on every puppet run [puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/174429 (owner: 10Giuseppe Lavagetto) [16:19:28] <_joe_> ottomata: ^^ [16:19:44] <_joe_> would you do the honours of merging this into the main puppet tree? [16:20:06] <_joe_> we've been without ganglia since you and ori did the varnishkafka change yesterday [16:20:29] <_joe_> (as I told you guys, btw) [16:21:10] (03PS1) 10Yuvipanda: icinga: Scream in -operations too when betacluster has issues [puppet] - 10https://gerrit.wikimedia.org/r/174430 [16:21:25] <_joe_> -2 Abandoned [16:21:42] <_joe_> YuviPanda: we have too much noise here already, IMO [16:21:48] https://phabricator.wikimedia.org/T1334 [16:22:16] betacluster has had broken puppet stuff for about a month because changes to prod-related stuff that nobody checked on betacluster [16:22:28] and 5 instances still have broken puppet, and nobody's workin on 'em atm. [16:22:31] I think it should be here [16:24:11] Yeah, I think we need some system for making Ops care when they break beta. [16:24:16] Not that I'm a fan of more noise in here... [16:24:23] hm, aye, thanks _joe_ [16:24:36] what, we've been wtihout ganglia? [16:24:40] <_joe_> ottomata: are you merging it then? [16:24:43] yes can do [16:24:49] (was still reading backlog) [16:25:02] <_joe_> ottomata: I told you guys when you applied your changes on cp1056 [16:25:09] <_joe_> that ganglia was failing [16:25:10] springle: https://phabricator.wikimedia.org/T1254, in case you missed it/emails :) [16:25:12] _joe_ I did not hear it! [16:25:25] <_joe_> the reason, turns out, is that ganglia is being restarted on every run [16:25:53] (03PS1) 10Ottomata: Update varnishkafka module with ganglia fix [puppet] - 10https://gerrit.wikimedia.org/r/174431 [16:25:55] <_joe_> kart_: I have one apache change of yours in CR, it seeems good, should I merge it? [16:26:11] (03CR) 10Ottomata: [C: 032 V: 032] Update varnishkafka module with ganglia fix [puppet] - 10https://gerrit.wikimedia.org/r/174431 (owner: 10Ottomata) [16:26:39] _joe_: Feel free. We're holding of adding woff2 fonts as of now, but config is okay. [16:26:41] _joe_ merged. [16:26:44] thank you. [16:26:56] <_joe_> ottomata: ok, checking now [16:27:28] <_joe_> ottomata: when we're in SF, I'll make you offer me a beer for every submodule you created :P [16:28:28] hahah [16:28:31] _joe_, ok! [16:28:55] and all these vagrant and labs users will buy me a beer! :p [16:29:11] or...maybe that's just me [16:29:15] maybe i'll buy myself a beer [16:29:17] so many beers! [16:29:25] <_joe_> ottomata: I don't think labs, but vagrant maybe [16:29:33] labs-vagrant, yo! :) [16:29:43] * bd808 would like some time to work on that again [16:30:03] <_joe_> ottomata: http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Text%2520caches%2520eqiad&tab=m&vn=&hide-hf=false [16:30:10] Jeff_Green: will buy me a beer then [16:30:13] for kafkatee in frack submodule [16:30:17] AT LEAST! [16:30:22] I want to get the lxc provider working with mw-vagrant so that labs-vagrant is really running vagrant [16:32:48] no beer for you! [16:32:56] mwuhahahhaha. [16:33:20] rats! [16:33:36] that's good anyway, i can really only handle one beer anyway [16:33:40] any more than that and I just get sleepy [16:34:10] _joe_...is that good? or just what was happening? [16:34:54] <^d> beer? [16:34:58] <^d> Somebody say something about beer? [16:35:08] <^d> (i guess it's 5 o'clock somewhere) [16:35:13] it's 10PM here [16:35:18] so about 6hours before my beer o'clock [16:36:19] RECOVERY - CI: Low disk space on /var on labmon1001 is OK: OK: All targets OK [16:36:44] !log upgrading virt1000 [16:36:49] Logged the message, Master [16:36:58] (03CR) 10Andrew Bogott: [C: 032] Update virt1000 (labs controller node) to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174426 (owner: 10Andrew Bogott) [16:39:15] YuviPanda: oh. I thought Kerala is in different timezone ;) [16:39:21] kart_: :) [16:39:29] kart_: you can still get illegal beer here [16:39:35] quasi legal, at least. is wrapped in paper [16:39:56] <_joe_> ottomata: that was what was happening [16:40:07] <_joe_> do you see the 20-min cycle of all of it? [16:41:42] ja [16:42:54] PROBLEM - DPKG on virt1000 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [16:44:03] RECOVERY - DPKG on virt1000 is OK: All packages OK [16:46:08] !log starting trusty upgrade of analytics1034 [16:47:26] (03PS1) 10Chad: enwiki gets Cirrus as primary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174434 [16:48:09] (03CR) 10Giuseppe Lavagetto: [C: 031] "\o/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174434 (owner: 10Chad) [16:48:32] (03PS2) 10Andrew Bogott: Update labs compute nodes to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174428 [16:48:34] (03PS2) 10Andrew Bogott: Upgrade labnet1001 to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174427 [16:48:36] (03PS1) 10Andrew Bogott: Update keystone config file for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174435 [16:49:50] * anomie is going to SWAT after all [16:50:02] (03PS3) 10Andrew Bogott: Update labs compute nodes to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174428 [16:50:03] (03PS3) 10Andrew Bogott: Upgrade labnet1001 to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174427 [16:50:05] (03PS2) 10Andrew Bogott: Update keystone config file for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174435 [16:50:07] (03CR) 10Yuvipanda: "(=｀ω´=)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174434 (owner: 10Chad) [16:50:13] <^d> T-10 minutes on Cirrus for enwiki. [16:50:33] <^d> Make sure to have your hard hats. [16:50:35] ^d: I love that I got it twice. [16:50:38] two offers of cake! [16:50:38] <^d> Make note of the closest exit. [16:50:44] you want me to sync? [16:51:00] (03CR) 10Jforrester: "Yay." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174434 (owner: 10Chad) [16:51:05] (03CR) 10Andrew Bogott: [C: 032] Update keystone config file for icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174435 (owner: 10Andrew Bogott) [16:51:11] <^d> manybubbles: I don't care either way :) [16:51:15] <^d> 9 minutes. [16:51:43] PROBLEM - DPKG on analytics1034 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [16:51:51] * bd808 grabs popcorn [16:51:59] hmm, I've no food item with me [16:52:34] <^d> Well you have 8 minutes to remedy that. [16:52:53] Just ran one last performance test. seems to be running just fine [16:53:05] everything is on the beach, and if I leave now I'm not coming back in 8mins. [16:53:09] I'll just grab some cold water [16:54:00] !log anomie Synchronized php-1.25wmf8/extensions/SecurePoll: SWAT: Fix SecurePoll jump wiki jumping [[gerrit:174436]] (duration: 00m 10s) [16:54:07] * anomie tests [16:55:16] Works for enwiki, accidentally. That's probably good enough until the train tomorrow. [16:55:49] anomie: Train is today :) [16:55:58] Reedy: I forgot that! Even better. [16:58:15] ^d: I'll build the commit then [16:58:39] <^d> I did already :) [16:58:52] <^d> https://gerrit.wikimedia.org/r/#/c/174434/ [16:59:12] (03PS1) 10Manybubbles: Use CirrusSearch by default for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174438 [16:59:26] Yay [16:59:32] <^d> Now we have 2 patches :D [16:59:57] <^d> Now to decide which one is better :D [17:00:04] manybubbles, ^d: Dear anthropoid, the time has come. Please deploy Search (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141119T1700). [17:00:14] (03Abandoned) 10Manybubbles: Use CirrusSearch by default for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174438 (owner: 10Manybubbles) [17:01:01] (03CR) 10Manybubbles: [C: 032] enwiki gets Cirrus as primary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174434 (owner: 10Chad) [17:01:10] (03Merged) 10jenkins-bot: enwiki gets Cirrus as primary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174434 (owner: 10Chad) [17:01:20] ^d: ok. I will sync then? [17:01:32] die lucene, die die [17:01:33] <^d> Make the sync msg fun :) [17:01:33] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 13 failures [17:01:41] <^d> "die lucene die die" is acceptable :) [17:02:14] PROBLEM - DPKG on virt1000 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [17:02:44] RECOVERY - DPKG on analytics1034 is OK: All packages OK [17:03:27] !log manybubbles Synchronized wmf-config/InitialiseSettings.php: Make CirrusSearch default search for enwiki. I am so excited. (duration: 00m 04s) [17:03:40] hmm pool queue errors [17:04:08] And you just can't hide it? [17:04:17] on prefix search [17:04:21] * ^d goes swimming in the pool [17:04:41] they look to be stopping [17:05:31] ^d: that was more load then the performance tests said we'd get: http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&c=Elasticsearch+cluster+eqiad&h=&tab=m&vn=&hide-hf=false&m=cpu_report&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name [17:06:42] <^d> Still not bad though. [17:07:59] <^d> manybubbles: Think we need to bump the queue settings a tad? [17:08:18] ^d: maybe a tad but we're not hitting them any more [17:08:23] RECOVERY - DPKG on virt1000 is OK: All packages OK [17:08:24] that was mostly a cache warming thing [17:08:25] (03PS1) 10Ottomata: Include spark CDH .deb in apt [puppet] - 10https://gerrit.wikimedia.org/r/174440 [17:08:29] <^d> Well probably because the cache filled. [17:08:30] <^d> Yeah [17:08:38] (03PS2) 10Ottomata: Include spark CDH .deb in apt [puppet] - 10https://gerrit.wikimedia.org/r/174440 [17:10:52] ^d: if we have trouble when we do the next rolling restart we need to have a think about it [17:11:01] <^d> Mhmm [17:11:09] <^d> Nother spat of pool counter full. [17:11:24] PROBLEM - Host analytics1034 is DOWN: PING CRITICAL - Packet loss = 100% [17:11:43] (03PS3) 10Ottomata: Include spark CDH .deb in apt [puppet] - 10https://gerrit.wikimedia.org/r/174440 [17:12:05] <^d> manybubbles: Maybe we should segment enwiki from the rest like you'd proposed before. [17:12:13] RECOVERY - Host analytics1034 is UP: PING OK - Packet loss = 0%, RTA = 2.04 ms [17:12:38] ^d: I think we should. [17:12:43] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:12:44] <^d> Let's do it [17:12:47] no need to rush into it, but I think we should at some point [17:12:48] k [17:12:54] you wanna propose a patch? [17:12:57] its 100% config I think [17:14:13] <^d> Working on it [17:14:14] <^d> Yeah [17:15:22] (03PS1) 10Chad: Segment enwiki's cirrus poolcounter traffic from the rest [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174442 [17:15:27] <^d> ^ manybubbles [17:16:01] (03CR) 10Ottomata: [C: 032] Include spark CDH .deb in apt [puppet] - 10https://gerrit.wikimedia.org/r/174440 (owner: 10Ottomata) [17:16:19] (03CR) 10Manybubbles: "Do we also want to lower the queue depths or are we ok there?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174442 (owner: 10Chad) [17:17:36] <^d> I think we'll be fine [17:19:27] !log starting trusty upgrade of analytics1035 [17:20:38] (03CR) 10Chad: [C: 032] "It'll probably be fine for now." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174442 (owner: 10Chad) [17:20:47] (03Merged) 10jenkins-bot: Segment enwiki's cirrus poolcounter traffic from the rest [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174442 (owner: 10Chad) [17:21:35] ^d: cool [17:21:48] ^d: I'm still logged into tin if you want me to sync it [17:22:03] !log demon Synchronized wmf-config/CirrusSearch-production.php: two pools > one pool (duration: 00m 04s) [17:22:06] <^d> ALready done [17:22:11] cool [17:22:23] going to go help give meds to my 2 year old...... [17:22:43] !log puppetmaster dead on vir1000, investigating. Port seems already in use. [17:22:52] <^d> manybubbles|away: Care care. [17:24:07] <^d> *take care [17:24:11] <^d> Words are hard [17:26:34] PROBLEM - DPKG on analytics1035 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [17:27:33] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 20 failures [17:31:14] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [17:31:46] "PHP Warning: Division by zero in /srv/mediawiki/php-1.25wmf8/extensions/Wikidata/vendor/data-values/geo/src/Formatters/GeoCoordinateFormatter.php on line 212" <-- known issue? [17:36:32] bd808: Seems there's a bug for it already [17:36:56] Reedy: Cool. I saw you even asked in the right place to find out :) [17:38:37] bd808: already fixed but need to backport [17:39:11] (03PS4) 10Andrew Bogott: Update labs compute nodes to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174428 [17:39:13] (03PS4) 10Andrew Bogott: Upgrade labnet1001 to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174427 [17:39:15] (03PS1) 10Andrew Bogott: Replaced some broken single-quotes [puppet] - 10https://gerrit.wikimedia.org/r/174447 [17:41:24] (03CR) 10Andrew Bogott: [C: 032] Replaced some broken single-quotes [puppet] - 10https://gerrit.wikimedia.org/r/174447 (owner: 10Andrew Bogott) [17:41:47] <^d> manybubbles: Segmenting enwiki did the trick. Just a few trickle of ArticleView timeouts which are normal. [17:42:32] ^d: I wasn't seeing any timeouts from it before the segmenting either though [17:42:39] I only saw them right when we went live [17:43:02] <^d> I'm not worried about 3 or 4 of those trickling in. [17:45:24] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:47:12] (03PS1) 10BBlack: s/eqiad/ulsfo/ as backup for AS/* + RU with esams primary [dns] - 10https://gerrit.wikimedia.org/r/174449 [17:49:05] (03CR) 10BBlack: [C: 032] s/eqiad/ulsfo/ as backup for AS/* + RU with esams primary [dns] - 10https://gerrit.wikimedia.org/r/174449 (owner: 10BBlack) [17:50:58] (03PS5) 10Andrew Bogott: Update labs compute nodes to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174428 [17:51:00] (03PS5) 10Andrew Bogott: Upgrade labnet1001 to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174427 [17:51:02] (03PS1) 10Andrew Bogott: Turn off keystone/redis for the icehouse upgrade. [puppet] - 10https://gerrit.wikimedia.org/r/174450 [17:52:06] (03CR) 10Andrew Bogott: [C: 032] Turn off keystone/redis for the icehouse upgrade. [puppet] - 10https://gerrit.wikimedia.org/r/174450 (owner: 10Andrew Bogott) [17:53:58] (03PS1) 10Gilles: Revert "Revert "Enable JPG thumbnail chaining on all wikis except commons"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/174451 [17:57:09] !log disabled keystone-redis because the current package doesn't work with icehouse [17:57:13] Logged the message, Master [18:00:04] tonythomas, Jeff_Green: Dear anthropoid, the time has come. Please deploy BounceHandler (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141119T1800). [18:02:35] lol [18:05:27] * tonythomas googles anthropoid [18:05:51] :) [18:06:01] ha [18:06:22] anthropoid = a higher primate, especially an ape or apeman. [18:06:23] :D [18:06:29] Reedy: time to merge ? [18:06:41] I'm just looking at the backports [18:06:45] okey [18:06:59] (03PS1) 10BBlack: Add ESAMS-EQIAD comments matching prev commit [dns] - 10https://gerrit.wikimedia.org/r/174454 [18:07:40] (03CR) 10BBlack: [C: 032] Add ESAMS-EQIAD comments matching prev commit [dns] - 10https://gerrit.wikimedia.org/r/174454 (owner: 10BBlack) [18:09:20] c'mon jenkins [18:13:06] tonythomas: I don't see any benefit from backporting the other 3 commits [18:14:03] Reedy: true [18:14:03] RECOVERY - DPKG on analytics1035 is OK: All packages OK [18:15:26] (03PS8) 10Reedy: Deploy BounceHandler extension to production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [18:15:32] (03CR) 10Reedy: [C: 032] Deploy BounceHandler extension to production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [18:15:41] (03Merged) 10jenkins-bot: Deploy BounceHandler extension to production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [18:16:03] Reedy: legoktm Jeff_Green ! yay ! [18:16:08] finally ! [18:16:13] :D [18:16:18] well, not synced yet :P [18:16:21] true [18:16:26] and generate VERP = false :\ [18:16:42] when can we switch that on atleast here in test2wiki ? [18:17:15] !log reedy Synchronized php-1.25wmf8/extensions/BounceHandler/: Bump (duration: 00m 14s) [18:17:17] Logged the message, Master [18:18:50] next should be https://gerrit.wikimedia.org/r/#/c/168622/ [18:18:57] !log reedy Synchronized wmf-config/: BounceHandler (duration: 00m 15s) [18:19:00] Logged the message, Master [18:19:01] waiting for things to show up in https://test2.wikipedia.org/wiki/Special:Version :) [18:19:44] It's there now [18:19:47] yay ! its showing up! [18:21:44] are we ready for the exim config? [18:22:30] and guess what - $wgGenerateVERP = true; [18:22:39] and now I got my email with VERPed return PATH [18:22:46] andrewbogott: hahaha, puppet on labs now thinks it's on EC2 and tries to connect to a hardcoded EC2 related address [18:22:50] 169.254.169.254 [18:22:59] https://dpaste.de/6bOm [18:23:08] um… ok, I'm going to ignroe that for now :) [18:23:18] andrewbogott: yeah, ok. [18:24:17] Jeff_Green: Yup, should be [18:24:47] should I be seeing outgoing messages with the verp envelop address yet? [18:25:17] YuviPanda: you mean every instance is trying to connect to that? [18:25:22] Or just on virt1000? [18:25:22] Reedy: why am I not seeing the API yet ? https://test2.wikipedia.org/wiki/api.php?action=bouncehandler [18:25:22] yep. [18:25:26] every instance [18:25:29] nothing wrong with virt1000 [18:25:32] um… what the heck? [18:25:32] I'm digging [18:25:34] yeah [18:25:39] it's a facter feature :) [18:25:54] PROBLEM - Host analytics1035 is DOWN: PING CRITICAL - Packet loss = 100% [18:25:56] I can't imagine how my changes caused that [18:25:59] although surely they did :( [18:26:38] facter 1.7.5 has a regression causing this bug again, downgrading to 1.6.5 fixes it. [18:26:43] RECOVERY - Host analytics1035 is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms [18:27:14] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:27:48] YuviPanda: but… did I somehow upgrade facter on the instances? [18:27:56] I guess maybe it's because of pluginsync [18:27:57] * legoktm reads up [18:28:16] (03PS6) 10Andrew Bogott: Update labs compute nodes to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174428 [18:28:18] (03PS6) 10Andrew Bogott: Upgrade labnet1001 to icehouse [puppet] - 10https://gerrit.wikimedia.org/r/174427 [18:28:20] (03PS1) 10Andrew Bogott: Another icehouse glance update. [puppet] - 10https://gerrit.wikimedia.org/r/174459 [18:28:25] tonythomas: I think you may have to rebase https://gerrit.wikimedia.org/r/#/c/168622/ ? [18:28:35] tonythomas: it's at https://test2.wikipedia.org/w/api.php?action=bouncehandler [18:28:39] 13:27 < whatami> This might have been mentioned already, but AT&T users are having trouble logging in. [18:28:42] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Can_not_log_in_on_AT.26T has all the information that I have. [18:28:46] /w instead of /wiki [18:29:20]