[00:01:16] New review: Dzahn; "this is not so much about giving Chad access, i added you reviewers for the puppet bikeshedding part..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/42791 [00:04:30] LeslieCarr: looks ok now ?! [00:06:55] New review: Aaron Schulz; "I recall this being really noisy, though my memory may exaggerate." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/42970 [00:11:30] root@fenari:/# puppet agent --configprint splay [00:11:30] true [00:11:35] which I suppose is the correct test [00:14:22] RECOVERY - NTP on db1036 is OK: NTP OK: Offset -1.430511475e-06 secs [00:18:06] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [00:18:07] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: Puppet has not run in the last 10 hours [00:18:07] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: Puppet has not run in the last 10 hours [00:18:07] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [00:18:07] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [00:26:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:20] New patchset: Tim Starling; "Fix mw-deployment-vars.erb" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43109 [00:29:29] New patchset: Pyoungmeister; "coredb: testing: ability to mark hosts as mariadb per node" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43110 [00:29:43] mutante: right now it is (post snmptt restart) [00:29:56] mutante: i also enabled debugging to see if it will core dump or something [00:30:25] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43109 [00:42:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.025 seconds [00:48:47] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43110 [00:50:22] LeslieCarr: i see.. so is that "snmptt on neon appears to freeze when logrotate runs" ? [00:50:43] yes/no ? [00:50:50] it looks like it did last night when logrotate ran [00:50:52] LeslieCarr: unrelated.. looks like we also need to look at that "check_ganglios_generic_value" [00:51:02] but it also forze at 17:46:53 and nothing excited happened [00:51:09] hrmm..ok [00:51:15] tstarling@tin:/srv/deployment/mediawiki/common$ mwscript eval.php --wiki=enwiki [00:51:15] PHP Warning: dba_open(/srv/deployment/mediawiki/common/wikiversions.cdb): failed to open stream: No such file or directory in /srv/deployment/mediawiki/common/multiversion/MWMultiVersion.php on line 260 [00:51:20] that's progress, I guess [00:52:01] Ryan_Lane/Reedy: is that file moving somewhere? [00:52:21] I don't think so [00:52:23] does it need to? [00:52:29] ditto [00:52:38] Just built a wikiversions.cdb [00:53:05] mutante: i think that's because those classes are commented out on neon -- i think both they should be uncommented and probably moved into the icinga class [00:53:05] we're also missing a LocalSettings.php [00:53:21] I was going to get git-deploy going in labs. is that something I should prioritize, or are you guys working on tin, now? [00:53:30] (in deployment-prep) [00:54:50] well, anomie was complaining about not being able to run maintenance scripts on tin [00:54:55] the work is probably much the same either way [00:55:06] which would you guys prefer? [00:55:43] I think things are mostly in a working state on tin right now for the rest of the work to continue [00:55:48] LeslieCarr: hrmm.yea. not a single line NOT containing "Normal" in snmptt.log ..will take a look again tomorrow.. [00:55:52] I would like both to be working with the same configuration [00:55:57] * Ryan_Lane nods [00:55:59] ok [00:56:09] I'll get deployment-prep going, then [00:56:55] mutante: if you see it stop processing check out snmptt.debug as well (just turned that on though) [00:57:20] btw, I changed the sync scripts slightly. when you run git deploy sync, it'll pause after the fetch state and provide a prompt where you can check the fetch status before continuing [00:57:23] yep, i saw it, not much in it yet though [00:57:52] so you can wait till all systems are finished with the fetch before doing a checkout [00:58:00] Ryan_Lane: ok [00:59:55] New patchset: Pyoungmeister; "coredb: need to pass mariadb to common" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43114 [01:01:01] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43114 [01:04:42] New review: Dereckson; "Already done. Ib1bb0040556c13e5d05ea610155e3cd7b7cce02c." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/40561 [01:05:33] Is it work pushing these commits to a branch on gerrit? [01:13:07] Reedy: shouldn't be [01:13:15] make a remote branch and push it in [01:13:24] you can push to a branch using git review [01:13:25] That's what I meant [01:13:27] ignore the gerrit ;) [01:13:55] notice that I moved the private stuff into a private repo [01:14:21] which does suck a little [01:14:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:34] we could treat each repo as dependent on each other [01:14:47] where when you start a deploy in one it also locks and deploys the other [01:15:31] I'm going to have to put in some logic to avoid infinite recursion there, though [01:15:40] require(/srv/deployment/mediawiki/common/1.21wmf7/../wmf-config/wgConf.php): failed to open stream: No such file or directory in /srv/deployment/mediawiki/common/wmf-config/CommonSettings.php on line 148 [01:15:55] needs to currently be /srv/deployment/mediawiki/common/1.21wmf7/../common/wmf-config/wgConf.php [01:15:57] :| [01:16:38] ah. yeah [01:17:00] hmm, there's gonna be a load of those [01:17:01] why not have a config variable for the common repo location? [01:17:37] is there a var for that? [01:17:55] if there isn't, that's what I'm suggesting :) [01:18:11] I've got a feeling I've seen one somewhere [01:18:46] we have 9 usages of $IP/.. in CommonSettings [01:19:08] maybe I was thinking of $wmfConfigDir [01:27:43] New patchset: Pyoungmeister; "testing: set db1036 back to noon-module for retest" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43119 [01:28:54] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43119 [01:29:27] [01:05:33] Is it work pushing these commits to a branch on gerrit?s [01:29:31] s/work/worth/ [01:30:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [01:32:03] TimStarling: any suggestions for a sensible way to determine where common is? [01:32:22] PROBLEM - Host db1036 is DOWN: PING CRITICAL - Packet loss = 100% [01:38:14] RECOVERY - Host db1036 is UP: PING OK - Packet loss = 0%, RTA = 26.49 ms [01:42:24] PROBLEM - MySQL Slave Delay on db1036 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:42:25] PROBLEM - MySQL Idle Transactions on db1036 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:42:25] PROBLEM - MySQL Slave Running on db1036 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:42:52] PROBLEM - MySQL Recent Restart on db1036 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:43:01] PROBLEM - MySQL disk space on db1036 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:43:11] PROBLEM - MySQL Replication Heartbeat on db1036 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:43:26] Reedy: how about __DIR__.'/..' ? [01:43:28] PROBLEM - Full LVS Snapshot on db1036 is CRITICAL: Connection refused by host [01:43:52] I guess my question was more "do we have to still use $IP?" [01:43:53] actually there is already MULTIVER_COMMON [01:44:11] Ah [01:44:12] I thought I'd seen something like that [01:44:25] from multiversion/defines.php [01:44:31] so yeah, you can just use that [01:44:51] * AaronSchulz snickers in the background [01:49:01] reedy@tin:/srv/deployment/mediawiki/common$ mwscript eval.php --wiki=enwiki [01:49:01] > echo $wgVersion; [01:49:01] 1.21wmf7 [01:49:21] one step at a time [01:49:46] I think I might've broken getExtendedVersionNumber though [01:49:46] RECOVERY - MySQL Recent Restart on db1036 is OK: OK seconds since restart [01:49:55] RECOVERY - MySQL disk space on db1036 is OK: DISK OK [01:49:55] RECOVERY - MySQL Replication Heartbeat on db1036 is OK: OK replication delay seconds [01:50:39] RECOVERY - Full LVS Snapshot on db1036 is OK: OK no full LVM snapshot volumes [01:50:58] RECOVERY - MySQL Slave Delay on db1036 is OK: OK replication delay seconds [01:50:59] RECOVERY - MySQL Idle Transactions on db1036 is OK: OK longest blocking idle transaction sleeps for seconds [01:50:59] RECOVERY - MySQL Slave Running on db1036 is OK: OK replication [01:54:56] re puppet splay fail: puppet only sleeps a random amount of time once, immediately after it starts up [01:55:17] then it sleeps for $runinterval *after* each run [01:55:32] so the interval is actually the amount of time it takes to run, plus $runinterval [01:57:46] so if the server stalls for 10 minutes and then allows a whole lot of clients to finish their runs simultaneously, they will stay together for the next run [02:01:25] Think we're going to have to chase a few files around to get them into version control [02:01:41] like LocalSettings.php as part of making the branch, rather than from checkoutMediaWiki [02:01:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:02:30] as much as I'd like to avoid it, it's possible to do tasks at the end of a checkout run [02:02:35] we do it in parsoid, for instance [02:02:51] in parsoid, we link to the configuration file in the config repo [02:02:56] Can't see any reason why it can't be in the wmf branches [02:02:59] ok [02:03:11] that's easier and less prone to failure [02:03:11] < 5 lines of code ;D [02:03:25] even copy paste most of it [02:03:51] this is going to cause me issues on other sites where I use the wmf branches, but I'll just deal with that :D [02:04:27] PROBLEM - NTP on db1036 is CRITICAL: NTP CRITICAL: Offset unknown [02:05:49] TimStarling: That's most of it done... php- prefixes stripped, dblists moved [02:07:54] RECOVERY - NTP on db1036 is OK: NTP OK: Offset -0.006457686424 secs [02:08:38] who the fuck delete my mariadb packages from the wikimedia repo and didn't email or otherwise tell me? [02:09:23] larry ellison? [02:09:40] someone deleted them? [02:09:54] who deletes packages at all? [02:09:57] reprepro changes: [02:09:57] remove precise-wikimedia deb universe i386 mysql-common 5.5.28-mariadb-wmf201212041~precise -- pool/universe/m/mariadb-5.5/mysql-common_5.5.28-mariadb-wmf201212041~precise_all.deb [02:10:02] someone who should be fired [02:10:04] wtf [02:10:30] someone who, if they're remote, that won't protect them [02:16:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.365 seconds [02:27:16] !log LocalisationUpdate completed (1.21wmf7) at Thu Jan 10 02:27:16 UTC 2013 [02:27:28] Logged the message, Master [02:32:58] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 218 seconds [02:33:42] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 237 seconds [02:33:43] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 247 seconds [02:33:51] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 242 seconds [02:34:50] New patchset: Asher; "changing log level to verbose in order to log key fingerprints for all logins" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43125 [02:35:11] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43125 [02:38:24] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Thu Jan 10 02:38:17 UTC 2013 [02:41:03] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [02:41:21] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Thu Jan 10 02:41:05 UTC 2013 [02:41:22] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Thu Jan 10 02:41:06 UTC 2013 [02:41:58] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 0 seconds [02:42:25] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [02:42:25] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 0 seconds [02:43:55] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Thu Jan 10 02:43:48 UTC 2013 [02:45:15] RECOVERY - Puppet freshness on mc16 is OK: puppet ran at Thu Jan 10 02:44:51 UTC 2013 [02:46:09] RECOVERY - Puppet freshness on mc5 is OK: puppet ran at Thu Jan 10 02:45:53 UTC 2013 [02:46:19] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Thu Jan 10 02:46:03 UTC 2013 [02:48:24] RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Thu Jan 10 02:47:58 UTC 2013 [02:48:25] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Thu Jan 10 02:47:59 UTC 2013 [02:48:25] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Thu Jan 10 02:48:00 UTC 2013 [02:49:55] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Thu Jan 10 02:49:34 UTC 2013 [02:51:36] !log LocalisationUpdate completed (1.21wmf6) at Thu Jan 10 02:51:34 UTC 2013 [02:51:46] Logged the message, Master [02:55:18] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Thu Jan 10 02:54:55 UTC 2013 [02:56:49] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Thu Jan 10 02:56:37 UTC 2013 [02:57:25] RECOVERY - Puppet freshness on kuo is OK: puppet ran at Thu Jan 10 02:57:08 UTC 2013 [02:57:25] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Thu Jan 10 02:57:12 UTC 2013 [02:57:52] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Thu Jan 10 02:57:34 UTC 2013 [03:00:16] RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Thu Jan 10 02:59:50 UTC 2013 [03:01:19] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Thu Jan 10 03:01:15 UTC 2013 [03:04:19] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Thu Jan 10 03:03:54 UTC 2013 [03:05:22] RECOVERY - Puppet freshness on mc13 is OK: puppet ran at Thu Jan 10 03:04:52 UTC 2013 [03:05:22] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Thu Jan 10 03:05:04 UTC 2013 [03:06:52] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Thu Jan 10 03:06:42 UTC 2013 [03:14:21] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Thu Jan 10 03:13:57 UTC 2013 [03:17:13] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [03:19:10] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [03:21:07] New patchset: Reedy; "Add .deploy to gitignore" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43127 [03:21:35] New patchset: Reedy; "Remove php- and ExtensionMessages from .gitignore" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43128 [03:22:16] New patchset: Reedy; "Updated all docroot symlinks" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43129 [03:22:46] New patchset: Reedy; "Move dblists to their own folder (dblists)" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43130 [03:23:13] New patchset: Reedy; "Update noc related symlinks (non http config)" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43131 [03:23:37] New patchset: Reedy; "Drop the php- prefix" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43132 [03:24:22] New patchset: Reedy; "Various updates and changes to make it all work..." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43133 [03:24:34] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43127 [03:24:42] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43128 [03:25:19] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43129 [03:25:55] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43130 [03:26:20] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43131 [03:26:26] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43132 [03:26:38] New patchset: Ryan Lane; "(bug 43339) deployment roles for beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/42549 [03:26:39] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43133 [03:28:11] New patchset: Ryan Lane; "(bug 43339) deployment roles for beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/42549 [03:29:24] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/42549 [03:32:05] New patchset: Ryan Lane; "Remove trailing slash on hostname for common repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43134 [03:32:21] PROBLEM - Host srv266 is DOWN: PING CRITICAL - Packet loss = 100% [03:32:47] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43134 [03:35:40] RECOVERY - Puppet freshness on lvs1004 is OK: puppet ran at Thu Jan 10 03:35:26 UTC 2013 [03:36:14] New patchset: Reedy; "Fixup live-1.5 symlinks" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43135 [03:37:03] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43135 [03:39:46] New patchset: Reedy; "Attempt at fixing phpunit tests for wmf-config.." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [03:42:33] New patchset: Reedy; "Attempt at fixing phpunit tests for wmf-config.." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [03:47:07] New patchset: Reedy; "Attempt at fixing phpunit tests for wmf-config.." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [03:47:58] New patchset: Ryan Lane; "Include packages conditionally" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43137 [03:50:13] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43137 [03:51:22] New patchset: Reedy; "Attempt at fixing phpunit tests for wmf-config.." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [03:55:26] New patchset: Reedy; "Attempt at fixing phpunit tests for wmf-config.." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [03:56:09] New patchset: Reedy; "Attempt at fixing phpunit tests for wmf-config.." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [03:57:03] New patchset: Reedy; "Attempt at fixing phpunit tests for wmf-config.." [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [03:57:18] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43136 [04:05:13] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [04:12:45] New patchset: Reedy; "Ignore "SHA-1 metadata" in fatalmonitor" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/40766 [04:12:50] New patchset: Reedy; "Log sync-docroot calls" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/40311 [04:13:09] New review: Reedy; "Needs rebasing" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/37968 [04:13:15] New review: Reedy; "Needs rebasing" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/42133 [04:22:22] New patchset: Ryan Lane; "Disable management of the pillars top file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43138 [04:22:30] err: Failed to apply catalog: Could not find dependency File[/usr/local/bin/mw-update-l10n] for File[/usr/local/bin/wmf-beta-autoupdate] at /etc/puppet/manifests/misc/beta.pp:30 [04:22:31] * Ryan_Lane sighs [04:23:03] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43138 [04:31:52] New patchset: Tim Starling; "Restart puppet every 4 hours instead of every 24" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43139 [04:33:54] New patchset: Tim Starling; "Restart puppet every 4 hours instead of every 24" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43139 [04:34:53] any objections to me merging that? [04:35:06] nope [04:35:39] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43139 [05:12:09] Change abandoned: Tim Starling; "Per Ryan's comment" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/13293 [05:12:52] New patchset: Tim Starling; "(bug 20409) Use NE flag for rewrites that probably need to deal with special chars." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/34113 [05:13:16] Change merged: Tim Starling; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/34113 [05:20:51] Sweet. :-) [05:29:30] ah. forgot to puppetize redis [05:39:02] New patchset: Ryan Lane; "Add redis to deployment hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43142 [05:44:08] New patchset: Ryan Lane; "Add sudo policy to deployment hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43144 [05:44:44] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43142 [05:45:12] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43144 [05:48:04] New patchset: Ryan Lane; "Remove duplicate def" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43145 [05:48:37] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43145 [05:49:30] New patchset: Tim Starling; "Don't proxy secure.wikimedia.org to the main apache cluster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43146 [05:50:04] TimStarling: \o/ [05:50:23] die secure die. heh [05:59:42] well, obviously repo dependencies are working [06:00:12] I purged redis's data when moving the database location, and force sync'd slot0 [06:00:35] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43146 [06:00:35] data showed up in deploy-info for both slot1 and l10n-slot0 [06:00:53] err [06:00:58] slot0 and l10n-slot0 [06:01:13] good [06:06:07] that last puppet change of mine didn't appear to do anything [06:06:18] lemme see [06:06:39] the secure.wikimedia.org file on singer never matched the one in puppet [06:06:50] o.O [06:06:59] maybe it's always been broken in puppet? [06:07:11] yes, probably [06:07:23] let me look at what apache_site does [06:07:51] there's a vim backup file on singer that suggests faidon just edited the file directly when he made his changes [06:08:14] ugh [06:08:29] apache_site only adds the link [06:08:48] the file isn't being included anywhere [06:08:51] let me fix that [06:08:59] just looking at planet.pp for comparison [06:09:05] it has a file {} for actually creating the file [06:09:12] yep [06:11:57] PROBLEM - Host bits-lb.esams.wikimedia.org_ipv6_https is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::a [06:13:04] ipv6 down for just one single -lb? [06:14:25] nagios-wm: you're full of shit [06:15:12] hrm [06:15:15] i'm looking [06:15:47] New patchset: Ryan Lane; "Make secure actually pull its config from puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43147 [06:16:15] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43147 [06:17:49] RECOVERY - Host bits-lb.esams.wikimedia.org_ipv6_https is UP: PING OK - Packet loss = 0%, RTA = 112.27 ms [06:19:12] New patchset: Tim Starling; "Changes for git-deploy" [operations/apache-config] (newdeploy) - https://gerrit.wikimedia.org/r/43148 [06:19:23] i am getting some ping loss when pinging directly to cr2-eqiad though not thoruhg [06:19:33] at this utilization level i shouldn't be getting any ping loss though [06:20:04] TimStarling: ok, it's pulling from puppet now [06:20:07] thanks [06:20:16] yw [06:26:37] New patchset: Tim Starling; "Fix secure.wikimedia.org VirtualHost" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43149 [06:27:02] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43149 [06:30:03] so it looks like switching over to git-deploy will be as simple as "mv common common-old && ln -s /srv/deployment/mediawiki/common common" [06:30:17] oh? [06:30:27] localization stuff still needs to happen... [06:31:07] I mean the particular part of the deployment process where you make apache use the new directory [06:31:17] ah [06:31:17] cool [06:31:37] actually it is: mv common-local common-old && ln -s /srv/deployment/mediawiki/common common-local [06:31:52] since there were one or two references to common-local [06:32:54] what still needs to be done on the localisation stuff? [06:33:13] did anomie finish the script that writes into the repos? [06:33:38] the repos are there, and repo dependencies work, but there's no data in them [06:33:50] also, are the references to the files good? [06:34:17] are we going to make a link from common to the repo? [06:34:42] I don't see any relevant commits from him [06:35:02] well, I'll work this out with him tomorrow, I guess [06:35:17] I'm finishing up adding deployment to beta [06:55:27] New patchset: Ryan Lane; "Use localhost for deploy-info util" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43152 [06:56:47] hm. I guess I should pull the pillar data for that hostname. [06:58:06] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43152 [07:26:04] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [07:30:22] New patchset: Ryan Lane; "Add private repo and fix reference to common repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43155 [07:33:15] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43155 [07:33:38] TimStarling: have the references to the private files also been updated? [08:05:43] New patchset: Adamw; "Move default config into a file" [operations/dumps] (master) - https://gerrit.wikimedia.org/r/43156 [08:05:55] New patchset: Hashar; "beta: fix autoupdater class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43157 [08:07:25] New patchset: Hashar; "(bug 43811) beta: fix autoupdater class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43157 [08:10:54] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43157 [08:20:08] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 221 seconds [08:20:26] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 231 seconds [08:25:51] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [08:29:18] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.020 second response time on port 11000 [08:32:36] New patchset: Hashar; "attempt to get mediawiki master under git deploy" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43158 [08:38:42] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43158 [08:53:03] New patchset: Hashar; "mw master branch to slotmaster" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43160 [08:53:32] Change merged: Hashar; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/43160 [08:56:42] New patchset: Ryan Lane; "wikidev group doesn't exist in labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43162 [08:57:29] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [08:57:30] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [08:57:36] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43162 [08:59:20] ok [08:59:22] perms changed [09:17:41] New patchset: MaxSem; "Try disabling noindex on some enwiki pages per discussion with Google" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43163 [09:25:41] New patchset: Hashar; "git-deploy: rename slotmaster to slotbeta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43164 [09:27:17] New patchset: Hashar; "git-deploy: rename slotmaster to beta0" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43164 [09:27:56] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43164 [09:52:38] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [09:53:53] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [10:00:08] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43163 [10:11:23] !log maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/43163/ - Try disabling noindex on some enwiki pages per discussion with Google' [10:11:33] Logged the message, Master [10:18:56] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: Puppet has not run in the last 10 hours [10:18:57] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: Puppet has not run in the last 10 hours [10:18:57] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [10:18:57] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [10:18:57] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [10:29:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [10:56:13] New patchset: Hashar; "contint: "luajit", "libluajit-5.1-dev" packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43182 [10:56:31] New review: Hashar; "Already in production, so feel free to merge whenever you want." [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/43182 [11:07:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.107 seconds [11:47:39] New patchset: Hashar; "contint: luajit, libluajit-5.1-dev and g++ packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43182 [11:48:08] !log gallium: manually installed luajit, libluajit-5.1-dev and g++ packages . Pending gerrit change is {{gerrit|43182}} [11:48:17] Logged the message, Master [11:48:46] New review: Hashar; "PS2 adds g++" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/43182 [11:50:27] New patchset: Hashar; "contint: packages needed to build WMF PHP extensions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43182 [11:50:54] New review: Hashar; "PS3 adds libthai-dev needed by wikidiff2" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/43182 [11:54:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:37] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/43182 [11:56:14] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 183 seconds [11:57:53] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 227 seconds [11:57:55] New review: Hashar; "it would list the list of all languages for each branch. Maybe I could hack something to have it un..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/42970 [12:02:59] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 0 seconds [12:03:08] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 0 seconds [12:07:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.034 seconds [12:47:40] New patchset: MaxSem; "Rm GeoData exceptions" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43202 [12:48:22] guillom: pardon j'ai quitté la mauvaise fenetre [12:48:25] guillom: le plus important pour décembre c'est le déploiement de Zuul et du nouveau workflow https://www.mediawiki.org/wiki/Continuous_integration/Workflow [12:48:36] pas de souci [12:49:01] et effectivement les tests ne sont plus lancés pour tout le monde. [12:49:25] en revanche on vérifie toujours quelques tests simple (genre que le PHP est valide, mais sans l'exécuter) [12:49:30] mais "trusted users" c'est toujours mieux que "juste les employés" =) [12:49:34] j'ai modifié [12:49:38] nice [12:49:47] omg French cabal:P [12:50:11] :P [12:50:26] New patchset: Hashar; "Contint requires a 512M tmpfs file fs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/35159 [12:50:46] MaxSem: feel free to join in :-] [12:51:07] guillom: pas grand chose à dire sur le projet "beta" pour le mois de décembre [12:51:08] по французски не бум-бум [12:51:24] hashar: pas de souci :) [12:51:53] guillom: though we had a small meeting with MaxSem + Arthur + Michelle to kick of Mobile support on the beta cluster [12:52:29] clope time :-D [12:52:45] MaxSem: Google Translate says "French for no boom-boom" [12:52:53] yes [12:53:24] "I don't speak French" to be precise [12:53:40] Ah. I would definitely not have guessed :) [12:54:37] (the meaning) [13:02:12] MaxSem: your beer drinking skill is good enough, that is the only thing that matter [13:02:44] I thought you French prefer wine? [13:03:29] احلاهاشر [13:03:31] :D [13:03:36] Oo, [13:03:45] MaxSem: I do both. Indeed prefer red wine [13:03:52] hello hashar [13:04:04] hiii [13:05:47] i am curious when we do the data center migration, if we changing anything like version of php we use? [13:05:58] we are changing [13:06:00] no [13:06:04] MaxSem: btw, when are we supposed to start working on bringing up Mobile support on beta? I got an account on mingle but that is about it for now [13:06:04] phew :) [13:06:08] both are running precise [13:06:36] though you should really write code agnostic to PHP version [13:06:39] aude: I think we do it January 21st till 23rd [13:06:49] * aude running my development environment on same version as wikimedia uses [13:06:52] MaxSem: indeed [13:06:53] aude: don't quote me on this though, just hearsays. I guess there will be a proper announcement. [13:07:14] * aude just wants to do extra testing on same version wikimedia uses [13:07:40] hashar, next week [13:07:47] MaxSem: good [13:07:53] (mobile on beta) [13:16:56] New review: Faidon; "Shouldn't it be 'requires => [ User['jenkins'], Group['jenkins'] ]." [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/35159 [13:18:53] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [13:20:50] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [13:22:56]