[00:22:40] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 16 Mar 2014 12:19:26 PM UTC [00:23:40] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [01:23:50] (03PS1) 10Springle: dbstore partman use regular db.cfg [operations/puppet] - 10https://gerrit.wikimedia.org/r/119000 [01:25:25] (03CR) 10Springle: [C: 032] dbstore partman use regular db.cfg [operations/puppet] - 10https://gerrit.wikimedia.org/r/119000 (owner: 10Springle) [01:29:10] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [01:50:57] it seems west coast verizon users have been reporting connection troubles over the past 2 weeks. [01:51:00] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#server_problems.3F [01:51:49] eh east coast [02:01:09] !log LocalisationUpdate failed: mwversionsinuse returned empty list [02:01:18] Logged the message, Master [02:44:20] (03CR) 10BryanDavis: "> drwxrws--- 6 root wikidev 4096 Mar 10 03:37 scap" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118946 (owner: 10Reedy) [02:49:23] and again [03:22:53] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 16 Mar 2014 12:19:26 PM UTC [03:23:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [05:25:35] (03PS1) 10ArielGlenn: remove WikiDump dependency, pep8-ify, cleanup usage message [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119003 [05:29:54] (03CR) 10ArielGlenn: [C: 032] remove WikiDump dependency, pep8-ify, cleanup usage message [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119003 (owner: 10ArielGlenn) [05:52:53] (03PS1) 10ArielGlenn: detabify and whitespace cleanup of html and other templates [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119004 [05:54:56] (03CR) 10ArielGlenn: [C: 032] detabify and whitespace cleanup of html and other templates [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119004 (owner: 10ArielGlenn) [06:23:53] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 16 Mar 2014 12:19:26 PM UTC [06:24:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [06:35:00] (03PS1) 10ArielGlenn: dumps templates, configs, dblists added to snapshot module [operations/puppet] - 10https://gerrit.wikimedia.org/r/119006 [06:38:00] (03CR) 10ArielGlenn: [C: 032] dumps templates, configs, dblists added to snapshot module [operations/puppet] - 10https://gerrit.wikimedia.org/r/119006 (owner: 10ArielGlenn) [06:47:50] (03PS1) 10ArielGlenn: eqiad snapshot hosts to use puppetized templates and configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119007 [06:48:34] (03CR) 10jenkins-bot: [V: 04-1] eqiad snapshot hosts to use puppetized templates and configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119007 (owner: 10ArielGlenn) [06:49:58] (03PS2) 10ArielGlenn: eqiad snapshot hosts to use puppetized templates and configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119007 [06:52:35] (03CR) 10ArielGlenn: [C: 032] eqiad snapshot hosts to use puppetized templates and configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119007 (owner: 10ArielGlenn) [07:11:44] (03PS1) 10ArielGlenn: snapshot central auth dump to use puppetized configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119008 [07:13:09] (03CR) 10ArielGlenn: [C: 032] snapshot central auth dump to use puppetized configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119008 (owner: 10ArielGlenn) [07:17:54] (03PS1) 10ArielGlenn: on snapshots fix config file names [operations/puppet] - 10https://gerrit.wikimedia.org/r/119009 [07:19:28] (03CR) 10ArielGlenn: [C: 032] on snapshots fix config file names [operations/puppet] - 10https://gerrit.wikimedia.org/r/119009 (owner: 10ArielGlenn) [07:54:51] (03PS1) 10ArielGlenn: puppetize page titles generation on snapshot hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/119011 [07:57:30] (03CR) 10ArielGlenn: [C: 032] puppetize page titles generation on snapshot hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/119011 (owner: 10ArielGlenn) [08:28:26] (03PS1) 10ArielGlenn: puppetize conf file for pagetitles dump on snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119013 [08:31:11] (03PS2) 10ArielGlenn: puppetize conf file for pagetitles dump on snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119013 [08:32:45] (03CR) 10ArielGlenn: [C: 032] puppetize conf file for pagetitles dump on snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119013 (owner: 10ArielGlenn) [08:59:58] (03PS1) 10Ori.livneh: GeoIP cookie: expand deployment from Labs to production [operations/puppet] - 10https://gerrit.wikimedia.org/r/119014 [09:04:03] PROBLEM - SSH on lvs4001 is CRITICAL: Server answer: [09:06:03] RECOVERY - SSH on lvs4001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [09:24:53] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 16 Mar 2014 12:19:26 PM UTC [09:25:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [09:45:26] (03CR) 10Mark Bergsma: Initial commit of pmacct module and role (036 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 (owner: 10Jkrauska) [10:28:20] (03PS1) 10ArielGlenn: detabify and whitespace fixes for one more dumps template [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119022 [10:29:12] (03CR) 10ArielGlenn: [C: 032] detabify and whitespace fixes for one more dumps template [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119022 (owner: 10ArielGlenn) [10:31:07] (03PS1) 10ArielGlenn: detabify, white space fixes for adds-changes template [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119023 [10:31:48] (03CR) 10ArielGlenn: [C: 032] detabify, white space fixes for adds-changes template [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119023 (owner: 10ArielGlenn) [10:37:52] (03PS1) 10ArielGlenn: update language about maxrevid content for addschanges dumps [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119024 [10:40:43] (03CR) 10ArielGlenn: [C: 032] update language about maxrevid content for addschanges dumps [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119024 (owner: 10ArielGlenn) [10:51:03] PROBLEM - LDAP on virt1000 is CRITICAL: Connection refused [11:01:36] (03PS1) 10ArielGlenn: puppetize adds-changes dumps (so-called incrementals) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119026 [11:04:33] (03CR) 10ArielGlenn: [C: 032] puppetize adds-changes dumps (so-called incrementals) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119026 (owner: 10ArielGlenn) [11:05:13] PROBLEM - LDAPS on virt1000 is CRITICAL: Connection refused [11:06:03] (03PS1) 10Gilles: Expose some headers to CORS requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/119027 [11:11:39] (03PS1) 10ArielGlenn: fix up 'maintained by puppet' notices for couple of dumps templates [operations/puppet] - 10https://gerrit.wikimedia.org/r/119028 [11:13:19] (03CR) 10ArielGlenn: [C: 032] fix up 'maintained by puppet' notices for couple of dumps templates [operations/puppet] - 10https://gerrit.wikimedia.org/r/119028 (owner: 10ArielGlenn) [11:28:50] (03PS2) 10Alexandros Kosiaris: Support spatial_ref_sys and comments in postgres [operations/puppet] - 10https://gerrit.wikimedia.org/r/117920 [11:43:07] (03PS1) 10Alexandros Kosiaris: Creating a spatially enabled db on osmlab dbs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119031 [11:56:15] (03CR) 10Faidon Liambotis: [C: 032] Expose some headers to CORS requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/119027 (owner: 10Gilles) [12:00:27] (03PS1) 10Faidon Liambotis: Remove language subdomains for wikidata.org [operations/dns] - 10https://gerrit.wikimedia.org/r/119032 [12:03:08] (03CR) 10Faidon Liambotis: "Also see I084175ffc2545ca68ddd21d1ecf7760f959f6c6b." [operations/dns] - 10https://gerrit.wikimedia.org/r/119032 (owner: 10Faidon Liambotis) [12:03:57] (03CR) 10Faidon Liambotis: "Also see I313d8643ccdd5c3790a87883a4f8ec258f91c103." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/113972 (owner: 10Thiemo Mättig (WMDE)) [12:07:08] (03PS1) 10ArielGlenn: puppetize generation of lists of good dumps for rsyncers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119033 [12:08:24] (03CR) 10jenkins-bot: [V: 04-1] puppetize generation of lists of good dumps for rsyncers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119033 (owner: 10ArielGlenn) [12:11:10] (03PS2) 10ArielGlenn: puppetize generation of lists of good dumps for rsyncers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119033 [12:13:07] (03PS1) 10Faidon Liambotis: autoinstall: add ms-be3004's second NIC to dhcpd [operations/puppet] - 10https://gerrit.wikimedia.org/r/119035 [12:13:29] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: add ms-be3004's second NIC to dhcpd [operations/puppet] - 10https://gerrit.wikimedia.org/r/119035 (owner: 10Faidon Liambotis) [12:14:08] (03CR) 10Faidon Liambotis: [V: 032] autoinstall: add ms-be3004's second NIC to dhcpd [operations/puppet] - 10https://gerrit.wikimedia.org/r/119035 (owner: 10Faidon Liambotis) [12:14:24] (03PS3) 10ArielGlenn: puppetize generation of lists of good dumps for rsyncers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119033 [12:15:58] (03CR) 10ArielGlenn: [C: 032] puppetize generation of lists of good dumps for rsyncers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119033 (owner: 10ArielGlenn) [12:23:29] (03PS1) 10ArielGlenn: more pep8 for script to list last good dumps [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119037 [12:23:56] (03CR) 10ArielGlenn: [C: 032] more pep8 for script to list last good dumps [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/119037 (owner: 10ArielGlenn) [12:25:53] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 16 Mar 2014 12:19:26 PM UTC [12:26:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [13:04:04] (03PS1) 10Alexandros Kosiaris: Convert carbon to RAID5+LVM [operations/puppet] - 10https://gerrit.wikimedia.org/r/119039 [13:06:23] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Mon Mar 17 13:06:16 UTC 2014 [13:11:53] RECOVERY - NTP on carbon is OK: NTP OK: Offset -0.04691720009 secs [13:29:03] PROBLEM - Certificate expiration on virt1000 is CRITICAL: SSL error: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed [13:29:03] RECOVERY - LDAP on virt1000 is OK: TCP OK - 0.000 second response time on port 389 [13:29:13] RECOVERY - LDAPS on virt1000 is OK: TCP OK - 0.000 second response time on port 636 [13:29:22] (03PS1) 10Faidon Liambotis: Revert "Initial commit of mwprof::reporter" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119042 [13:30:59] (03PS2) 10Faidon Liambotis: Revert "Initial commit of mwprof::reporter" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119042 [13:31:05] (03CR) 10Faidon Liambotis: [C: 032] Revert "Initial commit of mwprof::reporter" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119042 (owner: 10Faidon Liambotis) [13:32:53] ori: see ^^^ -- https://gerrit.wikimedia.org/r/119042 [13:34:53] RECOVERY - Puppet freshness on tungsten is OK: puppet ran at Mon Mar 17 13:34:43 UTC 2014 [13:37:51] apergos: labstore1001 has a disk critical alert for /exp/dumps (/exp?! wtf?), is that you or Coren? [13:38:32] (03PS1) 10ArielGlenn: puppetize generation of media directory lists for snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119043 [13:39:29] I guess that's himswitching the job from gluster to nfs without having a notion of the space requirements [13:39:44] okay [13:39:56] I have no idea really, I just saw "dumps" and pinged you :-) [13:40:08] I can turn off that job in about 2 minutes [13:40:09] (03CR) 10Alexandros Kosiaris: [C: 032] Support spatial_ref_sys and comments in postgres [operations/puppet] - 10https://gerrit.wikimedia.org/r/117920 (owner: 10Alexandros Kosiaris) [13:40:15] 3 minutes [13:40:27] (03CR) 10Alexandros Kosiaris: [C: 032] Creating a spatially enabled db on osmlab dbs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119031 (owner: 10Alexandros Kosiaris) [13:40:31] (03PS2) 10ArielGlenn: puppetize generation of media directory lists for snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119043 [13:40:42] no worries, let's just ping him when he wakes up [13:40:48] yep I sure will [13:41:02] (03CR) 10Alexandros Kosiaris: [C: 032] Convert carbon to RAID5+LVM [operations/puppet] - 10https://gerrit.wikimedia.org/r/119039 (owner: 10Alexandros Kosiaris) [13:41:09] paravoid: It's me. /exp isn't actual disks, it's the NFS4 export tree. Lemme go look at this. [13:41:15] oh here he is [13:41:19] ah, you're here [13:41:21] hello :) [13:41:58] !log rebuilding CirrusSearch index for enwiki - its mapping is out of date, making the search results worse. [13:42:01] Logged the message, Master [13:42:08] (03PS3) 10ArielGlenn: puppetize generation of media directory lists for snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119043 [13:42:38] Ah. Fun. 10T wasn't enough for the dumps, it seems. [13:42:46] ah no, not quite [13:42:50] * Coren adds moar space. [13:42:56] I can cut back the number we transfer or something [13:43:13] Hot much is actually needed? [13:43:21] I can tell you in about 5mins [13:43:22] sorry [13:43:23] This is lvs, and I have some room to spare. [13:43:29] lvm* [13:43:56] perhaps a very silly question, but why do we need to copy them rather than mount them off the snapshot hosts? [13:44:44] well it used to be that they wanted a public share in labs for 'stuff' [13:44:53] paravoid: I'm just doing it the way they were done previously with gluster. A priori, I don't think we /need/ to copy them, but I think that was done to insulate labs from them. [13:45:00] so a subset of dumps wound up on that share [13:45:39] any way to limit bandwidth use of folks accessing those files over nfs? [13:46:14] paravoid: Also, that'd mean we either (a) let labs instance mount directly or (b) mount and reexport from labstore, which seems sucky to me. [13:46:57] apergos: Not trivially. NFS doesn't have any support for traffic shaping; we could probably hack something with suitable iptables rules but that seems brittle. [13:47:12] (03PS4) 10ArielGlenn: puppetize generation of media directory lists for snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119043 [13:47:20] why is to hacky to mount & reexport? [13:47:27] sucky, even [13:47:46] paravoid: Well, it doubles network traffic for one. :-) [13:47:48] I mean, sure, it's not great, but copying >10T of data for no reason is worse [13:47:56] probably? [13:48:15] Whereas the copy just costs a rsync of the new stuff. [13:48:33] yes, it's only 10t this first time cause it's a new share that's empty [13:49:21] apergos: I just looked at things a little closer, and now realize that /that/ disk isn't lvm. Crap. (It lives on the non-redundant disks) [13:49:28] oh hm [13:49:40] I can always send over 3 instead of 5 or whatever [13:50:14] That'd work. I should say that 99% of labs' uses of dumps cluster around the latest ones anyways. [13:51:26] (03PS5) 10ArielGlenn: puppetize generation of media directory lists for snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119043 [13:51:30] gahhh rebasaes [13:51:44] apergos: But first, do we want to export/reexport the actual partitions instead? Now's a good time to ask that question. [13:52:19] I don't want a pile of lab users copying at top speed to compromise bandwidth for dumps production [13:52:36] that would be a very bad deal indeed, so I would want limits [13:53:24] (03CR) 10ArielGlenn: [C: 032] puppetize generation of media directory lists for snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/119043 (owner: 10ArielGlenn) [13:57:41] apergos: It's probably safest to keep doing a copy then. [13:59:34] pagecounts [13:59:38] they wanted those too... meh [14:00:46] 3.6T for those and they will only get larger over time [14:02:22] I'll do last 2 good for the umps [14:02:24] dumps [14:03:51] (03PS1) 10Alexandros Kosiaris: install-server::dhcp-server made precise aware [operations/puppet] - 10https://gerrit.wikimedia.org/r/119046 [14:04:52] (03PS1) 10ArielGlenn: adjust number of rsyncers to labs public data from datasets [operations/puppet] - 10https://gerrit.wikimedia.org/r/119047 [14:05:30] (03CR) 10Alexandros Kosiaris: [C: 032] install-server::dhcp-server made precise aware [operations/puppet] - 10https://gerrit.wikimedia.org/r/119046 (owner: 10Alexandros Kosiaris) [14:05:52] (03PS2) 10ArielGlenn: adjust number of rsyncers to labs public data from datasets [operations/puppet] - 10https://gerrit.wikimedia.org/r/119047 [14:05:54] (03CR) 10jenkins-bot: [V: 04-1] adjust number of rsyncers to labs public data from datasets [operations/puppet] - 10https://gerrit.wikimedia.org/r/119047 (owner: 10ArielGlenn) [14:06:38] riiiigght [14:06:47] bogus [14:07:32] (03PS3) 10ArielGlenn: adjust number of rsyncers to labs public data from datasets [operations/puppet] - 10https://gerrit.wikimedia.org/r/119047 [14:08:43] apergos: Do you need me to rm some stuff on the destination? [14:09:06] yeah if you don't mind tossing enwiki 4 and 5 [14:09:08] oldest [14:09:11] kk [14:09:12] that would be awesome [14:09:25] (03CR) 10ArielGlenn: [C: 032] adjust number of rsyncers to labs public data from datasets [operations/puppet] - 10https://gerrit.wikimedia.org/r/119047 (owner: 10ArielGlenn) [14:10:22] (03PS1) 10Alexandros Kosiaris: Fix dependency in install-server::dhcp-server [operations/puppet] - 10https://gerrit.wikimedia.org/r/119049 [14:10:33] I got six in there actually. [14:11:12] Oct-Mar inclusive [14:11:23] toss oldest 3? [14:11:47] (03CR) 10Alexandros Kosiaris: [C: 032] Fix dependency in install-server::dhcp-server [operations/puppet] - 10https://gerrit.wikimedia.org/r/119049 (owner: 10Alexandros Kosiaris) [14:12:30] yes [14:12:49] that wil give plenty o room for rsync to finish its job on the next run [14:13:13] * Coren exterminates. [14:13:13] RECOVERY - Disk space on labstore1001 is OK: DISK OK [14:13:23] sweet [14:13:51] That just freed 2.3T [14:14:45] any progress with labstore1001's puppet btw? [14:14:46] :) [14:14:49] enwiki is crazy big. [14:14:51] (03PS1) 10Alexandros Kosiaris: Fix typo introduced in d7d97a0 [operations/puppet] - 10https://gerrit.wikimedia.org/r/119050 [14:15:06] yep it's huge [14:15:26] paravoid: I'll be able to reenable it in about 6h [14:15:30] awesome [14:16:26] (03CR) 10Alexandros Kosiaris: [C: 032] Fix typo introduced in d7d97a0 [operations/puppet] - 10https://gerrit.wikimedia.org/r/119050 (owner: 10Alexandros Kosiaris) [14:24:19] !log disabling puppet on dataset2 for shuffling data sources around [14:24:23] Logged the message, Master [14:30:34] (03PS1) 10Alexandros Kosiaris: install-server module. Precise has squid3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/119052 [14:32:32] (03CR) 10Alexandros Kosiaris: [C: 032] install-server module. Precise has squid3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/119052 (owner: 10Alexandros Kosiaris) [14:39:02] (03PS1) 10coren: Labs: remove mode from mountpoints files {} [operations/puppet] - 10https://gerrit.wikimedia.org/r/119053 [14:40:29] (03PS1) 10Alexandros Kosiaris: role::installserver pin packages varied on distro [operations/puppet] - 10https://gerrit.wikimedia.org/r/119054 [14:40:37] (03CR) 10Hashar: [C: 031] Labs: remove mode from mountpoints files {} [operations/puppet] - 10https://gerrit.wikimedia.org/r/119053 (owner: 10coren) [14:43:04] (03CR) 10Alexandros Kosiaris: [C: 032] role::installserver pin packages varied on distro [operations/puppet] - 10https://gerrit.wikimedia.org/r/119054 (owner: 10Alexandros Kosiaris) [14:43:25] (03PS2) 10coren: Labs: remove mode and owner from mountpoints dirs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119053 [14:44:51] greg-g: I can take the SWAT deploy today if no one else is jumping up and down [14:45:13] (03CR) 10coren: [C: 032] "Less is more?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119053 (owner: 10coren) [14:48:49] (03CR) 10BryanDavis: "Sam's hack of removing the $BINDIR specification works because there is another mwversionsinuse in the search path:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118946 (owner: 10Reedy) [14:50:01] manybubbles: I was about to ping you and MaxSem to see which of us wanted to do it. If you want to do it, that's fine with me as I have no burning desire to. The one patch on the list today LGTM. [14:50:12] (03CR) 10Reedy: "Fixing it properly certainly sounds more sensible ;)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118946 (owner: 10Reedy) [14:50:22] sweet [14:50:59] ebernhardson: looks like we're ready to do the SWAT deploy in ten minutes for your change [14:52:37] is someone fiddling with gerrit? [14:54:12] manybubbles: that'd be https://gerrit.wikimedia.org/r/#/c/118741/, right? [14:54:43] I'd +1 it, but wifi/gerrit is slow/unresponsive [14:54:59] * aude can't access gerrit [14:55:15] hey anomie, go ahead. I'll push a few mobile conf changes later [14:55:44] mlitn: yeah, that is the one. [14:55:52] grrit! [14:55:55] why you slow [14:56:51] might be simplest if extension maintainer +2 it into the release branch of the extension then I build the submodule update for production and +2 that myself [14:57:13] PROBLEM - DPKG on analytics1004 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [14:57:15] so, that means you (or someone else in flow) +2 it [14:57:18] oh, sure, I'll +2 instead ;) [14:57:42] thanks! [14:59:27] +2'ed [15:00:01] thanks [15:00:26] my submodule update one core is taking a while so I won't be able to deploy it for a few minutes. because, submodules [15:00:31] ^d: ^^ :( [15:01:25] don't worry, I wasn't planning on leaving in the next minute ;) [15:01:52] <^d> Hmm? [15:02:03] geeeerit! [15:02:25] <^d> meh. [15:03:40] manybubbles: yessir, that's mostly your call :) [15:03:44] I should really kick off the submodule update 20 minutes before hand. that way it is fast to get just what I need [15:04:44] greg-g: it's great to have a deployment time slot this early in the day :) [15:04:58] (have nothing for today) [15:05:15] aude: you guys were part of the reasoning :) [15:05:31] great [15:05:33] holy crap, gerrit is dead in the water for me [15:07:05] <^d> wfm. [15:07:22] (03PS1) 10Chad: Revoke my gerrit admin privs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119060 [15:07:24] <^d> See ^ [15:08:42] * bd808 looks around to see who he'll bug about gerrit if that patch is merged [15:08:52] wait, what? [15:08:53] hah [15:08:54] yep, works for me now [15:09:27] anomie, are you deploying? [15:09:33] MaxSem: manybubbles is [15:09:40] ah [15:09:53] manybubbles, if you're not ready yey, I'd like to push https://gerrit.wikimedia.org/r/#/c/114634/ [15:10:23] MaxSem: sounds good to me [15:10:31] you have the conch [15:10:35] paravoid: don't worry, ^d is known for making such patchsets [15:10:49] ^d: and don't think it'll work: we'll still bug you even if you didn't have admin rights [15:11:06] <^d> Yeah, but it's easier for my to wash my hands and say "Sorry, can't help!" [15:11:18] <^d> "I totally would, if I had access!" [15:11:23] (03CR) 10MaxSem: [C: 032] Don't collapse sections on Wiktionaries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114634 (owner: 10MaxSem) [15:11:38] ^d: then you'll just be a liar! [15:12:35] (03Merged) 10jenkins-bot: Don't collapse sections on Wiktionaries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114634 (owner: 10MaxSem) [15:12:59] (03CR) 10Chad: [C: 031] "Less users with sudo is good for security. Please merge immediately." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119060 (owner: 10Chad) [15:13:10] MaxSem: I'm ready to +2 the wmf18 core change to get flow deployed. let me know when you are done with your config change. [15:13:45] (03Abandoned) 10Chad: Revoke my gerrit admin privs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119060 (owner: 10Chad) [15:13:48] !log maxsem synchronized wmf-config/ 'https://gerrit.wikimedia.org/r/#/c/114634/' [15:13:52] Logged the message, Master [15:13:59] manybubbles, ^^ [15:14:10] MaxSem: thanks! [15:14:22] * manybubbles has the conch [15:15:19] ^d: "fewer" [15:15:59] <^d> My English teachers never managed to drill that one into me. [15:17:13] PROBLEM - jmxtrans on analytics1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args -jar jmxtrans-all.jar [15:18:49] (03CR) 10BryanDavis: "We really do need to get broader coverage for gerrit support. Chad and Antoine get pinged to look into slowness in the gerrit/zuul/jenkins" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119060 (owner: 10Chad) [15:19:37] ^d: are you that pissed off by Gerrit ? [15:19:50] mlitn: ready to sync [15:20:05] <^d> hashar: Is it a day that ends in Y? [15:20:15] not in my language :D [15:20:35] you can surely brain dump your Gerrit knowledge and spread it to a bunch of folks in mwcore team [15:20:39] ping me and I'll mash the enter key and it'll go [15:20:40] I will be happy to help [15:21:04] ^d: I've only recently-ish gotten that one straight (fewer/less) [15:21:23] manybubbles: okidoki [15:22:14] <^d> hashar: I like the people on my team--why would I want to make them suffer with Gerrit? [15:22:32] !log manybubbles synchronized php-1.23wmf18/extensions/Flow/ 'SWAT Backport for fatal on Special:Contributions' [15:22:44] ^d: to share the pain? :-] [15:22:51] mlitn: all done [15:22:55] looks better [15:22:57] ^d: this way you will get more "free" time to work on a Gerrit replacement [15:23:03] manybubbles: great thanks [15:23:08] problem seems fixed :) [15:23:17] * manybubbles puts conch back into center of circle [15:27:13] RECOVERY - jmxtrans on analytics1021 is OK: PROCS OK: 1 process with command name java, args -jar jmxtrans-all.jar [15:27:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [15:33:40] manybubbles: doh, i thought that was 4pm for some reason [15:33:59] no big deal [15:34:22] mlitn had you covered [15:34:25] excellent :) [16:03:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:05:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:07:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:09:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:11:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:13:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:15:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:17:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:19:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:21:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:23:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:25:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:27:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 03:58:10 PM UTC [16:28:02] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Mon Mar 17 16:28:01 UTC 2014 [16:29:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 04:28:01 PM UTC [16:31:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 04:28:01 PM UTC [16:36:09] bd808: Sort of! [16:41:34] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 01:40:58 PM UTC [16:45:51] hi rachel, had fun? :) [16:48:14] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [16:58:34] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Mon Mar 17 16:58:26 UTC 2014 [17:02:14] RECOVERY - Puppet freshness on labsdb1004 is OK: puppet ran at Mon Mar 17 17:02:07 UTC 2014 [17:20:34] PROBLEM - Puppet freshness on dataset2 is CRITICAL: Last successful Puppet run was Mon 17 Mar 2014 02:19:32 PM UTC [17:21:04] PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 57.466667 [17:31:38] springle: hi [17:32:02] probably away, I'll mail [17:32:24] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2791.93335 [17:33:04] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Mon Mar 17 17:32:55 UTC 2014 [17:35:04] RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:35:22] akosiaris, since you are on rt duty... https://rt.wikimedia.org/Ticket/Display.html?id=6961 [17:35:24] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:36:10] gwicke: ok, I look into it [17:41:10] !log re-enabled dataset2 puppet, move done [17:41:18] a little after the fact, oh well [17:43:47] where is grrrit-wm ? [17:44:06] <^d> We got rid of it. Decided we didn't like bots. [17:45:00] great ^d :) [17:45:46] which channel is it this time?:) [17:47:13] #wikimedia-operations-bots-only-watch-out-for-spam [17:47:58] apergos: "move done" ? wot? [17:48:02] well [17:48:06] not exacly; [17:48:14] so ther eis the hostname (download.wm.o) [17:48:14] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [17:48:20] and there is a user (eric z) [17:48:55] which I have turned off his rsyncs but I'mreluctant to just edit scripts of his on stat1002, maybe I'd miss some, so I might have to turn those backon til I can get a hole of him [17:48:56] is the hostname moving to the new host for downloads [17:49:02] yes [17:49:04] where the mw-release people have access [17:49:06] no [17:49:13] :-D [17:49:16] heh [17:49:18] releases.wm.o is that host [17:49:27] downlaods willmove to eqiad is all [17:49:29] ok [17:49:49] but all the cron jobs are in and moved and puppeted up, there's one where I need to do a bunch of back runs on if before I can enable it [17:50:26] cool! [17:50:29] anyways... right nowif there wre an easy way to reach erik z in real time and say 'hey can you point your stuff to dataset1001', I would love it [17:50:52] put a banner on stats.wm.org :) [17:50:59] hahaha noooooooo [17:51:16] motd [17:51:19] wall [17:51:59] and if he doesn't log in for 3 days then what? [17:52:39] check contacts page on office for phone number and text him :p [17:54:14] RECOVERY - DPKG on analytics1004 is OK: All packages OK [17:54:59] milimetric: ping [17:55:17] hi paravoid, what's up [17:55:19] hi [17:55:37] are you on verizon right now? [17:55:51] mark: sent you a mail [17:56:04] yes paravoid, on verizon [17:56:12] apergos: I'm told Skype can work [17:56:19] can you give me your IP (in private if you prefer)? [17:56:32] a traceroute to e.g. bast1001 wouldn't hurt, but I'm more interested in your IP [17:56:50] 108.16.110.132 paravoid, tracerouting now [17:58:12] heh, sorry, no traceroute installed - one sec [17:58:28] and do you have ipv6? [17:58:37] no, I don't [17:58:51] ok, thanks [18:28:34] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [18:57:42] jgage: around ? [18:58:42] hi [18:59:27] hi, regarding RT 7050, i have something i have been using for some time [18:59:49] do you have a few sec's to glace around it ? [19:01:11] re:RT 7050: ip link show [19:01:31] if NO-CARRIER then bad [19:02:13] * jgage looks [19:02:30] mutante: for that you need to be inside the device or poll using snmp [19:02:57] what's the alternative, monitor what the router has negotiated for that port? [19:03:25] i think a normal snmp query will be enough [19:03:50] iirc query 1.3.6.1.2.1.2.2.1.14 [19:04:05] for IfInErrors [19:04:21] * matanya is googling to see if he remembers the oid correctly [19:04:40] disregard the command i said, but we can execute whatever we want locally and then send out to NSCA (as alternative to snmptrap) [19:04:49] nrpe should be fine [19:05:18] running this under netmon1001 (just an e.g.) using nrpe will do [19:06:03] we don't have snmpd running on our servers nor do we intend to [19:06:28] paravoid: we do have snmptrapd [19:06:45] not on every box [19:06:46] then just nrpe and if for some reason you want passive then nsca [19:07:32] so how do you plan to query paravoid ? [19:07:33] looks like we can get the value directly from /sys, though hopefully there's a nicer path: /sys/devices/pci0000:00/0000:00:11.0/0000:02:01.0/net/eth0/speed [19:07:45] /sys/class/net/eth0? [19:08:02] but you need /duplex too [19:08:16] ah yeah, perfect [19:08:30] and you need to only do the check for all nics where /carrier == 1, as we have tons of boxes with unplugged cards [19:09:09] are supported speeds anywhere under sysfs, I wonder [19:09:23] checking for 100mbps is easy, but you should also check if e.g a 10g card has negotiated at 1gbps [19:09:32] that shouldn't happen as often [19:09:49] but if we're going to have a check... [19:10:41] /sys/class/net/eth*/statistics/ has tons of metrics that we should check too, but these are probably better being check_graphited on or ganglios or whatever [19:14:48] ethtool will show supported link modes as well as current speed [19:15:59] yes [19:16:25] check exchange.nagios.org for existing ones? some are crap but some are not:) [19:16:32] looks like it's not getting that info from proc/sys though [19:16:44] netlink, most probably [19:16:59] yeah [19:17:00] socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 [19:17:01] fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 [19:22:14] wo ist morebots? [19:22:29] mehrbots [19:23:13] mehrroboter :p [19:23:18] weiss ich auch nicht [19:23:29] er ist gestorben [19:23:40] Coren/andrewbogott: any idea on why we're getting Wikitech spam at our noc@ alias? [19:23:49] Subject: Wikitech page Nova Resource:I-0000017b.eqiad.wmflabs has been changed by anonymous user 127.0.0.1 [19:23:53] etc. [19:24:13] ori: grrrit-wm is also gone. have all bots been moved out of channel? [19:24:25] Coren is moving tools to eqiad [19:24:25] morebots is schlimm, aber nicht schlimmer als meiner deutsche [19:24:28] paravoid: probably wikitech is configured to do that, although I don't know where... [19:24:40] it only happened recently, I'm guessing eqiad [19:25:04] Hm... [19:25:14] ori: labs migration related? [19:25:35] well, that's a notice of an instance being created. But as far as I know that's configured to only update at virt0 [19:25:36] wahrscheinlich [19:25:37] which is in tampa [19:26:04] paravoid: Someone set the labslogbot wikitech account's email to noc@, and echo got turned on. [19:26:17] oh heh [19:26:38] That someone was probably me, but if I did it it was ages ago [19:26:38] ori: bestimmt. Frag mal den Andreas Bogott [19:26:55] please fix :) [19:27:18]