[00:09:24] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:12] New patchset: Bhartshorne; "changed health check URL for swift's lvs to something small that I know exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2159 [00:20:33] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2159 [00:20:34] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2159 [00:31:35] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:49] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.078 seconds [00:42:59] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 260 MB (3% inode=60%): /var/lib/ureadahead/debugfs 260 MB (3% inode=60%): [00:51:51] !log applied articlefeedback v5 schema changes to enwiki, testwiki, en_labswikimedia [00:51:53] Logged the message, Master [00:54:10] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [00:54:10] New patchset: Bhartshorne; "adding the IP for ms-fe to the lvs node definition so it will listen on that IP" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2160 [00:54:41] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2160 [00:54:42] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2160 [01:05:19] RECOVERY - Disk space on srv220 is OK: DISK OK [01:05:19] RECOVERY - Disk space on srv221 is OK: DISK OK [01:07:29] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 0.024 seconds [01:15:30] maplebed: ip addr del 10.2.1.13/32 dev lo [01:17:02] Ryan_Lane: I updated http://wikitech.wikimedia.org/view/LVS#Diagnosing_problems and added the section "Incorrectly bound interfaces" [01:24:11] hi gues [01:24:14] guys* [01:24:18] so, wanna push a new branch [01:24:20] how do I do that? [01:24:25] to gerrit? [01:24:27] yeah [01:24:34] Ryan_Lane: ^^^^ [01:24:34] :D [01:25:02] push a branch to which repo? [01:25:31] ssh://gerrit.wikimedia.org:29418/analytics/reportcard.git [01:25:34] ah [01:25:38] just want to work on something other than master [01:25:41] so I don't mess things up more [01:25:49] well, it's gated trunk [01:26:00] so, you can push into a change, and before it goes it, someone can code review it [01:26:08] *goes in [01:26:13] right, but what's the push-for-review for a new branch? [01:26:26] does it need to be a branch on the server? [01:26:42] um, guess not, although I'd like my work to be saved elsewhere than just my lcoal [01:26:52] I'm pretty sure our version of gerrit requires you to do this through the interface [01:26:58] create the branch? [01:27:01] remotely? [01:27:05] Ryan_Lane: I've pushed a branch remotely before. [01:27:10] maplebed: to the server? [01:27:11] I don't remember how I did it though... [01:27:16] :P [01:27:19] as a new branch? [01:27:25] last time I tried it denied me [01:27:33] I bet roan knows how to do this [01:27:41] we should have it set so that branches can be pushed without review [01:27:43] it may be in the documentation too [01:27:46] but merges or commits to master need reviewed [01:27:53] possible? [01:28:23] yeah, not sure if this works [01:28:25] but I tried this: [01:28:26] git push ssh://gerrit.wikimedia.org:29418/analytics/reportcard.git HEAD:refs/for/otto/pipeline [01:28:33] got To ssh://gerrit.wikimedia.org:29418/analytics/reportcard.git [01:28:33] ! [remote rejected] HEAD -> refs/for/otto/pipeline (branch otto/pipeline not found) [01:28:33] error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/analytics/reportcard.git' [01:28:36] +1 ottomata - that would make it *way* easier to work with other folks with branches. [01:28:36] but i have no idea what I am doing [01:28:46] I don't know either [01:28:49] yeah, kinda the point of git is to be able to create branches willy nilly [01:28:56] well, yeah [01:29:08] you can do so through the gerrit interface, for sure [01:29:13] let's see... [01:29:21] Admin->Projects [01:29:29] choose your project [01:29:41] i see Branches listed [01:30:08] under branches, there should be an interface for creating a branch [01:31:44] hi otto [01:34:39] RECOVERY - Disk space on ms-fe1 is OK: DISK OK [01:38:56] New patchset: Ottomata; "moving pipeline files into pipeline/ directory. This should be a standalone python pacakge. Launching and parallelizing individual pipelines might go in here later" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2161 [01:38:58] New patchset: Ottomata; "Rearranging test and file structures, organizing things a bit" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2162 [01:39:21] coooool, got it [01:39:25] didn't have to use gerrit (I think) [01:39:27] used git-review [01:39:36] oh hi diederik :) [01:39:38] thanks! [01:39:59] whaa, topic otto/pipeline [01:40:00] hmm [01:40:10] that's your branch :) [01:40:24] from gerrit [01:40:26] Change-Id: [01:40:26] I4399e6bbd8b54c49f5b4faf0f1793ac8df6640d7 [01:40:27] Owner Ottomata [01:40:27] Project analytics/reportcard [01:40:27] Branch master [01:40:27] Topic otto/pipeline [01:40:34] Branch master [01:40:34] ? [01:40:41] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2161 [01:40:42] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2161 [01:41:24] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2162 [01:41:25] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2162 [01:43:18] New patchset: Ottomata; "Adding reorganized files. I should have added these with the last commit." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2163 [01:44:07] oook, gotta run soon, thanks for the help boys [01:44:44] laterz [01:45:39] agghg [01:45:42] it totally merged to master when you approved [01:45:43] aggghhhh [01:45:47] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2163 [01:45:47] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2163 [01:46:01] NICE! [01:46:03] no! [01:46:06] :( [01:46:06] boooo [01:46:10] [01:46:14] ah ok, phew [01:46:14] haha [01:46:23] that's stupid actually [01:46:25] yeah, wanted them to be isolated from master for a bit [01:46:26] yeah [01:46:28] not sure why it does that [01:46:31] me neither [01:46:42] maybe ryan/roan know more about this [01:46:42] this is my .gitreview file [01:46:42] [gerrit] [01:46:43] host=gerrit.wikimedia.org [01:46:43] port=29418 [01:46:43] project=analytics/reportcard.git [01:46:43] defaultbranch=master [01:46:49] maybe defaultbranch [01:46:55] oh, maybe I have to tell git review the branch [01:47:00] lemme try real quick, i have a dumb commit to push [01:47:59] git review --no-rebase otto/pipeline [01:47:59] Had trouble running git log --color=auto --decorate --oneline otto/pipeline --not remotes/gerrit/otto/pipeline [01:47:59] fatal: ambiguous argument 'remotes/gerrit/otto/pipeline': unknown revision or path not in the working tree. [01:47:59] Use '--' to separate paths from revisions [01:48:00] (env)[~/Projects/wm/analytics/reportcard] ( [01:48:03] grr, i dunno [01:48:26] will ask ryan about it on thursday I guess [01:48:27] ok, i'm out [01:48:28] laaaters! [01:48:56] laterz@ [02:07:22] PROBLEM - Disk space on ms-fe1 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:10:42] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1068s [02:23:02] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:42] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 2028s [02:38:02] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 8s [02:44:22] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 30s [02:49:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:59:02] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 144 MB (2% inode=60%): /var/lib/ureadahead/debugfs 144 MB (2% inode=60%): [02:59:12] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [02:59:13] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 199 MB (2% inode=60%): /var/lib/ureadahead/debugfs 199 MB (2% inode=60%): [02:59:43] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [03:00:32] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.717 seconds [03:09:13] New patchset: Diederik; "Initial commit, feedback Catrope incorporated, feedback Tim Starling (3x) incorporated" [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2142 [03:10:12] RECOVERY - Disk space on srv224 is OK: DISK OK [03:10:22] RECOVERY - Disk space on srv220 is OK: DISK OK [03:10:22] RECOVERY - Disk space on srv222 is OK: DISK OK [03:11:12] RECOVERY - Disk space on srv219 is OK: DISK OK [03:14:15] New review: Tim Starling; "(no comment)" [analytics/udp-filters] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2142 [03:18:42] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.261 seconds [03:56:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:32] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 140 MB (1% inode=60%): /var/lib/ureadahead/debugfs 140 MB (1% inode=60%): [04:18:42] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.211 seconds [04:20:02] RECOVERY - Disk space on es1004 is OK: DISK OK [04:22:22] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:28:42] RECOVERY - Disk space on srv223 is OK: DISK OK [04:44:12] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [04:44:13] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [04:48:02] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [04:51:02] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 258 MB (3% inode=60%): /var/lib/ureadahead/debugfs 258 MB (3% inode=60%): [05:02:12] RECOVERY - Disk space on srv221 is OK: DISK OK [06:42:31] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [07:10:21] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 198 MB (2% inode=60%): /var/lib/ureadahead/debugfs 198 MB (2% inode=60%): [07:21:41] RECOVERY - Disk space on srv223 is OK: DISK OK [07:32:51] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 91 MB (1% inode=60%): /var/lib/ureadahead/debugfs 91 MB (1% inode=60%): [07:44:01] RECOVERY - Disk space on srv220 is OK: DISK OK [08:02:07] New patchset: Dzahn; "add https monitoring for puppetmaster on stafford (RT 2361)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [08:18:29] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2164 [08:18:29] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [08:29:01] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 196 MB (2% inode=60%): /var/lib/ureadahead/debugfs 196 MB (2% inode=60%): [08:47:28] PROBLEM - Puppet freshness on mw8 is CRITICAL: Puppet has not run in the last 10 hours [08:52:58] RECOVERY - Disk space on srv219 is OK: DISK OK [08:56:26] New patchset: Dzahn; "adding the puppetmaster http(s) monitoring to sockpuppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2165 [08:56:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2165 [08:57:23] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2165 [08:57:23] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2165 [09:27:40] New review: Dzahn; "did somebody else work on this meanwhile? at least re: the duplicate checks it looks a lot better th..." [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1726 [09:56:11] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 404752 MB (3% inode=99%): [09:57:01] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 401042 MB (3% inode=99%): [10:13:41] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [10:21:01] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [11:20:51] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:21:00] I got user report that subscribing to mediawiki-announce at lists.wikimedia.org times out [12:27:32] New review: Mark Bergsma; "Daniel, can you please move this out of site.pp and put it where it belongs, i.e. puppetmaster.pp?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [12:38:54] New review: Dzahn; "sure, will do. makes more sense to put in service class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [12:42:11] PROBLEM - Host db1018 is DOWN: PING CRITICAL - Packet loss = 100% [12:44:31] New patchset: Dzahn; "move puppetmaster monitoring out of site.pp into puppetmaster.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2166 [12:47:08] New patchset: Dzahn; "move puppetmaster monitoring out of site.pp into puppetmaster.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2166 [12:47:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2166 [12:55:11] RECOVERY - Host ms-fe.pmtpa.wmnet is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [14:01:41] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:01] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:51] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [14:54:51] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [14:58:41] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.359 seconds [15:02:37] New patchset: Mark Bergsma; "Simplify lvs_realserver_ips configuration in role/cache.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2167 [15:02:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2167 [15:03:12] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2167 [15:03:12] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2167 [15:07:01] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when using parentheses" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:07:17] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2168 [15:08:14] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when using parentheses" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:08:31] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2168 [15:11:59] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when using parentheses" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:13:37] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when not using quotes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:13:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2168 [15:14:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2168 [15:14:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:19:18] New patchset: Mark Bergsma; "Retab squid.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2169 [15:19:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2169 [15:21:42] New patchset: Mark Bergsma; "Fix modes, simplify file definitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2170 [15:21:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2170 [15:23:41] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2169 [15:23:41] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2169 [15:23:59] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2170 [15:23:59] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2170 [15:32:51] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:52] New patchset: Mark Bergsma; "Remove usage of old wikimedia-task-squid package, do everything in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2171 [15:49:08] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2171 [15:49:39] New patchset: Mark Bergsma; "Remove usage of old wikimedia-task-squid package, do everything in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2171 [15:49:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2171 [15:50:29] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2171 [15:50:30] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2171 [15:52:26] New patchset: Mark Bergsma; "xfsprogs is installed on all servers, in base.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2172 [15:52:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2172 [15:52:38] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2172 [15:52:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2172 [15:55:21] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.097 seconds [15:55:44] New patchset: Mark Bergsma; "Fix mode on /aufs dir" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2173 [15:56:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2173 [15:56:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2173 [15:56:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2173 [16:07:36] !log Removed now obsolete package wikimedia-task-squid from the karmic-wikimedia and lucid-wikimedia APT repositories, and deleted in svn.wikimedia.org [16:07:38] Logged the message, Master [16:18:11] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 0.019 seconds [16:22:09] !log dataset1001 down for controller replacement [16:22:10] Logged the message, RobH [16:23:15] yayayayay [16:36:12] robh: db47...disk are hot swap? i am going to reseat disk3 [16:37:35] drive 2 (0,1,2) [16:38:54] cmjohnson1: they are but someone needs to ensure that the raid array only has that single missing drive. i think i dispatched it to you [16:38:58] so lemme take a quick peek [16:40:56] cmjohnson1: Ok, so i can confirm its just missing that one disk [16:41:03] and that i can see when you swapped it last [16:41:11] i dont think reseating it is going to fix it, but lets go ahead and try [16:41:24] yes..back in sept [16:41:42] !log reseating drive2 in db47 [16:41:43] Logged the message, Master [16:45:25] cmjohnson1: let me know when it reseats and goes green [16:46:00] robh: it didn't work....it has never gone green after replacement. [16:46:07] yea, its bad [16:46:14] so if that is from dell, reopen the old case [16:46:19] it is [16:47:44] New patchset: Hashar; "web holder for WiktionaryMobile nightly builds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2174 [16:48:01] !log dataset1001 controller replaced [16:48:02] Logged the message, RobH [16:48:03] New patchset: Mark Bergsma; "Add removal support for system roles" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2175 [16:48:04] apergos: ^ [16:48:11] yay! [16:48:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2174 [16:48:24] its not installed, just in the raid bios, feel free to reboot it or whatever [16:48:27] tomorrow will be a dataset1001-derful day :-) [16:48:31] i didnt setup raid, i assumed you would like to =] [16:48:35] yes, I will take it from there [16:48:37] thank you! [16:49:48] quite welcome =] [16:50:12] apergos: so you now, months and months and months...(years) have enough hardware in working order to do dumps without spof [16:50:14] hows it feel? [16:50:20] =] [16:50:37] (except that entire push a site change break the current run thing ;) [16:50:43] don't ask me til I acutally have it [16:51:04] really when I will reast easy for more than 2 seconds is as soon as the first full rsync completes [16:51:11] then I'll be able to sleep nights [16:51:29] (yeah, years. it's actually been frigging *years*.) [16:51:31] oh so. [16:51:32] so one will present the dumps to the public? [16:51:37] speaking of things we want to kill [16:51:40] and the other will do active storage for the snapshots? [16:51:47] yes, for now [16:51:58] apergos: if we never mention the server that shall not be named, maybe we can all forget it exists. [16:52:13] can we force those guys to take back ds1 and either give us a new box or give us a new box with similar hardware that sucks lots less? [16:52:21] nope, its out of warranty [16:52:25] cause ct will back you if you go to the vendor on it [16:52:26] the only open case we have is it [16:52:30] yeah but they didn't close the ticket [16:52:35] i dont want anything else from them though [16:52:37] cause it's still broken [16:52:44] and they wont credit it [16:52:48] and we dont want anything they sell [16:52:55] well I'm thinking if we can get them to give us something that's truly a jbod and a head we can do something else with [16:53:02] rather than this broken *&^%$# [16:53:06] the disk shelf is already that [16:53:14] I have said this before [16:53:21] I know you have mark [16:53:21] just hook up the drives of dataset1 to some other server [16:53:23] use it as a jbod [16:53:31] especially the existing jbod part of ds1 :P [16:53:32] i have never disagreed with this. [16:53:47] the only loss of dataset1 chassis are the disks, which we can pull and use elsewhere. [16:53:51] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [16:53:53] yeah [16:53:55] please do it [16:53:57] how many of those are there? [16:54:01] 24 [16:54:04] ah [16:54:11] i think. [16:54:15] ok well it's a bummer and it's also too dang bad [16:54:19] i really dont recall exactly, i hate that goddamned server [16:54:24] but its 4 U [16:54:29] which is 24 in my mind. [16:54:41] no internal disks, front disks only [16:54:50] I was gonna say, once dataset1001... but you know what, we've dicked around all this time and not had dataset1001 [16:54:56] so ops in agreement to kill dataset1 [16:54:57] so I'm officially declairnng and end to it [16:55:02] stabba stabba stabba [16:55:02] huzzah [16:55:03] die! [16:55:18] its down now right? [16:55:26] yea memory issue or some shit... [16:55:29] we need to ask chris to send email to the vendor saying "well it's been real", close the ticket [16:55:33] that's it and that's all [16:55:41] i will update the ticket for it later today [16:55:54] then figure out where you guys can best use some storage [16:56:24] well, any server we want to add it to will need the purchase of an h800 controller if it is a dell [16:56:26] it may be "up" but it can certainly be shut down [16:56:31] right [16:57:05] or we use the controller in dataset1, which i hate. [16:57:07] so lets not. [16:57:14] its a one off odd chipset using shitty software [16:58:09] let's not. that's right [16:59:29] die die die [16:59:34] :-D :-D [16:59:38] * apergos does a happy dance [16:59:54] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/2166 [17:01:40] what size drives are they? [17:01:48] we can perhaps use them in the thumpers or so [17:02:23] I don't remember any more [17:04:18] !log cp1019 console redirection fixed per rt2353, ready for OS install [17:04:19] Logged the message, RobH [17:08:58] !log investigating errors on cp1014 [17:09:00] Logged the message, RobH [17:22:00] Could someone run sync-apache then apache-graceful-all for me please? I've added be.wikimedia.org to /h/w/c/httpd/wikimedia.org [17:33:51] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:37:51] !log cp1014 memory was improperly installed (from factory?), installed in supported configuration and system is now ready for OS install per RT2351 [17:37:53] Logged the message, RobH [17:45:01] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [17:45:17] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/2175 [17:47:24] Jeff_Green: I guess you want me to handle the renaming dns stuff? [17:47:32] on rt 2291? [17:50:46] oh sorry--I stalled on that because it looks like we're going to move them into the new rack and all anyway [17:51:00] they won't even be on the main prod network [17:51:26] does that make sense? [17:52:05] well, the mgmt and such still changes [17:52:20] if they are goign to eventually be on their own mgmt we can do that then, but if its being renamed now it shouldnt remain half done [17:52:30] they'll be on their own management too [17:52:46] right... so then would you like them to be in the racks unaccessible in the mean time? [17:52:54] cuz i am going to remove the old dns entries for the mgmt names [17:52:58] yeah that's fine [17:53:01] so i can rename them so you can access them in the meantime [17:53:08] or i can leave them unaccessible, which i dont like doing [17:53:24] as far as I'm concerned we can pull them from the rack and rip out any dns etc config for now [17:53:24] cuz then if you wanted to check the hardware spec, you have to call me [17:53:45] well, i rather leave them in rack and accessible to some kind of lights out management [17:53:58] leaving them on the floor isnt allowed [17:54:07] ok--up to you. i don't forsee doing anything with them until they're in the new rack [17:54:08] and carring them in and out of storage is a waste of energy [17:54:10] ok. [17:54:20] !log updated dns for payments boxen renames in eqiad [17:54:21] Logged the message, RobH [17:56:31] RobH: thank you for looking at those CP boxes! [17:57:40] notpeter: welcome, they arent loaded right? [17:57:50] cuz i see some memory errors on some dracs i wanna investigate [17:57:58] not in rotation i mean [17:58:11] RobH: they won't be doing anything until mark gives me the green light to spin them up fully [17:58:17] so, go nuts [17:58:23] cool [17:58:25] and I'll check in with you before I start the next step [18:01:25] notpeter: cp1019 has memory error so leave it for now [18:01:29] cuz i need to bring it down [18:02:21] kk [18:10:11] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:32] !log cp1017 memory error repaired [18:14:34] Logged the message, RobH [18:15:15] !log cp1019 memory error repaired, now it is ready for OS install [18:15:16] Logged the message, RobH [18:15:23] notpeter: ok, its all set now, thx for holding off [18:15:31] dimm was not quite seated i suppose, reseating repaired it [18:16:07] sweet [18:22:56] robh: when you get a chance check db47 drive2...i think i figured it out [18:23:04] ? [18:23:05] * cmjohnson1 operator error [18:23:16] was in wrong mounting hole on bracket? [18:23:20] thus not seating properly [18:23:21] ? [18:23:22] yep [18:23:32] ok, did it turn out to need to go deeper? [18:23:38] just occurred to me [18:23:38] cuz the alternative usually is broken backplanes [18:23:59] if you mix it up putting it in the forward ones it shouldnt harm anything, just not work [18:24:02] right...looking at all these drives I am swapping to wipe and the light bulb went on [18:24:17]