[00:17:40] PROBLEM - mysqld processes on db1044 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:33:40] RECOVERY - mysqld processes on db1044 is OK: PROCS OK: 1 process with command name mysqld [01:39:09] PROBLEM - Disk space on elastic1008 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 11168 MB (3% inode=99%): [01:44:25] Oh no! We've finally run out of space! [02:01:53] !log LocalisationUpdate completed (1.23wmf10) at 2014-01-26 02:01:52+00:00 [02:02:40] !log LocalisationUpdate completed (1.23wmf11) at 2014-01-26 02:02:40+00:00 [02:09:05] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-01-26 02:09:05+00:00 [02:11:19] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:49] PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:39] RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.071 second response time [03:02:09] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.253 second response time [03:11:19] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [03:12:09] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.43 ms [03:54:09] huh re elastic1008 diskspace [03:57:54] (03PS1) 10Hoo man: Disable global AbuseFilter for private/fishbowl wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 [04:49:09] PROBLEM - Disk space on elastic1008 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 11253 MB (3% inode=99%): [05:12:09] PROBLEM - Disk space on elastic1008 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 11226 MB (3% inode=99%): [05:13:06] (03CR) 10Techman224: [C: 031] Disable global AbuseFilter for private/fishbowl wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 (owner: 10Hoo man) [05:15:09] PROBLEM - Disk space on elastic1008 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 11259 MB (3% inode=99%): [05:17:09] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:20:59] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [05:21:59] well, the disk free space is going up at least [06:37:30] (03CR) 10Reedy: "I wonder if this will actually work" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 (owner: 10Hoo man) [06:39:21] (03CR) 10Reedy: "// Do tag settings" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 (owner: 10Hoo man) [06:39:46] (03CR) 10Techman224: "I thought settings were applied in the order given in the file. So the two groups would supersede the small wiki groups" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 (owner: 10Hoo man) [06:41:34] !log reedy updated /a/common to {{Gerrit|I3f9759db9}}: Let admins add users to new groups on zhwikivoyage [06:41:42] Logged the message, Master [06:43:35] (03CR) 10Reedy: "reedy@tin:/a/common$ git fetch https://reedy@gerrit.wikimedia.org/r/operations/m " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 (owner: 10Hoo man) [07:19:30] (03PS1) 10Matanya: imagescaler: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109502 [08:23:49] (03CR) 10Nemo bis: "This comment just to note that as per consensus and priority this is a +1. :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 (owner: 10Hoo man) [08:24:33] Reedy: Did you do it? [08:25:04] he didn't [08:25:10] (03CR) 10Techman224: [C: 031] Disable global AbuseFilter for private/fishbowl wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109498 (owner: 10Hoo man) [08:25:21] ;-; [09:49:17] (03CR) 10DrTrigon: [C: 031] "Can we merge this? Thanks." [operations/puppet] - 10https://gerrit.wikimedia.org/r/109246 (owner: 10Tim Landscheidt) [09:50:17] (03PS2) 10Ori.livneh: Tools: Install python-magic [operations/puppet] - 10https://gerrit.wikimedia.org/r/109246 (owner: 10Tim Landscheidt) [09:50:22] (03CR) 10Ori.livneh: [C: 032 V: 032] Tools: Install python-magic [operations/puppet] - 10https://gerrit.wikimedia.org/r/109246 (owner: 10Tim Landscheidt) [10:08:12] (03PS1) 10Matanya: site: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109507 [10:41:46] (03PS2) 10Matanya: site: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109507 [11:48:11] (03PS1) 10Matanya: mobile: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 [12:47:34] (03PS1) 10Matanya: mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 [12:47:49] hi andrewbogott [12:47:57] * andrewbogott waves [12:58:41] (03PS1) 10Matanya: emery: remove one udp2log logger. [operations/puppet] - 10https://gerrit.wikimedia.org/r/109516 [14:01:54] andrewbogott: what is harmon used for? [14:08:38] matanya: no idea. [14:08:45] I'd expect to learn here but it isn't described… https://wikitech.wikimedia.org/wiki/Tampa [14:09:01] Heh, dig this: https://wikitech.wikimedia.org/wiki/Harmon [14:09:03] andrewbogott: I have: https://rt.wikimedia.org/Ticket/Display.html?id=6267 [14:09:10] and https://rt.wikimedia.org/Ticket/Display.html?id=3213 [14:09:18] Best doc page ever [14:09:39] yeah :) [14:10:24] So I guess mutante|away thinks it's for nothing at all. [14:10:30] He's probably the only one who knows for sure [14:10:38] Unless you can tell from site.pp [14:10:59] site.pp saysnode "harmon.pmtpa.wmnet" { [14:10:59] $cluster = "misc" [14:10:59] include standard, [14:10:59] admins::roots [14:10:59] } [14:11:05] which is nothing [14:11:22] i'll poke him tomorrow [14:12:51] i'm out, see you later [19:11:09] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:12:09] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [19:35:55] (03PS1) 10Matanya: wgSitename: consistent using ' instead of " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109590 [19:46:17] (03PS1) 10Matanya: wgWhitelistRead: remove wikimedia Isreal related. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109591 [19:47:17] (03PS2) 10Matanya: wgWhitelistRead: remove wikimedia Israel related. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109591 [19:48:53] (03CR) 10jenkins-bot: [V: 04-1] wgWhitelistRead: remove wikimedia Israel related. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109591 (owner: 10Matanya) [19:51:28] (03PS3) 10Matanya: wgWhitelistRead: remove wikimedia Israel related. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109591 [19:59:55] (03PS1) 10Matanya: hewikinews: tidy NS look [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109593 [20:21:42] (03CR) 10Addshore: [C: 031] hewikinews: tidy NS look [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109593 (owner: 10Matanya) [20:35:59] PROBLEM - SSH on searchidx1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:37:49] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [21:08:15] (03PS2) 10Tim Landscheidt: hewikinews: tidy NS look [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109593 (owner: 10Matanya) [21:08:48] (03CR) 10Tim Landscheidt: "PS2: Removed trailing whitespace." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109593 (owner: 10Matanya) [21:12:06] (03CR) 10Tim Landscheidt: [C: 04-1] "var_dump ("Viquip\xc3\xa8dia"); => string(11); var_dump ('Viquip\xc3\xa8dia'); => string(17) "Viquip\xc3\xa8dia" :-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109590 (owner: 10Matanya) [21:12:49] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Last successful Puppet run was Sun 26 Jan 2014 06:12:37 PM UTC [21:12:49] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Sun Jan 26 21:12:48 UTC 2014 [21:16:23] (03PS2) 10Matanya: wgSitename: consistent using ' instead of " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109590 [21:16:25] (03CR) 10jenkins-bot: [V: 04-1] wgSitename: consistent using ' instead of " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109590 (owner: 10Matanya) [21:17:03] (03PS3) 10Matanya: wgSitename: consistent using ' instead of " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109590 [21:27:39] (03CR) 10Guido.iaquinti: [C: 031] emery: remove one udp2log logger. [operations/puppet] - 10https://gerrit.wikimedia.org/r/109516 (owner: 10Matanya) [21:29:02] hmpf, just wanted to ask matanya something [21:29:11] (03CR) 10Guido.iaquinti: [C: 031] Add gerrit ferm rules for production [operations/puppet] - 10https://gerrit.wikimedia.org/r/96980 (owner: 10Alexandros Kosiaris) [21:29:23] (03CR) 10Tim Landscheidt: [C: 031] "No backslashes and the strings are identical :-)." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109590 (owner: 10Matanya) [21:31:06] (03CR) 10Guido.iaquinti: [C: 04-1] turn planet into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 (owner: 10Dzahn) [21:32:04] (03CR) 10Guido.iaquinti: [C: 031] realm: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/109074 (owner: 10Matanya) [21:33:47] (03CR) 10Guido.iaquinti: [C: 031] dyanmicproxy: Add license headers to files [operations/puppet] - 10https://gerrit.wikimedia.org/r/109291 (owner: 10Yuvipanda) [21:34:49] (03CR) 10Guido.iaquinti: [C: 031] remove professor from site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/109285 (owner: 10Dzahn) [21:42:31] !log Recreating GeoData index [21:42:39] Logged the message, Master [22:08:09] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [22:09:39] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 36.74 ms [22:26:07] (03CR) 10Guido.iaquinti: [C: 031] webperf: Remove misleading references to Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/108831 (owner: 10Tim Landscheidt) [23:50:53] (03PS1) 10Ori.livneh: gdash: fix graph label to accord with metric [operations/puppet] - 10https://gerrit.wikimedia.org/r/109606 [23:51:24] (03CR) 10Ori.livneh: [C: 032 V: 032] gdash: fix graph label to accord with metric [operations/puppet] - 10https://gerrit.wikimedia.org/r/109606 (owner: 10Ori.livneh)