[00:03:14] New patchset: Bhartshorne; "taking stat1 out of a pseudo-analytics cluster. reverts changes 2377 and 2378" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2379 [00:03:41] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2379 [00:03:41] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2379 [00:24:54] PROBLEM - MySQL Idle Transactions on stat1 is CRITICAL: Connection refused by host [00:25:34] gn8 folks [00:28:24] PROBLEM - mysqld processes on stat1 is CRITICAL: Connection refused by host [00:28:24] PROBLEM - MySQL Replication Heartbeat on stat1 is CRITICAL: Connection refused by host [00:30:14] PROBLEM - MySQL Slave Delay on stat1 is CRITICAL: Connection refused by host [00:31:34] PROBLEM - MySQL Recent Restart on stat1 is CRITICAL: Connection refused by host [00:34:14] PROBLEM - Full LVS Snapshot on stat1 is CRITICAL: Connection refused by host [00:34:24] PROBLEM - MySQL disk space on stat1 is CRITICAL: Connection refused by host [00:35:34] PROBLEM - MySQL Slave Running on stat1 is CRITICAL: Connection refused by host [00:40:52] New patchset: Ottomata; "device_pipeline.py - getting rid of debug() function, using main()." [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2380 [00:40:54] New patchset: Ottomata; "Hacky first work on loader classes." [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2381 [00:42:22] New patchset: Ottomata; "base.py - adding schema in comments. Got lots of work to do to make this prettier" [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2382 [00:44:13] New patchset: Bhartshorne; "backing out mysql from stat1 until it moves to an internal IP or something else" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2383 [00:44:52] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2383 [00:44:52] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2383 [01:24:15] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [01:24:15] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [01:27:37] New patchset: Lcarr; "Fixing lvs puppet config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2384 [01:28:05] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2384 [01:28:06] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2384 [02:24:25] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1770s [02:26:55] !log LocalisationUpdate completed (1.18) at Wed Feb 8 02:26:55 UTC 2012 [02:27:03] Logged the message, Master [02:28:15] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 2000s [02:34:52] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 3s [02:43:02] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 51s [07:18:08] zzz [08:06:51] RECOVERY - MySQL Slave Delay on db38 is OK: OK replication delay 0 seconds [08:15:11] RECOVERY - MySQL Replication Heartbeat on db38 is OK: OK replication delay 0 seconds [08:39:36] PROBLEM - Puppet freshness on srv226 is CRITICAL: Puppet has not run in the last 10 hours [08:48:34] can more than 6 users signup from the same ip? [08:54:05] No [08:54:18] Hmm... [08:54:21] Not that I know of [08:56:12] so I can open an unlimited nuber of accounts? [08:56:52] no [08:57:09] unless you are in the account creator group, or we put a exemption in [08:57:27] (which is what sometimes happens for conferences and the likes) [09:00:48] I need that [09:00:56] need a bz entry? [09:02:54] yes, and a link to information about the event on wiki [09:03:54] ok, thanks [10:04:07] RECOVERY - MySQL Slave Delay on db1043 is OK: OK replication delay 0 seconds [10:06:37] RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay 0 seconds [11:34:52] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [11:34:52] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [11:37:51] New patchset: Mark Bergsma; "Add cp3001/cp3002 to the Bits caches esams aggregates" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2385 [11:38:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2385 [11:38:31] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2385 [11:38:32] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2385 [11:40:56] New patchset: Mark Bergsma; "Use %sumfunc% instead of %sumsep%" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2386 [11:41:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2386 [11:41:36] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2386 [11:41:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2386 [11:48:37] New patchset: Mark Bergsma; "More coffee is needed. Add missing comma" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2387 [11:49:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2387 [11:49:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2387 [11:56:09] New patchset: Mark Bergsma; "Remove old knsq hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2388 [11:56:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2388 [11:56:36] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2388 [11:56:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2388 [12:19:03] New patchset: Mark Bergsma; "Add eqiad bits varnish servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2389 [12:19:23] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2389 [12:19:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2389 [12:19:30] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2389 [12:19:30] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2389 [12:44:08] New patchset: Mark Bergsma; "Add eqiad hosts to aggregates (incomplete)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2390 [12:44:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2390 [12:45:31] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2390 [12:45:32] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2390 [13:09:37] New patchset: Mark Bergsma; "Add eqiad servers as empty DS references are not allowed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2391 [13:09:38] !log Batch-deleting 9,760 redirects on cswiktionary, requested by Danny B [13:09:40] Logged the message, Mr. Obvious [13:10:07] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2391 [13:10:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2391 [14:05:18] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2381 [14:05:47] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2380 [14:05:47] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2381 [14:05:47] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2380 [14:06:09] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2382 [14:06:09] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2382 [14:23:37] ACKNOWLEDGEMENT - Host srv189 is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn hardware problem - RT 2413 [14:29:48] New review: Dzahn; "basics for stat1 server" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2293 [14:42:11] New patchset: Dzahn; "add class for stat server per ezachte (RT 2162), apply on stat1, rebased" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2293 [14:42:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2293 [14:42:49] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2293 [14:42:50] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2293 [14:50:58] New patchset: Dzahn; "ensure mountpoints exist (btw, puppet feature req. 5015)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2392 [14:51:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2392 [14:51:31] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2392 [14:51:31] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2392 [15:10:26] PROBLEM - check_minfraud1 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:26] RECOVERY - check_minfraud1 on payments2 is OK: HTTP OK: HTTP/1.1 200 OK - 8643 bytes in 0.313 second response time [15:49:04] New patchset: Dzahn; "geoip support for stats1 per RT 2164" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2393 [15:49:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2393 [15:51:39] New patchset: Dzahn; "geoip support for stats1 per RT 2164" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2393 [15:57:38] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2393 [15:57:39] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2393 [16:05:28] New review: Dzahn; "see inline comments, can be improved, but current version to start. manual changes applied for BZ 23..." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2333 [16:05:29] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2333 [16:13:09] <|RicZzz|> Hi, have been IP-autoblocked for the second time because I am behind a NAT of a large provider ( http://en.wikipedia.org/w/index.php?title=User_talk:Richiez&curid=8678248&diff=475776627&oldid=454051650 ) Is there anything I can do to protect myself from such trouble? Should not whoever does the autoblocks check for such obvious cases? [16:14:30] |RicZzz|: ah, so it was a block and not a network issue? (but that wouldnt reset TCP connections??) [16:14:57] <|RicZzz|> its a different issue from yesterday. [16:15:36] |RicZzz|: oh, ok. unlucky,hm? sorry [16:16:25] |RicZzz|: but that means the network issue is gone at least? (or you wouldnt have noticed the block now?) [16:16:47] |RicZzz|: ask for adding the range to [[MediaWiki:Autoblock whitelist]] (we did that for o2 on dewp) [16:17:09] <|RicZzz|> I was able to get around it by reconnecting - got assigned a different IP in a very different block. No idea if the issue is gone for real. [16:18:42] <|RicZzz|> thanks for the whitelist tip [16:20:46] (I SO hope that this kind of problem end when there is IPv6…) [16:20:56] end → ends [16:24:33] New patchset: Dzahn; "minor formatting in noc index.html" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2394 [16:24:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2394 [16:24:59] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2394 [16:25:00] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2394 [16:25:04] <|RicZzz|> where exactly do I ask for an entry in the whitelist? Can't edit it and don't see any instructions. [16:25:35] <|RicZzz|> IPv6 has privacy extensions so it may not help that much? [16:28:34] |RicZzz|: I have no idea where enwp handles that, but [[Wikipedia:Administrators'_noticeboard]] looks good [16:30:50] <|RicZzz|> ok, i will log off and reconnect to get around the block and ask for help there.. [17:09:05] Mobile wikisource doesn't use wikisource logo but wikipedia's one. Can someone fix it ? http://en.m.wikisource.org/ [17:16:20] New patchset: Mark Bergsma; "Add node lists to role/cache.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2395 [17:21:07] !log reboot of dataset1001 for testing. [17:21:09] Logged the message, Master [17:23:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2395 [17:23:13] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2395 [17:23:19] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2395 [17:41:34] New patchset: Mark Bergsma; "Automatically generate squid list for eqiad text squids" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2396 [17:42:36] New patchset: Mark Bergsma; "Automatically generate squid list for eqiad text squids" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2396 [17:43:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2396 [17:43:52] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2396 [17:43:56] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2396 [17:47:43] New patchset: Mark Bergsma; "Actually include role::cache::configuration" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2397 [17:48:52] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2397 [17:48:54] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2397 [18:05:47] New patchset: Mark Bergsma; "Attempt to generate RRD expression with erb" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2398 [18:10:37] New patchset: Mark Bergsma; "Attempt to generate RRD expression with erb" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2398 [18:12:59] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2398 [18:12:59] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2398 [18:15:12] New patchset: Mark Bergsma; "Wrong file name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2399 [18:16:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2399 [18:16:36] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2399 [18:17:05] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2399 [18:19:33] New patchset: Mark Bergsma; "Wrong function name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2400 [18:21:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2400 [18:21:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2400 [18:28:02] New patchset: Mark Bergsma; "Generate eqiad squid aggregate RPN as well" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2401 [18:28:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2401 [18:28:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2401 [18:50:54] PROBLEM - Puppet freshness on srv226 is CRITICAL: Puppet has not run in the last 10 hours [18:57:46] New patchset: Bhartshorne; "changed swift stats gathering logic to ignore authentication requests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2402 [18:58:09] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2402 [18:58:09] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2402 [19:12:35] New patchset: Mark Bergsma; "Temporarily disable upload.eqiad while it doesn't exist" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2403 [19:13:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2403 [19:13:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2403 [19:30:23] New patchset: Mark Bergsma; "Refactor" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2404 [19:31:15] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2404 [19:31:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2404 [19:42:04] New patchset: Mark Bergsma; "Move RPN generation into a varnish_rpn function" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2405 [19:42:40] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2405 [19:42:41] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2405 [19:44:10] New patchset: Mark Bergsma; "Add missing 'end'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2406 [19:44:44] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2406 [19:44:45] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2406 [19:50:16] New patchset: Mark Bergsma; "Let's not use erb tags inside ruby functions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2407 [19:50:45] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2407 [19:50:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2407 [20:06:44] New patchset: Mark Bergsma; "Convert all aggregate RPN lists to functions and puppet hashes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2408 [20:07:31] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2408 [20:07:32] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2408 [20:19:52] New patchset: Mark Bergsma; "Support empty lists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2409 [20:20:13] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2409 [20:20:20] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2409 [20:20:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2409 [20:22:37] New patchset: Mark Bergsma; "Add missing 'api' key to decommissioned nodes hash" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2410 [20:24:10] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2410 [20:24:11] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2410 [20:33:16] New patchset: Mark Bergsma; "Allow difference between path and node list" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2411 [20:33:41] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2411 [20:33:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2411 [20:40:10] New patchset: Mark Bergsma; "Correct esams host names" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2412 [20:40:30] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2412 [20:40:40] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2412 [20:40:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2412 [20:45:05] New patchset: Mark Bergsma; "knsq* use .knams.wikimedia.org still" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2413 [20:45:33] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2413 [20:45:34] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2413 [20:49:16] New patchset: Mark Bergsma; "More domain name corrections" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2414 [20:49:44] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2414 [20:49:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2414 [20:57:28] New patchset: Mark Bergsma; "Fix graph stacking" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2415 [20:58:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2415 [20:58:05] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2415 [21:12:06] New patchset: Andre Engels; "A new attempt, adding some of Diederik's comment, and a new file simplepipelines.py, containing two basic pipelines to show the value of a single variable or two variables seen as independent and dependent." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2416 [21:12:08] New patchset: Andre Engels; "ignoring a few files" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2417 [21:12:09] New patchset: Andre Engels; "Partly reacting to Diederik's comments; adding simplepipelines.py with some useful pipelines for the case when speed of programming is more important than beauty of output." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2418 [21:12:10] New patchset: Andre Engels; "Revert "Partly reacting to Diederik's comments; adding simplepipelines.py"" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2419 [21:35:47] New patchset: Andre Engels; "ignoring a few files" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2417 [21:46:16] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [21:46:16] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [22:51:46] PROBLEM - Host cp1017 is DOWN: PING CRITICAL - Packet loss = 100% [23:14:45] New patchset: Bhartshorne; "adding 304s to the list of interesting respnose codes for swift" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2420 [23:15:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2420 [23:15:16] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2420 [23:15:17] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2420 [23:17:36] PROBLEM - MySQL Replication Heartbeat on db11 is CRITICAL: CRIT replication delay 1781 seconds [23:22:06] PROBLEM - MySQL Slave Running on db11 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Cant drop database centralauth: database doesnt exist on [23:29:06] RECOVERY - MySQL Replication Heartbeat on db11 is OK: OK replication delay 0 seconds [23:33:36] RECOVERY - MySQL Slave Running on db11 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [23:49:35] !log asher synchronized wmf-config/db.php 'pulling db11' [23:49:37] Logged the message, Master [23:52:41] !log asher synchronized wmf-config/db.php 'returning db11' [23:52:42] Logged the message, Master [23:56:23] gn8 folks