[00:00:35] PROBLEM - HTTP radosgw on ms-fe1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:14] PROBLEM - HTTP radosgw on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:18] Error 400 on SERVER: Duplicate definition: Exec[apt-update-for-${name}] is already defined in file /var/lib/git/operations/puppet/modules/apt/manifests/repository.pp at line 59; cannot redefine at /var/lib/git/operations/puppet/modules/apt/manifests/repository.pp:59 on node virt0.wikimedia.org [00:02:19] * Ryan_Lane sighs [00:02:46] Yeah, that is the single worst thing about puppet. It should accept multiple non-conflicting definitions. [00:03:07] well, that one specifically is because single quotes were used [00:03:15] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 00:03:09 UTC 2013 [00:04:14] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [00:04:30] New patchset: Ryan Lane; "Use double quotes where dereferencing is needed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74302 [00:05:12] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74302 [00:06:36] PROBLEM - HTTP radosgw on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:46] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:18] New review: Catrope; "Looks good to me. Scheduling this for deployment in tomorrow's lightning deploy window (23 hours fro..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/72356 [00:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 00:24:49 UTC 2013 [00:25:26] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [00:28:06] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 00:27:58 UTC 2013 [00:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [00:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 00:28:54 UTC 2013 [00:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [00:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 00:32:45 UTC 2013 [00:33:16] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [00:45:36] PROBLEM - Disk space on ms-be1003 is CRITICAL: DISK CRITICAL - free space: / 5666 MB (3% inode=98%): [00:53:16] New patchset: TTO; "(bug 51328 and bug 51232) add autopatrolled and uploader groups to ckbwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74308 [00:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 00:54:45 UTC 2013 [00:55:26] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [00:55:56] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:36] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 00:57:33 UTC 2013 [00:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:46] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 00:58:45 UTC 2013 [00:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [00:59:56] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [01:00:56] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [01:02:36] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 01:02:35 UTC 2013 [01:03:16] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [01:25:00] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 01:24:53 UTC 2013 [01:25:30] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [01:27:50] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 01:27:45 UTC 2013 [01:28:20] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [01:29:00] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 01:28:55 UTC 2013 [01:29:10] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [01:32:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:30] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 01:33:28 UTC 2013 [01:33:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [01:34:10] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [01:35:30] PROBLEM - Disk space on ms-be1005 is CRITICAL: DISK CRITICAL - free space: / 5690 MB (3% inode=97%): [01:36:00] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [01:36:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:36:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:36:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [01:36:00] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [01:36:00] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [01:36:00] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [01:54:50] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 01:54:43 UTC 2013 [01:55:30] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [01:57:40] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 01:57:38 UTC 2013 [01:58:20] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [01:58:50] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 01:58:44 UTC 2013 [01:59:10] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [02:02:50] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 02:02:43 UTC 2013 [02:03:10] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:04] !log LocalisationUpdate completed (1.22wmf10) at Thu Jul 18 02:15:04 UTC 2013 [02:15:16] Logged the message, Master [02:24:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 02:24:51 UTC 2013 [02:25:27] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:17] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 02:28:07 UTC 2013 [02:28:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:28] !log LocalisationUpdate completed (1.22wmf9) at Thu Jul 18 02:28:28 UTC 2013 [02:28:39] Logged the message, Master [02:28:57] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 02:28:48 UTC 2013 [02:29:17] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [02:33:17] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 02:33:13 UTC 2013 [02:34:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [02:43:17] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 18 02:43:17 UTC 2013 [02:43:29] Logged the message, Master [02:54:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 02:54:47 UTC 2013 [02:55:27] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [02:57:47] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 02:57:46 UTC 2013 [02:58:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [02:58:27] PROBLEM - HTTP Apache on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:57] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 02:58:47 UTC 2013 [02:59:17] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [03:02:47] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 03:02:46 UTC 2013 [03:03:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [03:04:27] PROBLEM - HTTP Apache on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:53] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 03:24:49 UTC 2013 [03:25:33] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [03:27:09] New patchset: Ryan Lane; "Explicitly disable php for uploads directory" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74319 [03:27:43] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 03:27:38 UTC 2013 [03:28:03] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74319 [03:28:23] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [03:28:53] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 03:28:49 UTC 2013 [03:29:13] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [03:33:03] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 03:32:54 UTC 2013 [03:33:13] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [03:35:43] PROBLEM - HTTP Apache on ms-fe1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:43] PROBLEM - HTTP Apache on ms-fe1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:40:52] New patchset: Ryan Lane; "Bind memcached to 127.0.0.1 on openstackmanager nodes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74321 [03:45:00] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74321 [03:54:13] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [03:54:53] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 03:54:46 UTC 2013 [03:55:33] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:13] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 03:58:12 UTC 2013 [03:58:23] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:53] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 03:58:52 UTC 2013 [03:59:13] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [04:03:23] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 04:03:17 UTC 2013 [04:04:13] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [04:07:24] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:04] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 04:24:59 UTC 2013 [04:25:34] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [04:27:44] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 04:27:41 UTC 2013 [04:28:24] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [04:28:54] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 04:28:49 UTC 2013 [04:29:14] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [04:36:01] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 04:35:47 UTC 2013 [04:36:14] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [04:36:15] New patchset: Ori.livneh; "Expand documentation of EventLogging module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74322 [04:54:54] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 04:54:47 UTC 2013 [04:55:34] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:44] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 04:57:34 UTC 2013 [04:58:24] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [04:58:54] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 04:58:45 UTC 2013 [04:59:14] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [05:02:54] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 05:02:48 UTC 2013 [05:03:14] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:19] RECOVERY - HTTP Apache on ms-fe1001 is OK: HTTP OK: HTTP/1.1 200 OK - 229 bytes in 3.065 second response time [05:06:19] RECOVERY - HTTP Apache on ms-fe1003 is OK: HTTP OK: HTTP/1.1 200 OK - 229 bytes in 6.342 second response time [05:06:19] RECOVERY - HTTP Apache on ms-fe1002 is OK: HTTP OK: HTTP/1.1 200 OK - 229 bytes in 0.001 second response time [05:06:19] RECOVERY - HTTP radosgw on ms-fe1003 is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.004 second response time [05:06:19] RECOVERY - HTTP radosgw on ms-fe1004 is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.003 second response time [05:06:39] RECOVERY - HTTP radosgw on ms-fe1001 is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 1.839 second response time [05:07:09] RECOVERY - HTTP Apache on ms-fe1004 is OK: HTTP OK: HTTP/1.1 200 OK - 229 bytes in 0.001 second response time [05:07:09] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.004 second response time [05:07:12] RECOVERY - HTTP radosgw on ms-fe1002 is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.004 second response time [05:07:35] ignore that [05:07:37] that would be me [05:07:44] I should really make this non-paging again... [05:10:48] New patchset: Faidon; "Remove paging ms-fe.eqiad.wmnet check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74324 [05:11:43] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74324 [05:14:39] RECOVERY - Disk space on ms-be1003 is OK: DISK OK [05:17:29] RECOVERY - Disk space on ms-be1001 is OK: DISK OK [05:17:39] RECOVERY - Disk space on ms-be1005 is OK: DISK OK [05:24:59] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 05:24:55 UTC 2013 [05:25:39] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [05:27:39] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 05:27:36 UTC 2013 [05:28:19] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [05:28:49] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 05:28:47 UTC 2013 [05:29:19] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [05:33:09] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 05:33:01 UTC 2013 [05:33:09] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [05:54:59] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 05:54:49 UTC 2013 [05:55:39] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [05:58:09] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 05:58:07 UTC 2013 [05:58:19] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [05:58:49] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 05:58:48 UTC 2013 [05:59:19] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [06:02:59] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 06:02:56 UTC 2013 [06:03:09] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:19] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74322 [06:20:08] thanks paravoid [06:25:01] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 06:24:52 UTC 2013 [06:25:31] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:21] PROBLEM - Disk space on ms-be1002 is CRITICAL: DISK CRITICAL - free space: / 5660 MB (3% inode=98%): [06:28:01] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [06:28:51] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 06:28:48 UTC 2013 [06:29:01] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 06:28:58 UTC 2013 [06:29:21] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:21] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:21] RECOVERY - Disk space on ms-be1002 is OK: DISK OK [06:32:41] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 06:32:39 UTC 2013 [06:33:11] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [06:55:01] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 06:54:51 UTC 2013 [06:55:31] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [06:57:41] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 06:57:39 UTC 2013 [06:58:21] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [06:58:51] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 06:58:50 UTC 2013 [06:59:21] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [07:02:41] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 07:02:39 UTC 2013 [07:03:11] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [07:25:00] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 07:24:50 UTC 2013 [07:25:30] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [07:27:40] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 07:27:38 UTC 2013 [07:28:20] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [07:28:50] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 07:28:49 UTC 2013 [07:29:20] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [07:31:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [07:33:40] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 07:33:30 UTC 2013 [07:34:10] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:50] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 07:54:47 UTC 2013 [07:55:30] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [07:56:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [07:58:20] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 07:58:17 UTC 2013 [07:58:20] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:00] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 07:58:54 UTC 2013 [07:59:20] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:20] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 08:03:10 UTC 2013 [08:04:10] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 08:24:54 UTC 2013 [08:25:37] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [08:29:07] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 08:28:59 UTC 2013 [08:29:07] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 08:28:59 UTC 2013 [08:29:17] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [08:29:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [08:32:47] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 08:32:41 UTC 2013 [08:33:07] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 08:54:47 UTC 2013 [08:55:37] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [08:57:47] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 08:57:40 UTC 2013 [08:58:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [08:58:57] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 08:58:47 UTC 2013 [08:59:17] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [09:02:47] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 09:02:41 UTC 2013 [09:03:07] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [09:23:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [09:25:06] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 09:24:59 UTC 2013 [09:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [09:28:06] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 09:28:02 UTC 2013 [09:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [09:29:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 09:29:48 UTC 2013 [09:30:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [09:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 09:32:42 UTC 2013 [09:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [09:46:49] New patchset: Ori.livneh; "Clean-up: port 'analysis' class to a role class." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74334 [09:46:49] New patchset: Ori.livneh; "Add eventlogging::plugin custom resource type" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74335 [09:49:48] New patchset: Mark Bergsma; "Factor out LVS realserver ip include, add text-varnish IPs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74336 [11:38:26] PROBLEM - SSH on searchidx1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:01] probably we can't deploy traffic today [11:54:07] ok [11:54:08] and on friday that usually doesn't make people happy [11:54:13] yeah [11:54:20] so perhaps early next week we'll do that [11:54:28] hopefully they notice stuff gets better and faster :) [11:54:37] ok [11:54:37] hehe while it lasts [11:54:43] while wikidata is the only one on those varnish servers ;) [11:54:49] ah, ok [11:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 11:54:48 UTC 2013 [11:55:19] wikidata will have 6.4 TB of cache space available in eqiad alone [11:55:28] but if it goes well, the other wikis will follow soon ;) [11:55:31] :D [11:55:59] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [11:56:49] I guess, just let the community know now that we're gonna do this [11:56:54] that it will likely happen early next week, we'll post a new notice when we're switching traffic [11:56:54] and to let us know of any problems that may be related? [11:56:54] ok [11:57:36] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 11:57:32 UTC 2013 [11:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [11:59:14] ok, the PURGE hit/miss ratio patch compiles [11:59:18] I guess I can roll it out ;) [11:59:48] EVERYWHERE! [11:59:50] :) [12:01:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 12:01:53 UTC 2013 [12:02:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [12:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 12:02:39 UTC 2013 [12:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [12:04:45] something's fucked hehe [12:04:52] the init script doesn't finish starting varnish, [12:04:59] and both varnish and vhtcpd are consuming 100% cpu [12:07:02] perhaps that was just vhtcpd backlog processing [12:07:59] because it seems to work [12:12:54] deployed it on cp1046 (mobile cache) now [12:15:02] 34% cache ratio? [12:15:07] that can't be right [12:16:59] it was around 20 [12:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 12:24:49 UTC 2013 [12:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [12:27:27] !log Inserted varnish 3.0.3plus~rc1-wm14 packages into the precise-wikimedia APT repository [12:27:36] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 12:27:32 UTC 2013 [12:27:38] Logged the message, Master [12:28:19] New patchset: Mark Bergsma; "varnish (3.0.3plus~rc1-wm14) precise; urgency=low" [operations/debs/varnish] (testing/3.0.3plus-rc1) - https://gerrit.wikimedia.org/r/74354 [12:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [12:28:52] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/3.0.3plus-rc1) - https://gerrit.wikimedia.org/r/74344 [12:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 12:28:53 UTC 2013 [12:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [12:29:51] wouldn't it be interesting to have counters for Vary based hits/misses? [12:32:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:56] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 12:32:47 UTC 2013 [12:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:36:27] /* No Vary: header, no worries */ [12:36:31] excellent comment [12:37:12] lol [12:39:23] New patchset: Hashar; "beta: text cache in $wgSquidServerNoPurge" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74355 [12:39:54] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74355 [12:55:06] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 12:55:00 UTC 2013 [12:55:37] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [12:58:26] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 12:58:19 UTC 2013 [12:58:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:26] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 12:59:16 UTC 2013 [12:59:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [12:59:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.147 second response time [13:00:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [13:02:56] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 13:02:45 UTC 2013 [13:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:16] New review: Hashar; "Also need to add DataTypes in the wmf branches." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74291 [13:09:17] New review: Aude; "needs testing. " [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/74291 [13:21:42] wow [13:21:46] while reading the code [13:21:54] varnish is doing a lot of unnecessary work just to purge objects [13:22:14] it's going through all variants and expired objects and all that to find the best one [13:22:20] only to purge them all later [13:24:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 13:24:44 UTC 2013 [13:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [13:26:32] New patchset: Ottomata; "Granting Christian access to privatedata on stat1002. RT 5467" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74359 [13:28:16] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 13:28:08 UTC 2013 [13:28:23] New patchset: Ottomata; "Granting Christian access to privatedata on stat1002. RT 5467" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74359 [13:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [13:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 13:28:54 UTC 2013 [13:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [13:29:18] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74359 [13:30:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.030 second response time [13:33:16] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 13:33:12 UTC 2013 [13:34:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [13:37:05] paravoid: hi! have you done the puppet glue to get gdnsd installed on jenkins servers? :) [13:38:53] not yet, sorry [13:43:12] https://integration.wikimedia.org/ci/job/operations-puppet-pep8/3693/violations/file/modules/authdns/files/authdns-gen-zones.py/? [13:43:15] stupid pep8 [13:43:20] :D [13:44:11] the suggestion is completely stupid [13:44:31] oh that one [13:46:39] hey ^demon - I keep missing you on irc. we should talk about merging CirrusSearch's elasticsearch stuff at some point. [13:47:25] <^demon> Yeah. I was going to merge yesterday but I saw that the submodule for Elastica was causing problems with jenkins. [13:47:51] ah! now it all makes sese. [13:48:14] so you think it'd be better to move that submodule to gerrit? [13:48:33] also, are you still jet lagged? it is super early there [13:48:50] <^demon> No, no, been here a week (and I drove, so no jets). [13:48:59] <^demon> cmjohnson1 and I have some work to get done early today :) [13:49:04] car lagging .. [13:49:20] what is the issue with the submodules in jenkins ? [13:49:28] the job might not be properly configured [13:49:44] <^demon> https://integration.wikimedia.org/ci/job/mwext-CirrusSearch-lint/45/console - submodule tries to clone from on-disk, but it's a github submodule. [13:50:28] driving sounds like it'd be equal parts fun and torture [13:50:32] New patchset: Petr Onderka; "reading dump; command-line parameters" [operations/dumps/incremental] (gsoc) - https://gerrit.wikimedia.org/r/74361 [13:50:33] New patchset: Aude; "Update and cleanup settings for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74362 [13:50:48] <^demon> hashar: I figured the path of least resistance would be to just copy the library from github to gerrit. [13:51:03] <^demon> manybubbles: About 60/40 torture/fun ;-) [13:51:18] New review: Aude; "although I am quite confident this patch does everything correct, it needs careful sanity check and ..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/74362 [13:51:29] ^demon: hmm that is crazy [13:51:31] ^demon and hashar: that also has the advantage of making sure it doesn't go away - as unlikely as that is. [13:51:38] New patchset: Jeremyb; "rv tab in the middle of a line" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74363 [13:51:42] ^demon: seems Jenkins rewrite the github URL to be a relative path [13:52:14] <^demon> Yeah, which makes sense for most things since they are in gerrit and would be on disk, but not in this case. [13:54:19] <^demon> hashar: It might make sense to kill zuul for the bit while gerrit is down, so it doesn't keep trying to reconnect pointlessly. [13:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Thu Jul 18 13:54:49 UTC 2013 [13:55:02] New patchset: Aude; "Update and cleanup settings for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74362 [13:55:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [13:56:13] ^demon: sure [13:56:18] fucking pep8 [13:56:26] so stupid [13:56:32] New patchset: Aude; "Update and cleanup settings for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74362 [13:56:34] ^demon: feel free to shut zuul down whenever you want :) [13:56:36] if not args.force and \ [13:56:36] (os.path.getmtime(filepath) <= os.path.getmtime(zonefilepath)): [13:56:39] continue [13:56:44] files/authdns-gen-zones.py:77:13: E122 continuation line missing indentation or outdented [13:56:51] if not args.force and \ [13:56:51] (os.path.getmtime(filepath) <= os.path.getmtime(zonefilepath)): [13:56:54] continue [13:56:55] files/authdns-gen-zones.py:77:17: E125 continuation line does not distinguish itself from next logical line [13:56:55] New review: Aude; "still needs additional testing and sanity check" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/74362 [13:56:55] <^demon> hashar: Will do, just wanted to let you know :) [13:57:06] what the heck am I supposed to do then? [13:57:35] and if I put a parentheses around the second line gets at 80 chars and I get a warning for line length [13:57:46] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Jul 18 13:57:38 UTC 2013 [13:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:40] paravoid: https://gist.github.com/hashar/6029529 :-D [13:58:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Thu Jul 18 13:58:50 UTC 2013 [13:59:15] and that's supposed to be more readable? [13:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [13:59:18] because it's not really [14:00:40] <^demon> cmjohnson1: Ok, let's do this shiznit [14:01:16] paravoid: but it doesn't have the \ in it which makes the python gods happy [14:01:31] okay [14:02:05] !log removing/swapping sdb2 on manganese [14:02:16] Logged the message, Master [14:02:37] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Jul 18 14:02:34 UTC 2013 [14:02:45] <^demon> !log stopped gerrit & zuul services on manganese and gallium respectively [14:02:49] paravoid: another possibility is to use a variable for some of the long boolean parts [14:02:56] Logged the message, Master [14:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [14:03:39] stop the presses! [14:03:43] !log shutting down manganese [14:03:53] Logged the message, Master [14:04:36] PROBLEM - zuul_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-server [14:06:26] PROBLEM - Host manganese is DOWN: PING CRITICAL - Packet loss = 100% [14:07:25] paravoid: or you can slightly adapt your code https://gist.github.com/hashar/6029608 [14:07:58] > 79 :) [14:08:13] don't worry, I'll rework it [14:08:22] ǃlog Gerrit sleeping [14:08:26] it's just such a waste of time to adapt perfectly readable code [14:08:37] paravoid: or import getmtime with a shorter name [14:08:54] hashar: morebots sleeping? [14:09:24] paravoid: I tend to agree. My editor just reports me all the errors whenever I save and I got used to fix them [14:10:34] I had pep8 from wheezy which didn't show these :) [14:10:44] (1.2) [14:10:49] ahhh [14:11:10] one day I will get tox back ported on Precise and make use of virtual env to do the checks [14:11:20] this way you can pin which version of pep8 to use :) [14:11:34] hmm wait no. That would install the package from pip [14:11:36] RECOVERY - Host manganese is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [14:14:28] ^demon the disk is replaced and is synchronizing [14:14:39] <^demon> Mmk [14:19:50] <^demon> cmjohnson1: So, I had someone ask me a question that I didn't know the answer to. "Why does a disk replacement require downtime if there's no-interrupt RAIDs?" [14:20:07] <^demon> I know zilch here so was like "Err, cuz ops told me it had to come down :))" [14:21:14] in this case, the disks are located inside the server so I had to take it down to remove it [14:21:50] yay, no hotswap [14:22:11] <^demon> cmjohnson1: And that answer makes absolute perfect sense now. Thanks :) [14:22:52] ^demon the sync process is taking a little long...you can monitor to 'watch cat /proc/mdstat' [14:23:20] 2.7% [14:23:24] How much data is there? [14:23:34] <^demon> All the data! [14:24:56]