[00:19:14] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [00:22:24] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:51:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.658 second response time [01:21:10] (03PS1) 10Dereckson: Add filemover group on it.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83386 [01:39:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [01:49:35] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [01:52:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:45] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.138 second response time [01:57:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:59] !log LocalisationUpdate failed: git pull of extensions failed [02:01:03] Logged the message, Master [02:01:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [02:29:39] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [02:32:49] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:43:51] Is there a bug about LocalisationUpdate failing? [03:02:43] https://bugzilla.wikimedia.org/show_bug.cgi?id=53890 [03:14:26] PROBLEM - Puppet freshness on sq42 is CRITICAL: No successful Puppet run in the last 10 hours [03:14:44] (03PS1) 10Ori.livneh: Log more Navigation Timing events on mobile [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83388 [03:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [04:16:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [04:30:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:31:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.326 second response time [04:34:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:36:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [04:51:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.323 second response time [05:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [05:52:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.148 second response time [05:57:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:58:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.192 second response time [06:14:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.242 second response time [06:36:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.947 second response time [07:06:31] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: No successful Puppet run in the last 10 hours [07:40:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [08:48:21] PROBLEM - Disk space on analytics1011 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/g 74662 MB (3% inode=99%): [09:28:34] (03CR) 10Ori.livneh: [C: 04-1] "Not sure yet if that's what I want to do." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83388 (owner: 10Ori.livneh) [09:31:39] (03PS1) 10Ori.livneh: Ganglia backend for StatsD: Emit metadata every send_metadata_interval [operations/puppet] - 10https://gerrit.wikimedia.org/r/83396 [09:33:34] (03PS2) 10Ori.livneh: NavigationTiming StatsD instance: flush every 5 mins [operations/puppet] - 10https://gerrit.wikimedia.org/r/83230 [09:50:40] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 42 copy to table, 0 statistics [09:53:40] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [10:55:32] (03CR) 10Jalexander: [C: 031] "looks good to me, adding a couple who may be able to push." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83218 (owner: 10TTO) [12:14:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.938 second response time [13:15:11] PROBLEM - Puppet freshness on sq42 is CRITICAL: No successful Puppet run in the last 10 hours [13:40:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [14:01:24] (03CR) 10Ottomata: "(1 comment)" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/83191 (owner: 10Ottomata) [14:04:01] (03PS3) 10Ottomata: Updating kafka script and init scripts with recent changes in Kafka bin/*.sh scripts from 0.8 branch. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/83137 [14:04:16] (03PS4) 10Ottomata: Updating kafka scripts with recent changes from 0.8 branch. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/83137 [14:05:32] (03CR) 10Ottomata: [C: 032 V: 032] Updating kafka scripts with recent changes from 0.8 branch. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/83137 (owner: 10Ottomata) [14:06:08] hiaaaooo [14:06:19] can any shell wizards out there find a better way to do this? [14:06:21] https://gerrit.wikimedia.org/r/#/c/83191/1/debian/bin/kafka [14:15:54] (03PS3) 10Ottomata: Adding environment var ZOOKEEPER_URL. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/83191 [14:16:00] (03CR) 10Ottomata: [C: 032 V: 032] Adding environment var ZOOKEEPER_URL. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/83191 (owner: 10Ottomata) [14:24:35] (03CR) 10Ottomata: [C: 032 V: 032] Added README.md markdown file [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/83347 (owner: 10Edenhill) [14:44:50] (03PS8) 10Ottomata: (WIP) Initial Debian version [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis) [14:45:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.031 second response time [14:47:28] (03PS9) 10Ottomata: (WIP) Initial Debian version [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis) [15:00:02] (03CR) 10Edenhill: [C: 031] "(1 comment)" [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis) [15:19:30] hi, is someone arround who can put debian packages to apt.wikimedia.org ? [15:59:34] ottomata: [15:59:39] args="a b c --zookeeper d e f" [15:59:45] TMP=${args/--zookeeper} [15:59:51] if [ "${#args}" -ne "${#TMP}" ]; then echo "has zookeeper arg"; fi [16:01:00] it's the standard hack: the '/' strips '--zookeeper' if it finds it, and then you compare variable lengths to see if they're the same (meaning no --zookeeper was matched and stripped) or unequal (meaning args had '--zookeeper') [16:01:36] but you'd have to switch the shebang to #!/bin/bash; i'm not sure dash (which ubuntu uses as /bin/sh) supports that kind of parameter expansion [16:04:04] i'd keep the grep probably but express it as: grep -- '--zookeeper' <<<"$*" [16:11:32] yeah, ori-l, i could do this in bash [16:11:40] [[ ]] has wildcard support [16:11:41] can do [16:11:56] [[ $@ != *—zookeeper* ]] [16:11:57] or somethign [16:12:04] but since this is going to be part of a shipped .deb [16:12:08] faidon wants me to use sh [16:12:38] well, use grep then [16:12:40] https://wiki.ubuntu.com/DashAsBinSh#A.24.7Bparm.2BAC8.3F.2BAC8-pat.5B.2BAC8-str.5D.7D [16:12:43] hehe :p [16:12:52] and merge a couple of patches for me! [16:12:57] it will help you in unspecified ways [16:13:04] https://gerrit.wikimedia.org/r/#/c/83230/ [16:13:10] haha, sure! (i'm in a meeting right now though) [16:13:21] it will help the meeting! [16:13:42] just kidding. don't worry about it if you're in a meeting. [16:14:34] i got it open in a tab, will check it soon [16:15:09] k. the js changes are just ported from analytics/statsd-ganglia, don't need to be reviewed [16:48:20] (03PS1) 10Nikerabbit: ULS event logging for wiktionaries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83418 [17:00:06] (03Abandoned) 10Matthias Mullie: Add cron to generate CSVs for ee-dashboard [operations/puppet] - 10https://gerrit.wikimedia.org/r/71282 (owner: 10Matthias Mullie) [17:06:55] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: No successful Puppet run in the last 10 hours [17:09:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.026 second response time [17:13:17] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.399 second response time [17:16:08] hi, is someone arround who can put debian packages to apt.wikimedia.org ? [17:16:29] hey physikerwelt :-] [17:16:35] hi [17:16:40] ops could, they are together talking to each other face to face [17:16:51] so most probably not reading this channel right now :D [17:17:17] was it for latexml ? [17:17:27] ok I was wondering since I have put this questions approxmently 10 times to the channel [17:17:28] ja [17:17:30] ye [17:17:31] s [17:18:34] should I file a bugreport at bugzilla`? [17:24:11] physikerwelt: Has anyone reviewed them? [17:24:57] hmz, saves on mediawiki.org are slow / sometimes timing out for me [17:25:20] Big/complex pages? [17:25:26] The whole LaTeXML code? No, not someone at wikimedia just the debian installation script [17:25:26] nope [17:25:53] I think there is a internal code review at NIST but I'm not sure [17:26:39] does that mean the NSA reviewed it? :-P [17:26:52] physikerwelt: depends on the group that produced the code; but it is strongly encouraged [17:27:19] I'd say that probably about 40% of the code is at least reviewed by team members :p [17:27:20] small, simple pages. not seeing the issue on en.wp. getting actual timeouts / wikimedia errors on mediawiki.org though [17:27:41] saves end up going through anyway, but error reported to user [17:29:04] The idea with the package in the repository can be changed quickly if there are errors in the code [17:29:29] I think it would be hard to find someone to review the whole latexml code [17:29:32] Reedy, try editing e.g. https://www.mediawiki.org/wiki/Project:Sandbox [17:31:15] does timeout for me apparently weird [17:34:05] Reedy: I am wondering if that is varnish related [17:34:33] err squid [17:34:49] from 208.80.154.136 via cp1005.eqiad.wmnet (squid/2.7.STABLE9) to 10.64.0.131 (10.64.0.131) Error: ERR_READ_TIMEOUT, errno [No Error] [17:36:07] 402 entries from mediawikiwiki in slow-parse.log, 338 of them from VisualEditor/TranslationCentral [17:36:13] not sure if related [17:37:19] concerning https://git.wikimedia.org/summary/operations%2Fdebs%2Flatexml can I do anything to get in the apt repository? [17:37:36] physikerwelt: e-mail ops-requests@rt.wikimedia.org [17:38:03] ori-l: thanks [17:43:18] (03CR) 10Ryan Lane: [C: 032] Add GCM cipher and remove DES [operations/puppet] - 10https://gerrit.wikimedia.org/r/83043 (owner: 10Ryan Lane) [17:43:46] !log adding GCM cipher and removing DES from SSL config [17:43:50] Logged the message, Master [17:44:12] what's up all these? "logmsgbot: LocalisationUpdate failed: git pull of extensions failed" https://wikitech.wikimedia.org/wiki/Server_admin_log [17:44:21] robla: Git repo problems after the migration [17:44:31] Missing stuff [17:44:45] !log depooling ssl1001 [17:44:48] Logged the message, Master [17:45:06] robla: https://bugzilla.wikimedia.org/show_bug.cgi?id=53890 & https://bugzilla.wikimedia.org/show_bug.cgi?id=53841 [17:49:04] (03PS1) 10Springle: add springle to icinga sms [operations/puppet] - 10https://gerrit.wikimedia.org/r/83421 [17:50:26] (03PS2) 10Hashar: Update some rss/atom feeds in the planet wikimedia config [operations/puppet] - 10https://gerrit.wikimedia.org/r/83348 (owner: 10Jeroen De Dauw) [17:51:27] (03PS2) 10Springle: add springle to icinga sms [operations/puppet] - 10https://gerrit.wikimedia.org/r/83421 [17:51:40] (03CR) 10Springle: [C: 032 V: 032] add springle to icinga sms [operations/puppet] - 10https://gerrit.wikimedia.org/r/83421 (owner: 10Springle) [17:52:19] ori-l: I'm even to stupid to send an email... I got the reply User 'schubotz@tu-berlin.de' could not be loaded in the mail gateway do I have to use a special address? [17:53:29] * bawolff attampts to be naggy about RT 5735 (vhtcpd not working on cp1061) [17:54:09] bawolff: half hearted attempt, there [17:54:31] Well you know, don't want to be too annoying :P [17:54:38] !log pooling ssl1001 and depooling ssl1002-5 [17:54:41] Logged the message, Master [17:55:06] I was naggy on sunday, but that kind of fell on deaf ears due to it being sunday [17:55:25] renaggify [17:56:03] i removed {{Please leave this line alone and write below (this is the coloured heading)}} , page saves again [17:56:31] snappy, even. [17:57:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:26] bawolff: you should be able to see https://rt.wikimedia.org/Ticket/Display.html?id=5614 now :) [17:57:59] (03PS1) 10RobH: fixing tab spacing [operations/puppet] - 10https://gerrit.wikimedia.org/r/83423 [17:58:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [17:58:33] physikerwelt: e-mail the request to me and i'll forward it and cc you [17:58:36] mutante_: If upgrading tzdata breaks other stuff we've got big problems ;) [17:59:51] jeremyb: I can indeed. I've now moved on to different and other varnish servers not working ;) [18:00:16] reintroduce template, timeout back [18:00:16] (03CR) 10RobH: [C: 032] "spaces > tabs, plus removed the now defunct HKT paging hours" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83423 (owner: 10RobH) [18:04:35] !log repooling ssl1002-5 and depooling ssl1006-9 [18:04:38] Logged the message, Master [18:06:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.246 second response time [18:11:07] !log repooling ssl1006-9 [18:11:10] Logged the message, Master [18:14:06] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: various misc wikis to 1.22wmf16 [18:14:08] Logged the message, Master [18:15:12] (03PS1) 10RobH: cmjohnson new sshkey [operations/puppet] - 10https://gerrit.wikimedia.org/r/83425 [18:15:22] cmjohnson1: CAn you login to gerrit and +1 that change for me? [18:15:33] just seems extra secrute if the person asking for the change approves it =] [18:15:36] secure [18:16:13] can i get a merge on https://gerrit.wikimedia.org/r/#/c/83230/ and the patch it depends on? [18:16:43] (03CR) 10Cmjohnson: [C: 031 V: 031] cmjohnson new sshkey [operations/puppet] - 10https://gerrit.wikimedia.org/r/83425 (owner: 10RobH) [18:16:51] also, who changed the jquery version for ganglia-web? [18:16:58] (03CR) 10Cmjohnson: cmjohnson new sshkey [operations/puppet] - 10https://gerrit.wikimedia.org/r/83425 (owner: 10RobH) [18:17:40] ori-l, are you still poking at the mediawiki.org breakage? [18:18:12] Eloquence: i am, but i'm a bit stuck. i've narrowed it down to Template:Please_leave_this_line_alone_and_write_below_(this_is_the_coloured_heading) [18:18:28] ori-l, it seems to occur on other pages as well though, so perhaps a general template issue of some kind [18:18:42] (03CR) 10RobH: [C: 032] cmjohnson new sshkey [operations/puppet] - 10https://gerrit.wikimedia.org/r/83425 (owner: 10RobH) [18:19:01] !log depooling ssl3001-2 [18:19:04] Logged the message, Master [18:19:15] I'm not seeing any suspicious template-related changes in wmf16 - https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf16 - except for a modification of templatedata: https://gerrit.wikimedia.org/r/#/q/d66b4c60,n,z [18:20:01] * ori-l looks [18:20:44] (03PS1) 10Lcarr: adding in some mmgt addresses in ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/83426 [18:20:58] anomie, saves on mediawiki.org are broken, ori-l is poking at it right now [18:21:25] broken as in "server reports time out but save completes" [18:21:47] some context in channel log [18:21:50] Eloquence: I'm not likely to be able to help, my next flight boards in about 10 minutes. But I'll take a quick look. [18:21:53] kk [18:22:09] <^d> Eloquence: I'm able to make edits to mw.org ok. [18:22:21] <^d> Just did https://www.mediawiki.org/w/index.php?title=User%3A%5Edemon%2Ftest&diff=779481&oldid=612875 [18:22:23] ^d: it seems to depend on the presence of templates, lemme see if I can still repro [18:22:47] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: various misc wikis to 1.22wmf16 [18:23:01] it looks very mw-related but I'm around if you folks need anything from the ops side [18:23:04] yep, try editing https://www.mediawiki.org/wiki/Project:Sandbox which just has a simple template at the top [18:25:14] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikibooks and wikisource to 1.22wmf16 [18:25:17] Logged the message, Master [18:26:07] !log upgrading mw1000-mw1009 [18:26:10] Logged the message, Mistress of the network gear. [18:26:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:38] I see messages in exception.log on fluorine, looks like "Lock wait timeout" in Database.php coming from Translate's use of the MessageGroupStats hook. But that could be unrelated, I suppose. [18:27:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 3.984 second response time [18:27:27] Reedy, we are updating other wikis while the save issue is still unresolved? [18:27:42] [09-Sep-2013 18:22:32] Catchable fatal error: Argument 2 passed to Wikibase\Api\ItemByTitleHelper::getEntityIds() must be an array, null given, called in /usr/local/apache/common-local/php-1.22wmf16/extensions/Wikibase/repo/includes/api/GetEntities.php on line 72 and defined at /usr/local/apache/common-local/php-1.22wmf16/extensions/Wikibase/repo/includes/api/ItemByTitleHelper.php on line 64 [18:28:05] You can ignore those [18:28:16] They should filter out from the recent list [18:28:35] wikibase is not on mediawiki.org [18:28:45] that error is on test wikidata [18:29:08] Reedy: are the timeouts only happening on mediawiki.org, or is it more widespread? [18:29:21] Reedy: you'll have newer tzdata once puppet ran on $all [18:29:28] I was just going to try editing wikiversity and see [18:29:38] mutante_: great [18:29:41] Thanks [18:29:44] Reedy: and no need to import anything.. it was actually an issue with the sync from Ubuntu mirror to our mirror [18:29:51] * anomie heads to the gate, good luck all [18:29:55] Even better! [18:30:16] !log repooling ssl3001/3002 and depooling ssl3003 [18:30:18] Logged the message, Master [18:30:23] PROBLEM - DPKG on mw108 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:30:33] PROBLEM - DPKG on mw10 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:30:33] PROBLEM - DPKG on mw103 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:30:43] PROBLEM - DPKG on mw105 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:30:43] PROBLEM - DPKG on mw102 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:30:43] PROBLEM - DPKG on mw101 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:30:47] that's me [18:30:48] ... [18:30:49] sorry [18:30:52] <- yea, so .. [18:30:54] Yay, tampa [18:30:59] since we looked at the tzdata thing.. [18:31:03] PROBLEM - DPKG on mw107 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:03] PROBLEM - DPKG on mw106 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:03] PROBLEM - DPKG on mw104 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:12] and it was really weird, there was the newer package but NOT the newer Packages.bz2 list [18:31:13] PROBLEM - DPKG on mw1007 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:13] PROBLEM - DPKG on mw1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:13] PROBLEM - DPKG on mw100 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:14] PROBLEM - DPKG on mw1009 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:14] PROBLEM - DPKG on mw1057 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:24] no problem editing https://en.wikiversity.org/wiki/Wikiversity:Sandbox [18:31:32] and then Alex ran the sync script manually and it fixed it, now all Apaches have upgrades to do [18:31:32] is https://ishmael.wikimedia.org/ defunct? [18:31:33] PROBLEM - DPKG on mw1071 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:33] PROBLEM - DPKG on mw1005 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:33] PROBLEM - DPKG on mw1023 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:33] PROBLEM - DPKG on mw1012 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:33] PROBLEM - DPKG on mw1016 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:33] PROBLEM - DPKG on mw1037 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:33] PROBLEM - DPKG on mw1004 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:34] PROBLEM - DPKG on mw1042 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:34] PROBLEM - DPKG on mw1003 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:35] PROBLEM - DPKG on mw1086 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:35] PROBLEM - DPKG on mw1028 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:36] PROBLEM - DPKG on mw1069 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:36] PROBLEM - DPKG on mw1019 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:37] PROBLEM - DPKG on mw1020 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:37] PROBLEM - DPKG on mw1059 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:38] PROBLEM - DPKG on mw1060 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:38] PROBLEM - DPKG on mw1052 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:39] PROBLEM - DPKG on mw1063 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:42] that's all me [18:31:43] PROBLEM - DPKG on mw1038 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:43] PROBLEM - DPKG on mw109 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:43] PROBLEM - DPKG on mw1024 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:43] PROBLEM - DPKG on mw1083 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:43] PROBLEM - DPKG on mw1026 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:43] PROBLEM - DPKG on mw1093 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:43] PROBLEM - DPKG on mw1014 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:44] PROBLEM - DPKG on mw1089 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:44] PROBLEM - DPKG on mw1036 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:45] PROBLEM - DPKG on mw1062 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:45] PROBLEM - DPKG on mw1006 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:46] PROBLEM - DPKG on mw1054 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:46] PROBLEM - DPKG on mw1095 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:47] and Leslie is running them . so Apache itself will also be upgraded [18:31:47] PROBLEM - DPKG on mw1088 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:47] PROBLEM - DPKG on mw1015 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:53] PROBLEM - DPKG on mw1010 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:53] PROBLEM - DPKG on mw1078 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:53] PROBLEM - DPKG on mw1033 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:31:59] i love breaking everything! [18:32:03] PROBLEM - DPKG on mw1047 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:03] PROBLEM - DPKG on mw1040 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:03] PROBLEM - DPKG on mw1022 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:03] PROBLEM - DPKG on mw1017 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:03] PROBLEM - DPKG on mw1048 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:03] PROBLEM - DPKG on mw1066 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:03] PROBLEM - DPKG on mw1099 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:04] PROBLEM - DPKG on mw1031 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:04] PROBLEM - DPKG on mw1029 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:05] PROBLEM - DPKG on mw1076 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:05] PROBLEM - DPKG on mw1079 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:13] PROBLEM - DPKG on mw1013 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:13] PROBLEM - DPKG on mw1008 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:13] PROBLEM - DPKG on mw1097 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:13] PROBLEM - DPKG on mw1055 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:13] PROBLEM - DPKG on mw1011 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:13] PROBLEM - DPKG on mw1084 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:13] PROBLEM - DPKG on mw1025 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:14] PROBLEM - DPKG on mw1034 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:14] PROBLEM - DPKG on mw1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:15] PROBLEM - DPKG on mw1090 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:15] PROBLEM - DPKG on mw1070 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:16] oh noessssss,the intertubes are broke! [18:32:16] PROBLEM - DPKG on mw1043 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:16] PROBLEM - DPKG on mw1072 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:17] PROBLEM - DPKG on mw1082 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:17] PROBLEM - DPKG on mw1068 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:18] PROBLEM - DPKG on mw1045 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:18] PROBLEM - DPKG on mw1067 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:19] PROBLEM - DPKG on mw1096 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:19] PROBLEM - DPKG on mw1049 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:20] PROBLEM - DPKG on mw1080 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:20] PROBLEM - DPKG on mw1039 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:21] PROBLEM - DPKG on mw1051 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:21] PROBLEM - DPKG on mw1021 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:22] PROBLEM - DPKG on mw1027 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:23] PROBLEM - DPKG on mw1035 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:23] PROBLEM - DPKG on mw1091 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:23] PROBLEM - DPKG on mw1032 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:24] PROBLEM - DPKG on mw1041 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:24] PROBLEM - DPKG on mw1074 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:33] PROBLEM - DPKG on mw1085 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:34] PROBLEM - DPKG on mw1092 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:34] PROBLEM - DPKG on mw1056 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:34] PROBLEM - DPKG on mw1044 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:34] PROBLEM - DPKG on mw1098 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:34] PROBLEM - DPKG on mw1058 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:34] PROBLEM - DPKG on mw1065 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:35] PROBLEM - DPKG on mw1030 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:35] PROBLEM - DPKG on mw1064 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:36] PROBLEM - DPKG on mw1094 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:36] PROBLEM - DPKG on mw1087 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:37] PROBLEM - DPKG on mw1073 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:37] PROBLEM - DPKG on mw1053 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:38] PROBLEM - DPKG on mw1018 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:38] RECOVERY - DPKG on mw10 is OK: All packages OK [18:32:39] RECOVERY - DPKG on mw103 is OK: All packages OK [18:32:41] it's fun to break them [18:32:44] RECOVERY - DPKG on mw109 is OK: All packages OK [18:32:44] PROBLEM - DPKG on mw1077 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:45] RECOVERY - DPKG on mw105 is OK: All packages OK [18:32:45] RECOVERY - DPKG on mw102 is OK: All packages OK [18:32:45] RECOVERY - DPKG on mw1015 is OK: All packages OK [18:32:45] PROBLEM - DPKG on mw1075 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:32:45] RECOVERY - DPKG on mw101 is OK: All packages OK [18:33:03] RECOVERY - DPKG on mw107 is OK: All packages OK [18:33:03] RECOVERY - DPKG on mw106 is OK: All packages OK [18:33:03] RECOVERY - DPKG on mw104 is OK: All packages OK [18:33:03] PROBLEM - DPKG on mw1081 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:33:11] MessageGroupStats is a good suspect, i don't have db access and ishmael looks defunct [18:33:13] RECOVERY - DPKG on mw1013 is OK: All packages OK [18:33:13] RECOVERY - DPKG on mw1008 is OK: All packages OK [18:33:13] RECOVERY - DPKG on mw1011 is OK: All packages OK [18:33:13] RECOVERY - DPKG on mw1007 is OK: All packages OK [18:33:13] RECOVERY - DPKG on mw1001 is OK: All packages OK [18:33:13] RECOVERY - DPKG on mw1002 is OK: All packages OK [18:33:13] RECOVERY - DPKG on mw100 is OK: All packages OK [18:33:14] RECOVERY - DPKG on mw1009 is OK: All packages OK [18:33:15] I can't decide if this is more annoying than the puppet out of date problems [18:33:23] RECOVERY - DPKG on mw108 is OK: All packages OK [18:33:24] ori-l: dbaccess? sql mediawikiwiki [18:33:33] RECOVERY - DPKG on mw1085 is OK: All packages OK [18:33:33] RECOVERY - DPKG on mw1012 is OK: All packages OK [18:33:33] RECOVERY - DPKG on mw1005 is OK: All packages OK [18:33:33] RECOVERY - DPKG on mw1016 is OK: All packages OK [18:33:33] RECOVERY - DPKG on mw1004 is OK: All packages OK [18:33:33] RECOVERY - DPKG on mw1003 is OK: All packages OK [18:33:43] RECOVERY - DPKG on mw1014 is OK: All packages OK [18:33:43] RECOVERY - DPKG on mw1006 is OK: All packages OK [18:33:53] RECOVERY - DPKG on mw1010 is OK: All packages OK [18:33:53] RECOVERY - DPKG on mw1078 is OK: All packages OK [18:34:03] RECOVERY - DPKG on mw1047 is OK: All packages OK [18:34:03] RECOVERY - DPKG on mw1022 is OK: All packages OK [18:34:03] RECOVERY - DPKG on mw1040 is OK: All packages OK [18:34:03] RECOVERY - DPKG on mw1017 is OK: All packages OK [18:34:03] RECOVERY - DPKG on mw1048 is OK: All packages OK [18:34:03] RECOVERY - DPKG on mw1066 is OK: All packages OK [18:34:03] RECOVERY - DPKG on mw1031 is OK: All packages OK [18:34:04] RECOVERY - DPKG on mw1079 is OK: All packages OK [18:34:04] RECOVERY - DPKG on mw1081 is OK: All packages OK [18:34:05] RECOVERY - DPKG on mw1076 is OK: All packages OK [18:34:05] RECOVERY - DPKG on mw1029 is OK: All packages OK [18:34:13] RECOVERY - DPKG on mw1097 is OK: All packages OK [18:34:13] RECOVERY - DPKG on mw1055 is OK: All packages OK [18:34:13] RECOVERY - DPKG on mw1025 is OK: All packages OK [18:34:13] RECOVERY - DPKG on mw1070 is OK: All packages OK [18:34:13] RECOVERY - DPKG on mw1084 is OK: All packages OK [18:34:13] RECOVERY - DPKG on mw1034 is OK: All packages OK [18:34:13] RECOVERY - DPKG on mw1072 is OK: All packages OK [18:34:14] RECOVERY - DPKG on mw1043 is OK: All packages OK [18:34:14] RECOVERY - DPKG on mw1090 is OK: All packages OK [18:34:15] RECOVERY - DPKG on mw1068 is OK: All packages OK [18:34:15] RECOVERY - DPKG on mw1045 is OK: All packages OK [18:34:16]