[00:10:39] (03CR) 10Dzahn: [C: 032] "for completeness, manual files on labs instance" [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/81581 (owner: 10Dzahn) [00:11:33] (03CR) 10Dzahn: [V: 032] "for completeness, manual files on labs instance" [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/81581 (owner: 10Dzahn) [00:14:53] (03PS1) 10Dzahn: up version in changelog, minor fix in control file that removes one lintian warning [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/81602 [00:15:38] (03PS4) 10coren: Tools: Allow bastions to access other hosts with HBA [operations/puppet] - 10https://gerrit.wikimedia.org/r/77144 (owner: 10Tim Landscheidt) [00:15:53] (03CR) 10coren: [C: 032] "+2 after rebase." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77144 (owner: 10Tim Landscheidt) [00:18:49] (03CR) 10coren: "The principle is sound." [operations/puppet] - 10https://gerrit.wikimedia.org/r/80332 (owner: 10Tim Landscheidt) [00:19:14] (03PS2) 10coren: Tools: Add python-oursql to exec_environ [operations/puppet] - 10https://gerrit.wikimedia.org/r/80327 (owner: 10Tim Landscheidt) [00:19:56] (03PS2) 10coren: role::pmtpa-proxy: file[] -> File[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77122 (owner: 10Hashar) [00:20:15] (03CR) 10coren: [C: 032] "Trivial package addition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/80327 (owner: 10Tim Landscheidt) [00:20:39] (03CR) 10coren: [C: 032] "re +2 after rebase." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77122 (owner: 10Hashar) [00:21:01] (03PS3) 10coren: role::pmtpa-proxy: file[] -> File[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77122 (owner: 10Hashar) [00:21:14] Oh bah -- out of order. [00:22:14] (03CR) 10coren: [C: 032] "re re +2 after rebasing again" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77122 (owner: 10Hashar) [00:27:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.862 second response time [00:30:23] !log clearing ~14M rows from db1025.pgehres.bannerimpressions and bannerimpression_raw [00:30:29] Logged the message, Master [00:32:09] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 191 seconds [00:32:30] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 215 seconds [00:32:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.688 second response time [00:35:30] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -0 seconds [00:36:09] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [00:38:30] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [00:40:09] PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 655 [00:41:39] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:45:09] RECOVERY - check_mysql on db1008 is OK: Uptime: 2953571 Threads: 1 Questions: 19141935 Slow queries: 50008 Opens: 65056 Flush tables: 2 Open tables: 64 Queries per second avg: 6.480 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [00:47:30] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [00:50:02] ^ That should really be moved from fenari [00:50:39] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:52:36] (03CR) 10Dzahn: [C: 032 V: 032] up version in changelog, minor fix in control file that removes one lintian warning [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/81602 (owner: 10Dzahn) [01:39:58] !log fundraising delete query on db1025 ran 1:10minutes -- serious lag on db1008 and db43 now expected for same [01:40:06] Logged the message, Master [01:50:07] PROBLEM - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 639 [01:51:39] (03PS7) 10Reedy: Add a sqldump script wrapper around mysqldump [operations/puppet] - 10https://gerrit.wikimedia.org/r/43844 [01:51:46] (03PS8) 10Reedy: Add a sqldump script wrapper around mysqldump [operations/puppet] - 10https://gerrit.wikimedia.org/r/43844 [01:55:07] PROBLEM - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 939 [02:00:07] PROBLEM - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1239 [02:05:07] PROBLEM - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1539 [02:10:05] PROBLEM - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1839 [02:14:03] !log LocalisationUpdate completed (1.22wmf14) at Thu Aug 29 02:14:03 UTC 2013 [02:14:10] Logged the message, Master [02:15:05] PROBLEM - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2139 [02:19:35] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 225 seconds [02:19:35] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 225 seconds [02:20:05] PROBLEM - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2439 [02:20:28] ACKNOWLEDGEMENT - check_mysql on db78 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2439 Matt Walker Known replag thing [02:25:35] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [02:25:35] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -1 seconds [02:26:16] !log LocalisationUpdate completed (1.22wmf13) at Thu Aug 29 02:26:16 UTC 2013 [02:26:22] Logged the message, Master [02:31:11] (03PS1) 10Reedy: Simplify sql slightly [operations/puppet] - 10https://gerrit.wikimedia.org/r/81616 [02:33:04] (03PS9) 10Reedy: Add a sqldump script wrapper around mysqldump [operations/puppet] - 10https://gerrit.wikimedia.org/r/43844 [02:33:25] RECOVERY - MySQL Replication Heartbeat on db1046 is OK: OK replication delay 0 seconds [02:40:47] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 29 02:40:47 UTC 2013 [02:40:53] Logged the message, Master [02:45:06] RECOVERY - check_mysql on db78 is OK: Uptime: 4278787 Threads: 1 Questions: 24636981 Slow queries: 59985 Opens: 94861 Flush tables: 2 Open tables: 64 Queries per second avg: 5.757 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [03:21:31] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 188 seconds [03:22:21] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 203 seconds [03:23:11] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [03:28:21] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 3 seconds [03:28:31] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 10 seconds [03:49:30] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 227 seconds [03:49:49] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 240 seconds [03:54:49] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 23 seconds [03:55:30] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -0 seconds [04:25:34] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 181 seconds [04:25:35] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 185 seconds [04:26:34] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [04:26:35] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -1 seconds [04:32:54] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [04:36:04] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:44] PROBLEM - DPKG on virt2 is CRITICAL: Timeout while attempting connection [04:54:35] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 181 seconds [04:54:35] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 184 seconds [05:02:34] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [05:02:35] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -1 seconds [05:09:29] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 200 seconds [05:09:39] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 200 seconds [05:12:39] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -1 seconds [05:13:29] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [05:18:47] Will this happen with the Wikimedia wikis?: https://www.eff.org/deeplinks/2013/08/pushing-perfect-forward-secrecy-important-web-privacy-protection [05:19:21] Bsadowski1: https://blog.wikimedia.org/2013/08/01/future-https-wikimedia-projects/ [05:19:49] "Consider enabling perfect forward secrecy. Enabling perfect forward secrecy is only useful if we also eliminate the threat of traffic analysis of HTTPS, which can be used to detect a user’s browsing activity, even when using HTTPS." [05:20:35] Nice :D [05:20:42] Thanks, legoktm. [05:21:04] np [05:32:29] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 184 seconds [05:32:39] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 189 seconds [05:39:34] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 240 seconds [05:40:34] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [05:40:35] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -1 seconds [05:52:34] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 219 seconds [05:52:35] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 223 seconds [05:56:35] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [05:56:35] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -1 seconds [06:10:39] (03PS1) 10Springle: relax alert levels for slave delay [operations/puppet] - 10https://gerrit.wikimedia.org/r/81627 [06:12:22] (03CR) 10Springle: [C: 032 V: 032] relax alert levels for slave delay [operations/puppet] - 10https://gerrit.wikimedia.org/r/81627 (owner: 10Springle) [06:19:06] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [06:25:06] PROBLEM - Puppet freshness on ssl1006 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:36] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 181 seconds [06:26:37] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 186 seconds [06:27:32] (03PS1) 10Ori.livneh: Set AuthName for graphite/icinga/ishmael to 'Wikimedia Labs' [operations/puppet] - 10https://gerrit.wikimedia.org/r/81630 [06:28:36] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [06:28:37] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -0 seconds [06:32:06] PROBLEM - Puppet freshness on ssl1008 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:37] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 209 seconds [06:33:37] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 212 seconds [06:37:06] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [06:38:59] Coren|Away, ksuhku reported in #wikimedia-tech he was frequently getting "An error has occurred while searching: HTTP request timed out." on Commons [06:43:36] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [06:43:36] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -0 seconds [06:46:06] PROBLEM - Puppet freshness on ssl1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:47:06] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [06:50:06] PROBLEM - Puppet freshness on ssl1003 is CRITICAL: No successful Puppet run in the last 10 hours [06:50:06] PROBLEM - Puppet freshness on ssl1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:50:06] PROBLEM - Puppet freshness on ssl4 is CRITICAL: No successful Puppet run in the last 10 hours [06:53:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [06:53:06] PROBLEM - Puppet freshness on ssl1007 is CRITICAL: No successful Puppet run in the last 10 hours [06:56:06] PROBLEM - Puppet freshness on ssl1002 is CRITICAL: No successful Puppet run in the last 10 hours [06:56:06] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:59:06] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:01:06] PROBLEM - Puppet freshness on ssl3003 is CRITICAL: No successful Puppet run in the last 10 hours [07:02:06] PROBLEM - Puppet freshness on ssl1009 is CRITICAL: No successful Puppet run in the last 10 hours [07:03:06] PROBLEM - Puppet freshness on ssl3 is CRITICAL: No successful Puppet run in the last 10 hours [07:03:06] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: No successful Puppet run in the last 10 hours [07:07:00] PROBLEM - Puppet freshness on ssl2 is CRITICAL: No successful Puppet run in the last 10 hours [07:09:29] (03PS1) 10Springle: db maintenance role for terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/81633 [07:10:49] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.106 second response time [07:12:01] (03CR) 10Springle: "Want to schedule db maintenance stuff on terbium (such as the query_digests purge and pt-kill jobs). It this the appropriate way to get pe" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81633 (owner: 10Springle) [07:24:18] lo [07:27:51] !log relaxed slave delay warnings to warn 150 crit 300 [07:27:57] Logged the message, Master [08:06:55] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:59] (03PS2) 10Hashar: adjust jobrunner/videoscaler role for beta [operations/puppet] - 10https://gerrit.wikimedia.org/r/77034 [08:18:14] (03CR) 10Hashar: "Rebased" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77034 (owner: 10Hashar) [08:22:54] (03PS3) 10Hashar: contint: generate .gitconfig files for all jenkins users [operations/puppet] - 10https://gerrit.wikimedia.org/r/75856 [08:23:22] (03PS5) 10Hashar: beta: symlink /a/common [operations/puppet] - 10https://gerrit.wikimedia.org/r/65254 [08:23:55] (03PS3) 10Hashar: deployment: abstract out MW_RSYNC_HOST [operations/puppet] - 10https://gerrit.wikimedia.org/r/72491 [08:24:05] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [08:29:38] !log apt-get upgrade on lanthanum.eqiad.wmnet [08:29:44] Logged the message, Master [08:32:59] !log apt-get upgrade on gallium.wikimedia.org [08:33:05] Logged the message, Master [08:39:13] mark: hello there , if you are around, I got a jenkins box with high CPU system usage and can't find what is causing it :( [08:52:32] !log rebooting lanthanum.eqiad.wmnet (high sys cpu) [08:52:38] Logged the message, Master [08:53:04] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 325 seconds [08:53:24] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 341 seconds [08:54:14] PROBLEM - DPKG on lanthanum is CRITICAL: Connection refused by host [08:54:35] PROBLEM - SSH on lanthanum is CRITICAL: Connection refused [08:55:24] PROBLEM - RAID on lanthanum is CRITICAL: Timeout while attempting connection [08:55:54] PROBLEM - Disk space on lanthanum is CRITICAL: Timeout while attempting connection [09:00:14] PROBLEM - Host lanthanum is DOWN: PING CRITICAL - Packet loss = 100% [09:03:25] :( [09:28:26] hashar: what's the hostname again? [09:28:38] mark : lanthanum.eqiad.wmnet [09:28:48] mark: I have rebooted it though and that killed the box :( sorry [09:29:09] will write down a hundred time "I shall not reboot a box" [09:32:08] RECOVERY - Host lanthanum is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [09:32:15] !log Powercycled lanthanum [09:32:17] RECOVERY - RAID on lanthanum is OK: OK: State is Optimal, checked 1 logical device(s) [09:32:20] Logged the message, Master [09:32:27] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 301 seconds [09:32:37] RECOVERY - SSH on lanthanum is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [09:32:47] RECOVERY - Disk space on lanthanum is OK: DISK OK [09:34:06] (03PS1) 10Hashar: contint: install python-virtualenv [operations/puppet] - 10https://gerrit.wikimedia.org/r/81649 [09:35:09] mark: 100% idle. Thank you! [09:35:39] how does python-virtualenv NOT violate our policy of "no third party repos"? :) [09:36:07] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 333 seconds [09:36:27] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 355 seconds [09:39:06] !log killing long-running research transaction on db1047, and setting innodb_max_purge_lag to reduce history list length [09:39:12] Logged the message, Master [09:39:29] (03CR) 10Mark Bergsma: [C: 04-2] "I'd like to see really strong argumentation for this, because as has often been discussed, python-virtualenv has no place for installing/m" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81649 (owner: 10Hashar) [09:43:03] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [09:43:26] (03CR) 10Hashar: "The use case is for pywikibot which provide a test suite via:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81649 (owner: 10Hashar) [09:43:35] mark: yeah venv is a hack, I hav ereplied [09:43:50] the goal is to be able to install a python module dependency which is provided by the pywikibot project [09:44:01] (when running the jenkins jobs, not actually deployment) [09:44:06] yeah [09:44:10] that's why I asked for argumentation [09:44:15] that might be a valid use ;) [09:44:54] we could apparently install the dep directly in the workspace with python setup.py test --root . [09:45:08] but then I have no clue how to get python to look up the dependency in the created dir [09:45:43] python has a search path for that [09:45:51] you can set it in a number of ways, including an environment var [09:45:56] and a cli argument I think [09:46:19] >>> import sys [09:46:19] hashar: SyntaxError: Unexpected reserved word [09:46:19] >>> print sys.path [09:46:20] hashar: SyntaxError: Unexpected identifier [09:46:30] er [09:46:31] that's broken [09:46:46] it expects javascript :) [09:46:55] >>> foo = 3 + 2; [09:46:55] hashar: (number) 5 [09:48:11] so can we have venv ? :] [10:04:41] (03PS1) 10Mark Bergsma: Unbreak Content-Length streaming check [operations/puppet] - 10https://gerrit.wikimedia.org/r/81651 [10:06:09] (03PS2) 10Mark Bergsma: Unbreak Content-Length streaming check [operations/puppet] - 10https://gerrit.wikimedia.org/r/81651 [10:07:02] (03CR) 10Mark Bergsma: [C: 032] Unbreak Content-Length streaming check [operations/puppet] - 10https://gerrit.wikimedia.org/r/81651 (owner: 10Mark Bergsma) [10:14:25] hashar: [10:14:26] >>> import sys [10:14:26] >>> print sys.path [10:14:26] ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/pymodules/python2.7'] [10:14:27] mark: SyntaxError: Unexpected reserved word [10:14:27] mark: SyntaxError: Unexpected identifier [10:14:29] ? :) [10:14:45] what is ecmabot? [10:15:18] mark: something setup by Krinkle I think that let you execute javascript snippet [10:15:22] to help people I guess [10:15:28] I am not sure why it is in this channel though [10:15:32] me neither [10:15:56] anyway, adjusting sys.path didn't work? [10:19:44] :q [10:19:44] ls [10:19:54] gah. [10:20:18] in my defense, my terminal crashed very suddenly [10:20:44] Password: [10:21:28] * ori-l squints suspiciously [10:31:43] mark: so yeah the sys.path does not contain a local directory :( [10:31:54] mark: venv would do that for us I guess. [10:31:57] add it? :) [10:32:42] i think '' is the local dir btw [10:34:33] but python setup.py test --root . , would end up installing the module under ./Library/Python/2.7/site-packages/ (on my mac) [10:34:48] I don't think it will find it there :/ [10:34:50] (03Abandoned) 10Mark Bergsma: Revert "add missing uploadlb6 ips" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81474 (owner: 10Mark Bergsma) [10:36:20] put it in PYTHONPATH? [10:49:37] yup that seems to yield something [10:50:10] though it lookup for a path named 'httplib' [10:50:14] when the module is named httplib2 [10:50:15] grbblb [10:57:56] i believe the 'site' module sets up the path for site packages like that [10:58:00] have a look at its code [10:59:22] (03PS1) 10Mark Bergsma: Increase thread count, dependent on CPU core count [operations/puppet] - 10https://gerrit.wikimedia.org/r/81658 [11:00:51] ahh [11:01:26] mark: so distribute/setuptools ends up calling easy_install which does not look up in PYTHONPATH :-] [11:02:24] PROBLEM - Apache HTTP on mw1158 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:34] (03CR) 10Mark Bergsma: [C: 032] Increase thread count, dependent on CPU core count [operations/puppet] - 10https://gerrit.wikimedia.org/r/81658 (owner: 10Mark Bergsma) [11:02:44] PROBLEM - Apache HTTP on mw1155 is CRITICAL: Connection timed out [11:02:44] PROBLEM - Apache HTTP on mw1159 is CRITICAL: Connection timed out [11:02:44] PROBLEM - Apache HTTP on mw1156 is CRITICAL: Connection timed out [11:02:53] hmm [11:02:54] PROBLEM - Apache HTTP on mw1154 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:54] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: Connection timed out [11:03:14] PROBLEM - Apache HTTP on mw1157 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:04:23] swift has a spike in traffic [11:04:24] PROBLEM - Apache HTTP on mw1160 is CRITICAL: Connection timed out [11:04:45] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 61007 bytes in 0.228 second response time [11:07:20] RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.062 second response time [11:07:20] RECOVERY - Apache HTTP on mw1158 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.069 second response time [11:07:40] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 2.470 second response time [11:07:40] RECOVERY - Apache HTTP on mw1156 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.311 second response time [11:08:30] RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.062 second response time [11:08:41] RECOVERY - Apache HTTP on mw1154 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.064 second response time [11:09:10] RECOVERY - Apache HTTP on mw1157 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.071 second response time [11:09:38] lunch time bbl [11:14:18] hey [11:14:25] I'm not home, I got some rendering pages [11:14:30] everything okay? [11:14:42] seems to have recovered [11:14:46] swift has a large bump in traffic [11:15:02] had, now [11:15:49] huh [11:15:54] but req/s are about the same [11:16:40] perhaps because of my varnish large files fix? [11:16:43] although I don't really see how [11:17:01] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&tab=v&vn=Media+storage is interesting [11:18:04] the traffic spike is 100% though, wow [11:18:26] I'm on the webchat, no irc client or SSH [11:19:16] you need quassel ;-) [11:20:23] hm, maybe imagescalers being DoSed by very large files [11:20:38] ls -la /tmp on imagescalers might be helpful [11:20:44] anyway, seems okay now [11:21:03] yep [11:21:55] ok, I'll be home in an hour or two [11:21:59] ttyl [11:22:48] Could not mmap SILO (/srv/sdb3/varnish.main2) at target 1163673600, was mapped at 3615244288 instead [11:22:50] so annoying [13:24:04] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [13:26:44] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 348 seconds [13:26:45] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 347 seconds [13:37:39] (03CR) 10Mark Bergsma: [C: 032] Let log.data.copy default to true. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/81388 (owner: 10Edenhill) [13:37:47] (03CR) 10Mark Bergsma: [V: 032] Let log.data.copy default to true. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/81388 (owner: 10Edenhill) [13:37:54] is cirrussearch dead [13:40:45] (03CR) 10Mark Bergsma: [C: 032 V: 032] Added optional %{}t formatting to %t [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/81389 (owner: 10Edenhill) [13:41:23] (03PS1) 10Petr Onderka: Performed some of Platonides' suggestions [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/81662 [13:45:16] (03CR) 10Mark Bergsma: [C: 032 V: 032] Added optional secondary formatter: format.key [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/81390 (owner: 10Edenhill) [13:45:43] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [13:45:44] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -1 seconds [13:46:36] Nemo_bis, on mw.org or where? [13:59:48] yo akosiaris1 [14:00:27] somehow we lost this in the debian branch: [14:00:27] https://gerrit.wikimedia.org/r/#/c/79927/ [14:00:44] can I merge or cherry pick it in? [14:01:43] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 322 seconds [14:01:43] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 321 seconds [14:04:43] (03CR) 10Petr Onderka: [C: 032 V: 032] Performed some of Platonides' suggestions [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/81662 (owner: 10Petr Onderka) [14:05:51] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 146 seconds [14:05:51] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay 138 seconds [14:08:29] ottomata: yes you can. I do wonder though how i lost it.... [14:09:23] k danke [14:11:06] for some reason it was not in my original branch... that is why... [14:12:55] hmmm i fetched but did not merge this branch. I only did it for 0.8/0.7.* and trunk [14:13:10] ahh [14:13:10] yeah [14:13:12] (03PS1) 10Petr Onderka: Fix and refactoring of previous commit [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/81664 [14:13:35] (03CR) 10Petr Onderka: [C: 032 V: 032] Fix and refactoring of previous commit [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/81664 (owner: 10Petr Onderka) [14:15:09] (03PS1) 10Ottomata: Installing kafka-mirror init.d and default scripts. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/81665 [14:17:31] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 320 seconds [14:17:41] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 328 seconds [14:25:04] (03PS1) 10Ottomata: analytics1003 and analytics1011 are ganglia aggregators [operations/puppet] - 10https://gerrit.wikimedia.org/r/81666 [14:25:23] (03CR) 10jenkins-bot: [V: 04-1] analytics1003 and analytics1011 are ganglia aggregators [operations/puppet] - 10https://gerrit.wikimedia.org/r/81666 (owner: 10Ottomata) [14:25:31] (03PS1) 10Hashar: missing A entries for some LVS ethernet interfaces [operations/dns] - 10https://gerrit.wikimedia.org/r/81667 [14:26:02] (03PS2) 10Ottomata: analytics1003 and analytics1011 are ganglia aggregators [operations/puppet] - 10https://gerrit.wikimedia.org/r/81666 [14:26:52] (03CR) 10Ottomata: [C: 032 V: 032] analytics1003 and analytics1011 are ganglia aggregators [operations/puppet] - 10https://gerrit.wikimedia.org/r/81666 (owner: 10Ottomata) [14:33:51] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:14] andre__: I was trying on mw.o [14:35:58] (03PS1) 10Ottomata: Including role::analytics on Ciscos [operations/puppet] - 10https://gerrit.wikimedia.org/r/81668 [14:36:47] (03CR) 10Ottomata: [C: 032 V: 032] Including role::analytics on Ciscos [operations/puppet] - 10https://gerrit.wikimedia.org/r/81668 (owner: 10Ottomata) [14:36:51] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [14:37:24] (03PS1) 10Hashar: install certificate for stats.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/81669 [14:44:38] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 142 seconds [14:45:27] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -1 seconds [14:46:32] could someone please an apache configuration change for the contint server please? https://gerrit.wikimedia.org/r/#/c/71968/ worked out on my labs instance [14:46:32] !log restarting gmetad on nickel to see if I can get it to pick up reinstalled analytics boxes [14:46:38] Logged the message, Master [14:51:07] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 313 seconds [14:51:37] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 323 seconds [14:53:37] PROBLEM - MySQL Slave Delay on db43 is CRITICAL: CRIT replication delay 354 seconds [14:58:27] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 310 seconds [14:58:27] PROBLEM - DPKG on ms-be1009 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:58:37] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 321 seconds [15:02:07] (03CR) 10Ottomata: [C: 032 V: 032] Installing kafka-mirror init.d and default scripts. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/81665 (owner: 10Ottomata) [15:02:27] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -1 seconds [15:02:37] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [15:05:37] RECOVERY - MySQL Slave Delay on db43 is OK: OK replication delay 0 seconds [15:06:04] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -0 seconds [15:23:24] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 341 seconds [15:23:34] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 352 seconds [15:24:10] (03PS1) 10Akosiaris: Adding various servers to backup [operations/puppet] - 10https://gerrit.wikimedia.org/r/81674 [15:25:07] (03PS2) 10coren: maintain-replicas: the script that does the magic [operations/software] - 10https://gerrit.wikimedia.org/r/81593 [15:26:00] (03CR) 10coren: [C: 032 V: 032] "This is merely documenting status quo" [operations/software] - 10https://gerrit.wikimedia.org/r/81593 (owner: 10coren) [15:26:34] (03CR) 10Akosiaris: [C: 032] Adding various servers to backup [operations/puppet] - 10https://gerrit.wikimedia.org/r/81674 (owner: 10Akosiaris) [15:30:50] (03PS1) 10Akosiaris: require nrpe package before collecting [operations/puppet] - 10https://gerrit.wikimedia.org/r/81676 [15:38:27] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -0 seconds [15:38:37] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [15:43:25] !log Recreating Solr index [15:43:31] Logged the message, Master [15:52:38] (03CR) 10coren: [C: 031] "Works for me, but this will have the consequence that everyone will have to authenticate anew and will lose stored passwords in their UA i" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81630 (owner: 10Ori.livneh) [15:53:25] (03PS1) 10Nemo bis: Enable CleanChanges extension on Meta-Wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81678 [16:03:27] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 302 seconds [16:03:37] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 313 seconds [16:16:32] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay -1 seconds [16:16:42] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [16:19:52] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:52] PROBLEM - Puppet freshness on ssl1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:52] PROBLEM - Puppet freshness on ssl1008 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:09] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:46:09] PROBLEM - Puppet freshness on ssl1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:09] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:09] PROBLEM - Puppet freshness on ssl1003 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:09] PROBLEM - Puppet freshness on ssl1005 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:09] PROBLEM - Puppet freshness on ssl4 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:09] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:09] PROBLEM - Puppet freshness on ssl1007 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:09] PROBLEM - Puppet freshness on ssl1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:09] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:16] !log reedy synchronized php-1.22wmf15/ [16:56:21] Logged the message, Master [16:56:53] !log reedy synchronized docroot and w [16:57:00] Logged the message, Master [16:59:09] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:09] PROBLEM - Puppet freshness on ssl3003 is CRITICAL: No successful Puppet run in the last 10 hours [17:02:09] PROBLEM - Puppet freshness on ssl1009 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:09] PROBLEM - Puppet freshness on ssl3 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:09] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:40] PROBLEM - Puppet freshness on ssl2 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:02] !log csteipp synchronized php-1.22wmf14/includes/api 'bug49090' [17:12:09] Logged the message, Master [17:14:10] !log csteipp synchronized php-1.22wmf14/includes/resourceloader/ 'bug46332' [17:14:16] Logged the message, Master [17:17:38] (03CR) 10Dzahn: [C: 031] "RT #5679 - thanks for fixing this, :) +1 afaik and since it's already been tested in labs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81676 (owner: 10Akosiaris) [17:20:07] (03CR) 10Dzahn: [C: 031] "RT #4749 - installs ssl cert where $site_name = "stats.wikimedia.org"" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81669 (owner: 10Hashar) [17:22:57] (03PS2) 10Dzahn: install certificate for stats.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/81669 (owner: 10Hashar) [17:24:19] (03CR) 10Dzahn: [C: 032] "looks good, cert and key are in place" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81669 (owner: 10Hashar) [17:27:31] (03CR) 10Dzahn: "notice: /Stage[main]/Misc::Statistics::Sites::Stats/Install_certificate[stats.wikimedia.org]/Create_chained_cert[stats.wikimedia.org]/Exec" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81669 (owner: 10Hashar) [17:38:47] !log reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf15 and build l10n cache [17:38:54] Logged the message, Master [17:43:05] (03PS1) 10Dzahn: enable ssl on apache site stats.wikimedia.org on stat1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/81691 [17:48:23] (03PS1) 10RobH: new targets for smokeping [operations/puppet] - 10https://gerrit.wikimedia.org/r/81693 [17:49:40] mw1089: Copying to mw1089 from mw1010.eqiad.wmnet...cannot delete non-empty directory: php-1.22wmf2/.git/modules/extensions/WikiLove [17:50:22] anybody know if mr. liambotis is around? i don't want to pull him away from nonwork-life unless necessary. think he gets beeped when his handle is used. [17:50:29] I am [17:50:30] ;) [17:50:46] thanks for the consideration :) [17:50:47] do you have a keyword on your name, too? :) [17:51:16] just first name [17:51:26] and nickname, obviously [17:52:01] so it was purely out of luck that I noticed this so quickly :) [17:53:03] what's up? [17:53:21] alright, good deal, i found the right way to not wake you up in the middle of the night! [17:53:32] dr0ptp4kt is anxious to have etsy's statsd in apt [17:53:35] that's my guess, at least [17:53:40] i could be wrong [17:54:04] I'm not up to the point of having irssi wake me up [17:54:08] but i'll think about it [17:54:09] paravoid, the dual factor auth message kicked in when attempting to login with ops@wikimedia.org. who's going to get the sms/call? [17:54:22] ori-l, ha [17:54:25] (03PS2) 10Dzahn: enable ssl on apache site stats.wikimedia.org on stat1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/81691 [17:54:42] (03CR) 10RobH: [C: 032] new targets for smokeping [operations/puppet] - 10https://gerrit.wikimedia.org/r/81693 (owner: 10RobH) [17:54:44] ori-l: saw the request, it's supposed to be my "relaxation" week/week off though, so I thought I wouldn't deal with it this week :) [17:54:48] (03CR) 10jenkins-bot: [V: 04-1] enable ssl on apache site stats.wikimedia.org on stat1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/81691 (owner: 10Dzahn) [17:54:51] dr0ptp4kt: you mean noc@? [17:54:52] paravoid, yeah, don't hook up the subdermal wires to your irssi client [17:54:59] paravoid, yes, sorry, noc@ [17:55:06] noc@ has two-auth? [17:55:08] wtf? [17:55:44] (03PS3) 10Dzahn: enable ssl on apache site stats.wikimedia.org on stat1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/81691 [17:55:49] paravoid, LOL, well, i suppose it's for the best. it was extra funny because it was based on geolocation, while i was originating from non-splittunneled vpn! [17:56:12] no idea who set it up like that, maybe try it and see who reacts? :) [17:56:17] this was set up before I was hired, so I wouldn't know [17:56:32] paravoid, this could be funny. may have to do this about ten times. i kid! [17:56:34] alternatively, if there's a google account associated with noc@, maybe OIT can reset the 2-factor auth [17:57:03] ok, hang on a sec here. loosk like i can specify the number. but i'm guessing if it doesn't match it's going to scream at me. [17:57:28] does anyone know if we use instant commons in production? [17:57:41] No [17:57:49] that's no, we don't [17:57:55] not no, no one knows [17:58:15] <^d> Who knows? [17:58:42] like the holy roman empire, instant commons is neither instant nor common [17:58:49] manybubbles: https://noc.wikimedia.org/conf/highlight.php?file=filebackend.php [17:59:17] Beau knows [17:59:48] sorry, Bo Knows [17:59:52] <^d> ori-l: It's common after we made it easy to turn on :p [17:59:56] https://en.wikipedia.org/wiki/Bo_Knows [18:00:00] (for the youngins) [18:00:24] paravoid, false alarm. i was able to use my (arbitrary) cellphone. ok, i think it was just trying to rate limit access. i'm guessing there have been some automated attack attempts on the id or something. regarding that, i guess should probably change the password on the account and also leave a note telling people to NOT turn on the 2-factor auth. can you think of any reason *not* to make the password stronger? [18:00:44] paravoid, btw, when prompted to enable 2-factor i told it no. [18:01:06] ok, so next question then, is what we use in production close enough that I can test using instant commons? it looks like it uses the wgFileBackends like instant commons. [18:01:44] dr0ptp4kt: yes, please make it stronger [18:01:57] !log reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf15 and build l10n cache [18:02:03] Logged the message, Master [18:02:11] <^d> manybubbles: They all use the same FileRepo configuration stuff. [18:02:13] dr0ptp4kt: and document the cellphone bit; noc@wm.org being bruteforced wouldn't surprise me [18:02:28] <^d> InstantCommons is just a fun wrapper around ForeignAPIRepo. At WMF we use ForeignDBViaLBRepo [18:02:34] <^d> Or some silly name like that :) [18:02:35] That was apparently well timed [18:02:46] paravoid, ok, will do. will check on a few other security/privacy things for obvious enhancements, too. yeah, we love the kiddies, but not so much to allow arbitrary login :) [18:02:50] ^d: I imagine I'll be ok. I mean, I have to implement instant commons anyway. I'll look some more. [18:04:26] paravoid, can you create an @wikimedia.org email alias with a small distribution of trusted people? i'd like to set a recovery email address for noc@wikimedia.org, but need to make sure it goes to a small set of people. [18:04:27] <^d> manybubbles: So, I got angry at PHP for not supporting HTML5 parsing. [18:04:43] ^d: fair [18:04:58] <^d> So I started looking at https://github.com/google/gumbo-parser :p [18:05:02] <^d> It's in C. [18:06:20] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf15 [18:06:25] Logged the message, Master [18:07:08] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:29] paravoid, although i guess maybe a recovery email address isn't necessary if the Google Apps administator(s) can do password changes. still, if the Google Apps administrator(s) isn't (aren't) available, i wonder if a recovery email address would make sense. i'm torn. [18:09:44] I don't think that's necessary tbh [18:09:53] the password will be well documented [18:10:04] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf14 [18:10:09] Logged the message, Master [18:10:17] (03PS1) 10Reedy: testwiki, test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81694 [18:10:18] and worse case, in the event of a big emergency we can call OIT [18:10:18] (03PS1) 10Reedy: Add wmf15 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81695 [18:10:19] (03PS1) 10Reedy: Move php symlink to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81696 [18:10:20] (03PS1) 10Reedy: Wikipedias to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81697 [18:10:29] ^d: there are several good HTML5 parsers for node ;) [18:10:53] (03CR) 10Reedy: [C: 032] testwiki, test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81694 (owner: 10Reedy) [18:11:02] (03CR) 10Reedy: [C: 032] Add wmf15 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81695 (owner: 10Reedy) [18:11:14] (03CR) 10Reedy: [C: 032] Move php symlink to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81696 (owner: 10Reedy) [18:11:21] (03CR) 10Reedy: [C: 032] Wikipedias to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81697 (owner: 10Reedy) [18:11:36] ^d: parsoid can also provide XHTML output in case that helps you [18:12:18] gumbo seems to be cool, I was looking at it the day it was released [18:12:29] I'd expect it to get bindings and become pretty standard soon [18:13:50] (03CR) 10Reedy: [V: 032] Add wmf15 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81695 (owner: 10Reedy) [18:15:01] paravoid: it does not build a standard DOM sadly [18:15:26] It doesn't seem to have things like getElementById and getElementsByTagName either [18:15:29] (03PS1) 10Petr Onderka: Changed handling of undeleted pages in diff dumps [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/81699 [18:15:35] (03CR) 10jenkins-bot: [V: 04-1] testwiki, test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81694 (owner: 10Reedy) [18:15:51] The examples show you have to implement a DFS yourself to find them all [18:15:55] HTML parsing is usually the smaller problem [18:15:59] paravoid, cool man. thx [18:16:23] dr0ptp4kt: np :) [18:16:25] there are basically no good stand-alone and efficient HTML DOM implementations [18:16:32] dr0ptp4kt: I had fun chasing down the maxmind account the other day [18:16:42] paravoid, that sounds dreadful [18:16:43] lots of hours of digging around, asking all kinds of people [18:16:48] (03CR) 10Reedy: [V: 032] testwiki, test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81694 (owner: 10Reedy) [18:16:56] ending up in a mail account from an ex-employee who has left the organization more than a year ago [18:17:04] (03CR) 10Reedy: [V: 032] Move php symlink to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81696 (owner: 10Reedy) [18:17:15] (03CR) 10Reedy: [V: 032] Wikipedias to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81697 (owner: 10Reedy) [18:17:18] paravoid, hope it didn't involve a fax machine. [18:17:28] plenty of XML DOM implementations, and a few good HTML DOM implementations that are completely intertwined with a huge browser code base [18:17:56] no, escalated to ken and he's following it up to maxmind support now [18:18:01] (03PS3) 10Reedy: Install Education Program extension on pt.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81511 [18:18:03] <^d> gwicke: Problem with node is I'd have to use node ;-) [18:18:15] ^d: that's a feature ;) [18:18:18] paravoid, hope he likes he using fax machines ;) [18:18:25] (03CR) 10Reedy: [C: 032] Install Education Program extension on pt.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81511 (owner: 10Reedy) [18:18:42] (03Merged) 10jenkins-bot: Install Education Program extension on pt.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81511 (owner: 10Reedy) [18:18:57] <^d> gwicke: I'd rather write a php extension that uses gumbo :) [18:19:01] dr0ptp4kt: so using noc@ for this sounded like a good idea :) [18:19:24] ^d: I'd concentrate on the DOM first, then figure out how to build it [18:19:36] as that determines your PHP interfaces much more than the parser you use [18:19:50] (03CR) 10Reedy: [C: 04-1] "Needs adding to wmf-config/extension-list" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81678 (owner: 10Nemo bis) [18:19:54] <^d> gwicke: Well my problem is pretty straightforward :) [18:20:03] paravoid, yeah, got it. i'll get stuff ironed out. Google Webmaster Tools is actually useful, so best to have it documented. [18:20:08] <^d> I want to strip