[00:02:53] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0] [00:03:53] RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [00:04:25] wait, is eqiad 5xx still at PROBLEM then? [00:04:54] * greg-g opens browser [00:06:03] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0] [00:06:27] yes [00:08:14] pl proc line of course in fatallog [00:08:18] and redis [00:08:31] not a ton new [00:09:01] yeah I'm not seeing anything unusual in 5xx logs on oxygen either [00:12:55] bd808: https://tools.wmflabs.org/versions/ isn't loading [00:13:54] * bd808 pokes it with a stick [00:17:07] bd808: thanks :) [00:28:53] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0] [00:35:45] greg-g: you are now a co-maintainer of versions. `webservice restart` if it poops out again [00:58:23] PROBLEM - Disk space on thumbor1002 is CRITICAL: DISK CRITICAL - free space: / 1623 MB (3% inode=97%) [01:02:14] I'll take a look :( [01:04:33] RECOVERY - Disk space on thumbor1002 is OK: DISK OK [01:30:10] 06Operations, 06Performance-Team, 10Thumbor, 13Patch-For-Review: add thumbor to production infrastructure - https://phabricator.wikimedia.org/T139606#2753594 (10fgiunchedi) [01:30:13] 06Operations, 06Performance-Team, 10Thumbor: thumbor imagemagick filling up /tmp on thumbor1002 - https://phabricator.wikimedia.org/T145878#2753592 (10fgiunchedi) 05Resolved>03Open Reopening, I've seen a reoccurence today on thumbor1002 where `/tmp` was going out of disk space. Thumbor recovered by itsel... [01:32:13] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:43:03] 06Operations, 06Performance-Team, 10Thumbor: thumbor imagemagick filling up /tmp on thumbor1002 - https://phabricator.wikimedia.org/T145878#2753612 (10fgiunchedi) via manhole ``` root@thumbor1002:~# sudo -u thumbor socat - unix-connect:/srv/thumbor/tmp/thumbor@8827/manhole-8827 ######### ProcessID=2, Threa... [01:59:59] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:26:36] !log l10nupdate@tin scap sync-l10n completed (1.28.0-wmf.23) (duration: 09m 06s) [02:26:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:27:19] 06Operations, 06Performance-Team, 10Thumbor: thumbor imagemagick filling up /tmp on thumbor1002 - https://phabricator.wikimedia.org/T145878#2643838 (10ori) cgroups don't offer any facility for disk quotas (the blkio subsystem can only be used to throttle the rate of I/O). But if you can call `setrlimit()` (o... [02:30:17] 06Operations, 10Phabricator, 13Patch-For-Review: networking: allow ssh between iridium and phab2001 - https://phabricator.wikimedia.org/T143363#2753637 (10Dzahn) a:03Dzahn [02:30:53] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 29 02:30:52 UTC 2016 (duration 4m 16s) [02:30:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:46:41] (03PS1) 1020after4: move config for git-ssh(phabricator) to hiera [puppet] - 10https://gerrit.wikimedia.org/r/318662 [02:51:09] (03PS2) 1020after4: Move config for git-ssh(phabricator) to hiera [puppet] - 10https://gerrit.wikimedia.org/r/318662 (https://phabricator.wikimedia.org/T143363) [02:52:20] (03CR) 10Dzahn: "we should replace ferm::rule with ferm::service and then have an "srange => "(@resolve((${allowed_hosts})) @resolve((${allowed_hosts}), AA" [puppet] - 10https://gerrit.wikimedia.org/r/318662 (https://phabricator.wikimedia.org/T143363) (owner: 1020after4) [02:54:48] (03CR) 10Dzahn: "Let's just do one small thing at a time here though." [puppet] - 10https://gerrit.wikimedia.org/r/318662 (https://phabricator.wikimedia.org/T143363) (owner: 1020after4) [02:54:50] (03CR) 1020after4: "That isn't source range, it's destination" [puppet] - 10https://gerrit.wikimedia.org/r/318662 (https://phabricator.wikimedia.org/T143363) (owner: 1020after4) [02:55:34] mutante: note that it's not a srange but a drange in that patch, does ferm::service support that as well? [02:55:56] I'm really not sure what the ferm rule is like that, I didn't write it [02:56:40] it seems to be just allowing all traffic to the dest port but listing the IPs explicitly instead of just allowing to * [02:57:04] err daddr not srange [02:58:16] you are right, ferm::service comment only makes sense if we are talking about srange. i dont know that either yet [02:58:33] the specifics of that rule template [02:58:41] will have to look more after weekend [02:58:53] mutante: cool, no rush obviously [02:59:04] it's 10:00pm here so I'm done too ;) [02:59:11] have a great weekend [02:59:13] just wanted to say ..many small patches are easier to merge [02:59:19] than one larger one [02:59:22] you too, cu around [03:24:28] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 641.52 seconds [03:30:48] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 148.67 seconds [04:24:11] RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [05:53:56] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:09:36] PROBLEM - thumbor@8836 service on thumbor1002 is CRITICAL: CRITICAL - Expecting active but unit thumbor@8836 is inactive [06:13:46] RECOVERY - thumbor@8836 service on thumbor1002 is OK: OK - thumbor@8836 is active [06:23:16] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [08:06:45] !log reboot dataset1001 for new kernel [08:06:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:17:39] 07Puppet, 07Beta-Cluster-reproducible, 13Patch-For-Review: puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2753756 (10mmodell) >>! In T131946#2627561, @Joe wrote: > It would mak... [09:32:39] (03CR) 10Marostegui: "Will replication also start or only MySQL?" [puppet] - 10https://gerrit.wikimedia.org/r/318572 (owner: 10Andrew Bogott) [09:43:36] (03CR) 10Marostegui: "I like this idea, specially if at some point we might combine 10.0 (or 10.1) and 5.7." [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/318557 (https://phabricator.wikimedia.org/T149422) (owner: 10Jcrespo) [09:52:32] 06Operations, 10ops-codfw, 10DBA: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2753791 (10jcrespo) a:05Papaul>03None I would say before touching dbstore2001- transferring its raw contents to dbstore2002 and labsdb1008. If possible, stop using labsdb1008 as a test and mov... [10:00:31] (03CR) 10Jcrespo: [C: 04-1] "Yeah, they basically want the regular package experience, and not handling mysql in any way, which is a legitimate need- but maybe they sh" [puppet] - 10https://gerrit.wikimedia.org/r/318572 (owner: 10Andrew Bogott) [10:07:25] (03CR) 10Jcrespo: "Note that our package, wmf-mariadb10 and others, includes both client and server, and installs on patch mysql, mariadb, mysqldump, and mar" [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/318557 (https://phabricator.wikimedia.org/T149422) (owner: 10Jcrespo) [10:08:16] 06Operations, 10ChangeProp, 10MediaWiki-JobQueue, 06Performance-Team, and 2 others: Asynchronous processing in production: one queue to rule them all - https://phabricator.wikimedia.org/T149408#2753795 (10Joe) >>! In T149408#2753404, @bd808 wrote: >>>! In T149408#2753327, @Pchelolo wrote: >> @bd808 What ar... [10:37:11] (03CR) 10Marostegui: [C: 031] "Thanks for the explanation!. I would say go ahead and test it yes. Makes sense to me to be able to choose from different packages" [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/318557 (https://phabricator.wikimedia.org/T149422) (owner: 10Jcrespo) [11:00:44] 06Operations, 10puppet-compiler: puppet compiler claims "no change" when catalogs are actually different - https://phabricator.wikimedia.org/T149432#2753843 (10Peachey88) [11:10:04] !log performing schema change on s4 (imagelinks) T139090 [11:10:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:10:10] T139090: Deploy I2b042685 to all databases - https://phabricator.wikimedia.org/T139090 [11:43:04] 06Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T149377#2750453 (10jcrespo) Disk 4 is down, please replace (this box is likely under HP warranty): ``` $ hpssacli ctrl all show config Smart Array P420i in Slot 0 (Embedded) (sn: 001438033746C30) Gen8 ServBP 12+2 a... [13:51:49] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 10.192.48.44 on port 6479 [13:52:49] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 3081737 keys, up 220 days 6 hours - replication_delay is 0 [13:59:41] (03PS11) 10BBlack: nginx (1.11.4-1+wmf12) jessie-wikimedia; urgency=medium [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318324 [13:59:43] (03PS1) 10BBlack: Add CVE-2016-1247 minimal patch from moritz [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318682 [14:00:19] (03Abandoned) 10BBlack: disable call to BIO_set_write_buffer_size() [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318368 (owner: 10BBlack) [14:00:49] (03Abandoned) 10BBlack: disable dynamic tls records patch [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318323 (owner: 10BBlack) [14:10:16] (03PS2) 10BBlack: Add CVE-2016-1247 minimal patch from moritz [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318682 [14:10:18] (03PS12) 10BBlack: nginx (1.11.4-1+wmf12) jessie-wikimedia; urgency=medium [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318324 [14:13:48] (03PS3) 10BBlack: Add CVE-2016-1247 minimal patch from moritz [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318682 [14:13:50] (03PS13) 10BBlack: nginx (1.11.4-1+wmf12) jessie-wikimedia; urgency=medium [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318324 [14:23:52] (03CR) 10BBlack: [C: 032 V: 032] Create nginx-{full,light,extras}-dbg by hand. [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/317790 (owner: 10Ema) [14:24:00] (03CR) 10BBlack: [C: 032 V: 032] patch upstream -> master @2016-10-27 [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318321 (owner: 10BBlack) [14:24:08] (03CR) 10BBlack: [C: 032 V: 032] build: remove --with-ipv6 (removed upstream) [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318322 (owner: 10BBlack) [14:24:14] (03CR) 10BBlack: [C: 032 V: 032] update stapling proxy and ecdhe curve logging patches [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318327 (owner: 10BBlack) [14:24:19] (03CR) 10BBlack: [C: 032 V: 032] fixup debian perl buildflags patch [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318328 (owner: 10BBlack) [14:24:24] (03CR) 10BBlack: [C: 032 V: 032] update cloudflare dynamic record size patch for 1.11.5+ [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318522 (owner: 10BBlack) [14:24:33] (03CR) 10BBlack: [C: 032 V: 032] no OpenSSL readahead [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318431 (owner: 10BBlack) [14:24:44] (03CR) 10BBlack: [C: 032 V: 032] Add CVE-2016-1247 minimal patch from moritz [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318682 (owner: 10BBlack) [14:24:51] (03CR) 10BBlack: [C: 032 V: 032] nginx (1.11.4-1+wmf12) jessie-wikimedia; urgency=medium [software/nginx] (wmf-1.11.4) - 10https://gerrit.wikimedia.org/r/318324 (owner: 10BBlack) [14:25:23] !log nginx-1.11.4-1+wmf12 uploaded to carbon for jessie-wikimedia [14:25:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:30:37] (03PS5) 10Giuseppe Lavagetto: Initial debianization [debs/mcrouter] - 10https://gerrit.wikimedia.org/r/288196 (https://phabricator.wikimedia.org/T132317) [14:38:56] PROBLEM - thumbor@8813 service on thumbor1002 is CRITICAL: CRITICAL - Expecting active but unit thumbor@8813 is inactive [14:44:16] RECOVERY - thumbor@8813 service on thumbor1002 is OK: OK - thumbor@8813 is active [14:47:39] (03CR) 10Giuseppe Lavagetto: [C: 032] "Correctly built on copper" [debs/mcrouter] - 10https://gerrit.wikimedia.org/r/288196 (https://phabricator.wikimedia.org/T132317) (owner: 10Giuseppe Lavagetto) [14:55:56] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.084 second response time [15:06:36] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.105 second response time [15:09:37] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.120 second response time [15:10:31] (03PS1) 10Jcrespo: mariadb: improve accuracy of replication lag check [puppet] - 10https://gerrit.wikimedia.org/r/318685 [15:13:06] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 2.272 second response time [15:16:16] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.099 second response time [15:18:26] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.122 second response time [15:30:46] (03PS3) 10Odder: Working Class Movement Library (Salford) throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/318645 (https://phabricator.wikimedia.org/T149443) [15:31:05] (03CR) 10Dereckson: [C: 031] Working Class Movement Library (Salford) throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/318645 (https://phabricator.wikimedia.org/T149443) (owner: 10Odder) [15:34:54] 06Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Requesting access to contint for niedzielski - https://phabricator.wikimedia.org/T149233#2746004 (10Dzahn) @hashar Is this request for contint-users, contint-admins, contint-roots or a new group that s... [15:40:50] 06Operations, 10Ops-Access-Requests: Requesting access researchers, statistics-privatedata-users, and analytics-privatedata-users for Zareen - https://phabricator.wikimedia.org/T149211#2745424 (10Dzahn) Hi @Zareenf could you please create a new SSH key for production access that is different from the key used... [15:44:41] (03PS1) 10Dzahn: admin: create shell account for Zareen Farooqui [puppet] - 10https://gerrit.wikimedia.org/r/318688 (https://phabricator.wikimedia.org/T149211) [15:45:01] (03CR) 10Dzahn: [C: 04-2] "needs a new key" [puppet] - 10https://gerrit.wikimedia.org/r/318688 (https://phabricator.wikimedia.org/T149211) (owner: 10Dzahn) [16:04:07] (03PS1) 10Madhuvishy: Reorganize and append packages to requirements [wheels/paws-internal] - 10https://gerrit.wikimedia.org/r/318689 [16:06:01] (03CR) 10Madhuvishy: [C: 032 V: 032] Reorganize and append packages to requirements [wheels/paws-internal] - 10https://gerrit.wikimedia.org/r/318689 (owner: 10Madhuvishy) [16:20:46] (03PS7) 10Madhuvishy: [WIP] jupyterhub: Add module to set up Jupyterhub [puppet] - 10https://gerrit.wikimedia.org/r/288086 [16:28:26] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:30:16] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.083 second response time [16:31:17] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.129 second response time [16:56:26] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:03:06] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.099 second response time [17:04:06] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.132 second response time [17:08:26] PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.087 second response time [17:21:16] RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.112 second response time [17:33:51] (03CR) 10Alex Monk: [C: 031] Also provide imagemagick wrapper in openstack::nova::manager [puppet] - 10https://gerrit.wikimedia.org/r/316545 (https://phabricator.wikimedia.org/T145811) (owner: 10Muehlenhoff) [18:28:16] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 10.192.48.44 on port 6479 [18:29:26] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 3081374 keys, up 220 days 10 hours - replication_delay is 0 [19:19:42] (03PS1) 10Aklapper: phabricator: Fix empty "parentProject" when new project is a milestone [puppet] - 10https://gerrit.wikimedia.org/r/318699 [20:11:46] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [1000.0] [20:19:16] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] [20:23:26] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [1000.0] [20:55:37] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:10:31] 06Operations, 10Adminbot, 07RfC: [IDEA] Backup bot for morebots - https://phabricator.wikimedia.org/T148694#2754387 (10Zppix) [21:23:36] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [21:24:06] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0] [21:31:17] (03Abandoned) 10Reedy: $wgMWOAuthCentralWiki = false; [mediawiki-config] - 10https://gerrit.wikimedia.org/r/315602 (owner: 10Reedy) [21:32:36] PROBLEM - thumbor@8801 service on thumbor1002 is CRITICAL: CRITICAL - Expecting active but unit thumbor@8801 is inactive [21:38:06] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] [21:44:16] RECOVERY - thumbor@8801 service on thumbor1002 is OK: OK - thumbor@8801 is active [21:49:40] 06Operations, 10Adminbot: [IDEA] Backup bot for morebots - https://phabricator.wikimedia.org/T148694#2754515 (10Aklapper) [21:52:34] 06Operations, 10Adminbot: [IDEA] Backup bot for morebots - https://phabricator.wikimedia.org/T148694#2754516 (10Zppix) @aklapper why remove RfC tag? [22:24:55] 06Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Requesting access to contint for niedzielski - https://phabricator.wikimedia.org/T149233#2754572 (10Legoktm) It's for contint-admins, since that's what gives him the ability to deploy zuul config changes. [22:26:43] !log reedy@tin Synchronized php-1.28.0-wmf.23/includes/EditPage.php: Fix for T149473 (duration: 00m 49s) [22:26:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:26:50] T149473: Edit warnings have escaped HTML - https://phabricator.wikimedia.org/T149473 [22:27:59] 06Operations, 10Adminbot: [IDEA] Backup bot for morebots - https://phabricator.wikimedia.org/T148694#2754576 (10Aklapper) @Zppix: #RfC means "that a task needs input and discussion in the style of a request for comments". If you think that such a broader discussion (outside of this task) is needed in this case... [22:31:45] 06Operations, 10Adminbot: [IDEA] Backup bot for morebots - https://phabricator.wikimedia.org/T148694#2754582 (10Zppix) Ah, anyways i think @stashbot replaced morebots, considering it is logging to sal on wikitech according to irc chat http://ur1.ca/edq22 (see around5:26 utc -5) [22:33:19] 06Operations, 10Wikimedia-Site-requests, 13Patch-For-Review: Private wiki for Project Grants Committee - https://phabricator.wikimedia.org/T143138#2754583 (10Dereckson) As the poll doesn't give strong and clear input, and excepted if someone gives a formal opposition on this task in the next days, **project... [22:38:45] 06Operations, 10Wikimedia-Site-requests, 13Patch-For-Review: Private wiki for Project Grants Committee - https://phabricator.wikimedia.org/T143138#2754588 (10Zppix) @dereckson ack no debate from my end with that decision