[00:11:32] (03CR) 10BryanDavis: [C: 032] Guard against empty upcoming list [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/338440 (https://phabricator.wikimedia.org/T158448) (owner: 10BryanDavis) [00:15:38] (03Merged) 10jenkins-bot: Guard against empty upcoming list [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/338440 (https://phabricator.wikimedia.org/T158448) (owner: 10BryanDavis) [00:17:17] jouncebot: next [00:17:52] * bd808 waits for it.... [00:18:01] doh [00:20:22] jouncebot: status [00:20:33] jouncebot: next [00:21:07] w u so slow jouncebot ? [00:21:28] jouncebot: refresh [00:21:31] I refreshed my knowledge about deployments. [00:21:33] jouncebot: next [00:21:47] jouncebot: grounded [00:22:08] it logged the command.. but said nothing [00:22:17] * bd808 looks at the code he changed [00:23:23] ah. [00:23:26] jouncebot: now [00:23:26] No deployments scheduled for the forseeable future! [00:23:49] so next was always quiet when there was nothing in the future [00:24:02] related but different bug [00:24:04] that seems like an annoying bug [00:24:07] yeah [00:24:10] least it's better than dieing [00:30:19] (03PS1) 10BryanDavis: Say something when asked 'next' and there's nothing [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/338501 [00:37:51] (03CR) 10BryanDavis: [C: 032] Say something when asked 'next' and there's nothing [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/338501 (owner: 10BryanDavis) [00:40:27] (03PS6) 10Volans: Add debian/ directory for packaging [software/cumin] (debian) - 10https://gerrit.wikimedia.org/r/338374 (https://phabricator.wikimedia.org/T154588) [00:41:19] (03Merged) 10jenkins-bot: Say something when asked 'next' and there's nothing [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/338501 (owner: 10BryanDavis) [00:59:31] jouncebot: now [00:59:32] No deployments scheduled for the forseeable future! [00:59:36] jouncebot: next [00:59:36] No deployments scheduled for the forseeable future! [00:59:47] * bd808 declares victory [01:00:07] jouncebot: please [01:02:01] 06Operations, 10Tool-Labs-tools-Other, 13Patch-For-Review, 15User-bd808: Jouncebot: Crashes when issued a command. - https://phabricator.wikimedia.org/T158448#3037703 (10bd808) 05Open>03Resolved a:03bd808 ``` [00:59] < bd808> jouncebot: now [00:59] No deployments scheduled for the fo... [02:18:32] !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.12) (duration: 06m 41s) [02:18:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:23:52] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Feb 18 02:23:52 UTC 2017 (duration 5m 20s) [02:23:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:58:39] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:06:29] PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 61897 MB (12% inode=99%) [03:26:39] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:30:29] RECOVERY - Disk space on elastic1024 is OK: DISK OK [03:37:09] PROBLEM - puppet last run on db1078 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:51:39] PROBLEM - puppet last run on seaborgium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:04:29] PROBLEM - puppet last run on phab2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:06:09] RECOVERY - puppet last run on db1078 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:19:39] RECOVERY - puppet last run on seaborgium is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [04:32:29] RECOVERY - puppet last run on phab2001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:15:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 62100 MB (12% inode=99%) [05:28:19] PROBLEM - puppet last run on neodymium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:28:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 62239 MB (12% inode=99%) [05:39:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61747 MB (12% inode=99%) [05:56:19] RECOVERY - puppet last run on neodymium is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:26:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 60259 MB (12% inode=99%) [06:29:09] PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64128 MB (12% inode=99%) [06:32:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [06:32:39] RECOVERY - Disk space on elastic1029 is OK: DISK OK [06:34:39] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:52:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 61834 MB (12% inode=99%) [07:02:39] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:04:09] PROBLEM - puppet last run on restbase1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:08:59] PROBLEM - Start a job and verify on Precise on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/precise - 185 bytes in 0.253 second response time [07:09:10] RECOVERY - Disk space on elastic1026 is OK: DISK OK [07:13:59] RECOVERY - Start a job and verify on Precise on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 1.049 second response time [07:31:49] PROBLEM - puppet last run on cp3032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:33:09] RECOVERY - puppet last run on restbase1018 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:42:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [07:59:49] RECOVERY - puppet last run on cp3032 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:36:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 62280 MB (12% inode=99%) [08:41:30] (03CR) 10ArielGlenn: [C: 032] little tool that displays the last page id in bz2 xml content file [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/337341 (owner: 10ArielGlenn) [08:43:27] (03CR) 10ArielGlenn: [V: 032 C: 032] little tool that displays the last page id in bz2 xml content file [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/337341 (owner: 10ArielGlenn) [08:44:49] (03CR) 10ArielGlenn: [V: 032 C: 032] tiny util to get last revision id from bz2 xml content dump file [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/337863 (owner: 10ArielGlenn) [08:45:40] (03CR) 10ArielGlenn: [V: 032 C: 032] write results from getlastpageid and getlastrevid to stdout, not stderr [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338280 (owner: 10ArielGlenn) [08:47:14] (03CR) 10ArielGlenn: [V: 032 C: 032] set bfile.marker to NULL in a bunch of places [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338402 (owner: 10ArielGlenn) [08:47:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [08:47:51] (03CR) 10ArielGlenn: [V: 032 C: 032] update .gitignore with the binaries for the new utilities [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338281 (owner: 10ArielGlenn) [08:48:46] (03CR) 10ArielGlenn: [V: 032 C: 032] script to check whether page range of bz2 checkpoint file is correct [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338282 (owner: 10ArielGlenn) [08:50:53] (03CR) 10ArielGlenn: [V: 032 C: 032] combine the getlastpageid and getlastrevid utils into one [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338403 (owner: 10ArielGlenn) [08:51:58] (03CR) 10ArielGlenn: [V: 032 C: 032] remove getlastpageid and getlastrevid source files, fix up Makefile [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338404 (owner: 10ArielGlenn) [08:52:52] (03CR) 10ArielGlenn: [V: 032 C: 032] remove page_info_t now that we have id_info_t, and convert utils using it [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338405 (owner: 10ArielGlenn) [08:53:35] (03CR) 10ArielGlenn: [V: 032 C: 032] update README with docs on the new utility and script [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338406 (owner: 10ArielGlenn) [08:53:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61271 MB (12% inode=99%) [08:54:07] (03CR) 10ArielGlenn: [V: 032 C: 032] add function to dump parts of bz_info_t structure [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338407 (owner: 10ArielGlenn) [08:56:11] (03CR) 10ArielGlenn: [V: 032 C: 032] clean up some gcc warnings from unused variables and such [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338408 (owner: 10ArielGlenn) [09:02:09] PROBLEM - puppet last run on mw1286 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:04:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 61209 MB (12% inode=99%) [09:05:45] (03PS1) 10ArielGlenn: trailing whitespace cleanup [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338519 [09:08:47] (03CR) 10ArielGlenn: [V: 032 C: 032] trailing whitespace cleanup [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338519 (owner: 10ArielGlenn) [09:10:34] (03PS2) 10ArielGlenn: bump version to 0.0.6 [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338409 [09:11:40] (03CR) 10ArielGlenn: [V: 032 C: 032] bump version to 0.0.6 [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/338409 (owner: 10ArielGlenn) [09:30:09] RECOVERY - puppet last run on mw1286 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [09:33:29] PROBLEM - Disk space on elastic1021 is CRITICAL: DISK CRITICAL - free space: /srv 60843 MB (12% inode=99%) [09:34:39] RECOVERY - Disk space on elastic1029 is OK: DISK OK [09:37:29] RECOVERY - Disk space on elastic1021 is OK: DISK OK [09:49:39] PROBLEM - puppet last run on prometheus2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:46] (03PS3) 10ArielGlenn: dumps: Centeralize CSS in one file, make it wider and apply to more files [puppet] - 10https://gerrit.wikimedia.org/r/337264 (https://phabricator.wikimedia.org/T155697) (owner: 10Ladsgroup) [09:59:29] PROBLEM - Disk space on elastic1021 is CRITICAL: DISK CRITICAL - free space: /srv 62018 MB (12% inode=99%) [10:11:29] PROBLEM - Disk space on elastic1021 is CRITICAL: DISK CRITICAL - free space: /srv 61133 MB (12% inode=99%) [10:17:13] (03PS4) 10ArielGlenn: dumps: Centeralize CSS in one file, make it wider and apply to more files [puppet] - 10https://gerrit.wikimedia.org/r/337264 (https://phabricator.wikimedia.org/T155697) (owner: 10Ladsgroup) [10:17:40] RECOVERY - puppet last run on prometheus2002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [10:18:07] (03PS5) 10ArielGlenn: dumps: Centralize CSS in one file, make it wider and apply to more files [puppet] - 10https://gerrit.wikimedia.org/r/337264 (https://phabricator.wikimedia.org/T155697) (owner: 10Ladsgroup) [10:22:34] (03CR) 10ArielGlenn: [C: 032] dumps: Centralize CSS in one file, make it wider and apply to more files [puppet] - 10https://gerrit.wikimedia.org/r/337264 (https://phabricator.wikimedia.org/T155697) (owner: 10Ladsgroup) [10:27:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [10:31:29] RECOVERY - Disk space on elastic1021 is OK: DISK OK [10:48:29] PROBLEM - Disk space on elastic1021 is CRITICAL: DISK CRITICAL - free space: /srv 60599 MB (12% inode=99%) [10:53:29] RECOVERY - Disk space on elastic1021 is OK: DISK OK [10:54:29] PROBLEM - IPv6 ping to ulsfo on ripe-atlas-ulsfo is CRITICAL: CRITICAL - failed 22 probes of 269 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [10:59:29] RECOVERY - IPv6 ping to ulsfo on ripe-atlas-ulsfo is OK: OK - failed 16 probes of 269 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [11:06:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61697 MB (12% inode=99%) [11:19:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 62152 MB (12% inode=99%) [11:39:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 60802 MB (12% inode=99%) [11:43:09] PROBLEM - cassandra-a CQL 10.64.0.36:9042 on restbase-dev1001 is CRITICAL: connect to address 10.64.0.36 and port 9042: Connection refused [11:43:09] PROBLEM - cassandra-a service on restbase-dev1001 is CRITICAL: CRITICAL - Expecting active but unit cassandra-a is failed [11:43:19] PROBLEM - cassandra-a SSL 10.64.0.36:7001 on restbase-dev1001 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused [11:43:29] PROBLEM - puppet last run on restbase-dev1001 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 9 minutes ago with 3 failures. Failed resources (up to 3 shown): Service[cassandra-b],Service[cassandra-a],File[/srv/log/restbase/syslog.log] [12:07:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 60079 MB (12% inode=99%) [12:12:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [12:19:39] RECOVERY - Disk space on elastic1029 is OK: DISK OK [12:57:29] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:26:29] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:28:09] PROBLEM - puppet last run on einsteinium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:47:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 60312 MB (12% inode=99%) [13:52:40] PROBLEM - puppet last run on labvirt1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:54:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [13:58:09] RECOVERY - puppet last run on einsteinium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [14:11:39] PROBLEM - Check systemd state on labstore1005 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [14:11:59] PROBLEM - Ensure mysql credential creation for tools users is running on labstore1005 is CRITICAL: CRITICAL - Expecting active but unit maintain-dbusers is failed [14:21:39] RECOVERY - puppet last run on labvirt1005 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:36:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 61769 MB (12% inode=99%) [14:39:39] RECOVERY - Check systemd state on labstore1005 is OK: OK - running: The system is fully operational [14:39:59] RECOVERY - Ensure mysql credential creation for tools users is running on labstore1005 is OK: OK - maintain-dbusers is active [14:58:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [15:03:09] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:04:39] PROBLEM - Disk space on elastic1021 is CRITICAL: DISK CRITICAL - free space: /srv 61755 MB (12% inode=99%) [15:18:49] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:21:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61675 MB (12% inode=99%) [15:30:09] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [15:37:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 60301 MB (12% inode=99%) [15:39:59] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:47:49] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [15:50:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 61701 MB (12% inode=99%) [15:59:47] bd808 i saw the jouncebot update thanks! [16:05:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61426 MB (12% inode=99%) [16:07:51] hi! there's something wrong in operational system at dewiki, is here anybody for a small talk? [16:07:59] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:09:12] Raymond_ or yuvipanda? [16:10:06] doctaxon: I am here but I do not have access to the servers [16:10:26] and at least the watchlist works for me :) [16:10:40] Krenair ^ [16:11:18] look at the page Benutzer:Delta456/Carcassonne (Spiel)/Eigenständige Spiele [16:11:31] https://de.wikipedia.org/w/index.php?title=Benutzer:Delta456/Carcassonne_(Spiel)/Eigenst%C3%A4ndige_Spiele&action=history [16:11:48] say me please, who is the user "Unknown user" [16:12:12] he's not existing, but edits in dewiki [16:12:21] what is wrong [16:12:30] uhhh a really great and secret feature :) [16:12:49] sorry?! [16:12:57] !explain [16:13:03] sarcasm [16:13:05] sorry [16:13:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 62209 MB (12% inode=99%) [16:13:51] I guess something went wrong while importing old versions [16:14:59] Raymond, the first revision I edited not as import [16:15:01] all but today edits are related to enwiki: https://de.wikipedia.org/wiki/Spezial:Beitr%C3%A4ge/Unknown_user [16:15:04] unknown user is just a bad import edit [16:15:21] the first revision was a simple bot edit [16:15:28] nothing import [16:15:38] the import was the following [16:16:08] this was the first: 2017-02-18T15:44:04‎ TaxonBot (Diskussion | Beiträge | Sperren)‎ . . (162.779 Bytes) (+162.779)‎ . . (Bot: Vorbereitung zur Auslagerung) [16:16:27] the import can happen at anytime [16:16:35] ti doesnt have to be the first edit [16:16:52] but the first edit was emptied too [16:16:59] before import [16:17:33] no idea what happened, sorry [16:17:53] i suggest filing a task on phabricator [16:18:02] www.phabricator.wikimedia..org [16:18:08] www.phabricator.wikimedia.org [16:18:39] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 61225 MB (12% inode=99%) [16:18:39] RECOVERY - Disk space on elastic1029 is OK: DISK OK [16:19:44] hoo|away knows anything about too [16:19:57] but I don't know whether he has a solution [16:20:17] Like i said please file a task on phabricator [16:20:44] <-><- [16:20:49] woops [16:21:17] my operational tasks at phab are not working on of anybody. I don't do a phab task any more [16:22:05] my eldest phab task is dumpling a year now without reply [16:22:48] doctaxon sometimes backlogs happen and we have to priortize [16:23:09] we're mostly ran by volunteers myself included [16:23:25] i know, but that's not the point [16:24:03] i understand, so we're not always around and considering myself and Raymond_ don't have access to the servers or ability to tell what is going on filing a phab task is your best option [16:24:21] okay okay [16:25:00] doctaxon: some of my bugs are nearly 10 years old but I am still filing new bugs week for week [16:32:39] PROBLEM - Disk space on elastic1021 is CRITICAL: DISK CRITICAL - free space: /srv 61706 MB (12% inode=99%) [16:40:45] 06Operations: User:Unknown_user editing in dewiki - https://phabricator.wikimedia.org/T158474#3038239 (10doctaxon) [16:42:39] RECOVERY - Disk space on elastic1021 is OK: DISK OK [16:43:39] RECOVERY - Disk space on elastic1022 is OK: DISK OK [16:48:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 60743 MB (12% inode=99%) [17:03:45] 06Operations: User:Unknown_user editing in dewiki - https://phabricator.wikimedia.org/T158474#3038239 (10Reedy) These come from imports https://github.com/wikimedia/mediawiki/blob/master/includes/import/WikiImporter.php#L704 https://github.com/wikimedia/mediawiki/blob/master/includes/import/WikiImporter.php#L90... [17:08:39] RECOVERY - Disk space on elastic1029 is OK: DISK OK [17:29:05] !log reedy@tin Synchronized wmf-config/CommonSettings.php: Add Unknown user to reservedusernames (duration: 00m 48s) [17:29:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:36:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 60872 MB (12% inode=99%) [17:47:09] PROBLEM - Disk space on elastic1023 is CRITICAL: DISK CRITICAL - free space: /srv 60669 MB (12% inode=99%) [17:51:08] (03PS1) 10Tim Landscheidt: puppet: Remove templatedir setting [puppet] - 10https://gerrit.wikimedia.org/r/338540 (https://phabricator.wikimedia.org/T95158) [17:54:42] (03CR) 10Tim Landscheidt: "(Cf. also I372092ecde2730729dc31d474834742f1f3fa275 which only removes DO_NOT_ADD_FILES_HERE.)" [puppet] - 10https://gerrit.wikimedia.org/r/338540 (https://phabricator.wikimedia.org/T95158) (owner: 10Tim Landscheidt) [17:56:30] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [50.0] [17:56:30] PROBLEM - MediaWiki exceptions and fatals per minute on graphite2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [50.0] [17:59:17] !log reedy@tin Synchronized php-1.29.0-wmf.12/includes/DefaultSettings.php: Unknown user to reserved usernames in defaultsettings (duration: 00m 45s) [17:59:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:00:26] !log reedy@tin Synchronized wmf-config/CommonSettings.php: Rv reservedusernames addition from CS (duration: 00m 42s) [18:00:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:01:29] RECOVERY - MediaWiki exceptions and fatals per minute on graphite2001 is OK: OK: Less than 70.00% above the threshold [25.0] [18:04:29] PROBLEM - MediaWiki exceptions and fatals per minute on graphite2001 is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [50.0] [18:08:09] RECOVERY - Disk space on elastic1023 is OK: DISK OK [18:13:29] RECOVERY - MediaWiki exceptions and fatals per minute on graphite2001 is OK: OK: Less than 70.00% above the threshold [25.0] [18:14:29] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 70.00% above the threshold [25.0] [18:20:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61641 MB (12% inode=99%) [18:22:29] PROBLEM - MediaWiki exceptions and fatals per minute on graphite2001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [50.0] [18:24:29] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [50.0] [18:27:29] RECOVERY - MediaWiki exceptions and fatals per minute on graphite2001 is OK: OK: Less than 70.00% above the threshold [25.0] [18:28:10] Reedy there seems to be alot of failures ^^ [18:28:29] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 70.00% above the threshold [25.0] [18:32:39] RECOVERY - Disk space on elastic1029 is OK: DISK OK [18:34:16] paladox: Seems to be replag [18:34:20] oh [18:34:56] Or that's what logstash shows.. [18:37:28] oh [18:37:29] PROBLEM - MediaWiki exceptions and fatals per minute on graphite2001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [50.0] [18:49:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61250 MB (12% inode=99%) [18:50:59] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:52:29] RECOVERY - MediaWiki exceptions and fatals per minute on graphite2001 is OK: OK: Less than 70.00% above the threshold [25.0] [18:57:09] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:12:28] 06Operations, 10Gerrit: Decide how to support polygerrit - https://phabricator.wikimedia.org/T158479#3038496 (10Paladox) [19:12:39] PROBLEM - puppet last run on db1083 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:13:14] 06Operations, 10Gerrit: Decide how to support polygerrit - https://phabricator.wikimedia.org/T158479#3038497 (10demon) 05Open>03Resolved a:03demon Decision: let's just wait for them to fix the bug. [19:18:59] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:26:09] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:30:09] PROBLEM - CirrusSearch eqiad 95th percentile latency on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [1000.0] [19:33:09] PROBLEM - CirrusSearch eqiad 95th percentile latency on graphite2001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [1000.0] [19:40:39] RECOVERY - puppet last run on db1083 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:40:39] PROBLEM - Disk space on elastic1029 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 61449 MB (12% inode=99%) [19:49:39] RECOVERY - Disk space on elastic1029 is OK: DISK OK [20:02:09] RECOVERY - CirrusSearch eqiad 95th percentile latency on graphite2001 is OK: OK: Less than 20.00% above the threshold [500.0] [20:02:09] RECOVERY - CirrusSearch eqiad 95th percentile latency on graphite1001 is OK: OK: Less than 20.00% above the threshold [500.0] [20:06:49] PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 61543 MB (12% inode=99%) [20:14:40] MatmaRex: I am already uploading it, it's very big, give me some minutes more [20:16:49] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:20:49] RECOVERY - Disk space on elastic1024 is OK: DISK OK [20:26:38] (03PS3) 10Tim Landscheidt: Redirect wiki.toolserver.org to www.mediawiki.org [puppet] - 10https://gerrit.wikimedia.org/r/227079 (https://phabricator.wikimedia.org/T62220) (owner: 10Nemo bis) [20:27:09] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:32:06] (03CR) 10Tim Landscheidt: "Tested this successfully on relic.toolserver-legacy.eqiad.wmflabs (aka toolserver.org/stable.toolserver.org/www.toolserver.org/wiki.toolse" [puppet] - 10https://gerrit.wikimedia.org/r/227079 (https://phabricator.wikimedia.org/T62220) (owner: 10Nemo bis) [20:44:49] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:45:01] (03CR) 10Tim Landscheidt: "@BBlack: Is this patch and its compagnons in the what-is-normal topic still current? (They would need to be manully rebased in any case.)" [puppet] - 10https://gerrit.wikimedia.org/r/274089 (https://phabricator.wikimedia.org/T127387) (owner: 10BBlack) [20:45:57] (03CR) 10Tim Landscheidt: "@Mark Bergsma: Is this patch still current? (It would need to be manually rebased.)" [puppet] - 10https://gerrit.wikimedia.org/r/29805 (owner: 10Mark Bergsma) [20:56:09] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [21:06:20] MatmaRex: I added the xml to the task [21:08:29] PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 20 seconds [21:13:09] RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.023 second response time [21:25:49] PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 60856 MB (12% inode=99%) [21:27:49] PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 60227 MB (12% inode=99%) [21:43:29] PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 20 seconds [21:48:29] RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 11.681 second response time [21:48:59] PROBLEM - puppet last run on db1079 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:50:49] RECOVERY - Disk space on elastic1024 is OK: DISK OK [22:09:59] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:16:59] RECOVERY - puppet last run on db1079 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:37:59] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:49:39] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:59:09] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:59:39] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:06:49] PROBLEM - Disk space on elastic1022 is CRITICAL: DISK CRITICAL - free space: /srv 61666 MB (12% inode=99%) [23:18:39] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:20:49] RECOVERY - Disk space on elastic1022 is OK: DISK OK [23:23:09] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:27:39] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:28:09] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:29:44] (03PS2) 10Zppix: Fix typo in realm.pp [puppet] - 10https://gerrit.wikimedia.org/r/335548 (owner: 10Tim Landscheidt) [23:31:43] (03CR) 10Zppix: [C: 031] "Good catch :)" [puppet] - 10https://gerrit.wikimedia.org/r/335548 (owner: 10Tim Landscheidt) [23:38:09] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:40:49] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:52:09] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures