[00:12:41] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [00:30:41] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [00:48:25] 06Operations, 06DC-Ops: determine/process/document bios firmware tracking/updating policies - https://phabricator.wikimedia.org/T141128#2487936 (10Peachey88) > Should we always update the firmware at the time of receiving the servers? Personally, I probably would, If there are other pre-existing servers of th... [00:52:22] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 22 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [01:04:23] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [01:16:57] (03CR) 10Paladox: [C: 031] Gerrit: Run list_reviewer_counts cron as root [puppet] - 10https://gerrit.wikimedia.org/r/300711 (owner: 10Chad) [01:18:31] (03CR) 10Paladox: [C: 031] Gerrit: Remove googlebot from banned IPs. They ain't so bad [puppet] - 10https://gerrit.wikimedia.org/r/300692 (owner: 10Chad) [01:20:22] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [01:26:21] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [02:00:22] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [02:12:22] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [02:20:34] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 59s) [02:20:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:25:08] !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 24 02:25:08 UTC 2016 (duration 4m 34s) [02:25:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:39:50] 06Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864#2490319 (10sumanah) > @sumanah is on the Mailman team. I'll bet she has an insider perspective on what kind of challenges you'll face when migrating fr... [02:52:11] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [03:10:01] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [03:19:52] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [03:25:52] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [03:35:43] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: Puppet has 1 failures [04:01:51] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:01:22] PROBLEM - IPv6 ping to eqiad on ripe-atlas-eqiad is CRITICAL: CRITICAL - failed 20 probes of 245 (alerts on 19) - https://atlas.ripe.net/measurements/1790947/#!map [05:07:21] RECOVERY - IPv6 ping to eqiad on ripe-atlas-eqiad is OK: OK - failed 19 probes of 245 (alerts on 19) - https://atlas.ripe.net/measurements/1790947/#!map [05:30:01] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [05:35:23] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [05:35:52] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 18 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [05:37:22] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [05:47:08] 06Operations, 06Commons, 10MediaWiki-Page-deletion, 10media-storage, and 3 others: Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" - https://phabricator.wikimedia.org/T132921#2490346 (10Pokefan95) 05Resolved>03Open When does "because it is rare" became a... [06:27:51] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [06:30:51] PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:01] PROBLEM - puppet last run on db2060 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:52] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:52] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:53] PROBLEM - puppet last run on wtp2017 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:52] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 18 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [06:34:41] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:41] PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Puppet has 2 failures [06:34:52] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures [06:39:32] PROBLEM - Disk space on lithium is CRITICAL: DISK CRITICAL - free space: /srv/syslog 14783 MB (3% inode=99%) [06:55:43] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:56:32] RECOVERY - puppet last run on analytics1047 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:56:32] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:56:42] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:56:42] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:56:43] RECOVERY - puppet last run on mw2208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:43] RECOVERY - puppet last run on wtp2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:01] RECOVERY - puppet last run on db2060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:22] RECOVERY - Disk space on lithium is OK: DISK OK [07:02:02] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [07:08:01] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [07:31:02] PROBLEM - puppet last run on mc2007 is CRITICAL: CRITICAL: puppet fail [07:53:52] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 21 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [07:56:51] RECOVERY - puppet last run on mc2007 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:59:52] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 18 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [08:39:32] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 21 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [08:45:33] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 18 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [09:19:02] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: puppet fail [09:41:11] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: puppet fail [09:45:02] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [09:49:13] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [09:55:14] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 237 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [10:05:11] PROBLEM - check_mysql on fdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2901 [10:07:41] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:10:11] RECOVERY - check_mysql on fdb2001 is OK: Uptime: 3498876 Threads: 2 Questions: 48332262 Slow queries: 21226 Opens: 4176 Flush tables: 2 Open tables: 584 Queries per second avg: 13.813 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 583 [11:48:52] (03PS4) 10MarcoAurelio: Configuration changes for mk.wiktionary.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300177 (https://phabricator.wikimedia.org/T140566) [11:53:28] (03CR) 10MarcoAurelio: Initial configuration for tcy.wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:54:44] (03CR) 10Paladox: Initial configuration for tcy.wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [14:00:12] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: puppet fail [14:27:02] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [14:48:13] PROBLEM - puppet last run on relforge1001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:57:01] PROBLEM - puppet last run on mw2144 is CRITICAL: CRITICAL: puppet fail [15:03:22] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 236 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [15:09:23] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 19 probes of 236 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [15:23:53] RECOVERY - puppet last run on mw2144 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:46:54] Hi, how can you do https://gerrit.wikimedia.org/r/#/c/280456/2/portals,unified ? [15:51:49] mafk: What do you mean? [16:05:06] Reedy: hi. How to upload that subproject commit patch? I mean, how to create that? [16:05:23] it doesn't seem it's the same when I commit other changes to gerrit [16:05:32] clone operations/mediawiki-config [16:05:38] git submodule update --init --recursive [16:05:43] cd portals [16:05:49] git checkout master [16:05:50] git pull [16:05:53] cd .. [16:05:58] git commit portals [16:06:01] [16:06:38] --init too? [16:07:42] I do it as a matter of course [16:07:47] but if it's a new repo clone, it's needed [16:08:26] I already have operations/mediawiki-config cloned in my laptop [16:08:54] I want to do that for the pywikibot/i18n submodule, but I didn't knew the procedure to do so [16:09:27] ty [16:09:31] np [16:10:37] * mafk tests [16:36:20] Reedy: would git clone ssh://maurelio@gerrit.wikimedia.org:29418/pywikibot/core --recursive clone also the submodule without having to later do that --init --recursive ? [16:36:52] Honestly no idea [16:37:02] Why are you so adversed to running --init --recursive? :P [16:37:06] k, sorry for disturb [16:37:23] It's fine [16:37:30] mafk: I think you will need to do git submodule init [16:38:12] mafk: [16:38:13] --recursive, --recurse-submodules [16:38:13] After the clone is created, initialize all submodules within, using their default settings. This is equivalent to running git submodule update --init --recursive [16:38:13] immediately after the clone is finished. This option is ignored if the cloned repository does not have a worktree/checkout (i.e. if any of --no-checkout/-n, --bare, [16:38:13] or --mirror is given) [16:38:25] Platonides: looks like, according to man git clone if you've got a new enough version... [16:38:44] git clone ssh://maurelio@gerrit.wikimedia.org:29418/pywikibot/core --recurse-submodules [16:39:17] ooh [16:39:17] --separate-git-dir= [16:39:17] Instead of placing the cloned repository where it is supposed to be, place the cloned repository at the specified directory, then make a filesystem-agnostic Git [16:39:17] symbolic link to there. The result is Git repository can be separated from working tree. [16:39:41] Platonides: me estoy volviendo loco xD [16:41:56] * Platonides lanza un cubo de agua a mafk [16:42:25] * mafk encarcela a Platonides [16:42:46] that thing I know how to do it :P [16:48:58] Platonides: bueno, pues cloné localmente con --recursive y ya tengo todo, junto con el submodule: "Submodule path 'scripts/i18n': checked out '8e949fce7f77cc97f682780010c0bcc2e2de8f14'" [16:50:57] looks like it worked [16:57:04] yep, but when I do git commit, after having updated the submodule, it gives me that there's nothing to commit [16:57:07] :| [16:57:25] pity that thcipriani|afk is afk [16:57:45] I've done that submodule dance many many times [16:59:10] * Reedy clones [16:59:34] You can make gerrit track the master of the submodule and do it automatically [17:00:26] $ git submodule status [17:00:27] 06Operations, 06Commons, 10MediaWiki-Page-deletion, 10media-storage, and 3 others: Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" - https://phabricator.wikimedia.org/T132921#2490762 (10aaron) For random "can't lock", "lock wait timeout", and "deadlock" erro... [17:00:28] 8e949fce7f77cc97f682780010c0bcc2e2de8f14 scripts/i18n (8e949fc) [17:03:23] mafk: git reckons it's already tracking the master [17:04:25] hmm [17:05:19] that can't be right [17:05:26] Date: Thu Jul 21 08:27:35 2016 +0200 [17:06:53] mafk: ah [17:06:58] I think it's just done in the .gitmodules [17:07:09] it suggests it's tracking master [17:07:19] git pull && git submodule update [17:07:26] *should* be all that's necessary [17:07:38] [submodule "i18n"] [17:07:38] path = scripts/i18n [17:07:38] url = https://gerrit.wikimedia.org/r/p/pywikibot/i18n.git [17:07:38] branch = master [17:08:47] (in the root core folder) [17:09:45] mafk: feels like you're fighting a problem that doesn't exist :P [17:10:00] 9 [17:10:01] Reedy: Yep, that's for updating your local repo to master. However I want that people cloning the repo get the at least more updated translations than 8fb14... [17:10:41] thus updating the remote submodule commit shall be done? [17:10:50] You don't nee dto [17:10:55] Clone it [17:10:58] run git submodule update [17:11:06] or git pull and git submodule update [17:11:49] One a clean clone now [17:11:58] using --recurse-submodules or whatever it was [17:11:58] commit 2cfb23cbeb55440b77b07b442ca6f5ab074c00e3 [17:11:58] Author: Translation updater bot [17:11:58] Date: Thu Jul 21 08:27:35 2016 +0200 [17:12:03] 3 days ago [17:12:07] I'd say that's up to date [17:13:09] yep, yep [17:13:30] not sure when I cloned the repo for the first time I got such outdated submodule though [17:13:46] because when I updated it, 700+ files were changed [17:13:54] it was terribly outdated [17:14:18] using git pull origin master on said submodule [17:16:29] (03PS1) 10MarcoAurelio: Disabling local uploads on ms.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300758 (https://phabricator.wikimedia.org/T141227) [17:21:17] Nemo_bis: ^^ [17:22:13] (03CR) 10Nemo bis: "Thanks" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300758 (https://phabricator.wikimedia.org/T141227) (owner: 10MarcoAurelio) [17:25:23] I think http://stackoverflow.com/questions/8191299/update-a-submodule-to-the-latest-commit will help, it's what Reedy said ab initio, but maybe I did it wrong. Will try later, got to go now. Thanks. [17:57:49] 06Operations, 06Project-Admins, 05codfw-rollout, 03codfw-rollout-Jan-Mar-2016: Archive #codfw-rollout-Jan-Mar-2016 - https://phabricator.wikimedia.org/T139711#2440481 (10Luke081515) Guess we should wait till T122134 is closed. [18:20:42] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 20 probes of 236 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [18:26:43] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 18 probes of 236 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [18:41:47] (03CR) 10Yuvipanda: [C: 032] Add python (aka python3) images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/298557 (owner: 10Yuvipanda) [18:42:19] (03Merged) 10jenkins-bot: Add python (aka python3) images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/298557 (owner: 10Yuvipanda) [19:22:03] PROBLEM - puppet last run on wasat is CRITICAL: CRITICAL: puppet fail [19:33:23] 06Operations, 06Project-Admins, 05codfw-rollout, 03codfw-rollout-Jan-Mar-2016: Archive #codfw-rollout-Jan-Mar-2016 - https://phabricator.wikimedia.org/T139711#2490832 (10Danny_B) @Luke081515 That's what is actually happening. When this task was created, the status quo was as described. After my poking of @... [19:38:48] 06Operations, 06Labs, 10Labs-Infrastructure: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1753690 (10AlexMonk-WMF) Do we have any docs about the current setup with promethium (promethium.wikitextexp.eqiad.wmflabs is 10.68.16.2)? I noticed these strange production DN... [19:51:41] RECOVERY - puppet last run on wasat is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:14:11] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 1 failures [20:39:51] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [21:35:27] 06Operations, 10MediaWiki-extensions-UniversalLanguageSelector, 07I18n: MB Lateefi Fonts for Sindhi Wikipedia. - https://phabricator.wikimedia.org/T138136#2490972 (10Nemo_bis) [21:35:57] 06Operations, 10MediaWiki-extensions-UniversalLanguageSelector, 07I18n: MB Lateefi Fonts for Sindhi Wikipedia. - https://phabricator.wikimedia.org/T138136#2391310 (10Nemo_bis) [23:12:13] (03PS1) 10Yuvipanda: Add golang images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/300800 [23:12:31] (03CR) 10jenkins-bot: [V: 04-1] Add golang images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/300800 (owner: 10Yuvipanda) [23:12:37] (03PS2) 10Yuvipanda: Add golang images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/300800