feat(asset): move download_link property to an asset #2029

[pdavid-cssopra]

From #1929

Task

move eodag:download_link property to an asset when it is downloadable (do not keep it has a fake-value and when product download was performed from other assets). If the download_link is not required, remove it and do not add it to assets.

• use assets_mapping, see feat(plugins): new search config for assets mapping #1711
• use download_link as key
• use archive, data as roles

When downloading a whole product:

• download only assets having archive in roles
• if no asset has archive in roles, download all assets

Note:

• do not keep assets that do not have href or order_link

Work on

• update/complete asset mapping behaviour
• isolate in separated code extra behaviour in asset config interpretation.
• update provider configuration when find mappable extra data
• update specific providers
• centralize/deduplicate source on asset management if possible

[github-actions]

Test Results

  2 files   -     2    2 suites   - 2   35s ⏱️ - 2m 38s
685 tests ±    0  646 ✅  -    38  1 💤 ±0  38 ❌ +38 
706 runs   - 2 076  667 ✅  - 2 109  1 💤  - 5  38 ❌ +38 

For more details on these failures, see this check.

Results for commit 1a81d2b. ± Comparison against base commit 798a34a.

♻️ This comment has been updated with latest results.

[eodag-bot]

Code Coverage (Ubuntu)

Details

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         251      11  95.62%   104-115, 377, 644
config.py                                      305      24  92.13%   70-72, 75, 78, 81, 85, 89, 93-95, 583-585, 707-709, 728, 736, 766-771, 773
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/collection.py                              150       8  94.67%   176, 212, 215, 322, 361, 364, 382, 385
api/core.py                                    787      52  93.39%   266, 552, 600, 643, 683, 703, 744-749, 774, 858-877, 891, 897, 1050, 1055, 1154, 1193-1194, 1290-1291, 1317, 1348-1349, 1375, 1388, 1449-1450, 1467, 1481-1482, 1558, 1573-1576, 1688, 2075, 2197, 2285-2286
api/provider.py                                381      35  90.81%   186, 190-191, 316-337, 405, 521, 525-526, 531-534, 545, 621-631, 821-822, 870, 877, 908-911, 947-948, 956-957
api/search_result.py                           181      27  85.08%   111, 123, 133, 154, 203, 220, 320, 375-378, 397, 448, 453-454, 489-505, 507, 525-526, 532
api/product/__init__.py                         18       2  88.89%   57, 59
api/product/_assets.py                         134       4  97.01%   176, 311, 314-318
api/product/_product.py                        248      20  91.94%   189, 305-306, 324-325, 346, 434, 463, 577, 597, 620-623, 632-635, 689, 755, 767
api/product/metadata_mapping.py                813      67  91.76%   126-128, 223-228, 252, 310-311, 399, 420, 472-473, 510, 531-534, 557, 569-570, 611, 634, 664-669, 734-739, 751, 759, 795-808, 1013, 1188, 1197-1201, 1218-1223, 1356, 1379, 1388, 1410, 1415, 1467, 1539, 1567, 1593, 1607, 1632, 1678, 1747, 1822
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     29       0  100.00%
api/product/drivers/generic.py                  11       0  100.00%
api/product/drivers/sentinel1.py                33       0  100.00%
api/product/drivers/sentinel2.py                33       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 22       2  90.91%   48, 55
plugins/manager.py                             173      16  90.75%   102-107, 179, 201, 219-220, 232, 271-272, 372-375, 387-388
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                          101      34  66.34%   178-180, 210, 213-272
plugins/apis/usgs.py                           184      27  85.33%   137-138, 159, 265, 299, 341-343, 348, 376-377, 382, 412-419, 430-435, 457-463
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 288-289, 339-343
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       54      39  27.78%   50-55, 68-115
plugins/crunch/filter_latest_tpl_name.py        31      20  35.48%   46-54, 64-95
plugins/crunch/filter_overlap.py                63      25  60.32%   62-65, 72-75, 81, 85, 89, 100-116, 131-157
plugins/crunch/filter_property.py               30       5  83.33%   56-61, 64-65
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        401      76  81.05%   270, 304, 353-356, 386-387, 395-399, 479-482, 522-524, 528, 559-560, 566-570, 601, 666-674, 738-833, 845-850, 888, 914, 959-961, 1013
plugins/download/base.py                       280      35  87.50%   133, 162, 309-310, 368-369, 411, 415-426, 440, 517-521, 551, 586-587, 612-621, 680, 701, 723, 731, 765
plugins/download/http.py                       564      80  85.82%   233, 275-278, 340-343, 346, 353-358, 376-391, 408, 423, 483, 518, 532, 546, 556-560, 576-581, 592, 611, 648-651, 672, 682, 689, 854, 886, 916-925, 961, 986-987, 1006-1011, 1020, 1035-1037, 1041, 1044, 1059-1060, 1144, 1196, 1238-1239, 1251, 1261, 1311-1312, 1342, 1362, 1403-1405, 1459-1460
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         318      21  93.40%   114, 118, 142-148, 205-208, 256, 288, 351, 372, 497, 547, 708-709, 735-736
plugins/search/build_search_result.py          505     102  79.80%   258-259, 295, 299, 319, 547-558, 573-575, 602-603, 698, 722, 724, 791, 799-803, 824, 834, 860, 905, 929, 957, 975-990, 1040, 1065, 1068, 1072, 1081, 1087, 1125-1146, 1187, 1214-1215, 1224-1233, 1300, 1314-1364, 1470-1471, 1515, 1524-1526, 1583, 1631-1641
plugins/search/cop_marine.py                   267      56  79.03%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 219, 243, 249, 253, 257, 268, 279-280, 293, 323-326, 332, 353, 357, 361, 365, 369-373, 379-382, 385-402, 419-422, 475-479, 484, 496
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          112      87  22.32%   99-100, 104-105, 113-170, 176-189, 197-229, 247-288
plugins/search/qssearch.py                     813      90  88.93%   530-531, 554-555, 567-571, 786-792, 850, 946, 953, 1024, 1045, 1048-1049, 1067, 1076-1077, 1104, 1115-1116, 1176, 1185, 1190-1207, 1216, 1231, 1240-1243, 1253, 1274, 1352, 1375, 1448-1449, 1455, 1545, 1652-1656, 1722, 1725, 1737, 1751-1754, 1766, 1788-1800, 1808, 1868-1874, 1881, 1935, 1958, 1963-1964, 1979, 1985, 1995, 2085, 2089, 2100, 2124, 2137, 2145-2155, 2193-2197
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            84      18  78.57%   99-127, 166-169, 182, 224
resources/__init__.py                            0       0  100.00%
resources/shp/__init__.py                        0       0  100.00%
types/__init__.py                              167      43  74.25%   58, 62, 71-75, 86-98, 126-128, 135-140, 216, 219, 257, 267-283, 288, 290, 312, 317, 325, 335
types/bbox.py                                   39      19  51.28%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                          10       0  100.00%
types/queryables.py                            112       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/stac_extensions.py                       114       1  99.12%   282
types/stac_metadata.py                         116      16  86.21%   92, 113-114, 147, 177-191, 200-207
utils/__init__.py                              608      45  92.60%   71, 206, 237-238, 247-273, 276, 291, 371-375, 450-454, 534, 574-575, 604, 982-985, 1026-1028, 1036, 1055-1056, 1085, 1103-1104, 1216, 1304, 1468, 1557, 1572, 1581, 1593, 1783
utils/cache.py                                  22       0  100.00%
utils/dates.py                                 113       4  96.46%   171, 329-331
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       0  100.00%
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    239      13  94.56%   200-203, 245, 263, 489, 534, 539-540, 581, 656, 682
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                        10303    1396  86.45%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
api/core.py                                  0      -5  +0.63%
api/search_result.py                         0      +8  -4.42%
api/product/_assets.py                     +82       0  +4.70%
api/product/metadata_mapping.py            +22     +14  -1.54%
plugins/base.py                              0      -2  +9.09%
plugins/apis/ecmwf.py                        0     +26  -25.74%
plugins/apis/usgs.py                        +2      +2  -0.93%
plugins/download/http.py                     0      +3  -0.53%
plugins/search/base.py                    +122      +3  +2.58%
plugins/search/build_search_result.py       +1     +19  -3.73%
plugins/search/cop_marine.py                -1       0  -0.07%
plugins/search/qssearch.py                 -10      -4  +0.35%
utils/__init__.py                            0      +2  -0.33%
utils/s3.py                                 -1      +1  -0.44%
TOTAL                                     +217     +67  -0.37%

Results for commit: 1a81d2b

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results