Innovation co-development for viticulture and enology: Novel tele-detection web-service fuses vineyard data

.


Introduction
Understanding intra-vineyard variability is essential to putting efficient precision viticulture strategies into reality, particularly in the Mediterranean region where the land use patterns are widely fragmented and the vineyards exhibit significant heterogeneity due to the diversity in the soil, morphology, and microclimate.Studies comparing various remote sensing platforms, showed that varied resolutions produce findings that are comparable in vineyards with strong vegetation gradients and substantial vegetation clusters.Low resolution photos cannot accurately depict intra-vineyard variability and its patterns in vineyards with modest vegetation gradients and strong vegetation patchiness, on the other hand.Research also highlighted the difficulty in identifying canopy and inter-rows in low-resolution photos, which restricts the use of these platforms in the case of variable rates praying.This is due to the distinctive crop structure of vineyards.The cost study demonstrates that, outside technical considerations, there is an economic break-even between UAV and the other platforms between 5 and 50 ha of coverage, and that, above this point, satellite remote sensing is still competitive with airplane remote sensing [1].
We shall refer to "remote sensing" in its wider scope encompassing not only satellite source dimagery, but also those obtained with aircraft and unmanned aerial vehicles (UAV).
The key variables when comparing data sources for vineyard management applications are: ground sampling distance (or spatial resolution), the bands acquired and cost.Free satellite multispectral imagery such as that provided by the European Union Copernicus Programme (through Sentinel-2A and 2Bsatellites) starts at 10 m/pixel and has more bands than the typical UAV image.Paid satellite imagery starts at ~30 cm/pixel, requires tasking a satellite and implies a minimum area, which drives up costs for small areas such as vineyards.Cost is a relevant variable not only for a cost-benefit analysis in the context of user adoption, but is also closely related with temporal resolution.The cost of each drone image limits the number of images through the season, whereas free satellite data is available every 5 days.In general terms and taking into account costs, UAV imagery has an advantage in spatial resolution, while free satellite imagery from Copernicus has an advantage in temporal resolution (i.e.revisit times).
Studies comparing drone and free satellite imagery from Sentinel-2 suggest that satellite imagery cannot be directly used to reliably describe vineyard variability due to the "contribution of the inter-row surfaces" [2].Good correlations were detected between unfiltered Sentinel-2 and UAV NDVI [3], which only means that the contribution of the inter-row surfaces is similarly represented in both images.In vineyards without interrow vegetation, in plots larger than 0.5 ha, and when border pixels were removed, Sentinel-2 can reliably be used to map variability [4].
Current commercial approaches are limited by either using Sentinel-2 or UAV, but typically not both.The solution presented here uses multispectral UAV imagery derived data for decision support together with Sentinel-2 data for continuous monitoring of trends and potential anomalies during the growing season.

Study area
The study area comprises twenty vineyard holdings located in six Portuguese wine regions (Vinho Verde, Douro, Dão, Bairrada, Bucelas, and Alentejo), totalling 1018 hectares of vineyard area and 1811 hectares of total area.The vineyards are planted with more than 100 different grape varieties and are typical of very different situations.Near 600 hectares are located in mountain areas, displaying steep slopes, high slope aspect variation, and diverse vineyard structural systems.Overall, the 20 properties present a very rich and diverse sample in terms of characteristics, structures, and complexities.

UAV data
MAPP.it uses very high-resolution (5-10 cm/pixel) multispectral imaging gathered at low altitude (typically <120 m above-ground level -AGL), using micro-UAV platforms and COTS cameras.Typical platforms used are Spin.Works' own S20 fixed-wing UAV, Wingtra WingtraOne or several DJI multi-rotor UAVs, the latter typically used for coverage of smaller areas.MAPP.it processes Red, Green, Blue, Red, Edge and Near Infra-Red (NIR) bands available from cameras like the Micasense RedEdge MX, RedEdge-P or Altum-PT (the latter adding a low-resolution thermal band).
Adequate data gathering necessitates correct calibration using a sun sensor and photographing a calibration panel under the same illumination conditions, in order to guarantee accurate white balance and correct relative gains for each band in the processed images.
The data gathering process also requires maintaining an approximately constant altitude above ground, in order to maintain constant resolution, and a large (>80%) longitudinal and lateral (>75%) overlap between individual images, in order to ensure a successful photogrammetric recovery of a continuous dense 3D point-cloud, without gaps.

Satellite data
MAPP.it uses medium-resolution (10-60 m/pixel) multispectral imaging sourced from Copernicus, the European Union Earth observation programme.The images are obtained by the Sentinel-2satellites, which provide global land coverage every five days over any given location, provided that no clouds are present.The freely available data complements the UAV-gathered data by providing high temporal resolution data at medium resolution, as opposed to high-resolution drone data that is typically gathered 1-3 times per year.Satellite data does not allow identification and classification of individual vines, but provides a coarser view that enables identification of trends and potential anomalies in the vineyard that may show as deviations from expected evolution of the monitored indices.MAPP.it uses bands B2 (blue), B3 (green), B4 (red), B8 (NIR), B5 (red-edge), and B11 (SWIR).

Field observations
MAPP.it is deployed both as web and mobile applications.The mobile application runs on smartphones or tablets and provides quick access to each estate data in the field.The data includes the estate layout with all the vineyard blocks, the most recent plant-level data and contextual RGB ortho-mosaics.Aided by the device's built-in GPS and camera, the users may locate themselves in the vineyard with data displayed around their location at any scale of perspective and create georeferenced notes comprising a photo and a description, which are uploaded to the estates' database.These notes can then be correlated with the remaining data and accessed on both the web and mobile applications.

Data processing
MAPP.it has been developed from the ground up as agnostic to data source, and currently supports multispectral 2D ortho-mosaics, Sentinel2 data, and vector data.It also includes automatic processing pipelines, co-registration of geo-referenced data and analysis and decision support tools.

1 Photogrammetry
Raw UAV imagery is processed in a photogrammetry pipeline, first building a continuous dense 3D point-cloud from matching features and then generating an orthomosaic obtained from a stitching process of patches of the original photos.This process outputs RGB and RG+NIR (for context/visualization purposes), as well as NDVI (normalised difference vegetation index) and NDRE (normalised difference red-edge) single-channel ortho-mosaics to be used further downstream to compute plant-level data.

AI-based processing to plant-level data
The generated ortho-mosaics are normalized to a 10 cm/pixel resolution, and subsequently fed to a convolutional neural network (CNN) for automatic detection of vines.The CNN is periodically retrained with a data set that is augmented every year with the previous year data.The data comprises pairs of orthomosaics and masks that have been validated in a manual quality control process (ground truth).The output of the CNN is a 1-bit depth mask only containing the vine canopy.
The mask (raster file) is vectorised by extracting the skeletons (approximated centre lines of the canopy), which are finally discretised to points taking into account the known plant spacing.
The resulting plant map is used to compute an index for each plant based on the intersection of the 1bit mask raster and the median index value in the vicinity of each plant computed from the singlechannel index ortho-mosaic.
The skeletons vector file is also fed to another algorithm that detects gaps in the canopy where at least one vine could fit.This algorithm outputs a vector file complementary to the skeletons that in this case contains the gaps in the canopy where plants are missing.

Co-registration
A module for co-registration of heterogeneous data developed in-house is included that automatically coregisters 2D ortho-mosaics and 3D point clouds.

Derived data
Various data are derived from the processing described above.They are of three main types: plant-level, blocklevel, and pixel-level data.
Plant-level data currently provided for vineyard applications includes NDVI and NDRE.NDVI is computed from the well-known formula below: NDRE is computed from the same formula but replacing the RED by the Red-Edge channel.NDVI is possibly the most well-known vegetation index providing a robust measure of overall plant vegetative growth.NDRE provides a measure of plant health more closely associated with chlorophyll content.Significantly, whereas NDVI tends to saturate at later stages of development providing little differentiation between plants, NDRE does not saturate and is more sensitive to variability.
Block-level data is computed by taking into account the information from all its plants and producing simple metrics such as the number of plants, plant density, average value of an index, or its variability (standard deviation).
Finally, pixel-level data is computed from satellite data with spatial resolutions of 10-20 m/pixel with which trends can be observed, but further direct inferences should be avoided given that a patch of terrain of that size (10-20 m) contains many vines and a comparable amount of inter-row that can be anything from bare soil to a cover crop.The pixel-level data derived from satellite includes the NDVI, NDRE, and NDMI (Normalised Difference Moisture Index).The first two are computed from Sentinel-2's channels B4 (red), B8 (NIR), and B5 (rededge).The latter is computed from a similar formula to 1 but where the Red channel is replaced by the SWIR (Short WaveInfra-Red), in particular through the B11channel in Sentinel-2.

Data analysis and decision support tools
Data analysis and decision support tools were developed and improved through iterations between the service provider and users from the wine industry, incorporating elements and adjusting features to optimize the value derived by the latter when using the tools.The result was aweb-service named MAPP.it.MAPP.it provides the user with real-time data-manipulation tools to enable the creation of zones based on the selected variable (NDVI, NDRE, NDMI, etc.).It also enables selection of the observed universe based on an arbitrary number of metadata fields.
The default analysis universe includes all the blocks in an estate.The information shown is always contextual and is applicable to these to selected blocks.This information shown on screen includes a metrics panel (with number of plants, plant density and selected index average), and a tools panel containing a histogram (with the distribution of plants according to index value) and the user interface to create and manipulate zones.The plots panel contains the user interface to filter these selected blocks according the arbitrary metadata fields uploaded, such as block identifiers, year of plantation, plant variety, etc.A block or a group of blocks can also be directly selected with the mouse.
The creation of zones is done in the tools panel by choosing 2 to 5 optimal zones or alternatively by choosing pre-defined percentiles.The optimal zones are computed using the k-means [5] clustering algorithm.The zoning using percentiles groups the plants into predefined percentiles: (25, 50, 75) or (10, 25, 75, 90).The zones are shown in the map and histogram using a discrete colour map.The total number of plants (or hectares, depending on user selection) in each zone is shown in the histogram together with the index thresholds between each zone.These thresholds can be adjusted by the user according to the application and management practices and goals.In the case of selective harvesting the user might, for instance, want to have a minimum size (hectares) or number of plants for each zone such that it is viable to vinify in the cellar.

Understanding intra-block variability
Intra-block variability may be driven by several factors that may be grouped as structural (geomorphology, soil composition and structure, grapevine rootstock and variety, training system, plant density) or temporary (annual climate, nutrient and water availability, grapevine age, pests and diseases, accidents).Heterogeneities reduce the efficiency of the vineyard production system by reducing productivity, reducing the effect of scale for cost management and increasing the costs for quality management, namely in segmenting fruit to make homogeneous batches fit for the wines to be produced.Hence, vineyard managers strive to reduce heterogeneities or at least to define and control the boundaries of vineyard areas with contrasting characteristics, so they can be cultivated accordingly in a cost-effective way.

Evaluation of management decisions
When applying management decisions, vineyard managers usually have to wait several days, weeks, or even months before they can evaluate the outcome and the goodness of the decision.If the decision is meant to decrease the heterogeneity of a block, measuring the level of variability within the block becomes an actionable metric, provided the characteristic's (NDVI, NDRE, or NDMI) variability provides a good indication of the effect being sought and is sensitive to the management decision being applied.When this happens, variability of that characteristic can indeed be used to monitor and evaluate management decisions.On the other hand, management decisions may be made not to decrease heterogeneities but to segment and differentiate actions according to the different characteristics of areas in the vineyard (soil types, varieties, age, vigour, etc.), be them different blocks or sub-blocks (areas within blocks showing different behaviours and/or characteristics).Sometimes, sub-blocks from adjacent blocks show similar behaviours and characteristics between them, making them the target of the same management decisions, despite being indifferent blocks.In this case, management success will be measured by the enhancement of the contrasting characteristics of the target areas, blocks, or sub-blocks, according to the desired agronomic and oenological outcomes.

Planning at different time-scales
In vineyard management, some decisions require advance planning of weeks and even months.This is the case when more structural characteristics are to be changed: regrafting with different varieties or clones, installing irrigation or changing a training system.Consequently, the outcomes of those decisions will also require weeks to months to be correctly assessed.
However, other decisions are made and assessed in shorter time-scales.These are the cases of green pruning (removal of shoot tips, leaves, and shoots to provide aeration and solar exposure for grape bunches or to control vegetative growth), sanitary operations (spraying, pheromone diffusion, UV or heated air application to prevent or stop disease or pest outbreaks) or correcting nutrition through foliar applications.The typical timescale for these decisions may range from a few hours to a few days.
Finally, there are hybrid cases, where the operation requires planning both at long and short time scales.Examples of these are winter pruning, sanitary protection strategy, and harvest.These operations are usually highly dependent on the availability of human labour or contract machinery and need to be planned well in advance to ensure they will be available when needed at a reasonable cost, while retaining enough flexibility for last minute changes because of an unexpectedly long cold spell, high rainfall volume, or long drought duration.

Harvest planning
Of hybrid decisions, the most challenging are the ones related to harvest planning.This is the moment when fruition of a whole year of work and of the initial investment in planting the vineyard is expected to happen, the moment when the yields will provide the expectable value surplus against the cost of production and the quality of the grapes will be transferred to the wine they will originate.The ideal harvest window is often an elusive and fast-moving target.Mainly depending on the type or types of wine to be produced and available grapes (conditioned by the variety and the level of maturation at the picking moment), this window may range from a couple of weeks to a couple of days, being highly dependent on the weather conditions in the week or two before the ideal grape composition is attained.To match winery specifications, the vineyard manager needs to monitor very closely the evolution of the grape quality during this period and foresee with sufficient lead time where the ideal composition is expected to be obtained.Besides the weather, several other factors will play an important role, namely logistic issues such as availability of labour at the time of picking, capacity of the winery to receive grapes in that same date and availability of transportation between vineyards and the winery.Usually, the two weeks leading to the picking date are highly intensive in decision-making, making this time the most critical in generating value from making wine out of grapes.

Plant detection consistency
In assessing the plant detection consistency and accuracy it is important to take into account both the detected plants and gaps.To that end, the scatter plot below shows the sum of detected plants and gaps for each of the 864 blocks imaged and processed in 2020 and 2021.In 2019 comparatively less blocks were imaged (198) and at the time of this analysis the complete 2022 data set was not yet available.The scatter plot shows good consistency (R 2 =0.9939) in the total number of plants and gaps detected in both years, with a total relative error of 0.19% (2,299,067 and 2,294,769 total plants and gaps in 2020 and 2021, respectively).

Large-scale trends
The purpose of this test is to show that the variations observed by comparing several years, are seen in all the estates (or in the same region) showing that the determining factor is the state of the crop due to large scale climate/weather and not due to noise/error in the remote sensing pipeline.
Direct yearly comparison of remote sensing-based data on crop status cannot be done without taking into account the main factors potentially contributing to the differences: data acquisition/processing noise/errors and crop status.In general terms the crop status can vary from year to year due to: weather/climate, water and nutrient availability, diseases/pests, and management practices.Of these, weather and water availability tend to have at least a regional scale, whereas diseases/pests and management can be more localised.Figure 7 shows the average NDVI per estate for the years 2019 to 2022.On the left side are grouped the estates located in wetter wine regions (Dão, Bucelas, Bairrada, Vinho Verde), whereasthe right-side group are located in the drier regions of the Douro (mostly nonirrigated) and the Alentejo (mostly irrigated but with limited water availability).
The plot shows 2021 as the year with highest average vigour (NDVI) in all the estates but two.Conversely, it shows the current year (2022) corresponding to the lowest average vigour in all but two estates of the Douro and Alentejo regions.Of those two, DS4 is deficitirrigated and DCC6 is located above 500 meters relative to sea level in the wettest and coolest area of the valley.With the exception of one small holding (BA1 with 4.75 hectares of vineyards), the average vigour in the wetter regions appears to have an overall smaller dispersion through the years.
These large-scale trends show, as expected, that the determining factor in vine vigour is the weather/climate which is associated to water and nutrient availability.Furthermore, it also shows the errors associated with image acquisition and processing are sufficiently small as to allow a multi-year comparative analysis.

UAV vs. Sentinel-2
Sentinel-2 and UAV imagery is intrinsically different (in the acquisition geometry, sensors, atmospheric interferences, illumination, scale, etc.) and one cannot say in absolute terms that one is better than the other.Each has its advantages and disadvantages.
The purpose of this comparison is thus not to validate one against the other, but rather to understand under what circumstances they correlate and under which they do not.The weighted average NDVI per block is computed from Sentinel-2 imagery by introducing in the computation as many pixel samples as there are plants, even if that means that the same Sentinel-2 pixel is used several times (asthere are tens of plants in each Sentinel-2 pixel).The reasoning behind this approach as opposed to a simple pixel average in the block, is that such method slightly reduces the bias produced by the scale difference: tens of different vines and inter-row in one 10x10 m pixel means that even if all the plants had NDVI=1, the NDVI value for that pixel would never be 1.The UAV-derived average plant NDVI per plot on the other hand, is a simple average of all the NDVI values of all the plants in a block.
The scatter plot above shows that these two different average values per block do correlate, albeit, as expected, not in a simple linear fashion.The Sentinel-2-based values have an overall average of 0.35 (as opposed to 0.68 in the UAV case) and both its bounds are lower than those from the UAV case as seen in the histogram above.This behaviour is expected since the Sentinel-2 pixel will always contain some soil, whereas the plant values obtained from UAV imagery are by definition strictly from grapevine vegetation.Another consequence of this intrinsic difference is observed at the extremes of the range: at very high vigour the plant-level NDVI obtained with UAV tends to saturate, while the Sentinel-2-based NDVI rarely saturates due to the inter-row interference.The confusion matrix above shows another measure of correlation between Sentinel-2 and UAV derived data, this time at a sub-block scale.Each estate's entire vineyard was divided in 3 macro classes (tertiles) of increasing vine vigour, with each containing 33% of the plants (UAV) or area (Sentinel-2).The macro classes are generated by making each point (plant and pixel in the UAV and Sentinel-2 cases, respectively) equal to the average of itself and the surrounding points in a radius of 10 m.
The results in the confusion matrix show the following: 56% percent of the plants predicted (with Sentinel-2) in the correct tertile (i.e., accuracy); 38% of the plants predicted in the adjacent tertile; and 6% of the plants not predicted in the adjacent or correct tertiles.

Application to vineyard management
The resultant development of plants is an effective integrator of the environmental constraints it has to deal with.NDVI correlates with grapevine vegetative expression.Measuring NDVI at veraison allows for spatial variability assessment of plant vigour at both intra-annual and inter-annual time-scales [6].
In this way, NDVI becomes an important tool in assessing the effects of environmental constraints (climate, soil, nutrition, diseases, farming) in crop outcomes (yield, quality) and resources usage (water, fertilizers, plant protection, financial).
The use of these tools in vineyard management during the test phase by vineyard managers of all 20 holdings considered paved the way for having a general bird's eye view of the whole property, regardless of its size, while retaining the ability to zoom into the individual vine block or individual grapevine plant.In this way, the vineyard manager becomes able to identify and map boundaries of vigour heterogeneities down to the resolution of the individual grapevine.This enables them to understand the factors driving such heterogeneities and address the unwanted ones.
Property-wide average NDVI values also serve as an indicator of overall grapevine vegetative development allowing for inter-annual comparisons, determination of the natural and man-made drivers for inter-annual differences or differences between vine blocks in the same year.In this way, management interventions such as fertilizations, disease and pest prevention, water nutrition, canopy management operations (pruning, leaf-thinning, tipping, irrigation, etc.) are evaluated through comparison of before and after images, through the calculation of average index values or the variability within each vine block.The mapped representation of this variability assessment allows for managers to quickly identify where higher variabilities exist and address them in a targeted way, whenever necessary.
Counts of missing (grapevine mortality or morbidity) plants are quite useful to locate and address zones of higher incidence which may hint of soil nutrition problems, excessive heat and water stress or an origin point of trunk or other degenerative diseases.
Not less important, NDRE values are a proxy for chlorophyll levels and thus are a good indicator of plant health, being less prone to saturation, NDRE values provide more accurate assessments and segmentation in areas with high NDVI values.In this way, the combined use of drone-based NDVI and NDRE maps at veraison provide managers a quick, detailed and valuable tool for analyzing in a few minutes what would take hours to do in the terrain.It helps plan field visits to focus the areas with more serious situations and, therefore, increase the manager's efficiency by assisting the priority-setting of which are as to visit first and which blocks to harvest first.This is of particular value during the decisionintense weeks preceding harvest supplying a visually effective tool to support communication between viticulturists and winemakers, grape-growers and wineries.
The availability of satellite imagery, despite their lower resolution are a value-adding feature as they allow for effective temporal control of the vegetation evolution during the growth cycle.Combining satellite information with the data from the last drone flight, it becomes easy to detect those areas that are not following the usual spatial structure of any index, making it a timely signal of something that requires attention and needs a field check to find out the underlying reasons and, if appropriate, take corrective measures.
Finally, the histograms of NDVI and NDRE distribution help to provide objective and measurable assessments of the inter-annual variation of these indices, signaling years with different situations or the efficacy of management decisions.They may also be used to establish block-level or property-level management targets that can be monitored and measured interannually or intra-annually if using satellite-based data.

Conclusion
An oveltele-detection web-service has been developed by combining imagery data from drone and satellite platforms.The web-service is deployed both through computer and smart phone app interfaces.The added value of the web-service for users in the wine sector was optimized through a co-creation interaction between the service provider and wine sector users that effective incorporated tools and solutions supporting decision making needs for several processes and moments of the annual process of growing wine-grapes.
The service provides data of vegetation indices annually at a 10-cm spatial resolution (NDVI, NDRE) and every 5 days at a 10-meter spatial resolution (NDVI, NDRE, and NDMI).It also stores historical data allowing for fast and effective temporal comparisons enabling several vineyard management routines to be performed at unprecedented speed and detail.The mobile application allows for real-time on-site navigation with visualization of vegetation indices offering immediate ground-truthing data and identifying areas of concern.The built-in feature of picture and notes attachment to individual plants in the mobile application is a further user-centric enhancement clearly resulting from the co-creative undertaken approach.
We demonstrated the service to offer a relevant analysis of vegetative evolution of vineyards through the pure vegetation pixel methodological approach for dronebased imagery and by fusioning high-spatial resolution drone-based data with high-temporal resolution satellitebased data.Indeed, the multiple year analysis of sums of plants and gaps confirmed that the signal-to-noise ratio of the detected data is high, reliable, and consequential in supporting management decisions at the inter-annual scale.The comparative analysis of drone and satellite data further confirmed the complementarity of both sources and the relevance of their fusion in a single interface to achieve a highly resolved view, in the spatial and temporal dimensions, of vineyards, regardless of their characteristics, structures or complexities.
The user tests have shown the capacity of the webservice to provide actionable and valuable insights into vineyards and grapevines development and outcomes, positively influencing the efficiency of management and final agronomic and oenological results.Minimal qualification and training was required, something resulting from the co-creative process and user-centric approach, namely the quality of the intuitive and visuallyexplicit interface.
The service opens also the possibility for further research to be deployed, namely in understanding the drivers of block and vineyard variability but also by providing a powerful tool to integrate data at the plant level from different commercial-scale production options systematized in field trials with adequate experimental designs.In viticulture research, this provides a much needed platform to merge trial-scale and field-scale scientific experiments able to address the high-variability environment of commercial vineyards.

Figure 3 .
Figure 3. Data processing work-flow and modules.

Figure 6 .
Figure 6.Comparison of sum of plants and gaps detected per block.

Figure 7 .
Figure 7. Average NDVI per estate for different years.