rus
Issue #2/2019
M. V. Agrinsky, A. V. Golitsin, V. V. Startsev
Project of hyperspectral remoteland sensing complex using UAVS. Part 1
Project of hyperspectral remoteland sensing complex using UAVS. Part 1
Using unmanned aerial vehicles equipped with hyperspectral equipment, it is possible to collect high-resolution spectral and topological information about the local area of interest. Such opportunities open up a great potential for the use of hyperspectral complexes in monitoring the use and condition of agricultural land. For the purpose of obtaining detailed, rather than integral spectral information, the element base has been considered, structural elements have been selected and a practical model of the domestic complex of hyperspectral sensing has been created. The first part of the article describes the design of a television camera and its interaction with the mating assemblies.
DOI: 10.22184/1993-7296.FRos.2019.13.2.184.201
DOI: 10.22184/1993-7296.FRos.2019.13.2.184.201
Теги: hyperspectral equipment monitoring the use and condition of agricultural land television camera uavs бпла гиперспектральное оборудование мониторинг за использованием и состоянием земель сельскохозяйств телевизионная камера
Аrticle was received on 16.11.2018
Article was accepted for publication 24.11.2018
Monitoring the use and condition of agricultural land is a necessary element of the regulation system of the agricultural and industrial complex. Information on the availability and use of arable land should contain the data on the spatial distribution of arable land used and crops, as well as the data on the rapid detection of diseases of economically significant plants, the extent of damage to crops by various pathogens and assessments of the state of the land. Thus, the creation of a complex for monitoring agricultural lands is associated with the tasks of providing interested users with information about the areas of arable land and crops of various types, their productivity, and current information about their condition.
To implement the tasks of monitoring agricultural land and obtaining objective information about their use and condition, it is proposed to use methods of remote sensing of agricultural land using unmanned aerial vehicles equipped with hardware for conducting aerial digital surveys of extended areas and objects. The use of modern remote sensing data allows for optimizing and improving the efficiency of the territorial organization of agriculture.
The typical tasks are as follows:
• ensuring current monitoring of the state of crops;
• early forecasting of crop yields;
• monitoring the pace of harvesting simultaneously across the territories of large regions;
• operational monitoring of the detection of plant diseases and the extent of damage by various phytopathogens;
• determination of the capacity of pastures of various types and productivity of hayfields, etc.
In remote sensing (RS), the hyperspectral systems installed on an unmanned aerial vehicles (UAV) are becoming more widely used along with multispectral systems. The uniqueness of the hyperspectral system lies in its ability to record radiation in hundreds of very narrow spectral ranges, which allow for assessing the physicochemical properties of the objects under study. The most relevant problems of monitoring solved with the help of hyperspectral sensors include, inter alia, the following:
• determination of phenophases of plant development and timely detection of their anomalies;
• detection of the processes of lodging, soaking and binding crops, associated with a lack of moisture;
• control of the phytosanitary condition of crops and woodlands;
• observation of the dynamics of the development of crops and forecasting crop yields.
These tasks can be successfully solved with the help of an unmanned aerial vehicle equipped with hyperspectral instrumentation, which can ensure the collection of high-resolution spectral-topological data in the region of interest without significant costs.
Analysis of modern developments of hyperspectral equipment showed that the composition of domestic advanced developments is mainly represented by hyperspectral instruments for space purposes. For use in ground-based instruments of hyperspectral imaging, installed on UAVs, the market offers products by XIMEA, Resonon. Cubert and some others. However, these instruments provide only integral (areal), and not detailed information and, moreover, are expensive. We have created a practical model of a domestic hyperspectrometer complex designed to monitor the use and condition of agricultural land with an unmanned aerial vehicle (UAV). This work describes the analysis of the principles of work, considers the element base, justifies the choice of structural elements.
HYPERSPETCROMETER ARRANGEMENT
We will analyze the hyperspectrometer arrangement in order to understand the principles of its operation and determine the possible components of its design.
A hyperspectrometer is a device that captures images using following surface, and for each point of this image you can get the brightness spectrum of the reflected radiation in a given range of electromagnetic radiation. The luminance spectrum is represented by a limited set of spectral channels with given bandwidths.
The most common today are hyperspectrometers, which at each point in time register a narrow segment of the surface below. Such hyperspectrometers are of the «pushbroom» type. The functional diagram of the pushbroom-type hyperspectrometer is shown in Fig. 1. The hyperspectrometer includes an optical imaging system, a spectral divider and a photodetector.
The image formation of a narrow segment of the surface is performed by a slit, which is installed on the rear focal plane of the input lens. After the collimating lens, the image in parallel rays falls on the spectrometer, which can be used as a flat one-dimensional diffraction grating, where decomposition occurs in the spectrum, and then projected onto a photo-receiving array of a television camera.
The optical system and the diffraction grating of the hyperspectrometer form an image on the photodetector array. Along one axis of the image, the Y coordinate of a narrow strip of the Earth“s surface (see Fig. 1) is plotted; on the other, λ is the wavelength of radiation reflected from the Earth“s surface, and the amount of charge accumulated inside each array element (pixel) is proportional to the spectral radiation density at a given wavelength. Thus, a set of spectral dependences of the radiation reflected from the Earth“s surface is obtained on the photodetector array, depending on the Y coordinate of a certain part of the Earth“s surface.
The magnitude of the band in terms of Y is determined by the angle of view of the camera of the television α and the altitude of the UAV.
Due to the fact that the UAV, on which the hyperspectrometer is located, moves along the X coordinate (see Fig. 1), the Earth“s surface is scanned in the X direction.
A array photoreceiving device (array) is used as a photodetector. To control the array, the acquisition, storage and output of digital video data, a programmable logic integrated circuit (FPGA) with an integrated processor is used, which is a television camera as a whole [1−3].
Further we present the analysis of the main components of the hyperspectrometer for a reasonable choice of structural elements.
ANALYSIS OF THE TELEVISION CAMERA DESIGN
The television camera (TC) is designed to work as part of a hyperspectral camera on board an unmanned aerial vehicle (UAV). A TC transforms an optical image projected by the optical system onto a photodetector of a television camera into a video signal of an image and issues it to interfacing systems in digital and analog form.
Main technical specifications
1. Number of photosensitive elements of the array used to form the output video signal of 2048 Ч 2048 elements (pixels), with a photosensitive element size of 5.5 Ч 5.5 µm.
2. Size of the active zone of the photosensitive field of the array is 11.264 mm horizontally and 11.264 mm vertically (diagonal 15.8 mm);
3. Spectral range of the TC is determined by the spectral characteristics of the photodetector (reference parameter) and ranges from 0.4 to 1.0 µm.
4. TC operates in progressive scan mode and has a line resolution on the digital output in the central zone of the field of view of at least 750 television lines with an illumination in the plane of the photosensitive array surface of at least 10 lux and a modulation depth of 5%.
5. Operating range of illumination on the photosensitive array TC extends from a minimum of 0.2 lux to a maximum of 700 lux with automatic adjustment of sensitivity in the whole range of illumination on the photodetector.
6. TC provides digital video output via Base Camera Link without video compression, scan type – progressive with a frame rate of at least 50 Hz.
7. TC provides an analog video output signal in accordance with GOST 7845–92 (for a black and white image) used in process checks and settings.
8. Power supply of the TC is carried out from the onboard power supply system of the direct current of the main object with a voltage of 22 to 29 V, the power consumption does not exceed 10 watts.
9. Availability time of the TC for operation from the moment of power supply does not exceed 10 s.
Selecting the type of light-signal converter
The main structural element of a digital television camera, ensuring the realization of the main characteristics of the camera (resolution, frame rate, dynamic range, etc.) is a light-signal converter, which uses semiconductor solid-state array photodetectors or arrays.
Currently, modern digital television cameras mainly use two types of arrays: CCD arrays (Fig. 2) (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor, CMOS).
In the CCD and CMOS arrays, photodiodes are used to convert light into an electrical image signal. However, their operating principle varies significantly.
In charge coupled devices, the incident light detected by the photodiode of each pixel is converted into an electric charge (Fig. 3). The pixel charge is moved to the vertical «transport bus», located on the side of the pixel. The applied voltage then moves the charges along the vertical and horizontal transport buses until they reach the amplifier. The analog signal with different voltages (depending on the amount of light per pixel) is obtained at the output. Before processing, this signal is sent to a separate (located outside the chip) analog-to-digital converter, and the resulting digital data is converted into bytes representing the line of the image received by the sensor.
Since the CCD transmits electric charge, which has a low resistance and is less susceptible to interference from other electronic components, the resulting signal, as a rule, contains less different noise compared to the CMOS sensor signal.
Information from each sensitive cell is read sequentially, which prevents the next frame from being taken before all the data from the previous frame are considered (at the moment, this problem has been partially eliminated by increasing the memory buffer). This does not allow to use in streaming video, because these arrays are gradually being superseded by CMOS technology, the arrays of which can produce video through the array itself.
In the design of CCD arrays, there is a problem of «smearing effect» («slur»). It occurs when a very bright incident light falls on a vertical transport bus due to a leak and creates an excess charge, which manifests itself in the image as a bright vertical strip. However, the problem of slurring is radically solved in devices with interline transfer, which have gained a dominant position in the consumer video market.
Unlike arrays with frame transfer, the functions of charge accumulation and its transfer are separated. The charge from the accumulation elements (these are usually photodiodes; they also have capacity and are able to accumulate charge) is transmitted to the CCD transfer registers closed from light, i. e., the transfer section is sort of inserted into the accumulation section. Now the transfer of the charge relief of the entire frame occurs in a single cycle, and the «slurring» associated with the transfer does not occur. Compared to the frame transfer arrays, the filling factor in the interline transfer arrays is about half as much, since about half of the photosensitive surface area is protected from light.
Such a structure also requires high voltages to alternately open and close the shutters, which should be included in all pixels to control the time sequence of the flow of charges.
The power consumed by CCD arrays is especially great for high definition when you need to quickly read a large number of pixels. The CCDs have better quantum efficiency and low noise and provide flexibility in terms of cost when designing a system. They continue to dominate where the best image quality is required, e. g., in most industrial, scientific and medical applications.
In CMOS sensors, each pixel has its own charge-to-voltage converter (Fig. 4). And the sensor often contains circuits for digitizing, so that the digital signal is output at the chip output. The CMOS pixel cross section is shown in Fig. 5.
These additional functional assemblies take the area of the crystal available to collect incident light. Furthermore, the uniformity of the outputs (a key factor in image quality) of these sensors is worse, since each pixel has its own converter. But, on the other hand, the CMOS sensor requires fewer external circuits to perform basic operations.
The problem with smearing effect is absent here, since the incident light does not affect the electrical signal. Instead of shutters, switches and internal circuits are used in the CMOS array to control the sequence of output signals. The use of internal switches can significantly reduce power consumption while accelerating the process of simultaneously reading a large number of pixels. The reading efficiency is quite sufficient to support the progressive decomposition of HD images. In single-chip CMOS sensors, it becomes fundamentally possible to simultaneously read the R, G and B signals.
The CMOS sensors provide greater integration (more functions on the chip), less power dissipation (at the chip level) and a smaller system size due to image quality and flexibility. They are well suited for small-sized products.
The cost of crystals for both types of sensors (CCD and CMOS) is about the same. Previously, the supporters of the CMOS sensors installation claimed that they are much cheaper, since they can be manufactured on the same process lines as most high-density logic and memory chips. It turned out to be wrong. In order to obtain good image quality, the production of CMOS sensors requires special technological processes inherent to low-density mixed-signal processing devices. The CMOS sensors also require more silicon per pixel. A CMOS camera may contain fewer components and consume less power, but it may also require signal processing to compensate for the loss of image quality.
To increase the efficiency of photon collection, a micro-array of small microlenses is used. Its formation is very simple: a layer of optical low-melting plastic is applied onto the surface of a plate with already formed array structures, from which isolated squares are cut out above each element using photolithography. The gap between the individual squares is small. Then the plate heats up, the plastic melts and the surface of the individual squares acquires a close to spherical shape, focusing the light incident on its surface onto the photosensitive element of the array (Fig. 6).
To obtain the highest quality images with a minimum of noise, it is better to use the CCD arrays. However, if you need to conduct high-speed shooting, the CMOS array will be the optimal choice. High-speed video cameras based on CMOS arrays are widely used for surveying scientific experiments, registering fast processes, setting up and monitoring technological processes in production.
Currently, there is significant progress in the technology of CMOS arrays, their characteristics are close to the characteristics of CCD arrays. However, in a number of tasks, the CMOS arrays have significant advantages, e. g., if arbitrary sampling is required by coordinates, when tracking processes are performed using an object (window) of a given size, which can change during the tracking process, and also when hardware processing is required (in the CMOS array itself) in real time.
The CCD technology has been around for more than 30 years, and today, as part of this technology, numerous modifications have been developed that are adapted to solve various problems.
The lack of arrays produced by this technology is the limitation of the clock frequency of reading of 30–40 MHz. To overcome this drawback, the array developers have to make several output devices, which in turn leads to complex problems of black level alignment and gain linearity. In the image, this appears as a different brightness of the halves or quarters of the image (depending on whether two or four outputs are used).
Furthermore, the circuit design of TV cameras developed based on CCD arrays is significantly more complicated than that of TV cameras built based on CMOS arrays, which also leads to an increase in the consumption of these TV cameras.
The CMOS technology has the advantage of low production (the arrays are produced using well-proven CMOS technology) and the ability to place circuit solutions for processing the signal from the array on the same chip as the array itself. Thus, a microcircuit can contain not only a light-signal converter, but also an analog-to-digital converter (ADC), a synchronous generator, etc.
The disadvantage of this technology until recently was the lower sensitivity of the arrays and the limited adjustment of the accumulation time (shutter).
Recently, however, the new technological advances in the development of the CMOS arrays have largely overcome these shortcomings and have led to the emergence of modern sensors comparable to the CCD arrays in sensitivity and noise level, which, together with a large frame frequency, make them indispensable for a huge number of applications.
The family of high-speed CMOS sensors with a frame shutter has a resolution from VGA to 20 million pixels. The sensors consist of arrays of conveyor-type pixels with a «frame» shutter, allowing exposure during the reading of the previous frame, as well as performing double correlated sampling (DCS), which significantly reduces the proportion of noise and dark currents in the useful signal. A distinctive feature of modern CMOS sensors is their ultra-high frame frequency at full resolution and the possibility of its increase through partial reading, window mode and sub-sampling mode.
The readout circuit consists of LVDS digital serial outputs. The sensor includes an amplifier with a programmable gain and has the ability to adjust the offset. All settings are typically made through the SPI serial peripheral interface. The internal clock generates the synchronization signals necessary for reading and controlling the exposure. External exposure control is also possible. Depending on the model, the sensors support 8-, 10- and / or 12-bit ADC.
Distinctive features of the CMOS sensors:
• high frame rate;
• ability to highlight several areas of interest;
• built-in PLL (phase locked loop);
• built-in temperature sensor;
• built-in clock generator;
• SPI interface;
• monochrome and color versions.
The CMOS sensors are widely used in the following areas:
• machine vision;
• high-speed control;
• television broadcasting;
• aerial photography;
• space / astronomy;
• CTCV.
For comparison, we give the characteristics of the most high-quality arrays created by CCD and CMOS technologies (see Table 1).
The KAI‑02050 array was created using CCD technology, it has 4 outputs (to ensure a high frame rate) and, therefore, requires four ADCs, many drivers for controlling the phases of the array, and several power sources for the operation of the array. Furthermore, its sensitivity is inferior to the EV76C560 array.
The EV76C560 array (by E76V) when tested, showed the following results: when the illumination in the array plane is 5 · 10–3 lux, the signal-to-noise ratio S / N = 4 with an exposure of 40 ms. This array also has a possibility to use 2 different methods of reading the signal from the array – the so-called Global shutter and Rolling shutter.
The Global shutter is the reading of the charge from all pixels of the array at the same time, and the Rolling shutter is the sequential reading of lines of the array. Therefore, if there is a moving object in the frame, the exposure of the Global shutter type will be a «slur», the value of which depends on the speed of the observed object and the exposure time, and with the Rolling shutter type of exposure, the object will be distorted geometrically, since its different parts will be read at different times.
However, with a rolling shutter in a CMOS array, the noise is substantially less (signal-to-noise ratio S / N = 8 at illumination of 5 · 10–3 lux and exposure at 40 ms). The reading modes are switched by commands via the serial interface with the array during operation.
Also, a binning mode was introduced in this array (combining four adjacent pixels into one) for the first time for arrays manufactured using CMOS technology. This provides in low light conditions increased sensitivity by reducing the spatial resolution by half.
The sensitivity level of this array in the binning and the Rolling shutter mode approaches the characteristics of the most sensitive CCD arrays (e. g., SONYICX429ALL arrays).
Our tests performed using CMV4000 array by CMOSIS showed that the sensitivity of this array is somewhat inferior to that of the EV76C560array, besides, there is no binning mode and Rolling shutter in this array, but this array allows forming an image with a frequency of up to 180 fps.
Based on the analysis of the above materials, it is advisable to use the CMV4000ES‑3E5M1PP CMOS array and color CMV4000ES‑3E5C1PP array, respectively, in the television camera for the hyperspectral channel. Tab. 2 shows the main technical characteristics of CMV4000ES CMOS-array.
The analysis of the structural diagram of the television camera
Below is a description of the television camera developed by us for operation within a hyperspectrometer.
The structural diagram of the television camera (TC) is shown in Fig. 7
The television camera consists of the following modules:
• video sensor;
• digital processing module (DPM);
• power module.
The video sensor is an array photodetector (APD) operating in the visible wavelength range of 0.4–1.0 µm. The CMOS array is used as a video sensor (CMV4000, format of 2048 Ч 2048 pixels by CMOSIS (Belgium)).
The CMV4000 array video sensor transforms the image from the photosensitive surface of the array into a digital data array, which is then passed to the digital processing module.
The digital processing module (DPM) provides the following functions:
• input of digital data from the video sensor;
• processing signals from the video sensor and converting them into the required image format when transferring a pixel to a pixel;
• formation of the output digital video signal output with a frame rate of 50 Hz in 5 LVDS pairs using the Base Camera Link protocol;
• formation of a standard analog video signal according to GOST 7845–92 on the video output under load (75±7.5) Ohms for technological purposes;
• adjustment of the time of accumulation of video information in manual and automatic mode;
• ability to change the position and size of the video window;
• built-in readiness control;
• additional image processing.
The DPM is made on the basis of FPGA CYCLONEIII (ALTERA), which provides a flexibly programmable logic control of a television camera and the necessary signal processing. Reduced consumption compared with previous generations of FPGAs (CYCLONEII) provides the necessary thermal regime of a television camera.
The FPGA implements all types of signal processing, data buffering in RAM, digital video sensor control signals.
The DAC control is implemented on the FPGA to output an analog television signal according to GOST 7845–92, digital video signal over 5 LVDS pairs, using a protocol in the CameraLink format. The FPGA also receives control information from the mating equipment and provides telemetry via the CAN2.0b interface.
The power module, located in the chamber, forms the secondary power sources for the power supply of the FPGA, array, RAM, ROM, ADC, digital and analogue parts of the television camera. The power supply module provides output stabilized voltage to power the components of the television camera at an input supply voltage of 22–29 V.
The power module is galvanically isolated at the input, the common mode filter and output filters. The main technical characteristics of the television camera are given in Table. 3
Interaction with mating devices
The output information of the TC is digital and analog video signal of an image, output over LVDS5 pairs using the Base Camera Link protocol and GOST 7845-92, respectively. Analog video signal is used for technological purposes.
The control of the TC operating modes and the output modes of the video information is carried out by external commands (signals) received from the CAN2.0b digital interface using ISO‑11898 (It is possible to control the operating modes of the TC via the digital RS‑485 serial interface). Information on the state of the TC (readiness signals, health, etc.) is output to the opto-electronic system via the CAN2.0b digital interface.
In order to form the output video signal in the external synchronization mode, clock pulses with a repetition period of at least 10 ms are sent to the TC from the external equipment.
The LVDS (Low Voltage Differential Signaling) interface uses differential signaling with low signal levels. Most commonly, LVDS transmitter and receiver are used in a point-to-point configuration.
The LVDS interface provides high speed data transfer. The amplitude of the differential signal is 350 mV, which makes it possible to make sharper fronts of signals and a theoretically possible data transfer rate of 1.923 Gbit / s in a lossless environment. This interface is insensitive to common mode pickups up to ±1 V to differential inputs. Since LVDS uses the current switching mode when transmitting information, it is necessary to pay special attention to power consumption and electromagnetic pickups on adjacent buses when designing devices, which is the price of speed.
The transmitter controls the differential line. A marking impulse with a current of 3.5 mA is output to the line. The load lines are parallel-connected differential LVDS receiver and 100 Ohm resistor. The receiver itself has a high input impedance, and the main signal shaping occurs at the load resistor. When the line current is 3.5 mA, a voltage drop of 350 mV is formed, which is detected by the receiver. When switching the direction of the current in the line, the polarity of the voltage across the load resistor changes, forming the states of a logical zero and a logical unit.
The transmission scheme used has a number of advantages. The LVDS interface has a high data transfer rate. The maximum speed determined by the standard is up to 622 Mbit / s, which is comparable with the optical interface. At the same time, the actual speed of the LVDS channel is limited by two factors: the feed rate and data retrieval rate and the transmission medium parameters.
The differential signaling method used allows minimizing the effects of external interference. A small voltage change, symmetrical transmission lines and a low voltage rise (1 V / ns) cause a low level of interference from the interface itself. In turn, the low level of crosstalk and low sensitivity to external pickups make it possible to use a high density of connections both on the printed circuit board and in the cable. The LVDS transmission method does not depend on the supply voltage and operates at the transmitter and receiver supply voltage of 5 V, 3 V or 2.5 V. At the same time, pairing devices with different supply voltages is not a problem.
The transmission method used also makes it possible to significantly reduce the power consumption of the interface. At 100 Ohms the LVDS load with a voltage drop of 400 mV is 1.2 mW. For comparison, the RS‑422 interface has a power dissipation at a load of about 90 mW, which is 75 times more.
The simplicity of the termination signal LVDS-lines should also be noted: this requires only one resistor for each pair, and that“s it. For comparison, other high-speed methods require a pair of resistors, one of which is connected to ground, and the second is connected to the power supply, and often not directly, but through a special source.
The CameraLink digital interface, based on the LVDS interface, is a serial data transmission protocol designed for machine vision tasks. The frequency range of the Camera Link interface allows you to work with cameras with the resolution reaching megapixels, and the speed reaching several hundred frames per second. For cameras designed for a channel capacity of 100–850 MB / s, a standardized Camera Link interface is recommended.
Since Camera Link was designed specifically for machine vision cameras, it guarantees a very high video speed, ease of use and data integrity. The Camera Link interface has three configuration options, Base, Medium and Full. The speed for different configuration options: Base – 255 MB / s, Medium – 510 MB / s; Full – 850 MB / s [4 –6].
Informational interaction of TC with interfaced systems in the composition of the hyperspectral camera is made out by appropriate communication schemes and interaction protocols defining the interaction logic, communication parameters and signals developed and coordinated during the design phase of the design documentation [4−6].
Power supply and energy consumption evaluation
The TC power supply is carried out with a constant voltage from 22 to 29 V from the power source of the main object. The ripple factor in the power circuit from 22 to 29 V is not more than 7.4%. The power consumed by the TC from the power source is not more than 10 W. The TC has protection against overloads in electrical circuits and short circuits, in case the power supply circuits are not properly connected to power supplies.
Article was accepted for publication 24.11.2018
Monitoring the use and condition of agricultural land is a necessary element of the regulation system of the agricultural and industrial complex. Information on the availability and use of arable land should contain the data on the spatial distribution of arable land used and crops, as well as the data on the rapid detection of diseases of economically significant plants, the extent of damage to crops by various pathogens and assessments of the state of the land. Thus, the creation of a complex for monitoring agricultural lands is associated with the tasks of providing interested users with information about the areas of arable land and crops of various types, their productivity, and current information about their condition.
To implement the tasks of monitoring agricultural land and obtaining objective information about their use and condition, it is proposed to use methods of remote sensing of agricultural land using unmanned aerial vehicles equipped with hardware for conducting aerial digital surveys of extended areas and objects. The use of modern remote sensing data allows for optimizing and improving the efficiency of the territorial organization of agriculture.
The typical tasks are as follows:
• ensuring current monitoring of the state of crops;
• early forecasting of crop yields;
• monitoring the pace of harvesting simultaneously across the territories of large regions;
• operational monitoring of the detection of plant diseases and the extent of damage by various phytopathogens;
• determination of the capacity of pastures of various types and productivity of hayfields, etc.
In remote sensing (RS), the hyperspectral systems installed on an unmanned aerial vehicles (UAV) are becoming more widely used along with multispectral systems. The uniqueness of the hyperspectral system lies in its ability to record radiation in hundreds of very narrow spectral ranges, which allow for assessing the physicochemical properties of the objects under study. The most relevant problems of monitoring solved with the help of hyperspectral sensors include, inter alia, the following:
• determination of phenophases of plant development and timely detection of their anomalies;
• detection of the processes of lodging, soaking and binding crops, associated with a lack of moisture;
• control of the phytosanitary condition of crops and woodlands;
• observation of the dynamics of the development of crops and forecasting crop yields.
These tasks can be successfully solved with the help of an unmanned aerial vehicle equipped with hyperspectral instrumentation, which can ensure the collection of high-resolution spectral-topological data in the region of interest without significant costs.
Analysis of modern developments of hyperspectral equipment showed that the composition of domestic advanced developments is mainly represented by hyperspectral instruments for space purposes. For use in ground-based instruments of hyperspectral imaging, installed on UAVs, the market offers products by XIMEA, Resonon. Cubert and some others. However, these instruments provide only integral (areal), and not detailed information and, moreover, are expensive. We have created a practical model of a domestic hyperspectrometer complex designed to monitor the use and condition of agricultural land with an unmanned aerial vehicle (UAV). This work describes the analysis of the principles of work, considers the element base, justifies the choice of structural elements.
HYPERSPETCROMETER ARRANGEMENT
We will analyze the hyperspectrometer arrangement in order to understand the principles of its operation and determine the possible components of its design.
A hyperspectrometer is a device that captures images using following surface, and for each point of this image you can get the brightness spectrum of the reflected radiation in a given range of electromagnetic radiation. The luminance spectrum is represented by a limited set of spectral channels with given bandwidths.
The most common today are hyperspectrometers, which at each point in time register a narrow segment of the surface below. Such hyperspectrometers are of the «pushbroom» type. The functional diagram of the pushbroom-type hyperspectrometer is shown in Fig. 1. The hyperspectrometer includes an optical imaging system, a spectral divider and a photodetector.
The image formation of a narrow segment of the surface is performed by a slit, which is installed on the rear focal plane of the input lens. After the collimating lens, the image in parallel rays falls on the spectrometer, which can be used as a flat one-dimensional diffraction grating, where decomposition occurs in the spectrum, and then projected onto a photo-receiving array of a television camera.
The optical system and the diffraction grating of the hyperspectrometer form an image on the photodetector array. Along one axis of the image, the Y coordinate of a narrow strip of the Earth“s surface (see Fig. 1) is plotted; on the other, λ is the wavelength of radiation reflected from the Earth“s surface, and the amount of charge accumulated inside each array element (pixel) is proportional to the spectral radiation density at a given wavelength. Thus, a set of spectral dependences of the radiation reflected from the Earth“s surface is obtained on the photodetector array, depending on the Y coordinate of a certain part of the Earth“s surface.
The magnitude of the band in terms of Y is determined by the angle of view of the camera of the television α and the altitude of the UAV.
Due to the fact that the UAV, on which the hyperspectrometer is located, moves along the X coordinate (see Fig. 1), the Earth“s surface is scanned in the X direction.
A array photoreceiving device (array) is used as a photodetector. To control the array, the acquisition, storage and output of digital video data, a programmable logic integrated circuit (FPGA) with an integrated processor is used, which is a television camera as a whole [1−3].
Further we present the analysis of the main components of the hyperspectrometer for a reasonable choice of structural elements.
ANALYSIS OF THE TELEVISION CAMERA DESIGN
The television camera (TC) is designed to work as part of a hyperspectral camera on board an unmanned aerial vehicle (UAV). A TC transforms an optical image projected by the optical system onto a photodetector of a television camera into a video signal of an image and issues it to interfacing systems in digital and analog form.
Main technical specifications
1. Number of photosensitive elements of the array used to form the output video signal of 2048 Ч 2048 elements (pixels), with a photosensitive element size of 5.5 Ч 5.5 µm.
2. Size of the active zone of the photosensitive field of the array is 11.264 mm horizontally and 11.264 mm vertically (diagonal 15.8 mm);
3. Spectral range of the TC is determined by the spectral characteristics of the photodetector (reference parameter) and ranges from 0.4 to 1.0 µm.
4. TC operates in progressive scan mode and has a line resolution on the digital output in the central zone of the field of view of at least 750 television lines with an illumination in the plane of the photosensitive array surface of at least 10 lux and a modulation depth of 5%.
5. Operating range of illumination on the photosensitive array TC extends from a minimum of 0.2 lux to a maximum of 700 lux with automatic adjustment of sensitivity in the whole range of illumination on the photodetector.
6. TC provides digital video output via Base Camera Link without video compression, scan type – progressive with a frame rate of at least 50 Hz.
7. TC provides an analog video output signal in accordance with GOST 7845–92 (for a black and white image) used in process checks and settings.
8. Power supply of the TC is carried out from the onboard power supply system of the direct current of the main object with a voltage of 22 to 29 V, the power consumption does not exceed 10 watts.
9. Availability time of the TC for operation from the moment of power supply does not exceed 10 s.
Selecting the type of light-signal converter
The main structural element of a digital television camera, ensuring the realization of the main characteristics of the camera (resolution, frame rate, dynamic range, etc.) is a light-signal converter, which uses semiconductor solid-state array photodetectors or arrays.
Currently, modern digital television cameras mainly use two types of arrays: CCD arrays (Fig. 2) (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor, CMOS).
In the CCD and CMOS arrays, photodiodes are used to convert light into an electrical image signal. However, their operating principle varies significantly.
In charge coupled devices, the incident light detected by the photodiode of each pixel is converted into an electric charge (Fig. 3). The pixel charge is moved to the vertical «transport bus», located on the side of the pixel. The applied voltage then moves the charges along the vertical and horizontal transport buses until they reach the amplifier. The analog signal with different voltages (depending on the amount of light per pixel) is obtained at the output. Before processing, this signal is sent to a separate (located outside the chip) analog-to-digital converter, and the resulting digital data is converted into bytes representing the line of the image received by the sensor.
Since the CCD transmits electric charge, which has a low resistance and is less susceptible to interference from other electronic components, the resulting signal, as a rule, contains less different noise compared to the CMOS sensor signal.
Information from each sensitive cell is read sequentially, which prevents the next frame from being taken before all the data from the previous frame are considered (at the moment, this problem has been partially eliminated by increasing the memory buffer). This does not allow to use in streaming video, because these arrays are gradually being superseded by CMOS technology, the arrays of which can produce video through the array itself.
In the design of CCD arrays, there is a problem of «smearing effect» («slur»). It occurs when a very bright incident light falls on a vertical transport bus due to a leak and creates an excess charge, which manifests itself in the image as a bright vertical strip. However, the problem of slurring is radically solved in devices with interline transfer, which have gained a dominant position in the consumer video market.
Unlike arrays with frame transfer, the functions of charge accumulation and its transfer are separated. The charge from the accumulation elements (these are usually photodiodes; they also have capacity and are able to accumulate charge) is transmitted to the CCD transfer registers closed from light, i. e., the transfer section is sort of inserted into the accumulation section. Now the transfer of the charge relief of the entire frame occurs in a single cycle, and the «slurring» associated with the transfer does not occur. Compared to the frame transfer arrays, the filling factor in the interline transfer arrays is about half as much, since about half of the photosensitive surface area is protected from light.
Such a structure also requires high voltages to alternately open and close the shutters, which should be included in all pixels to control the time sequence of the flow of charges.
The power consumed by CCD arrays is especially great for high definition when you need to quickly read a large number of pixels. The CCDs have better quantum efficiency and low noise and provide flexibility in terms of cost when designing a system. They continue to dominate where the best image quality is required, e. g., in most industrial, scientific and medical applications.
In CMOS sensors, each pixel has its own charge-to-voltage converter (Fig. 4). And the sensor often contains circuits for digitizing, so that the digital signal is output at the chip output. The CMOS pixel cross section is shown in Fig. 5.
These additional functional assemblies take the area of the crystal available to collect incident light. Furthermore, the uniformity of the outputs (a key factor in image quality) of these sensors is worse, since each pixel has its own converter. But, on the other hand, the CMOS sensor requires fewer external circuits to perform basic operations.
The problem with smearing effect is absent here, since the incident light does not affect the electrical signal. Instead of shutters, switches and internal circuits are used in the CMOS array to control the sequence of output signals. The use of internal switches can significantly reduce power consumption while accelerating the process of simultaneously reading a large number of pixels. The reading efficiency is quite sufficient to support the progressive decomposition of HD images. In single-chip CMOS sensors, it becomes fundamentally possible to simultaneously read the R, G and B signals.
The CMOS sensors provide greater integration (more functions on the chip), less power dissipation (at the chip level) and a smaller system size due to image quality and flexibility. They are well suited for small-sized products.
The cost of crystals for both types of sensors (CCD and CMOS) is about the same. Previously, the supporters of the CMOS sensors installation claimed that they are much cheaper, since they can be manufactured on the same process lines as most high-density logic and memory chips. It turned out to be wrong. In order to obtain good image quality, the production of CMOS sensors requires special technological processes inherent to low-density mixed-signal processing devices. The CMOS sensors also require more silicon per pixel. A CMOS camera may contain fewer components and consume less power, but it may also require signal processing to compensate for the loss of image quality.
To increase the efficiency of photon collection, a micro-array of small microlenses is used. Its formation is very simple: a layer of optical low-melting plastic is applied onto the surface of a plate with already formed array structures, from which isolated squares are cut out above each element using photolithography. The gap between the individual squares is small. Then the plate heats up, the plastic melts and the surface of the individual squares acquires a close to spherical shape, focusing the light incident on its surface onto the photosensitive element of the array (Fig. 6).
To obtain the highest quality images with a minimum of noise, it is better to use the CCD arrays. However, if you need to conduct high-speed shooting, the CMOS array will be the optimal choice. High-speed video cameras based on CMOS arrays are widely used for surveying scientific experiments, registering fast processes, setting up and monitoring technological processes in production.
Currently, there is significant progress in the technology of CMOS arrays, their characteristics are close to the characteristics of CCD arrays. However, in a number of tasks, the CMOS arrays have significant advantages, e. g., if arbitrary sampling is required by coordinates, when tracking processes are performed using an object (window) of a given size, which can change during the tracking process, and also when hardware processing is required (in the CMOS array itself) in real time.
The CCD technology has been around for more than 30 years, and today, as part of this technology, numerous modifications have been developed that are adapted to solve various problems.
The lack of arrays produced by this technology is the limitation of the clock frequency of reading of 30–40 MHz. To overcome this drawback, the array developers have to make several output devices, which in turn leads to complex problems of black level alignment and gain linearity. In the image, this appears as a different brightness of the halves or quarters of the image (depending on whether two or four outputs are used).
Furthermore, the circuit design of TV cameras developed based on CCD arrays is significantly more complicated than that of TV cameras built based on CMOS arrays, which also leads to an increase in the consumption of these TV cameras.
The CMOS technology has the advantage of low production (the arrays are produced using well-proven CMOS technology) and the ability to place circuit solutions for processing the signal from the array on the same chip as the array itself. Thus, a microcircuit can contain not only a light-signal converter, but also an analog-to-digital converter (ADC), a synchronous generator, etc.
The disadvantage of this technology until recently was the lower sensitivity of the arrays and the limited adjustment of the accumulation time (shutter).
Recently, however, the new technological advances in the development of the CMOS arrays have largely overcome these shortcomings and have led to the emergence of modern sensors comparable to the CCD arrays in sensitivity and noise level, which, together with a large frame frequency, make them indispensable for a huge number of applications.
The family of high-speed CMOS sensors with a frame shutter has a resolution from VGA to 20 million pixels. The sensors consist of arrays of conveyor-type pixels with a «frame» shutter, allowing exposure during the reading of the previous frame, as well as performing double correlated sampling (DCS), which significantly reduces the proportion of noise and dark currents in the useful signal. A distinctive feature of modern CMOS sensors is their ultra-high frame frequency at full resolution and the possibility of its increase through partial reading, window mode and sub-sampling mode.
The readout circuit consists of LVDS digital serial outputs. The sensor includes an amplifier with a programmable gain and has the ability to adjust the offset. All settings are typically made through the SPI serial peripheral interface. The internal clock generates the synchronization signals necessary for reading and controlling the exposure. External exposure control is also possible. Depending on the model, the sensors support 8-, 10- and / or 12-bit ADC.
Distinctive features of the CMOS sensors:
• high frame rate;
• ability to highlight several areas of interest;
• built-in PLL (phase locked loop);
• built-in temperature sensor;
• built-in clock generator;
• SPI interface;
• monochrome and color versions.
The CMOS sensors are widely used in the following areas:
• machine vision;
• high-speed control;
• television broadcasting;
• aerial photography;
• space / astronomy;
• CTCV.
For comparison, we give the characteristics of the most high-quality arrays created by CCD and CMOS technologies (see Table 1).
The KAI‑02050 array was created using CCD technology, it has 4 outputs (to ensure a high frame rate) and, therefore, requires four ADCs, many drivers for controlling the phases of the array, and several power sources for the operation of the array. Furthermore, its sensitivity is inferior to the EV76C560 array.
The EV76C560 array (by E76V) when tested, showed the following results: when the illumination in the array plane is 5 · 10–3 lux, the signal-to-noise ratio S / N = 4 with an exposure of 40 ms. This array also has a possibility to use 2 different methods of reading the signal from the array – the so-called Global shutter and Rolling shutter.
The Global shutter is the reading of the charge from all pixels of the array at the same time, and the Rolling shutter is the sequential reading of lines of the array. Therefore, if there is a moving object in the frame, the exposure of the Global shutter type will be a «slur», the value of which depends on the speed of the observed object and the exposure time, and with the Rolling shutter type of exposure, the object will be distorted geometrically, since its different parts will be read at different times.
However, with a rolling shutter in a CMOS array, the noise is substantially less (signal-to-noise ratio S / N = 8 at illumination of 5 · 10–3 lux and exposure at 40 ms). The reading modes are switched by commands via the serial interface with the array during operation.
Also, a binning mode was introduced in this array (combining four adjacent pixels into one) for the first time for arrays manufactured using CMOS technology. This provides in low light conditions increased sensitivity by reducing the spatial resolution by half.
The sensitivity level of this array in the binning and the Rolling shutter mode approaches the characteristics of the most sensitive CCD arrays (e. g., SONYICX429ALL arrays).
Our tests performed using CMV4000 array by CMOSIS showed that the sensitivity of this array is somewhat inferior to that of the EV76C560array, besides, there is no binning mode and Rolling shutter in this array, but this array allows forming an image with a frequency of up to 180 fps.
Based on the analysis of the above materials, it is advisable to use the CMV4000ES‑3E5M1PP CMOS array and color CMV4000ES‑3E5C1PP array, respectively, in the television camera for the hyperspectral channel. Tab. 2 shows the main technical characteristics of CMV4000ES CMOS-array.
The analysis of the structural diagram of the television camera
Below is a description of the television camera developed by us for operation within a hyperspectrometer.
The structural diagram of the television camera (TC) is shown in Fig. 7
The television camera consists of the following modules:
• video sensor;
• digital processing module (DPM);
• power module.
The video sensor is an array photodetector (APD) operating in the visible wavelength range of 0.4–1.0 µm. The CMOS array is used as a video sensor (CMV4000, format of 2048 Ч 2048 pixels by CMOSIS (Belgium)).
The CMV4000 array video sensor transforms the image from the photosensitive surface of the array into a digital data array, which is then passed to the digital processing module.
The digital processing module (DPM) provides the following functions:
• input of digital data from the video sensor;
• processing signals from the video sensor and converting them into the required image format when transferring a pixel to a pixel;
• formation of the output digital video signal output with a frame rate of 50 Hz in 5 LVDS pairs using the Base Camera Link protocol;
• formation of a standard analog video signal according to GOST 7845–92 on the video output under load (75±7.5) Ohms for technological purposes;
• adjustment of the time of accumulation of video information in manual and automatic mode;
• ability to change the position and size of the video window;
• built-in readiness control;
• additional image processing.
The DPM is made on the basis of FPGA CYCLONEIII (ALTERA), which provides a flexibly programmable logic control of a television camera and the necessary signal processing. Reduced consumption compared with previous generations of FPGAs (CYCLONEII) provides the necessary thermal regime of a television camera.
The FPGA implements all types of signal processing, data buffering in RAM, digital video sensor control signals.
The DAC control is implemented on the FPGA to output an analog television signal according to GOST 7845–92, digital video signal over 5 LVDS pairs, using a protocol in the CameraLink format. The FPGA also receives control information from the mating equipment and provides telemetry via the CAN2.0b interface.
The power module, located in the chamber, forms the secondary power sources for the power supply of the FPGA, array, RAM, ROM, ADC, digital and analogue parts of the television camera. The power supply module provides output stabilized voltage to power the components of the television camera at an input supply voltage of 22–29 V.
The power module is galvanically isolated at the input, the common mode filter and output filters. The main technical characteristics of the television camera are given in Table. 3
Interaction with mating devices
The output information of the TC is digital and analog video signal of an image, output over LVDS5 pairs using the Base Camera Link protocol and GOST 7845-92, respectively. Analog video signal is used for technological purposes.
The control of the TC operating modes and the output modes of the video information is carried out by external commands (signals) received from the CAN2.0b digital interface using ISO‑11898 (It is possible to control the operating modes of the TC via the digital RS‑485 serial interface). Information on the state of the TC (readiness signals, health, etc.) is output to the opto-electronic system via the CAN2.0b digital interface.
In order to form the output video signal in the external synchronization mode, clock pulses with a repetition period of at least 10 ms are sent to the TC from the external equipment.
The LVDS (Low Voltage Differential Signaling) interface uses differential signaling with low signal levels. Most commonly, LVDS transmitter and receiver are used in a point-to-point configuration.
The LVDS interface provides high speed data transfer. The amplitude of the differential signal is 350 mV, which makes it possible to make sharper fronts of signals and a theoretically possible data transfer rate of 1.923 Gbit / s in a lossless environment. This interface is insensitive to common mode pickups up to ±1 V to differential inputs. Since LVDS uses the current switching mode when transmitting information, it is necessary to pay special attention to power consumption and electromagnetic pickups on adjacent buses when designing devices, which is the price of speed.
The transmitter controls the differential line. A marking impulse with a current of 3.5 mA is output to the line. The load lines are parallel-connected differential LVDS receiver and 100 Ohm resistor. The receiver itself has a high input impedance, and the main signal shaping occurs at the load resistor. When the line current is 3.5 mA, a voltage drop of 350 mV is formed, which is detected by the receiver. When switching the direction of the current in the line, the polarity of the voltage across the load resistor changes, forming the states of a logical zero and a logical unit.
The transmission scheme used has a number of advantages. The LVDS interface has a high data transfer rate. The maximum speed determined by the standard is up to 622 Mbit / s, which is comparable with the optical interface. At the same time, the actual speed of the LVDS channel is limited by two factors: the feed rate and data retrieval rate and the transmission medium parameters.
The differential signaling method used allows minimizing the effects of external interference. A small voltage change, symmetrical transmission lines and a low voltage rise (1 V / ns) cause a low level of interference from the interface itself. In turn, the low level of crosstalk and low sensitivity to external pickups make it possible to use a high density of connections both on the printed circuit board and in the cable. The LVDS transmission method does not depend on the supply voltage and operates at the transmitter and receiver supply voltage of 5 V, 3 V or 2.5 V. At the same time, pairing devices with different supply voltages is not a problem.
The transmission method used also makes it possible to significantly reduce the power consumption of the interface. At 100 Ohms the LVDS load with a voltage drop of 400 mV is 1.2 mW. For comparison, the RS‑422 interface has a power dissipation at a load of about 90 mW, which is 75 times more.
The simplicity of the termination signal LVDS-lines should also be noted: this requires only one resistor for each pair, and that“s it. For comparison, other high-speed methods require a pair of resistors, one of which is connected to ground, and the second is connected to the power supply, and often not directly, but through a special source.
The CameraLink digital interface, based on the LVDS interface, is a serial data transmission protocol designed for machine vision tasks. The frequency range of the Camera Link interface allows you to work with cameras with the resolution reaching megapixels, and the speed reaching several hundred frames per second. For cameras designed for a channel capacity of 100–850 MB / s, a standardized Camera Link interface is recommended.
Since Camera Link was designed specifically for machine vision cameras, it guarantees a very high video speed, ease of use and data integrity. The Camera Link interface has three configuration options, Base, Medium and Full. The speed for different configuration options: Base – 255 MB / s, Medium – 510 MB / s; Full – 850 MB / s [4 –6].
Informational interaction of TC with interfaced systems in the composition of the hyperspectral camera is made out by appropriate communication schemes and interaction protocols defining the interaction logic, communication parameters and signals developed and coordinated during the design phase of the design documentation [4−6].
Power supply and energy consumption evaluation
The TC power supply is carried out with a constant voltage from 22 to 29 V from the power source of the main object. The ripple factor in the power circuit from 22 to 29 V is not more than 7.4%. The power consumed by the TC from the power source is not more than 10 W. The TC has protection against overloads in electrical circuits and short circuits, in case the power supply circuits are not properly connected to power supplies.
Readers feedback
