The use in business of methods related to the discrete Fourier transform has significant potential. A limiting factor in realizing this potential is a high methodological entrance barrier.
The main focus of the work:
- data requirements for correct Fourier approximation of time series;
- the validity of expectations from the forecasts;
- a small set of harmonics is sufficient to approximate a complex series;
- what is a Fourier event;
- how and how Fourier events can help business;
- Fourier events in cash-flow analysis.
1. Forecast
The task was set by a large sea carrier and concerned forecasts for freight prices by ship types. The carrier had a series of subscriptions to forecasts of international analytical companies, but the quality of forecasts did not suit him. Analyst companies used multiple regression, had long-term statistics, and continually increased the dimension of their models. At the same time, they themselves admitted a fairly large percentage of errors in their forecasts.
The criterion for assessing the success of the new forecast was as follows: a fragment of historical data is given, a forecast is formed and the accuracy of the forecast of the already accomplished future is calculated. Eventfulness immediately became a clear methodological problem. If the United States did not sell oil at all until 2017, and then immediately became a leader, then how can this affect the conclusions based on historical data. Other events: wars, crises, - from the point of view of the forecast, are essentially the same events, but the situation with oil exports from the United States is extremely indicative in order to dismiss the event factor in the forecast methodology (weights produce linearity, and events produce a gap and singularity) ...
Many methods have been tried. The most interesting was the Fourier series approximation (Fourier approximation) of time series and its study from the point of view of forecasting for business. At the same time, there was a technical problem - all the time there was a shift in the approximation from the original series.
2. Formation of data for Fourier transform
Necessary preliminary explanations.
Discrete Fourier transform is applied to vectors consisting of real values. If a time series is viewed as a set of <time-value> points, then the Fourier transform is applied to a vector from a sequence of time series values.
There are subtleties of using the Fourier transform, which are associated with the number of values and characteristics of the intervals between them. For example, the original time series may have uneven intervals or missing values for certain time positions (weekends, holidays).
In many cases, the following procedure is helpful. The original time series is first interpolated, and then the required number of values at the desired time positions are taken from the interpolated function. Thus, the original time series is replaced by a more frequent regular series with the required number of interpolated values.
The following is the approach described by A. Dieckmann.
Discrete Fourier transform.
A vector of real values u = u [r] is transformed into a vector of complex values f [s] using the following formula (there are several formulas for F [s, r] that give equivalent results): f [s] = u [r] * F [s, r], where
, and the values of s, r vary from 1 to n.
Necessary data for obtaining the Fourier spectrum.
The resulting vector f [s] can be interpreted as a Fourier spectrum, since it contains information about the amplitudes, frequencies and phases of the fundamental harmonics.
Moreover, there are requirements for u [r]. The u [r] values must be specified at the split points of the interval when the interval has the length of an integer number of steps of the same size. The r value corresponds to the position (index) in the vector. In general, r defines a position in time (time series) or space (in other dimensions).
Suppose that the vector u [r] must be defined on the interval [tMin, tMax], the length of which is tt = (tMax-tMin). Let delta = tt / n correspond to the distance between adjacent points of the interval at which u [r] is calculated.
Let's consider what process should technically take place.
The complex exponential in the matrix F [s, r] can be interpreted as a vector probe (depends on s) that rotates in the complex plane with a frequency (s-1) / tt and moves sequentially (in time or space) along (r- 1) * tt / n. During matrix multiplication, the probe vector corresponding to r is multiplied by the specific u [r], and the vector sum is calculated over all r, giving the complex number f (s). And so it is repeated for all s from 1 to n. Each f [s] indicates the presence or absence of a component oscillating at the frequency associated with s.
How should u [r] be formed?
At this point, the original time series should be interpolated and displayed at the selected interval, a multiple of an integer number of steps. A sufficient number of points for an accurate approximation is chosen empirically.
At this stage, the main thing is how many points to take and which ones. The value of n is fixed to delta. In this case, we have a set of n + 1 points for all values of the interval partition.
In u = u [r] it is necessary to include points only from the first to the last but one, but not the last: only n.
Otherwise, the Fourier approximation will be slightly shifted relative to the original time series.
3. Visual interpretation of Fourier transforms
For the widespread use of the Fourier transform in practice, it is necessary to feel what it gives in addition to complex formulas, and correctly form the initial data.
Consider how the Fourier transform acts on a sinusoidal function. For this, it is useful to combine on one graph the behavior of the function and the characteristics that the Fourier transform gives at specific points and in general on the function under study.
Consider the function 1 + Sin [2πx] on the segment [0, π].
The amplitude of this function corresponds to 1Hz, since it repeats its movement after 2 π.
Let n = 20, then when dividing the interval into equal parts, you can get 21 values at the corresponding points of the split. But, following the above explanation, we will operate with only 20 points - without the last one (only black in the picture above).
The r parameter advances along the abscissa and has 20 values. The s parameter defines the speed in (s-1) Hz.
The following figures show the rotation of the probe vector. Each vector probe starts at the point u [r], for which the value F [s, r] is calculated. The parameters of the end of the probe vector are obtained as follows: the abscissa is the product u [r] * Re [F [s, r]], the ordinate is u [r] * Im [F [s, r]].
For clarity, a palette of moving probe vectors from beginning to end has been selected. It starts from brown, then through green to blue:
The figures below show the rotation of the probe vector reduced to the point on the graph for which the Fourier transform is calculated, as well as the path (vector sum) when adjacent probe vectors are directly adjacent.
The ordinate axis displays the amplitude of the original function and the imaginary part of the Fourier transform.
The abscissa axis is the position of the point on the time intervals of the original function and the real part of the Fourier transform.
The sum of the vectors shows the configuration of the motion of the probe vectors. The black dot denotes the start of the movement and the end of the movement (another black point if they do not match). For s = 3, the start and end are the same. For s = 1 and s = 2, the start and end do not match.
Start and end coordinates are shown separately, as well as rounded values (very close to zero).
The s value characterizes the tested frequency.
There is symmetry in behavior.
The center is s = 11.
For an example of symmetry, let us show the figures for s = 19 and s = 20, which are symmetric to s = 3 and s = 2.
What happens if you take not 20 points, but 21, including the last one. Example for s = 3. It shows the presence of a component that oscillates with a frequency associated with s = 3, while there are no such oscillations in the original function. There is only 1Hz fluctuation in the original function.
All of the above plots are intended to show the importance of correctly splitting the intervals and sampling data for these intervals without the last value. Only in this case will there be a correct Fourier approximation of the original series and the possibility of its periodic continuation.
The rest of the Fourier approximation aspects are presented in the reference literature quite fully.
4. Analysis of real time series
Let's return to the task that was described at the very beginning.
The following is a Fourier approximation of historical data on shipping prices for shipping.
In each picture in the first block on the left, there is a graph showing at what level of amplitude values (red dotted line) harmonics that make an insignificant contribution are cut off. The first block on the right shows the characteristics of the first 10 harmonics of the approximating series in decreasing amplitude.
The second block consists of plots with increasing the number of harmonics (in order from the largest amplitude) used for approximation. The result of the approximation is a red dotted line.
For a given time series, 5 harmonics are sufficient.
For this time series, you can limit yourself to 5 harmonics, if you do not consider very old data too important.
This time series is quite well approximated by the 8th harmonics.
In this case, it is desirable to take into account 11 harmonics.
Thus, historical data in a rather dynamic field of activity (prices for ships for sea transportation) can be well approximated by an average of 10 harmonics.
In general, the problem of forecasting, when the already existing future is restored from a fragment of historical data with some error, can be considered solved if the fundamental (some) harmonics of the approximation are known.
At the same time, it is clear that the forecast for the future that the Fourier approximation will give will in fact be completely wrong: this becomes clear due to the transparent mechanism for constructing the Fourier approximation.
With multiple regression, when we talk about 70% of the forecast reliability, everything is the same, but the opaque construction mechanism allows us to unreasonably hope that, in general (70%), the forecast will be correct.
5. Fourier events
Fourier events appear on the assumption that basic cyclical processes (harmonics) take place, which are superimposed and combined with important events, also represented by harmonics.
Thus, all harmonics of the Fourier approximation are divided into two parts: the basic harmonics of the process and the harmonics of events. It is important to remember that the sum of the basic and event harmonics gives an adequate approximation of the original series.
In this case, for a good forecast, it is enough to know the basic cycles and have a list of events and circumstances, according to which a rolling forecast should be formed based on expected or already occurred events or their chains. But this is a slightly different, not traditional technology of forecasting.
The following two methods for fixing Fourier events are methodologically justified.
The first method is associated with the subtraction from the complete time series of all possible combinations of harmonics that approximate it, and the comparison of known events to the resulting extrema or stable deviations. Since almost all industries have analytical companies that collect statistics and review trends (in some industries even weekly), finding important events for a date is not a difficult enough problem.
The second, let us call it the splitting method, is associated with dividing the complete time series into periods of different lengths and searching for "similar" periods by comparable harmonics. With the described approach to the Fourier approximation, such a task can be fully automated.
The splitting method is qualitatively different from the first method, since there is a nonlinear for the entire split the initial operation of isolating its trend (linear regression) from each component of the selected split.
6. Data analytics for oil prices through Fourier events
For example, consider the oil prices Europe Brent Spot Price FOB. Source: Thomson Reuters. US Energy Information Administration. Thomson Reuters. Data are daily in US dollars from May 20, 1987 to November 10, 2020.
Original time series.
We select a trend - linear regression.
We clear the initial data from the trend (a linear trend can always be restored).
Blue graph - raw data. Black is a trend. Orange - normalized data (no trend).
Find an approximation.
So far, everything is not very good: the 8th and 20th harmonics for such a series will be small.
For 30 harmonics, the result is quite acceptable.
Let's move on to the method of isolating Fourier events. Let us illustrate one of the approaches for the case of approximation of the original series by 8th harmonics.
Find all possible combinations of 8 harmonics. There will be 255 of them. For each of the 255 combinations, we calculate the absolute value from the point difference between the original row and the structure (row) generated by a specific combination of harmonics.
For a new series, we calculate the maximum, standard deviation and the total sum of values (it is possible that other indicators need to be calculated: mean, etc.).
In the figures, these indicators are shown sequentially. They correspond to the first hundred values, sorted in descending order of maximum.
Let's consider the first 60 of the selected 100. And then we will choose (visually) interesting ones. The graphs are shown below. The number under the picture corresponds to the ordinal number of the combination of 255. The gray graph is the original row, the red row is from the combination of harmonics.
What counts as "interesting" is a meaningful task for a business. Everything that has been there so far is just standard technique.
What happened in the end? From the set of harmonics that approximate the original series well, we have selected combinations that in some areas correspond very well to the original time series, and in others show a clear discrepancy. Just the last sites are candidates for the analysis of events that took place during this period (all charts are daily with an explicit date).
In addition, the presence of very well-adjoining areas of the graphs provides a basis for deriving the characteristics of the "norm" for the dynamics of the reflected processes.
The goal of the analysis is to identify harmonics that correspond to events. The inverse problem is the selection of basic cyclical processes.
The splitting method is important because the process represented by one time series can be essentially composite and depend on events of a higher order: a global crisis, etc.
7. Fourier events in cash-flow analysis
The process of analyzing a time series is associated with the expectation that cyclical processes dominate in it. In general, such expectations may not be met. The point is not even that there is no such dominance of cyclicity. Simply because of the particular way in which the time series is formed, it may not be possible to identify cyclicality in that particular series.
Cash-flow is another matter. As a matter of fact, production and commercial activities, most of the processes are initially cyclical in the way they are formed. Deviations from the norm are associated with events that violate this cyclicality. The use of the Fourier event method in the analysis of cash-flow makes it possible to identify an objective “norm”, as well as indicators of deviations.
In terms of Fourier events, the cash-flow analysis problem is well-algorithmic for applying artificial intelligence and neural networks (machine learning) methods.