Jump to: Step-by-step Guide
(A level of mathematical knowledge is recommended to follow the steps.)

Some fundamental “insight” into market trading and investing is still lacking. Over the last 120 years, various trading and investment strategies have appeared, but most funds, hedge funds, indexes, and stock prices exhibit erratic behaviors. On the home page of this website, we point out clues why these erratic behaviors exist.

Additionally, Wikipedia lists some different schools of thought on algorithms for predicting stock price. Nevertheless, most of these contemporary ideas are likely to be superseded with new concepts.

Curiously, entities in finance, fiduciary, or business roles do not have the tools to solve the persistent erratic returns problems. Technical computing and robust mathematical methods are rising around the world. And, some researchers are making significant discoveries. The venue of proficient modeling and prediction is with entities who are experts in signal processing, engineering, and applied mathematics. In particular, these entities study the data itself to learn its properties, and what is predictable.

This website posits a new viewpoint on real market physics. These thoughts are based on 35 years of experience with data exploration, extracting information from noisy data, and utilizing advanced modeling methods.

Through previous posts we have shown that the stock market has some random noise in the prices. We have also shown that the price energy varies roughly as a linear function of frequency over a certain range, and that this characteristic is often observed in systems which have Brownian motion.

About one third of the way down the page, Brownian motion is shown below the caption “White Noise and Brownian Noise in a Fourier Transform.”

We review. Brownian motion is the linear relationship that has a slope of minus one, plainly visible on the log-log scale. For comparison, Gaussian random noise appears as horizontal noise in the frequency spectrum. Brownian motion implies the possibility that some organized components in the energy might exist, wherever the slope is minus one. Thus, the trick is to search for these components, which are often buried in the spectrum.

Individual stock prices have distinct personalities. Yes, a collection of stocks might exhibit similar behavior, and sometimes they might be closely correlated. These cases are temporary. Each stock or index is a unique series with individual characteristics. We  examine the characteristics of each persona. This means we might examine correlations on a variable. But, we do not use correlations between series as an algorithm to “predict.”

Correlation is tricky. It is easy to suppose that there can only be 1 correlation number between two time series. But, we have shown an example where two time series might have a correlation of .96 at one sample rate, and a correlation of .08 or less at another sample rate. In other words, the calculated “correlation” value depends on the values of each data set, plus the sample rates of the data, and the lengths of the data sets.

What about cause and effect? In reality we simply do not know the inputs to stock prices. The sources that move the prices can arise from anywhere on the planet, and thus collectively form a multidimensional input set, that is invisible to us, and is changing from instant to instant. The only thing we “know” is that during the daily session, participants in the herd are generally trying to “figure out what will happen next.”  They “figure out” in different ways, depending on the school of thought they are using.

Indeed, herd dynamics is relevant in the stock market. The word “herd” generally refers to groups or flocks, such as schools of fish, herds of sheep, or massive flocks of starlings. Our resource Visual Nature Examples reveals characteristics of herds, such as “flow” or “pulsation” or “freezing” or “synchronized movement,” etc.

Remember, moments ago, we said that sample rate and “correlation” are related? We now mention sample rate again in another context related to predictability. What we “see” in dynamic systems is affected by the sample rate.  For example, the video “F19 Mouse Decomposition Maggots” (in Visual Nature Examples) reveals organized activity.  We “see” this because the filming time lapse rate (sample rate) is chosen so that humans can see the flowing movement, the pausing, the group vibrations, and then see the cycle repeating again and again. If the filming time lapse rate was different, these distinctive effects would not appear so clearly and plainly for us to see.

Next, we examine humans as a herd, affecting prices. Professor Shiller wrote a fun paper that  describes “Behavioral Finance,” a school of thought that seems to fit the market physics. On page 86 of that paper, he shows a graph that visually illustrates the erratic price behavior that we call “volatility.” The conclusion? Human behaviors create the volatility.  The jiggles are caused by the multidimensional input set of invisible forces, such as self-interest-based secrecy, arbitrage schemes, pairs trading, hedging concepts, speculation, outright manipulation, market makers who stop making a market when they perceive that the available market funds are drying up, and hundreds of other possible causes. And, the forces keep changing from moment to moment.

On the subject of “volatility,” the main page of this website lists three theorems near the top of the page: The Cobweb Theorem, System Dynamics, and Behavioral Finance. All of these ideas aptly describe the human causes of unstable market dynamics.

In addition to herd effects, and invisible inputs, data errors and low precision will affect prediction accuracy. For example, a moving average at -40 dB has a precision of two significant figures. Consequently, we can expect only a sloppy approximation from that. Meanwhile, a typical stock price has at least four significant figures, and up to seven figures when we also include indexes! Inconsistencies of this kind, in trading algorithms, makes “prediction” extremely hard to accomplish. We will return to precision near the end of the article.

Our Dynamics References quote various experts verbatim, who point out the great difficulty in using statistical methods for chaotic, dynamic systems. Also, verbatim quotes from the mathematician Edward Frenkel tell us exactly what we need to do to make good predictive models:

• Mathematical insight.
• Asking the right questions.
• Building a model.
• Not Statistical analysis.

To make good predictions, we cannot simply copy some popular algorithm code from the Internet that everyone is raving about. We must think about fundamental principles. Explore the data, look at it. Learn the truth from the data. Find something there that is useful. Ignore the stuff that is not useful. We know from experience that relationships in the data can be found that are not “recognized” by the experts. Yes. That is what we want to find.

Accordingly, we must make an adaptive model that predicts. Adaptive, AND not knowing the inputs.  An honest model test should run from about 2005 to the present. Good quality data. Watch out for data errors.

Once again, the sample rate, and what we “see.” The Rolling Shutter Explained video compares what we “see” with a cell phone camera versus what we see with an accurate camera, when viewing whirling propeller blades. Watch the video a few times (Super cool!). Then go back and watch the maggots, the fireflies, and the dogs herding the sheep. The key idea is “What prices do we pick, and at what rate?” The answer depends on what we wish to track.

Back issues of the Wall Street Journal provide great “insight.”

For example, the terms “support” and “resistance” were common in the 1900s. In other words, these ideas are 6 generations old. Ancient. If we analyze the pattern Open, High, Low and Close, for many years, we can see specific combinations and permutations. They break into distinct classes. The so-called “formations” of “support” and “resistance” are merely inevitable patterns in the set of possible permutations of O, H, L, C. That is all. Nothing more. However, because some market players believe the “formations” of “support” and “resistance” are meaningful, we do sometimes see changes in price level, or increases in volume, etc.

Other patterns in the set of permutations are more worthy of attention, such as the range between the High and the Low for the day. A year of daily data is a minimum, 5 years is better, and longer is even better. Calculate it as a percent change, sort it from small to large, and then plot it in Matlab. The plot shows us what is actually possible to achieve, from small margins to large margins, in a single day. Next, pick a desirable annual return such as 30% and then divide it by the number of trading days (such as 252), to obtain an average return per day (such as .119 percent per day). Compare that single number with the plot. The plot shows us that most professional traders are at the small margins range. It is quite a shock to see that graph. And then to realize the discrepancy between the  potential versus the actual achieved in real life by most professionals.

Wall Street Journal also reported “combines.” They were common from the 1920s to the 1940s.  Many schemes. The word “combine” refers to collusion of separate entities which work secretly to deliberately manipulate stock prices and indexes to create profits for themselves. Today the Internet unloads propaganda and disinformation in a torrent, which clearly affects security prices, indexes, and the masses. Thus, not much has changed.

Huge quantities of old, ancient ideas are still in the markets. Belief systems. Gurus. Garbage.

Here is a step-by-step suggestion for better precision insights of stock prices. A level of mathematical understanding is needed to follow the steps.

Set up the code for the classic Lorenz model and initial conditions. Then run the same model at two different precisions, such as 8 and 12 significant figures. Observe that the two series look the same for N time steps. Then, after step number N, the two series suddenly look different. For each series, calculate the mean and variance from 1 to N, and then from N+1 to the end.

For both precisions, we see the means and variances are different, particularly in the second set.  The point is that we know the original source equations exactly, and we know the initial conditions exactly.

The calculation for the means and variances depend on significant figures. Got it?

We might think further and deduce that concepts of mean and variance, in a chaotic system, are probably meaningless. We have no convenient Gaussian frame of reference.

In dynamic systems like the stock market, when the governing equations are unknown, we can expect the system dynamics to change over time, attributable to invisible forces.  If we focus on predicting upcoming changes in the short term over some short range of values, we  have a decent chance to predict something reasonably well if our algorithm is designed to calculate a net accuracy of about 5 to 7 significant figures.

If we do not do it this way, then we are in the “volatile” crowd. And, we are blind.

We now return to the discussion of precision and significant figures.

Most ordinary computers have a limit of about 16 significant figures.  A demonstration of machine limits is shown in Fig 3 of our article comparing two different filter designs. The Pascal filter response is cut off at -300 dB, due to machine roundoff effects.

How do we obtain more significant figures? We show a link on the front page of our website to Advanpix, which makes the “Multiprecision Computing Toolbox for Matlab.” A bunch of Universities are listed on the right-hand side of the page. They understand the importance of significant figures. Also, Figure 1 and Figure 2 demonstrate that even Matlab, a famous science and engineering package, will fail under certain computing circumstances, because it does not have enough significant figures.

In a chaotic system, precision matters. If our prediction is incorrect because of precision limits, no flashing neon sign will pop up to inform us. We need to keep a close eye on what we are doing and verify our assumptions as we go along.