No announcement yet.

Trading systems design.

  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    Data and Reverse Testing

    There is a point of view in trading systems that it takes at least 30 trades from a dataset to make a system worthwhile. In my opinion, this is mostly nonsense. I feel this especially strongly with short-term trading systems, and especially the daily markets (which we will deal with later in this series). Over 100 trades is a good number to see from the test set and one that I believe is likely to produce a potentially survivable system.

    One of the great aspects of building a trading system is that we can now use personal computers to test the results. One of the biggest weaknesses of traders diving into systems trading for the first time is to compose an idea, then test it in data, adjust the breaks for a while, and finally optimize it to produce the best possible system. Unfortunately, in this respect, the best possible system will actually be just an excellent facsimile of past data, curve fitted in the main. Reject it immediately and start again - unless you really think history will repeat itself down to the last notch!

    When doing reverse testing, it often makes sense to test self-correcting systems first without first stopping. You can then examine the fairness of the entry signals to a greater extent, before developing further with testing the process. However, with a non-self-correcting system, it is essential to start testing with a stop or exit policy right away, otherwise the risk of failure with this system is too high.


    • #17
      The dataset used for the test will of course have a big impact on the results. For example, parenthesized data akin to a tight-stop system will cause a lot of whipsaw losses. Also, remember the warning in an earlier article in this series about using daily data for the most accurate test results when trading tight stops that are smaller than the last trading area. Equally, a free stop will prevent such common small whipsaw losses (which are all cumulative, remember).

      Of course, big losses are naturally bad things by definition - but they may be the best way to get ahead. Even very small adjustments for tight shutdowns tend to make a large difference in revenue across the system. Modulating large shutdowns, even when modified by several hundred dollars at a time, tends not to make a big difference in gross revenue. Starting with a tight shutdown and gradually letting go, you will see a systemic change in results quickly (hopefully with income increasing!). The effect will then gradually change less rapidly as the stop size gets wider. Once you cross a certain threshold of volatility, an increase in the initial stop size adds a small amount to the overall systemic efficiency.

      Now let's take a look at our data. Here the more we have better. Many traders agree that 10 years is a good minimum and 12 years is preferable. However, this raises the problem of finding derivative markets with ten solid summer trades. While there are an increasing number of contracts with such a database, I could also add that system developers usually avoid the markets during their first 2 years of employment. The point is that markets tend to develop and grow during their first couple of years - at the point where they can be considered mature enough to make sense to take into account. We will discuss the possibility of synthesizing data from real world prices to allow for potentially unlimited testing at a later stage in this series.

      However, those who simply test their system against all available data are doomed to fit for perfection. Preferably, the best method is to split the data. System traders differ in their views on this. Many people divide data into two parts:

      Development kit and test suite. This allows you to use one dataset to actually build the system and a second dataset on which you can test the system, without any modification or optimization.

      I prefer the second school of dividing the data into three identical sections: the 'Development Kit', the 'Back Trial' set and the 'Go Through'.


      • #18
        With the first set of data, we are really developing our system by modifying and optimizing the parameters. Then, after developing the system, we apply a back test, after which we review the results again. If the results aren't impressive enough, then this is the downside on the drawing board. Providing a reasonable profit, which should add up first in two periods, we then move on to the ‘end-to-end’ test finale in the third piece of available data. The advantage is that we can observe the development of one-on-one trading and see how the market actually moves in favor of and against our system.

        'Get through' the test is thus an important prerequisite in system design as it:

        Can give us a significantly better 'feel for how our system operates, thus building our confidence in systemic abilities; and allows us to evaluate independently systemic operations that truly suit our trading style.

        As I mentioned in the opening sections, there is absolutely no point in implementing a system that trades in the short term with high volatility if you are a low volatility trader. The system should reflect your trading strength, not your weaknesses. So if you dislike the stress of extending positions that play against you for weeks at a time, such a system is not for you.

        Likewise, the more sensitive you need to understand how your system is performing, the more likely you are to tackle it during your first drawdown period, or faster. Murphy's Law almost guarantees that your first significant drawdown will begin shortly after you trade in real time with the system. It can even happen before the system has added up any reasonable income, and thus heighten your anxiety by eating up your initial capital. The more confidence you have in the system, the more likely you are to endure such a tough time.

        In terms of splitting data, if I used 12 years of data, 1986-1997, then as stated in the manual, I should split them as follows:

        Development: 1986-1989

        Reverse Testing: 1990-1993

        Pass Forward: 1994-1997

        Of course, once we have a suite of tests, we can examine the results to see where the tested system is performing. Obviously, if we originally designed the system in the dataset during an upward direction, it should come as no surprise that the system performs poorly in reverse (or general disadvantage) markets.

        So you need to use your intellect when choosing a development kit. Conceptually, you need a 4 year period that shows all forms of market movement: up, down, and sideways.

        Optimizing A system that uses a set of data skewed in one particular direction will not produce a robust system.

        Likewise, a system designed on a dataset that reflects very, very low volatility will produce dire results when the market overcomes its bracketed area.

        Whipsaw. A common disease among traders around the world, the term is used to describe the impact of reckless trading in volatility (and often in an unmanageable way) while the market remains trapped in a relatively narrow area. An example is the result of buying up a false gap signal enclosed within the market in some form, only the market immediately goes in the opposite direction, after which the trader reverses his position, but finds that the market is returning to the old trend.


        • #19
          System test results.

          Okay, you still stand, right? You want to skip this and future sections and count the time just to come up with your idea. Well, you know how - you have the right to go forward. When you lose 30% of your account on the system, which turned out to be lifeless, you can always go back and read on. Nobody said that systems engineering is easy - and it is undoubtedly a much more intense activity. This is why so many black box systems of dubious origins are selling so well.
          So if you want to enter blindly at this stage and just trade your systemic methodology, then please feel free in your decisions - there is always a chance that it might work. However, the likelihood that you will just give money to other merchants - like the author of this article, for a word ... Sounds good to me! If you want to donate some cash to the rest of the worldwide merchants, then just do it. On the other hand, you can donate unilaterally to ADT. Send us a check for 30% of your capital and save yourself from boring bickering, stress and strain and really bear all those losses!
          In the meantime, for those looking to tackle the process creating system, welcome to the sequel ...
          In his books, Cyber ​​Trading Strategies, Murray Ruggiero notes that:
          "Testing and evaluating a trading system involves three main questions:
          - How well can the system meet expectations and what are the risks?
          - How likely is it that the system will continue to work in the future?
          -Can we predict when and whether system execution will temporarily degrade or fail completely?

          “We will be looking at all of these aspects throughout this and future articles in the series. However, having arrived at a set of variables for our system, let's start with a simple test. The end result will be a lot of trading information, of which the amount of profit is only one important metric. in fact, a very good starting point is to look at the capital line. The capital line gives a graphical representation of how the system develops over the course of a trading life. An ideal system would have a nice smooth line gradually rising from the bottom left of the chart all the way to the top right side. very jagged peaks are not a viable system, as this implies a system that is not only volatile in income, but also relatively inconsistent in its overall revenues.
          Most good modern systems include a lot of statistical calculations of test / trade results. For example, the main areas to really deal with are commissions, slippage and margin requirements. In the case of the latter, keep in mind that in volatile markets, margins can change daily, as well as from day to day, weekly, etc. The margin requirement used during the system test should therefore reliably reflect what margin is likely to be in periods of volatility. Otherwise - in the real world - you may find that you are unable to use your system to trade due to excessive margin requirements. Likewise, in the sliding of the commission rate, one should avoid the risk and use the highest incremental rate.
          Other system statistics will include the following essentially self-explanatory result areas:

          1. The number of periods in the test.
          2. Total net income, gross profit and gross loss.
          3. Total number of trades, percentage of profitable ones, number of wins and failures.
          4. The most profitable and unsuccessful trades.

          5. Average snug and unsuccessful deals.
          6. Average trades (gains and losses).
          7. Maximum consecutive wins, maximum consecutive failures.
          8. Average the number of zones on winners and losers.
          9. Maximum daily drawdown.
          10. Profit indicator - the coefficient of victory in trading over a loss in trading, sometimes expressed as an index from -100 (very bad) to +100 (very good).
          11. Trading history (from deal to deal).


          • #20
            So, after our first test, let's take a look at some simple risk-adjusted analysis. There are three metrics that combine to create a trading result:
            By creating a risk-adjusted analysis method, we can most easily look at the maximum drawdown. If this is more than the amount of capital that you must invest in a systematic performance, then this system is not beneficial to you - at least not in its current format. Oh, and never be tempted to trade a system with less capital than the maximum of the implied drawdown. Let me guarantee you the end result will almost invariably include tears.
            One very simple measure of risk-adjusted performance is to calculate the net drawdown benefit / maximum. System development pioneers prefer to use the result for this calculation of 10 or more in order to verify the potential suitability of the system. Currently, the more popular measure of risk-corrective analysis, often used by CPOs and CTAs, is the Sharpe Ratio.
            Sharpe Coefficient is defined as:
            (Average Revenue - Risk Free Revenue) / A Standard Deviation Revenue
            The higher the Sharpe Ratio, the better the system, as the revenues are more sustainable. Where two different systems offer the same cash return, the one with the higher Sharpe Ratio has less risk.
            Another aspect to consider, before moving on to optimizing, is how many tests we want the system to vouch for. Now, while this is not a reckless pursuit of many tests (perhaps thousands) with potentially hundreds of proofs of profitability - and thus underpinning the robustness of the machine - can hardly be ignored. Consider the table below, which shows how a bunch of dough on a dough will significantly increase the amount of computer time it takes to return a test number:

            In other words, testing thousands of variables, hoping to achieve hundreds of profitable system results, it may not be desirable to test 5 indicators and more than 25 values, each may simply overflow. Even if your PC had huge memory reserves and a cool processor ...
            As a starting point, it is best to perform several tests initially. Then, when you start to find a trading system that looks like a nice promise, gradually increase the number of variables you are testing. Simply logging in with many variable parameters at any whim will simply take too much computer time (let alone your own latency) to be productive.
            And don't forget, once you've finished optimizing, when you move the system to Backtesting (and ultimately Walk Forward) data, there should be no further system modification. fit is the scourge of your research!
            Finally, for now, let's look at a couple of parameters convicted to be an advantage in evaluating a trading system. Larry Williams prefers an average profit per trade of more than $ 250, after considering a prudent amount of slippage and commission. This is a fine reasonable standard for its relatively short term trading. However, Williams also loves if his systems win more than 70% of the time. (Still, at least he's honest enough to admit the personal need to win that often.) However, really amazing systems - like many really good traders - can only make money on 40% of the trades.


            • #21
              What the test really tells us

              "The test results are not quite what they seem. You learned that a trading system is designed with an edge over hindsight. This is true because you know definitely how the market has behaved in the past. Any trading system you design or optimize reflects your a general view of past market action. You can point your understanding on a generalized path that will help avoid the dangers of curve-mounting. However, it makes sense to learn how the impact of flashback is difficult to eliminate. "

              So, you have a system, and a few results that tell you that it is profitable. So are you ready for rock and roll? No!
              At this point, you have a rough system and a very rough set of results. First, let's take a look at a few more numbers that system developers take into account in their methodologies.
              For example, there is the Payoff Ratio. It provides a revenue ratio for the average winning trades to the average losing trades. It is a useful marker for analyzing systems as it provides some indication of how the system is performing in general. This is good for building confidence and, as you test other systems, a useful indication of how they perform relative to each other over time. In part, similar to the legend about the turtle and the hare ...


              • #22
                Basically, whatever we're showing here, while the Outcome data is very useful, it doesn't remotely tell the full story about any system. For example, one metric that can be very useful for calculate is Recovery Factor. This is the absolute value of the ratio of net profit to the maximum of the daily drawdown. It is also a measure of how quickly the system recovers from deep drawdown. The Recovery Factor. Must be greater than 2 to indicate a survivable system. Obviously, the higher the number, the better.

                When we come to an analysis of your income, a terrifying trap may be where the market has accumulated exceptional gains from a single specific situation (for example, the moment of the 1987 stock market crash). So to ensure that the prudent part of the subconscious montage curve doesn't show an overly ruddy image of your income, subtract your biggest gain from your gross profit. This gives a number that many system developers refer to as 'Corrective Gross Benefit'. Keep in mind that you are not deleting the largest trade loss.
                Systemic backtesting copies, but does not really recreate, past market actions due to a range of weaknesses that we have previously discussed in this series - such as spread, rollover, slippage, etc. In other words, we allow backtesting to have realistic reflection of losses, but do not overestimate the profit estimate. After creating the Adjusted Gross Profit, recalculate the Profit factor and see - it is still more than one. (Profit factor is equal to Adjusted Gross Profit divided by Gross Loss.

                Likewise, keep in mind that when a system has one trade that makes 30% or more of the total return, then it is unlikely to be very important. Rather, it is a system that accidentally captured the main event in its reverse test. Even a single transaction making 20% ​​of the system revenue should, for safety's sake, be viewed with some suspicion.
                Once you have your Adjusted Gross Profit, you can now proceed to calculate 'Real Revenues'. Many system developers simply take the gross profit and split it in half. I prefer to go one step further, though, and divide the Adjusted Gross Benefit by 2. This gives a more realistic number of results that are likely to be in practice.

                Another metric that is most useful for calculating is multiplying the drawdowns by 2. Then, when you do make capital in the system, the chances of suddenly finding a program that cannot run due to lack of capital is greatly reduced. Likewise, no matter what the average margin level was during the trial, for safety's sake, multiply by 3 the highest amount. Moreover, it will keep you safe. Oh, and if you think this is unrealistic, then check out what the clearing houses are doing during the meltdowns ... (And then you can thank us for keeping you safe, or at least still in the game.)
                It is also useful to see the trading of systems visually. In addition to the capital charts described before, it is also a good idea to plot the system's revenues. With a tenacious system, they should form a normal distribution.

                And remember this nugget of helpful information from Tushar Chande:
                "In general terms, you should consider the results of any trading system with all due caution."


                • #23
                  System in operation / Invisible Risks

                  Here we present a list of things of sorts that can lead to loss when applying the system in real time. Some of the issues have already been covered in other articles in the series.

                  For example, it is very important to consider placing stops with your broker at this stage of entering a trade. Many traders choose to hold them back from the broker until they get close to fulfilling. This is good for those who trade in front of a live screen all day long and have sufficient discipline not to cancel them, holding a stop position when the market hits their level. However, for many traders, I recommend placing stops with your broker at the time of entry.

                  This relieves stress while also ensuring that orders are followed. Equally, the introduction of stops immediately after a trade entry neutralizes such risks as:

                  'Bathroom Risk' - the markets always move faster when you visit the toilet;

                  'Telephone Risk' - Once you are consumed with trying to teach your mother how to program her video, the markets are guaranteed to break through your unset stop levels.

                  In the case of multi-system traders covering a wide variety of markets, however, I tend to assure that it is not harmful for your broker to check and re-check orders at the beginning and end of each day trade. Likewise, introducing GTC (Good before Canceled) orders independently one at a time or by a dozen may seem like a good idea, but on the other hand I knew the merchant who gave the GTC order, forgetting to cancel it later, and then sought to backtrack when it was done.

                  In earlier ADT questions, we have spent a lot of time discussing brokerage etiquette - especially in Getting the Best from Your Broker. Also, with stops and systems, it is usually a good idea to ask your broker which order deployment method suits him best.

                  “You, the trader, are perhaps the most bizarre variable in the trading system. Since system trading is performed in an emotional vacuum, there is no guarantee that you will follow all signals from the trading system without rejection. source if you refuse to enter orders if necessary. " Tushar Chande, “Behind Technical Analysis”.


                  • #24
                    Disaster plan

                    If you think your PC / satellite or ISDN connection / info line etc. do not suffer from simple lack of power, people spoiling cables, electrical interference, etc.

                    The point is that no matter how tenacious the stations are (and in the case of electronic stations, they are by now more fragile than “open shouting”), trading time can and will be lost.

                    Generally in London, I experienced about 20 days of downtime (during some part of the day).

                    Having a simple system can be good in this situation, since at least you can try to manually correct it to check stop / exit positioning etc ...

                    Then, of course, there may be a problem with the PC, despite the fact that hardware and software developers are often ready to fall on the rails, promising 100% performance of their products (Windows - SUXX).

                    Hence, having a disaster recovery plan is a great idea.

                    To begin with, we can reduce the possible areas of potential collapse. Try to have your systems on a PC solely for trading. If you are at home and your children are playing on a computer used for trading, then give them their asses and send them to earn money on another computer for games.

                    Also, if you have a long-term lack of data, then think about exiting positions. It is very difficult to make money without price data unless a long upward or downward trend prevails.

                    Secondly, we need to think about how to restart the trade. Do we access the data to restart and then just jump into the market regardless of the price? Or should you re-enter the market when it returns to a certain marker? Where and how long do you actually wait to re-enter? Or are we just waiting for the next signal from the system? It depends to a large extent on the system, as well as the trader and his own psychology.

                    If you have automated the system, then this will make any process simple. In this regard, everything in the system should be automated if possible, even the disaster recovery procedures, which may even differ from point to point, depending on the kind of disaster that befalls your system.

                    And if you really want to have a super system that works well, write a manual about it. Describe the beginning of development and development in detail. Register the optimization tests that are done and, discuss your thoughts on the system from the beginning through time after development. Added to your trader's diary, this guide will serve many roles, ending with disaster recovery procedures. It will be an essential aspect of your trading technique, a great memoire helper and also a very useful tool for future systems development. It shouldn't be very long, about ten pages will probably be sufficient.

                    The headings and categories will obviously relate to your own personal need for a reminder / stimulus, etc., but will likely be in the direction of the following:

                    Introduction - why you want to come to systemic trading.

                    What kind of system are you installing to develop.

                    Your Goals and Expectations from the system.

                    Initial systemic principles, e.g. price concepts, daily, long term ...

                    The set of data used, that is, which markets, data - daily / ending days ...

                    An early systemic test, its results.

                    Initial Optimization.

                    Test analysis.

                    Disaster recovery initial conditions

                    Real world trading results (also remember your broker diary here).

                    Constant systems development theory and application, etc.

                    These items should also serve as a good checklist for future systems development.