Market data definitions & terminology

The goal of this article is to introduce the concepts, terminology and code structures required to develop applications that utilise real-time stock market data (e.g. trading applications). It discusses trading concepts, the different types of market data available, and provides a practical example on how to process data feed events into a market object model.

The article is aimed at intermediate to advanced developers who wish to gain an understanding of basic financial market data processing. I recommend that those who are already familiar with trading terminology skip ahead to the Market Data section.

Orders & Trades
A trade occurs when a seller agrees to transfer ownership of a specified quantity of stock to a buyer at a specified price. How do buyers and sellers meet? They use a centralised market place called a stock market. People come together and then announce their desire to buy or sell a specific stock; I want to buy 500 shares in BHP for $35.00, I want to sell 2,000 shares in RIO for $65.34. These are called orders. Buy orders are also referred to as bid orders. Sell orders are also referred to as ask or offer orders.

When a buy order's price is equal to or higher than the lowest priced sell order currently available, a trade occurs. When a sell order's price is equal to or lower than the highest priced buy order currently available, a trade occurs. This process is also known as a match, because the buy and sell prices must match or cross over for a trade to occur.

What happens if an order is submitted to the market, but it does not match? It is entered into a list of orders called the order book. The order will remain there until the trader cancels it, or it expires (e.g. end of day if it is a day order). Some orders expire immediately after matching against what is available on the order book. These are known as 'immediate or cancel' (IOC) or 'fill and kill' (FAK) orders. These orders are never entered into the order book regardless of whether they match or not.

Order Book
The order book contains all the offers to buy and sell a particular stock that have not been matched. It's like the classified advertisements for shares, everyone can see all the offers to buy and sell. The limit order book can be referred to as the book, the depth or the queue.

An order in the book can only be matched against an incoming order if it is the highest priority order (e.g. it must be at the top of the book). It's like standing in a queue: you must be at the head of the line to get served.

Orders in the order book are ordered by price-time priority. This means that orders are sorted first by price and then by time of submission. The highest priority buy order will be the buy order with the highest price that was submitted first. The highest priority sell order will be the sell order with the lowest price that was submitted first.

Here is an example order book showing two queues of orders in price-time priority: one queue for the buy (bid) orders and one queue for the sell (ask) orders. The time refers to the submission time of each order:

The bid (buy) orders and ask (sell) orders are arranged from top to bottom in priority. The ask orders are orders are ordered by time (earliest first) because they are the same price. Notice that the bid order for 100 shares is higher in priority than the order for 19 shares, even though it was submitted later, because it has a higher price.

Trade Example
I will now step through a trade match using the example order book shown above. Consider that a sell order for 150 shares at 23.34 is submitted into the market (the short-hand for this order would be: Sell 150@23.34). The new order is marked in yellow:

Notice the bid and ask prices are now crossed. After each order is submitted, the exchange checks for crossed prices and then performs the required number of matches to return the market to an uncrossed state. In this case, the exchange matches the 150 sell against the 100 buy at 23.34, generating a trade for 100 share at 23.34 (100@23.34). The remaining 50 share sell order is now the highest priority ask order in the book:

Market Quotes
Often traders are not interested in the entire order book of a stock, but just the highest bid and the lowest ask prices currently in the market, and the quantity available at those prices. All this information together is known as a market quote.

Spread
When a market is in continuous trading (not closed or in an auction state), the bid price must always be lower than the ask price. If they were crossed, a trade would occur. The difference between the lowest ask and highest bid price is called the spread.

Prices to buy and sell can be only changed in specified increments (e.g. you can't offer to buy at $35.0001). The minimum price change for a stock is called its tick size. Consequently, the minimum spread size for each stock is dictated by its the tick size. The lower the share price of the stock, the smaller the tick size will be. For example, tick sizes on the Australian market are:

• $0.001 for a share price under $0.10
• $0.005 for a share price under $2.00
• $0.01 for a share price equal to or greater than $2.00

Market Data
There are several different 'grades' of market data. Data quality is determined by its granularity and its detail.

Granularity
Granularity refers to the observational time interval of the data. Snapshot observations record a particular moment in time (for example daily closing prices, or market quotes for each minute of the day). Event-based observations are recorded each time a relevant field changes (e.g. trade update, change to the order book).

Event-level data sets are superior to snapshot data sets because a snapshot view can always be derived from event data, however the inverse does not hold true. If they are so much better, why is it that all data sets are not supplied in event form? A few reasons. They are big to store, harder to code against, they are slower to process and not everyone needs this level of detail.

Detail
Detail refers to what information is contained in the data set. There are three levels of market data detail: trades, quotes and depth. Trade & quote updates combined together are often referred to as Level 1 (L1) data. Depth updates are referred to as Level 2 (L2) data.

Trade Updates
The simplest level of detail occurs in the form of trade prices (e.g. daily closing prices). These are widely available and are often used by retail investors to select a stock to buy and hold for a longer investment period (e.g. months, years).
Below is a sample of daily closing prices for BHP traded on the ASX:

Date Open High Low Close Volume2013-02-05 37.80 37.94 37.68 37.92 56837822013-02-04 37.38 37.61 37.33 37.48 61406102013-02-03 37.50 37.64 37.42 37.62 66764102013-02-02 37.30 37.30 37.04 37.17 69365942013-02-01 37.25 37.27 36.95 37.10 137375222013-01-31 36.90 37.22 36.82 37.16 71746442013-01-30 37.00 37.15 36.86 37.06 91431362013-01-29 36.54 36.85 36.50 36.58 5569151

A big step up from daily closing prices is intraday trade history (also known as tick data), which contains a series of records detailing every trade that occurred for a stock. A good trade data set will contain the following fields:

• Symbol - Security symbol (e.g. BHP)
• Exchange - Exchange the trade occurred on (e.g. ASX, CXA)
• Price - Transaction price
• Quantity - Transaction quantity
• Time - Transaction date and time (this will be in milliseconds or microseconds if it's a good dataset)
• Trade Type (Condition Codes) - What type of trade it was (e.g. standard, off-market trade report, booking purpose trade)

Below is a sample of intraday trade records for FMG (Fortescue Metals Group) traded on the ASX for the 30th Sep 2011:

Date Time Symbol Exch Price Quantity Type20110930 11:14:24.475 FMG ASX 4.62 1000 20110930 11:14:24.475 FMG ASX 4.62 5000 XT20110930 11:14:24.475 FMG ASX 4.62 249 20110930 11:14:24.477 FMG ASX 4.62 25722 20110930 11:14:24.480 FMG ASX 4.62 1518 XT20110930 11:14:24.482 FMG ASX 4.62 113 XT20110930 11:14:25.046 FMG ASX 4.62 2702

NOTE: The 'XT' flag indicates that the trade was a crossing. This occurs when the same broker executes both sides of the trade (e.g. one client is buying through the broker and another is selling).

Intraday trade records are often used in retail trading software for intraday charting and technical analysis. Sometimes people will attempt to backtest a trading strategy (backtest means to evaluate performance using historical data) using trade records. Don't do this if you are testing intraday strategies, you're results will be useless because the trade price won't always reflect the actual price you could buy or sell for at that time.

The charts displayed by Google Finance for a security are generated using intraday trade records (with the trade time on the x-axis and trade price on the y-axis).

Quote Updates
The next step up in market data detail is the inclusion of market quote updates. A good market quote data set will contain a record of the following fields every time there is a change:

• Symbol - Security symbol (e.g. BHP)
• Exchange - Exchange the quote is from
• Time - Quote update time
• BidPrice - Highest market bid price
• BidQuantity - Total quantity available at market bid price
• AskPrice - Lowest market ask price
• AskQuantity - Total quantity available at market ask price

Some quote data sets will provide these additional fields, which are useful for certain types of analytics, but not particularly relevant for backtesting:

• BidOrders - Number of orders at market bid price
• AskOrders - Number of orders at market ask price

Below is a sample of real-time quote update events for NAB (National Australia Bank) on the 31st Oct 2012:

Date Time Update Symbol Exch Side Price Quantity2012-10-31 13:51:13.784 QUOTE NAB ASX Bid 25.81 150072012-10-31 13:51:14.615 QUOTE NAB ASX Bid 25.82 102012-10-31 13:51:14.633 QUOTE NAB ASX Bid 25.81 136232012-10-31 13:51:14.684 QUOTE NAB ASX Ask 25.82 25002012-10-31 13:52:09.168 QUOTE NAB ASX Bid 25.80 122232012-10-31 13:52:09.173 QUOTE NAB ASX Ask 25.81 12782012-10-31 13:52:39.750 QUOTE NAB ASX Ask 25.80 1362012-10-31 13:52:39.754 QUOTE NAB ASX Bid 25.79 126562012-10-31 13:54:20.870 QUOTE NAB ASX Ask 25.81 103752012-10-31 13:54:20.878 QUOTE NAB ASX Bid 25.80 1098

A good quality market quote dataset is all that is required for back-testing intraday trading strategies, as long as you are only planning to trade against the best market price rather than post orders in the order book and waiting for fills (putting aside issues of data feed latency, execution latency and other more advanced topics for now).

Depth Updates
The final level of market data detail is the inclusion of market depth updates. Depth updates contain a record of every change to every order in the order book for a particular security. Depth data sets can often be limited. Some depth data sets only provide an aggregated price view, the total quantity and number of orders at each price level. Others will only provide the first few price levels.

A good depth update data set will contain a record of the following fields every time one of them changes:

• Symbol - Security symbol (e.g. BHP)
• Exchange - Exchange the update is from
• Time - Order update time
• Order Position OR Unique Order Identifier - This identifies which order has been changed (either by relative position in the queue, or by an order Id)
• Quantity - Order quantity
• Update Type - NEW or ENTER (a new order has been entered), UPDATE (volume has been amended) or DELETE (order has been removed)

Market depth updates are required for accurately back-testing intraday trading strategies where you are submitting orders that will enter the order book queue rather than executing immediately at the best market price. > From <https://www.codeproject.com/Articles/553206/An-Introduction-to-Real-Time-Stock-Market-Data-Pro>