Dukascopy tickdata downloader

If you design an automatic trading system, the most common problem that you face is where to get the data for the testing (backtesting to be precise). There are several sources of this data (called tickdata) but none of them is user friendly. Typically, the task involves a lot of manual work, like downloading the files, unpacking them, building the database, etc. Also the quality of the data almost always is far from perfect. Perhaps one of the best free sources of the tick data is a Swiss company Dukascopy. However if you decide to go for its data, you will be disappointed. The retrieval of the data is so user unfriendly and time consuming that you would think twice before going for it.

However this task can be easily automated. The structure of the data stored on the Ducascopy servers is simple. Each hour of tick data is represented by the individual file and the file's URL is constructed from the date and a symbol name. The only intricacy is that the months start with zero. Once this file is downloaded, it has to be unpacked by LZMA compressor.

Using the Code
Constructing the URL for one day:

string
 BaseUrl = 
"
http://www.dukascopy.com/datafeed/
"
; 

DateTime lowValue = start; 


while
 (lowValue \<= end) 

{ 

    
string
 dr_val = symbol + 
"/"
 + lowValue.Year.ToString() + 

    
"/"
 + (lowValue.Month - 
1
).ToString(
"00"
) + 

    
"/"
 + (lowValue.Day).ToString(
"00"
) + 
"/"
; 

    

    DateTime Dt_base = 
new
 DateTime(lowValue.Year, lowValue.Month, lowValue.Day); 

    

    
for
 (
int
 h = 
0
; h \< 
24
; h++) 

    { 

        SymbolAndURLContainer dc = 
new
 SymbolAndURLContainer(); 

        dc.Url = dr_val + h.ToString(
"00"
) + 
"h_ticks.bi5"
; 

        TimeSpan ts = TimeSpan.FromHours(h); dc.dt = Dt_base + ts; Files_lst.Add(dc); 

    } 

}  





public
 
class
 HttpDownloader : BaseDownloader 

BaseUrl = "http://www.dukascopy.com/datafeed/"; DateTime lowValue = start; 
while (lowValue ≤ end) {  string dr_val = symbol + "/" + lowValue.Year.ToString() +  "/" + (lowValue.Month - 1).ToString("00") +  "/" + (lowValue.Day).ToString("00") + "/";   DateTime Dt_base = new DateTime(lowValue.Year, lowValue.Month, lowValue.Day);   for (int h = 0; h < 24; h++)  {  SymbolAndURLContainer dc = new SymbolAndURLContainer();  dc.Url = dr_val + h.ToString("00") + "h_ticks.bi5";  TimeSpan ts = TimeSpan.FromHours(h); dc.dt = Dt_base + ts; Files_lst.Add(dc);  } }
Running the Application
Just load the solution to Visual Studio 2012, compile and run. You have to choose the output folder before starting the app.
Downloading the File
This is done by:

public class HttpDownloader : BaseDownloader

This class downloads the file asynchronously. It runs in its own thread.

Unpacking
I use this implementation of LZMA:

• http://code.google.com/p/managed-lzma/

After the file is unpacked, it has to be parsed and interpreted. The bytes are stored as bigendian, so they have to be reversed. When we finish with unpacking, the data is written to the CSV file for further use.
Where to Get the Latest Version of this Software
The latest release of this software can be found here.
Also you can find the tick data server and the clients for MBTrading, Gaincapital, Histdata and Pepperstone brokers. Under certain conditions, the above products can be obtained free of charge (and many other useful components and tips).
Current version 1.0.04

From <http://www.codeproject.com/Tips/591422/Dukascopy-tickdata-downloader>