Open Source HFT/Algo model

[oddballstocks]

This isn't for 99% of this board, but for the 1%...

 

I saw a post on Hacker News where a guy linked to his GitHub page with an OS HFT model in Python.

 

The project looks like it's full cycle and uses Interactive Brokers to execute the trades. The project is missing a lot of pieces (risk management). But it's a good start.

 

If anyone is looking to get a start in automated trading or automated investing I think this is a good start. I'm sure it could be modified to do magic formula or something value-y.

 

Here is the GitHub page: https://github.com/jamesmawm/High-Frequency-Trading-Model-with-IB

HN comments are insightful: https://news.ycombinator.com/item?id=9686841

 

I've always been fascinated by automated or programmatic trading, and it's cool to see the guys here.

 

I know a few board members have worked in this area as well, would be cool to hear some thoughts.

[Jurgis]

There were couple of articles on real HFT (collocated variety) in Communications of ACM recently. I only found one online: http://www.cs.rit.edu/~rlc/Courses/ProComm/Research/onlineTrading.pdf In case anyone is interested.

[mbharadwaj]

oddballstocks - thank you for posting this.

 

HN comments mention Disruptor for scaling this up. A shameless plug - a disruptor example - https://github.com/mbharadwaj/disruptor-example

 

I recently was at a conference where Martin Thompson, one of the original founders of LMAX presented. His blog (http://mechanical-sympathy.blogspot.com/) is a good one to follow for optimizing approaches.

 

Martin Fowler - a renowned expert in related field had a article on LMAX this is helpful - http://martinfowler.com/articles/lmax.html

[meiroy]

Thanks for the post. This is fascinating, what would happen if this really become easy to install and use, how would that impact the markets when everyone and their savvy aunt runs it? Considering the algorithm is out in the open, it should be possible to create a counter algorithm to take advantage of people who use it and suck away their money.

 

oddballstocks, if you were to write a program to mine the net for sites such as google trend, news sites etc. for certain data or trends, which programming language would you use?

[oddballstocks]

oddballstocks, if you were to write a program to mine the net for sites such as google trend, news sites etc. for certain data or trends, which programming language would you use?

 

Interesting question, especially given the context of the original post.  The author wrote this in Python.  Python is an interpreted language and is viewed as much slower.

 

If I were to write something I'd do it in Java or C#, but that's because I know both.  I'm not sure which is actually faster, both have a layer of abstraction, the JVM and the .NET interpreter.  But each of these languages is magnitudes faster than Python.

 

The Disruptor library is REALLY interesting.  It's written in Java, and clearly these guys are squeezing every bit of performance out of Java as possible.  What I love about this is I've always thought this is what computing should be.  Getting every last bit of performance out of a given piece of hardware.  Things can be very bloated now because RAM is cheap and SSD's are everywhere.  There isn't the same penalty hitting the disk with SSD isn't there like it was with a 5400 RPM disk, or even slower.

 

I love that Disruptor has this inter thread messaging.  I really want to find a way to use that somehow, but really the only use case is in something that has to be extremely fast and deal with a lot of changing real time information.

[innerscorecard]

Thanks for posting this. I would be the crudest potential user of this, but I absolutely would want to implement my mechanical Magic Formula investments algorithmically.

[oddballstocks]

There were couple of articles on real HFT (collocated variety) in Communications of ACM recently. I only found one online: http://www.cs.rit.edu/~rlc/Courses/ProComm/Research/onlineTrading.pdf In case anyone is interested.

 

I'm enjoying this paper.  I never spent much time trying to think this through, but it's a huge challenge.  When looking at an investment it's a static analysis.  I have variable x, y, and z, if something happens to x then y changes etc.  Yet in a real-time algo situation while you're trying to decide whether to act on x y is happening and can change that decision.

[oddballstocks]

Thanks for posting this. I would be the crudest potential user of this, but I absolutely would want to implement my mechanical Magic Formula investments algorithmically.

 

That's the allure for me as well.  I look at the data I have, and look at how easy it is to program an algo and think "I wonder if an opportunity exists to write an automated bank trading program."

 

It seems like merging human intelligence with a model could be incredible.  The model does things like buy when every fiber is telling you to go cash, but it's not a stupid model.  It's based on fundamental things.  I think about a model that knows when credit quality is turning and immediately sells a bank for me. 

 

If someone ever gave me tens of millions of VC to burn one of the first things I'd do would be to buy a historical market data set and a Data Scientist to mine it and see if this was worth any time.  My gut says value investing merged with some of this stuff could be very profitable a la Magic Formula.  I know Greenblatt has raise a billion on the back of his value based algo.

[Jurgis]

I was thinking about doing some Machine Learning on fundamental data... but I don't have data and I never found time to do it... :) Not sure it would come out with anything better than Magic Formula though.

 

Do you think what you write can outperform Magic Formula and/or QVAL-and-friends?

[wachtwoord]

oddballstocks, if you were to write a program to mine the net for sites such as google trend, news sites etc. for certain data or trends, which programming language would you use?

 

Interesting question, especially given the context of the original post.  The author wrote this in Python.  Python is an interpreted language and is viewed as much slower.

 

If I were to write something I'd do it in Java or C#, but that's because I know both.  I'm not sure which is actually faster, both have a layer of abstraction, the JVM and the .NET interpreter.  But each of these languages is magnitudes faster than Python.

 

Python is compilable. If you want something truly fast you wont use a language with a VM such as JVM or .Net. C or even assembly for parts of it is a good choice.

[writser]

The project looks interesting but if you use the Interactive Brokers API to do trading I wouldn't call it 'high-frequency'. I think what programming language you are actually using is not important at all in this case (actually I would say in most cases, but that's a completely different discussion :) ). Even if the performance difference between, let's say, Assembly, Java and Python is a factor 100 we're talking about a few microseconds whereas random stuff about your internet connection (what IB gateway am I connected to, how close do I live to my internet backbone) can make a 10ms+ difference. Not to mention that dedicated HFT firms have leased lines, direct exchange access, dedicated hardware and microwave towers, among other stuff. You won't stand a chance. Besides that, IB has an 100 order limit per day or something so you can't build any serious strategy anyway.

 

In the 'medium-frequency' areas where you can do some interesting stuff from home (and I believe they exist) it is really not going to matter what programming language you are going to use. Your best bet is probably to use the language you are most familiar with (and that generates maintainable code - doing stuff in assembly would be a terrible idea imho).

 

p.s.: an interesting blog if you're interested in HF strategies: https://mechanicalmarkets.wordpress.com/ .

[oddballstocks]

I agree that IB is not HFT..that was one of my first thoughts on this as well.

 

Network latency is really the slowest link, but it seems a lot of focus is on computing time as well.  Look at the Disruptor paper or the ACM papers linked.  What's the difference in calculating an exponential mean vs a two pass median in Java?  Surely that incremental performance gain pales in comparison to the network latency, yet there's a focus on that.

 

I agree assembly would be a bad choice.  I did some assembly years back.  It takes forever to develop, and it's unmaintainable.  Imagine a team trying to manage an assembly codebase.  Systems level languages are fast enough.

 

One thing I've been thinking is if there's a use case here for fpga or SoC type setups.  I can imagine a circuit programmed to do a sort.  Imagine if a switch were to be built that pre-sorted or intelligently routed requests.  By the time it hit the server some level of decision making had already been done.

 

 

[writser]

Sure, but exponential vs. linear is not really a programming language thing either. Probably it's better to have the linear algorithm in Visual Basic than the exponential one in assembly. The FPGA stuff is already happening, just Google and you'll find plenty examples. People are building network cards with built-in trading capacities, routers / switches that preprocess price feeds (even cisco), all you can think of. The amount of resources being dedicated to this field is staggering.

 

By the way, you probably knew this already but FPGA's are also being used to mine bitcoins. Have you ever seen this?

. Fascinating stuff :) .

[mbharadwaj]

oddballstocks - please do check out Gil Tene's posts on his blog and github account: http://latencytipoftheday.blogspot.com/

 

He and Martin Thompson have worked together on a few projects and both discuss a lot of aspects related to squeezing the last drop of juice out of the hardware. Their twitter handles are good to follow as well.

[RichardGibbons]

I wrote an automated trading system on top of the Interactive Brokers API using C#.  In that case, my impression was that network latency mattered much more than computational time, so I didn't spend much time trying to speed up the algorithm.  (That said, the algorithm was pretty simple.)

 

I made a small amount of money at it, but for my algorithm and execution speeds, the slippage  didn't make it worth the risk and the complexity.  From memory, I traded several million dollars worth of stock, and, if I'd got the execution on the bid/ask that IB said I would, I would have made in the area of $40-$50K.  Instead, I made under $10K because the execution of the orders was too slow.