Bot Trading Final Thoughts

My software project for February was a stock trading bot that made trades based on twitter feeds. It was an interesting project that gave me a chance to learn a few new things.

I had the chance to apply a couple of machine learning techniques.  In particular using natural language processing to perform sentiment analysis and named entity recognition.  There is still a lot to improve on with the tools available in these areas but I was left with the impression that we are in the midst of a big shift with how AI algorithms will be applied to real world applications.

To give a more concrete example of that; I started working on the named entity recognition by trying to apply  the basic tooling that comes with the NLTK library in Python, this proved far too complex to complete without completing a computational linguistics course first.  My next attempt was to use the Stanford NER model which is the current best approach to the problem, however, out of the box it lacks the training to be useful, was still overly complex to work with, and gave bad results in some simple test cases.  The final approach that I took was to use the Google natural language APIs which was brilliant by comparison. Google is able to tie their entity matching to their knowledge graph and as a result something like ‘apple had a great 3rd quarter’ can identify ‘apple’ as the company which links to their wikipedia page, the names of executives and every bit of information google has about the company.

Admittedly, working with the knowledge graph is complicated, but the ability to pull things together in that way is stunning.

I think that as more of these AI algorithms become exposed as trained Software as a Service APIs, you’ll start to see more regular developers being able to embed them into the applications that we all use everyday.

The leverage provided by software and it’s distribution model allows for disruptions to happen very quickly.  The current bottleneck in my opinion is that there is a lack of people with the required skills to build these things from scratch.  Regular software shops don’t have the vast training data needed to make smart machine learning based algorithms, nor do they have the in-house expertise to apply the latest developments in Deep Learning.  With Google, Microsoft, Amazon and Baidu all announcing machine learning APIs usable by junior developers we will start to see a lot more intelligence in the software we use everyday.  And as more people become aware of what is and is not possible the application of these techniques could explode.

The future will be interesting.