Vijay Krishna Menon

Measuring stock price and trading volume causality among Nifty50 stocks: The Toda Yamamoto method

This paper analyzes the existence of a Granger causality relationship between stock prices and trading volume using minute by minute data (transformed from tick by tick data) of Nifty 50 companies traded at the National Stock Exchange, India for the period of one year from July 2014 to June 2015. Since the time series data taken is not integrated in of the same order, the Toda-Yamamoto methodology was applied to test for causality. The results show that 29 companies out 50 companies have two-way (bi-directional) causality between price and volume and 15 companies have one way (unidirectional) causal relationship where price causes volume and volume does not cause price and 6 other companies have no causal relationship in either way. The study suggests that the Efficient Markets Hypothesis does not hold true for these 29 companies during the period of this study.

Prediction of Malicious Domains Using Smith Waterman Algorithm

IT security is an issue in today world. This is due to many reasons such as, malicious domains. Predicting the malicious domain in a set of domains is important. Here we have proposed a method for analysing such domains. In this method Wireshark is used for capturing the network packets. These packets are further given to client server machine and store in server database which makes an interface between the wireshark and machine. The data from the server database are then compared with the dictionary to predict the malicious websites. It is identified in such a way that if a word in a domain matches with any one of the dictionary word then it is considered as non-malicious websites others are malicious websites.

Bulk Price Forecasting Using Spark over NSE Data Set

Financial forecasting is a widely applied area, making use of statistical prediction using ARMA, ARIMA, ARCH and GARCH models on stock prices. Such data have unpredictable trends and non-stationary property which makes even the best long term predictions grossly inaccurate. The problem is countered by keeping the prediction shorter. These methods are based on time series models like auto regressions and moving averages, which require computationally costly recurring parameter estimations. When the data size becomes considerable, we need Big Data tools and techniques, which do not work well with time series computations. In this paper we discuss such a finance domain problem on the Indian National Stock Exchange (NSE) data for a period of one year. Our main objective is to device a light weight prediction for the bulk of companies with fair accuracy, useful enough for algorithmic trading. We present a minimal discussion on these classical models followed by our Spark RDD based implementation of the proposed fast forecast model and some results we have obtained.

A synchronised tree adjoining Grammar for English to Tamil Machine Translation

Tree adjoining Grammar (TAG) is a rich formalism for capturing syntax and some limited semantics of Natural languages. The XTAG project has contributed a very comprehensive TAG for English Language. Although TAGs have been proposed nearly 40 years ago by Joshi et al, 1975, their usage and application in the Indian Languages have been very rare, predominantly due to their complexity and lack of resources. In this paper we discuss a new TAG system and methodology of development for Tamil Language that can be extended for other Indian languages. The trees are developed synchronously with a minimalistic grammar obtained by careful pruning of XTAG English Grammar. We also apply Chomskian minimalism on these TAG trees, so as to make them simple and easily parsable. Furthermore we have also developed a parser that can parse simple sentences using the above mentioned grammar, and generating a TAG derivation that can be used for dependency resolution. Due to the synchronous nature of these TAG pairs they can be readily adapted for Formalism based Machine Translation (MT) from English to Tamil and vice versa.

A New Evolutionary Parsing Algorithm for LTAG

Tree adjoining grammars (TAGs) are mildly context-sensitive psycholinguistic formalisms that are hard to parse. All standard TAG parsers have a worst-case complexity of O(n6), despite being one of the most linguistically relevant grammars. For comprehensive syntax analysis, especially of ambiguous natural language constructs, most TAG parsers will have to run exhaustively, bringing them close to worst-case runtimes, in order to derive all possible parse trees. In this paper, we present a new and intuitive genetic algorithm, a few fitness functions and an implementation strategy for lexicalised-TAG parsing, so that we might get multiple ambiguous derivations efficiently.

Deep Learning for Network Flow Analysis and Malware Classification

In this paper, we present the results obtained by applying deep learning techniques to classification of network protocols and applications using flow features and data signatures. We also present a similar classification of malware using their binary files. We use our own dataset for traffic identification and Microsoft Kaggle dataset for malware classification tasks. The current techniques used in network traffic analysis and malware detection is time consuming and beatable as the precise signatures are known. Deep learned features in both cases are not hand crafted and are learned form data signatures. It cannot be understood by the attacker or the malware in order to fake or hide it and hence cannot be bypassed easily.

Stock price prediction using dynamic mode decomposition

Stock price prediction is a challenging problem as the market is quite unpredictable. We propose a method for price prediction using Dynamic Mode Decomposition assuming stock market as a dynamic system. DMD is an equation free, data-driven, spatio-temporal algorithm which decomposes a system to modes that have predetermined temporal behaviour associated with them. These modes help us determine how the system evolves and the future state of the system can be predicted. We have used these modes for the predictive assessment of the stock market. We worked with the time series data of the companies listed in National Stock Exchange. The granularity of time was minute. We have sampled a few companies across sectors listed in National Stock Exchange and used the minute-wise stock price to predict their price in next few minutes. The obtained price prediction results were compared with actual stock prices. We used Mean Absolute Percentage Error to calculate the deviation of predicted price from actual price for each company. Price prediction for each company was made in three different ways. In the first, we sampled companies belonging to the same sector to predict the future price. In the latter, we considered sampled companies from all sectors for prediction. In the first and second method, the sampling as well as the prediction window size were fixed. In the third method the sampling of companies was done from all sectors considered. The sampling window was kept fixed, but predictions were made until it crossed a threshold error. Prediction was found to be more accurate when samples were taken from all the sectors, than from a single sector. When sampling window alone was fixed; the predictions could be made for longer period for certain instances of sampling.

Stock price prediction using LSTM, RNN and CNN-sliding window model

Stock market or equity market have a profound impact in todays economy. A rise or fall in the share price has an important role in determining the investors gain. The existing forecasting methods make use of both linear (AR, MA, ARIMA) and non-linear algorithms (ARCH, GARCH, Neural Networks), but they focus on predicting the stock index movement or price forecasting for a single company using the daily closing price. The proposed method is a model independent approach. Here we are not fitting the data to a specific model, rather we are identifying the latent dynamics existing in the data using deep learning architectures. In this work we use three different deep learning architectures for the price prediction of NSE listed companies and compares their performance. We are applying a sliding window approach for predicting future values on a short term basis. The performance of the models were quantified using percentage error.

Semantic Analysis Using Pairwise Sentence Comparison with Word Embeddings

Comparing the semantics of a pair of sentences has been an interesting yet unstructured problem. Semantic analysis is mostly elusive due to the fact that the semantics of Natural language constructs cannot be measured, let alone be compared to one another. Methods like Latent Semantic Analysis(LSA) and Latent Dichlaret Analysis(LDA) are able to capture broader semantics between documents, but their contribution in pairwise comparison tasks which require deeper semantics may be limited. In this paper we present a local alignment based scoring scheme for sentence pairs using word embeddings and how this can be used as a feature for some popular text analysis tasks such as summarization, paraphrase comparison, topic profiling and other semantic comparison tasks. We also present a theoretical analysis on the metrics used in this approach and a separability argument using t-SNE plots. Furthermore we detail our Spark implementation model for the pairwise comparison and summarization.