Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

 ___       _       _ _ _                  _     _____              _ _               ____        _ 
|_ _|_ __ | |_ ___| | (_) __ _  ___ _ __ | |_  |_   _| __ __ _  __| (_)_ __   __ _  | __ )  ___ | |_
 | || '_ \| __/ _ \ | | |/ _` |/ _ \ '_ \| __|   | || '__/ _` |/ _` | | '_ \ / _` | |  _ \ / _ \| __|
 | || | | | ||  __/ | | | (_| |  __/ | | | |_    | || | | (_| | (_| | | | | | (_| | | |_) | (_) | |_ 
|___|_| |_|\__\___|_|_|_|\__, |\___|_| |_|\__|   |_||_|  \__,_|\__,_|_|_| |_|\__, | |____/ \___/ \__|
                         |___/                                               |___/                   
₿   Ξ   ₳   ₮   ✕   ◎   ●   Ð   Ł   Ƀ   Ⱥ   ∞   ξ   ◈   ꜩ   ɱ   ε   ɨ   Ɓ   Μ   Đ  ⓩ  Ο   Ӿ   Ɍ  ȿ

https://t.me/intelligent_trading_signals 📈 Intelligent Trading Signals 📉 https://t.me/intelligent_trading_signals

Intelligent trading bot

The project is aimed at developing an intelligent trading bot for automatic trading cryptocurrencies using state-of-the-art machine learning approaches to data processing and analysis. The project provides the following major functions:

  • Analyzing historic data and training machine learning models as well as finding their best hyper-parameters. It is performed in batch off-line mode.
  • Signaling service which is regularly requests new data from the exchange and generates buy-sell signals by applying the previously trained models
  • Trading service which does real trading by buying or selling the assets according to the generated signals

Note that is an experimental project aimed at studying how various machine learning and feature engineering methods can be applied to cryptocurrency trading.

Intelligent trading channel

This software is running in a cloud and sends its signals to this Telegram channel:

📈 Intelligent Trading Signals 📉 https://t.me/intelligent_trading_signals

Everybody can subscribe to the channel to get the impression about the signals this bot generates.

Currently, the bot is configured using the following parameters:

  • Exchange: Binance
  • Cryptocurrency: ₿ Bitcoin
  • Frequency: 1 minute
  • Score between -1 and +1. <0 means decrease, and >0 means increase
  • Filter: notifications are sent only if score is greater than ±0.15
  • One increase/decrease sign is added for each step of 0.5 (after the filter threshold)
  • Prediction horizon 3 hours ahead. For example, if the score is +0.25 then the price is likely to increase 1-2% during next 3 hours
  • History taken into account for forecasts: 12-24 hours

There are silent periods when the score in lower than the threshold (currently 0.15) and no notifications are sent to the channel. If the score is greater than the threshold, then every minute a notification is sent which looks like

₿ 60,518 📉 📉 📉 Score: -0.26

The first number is the latest close price. The score -0.26 means that it is very likely to see the price 1-2% lower than the current close price during next few hours. The three decrease signs mean three 0.5 steps after the threshold 0.15.

Signaler

The signaler performs the following steps to make a decision about buying or selling an asset:

  • Retrieve the latest data from the server
  • Compute derived features based on the latest history
  • Apply several (previously trained) ML models by forecasting some future values (not necessarily prices) which are also treated as (more complex) derived features
  • Aggregate the results of forecasting produced by different ML models by computing the final score which reflects the strength of the upward or downward trend. Positive score means growth and negative score means fall
  • Apply a previously trained signal model to make a decision about buying or selling. The signal models are trained separately and are relatively simple in comparison to the forecasting ML models

The final result of the signaler is the score (between -1 and +1) and the signal (BUY or SELL). The score can be used for further decisions while signal is supposed to be used for executing real transactions.

Starting the service: python3 -m service.server -c config.json

Training machine learning models

The following batch scripts are used to train the models needed by the signaler:

  • Download the latest historic data: python -m scripts.download_data -c config.json
  • Merge several historic datasets into one dataset: python -m scripts.merge_data -c config.json
  • Generate feature matrix: python -m scripts.generate_features -c config.json
  • Train prediction models: python -m scripts.train_predict_models -c config.json

There exist also batch scripts for hyper-parameter tuning and signal model generation based on backtesting:

  • Generate rolling predictions which simulate what we do by regularly re-training the models and using them for prediction: python -m scripts.generate_rolling_predictions -c config.json
  • Train signal models for choosing best thresholds for sell-buy signals which produce best performance on historic data: python -m scripts.train_signal_models -c config.json

Configuration parameters

The configuration parameters are specified in two files:

  • service.App.py in the config field of the App class
  • -c config.jsom argument to the services and scripts. The values from this config file will overwrite those in the App.config

Here are some most important fields:

  • symbol it is a trading pair like BTCUSDT - it is important for almost all cases
  • data_folder - location of data files which are needed only for batch scripts and not for services
  • model_folder - location of trained ML models which are stored by batch scripts and then are loaded by the services
  • signaler is a section for signaler parameters
  • trader is a section for trader parameters

Trader

The trader is working but not thoroughly debugged, particularly, not tested for stability and reliability. Therefore, it should be considered a prototype with basic functionality. It is currently integrated with the Signaler.

Owner
Alexandr Savinov
AI, ML, Big/Fast Data, Forecasting, Anomaly detection, AIoT
Alexandr Savinov
Comments
  • Configs samples

    Configs samples

    Hello! Сould you update the sample configs? Because it seems there are not enough parameters for the full functionality of the scripts, for example "time_column" is missing at all, as well as "freq". I tried to start training several times, but the process was interrupted several times due to incorrect training configuration.

    Sincerely!

  • No data is being downloaded

    No data is being downloaded

    Hello , I made a simple config.json from config-sample-v0.2.0.json with Binaance API Key and telegram key, let it ran for a while, no data is being downloaded.

    It starts with : Start downloading 'BTCUSDT' ... File not found. All data will be downloaded and stored in newly created file. Downloading all available 1m data for BTCUSDT. Be patient..! ..and stays like that forever. Maybe some binance endpoint was changed in the meanwhile ?

  • Stuck at `Downloading all available 1m data for BTCUSDT. Be patient..!` for hours.

    Stuck at `Downloading all available 1m data for BTCUSDT. Be patient..!` for hours.

    my config file looks like this

    {
      "api_key": "my_key",
      "api_secret": "my_secret",
    
      "symbol": "BTCUSDT",
      "base_asset": "BTC",
      "quote_asset": "USDT",
    
      "data_folder": "some_path",
      "model_folder": "some_path"
    }
    

    what's going wrong? Can you please help me out here?

  • Error when running python -m scripts.generate_features -c config.json

    Error when running python -m scripts.generate_features -c config.json

    I have the following error :

    Loading data from source data file DATA_ITB/BTCUSDT/data.csv...
    Finished loading 2591 records with 23 columns.
    Start generating features for 2591 input records.
    Start generator klines...
    Unknown feature generator klines
    Traceback (most recent call last):
      File "/home/gitpod/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/home/gitpod/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/workspace/intelligent-trading-bot/scripts/generate_features.py", line 202, in <module>
        main()
      File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/workspace/intelligent-trading-bot/scripts/generate_features.py", line 66, in main
        df, new_features = generate_feature_set(df, fs, last_rows=0)
    TypeError: cannot unpack non-iterable NoneType object
    
    
  • Sample of config file

    Sample of config file

    Do you have a sample of your config file ? All scripts have "freq" option hard coded ?

    I have an error with "freq": "1h" but no idea oh problem trying to debug df

    lightgbm.basic.LightGBMError: Check failed: (num_data) > (0) at /Users/runner/work/1/s/python-package/compile/src/io/dataset.cpp, line 33

    Can we have a bit more documentation ?

    Thx

  • Model performance issue

    Model performance issue

    @asavinov. Thanks for you sharing. Very clear. Through your code, it is found that the trained model does not perform well (precision and recall) in predicting the price decline in the future. Do you have any good suggestions?

  • Missing LICENSE

    Missing LICENSE

    I see you have no LICENSE file for this project. The default is copyright.

    I would suggest releasing the code under the GPL-3.0-or-later or AGPL-3.0-or-later license so that others are encouraged to contribute changes back to your project.

  • Predict rolling

    Predict rolling

    PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html date_obj = stz.localize(date_obj) Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/root/intelligent-trading-bot/scripts/predict_rolling.py", line 281, in main() File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/root/intelligent-trading-bot/scripts/predict_rolling.py", line 100, in main prediction_start = find_index(df, P.prediction_start_str) File "/root/intelligent-trading-bot/common/utils.py", line 135, in find_index id = res.index[0] File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 5358, in getitem return getitem(key) IndexError: index 0 is out of bounds for axis 0 with size 0

  • Warnings while working

    Warnings while working

    Hello, during the process of working with an already trained model(predict_models), the following warnings occurs:

    1/1 [==============================] - 0s 13ms/step
    C:\Users\sanya\PycharmProjects\intelligent-trading-bot\venv\lib\site-packages\scipy\stats\_stats_mstats_common.py:175: RuntimeWarning: invalid value encountered in double_scalars
      slope = ssxym / ssxm
    C:\Users\sanya\PycharmProjects\intelligent-trading-bot\venv\lib\site-packages\scipy\stats\_stats_mstats_common.py:189: RuntimeWarning: invalid value encountered in sqrt
      t = r * np.sqrt(df / ((1.0 - r + TINY)*(1.0 + r + TINY)))
    C:\Users\sanya\PycharmProjects\intelligent-trading-bot\venv\lib\site-packages\scipy\stats\_stats_mstats_common.py:192: RuntimeWarning: invalid value encountered in double_scalars
      slope_stderr = np.sqrt((1 - r**2) * ssym / ssxm / df)
    C:\Users\sanya\PycharmProjects\intelligent-trading-bot\venv\lib\site-packages\scipy\stats\_stats_mstats_common.py:175: RuntimeWarning: invalid value encountered in double_scalars
      slope = ssxym / ssxm
    C:\Users\sanya\PycharmProjects\intelligent-trading-bot\venv\lib\site-packages\scipy\stats\_stats_mstats_common.py:189: RuntimeWarning: invalid value encountered in sqrt
      t = r * np.sqrt(df / ((1.0 - r + TINY)*(1.0 + r + TINY)))
    C:\Users\sanya\PycharmProjects\intelligent-trading-bot\venv\lib\site-packages\scipy\stats\_stats_mstats_common.py:192: RuntimeWarning: invalid value encountered in double_scalars
      slope_stderr = np.sqrt((1 - r**2) * ssym / ssxm / df)
    1/1 [==============================] - 0s 15ms/step
    

    What can this indicate, can this somehow affect the trained model and the whole prediction process? I see that linregress method is only used when in add_linear_trends method(class feature_generation_rollong_agg.py and in train_signal_models class which I have not touched.

    Is it possible to fail at the data downloading stage? I used download_data_binance.py. By the way, such errors did not occur while downloading generating features, merging data or generate labels and while traning too.

    I also use the latest version of the repository uploaded to github.

    I'm new to python, that's why I'm asking such simple questions. Hope for understanding 😁

    Regards!

  • I need your help

    I need your help

    Hello, thank you for publishing this project I have been researching digital currency exchange for some time and I really do not know where to start. I am more interested in working with Python and need guidance on what the database structure and data type should look like. Thanks for taking the time to guide me.

  • PytzUsageWarning

    PytzUsageWarning

    2022-02-07 06:07:46.418122: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2022-02-07 06:07:46.418169: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. /usr/local/lib/python3.8/dist-packages/statsmodels/compat/pandas.py:65: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. from pandas import Int64Index as NumericIndex Initializing server. Trade pair: BTCUSDT. 2022-02-07 06:07:49.759194: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2022-02-07 06:07:49.759243: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303) 2022-02-07 06:07:49.759266: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ubuntu-c-4-8gib-fra1-01): /proc/driver/nvidia/version does not exist 2022-02-07 06:07:49.760338: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. Finished health check (connection, server status etc.) Finished initial data collection. /usr/local/lib/python3.8/dist-packages/apscheduler/util.py:95: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html if obj.zone == 'local': /usr/local/lib/python3.8/dist-packages/apscheduler/triggers/cron/__init__.py:146: PytzUsageWarning: The normalize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html return self.timezone.normalize(dateval + difference), fieldnum Scheduler started.

    Any solution to this pytz issue

  • Freqtrade Hyperopt

    Freqtrade Hyperopt

    Hi,

    I just found your work, and that seem really good. Maybe you can work for integrate your Signal system as hyperopt in freqtrade, you maybe not lost more time in Trader part and you will help few people ! If not, I will try your project and check if my little knowledge of python can help for trader part.

    Thanks for your work, Tcheksa

Cryptocurrency Trading Bot - A trading bot to automate cryptocurrency trading strategies using Python, equipped with a basic GUI
Cryptocurrency Trading Bot - A trading bot to automate cryptocurrency trading strategies using Python, equipped with a basic  GUI

Cryptocurrency Trading Bot - A trading bot to automate cryptocurrency trading strategies using Python, equipped with a basic GUI. Used REST and WebSocket API to connect to two of the most popular crypto exchanges in the world.

Sep 15, 2022
Trading bot - A Trading bot With Python
Trading bot - A Trading bot With Python

Trading_bot Trading bot intended for 1) Tracking current prices of tokens 2) Set

Dec 1, 2022
Nov 16, 2022
A Advanced Powerful, Smart And Intelligent Group Management Bot With New And Powerful Features
 A Advanced Powerful, Smart And Intelligent Group Management Bot With New And Powerful Features

Vegeta Robot A Advanced Powerful, Smart And Intelligent Group Management Bot With New And Powerful Features ... Written with Pyrogram and Telethon...

Nov 16, 2022
Utilizing the freqtrade high-frequency cryptocurrency trading framework to build and optimize trading strategies. The bot runs nonstop on a Rasberry Pi.
Utilizing the freqtrade high-frequency cryptocurrency trading framework to build and optimize trading strategies. The bot runs nonstop on a Rasberry Pi.

Freqtrade Strategy Repository Please test all scripts and dry run them before using them in live mode Contact me on discord if you have any questions!

Nov 13, 2022
This program is an automated trading bot that uses TDAmeritrades Thinkorswim trading platform's scanners and alerts system.
This program is an automated trading bot that uses TDAmeritrades Thinkorswim trading platform's scanners and alerts system.

Python Trading Bot w/ Thinkorswim Description This program is an automated trading bot that uses TDAmeritrades Thinkorswim trading platform's scanners

Nov 23, 2022
Using GNU Radio and HackRF One to Receive, Analyze and Send ASK/OOK signals

play_with_ask NIS-8016 Lab A code: Recv.grc/py: Receive signals and match with ASK button using HackRF and GNU radio. I use AM demod block(can also in

Jul 4, 2022
Crypto-trading-simulator - Cryptocurrency trading simulator using Python, Streamlit
Crypto-trading-simulator - Cryptocurrency trading simulator using Python, Streamlit

Crypto Trading Simulator Run streamlit run main.py Dependency Python 3 streamli

Jul 2, 2022
A telegram bot script for generating session string using pyrogram and telethon on Telegram bot

String-session-Bot Telegram Bot to generate Pyrogram and Telethon String Session. A star ⭐ from you means a lot to us! Usage Deploy to Heroku Tap on a

Oct 28, 2022
Apr 22, 2022
Sep 11, 2022
This is a cryptocurrency trading bot that analyses Reddit sentiment and places trades on Binance based on reddit post and comment sentiment. If you like this project please consider donating via brave. Thanks.

This is a cryptocurrency trading bot that analyses Reddit sentiment and places trades on Binance based on reddit post and comment sentiment. The bot f

Oct 17, 2022
Python based Algo trading bot for Nifty / Banknifty futures and options

Fully automated Alice Blue Algo Trading with Python on NSE and MCX for Nifty / Crude / Banknifty futures and options , absolutely FREE ! This algo tra

Nov 30, 2022
Fully Dockerized cryptocurrencies Trading Bot, based on Freqtrade engine. Multi instances.

Cryptocurrencies Trading Bot - Freqtrade Manager This automated Trading Bot is based on the amazing Freqtrade one. It allows you to manage many Freqtr

Nov 25, 2022
This bot will pull a stream of tweets based on rules you set and automatically reply to them.

Twitter reply bot This bot will pull a stream of tweets based on rules you set and automatically reply to them. I built this bot in order to help comb

Feb 13, 2022
Hermes Bytecode Reverse Engineering Tool (Assemble/Disassemble Hermes Bytecode)
Hermes Bytecode Reverse Engineering Tool (Assemble/Disassemble Hermes Bytecode)

hbctool A command-line interface for disassembling and assembling the Hermes Bytecode. Since the React Native team created their own JavaScript engine

Dec 2, 2022
Info & tools for reverse engineering the M6 smart fitness band
Info & tools for reverse engineering the M6 smart fitness band

m6-reveng This repo contains information and tools for reverse engineering the $7 M6 smart fitness band. Hardware The SoC (system-on-a-chip) is a Teli

Nov 3, 2022