News/Tech News

InLong: Apache’s data integration framework

Published on Oct 12, 2022

The Apache InLong integration framework was originally developed at Tencent and has been used in production for over eight years in big data scenarios to support massive data reporting services. Three years after the project was introduced in the Apache Incubator, it graduated as an Apache top-level project.

InLong manages all stages of the data lifecycle, from collection to landing, and provides different processing modules according to the stage of the data. There are five main modules in the system: Agent, DataProxy, MQ, Sort, and Manager. Pluggable architecture allows modules to be plugged into the system according to specific protocols.

The InLong Agent collects data from heterogeneous sources and writes it to target systems. There is a Job/Task architecture in which each task contains a reader, a sink, and a channel. Connectors based on Apache Flink allow data to be read and written to/from files, SQL databases, Apache Kafka, Apache Pulsar, MongoDB, Elasticsearch, HDFS, and others; Using Apache Flume, the InLong DataProxy connects the InLong Agent with the message queue (MQ); InLong MQ was first developed by Tencent as TubeMQ, where it focused on the transmission of massive data and has a master/broker architecture and two tiers of storage based on memory and files; InLong Sort is an ETL service utilising Apache Flink SQL; InLong Manager provides comprehensive management and control capabilities for data services, including metadata, OpenAPIs, task flow, and authority.

The InLong platform also includes Dashboard and Audit modules for ease of use.

InLong can be deployed as standalone software with MySQL, Flink, and Docker as requirements, or as a Docker Compose YAML or a Helm Chart for Kubernetes deployment.

Currently, InLong is used in a variety of industries, including advertising, payments, social networking, games, artificial intelligence, etc., to provide efficient and convenient customer service.

Tencent Big Data donated Apache InLong to the Apache Incubator in November 2019, and it has reached over 100 contributors and almost 1.000 stars on their GitHub repository.

Tech News

VS Code v1.72

VS Code v1.72 now includes community discussions for extension authors

Visual Studio Code’s secret sauce is the vast array of extensions available in its marketplace…


Improved JavaScript interoperability in .NET 7 RC 1

As part of the .NET 7 RC 1 release, we are pleased to announce the introduction of the new wasm-experimental…

Our Latest Blog

Beyond the Basics Advanced Techniques and Tools for Ethical Hacking Professionals

Beyond the Basics: Advanced Techniques and Tools for Ethical Hacking Professionals

Are you an ethical hacking professional looking to take your skills to the next level?...
Read More
Mastering Full Stack Python Development with Django A Comprehensive Guide

Mastering Full Stack Python Development with Django: A Comprehensive Guide

Python is a powerful programming language that has taken the world of web development by...
Read More
Mastering Machine Learning A Beginner's Guide to Python

Mastering Machine Learning: A Beginner’s Guide to Python

Welcome to the world of machine learning! With the ever-increasing demand for artificial intelligence and...
Read More

Follow Us


Browse LSET presentations to understand interesting…

Explore Now

Get complete guides to empower yourself academically…

Explore Now

Learn about information technology and business…

Explore Now