Published on Oct 12, 2022
The Apache InLong integration framework was originally developed at Tencent and has been used in production for over eight years in big data scenarios to support massive data reporting services. Three years after the project was introduced in the Apache Incubator, it graduated as an Apache top-level project.
InLong manages all stages of the data lifecycle, from collection to landing, and provides different processing modules according to the stage of the data. There are five main modules in the system: Agent, DataProxy, MQ, Sort, and Manager. Pluggable architecture allows modules to be plugged into the system according to specific protocols.
The InLong Agent collects data from heterogeneous sources and writes it to target systems. There is a Job/Task architecture in which each task contains a reader, a sink, and a channel. Connectors based on Apache Flink allow data to be read and written to/from files, SQL databases, Apache Kafka, Apache Pulsar, MongoDB, Elasticsearch, HDFS, and others; Using Apache Flume, the InLong DataProxy connects the InLong Agent with the message queue (MQ); InLong MQ was first developed by Tencent as TubeMQ, where it focused on the transmission of massive data and has a master/broker architecture and two tiers of storage based on memory and files; InLong Sort is an ETL service utilising Apache Flink SQL; InLong Manager provides comprehensive management and control capabilities for data services, including metadata, OpenAPIs, task flow, and authority.
The InLong platform also includes Dashboard and Audit modules for ease of use.
InLong can be deployed as standalone software with MySQL, Flink, and Docker as requirements, or as a Docker Compose YAML or a Helm Chart for Kubernetes deployment.
Currently, InLong is used in a variety of industries, including advertising, payments, social networking, games, artificial intelligence, etc., to provide efficient and convenient customer service.
Tencent Big Data donated Apache InLong to the Apache Incubator in November 2019, and it has reached over 100 contributors and almost 1.000 stars on their GitHub repository.
