Published on feb 14, 2023
InfluxData has released Influx IOx, the new version of its database engine. It is now available for use in InfluxDB Cloud.
According to an official InfluxData tweet, the new engine was developed in Rust and can handle metrics, events, and traces. Typically, metrics are time series collected at regular intervals from the source and are used to evaluate a service’s performance and availability. Events are changes in state that occur as a result of certain conditions (i.e. deployed code, HTTP 5XX errors) and can be correlated with metrics to gain additional insight. Distributed systems use traces to show request propagation.
As a result of Influx IOx, InfluxDB Cloud is able to ingest the data in real time and derive metrics on the fly. Observability and distributed tracing, which rely on high cardinality data, can now be handled more efficiently by InfuxDB. As some components depend on others, errors, bottlenecks, and delays can impact the performance of the whole system in a distributed system.
An observability concept called tracing is used to understand how the different pieces work together. Traces provide a view of a request, operation, task, or other unit of work as it moves through the distributed system. By definition, tracing data have a high cardinality, and for a time series database, cardinality can be a problem for unbounded values such as user IDs, IP addresses, and container IDs. Due to how the database indexes data, high cardinality can affect performance.
With the new engine, InfluxDB Cloud can handle tag values of unbounded data without sacrificing performance. A new format for data persistence is Apache Parquet, which allows for better compression (and lower storage costs). When new data arrives, IOx writes it to the columns of the table and saves it to a new Parquet file. IOx writes data in Parquet using some hints to describe the column context. At query time, these hints are used by the query engine to skip over entire Parquet files and/or portions of files that are not interesting. The query engine also uses parallelism, in-memory caching, and pushdown concepts to improve low-latency query performance.
As a result of the use of Apace DataFusion, a Rust-based query engine, developers are now able to gather information using native SQL queries. For more advanced data processing, they can still use the Flux language.
In the time of writing, the new database engine is available in two AWS regions (Virginia (us-east-1) and Frankfurt (eu-central-1), but new availability zones are planned as well as the addition of Azure and Google Cloud support.
