[TOC]
IGinX is an open source polystore system. A polystore system provides an integrated data management service over a set of one or more potentially heterogeneous database/storage engines, serving heterogeneous workloads.
Currently, IGinX directly supports big data service over relational database PostgreSQL, time series databases InfluxDB/IoTDB/TimescaleDB/OpenTSDB, and Parquet data files.
Since ZooKeeper, IGinX and IoTDB are all developed using Java, Java needs to be installed first. If a running environment of JDK >= 1.8 has been installed locally, skip this step entirely.
First, visit the official Java website to download the JDK package for your current system.
Installation
$ cd ~/Downloads
$ tar -zxf jdk-8u181-linux-x64.gz # unzip files
$ mkdir /opt/jdk
$ mv jdk-1.8.0_181 /opt/jdk/
Edit the ~/.bashrc file and add the following two lines at the end of the file:
export JAVA_HOME = /usr/jdk/jdk-1.8.0_181
export PATH=$PATH:$JAVA_HOME/bin
Load the file with the changed configuration (into shell scripts):
$ source ~/.bashrc
$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
If the words above are displayed, it means the installation was successful.
Maven is a build automation tool used primarily to build and managa Java projects. If you need to compile from the source code, you also need to install a Maven environment >= 3.6. Otherwise, skip this step entirely.
$ wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
$ tar -xvf apache-maven-3.3.9-bin.tar.gz
$ sudo mv -f apache-maven-3.3.9 /usr/local/
Edit the ~/.bashrc file and add the following two lines at the end of the file:
export MAVEN_HOME=/usr/local/apache-maven-3.3.9
export PATH=${PATH}:${MAVEN_HOME}/bin
Load the file with the changed configuration (into shell scripts):
$ source ~/.bashrc
$ mvn -v
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
If the words above are displayed, that means the installation was successful.
The project requires Node.js installation. If you already have it installed on your device, skip this part.
$ cd ~
$ wget https://nodejs.org/dist/v18.16.1/node-v18.16.1-linux-x64.tar.xz
$ tar -zxvf node-v18.16.1-linux-x64.tar.xz
$ sudo mv -f node-v18.16.1-linux-x64 /usr/local/
Attach code below to file ~/.bashrc
export NODEJS_HOME=/usr/local/node-v18.16.1-linux-x64
export PATH=${PATH}:${NODEJS_HOME}/bin
Load file ~/.bashrc to apply changes.
$ source ~/.bashrc
Confirm the Installation by testing npm command
$ npm -v
9.5.1
ZooKeeper enables highly reliable distributed coordination. In IGinX, ZooKeeper makes it possible to have more than one IGinX instances deployed.
$ cd ~
$ wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.2/apache-zookeeper-3.7.2-bin.tar.gz
$ tar -zxvf apache-zookeeper-3.7.2-bin.tar.gz
$ cd apache-zookeeper-3.7.2-bin/
$ mkdir data
$ cp conf/zoo_sample.cfg conf/zoo.cfg
Modify fileconf/zoo.cfg.
# Original conf/zoo.cfg
dataDir=/tmp/zookeeper
# ↓ change to
dataDir=data
IoTDB is Apache’s Apache IoT native database with high performance for data management and analysis, deployable on the edge and the cloud.
The specific installation method is as follows:
$ cd ~
$ wget https://github.com/IGinX-THU/IGinX-resources/raw/main/resources/apache-iotdb-0.12.6-server-bin.zip
$ unzip apache-iotdb-0.12.6-server-bin.zip
Compile with source code. If you need to modify code yourself, you can use this installation method.
Fetch the latest development version and build it locally.
$ cd ~
$ git clone git@github.com:IGinX-THU/IGinX.git
$ cd IGinX
$ mvn clean install -Dmaven.test.skip=true
The following words are displayed, indicating that the IGinX build is successful:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for IGinX 0.7.1:
[INFO]
[INFO] IGinX .............................................. SUCCESS [ 20.674 s]
[INFO] IGinX Thrift ....................................... SUCCESS [01:18 min]
[INFO] IGinX Shared ....................................... SUCCESS [ 8.101 s]
[INFO] IGinX Session ...................................... SUCCESS [ 3.168 s]
[INFO] IGinX Antlr ........................................ SUCCESS [ 16.170 s]
[INFO] IGinX Core ......................................... SUCCESS [03:35 min]
[INFO] IGinX Client ....................................... SUCCESS [ 11.159 s]
[INFO] IGinX JDBC ......................................... SUCCESS [ 3.426 s]
[INFO] IGinX IoTDB12 ...................................... SUCCESS [ 31.081 s]
[INFO] IGinX InfluxDB ..................................... SUCCESS [ 17.516 s]
[INFO] IGinX OpenTSDB ..................................... SUCCESS [ 11.103 s]
[INFO] IGinX PostgreSQL ................................... SUCCESS [ 4.449 s]
[INFO] IGinX Parquet ...................................... SUCCESS [01:28 min]
[INFO] IGinX Redis ........................................ SUCCESS [ 9.875 s]
[INFO] IGinX MongoDB ...................................... SUCCESS [ 10.020 s]
[INFO] IGinX Example ...................................... SUCCESS [ 3.864 s]
[INFO] IGinX Test ......................................... SUCCESS [ 5.490 s]
[INFO] Zeppelin on IGinX .................................. SUCCESS [ 25.019 s]
[INFO] IGinX Tools for CSV export ......................... SUCCESS [ 2.860 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:25 min
[INFO] Finished at: 2023-06-20T10:28:24+08:00
[INFO] ------------------------------------------------------------------------
First of all, you need to launch IoTDB.
$ cd ~
$ cd apache-iotdb-0.12.6-server-bin/
$ ./sbin/start-server.sh
The following display of words means the IoTDB installation was successful：
2021-05-27 08:21:07,440 [main] INFO o.a.i.d.s.t.ThriftService:125 - IoTDB: start RPC ServerService successfully, listening on ip 0.0.0.0 port 6667
2021-05-27 08:21:07,440 [main] INFO o.a.i.db.service.IoTDB:129 - IoTDB is set up, now may some sgs are not ready, please wait several seconds...
2021-05-27 08:21:07,448 [main] INFO o.a.i.d.s.UpgradeSevice:109 - finish counting upgrading files, total num:0
2021-05-27 08:21:07,449 [main] INFO o.a.i.d.s.UpgradeSevice:74 - Waiting for upgrade task pool to shut down
2021-05-27 08:21:07,449 [main] INFO o.a.i.d.s.UpgradeSevice:76 - Upgrade service stopped
2021-05-27 08:21:07,449 [main] INFO o.a.i.db.service.IoTDB:146 - Congratulation, IoTDB is set up successfully. Now, enjoy yourself!
2021-05-27 08:21:07,450 [main] INFO o.a.i.db.service.IoTDB:93 - IoTDB has started.
Then, launch ZooKeeper.
$ cd ~
$ cd apache-zookeeper-3.7.2-bin/
$ ./bin/zkServer.sh start
You will see messages as below if ZooKeeper is successfully started.
ZooKeeper JMX enabled by default
Using config: /home/root/apache-zookeeper-3.7.2-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
Using source code to launch
$ cd ~
$ cd IGinX
$ chmod +x startIginX.sh # enable permissions for startup scripts
$ ./startIginX.sh
The following display of words means the IGinX installation was successful：
May 27, 2021 8:32:19 AM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [127.0.0.1:6666]
May 27, 2021 8:32:19 AM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
08:32:19.446 [Thread-0] INFO cn.edu.tsinghua.iginx.rest.RestServer - IGinX REST server has been available at http://127.0.0.1:6666/.
After the startup is complete, you can easily use the RESTful interface to write and query data to IGinX.
Create a file insert.json and add the following into it:
[
{
"name": "archive_file_tracked",
"datapoints": [
[1359788400000, 123.3],
[1359788300000, 13.2 ],
[1359788410000, 23.1 ]
],
"tags": {
"host": "server1",
"data_center": "DC1"
}
},
{
"name": "archive_file_search",
"timestamp": 1359786400000,
"value": 321,
"tags": {
"host": "server2"
}
}
]
Insert data into the database using the following command:
$ curl -XPOST -H'Content-Type: application/json' -d @insert.json http://127.0.0.1:6666/api/v1/datapoints
After inserting data, you can also query the data just written using the RESTful interface.
Create a file query.json and write the following data into it:
{
"start_absolute" : 1,
"end_relative": {
"value": "5",
"unit": "days"
},
"time_zone": "Asia/Kabul",
"metrics": [
{
"name": "archive_file_tracked"
},
{
"name": "archive_file_search"
}
]
}
Enter the following command to query the data:
$ curl -XPOST -H'Content-Type: application/json' -d @query.json http://127.0.0.1:6666/api/v1/datapoints/query
The command will return information about the data point just inserted:
{
"queries": [
{
"sample_size": 3,
"results": [
{
"name": "archive_file_tracked",
"group_by": [
{
"name": "type",
"type": "number"
}
],
"tags": {
"data_center": [
"DC1"
],
"host": [
"server1"
]
},
"values": [
[
1359788300000,
13.2
],
[
1359788400000,
123.3
],
[
1359788410000,
23.1
]
]
}
]
},
{
"sample_size": 1,
"results": [
{
"name": "archive_file_search",
"group_by": [
{
"name": "type",
"type": "number"
}
],
"tags": {
"host": [
"server2"
]
},
"values": [
[
1359786400000,
321.0
]
]
}
]
}
]
}
If you see the following information returned, it means you are able to successfully use RESTful interface to write and query data to IGinX.
For more interfaces, please refer to the official IGinX manual.
If you want to use a different interface, there is another option.
In addition to the RESTful interface, IGinX also provides the RPC data access interface. For this specific interface, please refer to the official IGinX manual.
At the same time, IGinX also provides some official examples, showing the most common usage of the RPC interface.
Below is a short tutorial on how to use it.
Since the IGinX jars have not been released to the Maven central repository, if you want to use it, you need to manually install it to the local Maven repository.
The specific installation method is as follows:
# download the newest IGinX release version source code package
$ wget https://github.com/IGinX-THU/IGinX/archive/refs/tags/v0.7.0.zip
# Unzip the source package
$ tar -zxvf v0.7.0.tar.gz
# go to the main project's directory
$ cd IGinX-rc-v0.7.0
# Install to local Maven repository
$ mvn clean install -DskipTests
Specifically, when using it, you only need to introduce the following dependencies in the pom file of the corresponding project:
<dependency>
<groupId>cn.edu.tsinghua</groupId>
<artifactId>iginx-core</artifactId>
<version>0.7.0</version>
</dependency>
Before accessing IGinX, you first need to create a session and try to connect. The Session constructor has 4 parameters, which are the ip and port IGinX will to connect to, and the username and password for IGinX authentication. The current authentication system is still being written, so the account name and password to access the backend IGinX can directly fill in root:
Session session = new Session("127.0.0.1", 6888, "root", "root");
session.openSession();
You can then try to insert data into IGinX. Since IGinX supports the creation of time-series when data is written for the first time, there is no need to call the relevant series creation interface in advance. IGinX provides row-style and column-style data writing interfaces.
The following is an example of using the column-style data writing interface:
private static void insertColumnRecords(Session session) throws SessionException, ExecutionException {
List<String> paths = new ArrayList<>();
paths.add("sg.d1.s1");
paths.add("sg.d2.s2");
paths.add("sg.d3.s3");
paths.add("sg.d4.s4");
int size = 1500;
long[] timestamps = new long[size];
for (long i = 0; i < size; i++) {
timestamps[(int) i] = i;
}
Object[] valuesList = new Object[4];
for (long i = 0; i < 4; i++) {
Object[] values = new Object[size];
for (long j = 0; j < size; j++) {
if (i < 2) {
values[(int) j] = i + j;
} else {
values[(int) j] = RandomStringUtils.randomAlphanumeric(10).getBytes();
}
}
valuesList[(int) i] = values;
}
List<DataType> dataTypeList = new ArrayList<>();
for (int i = 0; i < 2; i++) {
dataTypeList.add(DataType.LONG);
}
for (int i = 0; i < 2; i++) {
dataTypeList.add(DataType.BINARY);
}
session.insertColumnRecords(paths, timestamps, valuesList, dataTypeList, null);
}
After completing the data writing, you can use the data query interface to query the data just written:
private static void queryData(Session session) throws SessionException, ExecutionException {
List<String> paths = new ArrayList<>();
paths.add("sg.d1.s1");
paths.add("sg.d2.s2");
paths.add("sg.d3.s3");
paths.add("sg.d4.s4");
long startTime = 100L;
long endTime = 200L;
SessionQueryDataSet dataSet = session.queryData(paths, startTime, endTime);
dataSet.print();
}
You can also use the downsampling aggregation query interface to query the interval statistics of the data:
private static void downsampleQuery(Session session) throws SessionException, ExecutionException {
List<String> paths = new ArrayList<>();
paths.add("sg.d1.s1");
paths.add("sg.d2.s2");
long startTime = 100L;
long endTime = 1101L;
// MAX
SessionQueryDataSet dataSet = session.downsampleQuery(paths, startTime, endTime, AggregateType.MAX, 100);
dataSet.print();
// MIN
dataSet = session.downsampleQuery(paths, startTime, endTime, AggregateType.MIN, ROW_INTERVAL * 100);
dataSet.print();
// FIRST
dataSet = session.downsampleQuery(paths, startTime, endTime, AggregateType.FIRST, ROW_INTERVAL * 100);
dataSet.print();
// LAST
dataSet = session.downsampleQuery(paths, startTime, endTime, AggregateType.LAST, ROW_INTERVAL * 100);
dataSet.print();
// COUNT
dataSet = session.downsampleQuery(paths, startTime, endTime, AggregateType.COUNT, ROW_INTERVAL * 100);
dataSet.print();
// SUM
dataSet = session.downsampleQuery(paths, startTime, endTime, AggregateType.SUM, ROW_INTERVAL * 100);
dataSet.print();
// AVG
dataSet = session.downsampleQuery(paths, startTime, endTime, AggregateType.AVG, ROW_INTERVAL * 100);
dataSet.print();
}
After the session is completed, you need to manually close and release your connection from your terminal/backend:
session.closeSession();
For the full version of the code, please refer to: https://github.com/IGinX-THU/IGinX/blob/main/example/src/main/java/cn/edu/tsinghua/iginx/session/IoTDBSessionExample.java