Redis-v1 connector


Redis-v1 connector

Parent document: Connectors

BitSail Redis connector supports writing Redis. The main function points are as follows:

  • Support batch write to Redis.
  • Support 4 kinds of data type in Redis.

Maven dependency

<dependency>
   <groupId>com.bytedance.bitsail</groupId>
   <artifactId>connector-redis</artifactId>
   <version>${revision}</version>
</dependency>

Redis writer

Supported data types

We support four kinds of data types in Redis:

  • String, Set, Hash, Sorted Set.

Each data type has its own requirement for input data:

Data typeRequired column numbers1st column2nd column3rd column
String2keyvalue
Set2key of setvalue to insert into set
Hash3key of hashkey to insert to hashvalue to insert to hash
Sorted Set (Zset)3key of Sorted setscorevalue to insert to set

Parameters

The following mentioned parameters should be added to job.writer block when using, for example:

"writer": {
      "class": "com.bytedance.bitsail.connector.redis.sink.RedisSink",
      "redis_data_type": "hash",
      "redis_host": "localhost",
      "redis_port": 6379,
      "columns": [
        {
          "index": 0,
          "type": "string"
        },
        {
          "index": 1,
          "type": "string"
        },
        {
          "index": 2,
          "type": "string"
        }
      ]
    }

Necessary parameters

Param nameRequiredOptional valueDescription
classYesClass name of redis writer, com.bytedance.bitsail.connector.legacy.redis.sink.RedisOutputFormat
redis_hostYeshost of redis
redis_portYesport of redis
columnsYesDescribing fields' name and type, type can be binary or string or other basic types. The binary type is byte[].

Optional parameters

Param nameRequiredOptional valueDescription
writer_parallelism_numNoWriter parallelism num
client_timeout_msNoTimeout of redis connection. Default 60000 ms
ttlNoTtl of inserted data. Default -1 means not setting ttl
ttl_typeNo"DAY", "HOUR", "MINUTE", "SECOND"Time unit of ttl. Default "DAY"
write_batch_intervalNoRedis instruction write batch size. Default 50
redis_data_typeNo"string"
"set"
"hash"
"sorted_set"
"mhash"
Data type to insert. Default "string"
passwordNoPassword of redis
connection_pool_max_totalNoJedis pool max total connection
connection_pool_max_idleNojedis pool max idle connection
connection_pool_min_idleNoJedis pool min idle connection
connection_pool_max_wait_time_msNoJedis pool max wait time in millis
max_attempt_countNoRetryer retry count

Configuration examples: Redis-v1 connector example