BitSail

End-to-End Test

End-to-End Test

BitSail supports End-to-End (E2E) test for most connectors.

0. Prerequisite

E2E framework use docker to construct test data resources and test executors, so you need to install docker at first.

1. What connectors and data sources are supported?

Curent E2E test framework is aim at connector V2, e.g., connector-redis, connector-rocketmq.

Connectors under bitsail-connectors-legacy cannot be test.

  • Supported test cases: bitsail-test/bitsail-test-end-to-end/bitsail-test-e2e-connector-v1

2. How to run E2E test

1. Run in Local IDE

You can directly run E2E test in local IDE. You may need to run build.sh beforehand.

2. Run single case in console

You can use test-e2e.sh script to test specific case, for example:

bash test-e2e.sh bitsail-test-e2e-connector-v1-clickhouse

3. Run all cases

mvn clean verify -DskipUT=true -DskipITCase=true -DskipE2E=false -D"checkstyle.skip"=true -D"license.skipAddThirdParty"=true --no-snapshot-updates -am -P _maven.oracle.com_

3. How to prepare a test case

1. Prepare data source

There are some existing data sources are supported in bitsail-test/bitsail-test-end-to-end/bitsail-test-e2e-connector-v1.

If you cannot find target data source, then you need to implement it at first.

2. Prepare test script

Use connector-redis as example, you can use the following script:

{
   "job": {
       "common": {
       "job_id": -2413,
       "job_name": "bitsail_fake_to_redis_e2e_test",
       "instance_id": -20413,
       "user_name": "user"
   },
   "reader": {
       "class": "com.bytedance.bitsail.connector.fake.source.FakeSource",
       "total_count": 300,
       "rate": 100,
       "null_percentage": 0,
       "unique_fields": "fake_key",
       "columns": [
           {
               "name": "fake_key",
               "type": "string",
               "properties": "unique"
           },
           {
               "name": "fake_value",
               "type": "string"
           }
       ]
       },
   "writer": {
       "class": "com.bytedance.bitsail.connector.redis.sink.RedisSink",
       "redis_data_type": "string",
       "columns": [
           {
               "name": "fake_key",
               "type": "string"
           },
           {
               "name": "fake_value",
               "type": "string"
           }
       ]
       }
   }
}

3. Prepare test class

After data sources and scripts being prepared, you only need to read the scripts and submit to test.

Take fake-to-redis case as an example:

  • AbstractE2ETest: Abstract E2E test class. Users can extend this class to submit script to E2E test executor.
public class FakeToRedisE2ETest extends AbstractE2ETest {

  @Test
  public void testFakeToRedis() throws Exception {
    BitSailConfiguration jobConf = BitSailConfiguration.from(
        new File(Paths.get(getClass().getClassLoader()
            .getResource("fake_to_redis.json")
            .toURI()).toString()));
    jobConf.set(FakeReaderOptions.TOTAL_COUNT, 500);

    // Check if there are 500 keys in redis.
    submitFlink11Job(jobConf,
        "test_fake_to_redis",
        dataSource -> Assert.assertEquals(
            500,
            ((RedisDataSource) dataSource).getKeyCount()
        ));
  }
}