Skip to content

[Bug] kettle stream load插件bug和修复方法 #56355

@shuaizai88

Description

@shuaizai88

Search before asking

  • I had searched in the issues and found no similar issues.

Version

master分支

What's Wrong?

比如上个步骤表输入,包含 id,age,name doris输出的目标表 字段顺序是id,name,age。
在序列化的时候,因为age和name的顺序是调换了的,但是他按照record的下标来取数据,实际生成的csv或者json数据age和name没有调换位置。
并且因为name一般包含中文,导致 name的value在进行convertExternal的时候会报 string无法转换为integer。

Image

What You Expected?

在DorisStreamLoader中看到,data.keynrs 记录了正确的顺序,所以在构造DorisRecordSerializer把这个data.keynrs 传进来。然后通过 下面截图的代码进行数据匹配。这才是正确的顺序。

Image

How to Reproduce?

以下为复现步骤:
1 在mysql里创建表:
CREATE TABLE yutao_user (
id int NOT NULL COMMENT 'id',
name varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '姓名',
age int DEFAULT NULL COMMENT '年龄',
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci COMMENT='鱼涛用户';
insert 几条数据:
INSERT INTO yutao_user (id, name, age) VALUES (1, 'Hui Fat', 222);
INSERT INTO yutao_user (id, name, age) VALUES (2, 'Dong Zhiyuan', 467);
INSERT INTO yutao_user (id, name, age) VALUES (3, 'Hao Ziyi', 448);
2 在doris里创建表
CREATE TABLE yutao_user (
id VARCHAR(48) NOT NULL COMMENT '主键',
name VARCHAR(32) NULL COMMENT '名称',
age INT NULL COMMENT '年龄'
) ENGINE=OLAP
UNIQUE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS AUTO
PROPERTIES (
"replication_allocation" = "tag.location.default: 3")

3 表输入写:select id,age,name from yutao_user

4 doris输出里打乱其顺序, 字段匹配为id,name,age

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions