145 blog
0%

apache-doris(k8s部署)

发表于 更新于 分类于

apache-doris

介绍

该介绍来自官网

Apache Doris 是一个基于 MPP 架构的高性能、实时的分析型数据库，以极速易用的特点被人们所熟知，仅需亚秒级响应时间即可返回海量数据下的查询结果，不仅可以支持高并发的点查询场景，也能支持高吞吐的复杂分析场景。基于此，Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、数据湖联邦查询加速等使用场景，用户可以在此之上构建用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。

Apache Doris 最早是诞生于百度广告报表业务的 Palo 项目，2017 年正式对外开源，2018 年 7 月由百度捐赠给 Apache 基金会进行孵化，之后在 Apache 导师的指导下由孵化器项目管理委员会成员进行孵化和运营。目前 Apache Doris 社区已经聚集了来自不同行业数百家企业的 400 余位贡献者，并且每月活跃贡献者人数也超过 100 位。 2022 年 6 月，Apache Doris 成功从 Apache 孵化器毕业，正式成为 Apache 顶级项目（Top-Level Project，TLP）

Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体，截止目前， Apache Doris 已经在全球超过 2000 家企业的生产环境中得到应用，在中国市值或估值排行前 50 的互联网公司中，有超过 80% 长期使用 Apache Doris，包括百度、美团、小米、京东、字节跳动、腾讯、网易、快手、微博、贝壳等。同时在一些传统行业如金融、能源、制造、电信等领域也有着丰富的应用。

官网地址：https://doris.apache.org/

架构

Doris整体架构如下图所示，Doris 架构非常简单，只有两类进程

  • Frontend（FE），主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。
  • Backend（BE），主要负责数据存储、查询计划的执行。

Image description

在使用接口方面，Doris 采用 MySQL 协议，高度兼容 MySQL 语法，支持标准 SQL，用户可以通过各类客户端工具来访问 Doris，并支持与 BI 工具的无缝对接。

在存储引擎方面，Doris 采用列式存储，按列进行数据的编码压缩和读取，能够实现极高的压缩比，同时减少大量非相关数据的扫描，从而更加有效利用 IO 和 CPU 资源。

fe

fe角色

fe有两种角色：Observer和Follower

Follower是可读可写的角色，而Observer是只读的角色。其中master节点也是从Follower中选举出来的，Observer不参与选举。

单节点

单节点的fe没有特别需要注意的地方，挂掉重启也不会有任何影响，只要不是在进程工作的重启对数据和服务都没任何影响。单节点的fe只有FLLOWER身份的节点

多节点

多节点有两种形式

1.单Follower和多Observer ：相当于是一写多读，这个是最好运维的，也是最常见的方式，缺点是Follower依然是单点故障，优点是Follower出了故障可以手动把Observer配置为Follower，相当于Observer都是Follower的备份。

2.多Follower：每个节点都是Follower节点，缺点是数据出现故障或者节点全部同时挂掉难恢复，优点是高可用。

be

be作为fe的执行器无任何状态，只需要加入到FE集群即可。

broker

Broker 是用于访问外部数据源（如 hdfs）的进程，broker和be基本一致。

部署

这里部署只涉及Kubernetes部署，DOCKER以及二进制部署请参考官方文档：https://doris.apache.org/zh-CN/docs/dev/install/standard-deployment

kubernetes部署

1
2
3
关于K8S部署doris：首先这里doris本身对k8s容器部署的支持其实是比较差的，虽然新版本中doris支持k8s模式部署，但我测试使用k8s模式部署实际还是问题的，因为Doris
节点ip原本是固定的，所以pod重启更换ip集群是识别不到的，不知道新版本2.0是否有解决这个问题。
问题：doris的每一个节点都需要固定ip，但k8s pod的ip是无法固定的，除非以主机网络模式部署。
1
2
3
4
5
部署说明：
1.因为doris是以ip进行通信，所以这里每个节点的IP需要固定，这里要用到K8S主机网络模式，每一个doris节点需要直接监听宿主机ip以及端口。
2.集群中每个节点需要单独部署,无法通过depolyment或statefulset直接部署多节点。问题说明:由于每个pod的节点以及ip是固定的，所以每个pod必须部署在对应节点以及监听ip上，如果使用副本控制器很有可能会出现重启后更换部署的node。
3.当fe FLLOWER节点同时全部挂掉或者只剩下1个节点时，fe集群将不再正常工作，这里可以考虑使用单FLLOWER。但broker或者be节点无论如何重启都不会影响。
4.原本部署节点都需要手动在fe节点上添加其他fe节点以及be节点，但这里由于用到了init启动脚本（下面Dokcer打包的时候会用到），所以只需要传递变量即可在启动的时候自动添加节点到集群。如果不用init启动脚本用自身的start脚本就需要手动维护fe的数据库，手动添加每一个fe、be、broker节点。

fe部署

这里fe采用多Follower的方式部署，部署3节点高可用fe集群

首先需要构建fe镜像，dockerhub也有官方的镜像，但版本一般与官网差得比较远，如果对版本要求不高可以直接使用dockerhub的官方镜像。

dockerhub地址：https://hub.docker.com/r/apache/doris/tags

因为这里目前官方的稳定版本为1.2.4.1，但官方镜像还没有能提供到该版本，所以这里选择手动构建镜像部署。

镜像构建相关文档：https://doris.apache.org/zh-CN/docs/dev/install/construct-docker/construct-docker-image

1.下载doris fe的二进制包
1
wget https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.4.1-rc01/apache-doris-fe-1.2.4.1-bin-x86_64.tar.xz --no-check-certificate
2.Dockerfile文件准备
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
FROM openjdk:8u342-jdk

# set environment variables
ENV JAVA_HOME="/usr/local/openjdk-8/" \
	PATH="/opt/apache-doris/fe/bin:${PATH}"

# download the software to the mirror and replace it as needed
ADD ./apache-doris-fe-1.2.4.1-bin-x86_64.tar.xz /opt/

RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list && \
    sed -i 's/security.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list

# deploy software
RUN apt-get update && \
	apt-get install -y default-mysql-client && \
	apt-get clean && \
	mkdir /opt/apache-doris && \
	cd /opt && \
	mv apache-doris-fe-1.2.4.1-bin-x86_64 /opt/apache-doris/fe

ADD ./init_fe.sh /opt/apache-doris/fe/bin
RUN chmod 755 /opt/apache-doris/fe/bin/init_fe.sh

CMD ["/opt/apache-doris/fe/bin/init_fe.sh"]
3.init_fe文件准备（初始化启动脚本）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail
shopt -s nullglob

DORIS_HOME="/opt/apache-doris"

# Obtain necessary and basic information to complete initialization

# logging functions
# usage: doris_[note|warn|error] $log_meg
#    ie: doris_warn "task may fe risky!"
#   out: 2023-01-08T19:08:16+08:00 [Warn] [Entrypoint]: task may fe risky!
doris_log() {
    local type="$1"
    shift
    # accept argument string or stdin
    local text="$*"
    if [ "$#" -eq 0 ]; then text="$(cat)"; fi
    local dt="$(date -Iseconds)"
    printf '%s [%s] [Entrypoint]: %s\n' "$dt" "$type" "$text"
}
doris_note() {
    doris_log Note "$@"
}
doris_warn() {
    doris_log Warn "$@" >&2
}
doris_error() {
    doris_log ERROR "$@" >&2
    exit 1
}

# check to see if this file is being run or sourced from another script
_is_sourced() {
    [ "${#FUNCNAME[@]}" -ge 2 ] &&
        [ "${FUNCNAME[0]}" = '_is_sourced' ] &&
        [ "${FUNCNAME[1]}" = 'source' ]
}

docker_setup_env() {
    declare -g DATABASE_ALREADY_EXISTS BUILD_TYPE_K8S
    if [ -d "${DORIS_HOME}/fe/doris-meta/image" ]; then
        DATABASE_ALREADY_EXISTS='true'
    fi
}

# Check the variables required for   startup
docker_required_variables_env() {
    declare -g RUN_TYPE
    if [ -n "$BUILD_TYPE" ]; then
        RUN_TYPE="K8S"
        if [[ $BUILD_TYPE =~ ^([kK]8[sS])$ ]]; then
            doris_warn "BUILD_TYPE" $BUILD_TYPE
        else
            doris_error "BUILD_TYPE rule error！example: [k8s], Default Value: docker."
        fi
        return
    fi

    if [[ -n "$FE_SERVERS" && -n "$FE_ID" ]]; then
        RUN_TYPE="ELECTION"
        if [[ $FE_SERVERS =~ ^.+:[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}:[1-6]{0,1}[0-9]{1,4}(,.+:[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}:[1-6]{0,1}[0-9]{1,4})*$ || $FE_SERVERS =~ ^([0-9a-fA-F]{1,4}:){7,7}([0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,6}(:[0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,5}((:[0-9a-fA-F]{1,4}){1,2}|:)|([0-9a-fA-F]{1,4}:){1,4}((:[0-9a-fA-F]{1,4}){1,3}|:)|([0-9a-fA-F]{1,4}:){1,3}((:[0-9a-fA-F]{1,4}){1,4}|:)|([0-9a-fA-F]{1,4}:){1,2}((:[0-9a-fA-F]{1,4}){1,5}|:)|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6}|:)|:((:[0-9a-fA-F]{1,4}){1,7}|:)$ ]]; then
            doris_warn "FE_SERVERS" $FE_SERVERS
        else
            doris_error "FE_SERVERS rule error！example: \$FE_NAME:\$FE_HOST_IP:\$FE_EDIT_LOG_PORT[,\$FE_NAME:\$FE_HOST_IP:\$FE_EDIT_LOG_PORT]..."
        fi
        if [[ $FE_ID =~ ^[1-9]{1}$ ]]; then
            doris_warn "FE_ID" $FE_ID
        else
            doris_error "FE_ID rule error！If FE is the role of Master, please set FE_ID=1, and ensure that all IDs correspond to the IP of the current node."
        fi
        return
    fi
    doris_note $FE_MASTER_IP "  " $FE_MASTER_PORT "  " $FE_CURRENT_IP "  " $FE_CURRENT_PORT
    if [[ -n "$FE_MASTER_IP" && -n "$FE_MASTER_PORT" && -n "$FE_CURRENT_IP" && -n "$FE_CURRENT_PORT" ]]; then
        RUN_TYPE="ASSIGN"
        if [[ $FE_MASTER_IP =~ ^[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}$ || $FE_MASTER_IP =~ ^([0-9a-fA-F]{1,4}:){7,7}([0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,6}(:[0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,5}((:[0-9a-fA-F]{1,4}){1,2}|:)|([0-9a-fA-F]{1,4}:){1,4}((:[0-9a-fA-F]{1,4}){1,3}|:)|([0-9a-fA-F]{1,4}:){1,3}((:[0-9a-fA-F]{1,4}){1,4}|:)|([0-9a-fA-F]{1,4}:){1,2}((:[0-9a-fA-F]{1,4}){1,5}|:)|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6}|:)|:((:[0-9a-fA-F]{1,4}){1,7}|:)$ ]]; then
            doris_warn "FE_MASTER_IP" $FE_MASTER_IP
        else
            doris_error "FE_MASTER_IP rule error！example: \$FE_MASTER_IP"
        fi
        if [[ $FE_MASTER_PORT =~ ^[1-6]{0,1}[0-9]{1,4}$ ]]; then
            doris_warn "FE_MASTER_PORT" $FE_MASTER_PORT
        else
            doris_error "FE_MASTER_PORT rule error！example: \$FE_MASTER_EDIT_LOG_HOST."
        fi
        if [[ $FE_CURRENT_IP =~ ^[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}$ || $FE_CURRENT_IP =~ ^([0-9a-fA-F]{1,4}:){7,7}([0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,6}(:[0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,5}((:[0-9a-fA-F]{1,4}){1,2}|:)|([0-9a-fA-F]{1,4}:){1,4}((:[0-9a-fA-F]{1,4}){1,3}|:)|([0-9a-fA-F]{1,4}:){1,3}((:[0-9a-fA-F]{1,4}){1,4}|:)|([0-9a-fA-F]{1,4}:){1,2}((:[0-9a-fA-F]{1,4}){1,5}|:)|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6}|:)|:((:[0-9a-fA-F]{1,4}){1,7}|:)$ ]]; then
            doris_warn "FE_CURRENT_IP" $FE_CURRENT_IP
        else
            doris_error "FE_CURRENT_IP rule error！example: \$FE_CURRENT_IP"
        fi
        if [[ $FE_CURRENT_PORT =~ ^[1-6]{0,1}[0-9]{1,4}$ ]]; then
            doris_warn "FE_CURRENT_PORT" $FE_CURRENT_PORT
        else
            doris_error "FE_CURRENT_PORT rule error！example: \$FE_CURRENT_EDIT_LOG_HOST."
        fi
        return
    fi
    doris_error EOF "
                 Note that you did not configure the required parameters!
                 plan 1:
                 BUILD_TYPE
                 plan 2:
                 FE_SERVERS & FE_ID
                 plan 3:
                 FE_MASTER_IP & FE_MASTER_PORT & FE_CURRENT_IP & FE_CURRENT_PORT"
                EOF
}
get_doris_fe_args() {
    declare -g MASTER_FE_IP CURRENT_FE_IP MASTER_FE_EDIT_PORT CURRENT_FE_EDIT_PORT PRIORITY_NETWORKS CURRENT_FE_IS_MASTER
    if [ $RUN_TYPE == "ELECTION" ]; then
        local feServerArray=($(echo "${FE_SERVERS}" | awk '{gsub (/,/," "); print $0}'))
        for i in "${feServerArray[@]}"; do
            val=${i}
            val=${val// /}
            tmpFeName=$(echo "${val}" | awk -F ':' '{ sub(/fe/, ""); sub(/ /, ""); print$1}')
            tmpFeIp=$(echo "${val}" | awk -F ':' '{ sub(/ /, ""); print$2}')
            tmpFeEditLogPort=$(echo "${val}" | awk -F ':' '{ sub(/ /, ""); print$3}')
            check_arg "TMP_FE_NAME" $tmpFeName
            feIpArray[$tmpFeName]=${tmpFeIp}
            check_arg "TMP_FE_EDIT_LOG_PORT" $tmpFeEditLogPort
            feEditLogPortArray[$tmpFeName]=${tmpFeEditLogPort}
        done

        MASTER_FE_IP=${feIpArray[1]}
        check_arg "MASTER_FE_IP" $MASTER_FE_IP
        MASTER_FE_EDIT_PORT=${feEditLogPortArray[1]}
        check_arg "MASTER_FE_EDIT_PORT" $MASTER_FE_EDIT_PORT
        CURRENT_FE_IP=${feIpArray[FE_ID]}
        check_arg "CURRENT_FE_IP" $CURRENT_FE_IP
        CURRENT_FE_EDIT_PORT=${feEditLogPortArray[FE_ID]}
        check_arg "CURRENT_FE_EDIT_PORT" $CURRENT_FE_EDIT_PORT

        if [ ${MASTER_FE_IP} == ${CURRENT_FE_IP} ]; then
            CURRENT_FE_IS_MASTER=true
        else
            CURRENT_FE_IS_MASTER=false
        fi

        PRIORITY_NETWORKS=$(echo "${CURRENT_FE_IP}" | awk -F '.' '{print$1"."$2"."$3".0/24"}')
        check_arg "PRIORITY_NETWORKS" $PRIORITY_NETWORKS

        doris_note "FE_IP_ARRAY = ${feIpArray[*]}"
        doris_note "FE_EDIT_LOG_PORT_ARRAY = ${feEditLogPortArray[*]}"
        doris_note "MASTER_FE = ${feIpArray[1]}:${feEditLogPortArray[1]}"
        doris_note "CURRENT_FE = ${CURRENT_FE_IP}:${CURRENT_FE_EDIT_PORT}"
        doris_note "PRIORITY_NETWORKS = ${PRIORITY_NETWORKS}"

    elif [ $RUN_TYPE == "ASSIGN" ]; then
        MASTER_FE_IP=${FE_MASTER_IP}
        check_arg "MASTER_FE_IP" $MASTER_FE_IP
        MASTER_FE_EDIT_PORT=${FE_MASTER_PORT}
        check_arg "MASTER_FE_EDIT_PORT" $MASTER_FE_EDIT_PORT
        CURRENT_FE_IP=${FE_CURRENT_IP}
        check_arg "CURRENT_FE_IP" $CURRENT_FE_IP
        CURRENT_FE_EDIT_PORT=${FE_CURRENT_PORT}
        check_arg "CURRENT_FE_EDIT_PORT" $CURRENT_FE_EDIT_PORT

        if [ ${MASTER_FE_IP} == ${CURRENT_FE_IP} ]; then
            CURRENT_FE_IS_MASTER=true
        else
            CURRENT_FE_IS_MASTER=false
        fi

        PRIORITY_NETWORKS=$(echo "${CURRENT_FE_IP}" | awk -F '.' '{print$1"."$2"."$3".0/24"}')
        check_arg "PRIORITY_NETWORKS" $PRIORITY_NETWORKS
    fi

    # check fe start
    check_fe_status true

}

add_priority_networks() {
    doris_note "add priority_networks ${1} to ${DORIS_HOME}/fe/conf/fe.conf"
    echo "priority_networks = ${1}" >>${DORIS_HOME}/fe/conf/fe.conf
}

# Execute sql script, passed via stdin
# usage: docker_process_sql sql_script
docker_process_sql() {
    set +e
    mysql -uroot -P9030 -h${MASTER_FE_IP} --comments "$@" 2>/dev/null
}

docker_setup_db() {
    set +e
    # check fe status
    local is_fe_start=false
    if [ ${CURRENT_FE_IS_MASTER} == true ]; then
        doris_note "Current FE is Master FE!  No need to register again！"
        return
    fi
    for i in {1..300}; do
        docker_process_sql <<<"alter system add FOLLOWER '${CURRENT_FE_IP}:${CURRENT_FE_EDIT_PORT}'"
        register_fe_status=$?
        if [[ $register_fe_status == 0 ]]; then
            doris_note "FE successfully registered！"
            is_fe_start=true
            break
        else
            check_fe_status
            if [ -n "$CURRENT_FE_ALREADY_EXISTS" ]; then
                doris_warn "Same frontend already exists! No need to register again！"
                break
            fi
            if [[ $(($i % 20)) == 1 ]]; then
                doris_warn "register_fe_status: ${register_fe_status}"
                doris_warn "FE failed registered!"
            fi
        fi
        if [[ $(($i % 20)) == 1 ]]; then
            doris_note "ADD FOLLOWER failed, retry."
        fi
        sleep 1
    done
    if ! [[ $is_fe_start ]]; then
        doris_error "Failed to register CURRENT_FE to FE！Tried 30 times！Maybe FE Start Failed！"
    fi
}

# Check whether the passed parameters are empty to avoid subsequent task execution failures. At the same time,
# enumeration checks can fe added, such as checking whether a certain parameter appears repeatedly, etc.
check_arg() {
    if [ -z $2 ]; then
        doris_error "$1 is null!"
    fi
}


check_fe_status() {
    set +e
    declare -g CURRENT_FE_ALREADY_EXISTS
    if [[ ${CURRENT_FE_IS_MASTER} == true ]]; then
        doris_note "Current FE is Master FE!  No need check fe status！"
        return
    fi
    for i in {1..300}; do
        if [[ $1 == true ]]; then
            docker_process_sql <<<"show frontends" | grep "[[:space:]]${MASTER_FE_IP}[[:space:]]" | grep "[[:space:]]${MASTER_FE_EDIT_PORT}[[:space:]]"
        else
            docker_process_sql <<<"show frontends" | grep "[[:space:]]${CURRENT_FE_IP}[[:space:]]" | grep "[[:space:]]${CURRENT_FE_EDIT_PORT}[[:space:]]"
        fi
        fe_join_status=$?
        if [[ "${fe_join_status}" == 0 ]]; then
            if [[ $1 == true ]]; then
                doris_note "Master FE is started!"
            else
                doris_note "Verify that CURRENT_FE is registered to FE successfully"
            fi
            CURRENT_FE_ALREADY_EXISTS=true
            break
        else
            if [[ $(($i % 20)) == 1 ]]; then
                if [[ $1 == true ]]; then
                    doris_note "Master FE is not started, retry."
                else
                    doris_warn "Verify that CURRENT_FE is registered to FE failed, retry."
                fi
            fi
        fi
        if [[ $(($i % 20)) == 1 ]]; then
            doris_note "try session Master FE."
        fi
        sleep 1
    done
}

_main() {
    docker_required_variables_env
    if [[ $RUN_TYPE == "K8S" ]]; then
        start_fe.sh
    else
        docker_setup_env
        get_doris_fe_args

        if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
            add_priority_networks $PRIORITY_NETWORKS
        fi

        docker_setup_db
        check_fe_status
        doris_note "Ready to start CURRENT_FE！"

        if [ $CURRENT_FE_IS_MASTER == true ]; then
            start_fe.sh
        else
            start_fe.sh --helper ${MASTER_FE_IP}:${MASTER_FE_EDIT_PORT}
        fi
    fi

    exec "$@"
}

if ! _is_sourced; then
    _main "$@"
fi
4.构建镜像并推送
1
2
docker build . -t docker-registry.finsiot.com/finsiot/doris-be:1.4.2.1
docker push docker-registry.finsiot.com/finsiot/doris-be:1.4.2.1
5.k8s部署fe(3节点)
部署fe说明1：部署所需传递的变量

fe部署需要传两个变量：FE_ID,FE_SERVERS

FE_ID:这个主要定义节点是否为主节点，虽然fe只要为FLLOWER都是可写可读的主节点，但FLLOWER里面也需要选举一个MASTER节点，这里FE_ID为1的默认是MASTER节点

FE_SERVERS：fe以及be以及broker所有doris服务都需要这个变量，这个变量定义了所有FE的节点信息，格式为:[FE_NAME:FE_IP:FE_PORT,FE_NAME2:FE_IP:FE_PORT]

这里由于所有服务都需要定义这个变量，所以先部署FE_SERVERS的configmap，然后在后面的所有doris服务中都要引用这个configmap中的FE_SERVERS变量。

部署fe说明2：k8s部署相关配置

1.每一个fe节点有独立的部署yaml文件，例如这里部署3个节点则会由fe1.yaml、fe2.yaml、fe3.yaml三个文件，会分别部署。每个文件中的FE_ID变量是不一样的，分别定义自己的ID号，FE_SERVERS变量因为都是一样的所有单独取自一个独立的configmap文件这里上面说过，就不再赘述。

2.不需要部署svc控制器，每个节点直接监听本机IP以及对应端口，通过以下配置实现

  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

3.需要固定部署的NODE节点，每个fe节点需要单独指定,并且由于每个节点都需要占用宿主机自身的IP及端口号来监听，所以每一个fe节点需要部署在不同的node节点上。

1
nodeName: node2

这些配置下面部署的yaml中可以看到，这里只是说明其作用。

4.由于一个服务器上面有多个ip，要确保监听到指定的ip，需要通过fe配置文件中的priority_networks配置实现

1
priority_networks = 192.168.110.0/26 #这里之所以下面部署时掩码设置为26是因为服务器里面有一个192.168.110.200的高可用IP，要确保节点IP固定，所以不能使用这种流动性IP。一般情况使用24位掩码即可

5.挂载目录

除了自身配置文件目录/opt/apache-doris/fe/conf/fe.conf以外，还需要挂载数据目录：/opt/apache-doris/fe/doris-meta

fe-servers-env.yaml
1
2
3
4
5
6
7
apiVersion: v1
kind: ConfigMap
metadata:
  name: fe-servers
data:
  TZ: "Asia/Shanghai" #这里TZ是为了定义服务时区
  FE_SERVERS: 'fe1:192.168.110.30:9010,fe2:192.168.110.31:9010,fe3:192.168.110.32:9010'
fe1.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
apiVersion: v1
kind: ConfigMap
metadata:
  name: doris-fe-1
data:
 FE_ID: '1'
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: doris-fe-1
  labels:
    app: doris-fe-1
spec:
  serviceName: doris-fe-1
  replicas: 1
  selector:
    matchLabels:
      app: doris-fe-1
  template:
    metadata:
      labels:
        app: doris-fe-1
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      nodeName: node2
      containers:
        - name: doris-fe-1
          imagePullPolicy: Always
          image: docker-registry.finsiot.com/finsiot/doris-fe:1.2.4.1 #apache/doris:1.2.2-fe-x86_64
          volumeMounts:
          - name: conf
            mountPath: /opt/apache-doris/fe/conf/fe.conf
            subPath: fe.conf
          - mountPath: /opt/apache-doris/fe/doris-meta
            name: volume-fe
          livenessProbe:
            httpGet:
              path: /api/bootstrap
              port: 8030
            initialDelaySeconds: 300
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3     
          envFrom:
            - configMapRef:
                name: doris-fe-1 
            - configMapRef:
                name: fe-servers
          ports:
            - containerPort: 8030
              protocol: TCP
            - containerPort: 9020
              protocol: TCP
            - containerPort: 9030
              protocol: TCP
            - containerPort: 9010
              protocol: TCP
          resources:
            limits:
              cpu: 2
              memory: 4G
            requests:
              cpu: 200m
              memory: 1G
      volumes:
      - name: conf
        configMap:
          name: follower-conf-1
      - name: volume-fe
        persistentVolumeClaim:
          claimName: pvc-doris-fe-1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: follower-conf-1
data:
  fe.conf: |
    metadata_failure_recovery=false
    PPROF_TMPDIR="$DORIS_HOME/log/"
    sys_log_level = INFO
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    priority_networks = 192.168.110.0/26
fe2.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
apiVersion: v1
kind: ConfigMap
metadata:
  name: doris-fe-2
data:
 FE_ID: '2'
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: doris-fe-2
  labels:
    app: doris-fe-2
spec:
  serviceName: doris-fe-2
  replicas: 1
  selector:
    matchLabels:
      app: doris-fe-2
  template:
    metadata:
      labels:
        app: doris-fe-2
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      nodeName: node3
      containers:
        - name: doris-fe-2
          image: docker-registry.finsiot.com/finsiot/doris-fe:1.2.4.1  #apache/doris:1.2.2-fe-x86_64
          volumeMounts:
          - name: conf
            mountPath: /opt/apache-doris/fe/conf/fe.conf
            subPath: fe.conf
          - mountPath: /opt/apache-doris/fe/doris-meta
            name: volume-fe
          livenessProbe:
            httpGet:
              path: /api/bootstrap
              port: 8030
            initialDelaySeconds: 300
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3
          envFrom:
            - configMapRef:
                name: doris-fe-2
            - configMapRef:
                name: fe-servers
          ports:
            - containerPort: 8030
              protocol: TCP
            - containerPort: 9020
              protocol: TCP
            - containerPort: 9030
              protocol: TCP
            - containerPort: 9010
              protocol: TCP
          resources:
            limits:
              cpu: 2
              memory: 4G
            requests:
              cpu: 200m
              memory: 1G
      volumes:
      - name: conf
        configMap:
          name: follower-conf-2
      - name: volume-fe
        persistentVolumeClaim:
          claimName: pvc-doris-fe-2
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: follower-conf-2
data:
  fe.conf: |
    metadata_failure_recovery=false
    PPROF_TMPDIR="$DORIS_HOME/log/"
    sys_log_level = INFO
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    priority_networks = 192.168.110.0/26
fe3.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
apiVersion: v1
kind: ConfigMap
metadata:
  name: doris-fe-3
data:
  FE_ID: '3'
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: doris-fe-3
  labels:
    app: doris-fe-3
spec:
  serviceName: doris-fe-3
  replicas: 1
  selector:
    matchLabels:
      app: doris-fe-3
  template:
    metadata:
      labels:
        app: doris-fe-3
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      nodeName: node4
      containers:
        - name: doris-fe-3
          image:  docker-registry.finsiot.com/finsiot/doris-fe:1.2.4.1 #apache/doris:1.2.2-fe-x86_64
          volumeMounts:
          - name: conf
            mountPath: /opt/apache-doris/fe/conf/fe.conf
            subPath: fe.conf
          - mountPath: /opt/apache-doris/fe/doris-meta
            name: volume-fe
          livenessProbe:
            httpGet:
              path: /api/bootstrap
              port: 8030
            initialDelaySeconds: 300
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3
          envFrom:
            - configMapRef:
                name: doris-fe-3
            - configMapRef:
                name: fe-servers
          ports:
            - containerPort: 8030
              protocol: TCP
            - containerPort: 9020
              protocol: TCP
            - containerPort: 9030
              protocol: TCP
            - containerPort: 9010
              protocol: TCP
          resources:
            limits:
              cpu: 2
              memory: 4G
            requests:
              cpu: 200m
              memory: 1G
      volumes:
      - name: conf
        configMap:
          name: follower-conf-3
      - name: volume-fe
        persistentVolumeClaim:
          claimName: pvc-doris-fe-3
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: follower-conf-3
data:
  fe.conf: |
  	metadata_failure_recovery=false
    PPROF_TMPDIR="$DORIS_HOME/log/"
    sys_log_level = INFO
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    priority_networks = 192.168.110.0/26
部署
1
2
3
4
kubectl apply -f fe-servers-env.yaml
kubectl apply -f fe1.yaml
kubectl apply -f fe2.yaml
kubectl apply -f fe3.yaml

部署完以后访问任何一个节点的8030端口都可以访问到fe的web管理页面，例如http://192.168.110.30:8030

默认账号为root，无密码

进入web管理页面在system菜单下面的frontends中可以看到所有FLLOWER节点信息

image-20230510150020262

Alive字段为true就表示节点正常。如果不正常可以通过看到ErrMsg和fe节点本身日志信息来排错。

be部署

首先需要构建be镜像，dockerhub也有官方的镜像，但版本一般与官网差得比较远，如果对版本要求不高可以直接使用dockerhub的官方镜像。

dockerhub地址：https://hub.docker.com/r/apache/doris/tags

因为这里目前官方的稳定版本为1.2.4.1，但官方镜像还没有能提供到该版本，所以这里选择手动构建镜像部署。

镜像构建相关文档：https://doris.apache.org/zh-CN/docs/dev/install/construct-docker/construct-docker-image

1.下载doris be的二进制包
1
2
3
4
5
这里以1.2.4.1部署为例。但下载的时候需要注意be分为CPU型号分为X64 ( avx2 )和X64 ( no avx2 )以及ARM64，这里因为服务器CPU都是X64，所以主要需要注意CPU是否支持avx2，如果不支持需要下载no avx2版本
查看CPU是否支持avx2明显:
	lscpu | grep avx2
这里因为服务器不支持avx2，所以选择no avx2版本，下载be包命令：
	wget https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.4.1-rc01/apache-doris-be-1.2.4.1-bin-x86_64-noavx2.tar.xz --no-check-certificate
2.Dockerfile文件准备
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#---
# 选择基础镜像
FROM openjdk:8u342-jdk
# 设置环境变量
ENV JAVA_HOME="/usr/local/openjdk-8/" \
    PATH="/opt/apache-doris/be/bin:$PATH"

# 下载软件至镜像内，可根据需要替换
ADD ./be /opt/be

RUN apt-get update && \
    apt-get install -y default-mysql-client && \
    apt-get clean && \
    mkdir /opt/apache-doris && \
    cd /opt && \
    mv ./be /opt/apache-doris/be

ADD ./resource/init_be.sh /opt/apache-doris/be/bin
RUN chmod 755 /opt/apache-doris/be/bin/init_be.sh

CMD ["/opt/apache-doris/be/bin/init_be.sh"]
#---
3.init_be.sh文件准备（启动初始化脚本）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
#脚本取自官方github：https://github.com/apache/doris/blob/master/docker/runtime/be/resource/init_be.sh
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail
shopt -s nullglob

DORIS_HOME="/opt/apache-doris"

# Obtain necessary and basic information to complete initialization

# logging functions
# usage: doris_[note|warn|error] $log_meg
#    ie: doris_warn "task may be risky!"
#   out: 2023-01-08T19:08:16+08:00 [Warn] [Entrypoint]: task may be risky!
doris_log() {
  local type="$1"
  shift
  # accept argument string or stdin
  local text="$*"
  if [ "$#" -eq 0 ]; then text="$(cat)"; fi
  local dt="$(date -Iseconds)"
  printf '%s [%s] [Entrypoint]: %s\n' "$dt" "$type" "$text"
}
doris_note() {
  doris_log Note "$@"
}
doris_warn() {
  doris_log Warn "$@" >&2
}
doris_error() {
  doris_log ERROR "$@" >&2
  exit 1
}

# check to see if this file is being run or sourced from another script
_is_sourced() {
  [ "${#FUNCNAME[@]}" -ge 2 ] &&
    [ "${FUNCNAME[0]}" = '_is_sourced' ] &&
    [ "${FUNCNAME[1]}" = 'source' ]
}

docker_setup_env() {
  sysctl -w vm.max_map_count=2000000
  declare -g DATABASE_ALREADY_EXISTS
  if [ -d "${DORIS_HOME}/be/storage/data" ]; then
    DATABASE_ALREADY_EXISTS='true'
  fi
}

add_priority_networks() {
  doris_note "add priority_networks ${1} to ${DORIS_HOME}/be/conf/be.conf"
  echo "priority_networks = ${1}" >>${DORIS_HOME}/be/conf/be.conf
}

show_be_args(){
  doris_note "============= init args ================"
  doris_note "MASTER_FE_IP " ${MASTER_FE_IP}
  doris_note "CURRENT_BE_IP " ${CURRENT_BE_IP}
  doris_note "CURRENT_BE_PORT " ${CURRENT_BE_PORT}
  doris_note "RUN_TYPE " ${RUN_TYPE}
  doris_note "PRIORITY_NETWORKS " ${PRIORITY_NETWORKS}
}

# Execute sql script, passed via stdin
# usage: docker_process_sql sql_script
docker_process_sql() {
  set +e
  mysql -uroot -P9030 -h${MASTER_FE_IP} --comments "$@" 2>/dev/null
}

node_role_conf(){
  if [[ ${NODE_ROLE} == 'computation' ]]; then
    doris_note "this node role is computation"
    echo "be_node_role=computation" >>${DORIS_HOME}/be/conf/be.conf
  else
    doris_note "this node role is mix"
  fi
}

register_be_to_fe() {
  set +e
  # check fe status
  local is_fe_start=false
  for i in {1..300}; do
    docker_process_sql <<<"alter system add backend '${CURRENT_BE_IP}:${CURRENT_BE_PORT}'"
    register_be_status=$?
    if [[ $register_be_status == 0 ]]; then
      doris_note "BE successfully registered to FE！"
      is_fe_start=true
      break
    else
      check_be_status
      if [ -n "$BE_ALREADY_EXISTS" ]; then
        doris_warn "Same backend already exists! No need to register again！"
        break
      fi
      if [[ $(( $i % 20 )) == 1 ]]; then
          doris_warn "register_be_status: ${register_be_status}"
          doris_warn "BE failed registered to FE!"
      fi
    fi
    if [[ $(( $i % 20 )) == 1 ]]; then
      doris_note "Register BE to FE is failed. retry."
    fi
    sleep 1
  done
  if ! [[ $is_fe_start ]]; then
    doris_error "Failed to register BE to FE！Tried 30 times！Maybe FE Start Failed！"
  fi
}

check_be_status() {
    set +e
    local is_fe_start=false
    for i in {1..300}; do
        if [[ $(($i % 20)) == 1 ]]; then
            doris_warn "start check be register status~"
        fi
        docker_process_sql <<<"show backends;" | grep "[[:space:]]${CURRENT_BE_IP}[[:space:]]" | grep "[[:space:]]${CURRENT_BE_PORT}[[:space:]]"
        be_join_status=$?
        if [[ "${be_join_status}" == 0 ]]; then
            doris_note "Verify that BE is registered to FE successfully"
            is_fe_start=true
            return
        else
            if [[ $(($i % 20)) == 1 ]]; then
                doris_note "register is failed, wait next~"
            fi
        fi
        sleep 1
    done
    if [[ ! $is_fe_start ]]; then
        doris_error "Failed to register BE to FE！Tried 30 times！Maybe FE Start Failed！"
    fi
}


_main() {
  if [[ $RUN_TYPE == "K8S" ]]; then
      start_be.sh
  else
      docker_setup_env
      if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
        add_priority_networks $PRIORITY_NETWORKS
        node_role_conf
      fi
      show_be_args
      register_be_to_fe
      check_be_status
      doris_note "Ready to start BE！"
      start_be.sh
      exec "$@"
  fi
}

if ! _is_sourced; then
  _main "$@"
fi
4.构建镜像并推送
1
2
docker build . -t docker-registry.finsiot.com/finsiot/doris-be:1.4.2.1
docker push docker-registry.finsiot.com/finsiot/doris-be:1.4.2.1
5.k8s部署be(3节点)

be也需要传递FE_SERVERS变量，通过上面fe-servers-env.yaml中定义的fe-servers configmap，除此之外，be还需要定义一个环境变量BE_ADDR，用于定义自身节点的ip和端口号。

其他部署说明和FE一样，只是be本身没有master节点，所有节点都只需要通过定义FE_SERVERS来加入到集群中即可。

be的数据挂载目录为：/opt/apache-doris/be/storage ，配置文件挂载目录为：/opt/apache-doris/be/conf/be.conf

be1.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
apiVersion: v1
kind: ConfigMap
metadata:
  name: doris-be-1
data:
 BE_ADDR: '192.168.110.30:9050'
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: doris-be-1
  labels:
    app: doris-be-1
spec:
  serviceName: doris-be-1
  replicas: 1
  selector:
    matchLabels:
      app: doris-be-1
  template:
    metadata:
      labels:
        app: doris-be-1
    spec:
      nodeName: node2
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: doris-be-1
          image: docker-registry.finsiot.com/finsiot/doris-be:1.2.4.1  #apache/doris:1.2.2-be-x86_64-noavx2  #uhub.service.ucloud.cn/dataease/doris-be-1:v1.1.3
          envFrom:
            - configMapRef:
                name: doris-be-1
            - configMapRef:
                name: fe-servers
          livenessProbe:
            httpGet:
              path: /api/health
              port: 8040
            initialDelaySeconds: 300
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3            
          ports:
            - containerPort: 9060
              protocol: TCP
            - containerPort: 9070
              protocol: TCP
            - containerPort: 8040
              protocol: TCP
            - containerPort: 9050
              protocol: TCP
            - containerPort: 8060
              protocol: TCP
          resources:
            limits:
              cpu: 2
              memory: 2G
            requests:
              cpu: 200m
              memory: 1G
          volumeMounts:
            - mountPath: /opt/apache-doris/be/storage
              name: volume-be
            - name: conf
              mountPath: /opt/apache-doris/be/conf/be.conf
              subPath: be.conf            
      volumes:
      - name: conf
        configMap:
          name: be-conf-1
      - name: volume-be
        persistentVolumeClaim:
          claimName: pvc-doris-be-1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: be-conf-1
data:
  be.conf: |
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    sys_log_level = INFO
    PPROF_TMPDIR="$DORIS_HOME/log/"
    priority_networks = 192.168.110.0/26

be2.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
apiVersion: v1
kind: ConfigMap
metadata:
  name: doris-be-2
data:
 BE_ADDR: '192.168.110.31:9050'
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: doris-be-2
  labels:
    app: doris-be-2
spec:
  serviceName: doris-be-2
  replicas: 1
  selector:
    matchLabels:
      app: doris-be-2
  template:
    metadata:
      labels:
        app: doris-be-2
    spec:
      nodeName: node3
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: doris-be-2
          image: docker-registry.finsiot.com/finsiot/doris-be:1.2.4.1 # apache/doris:1.2.2-be-x86_64-noavx2  #uhub.service.ucloud.cn/dataease/doris-be-2:v1.1.3
          envFrom:
            - configMapRef:
                name: doris-be-2
            - configMapRef:
                name: fe-servers
          livenessProbe:
            httpGet:
              path: /api/health
              port: 8040
            initialDelaySeconds: 300
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3            
          ports:
            - containerPort: 9060
              protocol: TCP
            - containerPort: 9070
              protocol: TCP
            - containerPort: 8040
              protocol: TCP
            - containerPort: 9050
              protocol: TCP
            - containerPort: 8060
              protocol: TCP
          resources:
            limits:
              cpu: 2
              memory: 2G
            requests:
              cpu: 200m
              memory: 1G
          volumeMounts:
            - mountPath: /opt/apache-doris/be/storage
              name: volume-be
            - name: conf
              mountPath: /opt/apache-doris/be/conf/be.conf
              subPath: be.conf
      volumes:
      - name: conf
        configMap:
          name: be-conf-2
      - name: volume-be
        persistentVolumeClaim:
          claimName: pvc-doris-be-2
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: be-conf-2
data:
  be.conf: |
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    sys_log_level = INFO
    PPROF_TMPDIR="$DORIS_HOME/log/"
    priority_networks = 192.168.110.0/26
be3.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
apiVersion: v1
kind: ConfigMap
metadata:
  name: doris-be-3
data:
 BE_ADDR: '192.168.110.32:9050'
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: doris-be-3
  labels:
    app: doris-be-3
spec:
  serviceName: doris-be-3
  replicas: 1
  selector:
    matchLabels:
      app: doris-be-3
  template:
    metadata:
      labels:
        app: doris-be-3
    spec:
      nodeName: node4
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: doris-be-3
          image: docker-registry.finsiot.com/finsiot/doris-be:1.2.4.1  #apache/doris:1.2.2-be-x86_64-noavx2  #uhub.service.ucloud.cn/dataease/doris-be-3:v1.1.3
          envFrom:
            - configMapRef:
                name: doris-be-3
            - configMapRef:
                name: fe-servers
          livenessProbe:
            httpGet:
              path: /api/health
              port: 8040
            initialDelaySeconds: 300
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3            
          ports:
            - containerPort: 9060
              protocol: TCP
            - containerPort: 9070
              protocol: TCP
            - containerPort: 8040
              protocol: TCP
            - containerPort: 9050
              protocol: TCP
            - containerPort: 8060
              protocol: TCP
          resources:
            limits:
              cpu: 2
              memory: 2G
            requests:
              cpu: 200m
              memory: 1G
          volumeMounts:
            - mountPath: /opt/apache-doris/be/storage
              name: volume-be
            - name: conf
              mountPath: /opt/apache-doris/be/conf/be.conf
              subPath: be.conf
      volumes:
      - name: conf
        configMap:
          name: be-conf-3
      - name: volume-be
        persistentVolumeClaim:
          claimName: pvc-doris-be-3
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: be-conf-3
data:
  be.conf: |
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    sys_log_level = INFO
    PPROF_TMPDIR="$DORIS_HOME/log/"
    priority_networks = 192.168.110.0/26
部署
1
2
3
kubectl apply -f be1.yaml
kubectl apply -f be2.yaml
kubectl apply -f be3.yaml

部署完成以后在fe管理页面的system菜单下面的backends中可以看到所有be节点

image-20230510153551945

broker部署

Broker 是用于访问外部数据源（如 hdfs）的进程

broker无数据目录，配置文件使用默认即可，所以这里无需挂载目录

1.下载doris broker二进制包
1
wget https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.4.1-rc01/apache-doris-dependencies-1.2.4.1-bin-x86_64.tar.xz --no-check-certificate
2.Dockerfile文件准备
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 选择基础镜像
FROM openjdk:8u342-jdk

# 设置环境变量
ENV JAVA_HOME="/usr/local/openjdk-8/" \
    PATH="/opt/apache-doris/broker/bin:$PATH"

RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list && \
    sed -i 's/security.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list
# 下载软件至镜像内，此处 broker 目录被同步压缩至 FE 的二进制包，需要自行解压重新打包，可根据需要替换
ADD ./apache-doris-dependencies-1.2.4.1-bin-x86_64.tar.xz /opt/broker

RUN apt-get update && \
    apt-get install -y default-mysql-client && \
    apt-get clean && \
    mkdir /opt/apache-doris && \
    cd /opt && \
    mv ./apache-doris-dependencies-1.2.4.1-bin-x86_64/apache_hdfs_broker/ /opt/apache-doris/broker

ADD ./init_broker.sh /opt/apache-doris/broker/bin
RUN chmod 755 /opt/apache-doris/broker/bin/init_broker.sh

ENTRYPOINT ["/opt/apache-doris/broker/bin/init_broker.sh"]

3.init_broker.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail
shopt -s nullglob

DORIS_HOME="/opt/apache-doris"

# Obtain necessary and basic information to complete initialization

# logging functions
# usage: doris_[note|warn|error] $log_meg
#    ie: doris_warn "task may BROKER risky!"
#   out: 2023-01-08T19:08:16+08:00 [Warn] [Entrypoint]: task may BROKER risky!
doris_log() {
  local type="$1"
  shift
  # accept argument string or stdin
  local text="$*"
  if [ "$#" -eq 0 ]; then text="$(cat)"; fi
  local dt="$(date -Iseconds)"
  printf '%s [%s] [Entrypoint]: %s\n' "$dt" "$type" "$text"
}
doris_note() {
  doris_log Note "$@"
}
doris_warn() {
  doris_log Warn "$@" >&2
}
doris_error() {
  doris_log ERROR "$@" >&2
  exit 1
}

# check to see if this file is BROKERing run or sourced from another script
_is_sourced() {
  [ "${#FUNCNAME[@]}" -ge 2 ] &&
    [ "${FUNCNAME[0]}" = '_is_sourced' ] &&
    [ "${FUNCNAME[1]}" = 'source' ]
}

# Check the variables required for startup
docker_required_variables_env() {
  if [[ $FE_SERVERS =~ ^.+:[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}:[1-6]{0,1}[0-9]{1,4}(,.+:[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}:[1-6]{0,1}[0-9]{1,4})*$ ]]; then
    doris_warn "FE_SERVERS" $FE_SERVERS
  else
    doris_error "FE_SERVERS rule error！example: \$FE_NAME:\$FE_HOST_IP:\$FE_EDIT_LOG_PORT[,\$FE_NAME:\$FE_HOST_IP:\$FE_EDIT_LOG_PORT]..."
  fi
  if [[ $BROKER_ADDR =~ ^[a-zA-Z0-9]+:[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}:[1-6]{0,1}[0-9]{1,4}$ ]]; then
    doris_warn "BROKER_ADDR" $BROKER_ADDR
  else
    doris_error "BROKER_ADDR rule error！example: \$BROKER_NAME:\$BROKER_HOST_IP:\$BROKER_IPC_PORT"
  fi
}

get_doris_broker_args() {
  local feServerArray=($(echo "${FE_SERVERS}" | awk '{gsub (/,/," "); print $0}'))
  for i in "${feServerArray[@]}"; do
    val=${i}
    val=${val// /}
    tmpFeId=$(echo "${val}" | awk -F ':' '{ sub(/fe/, ""); sub(/ /, ""); print$1}')
    tmpFeIp=$(echo "${val}" | awk -F ':' '{ sub(/ /, ""); print$2}')
    tmpFeEditLogPort=$(echo "${val}" | awk -F ':' '{ sub(/ /, ""); print$3}')
    check_arg "tmpFeIp" $tmpFeIp
    feIpArray[$tmpFeId]=${tmpFeIp}
    check_arg "tmpFeEditLogPort" $tmpFeEditLogPort
    feEditLogPortArray[$tmpFeId]=${tmpFeEditLogPort}
  done

  declare -g MASTER_FE_IP BROKER_HOST_IP BROKER_IPC_PORT BROKER_NAME
  MASTER_FE_IP=${feIpArray[1]}
  check_arg "MASTER_FE_IP" $MASTER_FE_IP
  BROKER_NAME=$(echo "${BROKER_ADDR}" | awk -F ':' '{ sub(/ /, ""); print$1}')
  check_arg "BROKER_NAME" $BROKER_NAME
  BROKER_HOST_IP=$(echo "${BROKER_ADDR}" | awk -F ':' '{ sub(/ /, ""); print$2}')
  check_arg "BROKER_HOST_IP" $BROKER_HOST_IP
  BROKER_IPC_PORT=$(echo "${BROKER_ADDR}" | awk -F ':' '{ sub(/ /, ""); print$3}')
  check_arg "BROKER_IPC_PORT" $BROKER_IPC_PORT

  doris_note "feIpArray = ${feIpArray[*]}"
  doris_note "feEditLogPortArray = ${feEditLogPortArray[*]}"
  doris_note "masterFe = ${feIpArray[1]}:${feEditLogPortArray[1]}"
  doris_note "brokerAddr = ${BROKER_NAME}:${BROKER_HOST_IP}:${BROKER_IPC_PORT}"
  # wait fe start
  check_broker_status true
}

# Execute sql script, passed via stdin
# usage: docker_process_sql sql_script
docker_process_sql() {
  set +e
  mysql -uroot -P9030 -h${MASTER_FE_IP} --comments "$@" 2>/dev/null
}

# register broker
register_broker_to_fe() {
  set +e
  # check fe status
  local is_fe_start=false
  for i in {1..300}; do
    if [[ $(( $i % 20 )) == 1 ]]; then
      doris_note "Register BROKER to FE is failed. retry."
    fi
    docker_process_sql <<<"alter system add broker ${BROKER_NAME} '${BROKER_HOST_IP}:${BROKER_IPC_PORT}'"
    register_broker_status=$?
    if [[ $register_broker_status == 0 ]]; then
      doris_note "BROKER successfully registered to FE！"
      is_fe_start=true
      break
    else
      check_broker_status
      if [ -n "$BROKER_ALREADY_EXISTS" ]; then
        doris_warn "Same backend already exists! No need to register again！"
        break
      fi
      if [[ $(( $i % 20 )) == 1 ]]; then
          doris_warn "BROKER failed registered to FE!"
      fi
    fi
    sleep 1
  done
  if ! [[ $is_fe_start ]]; then
    doris_error "Failed to register BROKER to FE！Tried 30 times！MayBe FE Start Failed！"
  fi
}

# Check whether the passed parameters are empty to avoid subsequent task execution failures. At the same time,
# enumeration checks can BROKER added, such as checking whether a certain parameter appears repeatedly, etc.
check_arg() {
  if [ -z $2 ]; then
    doris_error "$1 is null!"
  fi
}

check_broker_status() {
  set +e
  for i in {1..300}; do
    if [[ $1 == true ]]; then
      docker_process_sql <<<"show frontends" | grep "[[:space:]]${MASTER_FE_IP}[[:space:]]"
    else
      docker_process_sql <<<"show proc '/brokers'" | grep "[[:space:]]${BROKER_HOST_IP}[[:space:]]" | grep "[[:space:]]${BROKER_IPC_PORT}[[:space:]]"
    fi
    broker_join_status=$?
    doris_warn "broker_join_status: " $broker_join_status
    if [[ "${broker_join_status}" == 0 ]]; then
      if [[ $1 == true ]]; then
        doris_note "MASTER FE is started!"
      else
        doris_note "Init Check - Verify that BROKER is registered to FE successfully"
        BROKER_ALREADY_EXISTS=true
      fi
      break
    fi
    if [[ $(( $i % 20 )) == 1 ]]; then
      if [[ $1 == true ]]; then
        doris_note "MASTER FE is not started. retry."
      else
        doris_note "BROKER is not register. retry."
      fi
    fi
    sleep 1
  done
}

_main() {
  docker_required_variables_env
  get_doris_broker_args
  register_broker_to_fe
  check_broker_status
  doris_note "Ready to start BROKER！"
  start_broker.sh
  exec "$@"
}

if ! _is_sourced; then
  _main "$@"
fi
4.构建镜像并推送
1
2
docker build . -t docker-registry.finsiot.com/finsiot/doris-broker:1.4.2.1
docker push docker-registry.finsiot.com/finsiot/doris-broker:1.4.2.1
5.k8s部署broker

broker和be几乎一样，只需要指定自身的ip端口，再指定fe集群的ip端口就行了

环境变量为：BROKER_ADDR，FE_SERVERS

BROKER_ADD格式为：BROKER_NAME:IP:PORT

broker.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: v1
kind: ConfigMap
metadata:
  name: doris-broker-1
data:
 BROKER_ADDR: 'broker1:192.168.110.30:8000'
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: doris-broker-1
  labels:
    app: doris-broker-1
spec:
  serviceName: doris-broker-1
  replicas: 1
  selector:
    matchLabels:
      app: doris-broker-1
  template:
    metadata:
      labels:
        app: doris-broker-1
    spec:
      nodeName: node2
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: doris-broker-1
          image: docker-registry.finsiot.com/finsiot/doris-broker:1.2.4.1  #apache/doris:1.2.2-be-x86_64-noavx2  #uhub.service.ucloud.cn/dataease/doris-broker-1:v1.1.3
          envFrom:
            - configMapRef:
                name: doris-broker-1
            - configMapRef:
                name: fe-servers
          ports:
            - containerPort: 8000
              protocol: TCP
          resources:
            limits:
              cpu: 2
              memory: 2G
            requests:
              cpu: 200m
              memory: 1G

部署完在fe管理页面system菜单中的brokers可以看到。

如果要部署多个broker只需和be一样部署即可，构建多个yaml文件多个不同名称不同节点的broker，指定环境变量 BROKER_ADDR，再部署启动即可。

doris问题及处理方法

FE Follower全部挂掉恢复方法

当fe Follower节点全部同时挂掉或者Follower仅存在一个节点达到一定时间的时候都会出现集群故障。此时程序无法启动，日志显示为

wait catalog to be ready. FE type UNKNOWN

恢复方法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
1、首先，停止所有 FE 进程，同时停止一切业务访问。保证在元数据恢复期间，不会因为外部访问导致其他不可预期的问题。

2、确认哪个 FE 节点的元数据是最新：
    首先，务必先备份所有 FE 的 meta_dir 目录。
    通常情况下，Master FE 的元数据是最新的。可以查看 meta_dir/image 目录下，image.xxxx 文件的后缀，数字越大，则表示元数据越新。
    通常，通过比较所有 FOLLOWER FE 的 image 文件，找出最新的元数据即可。
    之后，我们要使用这个拥有最新元数据的 FE 节点，进行恢复。
    
3、以下操作都在由第2步中选择出来的 FE 节点上进行。
    在 fe.conf 中添加配置：metadata_failure_recovery=true。并设置FE_ID以及FE_SERVERS的ID号，将该节点ID设置为1。
    例如这里ip为192.168.110.30的节点2环境变量为： FE_ID: '2'，FE_SERVERS: 'fe1:192.168.110.30:9010,fe2:192.168.110.31:9010,fe3:192.168.110.32:9010'
    就应该修改为FE_ID： '1',FE_SERVERS:'fe1:192.168.110.31:9010,fe2:192.168.110.30:9010,fe3:192.168.110.32:9010'
    这里其他节点的id不重要，保证本节点ID为1就行。然后启动这个节点
    ---
    备注：`metadata_failure_recovery=true` 的含义是，清空 "bdbje" 的元数据。这样 bdbje 就不会再联系之前的其他 FE 了，而作为一个独立的 FE 启动。这个参数只有在恢复启动时才需要设置为 true。恢复完成后，一定要设置为 false，否则一旦重启，bdbje 的元数据又会被清空，导致其他 FE 无法正常工作。
    ---
    
    如果正常，这个 FE 会以 MASTER 的角色启动，类似于前面 启动单节点 FE 一节中的描述。在 fe.log 应该会看到 transfer from XXXX to MASTER 等字样。
    通过 ALTER SYSTEM DROP FOLLOWER/OBSERVER 命令，将之前的其他的 FE 从元数据删除后，按加入新 FE 的方式，重新把这些 FE 添加一遍。
        命令示例： ALTER SYSTEM DROP FOLLOWER '192.168.110.30:9010';
    将 fe.conf 中的 metadata_failure_recovery=true 配置项删除，或者设置为 false，然后重启这个 FE（重要）。
    然后启动其他fe FLLOWER节点（因为配置文件中配置了FE_ID和FE_SERVERS，所以会自动加入集群，这里相当于是找到了有最新元数据的fe节点，删除集群并重新加入同步数据，完成了FE的集群修复工作。）