Clickhouse [un]installing

With ssl & digest support

Notes

• The only playbok at the moment is clickhouse.yml
• Zookeeper hosts are in the [zookeepers] group, Clickhouse hosts are in the [clickhouse] one
• Clickhouse clusters are described in the ch_clusters variable.
• The <ch_clusters.name>_shard macro is created for each Clickhouse host equal to a shard number of this host in the <ch_clusters.name> cluster.
• The installation packages (in the distrib subdirectory) & certs (in the certs subdirectory) or links to them must be placed to the roles/clickhouse/files/ directory. For example:

$ tree roles/clickhouse/files
roles/clickhouse/files
├── certs
│   ├── fqdn-X.crt
│   ├── fqdn-X.key
│   ├── ...
│   └── truststore.pem
└── distrib -> /distrib/ch/deb/24.3.5

• If ch_with_ssl: "yes", then fqdn-X.{crt,key} must exist for all hostnames from the inv file
• ch_use_keeper: "yes/no" - to use ClickHouse Keeper or Zookeeper as a coordinator. If "yes", then every host from the [zookeepers] group in inv must be included into the [clickhouse] group. If "no", then Zookeeper on all hosts of the [zookeepers] group in inv must be configured and started up separetedly.
• ch_default_cluster: "s1r3" - The default ClickHouse cluster name. It's used for convenience. The {cluster} macro gets this value, and the same value is used in the default_replica_path ClickHouse parameter. It's usually set to the name of one of the ClickHouse clusters, where replicated tables are created mostly to not specify it in the CREATE TABLE statement for ReplicatedMergeTree tables. For example:

CREATE TABLE ... ON CLUSTER '{cluster}' ... ENGINE = ReplicatedMergeTree()

• ch_clusters: - The ClickHouse clusters definition. One of the names must be equal to the value specified in ch_default_cluster. The hostnames of each cluster are specified by their indexes in the [clickhouse] group of the inv file. An example for 2 clusters:
  • s3r1: 3 shards, 1 replica
  • s1r3: 1 shard, 3 replica (the same name is specified in ch_default_cluster)

ch_clusters:
  - name: s3r1
    shards:
      - replicas:
        - "{{ groups[ch_group_name][0] }}"
      - replicas:
        - "{{ groups[ch_group_name][1] }}"
      - replicas:
        - "{{ groups[ch_group_name][2] }}"
  - name: "{{ ch_default_cluster }}"
    shards:
      - replicas:
        - "{{ groups[ch_group_name][0] }}"
        - "{{ groups[ch_group_name][1] }}"
        - "{{ groups[ch_group_name][2] }}"

Commands examples

• Installing

ansible-playbook clickhouse.yml -e "ch_config_only=no"

• Reconfig with restart

ansible-playbook clickhouse.yml

• Reconfig without restart

ansible-playbook clickhouse.yml -e "ch_reconfig_restart=no"

• Stopping

ansible-playbook clickhouse.yml -e 'play_action=stop'

• Starting

ansible-playbook clickhouse.yml -e 'play_action=start'

Useful commands

• Configs on the 1-st ZK node of the inventory

ansible -m shell -a "ls -l /opt/zookeeper/cert" $(grep -A1 -F [zookeepers inv | tail -1)
ansible -m shell -a "cat /opt/zookeeper/conf/zoo.cfg" $(grep -A1 -F [zookeepers inv | tail -1)
ansible -m shell -a "cat /opt/zookeeper/conf/zookeeper-env.sh" $(grep -A1 -F [zookeepers inv | tail -1)

• Zookeeper processes

ansible -m shell -a "ps -u zookeeper -o args= | grep 'zoo\.cfg$'" all
ansible -m shell -a "ps -fwwp \$(cat /home/zookeeper/data/zookeeper_server.pid)" all

• Zookeeper Log files
  On the 3-rd ZK node of the inventory

ansible -m shell -a "sh -c 'tail -100 \$(ls -tr /opt/zookeeper/logs/zookeeper-zookeeper-server-*.out | tail -1)'"  $(grep -A3 -F [zookeepers inv | tail -1)

• Clickhouse Log files

ansible -m shell -a "tail -100 /var/log/clickhouse-server/clickhouse-server.log" $(grep -A1 -F [clickhouse inv | tail -1)

• Zookeeper Connection & status

cert=/distrib/cert/certs/hdfs-01-03/zk/hdfs-03.crt
key=/distrib/cert/certs/hdfs-01-03/zk/hdfs-03.key
cafile=roles/zookeeper/files/cert/truststore.pem
h=$(grep -A1 -F [clickhouse inv | tail -1)

openssl s_client -connect ${h?}:2281 \
-cert ${cert?} \
-key ${key?} \
-CAfile ${cafile?} \
-state -quiet

echo "stat" | nc --ssl --ssl-cert ${cert?} --ssl-key ${key?} ${h?} 2281

• Clickhouse client Config file

cat <<EOF | tee ./clickhouse-client.xml
<config>
    <secure>true</secure>
    <openSSL>
      <client>
        <caConfig>${cafile?}</caConfig>
      </client>
    </openSSL>
</config>
EOF

clickhouse-client -h $(grep -A1 -F [clickhouse inv | tail -1) --multiquery <<EOF
SYSTEM RELOAD CONFIG ON CLUSTER s2r2
EOF

Uninstalling

TBD