发表于 | 分类于 |

Spark 当前支持使用共享密钥来认证。可以通过配置 spark.authenticate 选项来开启认证。这个参数用来控制 Spark 的通讯协议是否使用共享密钥来执行认证。这个认证是一个基本的握手，用来保证两侧具有相同的共享密钥然后允许通讯。如果共享密钥无法识别，他们将不能通讯。创建共享密钥的方法如下 :

  • 对于 Spark on YARN 的部署方式，配置 spark.authenticate 为 true，会自动生成和分发共享密钥。每个 application 会使用一个单独的共享密钥
  • 对于 Spark 的其他部署方式，需要在每一个 nodes 配置 spark.authenticate.secret 参数。这个密码将会被所有的 Master/Workers 和 aplication 使用
阅读全文 »

发表于 | 分类于 |

在 Kubernetes 上部署 Harbor

To deploy Harbor on Kubernetes, it requires some additional steps because

  1. When Harbor registry uses https, so we need cert or workaround to avoid errors like this:

    1
    2
    3
    Error response from daemon: invalid registry endpoint https://{HOST}/v0/: unable to ping registry endpoint https://{HOST}/v0/
    v2 ping attempt failed with error: Get https://{HOST}/v2/: EOF
    v1 ping attempt failed with error: Get https://{HOST}/v1/_ping: EOF. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry {HOST}` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/{HOST}/ca.crt

    There is a workaround if you don’t have a cert. The workaround is to add the host into the list of insecure registry by editting the

    file:
    1
    2
    ```bash
    sudo vi /etc/default/docker

    add the line at the end of file:

    1
    DOCKER_OPTS="$DOCKER_OPTS --insecure-registry={HOST}"

    restart docker service

    1
    sudo service docker restart
    阅读全文 »