quobyte-csi-driver

Multi-cluster setup

Quobyte access keys are highly recommended for multi-cluster deployments

Quobyte CSI Driver supports access to multiple Quobyte storage clusters (for example, production and testing) from single k8s cluster.

To use one Quobyte cluster, you always need one Quobyte CSI driver i.e. one-to-one mapping between Quobyte CSI driver and Quobyte cluster. To use n Quobyte clusters, you need n Quobyte CSI drivers.

For each driver the following must be met

  1. Unique quobyte.apiURL in driver configuration

  2. Unique quobyte.csiProvisionerName in driver configuration

  3. Unique quobyte.clientMountPoint in driver configuration

  4. k8s nodes with multiple Quobyte clients, each with their own mountPoint and registry configuration

Adjust the driver configuration and deploy driver with helm install <SOME_UNIQUE_NAME> ./quobyte-csi-driver

The value configured for quobyte.csiProvisionerName must be used as StorageClass.provisioner to refer this Quobyte Cluster/CSI driver combination

Installation of multiple native clients

Limitations

  1. Two native clients with different version cannot be installed on same machine
  2. install_quobyte add-client can only install single client on a machine, additional clients must be installed/cloned manually

Installation of clients (for systemd services)

  1. Install Quobyte client following client installation instructions

  2. Clone copies of service and configuration on each k8s node

     clusters="test1 test2" # cluster names separated by single space
 for cluster in $clusters;
 do
   sudo cp /usr/lib/systemd/system/quobyte-client.service /usr/lib/systemd/system/quobyte-client-$cluster.service
   sudo sed -i -e "s/EnvironmentFile=\/etc\/quobyte\/client-service.env/EnvironmentFile=\/etc\/quobyte\/client-service-${cluster}.env/g" /usr/lib/systemd/system/quobyte-client-$cluster.service
   sudo systemctl enable /usr/lib/systemd/system/quobyte-client-$cluster.service
   sudo cp /etc/quobyte/client-service.env /etc/quobyte/client-service-$cluster.env
   sudo sed -i -e "s/config_file=\/etc\/quobyte\/client-service.cfg/config_file=\/etc\/quobyte\/client-service-$cluster.cfg/g" /etc/quobyte/client-service-$cluster.env
   sudo cp /etc/quobyte/client-service.cfg /etc/quobyte/client-service-$cluster.cfg
 done

  3. Edit /etc/quobyte/client-service<-YOUR_CLUSTER_NAME>.cfg and configure registry, client mount points (as needed by CSI driver).

  4. Start clients

     clusters="test1 test2" # cluster names separated by single space
 for cluster in $clusters;
 do
   sudo systemctl start quobyte-client-$cluster.service
   sudo systemctl status quobyte-client-$cluster.service
 done

  5. To remove clients, please use following commands

     remove_cluster_client="test1 test2"  # cluster names separated by single space
 for cluster in $remove_cluster_client;
 do
   sudo systemctl stop quobyte-client-$cluster.service
   sudo rm -f /usr/lib/systemd/system/quobyte-client-$cluster.service
   sudo rm -f /etc/systemd/system/multi-user.target.wants/quobyte-client-$cluster.service
   sudo rm -f /etc/systemd/system/remote-fs.target.wants/quobyte-client-$cluster.service
   sudo rm -f /etc/quobyte/client-service-$cluster.env
   sudo rm -f /etc/quobyte/client-service-$cluster.cfg
 done