Best practice to use AWS Secrets Manager with EKS and rotate regularly

Cluster information:

Kubernetes version: 1.2x
Cloud being used: AWS
Installation method: AWS Console
Host OS: Linux

We are currently considering externalizing secrets for Kubernetes (EKS) by utilizing AWS Secrets Manager. Our proposed approach involves using the K8s Secrets Store CSI Driver and ASCP to retrieve secrets from Secrets Manager and then pass them as environment variables to the container.

Additionally, we have identified other potential options:

  • Utilizing the Secrets CSI driver and mounting secrets - However, we are unsure about the correct method for utilizing the mounted secrets and how to integrate them effectively.
  • Making direct API calls from the Java application.

Are there any alternative approaches we should consider?

Furthermore, we have a requirement to automatically rotate these secrets at regular intervals, such as every 7 days. It is crucial that the rotation remains synchronized with the secrets used by the application. We are seeking an optimal approach for achieving this synchronization with zero downtime. The secret auto rotation feature provided by the CSI driver relies on polling, but it’s important to note that AWS charges for each API call. Hence, we aim to minimize API calls to reduce costs while ensuring zero downtime. Setting the polling interval too low would increase costs, while setting it too high may result in downtime if rotation fails to occur within the expected time frame. Are there alternative methods to address this concern? If not, how can we appropriately determine the optimal polling interval?