data-engineering-storage-authentication

Secure authentication patterns for accessing cloud storage (S3, GCS, Azure Blob) and cloud services in data pipelines. Covers IAM roles, service principals, secret managers, and best practices for credential management.

| AWS | IAM roles (EC2/ECS/Lambda) | Environment variables, Secrets Manager | | GCP | Workload Identity / ADC | Service account keys (discouraged) | | Azure | Managed Identity | Service principal with certificate | | Local Dev | .env files + local credentials | Static keys (temporary only) |

❌ Hardcoding credentials - Committing to git → rotate immediately ❌ Using root/admin accounts - Create scoped users/service principals ❌ Long-lived keys - Rotate every 90 days or less ❌ Over-permissive roles - Grant s3:GetObject not s3: ❌ Missing environment separation - Dev credentials in prod