In Ansible, why is the PutMetricData operation failing with 'The security token provided in the request is invalid (InvalidClientTokenId)

Question

A master-worker architecture was created using Ansible, inside Master. The below code creates workers.

- name: Provisioning Spot instaces ec2: assign_public_ip: no spot_price: "{{ ondemand4_price }}" spot_wait_timeout: 300 assign_public_ip: no aws_access_key: "{{ assumed_role.sts_creds.access_key }}" aws_secret_key: "{{ assumed_role.sts_creds.secret_key }}" security_token: "{{ assumed_role.sts_creds.session_token }}" region: "{{ aws_region }}" image: "{{ image_instance }}" instance_type: "{{ large_instance }}" key_name: "{{ ssh_keyname }}" count: "{{ ninstances }}" state: present group_id: "{{ priv_sg }}" vpc_subnet_id: "{{ subnet_id }}" instance_profile_name: 'ML-Ansible' wait: true instance_tags: Name: Worker #delete_on_termination: yes register: ec2 ignore_errors: True

Worker instances were also created as ‘ML-Ansible’ with permissions. When I tried to execute the following AWS shell command aws cloudwatch put-metric-data … , it ends up with the following error

"stderr": "\nAn error occurred (InvalidClientTokenId) when calling the PutMetricData operation: The security token included in the request is invalid.",

Now, we have a fresh set of aws_access_key_id and aws_secret_access_key

My ~/.aws/credentials file contains the past credentials when I run an Ansible file. Should I have to edit with respect to the IAM profile also?

Answer 1

Try re-creating AMI after removing the credentials, if you were not using the AMI provided by AWS. As the credentials in  ~/.aws/credentials created might have got included before AMI was created and get baked into the AMI that are used to launch your workers.