Amazon AWS elasticsearch Kibana access from browser

Question

I know this issue has been already discussed before, Yet I feel my question is a bit different.

I'm trying to figure out how am I to enable access to the Kibana over the self-managed AWS elastic search which I have in my AWS account.

Could be that what am I about to say here is inaccurate or complete nonsense. I am pretty novice in the whole AWS VPC wise section and to ELK stuck.

Architecture: Here is the "Architecture":

I have a VPC.

Within the VPC I have several subnets.

Each server sends it's data to the elastic search using log stash which runs on the server itself. For simplicity, let's assume I have a single server.

The elastic search https URL which can be found in the Amazon console is resolved to an internal IP within the subnet that I have defined.

Resources:

I have found the following link which suggests using one of two option:

https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/

Solutions:

Option 1: resource-based policy

Either to allow resource-based policy for elastic search by introducing condition which specifies a certain IP address.

This was discussed in the following thread but unfortunately did not work for me.

The proper access policy for Amazon Elastic Search Cluster

When I try to implement it in the Amazon console, Amazon notifies me that because I'm using Security group, I should resolve it by using security group.

Security group rules:

I tried to set a rule which allows my personal computer(Router) public IP to access Amazon elastic search ports or even opening all ports to my public IP.

But that didn't work out. I would be happy to get a more detailed explanation to why but I'm guessing that's because the elastic search has only internal IP and not public IP and because it is encapsulated within the VPC I am unable to access it from outside even if I define a rule for a public IP to access it.

Option 2: Using proxy

I decline to use this solution unless I have no other choice.

I'm guessing that if I set another server with public and internal IP within the same subnet and VPC as that of the elastic search, and use it as a proxy, I would then be able to access this server from the outside by defining the same rules to them it's newly created security group. Like the article suggested.

Sources:

I found out of the box solution that someone already made for this issue using a proxy server in the following link:

Using either executable or docker container.

https://github.com/abutaha/aws-es-proxy

Option 3: Other

Can you suggest another solution? Is it possible to use Amazon Load balancer or Amazon API gateway to accomplish this task?

I just need proof of concept, not something which goes into the production environment.

Bottom line:

I need to be able to access Kibana from the browser in order to be able to search elastic search indexes.

Answer 1

You can use Cognito authentication for this.

https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-elasticsearch-service-simplifies-user-authentication-and-access-for-kibana-with-amazon-cognito/