When Cloud services are used in an application, it might be tricky to mock them during local development. Some approaches include: 1) doing nothing thus letting your application fail when it makes a call to a Cloud service; 2) creating sets of fake data to return from calls to AWS S3, for example; 3) using an account in the Cloud for development purposes. A nice in-between solution is using Localstack, a Cloud service emulator. Whereas the number of services available and the functionality might be a bit limited compared to the real AWS environment, it works rather well for our team.<\/p>\n\n\n\n
This article will describe how to set it up for local development in Docker. <\/p>\n\n\n\n
Docker-compose setup:<\/strong><\/p>\n\n\n\n In the services section of our docker-compose.yml<\/em> we have Localstack container definition: <\/p>\n\n\n\n Although we don’t need to connect to any AWS account, we do need dummy AWS variables (with any value). We specify which services we want to run using Localstack – in this case it’s SQS and S3.<\/p>\n\n\n\n We also need to set HOSTNAME_EXTERNAL because SQS API needs the container to be aware of the hostname that it can be accessed on.<\/p>\n\n\n\n Another point is that that we cannot use the entrypoint definition because Localstack has a directory docker-entrypoint-initaws.d<\/em> from where shell scripts are run when the container starts up. That’s why we’re mapping the container volume to a folder wherer those scripts are. In our case create_localstack_resources.sh<\/em> will create all the necessary S3 buckets and the SQS queue:<\/p>\n\n\n\n Note that AWS CLI command syntax is different to the real AWS CLI (otherwise you’d create resources in the account for which you have the credentials set up!), and includes Localstack endoint flag: –endpoint-url=http:\/\/localhost:4566<\/em><\/p>\n\n\n\n <\/p>\n\n\n\n Configuration files<\/strong><\/p>\n\n\n\n We use Scala with Play framework for this particular application, and therefore have .conf<\/em> files. In local.conf<\/em> file we have the following:<\/p>\n\n\n\n aws {\n localstack.endpoint=\"http:\/\/localstack:4566\"\n region = \"eu-central-1\"\n s3.bucket1 = \"bucket1\"\n s3.bucket2 = \"bucket2\"\n sqs.my_queue = \"my-queue\"\n sqs.queue_enabled = true\n}<\/p><\/pre>\n\n\n\n The real application.conf<\/em> file has resource names injected at the instance startup. They live in an autoscaling group launch template where they are created by Terraform (out of scope of this post).<\/p>\n\n\n\n <\/p>\n\n\n\n Initializing SQS client based on the environment<\/strong><\/p>\n\n\n\n The example here is for creating an SQS client. Below are snippets most relevant to the topic.<\/p>\n\n\n\n In order to initialize the SQS Service so that it can be injected into other services we can do this:<\/p>\n\n\n\n lazy val awsSqsService: QueueService = createsSqsServiceFromConfig()<\/p><\/pre>\n\n\n\n In createsSqsServiceFromConfig<\/span><\/em> we check if the configuration has a Localstack endpoint and if so, we build LocalStack client:<\/p>\n\n\n\n protected def createsSqsServiceFromConfig(): QueueService = {\n readSqsClientConfig().map { config =>\n val sqsClient: SqsClient = config.localstackEndpoint match {\n case Some(endpoint) => new LocalStackSqsClient(endpoint, config.region)\n case None => new AwsSqsClient(config.region)\n }\n new SqsQueueService(config.queueName, sqsClient)\n }.getOrElse(fakeAwsSqsService)\n}<\/p><\/pre>\n\n\n\n readSqsClientConfig<\/em> is used to get configuration values from .conf<\/em> files:<\/p>\n\n\n\n private def readSqsClientConfig = { Finally LocalStackSqsClient initialization looks like this:<\/p>\n\n\n\n class LocalStackSqsClient(endpoint: String, region:String) extends SqsClient with Logging { \noverride def BuildClient(): AmazonSQS = {\n log.debug(\"Initializing LocalStack SQS service\")\n client\n }\n}<\/p><\/pre>\n\n\n\n Real AWS Client for the test\/live environment (a snippet):<\/p>\n\n\n\n <\/p>\n\n\n\n Notice that we need fake BasicAWSCredentials<\/em> that allows us to pass in dummy AWS access key and secret key <\/em>and then we use AWSStaticCredentialsProvider<\/em>, an implementation of AWSCredentialsProvider<\/em> that just wraps static AWSCredentials<\/em>. When real AWS environment is used, instead of AWSStaticCredentialsProvider<\/em> we use DefaultAWSCredentialsProviderChain<\/em>, which picks the EC2 Instance Role if it’s unable to find credentials by any other methods.<\/p>\n\n\n\n <\/p>\n\n\n\n And that’s it. Happy coding!<\/p>\n\n\n","protected":false},"excerpt":{"rendered":" When Cloud services are used in an application, it might be tricky to mock them during local development. Some approaches include: 1) doing nothing thus letting your application fail when it makes a call to a Cloud service; 2) creating sets of fake data to return from calls to AWS S3, for example; 3) using … Continue reading “Setting up local AWS environment using Localstack”<\/span><\/a><\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-489","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=\/wp\/v2\/posts\/489","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=489"}],"version-history":[{"count":23,"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=\/wp\/v2\/posts\/489\/revisions"}],"predecessor-version":[{"id":641,"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=\/wp\/v2\/posts\/489\/revisions\/641"}],"wp:attachment":[{"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=489"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=489"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.67bricks.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=489"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}localstack:\n image: localstack\/localstack:latest\n hostname: localstack\n environment:\n - SERVICES=s3,sqs\n - HOSTNAME_EXTERNAL=localstack\n - DATA_DIR=\/tmp\/localstack\/data\n - DEBUG=1\n - AWS_ACCESS_KEY_ID=test\n - AWS_SECRET_ACCESS_KEY=test\n - AWS_DEFAULT_REGION=eu-central-1\n ports:\n - \"4566:4566\"\n volumes:\n - localstack-data:\/tmp\/localstack:rw\n - .\/create_localstack_resources.sh:\/docker-entrypoint-initaws.d\/create_localstack_resources.sh\n<\/pre>\n\n\n\n
EXPECTED_BUCKETS=(\"bucket1\" \"bucket2\" \"bucket3\")\nEXISTING_BUCKETS=$(aws --endpoint-url=http:\/\/localhost:4566 s3 ls --output text)\n\necho \"creating buckets\"\nfor BUCKET in \"${EXPECTED_BUCKETS[@]}\"\ndo\n echo $BUCKET\n if [[ $EXISTING_BUCKETS != *\"$BUCKET\"* ]]; then\n aws --endpoint-url=http:\/\/localhost:4566 s3 mb s3:\/\/$BUCKET\n fi\ndone\n\necho \"creating queue\"\nif [[ $EXISTING_QUEUE != *\"$EXPECTED_QUEUE\"* ]]; then\n aws --endpoint-url=http:\/\/localhost:4566 sqs create-queue --queue-name my-queue\\\n --attributes '{\n \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-central-1:000000000000:my-dead-letter-queue\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\",\n \"VisibilityTimeout\": \"120\"\n }'\nfi\n<\/pre>\n\n\n\n
val sqsName = config.get[String](\"aws.sqs.my_queue\")
val sqsRegion = config.get[String](\"aws.region\") <\/a>
val localStackEndpoint = config.getOptional[String](\"aws.localstack.endpoint\")<\/a>
SqsClientConfig(sqsName, sqsRegion, localStackEndpoint)
}<\/p><\/pre>\n\n\n\n
private val sqsEndpoint = new EndpointConfiguration(endpoint, region)
private val awsCreds = new BasicAWSCredentials(\"test\", \"test\")
private lazy val sqsClientBuilder = AmazonSQSClientBuilder.standard()
.withEndpointConfiguration(sqsEndpoint)
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
private lazy val client = sqsClientBuilder.build()<\/p> AmazonSQSClientBuilder.standard()\n .withCredentials(new DefaultAWSCredentialsProviderChain)\n .withRegion(region)<\/code><\/pre>\n\n\n\n