How to test AWS step functions locally
Sun, 15 Sep 2019

In mathematics, a function on the real numbers is called a step function if it can be written as a finite linear... Oh wait, wrong topic, sorry =)

Let me start again. For who doesn’t know AWS step functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly. You can basically build a multi-step workflow with parallel or serial execution of aws services or simple lambda functions, you can add state and conditions. In the AWS console you can build your logic and have a nice diagram of your whole process.

Awesome right? Well this awesomeness can become a debug(ness) nightmare(ness) especially if we cannot test it in local. And that is why Amazon offer us, mere mortals, a containerized version of step functions.

AWS docs are really good and offer plenty of information, often so much that you will end up puking knowledge. This, plus some "minor" key details missing on the aws examples and the infinite number of "hello world with step functions and serverless framework" tutorials on medium, will sure end up triggering your periodic career crisis. No worries, I have sacrificed myself already. In this blog post I will explain how I have managed to successfully run my step functions locally including DynamoDB.

Those are the tools that you need:

Yea that's a tons of stuff. Anyway, you can actually follow the instructions here, but there are some missing details and they do not include DynamoDB.

The first thing to do, and the most important, is to write your state machine. If you are not familiar with the state language, I suggest you navigate to the 

AWS console > step functions > create state machine.
The visual editor will help you to visualize what you are doing and tell you if there are any errors in your code. Once you finish copy the code snippet into a .json file in your root folder. This is not a tutorial on how to use step functions and lambda so I presume that your project is already setup with all your functions and a cloudformation template with all the roles and policies assigned.

Second, create a simple text file with your credentials, you can call it stepfunctions-credentials.txt, (it could be any name). It should look like this:

AWS_ACCOUNT_ID=<your id>
AWS_DEFAULT_REGION=ap-southeast-1
AWS_ACCESS_KEY_ID=<your access key>
AWS_SECRET_ACCESS_KEY=<your secret key>
LAMBDA_ENDPOINT=http://127.0.0.1:3001
DYNAMODB_ENDPOINT=http://<container name>:8000

I will explain later about DYNAMODB_ENDPOINT and LAMBDA_ENDPOINT. Note: If we want containers to be able to resolve IP addresses by name, we should use a user defined network: 

docker network create $NETWORK_NAME

Third, now we can start everything. First missing piece of information in the AWS docs is on the docker script to run our step functions container 

docker run -p 8083:8083 amazon/aws-stepfunctions-local
The missing piece is the flag --network host , without this flag your docker container will not be able to communicate with the sam local container. So, let us start everything:

  1. Start DynamoDB on a user defined network: 

    docker run -d --name $CONTAINER_NAME --network $NETWORK_NAME -p 8000:8000 amazon/dynamodb-local

  2. Start sam local on the same user defined network of DynamoDB: 

    sam local start-lambda --docker-network $NETWORK_NAME

  3. Start Step functions on host network, hypothetically we could run on the same user defined network, but then I wasn't able to find any information about the sam container endpoint: 

    docker run -d --name $CONTAINER_NAME --network host -p 8083:8083 \
--env-file /path/to/stepfunctions-credentials.txt \
amazon/aws-stepfunctions-local

After SAM local has started, you should see a message "running on http://127.0.0.1:3001/" and that’s your LAMBDA_ENDPOINT while the DYNAMODB_ENDPOINT is going to be the name of your dynamodb container plus its port, in this case: http://dynamodb:8000

Fourth, now we have to create and execute our state machine

  1. Create the state machine 

    aws stepfunctions \
--endpoint http://localhost:8083 \
  create-state-machine --definition "`cat path/to/your-statemachine-definition.json`" \
  --name "Your state machine name" \
  --role-arn "arn:aws:iam::012345678901:role/DummyRole" \

  2. Start the execution: after run the above script should echo a json like object with the property stateMachineArn  which you can copy and paste in the next script: 

    aws stepfunctions \
  --endpoint http://localhost:8083 \
  start-execution \
    --state-machine yourStateMachineArn \
    --input "{ \"yourInputKey\": \"yourInputVale\"}" \
    Also this script will echo another json object with the property executionArn. To make sure everything is running correctly, you can copy the executionArn and paste it in the following script: 
    aws stepfunctions \
  --endpoint http://localhost:8083 \
  describe-execution \
    --execution-arn yourExecutionArn

Lots of steps to make other steps work uhu?. To make this process less painful you can create a full script, however if you are too lazy you can check my gits with the full bash script.

I hope you liked this post, if you think you can make this flow better, please do not hesitate to send me an email at [email protected]. I really would love to hear your opinion

Cheers!

Thanks for reading!