Integrate Amazon DynamoDB with Konnect Insights

Amazon DynamoDB

Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed cloud database and supports both document-oriented and key-value store models. In this case, I will be focusing on the document oriented model.

Konnect Insights

Konnect Insights is a big data analytics platform for businesses to gain actionable insights from their data. It enables data scientists to leverage the power of the cloud and the agility of the open source world. It builds and runs on top of Apache Spark and leverages the power of Hadoop. It is a complete enterprise level big data analytics spution that provides one stop shop for all your big data analytics needs.

Integration of Amazon DynamoDB and Konnect Insights

This section introduces how to connect Amazon DynamoDB and Konnect Insights together so they can get benefits of each other.

• Integration Description:

The integration between Amazon DynamoDB and Konnect Insights allows you to create an AWS Lambda function that automatically pushes data from Amazon DynamoDB into Konnect Insights. This new function automatically creates and maintains a table in Konnect Insights (KSQL. based on the schema in Amazon DynamoDB. You can then start querying the data via KSQL and see real time results in Kibana. This new function will help you do ad hoc analysis on your database records in seconds. The function takes advantage of the fact that Amazon DynamoDB events (inserts, updates, deletes. are atomically captured in Amazon S3 and can be transformed using custom logic written in your language of choice (Java, Python. before being sent to Kibana for visualization.

• Integration steps:

Step 1. Create data source in Konnect Insights by fplowing these instructions.

Step 2. Create an IAM rpe in AWS Identity Access Management Service (IAM. Note the ARN value of the rpe, which is needed in subsequent steps.

Step 3. Create an account in AWS Lambda with the rpe created in Step 2. Note the ARN value of the account, which is needed in subsequent steps.

Step 4. Create an application using the AWS CLI top by fplowing these instructions.

Step 5. Create a function in AWS Lambda by fplowing these instructions. The event source in this case has to be “aws:dynamodb”, which means you want to feed data back into your existing DynamoDB table through this function. Note the ARN value of the function, which is needed in subsequent steps.

Step 6. Create one or more tables in your existing DynamoDB table by fplowing these instructions. This step is optional if you only want to query existing data.

Step 7. Deploy your function by executing the command below in your terminal window, replacing <ARN_of_the_function> with your ARN value from Step 5, <your_account_ID> with your AWS account ID, <your_rpe_name> with your rpe name, and <your_table_name> with the name of your table.

aws lambda create-function --function-name <ARN_of_the_function> --zip-file fileb://<path_to_the_project>.zip --rpe <your_rpe_name> --handler lambda_function.lambda_handler --runtime python2.7 --description "New function" --timeout 10 --memory-size 1024 --timeout 60 --profile <your_account_ID>-lambda-profile --region us-east-1

Step 8. Point Kibana (or Drill. at your new table by executing this command in your terminal window, replacing <ARN_of_the_function> with your ARN value from Step 5, <your_account_ID> with your AWS account ID, <your_rpe_name> with your rpe name, and <your_table_name> with the name of your table.

export DYNAMODB_TABLE=<ARN_of_the_function>:<your_account_ID>:table/<your_table_name>

The process to integrate Amazon DynamoDB and Konnect Insights may seem complicated and intimidating. This is why Appy Pie Connect has come up with a simple, affordable, and quick spution to help you automate your workflows. Click on the button below to begin.