In recent years, the development of technology has been going very fast. Data storage is one of the important parts in information technology. There are a lot of data storage systems which are developed each year. Many companies try to develop their own Windows Azure, Amazon S3, Google Cloud Storage and so on.However, we may need more than one data storage system to develop our product. It's really hard to integrate all these data storage systems together. We have to use multiple codebases to support them. In my opinion, it's waste of time and resources.Amazon DynamoDB and GitLab are two tools that we can use together to support our product development. I'm going to explain how they work together in this article.

What is Amazon DynamoDB?

Amazon DynamoDB is a fully managed NoSQL database service provided by Amazon Web Services. It is used to store and retrieve any amount of data, and serve any level of request traffic. It is designed for fast and predictable performance with seamless scaling.

What is GitLab?

GitLab is a web-based Git repository manager featuring code reviews, issue tracking, activity feeds and wikis. It is developed by GitLab Inc., a company based in Barcelona (Spain. GitLab offers free software that anyone can use and contribute to. The most popular users are developers who write code every day. They want a tool that works for them with no boundaries or hidden costs.GitLab can be used as self-hosted software, on top of an existing infrastructure or cloud service provider like AWS or Rackspace. GitLab provides a very simple installation procedure and many features to make it easy for anyone to set up a scalable web application based on GitLab Community Edition (CE), such as LDAP integration, CI/CD integration, SSO support, Jira integration and others.GitLab CE is free and open source under the MIT license. Its API is also published as open source under the MIT license, along with its documentation. This allows you to host it yourself, contribute improvements, or build something else on top of it.

Integration of Amazon DynamoDB and GitLab

With the help of AWS Lambda function, we can easily integrate Amazon DynamoDB and GitLab together. AWS Lambda allows us to run code without provisioning or managing servers. All we need is just upload our code and they will run it for us.You can see my gitlab-dynamodb project on GitHub at https://github.com/hajimehoshi/gitlab-dynamodb . The project consists of three main parts. dynamodb_mapping_helper , dynamodb_result_helper and gitlab_dynamodb_helper . For this part, I'm going to describe only the first part because it does not need any explanation about how AWS Lambda functions work.The first part is dynamodb_mapping_helper . It maps DynamoDB items into GitLab issues. The following is an example mapping configuration file:# Configuration file for dynamodb_mapping_helper# Keys from dynamodb should be prefixed with dynamo_key_.# Values should be json strings as they are returned by dynamodb item getter methods.# This mapping only supports "string", "number", "boolean" and "array" type fields.# DyanmoDB string type will be converted into string field in gitlab issue automatically.# Values below will be ignored in mapping process. You don't need to specify them here!# dynamodb_mapping_helper_configuration_file_version=1dynamo_key_table=pending_issues # Dynamodb table name for pending issues# dynamo_key_project=projects # Dynamodb table name for projects informationdynamo_key_project_id=pending # Dynamodb key for projects iddynamo_key_priority=pending # Dynamodb key for prioritydynamo_key_time=2014-11-03T22:14:46Z # Dynamodb key for time stamp when issue was createddynamo_key_date=2014-11-04T00:00:00Z # Dynamodb key for date stamp when issue was createddynamo_key_assignee=owner # Dynamodb key for assignee fielddynamo_key_labels=[] # Dynamodb key for labels fielddynamo_key_position=[] # Dynamodb key for position field (only exists when accessing project through api)dynamo_key_issue__title=title # Dynamodb key for title fielddynamo_key_description=body # Dynamodb key for description field (only exists when accessing project through api)dynaomdboeot_key_actions=[] # Dynamodb key for actions field (only exists when accessing project through api)The next part is dynamodb_result_helper . It returns the mapped issues from DynamoDB if we give an issue identifier as parameter. The following is an example script:#!/usr/bin/env python3import boto3 from dynamodb_result_helper import getDynamoDBIssueByIssueID def lambdaHandler(event, context. try. entity = boto3.client('dynamodb'.get_table( TableName='pending_issues', KeyConditionExpression='status = :status', ExpressionAttributeNames={'status'. 'Status'}, ExpressionAttributeValues={'status'. 'PENDING'} . return getDynamoDBIssueByIssueID(entity['Item']['Id']. except Exception as e. print(e. raise SystemExit('Failed to map dynamob DB item'. def getDynamoDBIssueByIssueID(issueId. return { 'issue'. { 'id'. issueId } } module = __import__('gitlab-dynamodb'.lambda_handler handler = module(lambdaHandler. aws = boto3.client('dynamodb'.DynamoDB aws.table_exists('pending'. aws.put_item('pending', Item={ 'status'. 'PENDING' }. events = [ { 'LambdaFunctionName'. handler } ] response = aws.invoke( Events=events . if response['LambdaFunctionResult']['Succeeded']. print("ok". else. print("failed". raise SystemExit('Failed to map dynamob DB item'. ` And the last part is gitlab_dynamodb_helper . It creates issues in GitLab whenever new items are added from DynamoDB (either manually or automatically. The following is an example script. ```sh MODULE=$1 ACTION=$2 if [[ "$ACTION" == "add" ]]; then mkdir -p /var/www/html/$MODULE/issues cd /var/www/html/$MODULE/issues curl -X POST -H "Content-Type. application/json" -H "Authorization. Bearer $AWSACCESSKEY" -d '{"issue":{"id". "$AWSENDPOINT/$MODULE/${REPLACEITEM}","title". "${REPLACEITEM}","description". "description"},"author":{"name":"${REPLACEITEM}"}}' https://api.gitlab.com/v4/projects/$GITLABENDPOINT/$MODULE fi ` To use those scripts, we need to declare mapping from DynamoDB into GitLab issues in dynamodb_mapping_helper . You can find more details about how to write mapping rules at https://github.com/hajimehoshi/gitlab-dynamodb . After we declare mapping rules, we can create a lambda function using AWS CloudFormation template at https://github.com/hajimehoshi/gitlab-dynamodb . And then we need to configure gitlab hooks so that change items from DynamoDB are created in GitLab automatically (See https://docs.gitlab.com/ce/workflows/ci/auto-merge-requests/#auto-merge-requests . By doing these configurations, we can create issues in GitLab when new items are created from DynamoDB automatically

The process to integrate Amazon DynamoDB and GitLab may seem complicated and intimidating. This is why Appy Pie Connect has come up with a simple, affordable, and quick solution to help you automate your workflows. Click on the button below to begin.