USING AMAZON KINESIS* AND DYNAMODB* ON THE INTEL® EDISON PLATFORM

Introduction

This article shows an example of how to create a Cognito* parameter, describe to the Amazon Kinesis* stream, put record, get record and monitor the Amazon Kinesis Stream. It also describes how to use the DynamoDB* cloud database service to store data. The Intel® Edison platform is a powerful controller, on which developers can develop their project in JavaScript*, C/C++, Wyliodrin*, Python*, and more. In this article, we are using JavaScript with Intel® XDK on the Intel Edison platform. For more information on the Intel XDK® IoT Edition, visit https://software.intel.com/en-us/articles/hands-on-intel-iot-developer-kit-using-intel-xdk.

Hardware Components

The hardware components used in this project are listed below:

Intel® Edison compute module
Arduino* expansion board
Two Micro B to Type A USB cables
A direct current (DC) power supply

For detailed instructions on how to assemble and power the Intel® Edison board, visit https://software.intel.com/en-us/assembling-intel-edison-board-with-arduino-expansion-board.

Development Board

Go to Intel® Edison Board Software Downloads to download the latest drivers, installers, firmware images, and Arduino IDE: https://software.intel.com/en-us/iot/hardware/edison/downloads. For information on how to configure the Intel® Edison platform, go to https://software.intel.com/en-us/articles/hands-on-intel-iot-developer-kit-using-intel-xdk.

Amazon Web Services* (AWS)

AWS is open source code that provides the JavaScript library with many AWS services, including Kinesis stream, DynamoDB, and more. To begin using AWS with the AWS SDK for JavaScript in Node.js on the Intel® Edison platform, install the AWS SDK JavaScript library and documentation on the serial terminal:

Figure 1: Installing the Amazon Web Services* SDK Library.

Amazon Kinesis Stream

Amazon Kinesis Stream enables real-time data streaming on a large scale and makes the data available for consumption. The data in the stream are the ordered sequence of data records. The data records in the stream are split into multiple shards. Splitting data into multiple shards is a technique used to spread out the load.

1. Creating the Kinesis Stream and Identity Pool

The first step is to create an AWS account, then create the Kinesis stream with the number of desired shard. Visit https://software.intel.com/en-us/articles/enabling-amazon-kinesis-on-the-intel-edison-board to create a Kinesis stream, an identity pool, an IAM role, and a policy. In the example below, the Kinesis stream IntelTinkKinesisStream has 5 shards and the name of the identity pool is IntelTinkPool.

Figure 2: Creating a Kinesis stream.

Figure 3: Creating an identity pool.

2. Creating Cognito Parameters for theKinesis Stream

Amazon Cognito is a web service to specify the temporary security credentials to mobile devices. The Amazon Cognito parameters are used to initialize the Cognito Credentials object.

Create the Cognito parameters for the Amazon Kinesis Stream as follows:

AccountId: From the Account Name in the upper-right corner, go to My Account.
RoleArn: From the Account Name in the upper-right corner, select Security Credentials, Roles, and then Cognito_IntelTinkPoolUnauth_Role.

Figure 4: Viewing Cognito* Roles ARN.

IdentityPoolId: Go to Cognito, select the Identity pool name created above, and then Edit identity pool.

Figure 5: Viewing the Amazon Cognito* Identify Pool ID.

var awsRegion = "us-east-1";
var streamName = 'IntelTinkKinesisStream';   // Kinesis stream
var tableName = "IntelTinkPool";

// Cognito parameters for Kinesis Stream
var cognitoParams = {
    AccountId: "648594433538",
    RoleArn: "arn:aws:iam::648594433538:role/Cognito_IntelTinkPoolUnauth_Role",
    IdentityPoolId: "us-east-1:3ad5e7f2-0a28-4fa3-b116-84a430da90be"
};

Code Example 1: Cognito parameters for the Amazon Kinesis* stream.

3. Creating Cognito Parameters for the Kinesis Stream

Now you are ready to write a sample application using the Amazon Cognito Identify service to create a credentials object.

var AWS = require('aws-sdk');
AWS.config.region = awsRegion;
AWS.config.credentials = new AWS.CognitoIdentityCredentials(cognitoParams);
AWS.config.credentials.get(function(err) {
    if (!err) {
        console.log("Cognito Identity Id: " + AWS.config.credentials.identityId);
    }
    else {
        console.log("Error in Cognito identiy credentials.");
        console.log(err);
    }
});

Code Example 2: Creating a new credentials object.

4. DescribeStream

DescribeStream returns the current status of the stream, the Amazon resource names (ARN), an array of shard objects, and it states whether there are more shards available.

The example below describe to the Amazon Kinesis stream to get the available shard.

kinesis.describeStream(describeParams, function(err, data) {
  if (err) {
      console.log(err, err.stack); // an error occurred
  }
  else {
        console.log("describeStream data");
        console.log(data.StreamDescription.Shards[4].ShardId);
  }
})

Code Example 3: DescribeStream

5. PutRecord

The sample code below puts a record in the JSON format into the Amazon Kinesis stream 'IntelTinkKinesisStream'. The record contains two data typed strings: the MAC address and the five-byte value. For more information about PutRecord, go to http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html.

function putRecordsToKinesisStream (){
    var json = {
        "PegID": "FF.FF.FF.FF.FF",
        "FiveBytesData": "0x1212121212",
    };

    var params = {
        Data: JSON.stringify(json),
        PartitionKey: partitionKey,
        StreamName: streamName
    };

    kinesis.putRecord(params, function(err, putData) {
        if (err) {
            console.log(err, err.stack);
        } else {
            try {
                sharId.value = putData.ShardId;
            } catch(err) {
                console.log("Error in Kinesis.putRecord()");
               console.log(err);
            }
        }
    });
};

Code Example 4: putRecord

6. getRecord

Before getRecords, we call getShardIterator to specify the name of the Kinesis stream, how to read the data records, and position in the shard from which to start reading records sequentially. For details description of getShardIterator, see http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html.

// Describe stream parameters
var describeParams = {
    StreamName: streamName
};

kinesis.describeStream(describeParams, function(err, data) {
    if (err) {
      console.log(err, err.stack); // an error occurred
    }
    else {
        var getParams = {
            ShardId: data.StreamDescription.Shards[4].ShardId,
            ShardIteratorType: "TRIM_HORIZON",   // get oldest package
            StreamName: streamName,
        };

        kinesis.getShardIterator(getParams, function(err, result) {
            if (err) {
                 console.log("Error in getShardIterator()");
                 console.log(err);
            } else {
                // Get records from the Kinesis stream
                getRecord(result.ShardIterator);
            }
        });
    }
});

Code Example 5: getShardIterator

Now we can GetRecords from the Amazon Kinesis stream 'IntelTinkKinesisStream'. For more information about GetRecords, look at http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html.

function getRecord(shard_iterator) {

    var getRecParams = {
        ShardIterator: shard_iterator
    };

    kinesis.getRecords(getRecParams, function(err, result) {
        if (err) {
            console.log("Error in getRecords() from the Kinesis stream.");
            console.log(err);
        } else {
            try {
                var StringDecoder = require('string_decoder').StringDecoder;
                var decoder = new StringDecoder('utf8');

                // If there are packages from getRecords()
                if(result.Records.length > 0) {
                    // Loop through all the packages
                    …
                }
            } catch(err) {
                console.log("Error parsing the package.");
                console.log(err);
            }

            if (result.NextShardIterator) {
                getRecord(result.NextShardIterator);
            }
        }
    });
}

Code Example 6: getRecords

Amazon DynamoDB

Amazon DynamoDB is a fully managed cloud service. Its flexible data management makes it great for Internet of Things (IoT) applications. With the Amazon DynamoDB, we can store the data broadcast from the Intel® Edison platform and analyze the data for specialized needs.

1. Creating a DynamoDB Table

Create a DynamoDB table by opening DynamoDB and clicking Create table. The example below creates a DynamoDB table named TinkSmartPegDb with two attributes PegID and FiveBytesData typed string.

Figure 6: Viewing the Amazon DynamoDB* table.

2. Editing Policies

The IAM policies allow us to specify any API action from any service that supports IAM. The policies below specify API actions for both the Amazon Kinesis stream and DynamoDB used in the example code throughout document.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1444052820000",
            "Effect": "Allow",
            "Action": [
                "kinesis:DescribeStream",
                "kinesis:PutRecord",
                "kinesis:PutRecords",
                "kinesis:GetShardIterator",
                "kinesis:GetRecords"
            ],
            "Resource": [
                "arn:aws:kinesis:us-east-1:648594433538:stream/IntelTinkKinesisStream"
            ]
        },
        {
            "Sid": "Stmt1445979793000",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-east-1:648594433538:table/TinkSmartPegDb"
            ]
        },
        {
            "Sid": "Stmt1445979793001",
            "Action": [
                "dynamodb:ListTables"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Figure 7: Policies for Kinesis* stream and DynamoDB*.

3. Calling listTable

listTable returns an array of the Amazon DynamoDB table names that were previously created. We call listTable to get the available Amazon DynamoDB table name.

var db = new AWS.DynamoDB();
var dbParams = {};

// If the DynamoDB table exists, get the DynamoDB table name.
db.listTables(dbParams , function(err, data) {
    if (err) {
        console.log("Error in dynamoDb.listTables(): ");
        console.log(err, err.stack); // an error occurred
        return;
    }
    else {
        tableStr.value = data.TableNames;
        console.log(tableStr.value.length);
    }
});

Code Example 7: listTables

4. Calling putItem

The function below receives the package from the Amazon Kinesis stream and stores it in the Amazon DynamoDB table.

putItem = function(tableName, pegId, fiveBytes) {
    // The item has two attributes: PegID type string and FiveBytesData type String
    var item = {
        'PegID': { 'S': pegId },
        'FiveBytesData': { 'S': fiveBytes }
    };

    var putItemParams = {
        TableName: tableName,
        Item: item
    };

    // Store the package in the DynamoDB
    db.putItem(putItemParams, function(err, data) {
    if (err) {
        console.log("Got error: ");
        console.log(err, err.stack); // an error occurred
    } else {
        console.log(JSON.stringify(data, null, 2));
    }
  });
};

Code Example 8: putItem

5. View DynamoDB Items

To view the list of items in the DynamoDB table, select the table name, and then select the Items tab. The TinkSmartPegDb below has two attributes PegID and FiveBytesData.

Figure 8: DynamoDB* items.

Monitoring the Amazon Kinesis Stream

We can collect, view, and analyze CloudWatch metrics for the Amazon Kinesis streams. The PutRecord and GetRecords appear in the graphs that show the number of successful records.

Figure 9: Monitor the PutRecord.

Figure 10: Monitor the GetRecords.

Sketch Examples

The following examples of sketches show how to repeatedly read the Kinesis stream, get the records from the Amazon Kinesis stream, and store them in the Amazon DynamoDB. Both of the examples below were developed and executed on the Intel® Edison platform.

Kinesis stream putRecord example

var awsRegion = "us-east-1";
var streamName = 'IntelTinkKinesisStream'; // Kinesis stream
var tableName = "IntelTinkPool";
var cognitoParams = {
    AccountId: "648594433538",
    RoleArn: "arn:aws:iam::648594433538:role/Cognito_IntelTinkPoolUnauth_Role",
    IdentityPoolId: "us-east-1:3ad5e7f2-0a28-4fa3-b116-84a430da90be"
};

var AWS = require('aws-sdk'); //package for AWS
AWS.config.region = awsRegion;
AWS.config.credentials = new AWS.CognitoIdentityCredentials(cognitoParams);
AWS.config.credentials.get(function(err) {
    if (!err) {
        console.log("Cognito Identity Id: " + AWS.config.credentials.identityId);
    }
    else {
        console.log(err);
    }
});

var kinesis = new AWS.Kinesis({apiVersion: '2013-12-02'});

function sharIdObj() {
    this.value = "";
}
var sharId = new sharIdObj();

function gateWayPackageObj() {
    this.value = "";
}
var package = new gateWayPackageObj();
var describeParams = {
        StreamName: streamName
    };

kinesis.describeStream(describeParams, function(err, data) {
  if (err) {
      console.log(err, err.stack); // an error occurred
  }
  else {
        console.log("describeStream data");
        console.log(data);           // successful response
  }
})

setInterval(putRecordsToKinesisStream, 5000);

////////////////////////////////////////////////////////////////////////////////////////////////
// Function Name: putRecordsToKinesisStream()
// This function puts the package into the Kinesis stream. The Kinesis stream is pre-created.
// Parameters: NA
////////////////////////////////////////////////////////////////////////////////////////////////

function putRecordsToKinesisStream (){

    var json = {
        "PegID": "FF.FF.FF.FF.FF",
        "FiveBytesData": "0x1212121212",
    };

    var params = {
        Data: JSON.stringify(json),
        PartitionKey: partitionKey,
        StreamName: streamName
    };

    kinesis.putRecord(params, function(err, putData) {
        if (err) {
            console.log(err, err.stack);
        } else {
            try {
                sharId.value = putData.ShardId;
            } catch(err) {
                console.log("Error in Kinesis.putRecord()");
               console.log(err);
            }
        }
    });
};

Code Example 9: Kinesis* stream putRecord() example.

Continuously read Kinesis stream example

var awsRegion = "us-east-1";
var streamName = 'IntelTinkKinesisStream';   // Kinesis stream
var tableName = "IntelTinkPool";

// Cognito parameters for Kinesis Stream and DynamoDB
var cognitoParams = {
    AccountId: "648594433538",
    RoleArn: "arn:aws:iam::648594433538:role/Cognito_IntelTinkPoolUnauth_Role",
    IdentityPoolId: "us-east-1:3ad5e7f2-0a28-4fa3-b116-84a430da90be"
};

var AWS = require('aws-sdk');          //package for AWS
AWS.config.region = awsRegion;
AWS.config.credentials = new AWS.CognitoIdentityCredentials(cognitoParams);
AWS.config.credentials.get(function(err) {
    if (!err) {
        console.log("Cognito Identity Id: " + AWS.config.credentials.identityId);
    }
    else {
        console.log("Error in Cognito identiy credentials.");
        console.log(err);
    }
});

var kinesis = new AWS.Kinesis({apiVersion: '2013-12-02'});
var db = new AWS.DynamoDB();
var dbParams = {};

// Object for the DynamoDB table name
function tableObj() {
    this.value = "";
}
var tableStr = new tableObj();

// If the DynamoDB table exists, get the DynamoDB table name.
db.listTables(dbParams , function(err, data) {
    if (err) {
        console.log("Error in dynamoDb.listTables(): ");
        console.log(err, err.stack);    // an error occurred
        return;
    }
    else {
        tableStr.value = data.TableNames;
        console.log(tableStr.value.length);
    }
});

// Describe stream parameters
var describeParams = {
    StreamName: streamName
};

kinesis.describeStream(describeParams, function(err, data) {
    if (err) {
      console.log(err, err.stack);    // an error occurred
    }
    else {
        var getParams = {
            ShardId: data.StreamDescription.Shards[4].ShardId,
            ShardIteratorType: "TRIM_HORIZON",     // get oldest package
            StreamName: streamName,
        };

        kinesis.getShardIterator(getParams, function(err, result) {
            if (err) {
                 console.log("Error in getShardIterator()");
                 console.log(err);
            } else {
                // Get records from the Kinesis stream
                getRecord(result.ShardIterator);
            }
        });
    }
});

////////////////////////////////////////////////////////////////////////////////////////////////
// Function Name: getRecord
// This function gets records from the kinesis stream and then puts the item into the
// DynamoDB.
//
// Parameters:
//     - shard_iterator: The shard position from which to start reading data records
//                                    sequentiallly.
////////////////////////////////////////////////////////////////////////////////////////////////
function getRecord(shard_iterator) {
    var getRecParams = {
        ShardIterator: shard_iterator
    };

    kinesis.getRecords(getRecParams, function(err, result) {
        if (err) {
            console.log("Error in getRecords() from the Kinesis stream.");
            console.log(err);
        } else {
            try {
                var StringDecoder = require('string_decoder').StringDecoder;
                var decoder = new StringDecoder('utf8');

                // If there are packages from getRecords()
                if(result.Records.length > 0) {
                    // Loop through all the packages
                    for(var i = 0; i < result.Records.length; i++) {
                        if(result.Records[i] != undefined) {
                            var getData = JSON.parse(decoder.write(result.Records[i].Data));

                            console.log("PegId =");
                            console.log(getData.PegID);

                            // Put the package into the DynamoDB table
                            putItem(tableStr.value[0], getData.PegID, getData.FiveBytesData);
                        }
                    }
                }
            } catch(err) {
                console.log("Error parsing the package.");
                console.log(err);
            }

            if (result.NextShardIterator) {
                getRecord(result.NextShardIterator);
            }
        }
    });
}

////////////////////////////////////////////////////////////////////////////////////////////////
// Function Name: putItem
// This function
//     - Creates an item using the pass in parameters: pegId and fiveBytes
//     - Calls dynamoDB.putItem() to put the item into the tableName
// Parameters:
//     - tableName: The name of the DynamoDB
//     - pegId:     UUID of the peg
//     - fiveBytes: The stock information of the peg
////////////////////////////////////////////////////////////////////////////////////////////////
putItem = function(tableName, pegId, fiveBytes) {
    // The item has two attributes: PegID type string and FiveBytesData type String
    var item = {
        'PegID': { 'S': pegId },
        'FiveBytesData': { 'S': fiveBytes }
    };

    var putItemParams = {
        TableName: tableName,
        Item: item
    };

    // Store the package in the DynamoDB
    db.putItem(putItemParams, function(err, data) {
    if (err) {
        console.log("Got error: ");
        console.log(err, err.stack); // an error occurred
    } else {
        console.log(JSON.stringify(data, null, 2));
    }
  });
};

Code Example 10: Repeatedly read Kinesis* stream example.

Summary

We have described how to use Amazon Kinesis stream and DynamoDB on Intel® Edison platform using JavaScript with Intel XDK. Try different Amazon Web Services (AWS) and hands-on experience with the Intel® Edison platform.

References

Amazon Kinesis Service API Reference:

http://docs.aws.amazon.com/kinesis/latest/APIReference/API_Operations.html

IoT References:

About the Author

Nancy Le is a software engineer at Intel Corporation in the Software and Services Group working on the Intel® Atom^TM processor and IoT scale enabling projects.

Notices

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Check with your system manufacturer or retailer or learn more at intel.com.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.

This sample source code is released under the Intel Sample Source Code License AgreementLike SubscribeAdd new comment Flag as spam .Flag as inappropriate Flag as Outdated

Intel, the Intel logo, and Intel Atom are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others