Tips on how to combine AWS IoT Core with Amazon MSK

Submit by Milo Oostergo, Principal Options Architect and Doron Bleiberg, Senior Resolution Architect, AWS Startups


Monitoring IoT gadgets in actual time can present invaluable insights that may assist you to keep the reliability, availability, and efficiency of your IoT gadgets. AWS IoT Core gives integrations with Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (“Amazon MSK”) to arrange real-time streaming knowledge pipelines. Amazon MSK is in style selection for purchasers who’re accustomed to Kafka, want infinite message retention, and are on the lookout for the bottom latency. On this weblog publish, we describe the right way to arrange AWS IoT Core to stream occasions to Amazon MSK and customary asks from our clients.


The diagram beneath illustrates the elements and performance you possibly can construct following this weblog publish or utilizing this pattern AWS CloudFormation template. As a part of this resolution, MQTT messages streamed to AWS IoT Core are routed to Amazon Managed Streaming for Apache Kafka (Amazon MSK) utilizing AWS IoT Guidelines actions. Entry to the Amazon MSK cluster is managed utilizing username and password which can be securely saved in AWS Secrets and techniques Supervisor and encrypted utilizing AWS Key Administration Service.


Solution overview


Step 1: Organising an Amazon MSK cluster

To ship messages from IoT gadgets to Amazon MSK utilizing AWS IoT Core rule actions, you’ll want to allow authentication in your Amazon MSK cluster. IoT rule actions can authenticate together with your Amazon MSK cluster with username and password authentication utilizing the SASL framework or through the use of TLS shopper authentication by way of AWS Certificates Supervisor. On this weblog publish, we arrange the cluster utilizing SASL/SCRAM authentication technique. As soon as a cluster is created, you possibly can’t modify the the authentication settings.

To create the Amazon MSK cluster with authentication enabled

  1. From the Amazon MSK console, select Create Cluster.
  2. Choose, enter a cluster title, and preserve the really helpful Apache Kafka model.Amazon MSK create cluster
  3. In Networking, choose your VPC and select “2” for Variety of Availability Zones. From the drop-downs, choose the 2 Availability Zones within the VPC, and select the personal subnets for every.
  4. In Entry management technique, select SCRAM/SASL authentication.Selecting the SASL/SCRAM authentication method for your Amazon MSK cluster
  5. Preserve the prevailing defaults and select Create cluster. It takes as much as quarter-hour to create the cluster and the standing is displayed within the Cluster Abstract panel.

Step 2: Create credentials in AWS Secrets and techniques Supervisor  

After the cluster is efficiently created, we create a set of credentials that can be utilized by the IoT rule to attach with the Amazon MSK cluster. The credentials have to be saved in AWS Secrets and techniques Supervisor and related to the cluster. Earlier than we create the credentials in AWS Secrets and techniques Supervisor, we first create a customer-managed key in AWS Key Management Service (KMS). Secrets and techniques encrypted with a AWS managed CMK can’t be used with an Amazon MSK cluster.

  1. Open the AWS KMS console and select Create key.
  2. Select symmetric key and observe the wizard to create the important thing. You don’t need to outline the important thing administrative permissions or key utilization permissions at this level. We set this up later.Now that the KMS key’s created, we will retailer the credentials in AWS Secrets and techniques Supervisor.
  1. Open the AWS Secrets Manager console and select Retailer a brand new credential.
  2. Select Different kind of secrets and techniques (e.g. API key) for the key kind.Store your secret in AWS Secrets Manager
  3. Enter the consumer and password knowledge, which have to be within the following format:
       "username": "msk",
       "password": "msk-secret"

  4. Choose the shopper managed key you created in earlier step.
  5. To affiliate secrets and techniques with the Amazon MSK cluster, the key title should have the prefix AmazonMSK_. On this instance, we use the title AmazonMSK_secret.
  6. Report the ARN (Amazon Useful resource Identify) worth on your secret.

Step 3: Affiliate secret with Amazon MSK cluster

As soon as the key is created in AWS Secrets and techniques Supervisor, we will affiliate the key with our Amazon MSK cluster.

  1. Return to the Amazon MSK console and choose your cluster.
  2. Select Affiliate secrets and techniques and copy-paste the ARN of the key you created in earlier step.

Associate secret with Amazon MSK cluster

Step 4: Arrange AWS Id and Entry Administration (IAM) position and coverage for AWS IoT rule

To grant AWS IoT entry to stream knowledge to our Amazon MSK cluster, you need to create an IAM position with a coverage that enables entry to the required AWS sources.

To create an IAM position utilizing AWS CLI

  1. Save the next belief coverage doc, which grants AWS IoT permission to imagine the position, to a file named iot-role-trust.json:
          "Effect": "Allow",
          "Principal": {
            "Service": ""
          "Action": "sts:AssumeRole"

  2. Use the create-role command to create an IAM position specifying the iot-role-trust.json file. Be sure to exchange the AWS account id and area iam create-role --role-name IoT-Rule-MSK-Position --assume-role-policy-document file://iot-role-trust.json
  3. Save the next JSON right into a file named iot-msk-policy.json.
             "Useful resource":"*"
             "Useful resource":" ""arn:aws:secretsmanager:area:123456789012:AmazonMSK_*"

    This JSON is an instance coverage doc that gives entry to create and handle elastic community interfaces in your Amazon Digital Non-public Cloud and retrieve the credentials to succeed in your Kafka brokers.

  4. Use the create-policy command to outline the actions and sources that AWS IoT Core can entry upon assuming the position, by passing within the iot-msk-policy.json file:aws iam create-policy --policy-name IoT-Rule-MSK-policy --policy-document file://iot-msk-policy.json
  5. Use the attach-role-policy command to connect your coverage and grant AWS IoT entry. Exchange the placeholder ARN by the coverage ARN returned within the earlier iam attach-role-policy --role-name IoT-Rule-MSK-Position --policy-arn "arn:aws:iam::123456789012:coverage/IoT-Rule-MSK-policy"

    To grant the IAM position entry to the KMS key
    In an effort to decrypt the key saved in AWS Secrets and techniques Supervisor, we should add the IAM position to the listing of key customers for the Buyer Managed KMS key we earlier created.
    1. Go to the AWS KMS console and choose the KMS key you created within the earlier step.
    2. In Key customers add the IAM position IoT-Rule-MSK-Position.

Step 5 – Create VPC vacation spot for AWS IoT core

Create a vacation spot to your VPC the place Apache Kafka clusters reside. This vacation spot is used to route messages from gadgets to your Amazon MSK cluster.

  1. Go to AWS IoT console, select Act, after which select Locations.
  2. Select Create a VPC vacation spot.
  3. Choose the VPC and identical subnets which can be used on your Amazon MSK cluster.
  4. Choose safety group that’s used on your Amazon MSK cluster.
  5. Choose the IoT-Rule-MSK-Position you created within the earlier step.

Step 6 – Create AWS IoT rule

  1. Go to AWS IoT console, select Act, select Guidelines, and create a brand new rule.
  2. In Actions select Add motion and choose Kafka.
  3. Choose the VPC vacation spot you created within the earlier step.
  4. Specify the Kafka subject.
  5. Specify the TLS bootstrap servers of your Amazon MSK cluster. You possibly can view the bootstrap server URLs in shopper data of your MSK cluster particulars.View client information to connect to your Amazon MSK cluster
  6. As we arrange our Amazon MSK cluster with the SCRAM SASL authentication technique, choose SSL_SASL as safety.protocol and choose SCRAM-SHA512 as sasl.mechanism.
  7. Specify the next variable in sasl.scram.username and exchange the title AmazonMSK_secret with the title of the key you saved in step 2.${get_secret(‘AmazonMSK_secret', 'SecretString', ‘username’, 'arn:aws:iam::123456789012:position/iot-msk-role)}
  8. Specify the next variable in sasl.scram.password and save the IoT rule motion.${get_secret(‘AmazonMSK_secret', 'SecretString', ‘password, 'arn:aws:iam::123456789012:position/iot-msk-role)}Testing the AWS IoT rule
    At this level, you might have created the Amazon MSK cluster and arrange an AWS IoT Core rule with the required IAM permissions. To confirm IoT occasions are streaming to your Amazon MSK cluster, you possibly can join a Kafka shopper to your bootstrap servers and ship an occasion to your IoT subject utilizing the MQTT take a look at shopper out there within the AWS IoT console.Sending a MQTT test messageThe Kafka shopper linked to your cluster can now obtain messages on the Amazon MSK subject. To be taught how one can connect with your Amazon MSK cluster, see the part Connecting to your cluster with a username and password within the Amazon MSK developer guide.Receiving IoT messages in your Kafka consumerOrganising the permissions incorrectly is a typical subject leading to clients not receiving occasions on their Amazon MSK cluster.  When AWS IoT is unable to ship occasions, the foundations engine triggers an Error motion. For instance, you possibly can arrange an error motion to ship the occasions to Amazon CloudWatch Logs and specify the CloudWatch log group to which the IoT rule motion sends the info. When an error happens whereas processing your rule, you possibly can view the stream of log occasions within the log group in CloudWatch Logs.

Cleansing up

When you adopted together with this resolution, full the next steps to keep away from incurring undesirable fees to your AWS account.

AWS IoT Core

  • Within the Act part, delete the rule and VPC vacation spot.

Amazon MSK


  • Delete the Buyer Managed Key used to encrypt the secrets and techniques saved in AWS Secrets and techniques Supervisor.

AWS Secrets and techniques Supervisor

  • Delete the key created to authenticate together with your Amazon MSK cluster.


  • Delete the insurance policies and roles created alongside the way in which.

Amazon CloudWatch

  • Delete the related Log teams.


On this publish, we gave you an summary of the right way to construct a real-time streaming knowledge pipeline on your IoT gadgets by integrating AWS IoT Core with Amazon MSK. We confirmed the way you setup Amazon MSK and IoT rule actions to ship messages to Apache Kafka.

In regards to the authors

Milo Oostergo
Milo is a Principal Resolution Architect for AWS Startups workforce in Amsterdam. Earlier than becoming a member of the Startup workforce, he labored as Principal Product Supervisor on varied AWS companies.

Doron Bleiberg
Doron is a senior Resolution Architect for AWS Startups workforce in Israel, specializing in AWS IoT companies

Source link

Compare items
  • Total (0)