Sending data involves data points and data streams from the Remote Manager point of view, and also a new concept called data point collection from the ConnectCore Cloud Services API:

  • A data point represents a value at a specific moment in time and possibly at a geographical localization. To complete the information, a data point must belong to a data stream, which contains metadata common to all the data points on it.

  • A data stream defines common information for all data points that belong to it, including its stream ID (the name given in Remote Manager), type (float, integer, string, etc.), and units (C, km/h, seconds, etc.).

  • A data point collection is a group of data points (from one or more data streams) pending to be uploaded to their Remote Manager stream(s). When sending a data point collection all the data points in it are pushed to their data stream(s) in the same transaction, minimizing network usage.

To send data points to Remote Manager, follow these steps:

  1. Create a data point collection

  2. Add a data stream to the collection

  3. Add a data point

  4. Send data point collection to Remote Manager

  5. Destroy data point collection

  6. View data stream series in Remote Manager

For a list of possible errors sending data points to Remote Manager, see Description of errors.

Digi Embedded Yocto includes a Data points example demonstrating how to send data points to Remote Manager.

1. Create a data point collection

The first step before working with data points is to create an empty collection. You can use the cccs_dp_create_collection() function.

cccs_dp_error_t cccs_dp_create_collection(cccs_dp_collection_handle_t *const collection);
Parameter Description

collection

Address of the handler to the struct that will hold the data point collection. May be NULL when returning if the creation process fails.
Creating a data point collection
cccs_dp_collection_handle_t collection;
cccs_dp_error_t error;

/* [...] */

/* 1. Create the data point collection */
error = cccs_dp_create_collection(&collection);
if (collection == NULL) {
	log_error("Cannot create data point collection, error %d", error);
}

Function cccs_dp_create_collection() returns a cccs_dp_error_t error code indicating whether a failure occurs during the process, CCCS_DP_ERROR_NONE if success. See Description of errors for more information.

2. Add a data stream to the collection

Before adding a data point to a collection, you must add at least one data stream to the collection. This step explains how many parameters the data stream will hold and in which order they are expected.

A data stream defines:

  • Common information for the data points belonging to a data stream, such as unit, type, or stream ID.

  • How a data point is added to a collection, that is, how many parameters and in which order they are expected.

There are two functions. The first one obviates the last two parameters:

cccs_dp_error_t cccs_dp_add_data_stream_to_collection(
                          cccs_dp_collection_handle_t const collection,
                          char const * const stream_id,
                          char const * const format_string,
                          bool add_local_timestamp);
cccs_dp_error_t cccs_dp_add_data_stream_to_collection_extra(
                          cccs_dp_collection_handle_t const collection,
                          char const * const stream_id,
                          char const * const format_string,
                          bool add_local_timestamp,
                          char const * const units,
                          char const * const forward_to);
Parameter Description

collection

Data point collection to add the data stream to.

It must have been initialized with cccs_dp_create_collection().

stream_id

Name of the data stream destination in Remote Manager.

It must be unique.

For example, name "sensor_temp" creates a stream in Remote Manager with the path <DeviceID>/sensor_temp, where <DeviceID> is the ID of your device.

format_string

Null-terminated string to define the data type and how it is expected.

See explanation below.

add_local_timestamp

If true, the timestamp of each data point in the stream is calculated when the sample is added to data stream using cccs_dp_add() function.

If false, the timestamp is set by the server when data is uploaded.

units[1]

String to define the unit of the data in the stream, such as, seconds, C, etc.

NULL for no units.

forward_to[1]

Name of the data stream to replicate data points to.[2]

Can be NULL if not used.

1. only for cccs_dp_add_data_stream_to_collection_extra() function.

2. as soon as a data point is stored in the data stream, it is also copied or replicated in the data stream whose name is in the forward_to parameter.

Adding a data stream to the collection
cccs_dp_collection_handle_t collection;
cccs_dp_error_t error;

/* [...] */

/*
 * 2. Add the data stream "sensor_temp". Its data points attributes are:
 *   - Float value
 *   - Location required
 *   - Timestamp set when data point added
 *   - Quality not present
 */
error = cccs_dp_add_data_stream_to_collection(collection,
					      "sensor_temp",
					      CCCS_DP_KEY_DATA_FLOAT " " CCCS_DP_KEY_LOCATION,
					      true);
if (error != CCCS_DP_ERROR_NONE) {
	log_error("Cannot add sensor_temp data stream to collection, error %d", error);
}

The format_string is a combination of some keywords separated by a white space. Each keyword defines an attribute of the data points that are going to be added to the data stream:

  • Data type. Required attribute to indicate the type of the data points value to be added to this data stream.

  • Location. Optional attribute to indicate if the data points location is going to be captured.

  • Quality. Optional attribute to indicate if the sample quality is going to be captured.

The following options are available for each attribute:

Information type Keyword Description

Data type

CCCS_DP_KEY_DATA_INT32

32-bit signed integer value

CCCS_DP_KEY_DATA_INT64

64-bit signed integer value

CCCS_DP_KEY_DATA_FLOAT

Floating point value

CCCS_DP_KEY_DATA_DOUBLE

Double precision floating point value

CCCS_DP_KEY_DATA_STRING

Null-terminated string

CCCS_DP_KEY_DATA_JSON

Null-terminated string that represents a JSON object

CCCS_DP_KEY_DATA_GEOJSON

Null-terminated string that represents a GeoJSON object

Location

CCCS_DP_KEY_LOCATION

Location structure, with latitude, longitude and altitude (optional) as floating point values

Quality

CCCS_DP_KEY_QUALITY

Data Point quality value, as an integer
The order of the keywords in the format_string is very important. It defines the expected order when adding a data point (like in printf/scanf functions).

For example:

  • The format_string CCCS_DP_KEY_DATA_INT32 " " CCCS_DP_KEY_LOCATION means that all data points added to the data stream must pass a 32-bit integer as the first argument and the location structure as the second one.

  • The format_string CCCS_DP_KEY_DATA_STRING " " CCCS_DP_KEY_LOCATION " " CCCS_DP_KEY_QUALITY means that all data points added to the data stream must pass a null-terminated string as the first argument, the location structure as the second one, and an integer for the quality as the third one.

Functions cccs_dp_add_data_stream_to_collection() and cccs_dp_add_data_stream_to_collection_extra() return a cccs_dp_error_t error code indicating whether any failure occurs during the process, CCCS_DP_ERROR_NONE if success. See Description of errors for more information.

3. Add a data point

A data point represents a single value that is stored at a specific time in a data stream in Remote Manager. A data point is defined by its attributes:

  • Value. Required attribute, that represents the sample value to store. Its type must match with the type selected in the format_string when the data stream was added to the collection.

  • Timestamp. Optional value to specify when the value was captured. Depending on the value of add_local_timestamp, it can be set when the value is added to the data stream or by the server when data is uploaded.

  • Location. Optional value (cccs_location_t) to establish the location of the device when the sample was taken. Only required if CCCS_DP_KEY_LOCATION keyword was included in the format_string.

  • Quality. Optional value (int32_t) to define the quality of the sample. Only required if CCCS_DP_KEY_QUALITY keyword was included in the format_string.

The data point value is required to add a data point to a collection. The rest of the attributes are optional, only those ones in the format_string of cccs_dp_add_data_stream_to_collection() need to be provided. See Add a data stream to the collection.

The function cccs_dp_add() allows you to add data points to a data point collection providing the name of the stream which they belongs and the required data point attributes.

cccs_dp_error_t cccs_dp_add(cccs_dp_collection_handle_t const collection,
                            char const * const stream_id, ...);
Parameter Description

collection

Data point collection to add the data stream to.

It must have been initialized with cccs_dp_create_collection().

stream_id

Name of the data stream destination in Remote Manager. It must be unique.

...

Attribute values of the data point to be added (value, location, quality).

They must match (in number and order) the format_string specified when the stream_id data stream was added with cccs_dp_add_data_stream_to_collection().

See Add a data stream to the collection.

Function cccs_dp_add() returns a cccs_dp_error_t error code indicating whether any failure occurs during the process, CCCS_DP_ERROR_NONE if success. See Description of errors for more information.

Adding a data point to the collection
cccs_dp_collection_handle_t collection;
cccs_dp_error_t error;
int const FIVE_MINUTES = 5 * 60;
int i;

/* [...] */

error = cccs_dp_add_data_stream_to_collection(collection, "sensor_temp", CCCS_DP_KEY_DATA_FLOAT " " CCCS_DP_KEY_LOCATION);

/* [...] */

for (i = 0; i < 10; i++) {
	float temperature;
	cccs_location_t location;

	temperature = get_sensor_temperature();
	get_gps_location(&location.latitude, &location.longitude);
	location.elevation = 90;

	/*
	 * 3. Add data points.
	 * Pass temperature as a float and the location as it is specified in the format_string
	 */
	error = cccs_dp_add(collection, "sensor_temp", temperature, &location);
	if (error != CCCS_DP_ERROR_NONE) {
		log_error("Cannot add temperature data point, error: %d", error);
	}
	sleep(FIVE_MINUTES);
}

You can get the number of data points stored in a collection using the cccs_dp_get_collection_points_count() function.

cccs_dp_error_t cccs_dp_get_collection_points_count(cccs_dp_collection_handle_t const collection,
                                                    uint32_t * const count);

You can also manage the streams and data points in a collection with the functions:

Function Description 
cccs_dp_error_t cccs_dp_remove_data_stream_from_collection(
	cccs_dp_collection_handle_t const collection,
	char const * const stream_id);

Removes the given stream (stream_id) from the collection data points collection

 
cccs_dp_error_t cccs_dp_get_collection_points_count(
	cccs_dp_collection_handle_t const collection,
	uint32_t * const count);

Returns in count the number of data points stored in collection

 
cccs_dp_error_t cccs_dp_clear_collection(
	cccs_dp_collection_handle_t const collection);

Removes all data streams information and the data points stored in collection

These functions return a cccs_dp_error_t error code indicating whether any failure occurs during the process, CCCS_DP_ERROR_NONE if success. See Description of errors for more information.

4. Send data point collection to Remote Manager

Once all data points are added to the proper data stream in your data point collection, send them to Remote Manager using one of the following functions:

cccs_comm_error_t cccs_send_dp_collection(cccs_dp_collection_handle_t const collection,
                                          cccs_resp_t *resp);
cccs_comm_error_t cccs_send_dp_collection_tout(cccs_dp_collection_handle_t const collection,
                                               unsigned long const timeout,
                                               cccs_resp_t *resp);
Parameter Description

collection

Data point collection to send to Remote Manager

timeout[1]

Number of seconds to wait for response from the daemon.

resp

Received response from CCCS daemon.

1. only for cccs_send_dp_collection_tout() function.

All data points in a collection are sent to their respective data stream when the collection is uploaded to Remote Manager, regardless the order of addition.

The maximum number of data points allowed per request is 250.

These functions return a cccs_comm_error_t error code indicating whether any failure occurs during the process, CCCS_SEND_ERROR_NONE if success. See Description of errors for more information.

Sending data points to Remote Manager
int send_data(cccs_dp_collection_handle_t collection)
{
	cccs_comm_error_t ret;
	cccs_resp_t resp;

	log_info("%s", "Sending temperature values!");

	/* 4. Send the data points in the collection to Remote Manager */
	ret = cccs_send_dp_collection(collection, &resp);
	if (ret != CCCS_SEND_ERROR_NONE) {
		log_error("%s: error sending data points: CCCSD error %d",
			__func__, ret);
	} else if (resp.code != 0) {
		if (resp.hint)
			log_error("%s: error sending data points: CCCSD error %s (%d)",
				__func__, resp.hint, resp.code);
		else
			log_error("%s: error sending data points: CCCSD error %d",
				__func__, resp.code);
	}

	free(resp.hint);

	return ret;
}

If data cannot be uploaded and data backlog service is enabled, data is stored locally in the configured data_backlog_path directory.

This service is enabled and configured in the ConnectCore Cloud Services (CCCS) configuration file, /etc/cccs.conf. For more information about the configuration file, see Configure ConnectCore Cloud Services and Include ConnectCore Cloud Services applications in Digi Embedded Yocto.

If samples cannot be sent nor saved in the configured directory, or data backlog service is disabled, they remain in the collection. You must create custom code to remove them from the collection.

5. Destroy data point collection

At any time, you can free a collection’s allocated resources by calling cccs_dp_destroy_collection(). You can also clear a collection with cccs_dp_clear_collection(), which removes all the data streams information and the data points stored.

cccs_dp_error_t cccs_dp_destroy_collection(cccs_dp_collection_handle_t const collection);
Parameter Description

collection

Data point collection to destroy

Function cccs_dp_destroy_collection() returns a cccs_dp_error_t error code indicating whether any failure occurs during the process, CCCS_DP_ERROR_NONE if success. See Description of errors for more information.

6. View data stream series in Remote Manager

You can view the data points stored in the cloud (in charts and in text format) from the Remote Manager platform. To do so follow these steps:

  1. Go to the Insights > Data Streams tab.

  2. Click on the Stream ID item of the stream in the table you want to analyze.

    It is similar to 00000000-00000000-00XXXXXX-XXXXXXXX/sensor_temp, where 00000000-00000000-00XXXXXX-XXXXXXXX is the device ID.

    If you have several streams on the list, you can filter using the stream identifier:

    1. Click on the filter text box and select Stream ID.

    2. Type the stream identifier, for example sensor_temp, and press Enter.

  3. On the right, select the format to represent the values (Tabular, Line Chart, Area Chart, etc.), for example, Tabular. A table with the values in the selected stream is displayed.

See Monitor device data for more information on how to retrieve stored data in Remote Manager or listen to new data events using Remote Manager web service APIs.
Remote Manager data streams

Description of errors

cccs_comm_error_t error Description

CCCS_SEND_ERROR_NONE

Operation completed successfully. No error found.

CCCS_SEND_ERROR_INVALID_ARGUMENT

In most cases this refers to empty strings or NULL pointers in structures.

CCCS_SEND_ERROR_OUT_OF_MEMORY

CCCS encountered problems allocating memory to perform receive operations.

Your system may have run out of resources.

CCCS_SEND_ERROR_LOCK

CCCS encountered problems using synchronization mechanisms.

Your system may have run out of resources.

CCCS_SEND_UNABLE_TO_CONNECT_TO_DAEMON

Unable to connect to CCCS daemon.

It is probably not running.

CCCS_SEND_ERROR_BAD_RESPONSE

Received bad response from CCCS daemon.

CCCS_SEND_ERROR_FROM_CLOUD

Error received from Digi Remote Manager.
cccs_dp_error_t error Description

CCCS_DP_ERROR_NONE

Operation finished successfully.

CCCS_DP_ERROR_INVALID_ARGUMENT

In most cases this refers to empty strings or NULL pointers in structures.

CCCS_DP_ERROR_INVALID_STREAM_ID

When adding a data stream to a collection this error might mean that the provided stream name contains invalid characters or it already exists in the collection.

When adding a data point, it means that such stream name does not exist in the collection.

CCCS_DP_ERROR_INVALID_FORMAT

When creating a data stream, the provided format string contains invalid keywords or characters or is ambiguous (repeated keywords).

CCCS_DP_ERROR_INVALID_UNITS

The units can be NULL, but not an empty string.

CCCS_DP_ERROR_INVALID_FORWARD_TO

When creating a data stream, the forward to must contain only valid characters.

CCCS_DP_ERROR_INSUFFICIENT_MEMORY

CCCS encountered problems allocating memory to perform the requested operation.

Your system may have run out of resources.

CCCS_DP_ERROR_LOCK_FAILED

CCCS encountered problems using synchronization mechanisms.

Your system may have run out of resources.

CCCS_DP_ERROR_INITIATE_ACTION_FAILED

An unexpected failure on lower communication layers occurred.

You can check the specific error on the logging output.

CCCS_DP_ERROR_RESPONSE_BAD_REQUEST

The collection was uploaded but Remote Manager response indicated that it has invalid data.

CCCS_DP_ERROR_RESPONSE_UNAVAILABLE

Remote Manager response indicates that service is not available at this moment and can be retried later.

The account also may not have sufficient permissions for data points service.

CCCS_DP_ERROR_RESPONSE_CLOUD_ERROR

Remote Manager encountered an error while handling the request.

CCSI_DP_ERROR_STATUS_CANCEL

The operation did not finish properly because it was canceled by the user (probably by stopping CCCS).

CCCS_DP_ERROR_STATUS_INVALID_DATA

Invalid data passed to lower communication layers; this error should never be returned.

CCCS_DP_ERROR_STATUS_SESSION_ERROR

Indicates an error on lower communication layers.

Data points example

The Upload data points example demonstrates the usage of the CCCS send data API. In this example, a integer value is incremented every five seconds and added to the "incremental" data stream in a data point collection. The collection is sent to Remote Manager when it contains 10 samples.

This example is included in Digi Embedded Yocto. Go to GitHub to look at the application source code.