API Endpoint Workflows for Records Ingestion and Invalid Records

Introduction

Purpose

This document outlines the non-authentication API records-related endpoint workflows for the Data Intake Gateway (DIG). These include:

1. Posting records into DIG — /records POST
2. Querying for invalid records — /invalidrecords GET

The document also includes best practices and sample implementation flow charts for each of the workflows.

Terms and definitions

Required reading

Before reading this document, you should read and understand:

1. The DIG API Authentication Workflow. All workflows identified here require a valid and unexpired bearer token. All DIG API endpoints (that are not authentication related endpoints) require the Authorization: Bearer HTTP header. Thus, integration systems need to implement the authentication workflow alongside the aforementioned flows.
2. The above Terms and Definitions.
3. For records there are concepts from the MyGeotab SDK that are relevant:
4. 1. Unit Of Measure
   2. Working With Dates

Overview

The DIG API provides a primary endpoint to enable integrators to send bulk custom telematics device records to MyGeotab using the /records endpoint.

Additionally, it provides secondary endpoints to help the integrators with troubleshooting, which include:

1. The /invalidrecords endpoint for invalid records that may have been deemed as invalid or rejected.

Records workflow

Systems that integrate with DIG are expected to send bulk requests containing valid records to the /records API endpoint. You can make these requests as frequently as required for your use case with a consideration of size per request. See the best practices for /records to learn more.

Figure 1 - Record Endpoint API Workflow

Records endpoint

A call to /records posts a data payload of records using JSON format as an array of record items. All records must contain the following [string] fields:

1. “DateTime” (timestamp)
2. “SerialNo” (device serial)
3. “Type” (record type)

You should provide all other record fields necessary for the given record type. See the DIG OpenAPI YAML spec for a list of supported record types and their respective fields.

See a sample workflow implementation flowchart below:

Figure 1 - Record Endpoint Sample Implementation Workflow

Endpoint

[https://dig.geotab.com:443/records] POST

Request Body

Array of valid records items

Returns

If the request is accepted:

1. 202 ACCEPTED with
2. 1. Data: returns a request ID
   2. Error: <empty>

If the request is NOT accepted:

1. 400 BAD_REQUEST or
2. 413 PAYLOAD_TOO_LARGE or
3. 401 UNAUTHORIZED

with

4. 1. Data: <empty>
   2. Error: populated with error message(s)

   The above class of responses should not be automatically retried, as they require corrective action.

   
   

   Other HTTP response codes should be retried per their specifications:

4. 4XX or 5XX; see “API References” and “HTTP Response Codes” sections in

Data integrity considerations

Please consider the following:

✱ NOTE: The solution ignores extraneous data fields in records that do not conform to the record specification. Additionally, as per the OpenAPI format specification, field names are case-sensitive and data MAY be ignored or dropped if it does not conform to the field name casing specified in the OpenAPI YAML specification. Geotab recommends all JSON payload field names use PascalCase naming.




✱ NOTE: Optional fields with null values or that are missing entirely are set to a default value, following the Default values of C# types. Depending on your device data reporting workflows and the record type, it may be required for an integration to explicitly include fields and set their values to avoid misinterpretation.

Size and frequency of bulk requests

The frequency and size of requests can have competing priorities. Consider that typically for any ingestion system, smaller bulk requests will have lower throughput performance, and larger bulk requests will have higher throughput. This is a matter of maximizing payload efficiency per request (TCP & HTTP request overhead) and minimizing latency.

We recommend sending requests with between 200 - 500 records on average for optimal performance in steady state. Typically, the maximum number of records threshold for your integration could be up to 1000 records per request. Larger requests are also acceptable should your system experience a backlog or burst and needs to recover.

There is no "one size fits all". Integrators need to determine what works best for their systems; they will typically employ logic for both “number of records per request” and “time since last request”. Practically, this means bulking requests up to a certain maximum size, or up until a maximum duration since the last request. The example flowchart above uses sane default starting values.




✱ NOTE: If you have experienced a technical issue and need to send larger than normal bulk processing requests to recover (beyond your current active production workload rates), we request that you schedule this during off-peak hours (relative to the timezone of the region). Please contact Geotab Support before sending extremely large requests for an extended period of time that are out of the ordinary. Typically, a minimum of two business days advance notice is necessary.

Curl examples

Generic Request

curl --request POST \

--url https://dig.geotab.com:443/records \

--header 'Authorization: Bearer eyJh--Bearer-token-clipped-for-brevity-TCsIA' \

--header 'content-type: application/json' \

--data '[{RECORD_1}, {RECORD_2}, {...}, ...]'




Success Request & Response

5. 1. RECORD_X is a GenericStatusRecord
   2. Successful POST example returns 202 ACCEPTED
   
   

   {"DateTime":"2020-01-01T01:00:00Z","SerialNo":"xxxxxxx","Type":"GenericStatusRecord","Code":X,"Value":X}

   
   

   Response:

   {

   "Error": [],

   "Data": "6184f7b2-clipped-for-brevity-fa93c1e53a5d"

   }

   Rejected Requests & Responses

   Invalid DateTime:

   3. RECORD_X is a VinRecord with an unspecified time zone
   4. Non-successful POST is rejected as 400 BAD_REQUEST
   
   

   {"DateTime":"2020-01-01T01:00:00","SerialNo":"xxxxxxx","Type":"VinRecord","VehicleIdentificationNumber":X}

   
   

   Response:

   {

   "Error": [

   "Bad Request: one or more validation errors occurred",

   "Encountered one or more records having their DateTime field with an unspecified time zone"

   ],

   "Data": null

   }

   
   

   Invalid JSON:

1. RECORD_X is a BaseRecord with invalid JSON (wrong quotes)
2. Non-successful POST is rejected as 400 BAD_REQUEST



{"DateTime":’2022-10-13T16:02:22.2216677Z’,"SerialNo":’xxxxxxx’,"Type":’None’}




Response:

{

"Error": [

"Bad Request: one or more validation errors occurred",

"Encountered an error in the JSON payload, verify it adheres to RFC8259"

],

"Data": null

}




✱ NOTE: For errors related to an invalid date, see the “Working with Dates” section in the Data Intake Gateway Guide.





Invalid records workflow

DIG provides the /invalidrecords endpoint to query for invalid records that may have been marked as invalid or rejected for various reasons. Systems that integrate with DIG should periodically query this endpoint to inspect, identify, and correct any problems. Records that are corrected can then be resent to the DIG /records endpoint.



Figure 3 - Invalid Record API Endpoint Workflow

Invalid records accumulate within DIG as they are processed and identified through the /records workflow. Invalid records are not sent to MyGeotab; they become available for your account to query and inspect.




✱ NOTE: Data records accepted by DIG that are marked as invalid will be available at this endpoint as quickly as possible. See the Data Processing Latency section in the Appendix for additional detail and considerations.




A request to /invalidrecords gets a data payload that may contain invalid records if they exist. The workflow and usage of the endpoint supports some basic parameters for controlling the maximum number of results per response and the next result key for walking results (aka pagination) and incremental calls over time. The response also includes metadata that is useful for the integrator to implement this workflow with the system.

The array of returned invalid records is a consistent structure that includes their type, original field values, the reason why they are considered invalid, along with other metadata.

DIG will expire invalid records after a period of time. To avoid data loss, it is expected that your integration queries this endpoint periodically to save them for manual inspection.

To recap: The integrator must periodically query /invalidrecords to investigate erroneous data, correct problems, and resend the data.



Figure 4 - Invalid Records Endpoint Sample Implementation Workflow




Invalid records endpoint

Endpoint

[https://dig.geotab.com:443/invalidrecords] GET

Params

NextResultKey: set to 0 initially; subsequent requests should set this to the previous request’s NextResultKey response value;.

NumberOfResults: the maximum number of invalid records to return for this request

Returns

If the request is accepted:

1. 200 OK with
2. 1. Data (for successful responses):
   2. 1. NextResultKey: use this for the next request’s NextResultKey parameter.
      2. CurrentResultKey: the value of NextResultKey that was specified in the current request.
      3. TotalNumberOfInvalidRecords: the total number of unexpired invalid records for your account.
      4. CurrentNumberOfInvalidRecords: the number of invalid records returned in this response.
      5. InvalidRecords: [array] of invalid records (if any) with metadata.
   2. Error (for unsuccessful responses, populated with error message)

If the request is NOT accepted:

1. 400 BAD_REQUEST or
2. 401 UNAUTHORIZED

with

1. Data: <empty>
2. Error: populated with error message(s)

The above class of responses should not be automatically retried, as they require corrective action.




Other HTTP response codes should be retried per their specifications:

Reasons for records to be marked as invalid and expected actions

Geotab recommends that all invalid records provided by DIG are stored for manual inspection and case-by-case assessment before resending or discarding. There are many reasons why a record may be marked as invalid. The system assesses validity in multiple stages. The most common reasons include, but are not limited to:

1. Invalid Serial NumberSending a record for a device that is invalid, unprovisioned, not active, or not owned by your account.
2. Missing Mandatory Data FieldSome record fields are mandatory.(contextual to the record type itself; does not apply to SerialNo, DateTime, and Type fields)
3. Bad or Corrupt DataSome record types have data value validations and restrictions per industry specification, DIG API specification, or some other requirement; additionally, some data types require a specific encoding.

All scenarios must have some action performed and a decision by the integrator whether to retry the data. Before retrying, actions may include:

1. Finishing administrative tasks (billing, device provisioning)
2. Correcting corrupt data from devices at the source
3. Fixing issues in the integration implementation’s data processing
4. Etc.

A final decision to discard the data may also be the outcome, and will depend on your business requirements.

Explanation for NextResultKey and CurrentResultKey

There is a relationship between:

1. NextResultKey in a request and CurrentResultKey in the response.
2. NextResultKey from a response and NextResultKey in the subsequent request.

NextResultKey can be thought of as a pagination key that will unpredictably increase monotonically, and must be tracked and maintained by the integration, as it is chained from request to request. If this tracker is lost, the integration will need to start over with 0 and would be forced to reprocess duplicate invalid records. Note that this value is not guaranteed to always increase.

CurrentResultKey in a response is simply metadata equal to the value of NextResultKey from the associated request.

The following simplified sequence diagram can help you understand these relationships:

Figure 5 - Invalid Records ResultKey Explanation

Explanation for CurrentNumberOfInvalidRecords and TotalNumberOfInvalidRecords

Both of these response fields are simple counters for the number of invalid records respectively.

CurrentNumberOfInvalidRecords will always be a subset of the TotalNumberOfInvalidRecords, and will be less than or equal to the request parameter NumberOfResults. While CurrentNumberOfInvalidRecords is >=0, there may be more invalid records returned in subsequent requests to /invalidrecords.

TotalNumberOfInvalidRecords will increase or decrease over time, but always reflects DIG’s state of total invalid records for your account at the time the request was made. It increases when there are new invalid records processed and it decreases when invalid records expire.

Curl example

Request

curl --request GET \

--url 'https://dig.geotab.com:443/invalidrecords?NextResultKey=0&NumberOfResults=100' \

--header 'Authorization: Bearer eyJh--Bearer-token-clipped-for-brevity-TCsIA'

Response

{

"Error": [],

"Data": {

"NextResultKey": 10514,

"CurrentResultKey": 0,

"TotalNumberOfInvalidRecords": 200,

"CurrentNumberOfInvalidRecords": 100,

"InvalidRecords": [{INVALID_RECORD_1}, {INVALID_RECORD_2}, ..., {INVALID_RECORD_100}]

}

}

Where: INVALID_RECORD_X (in example) is an invalid record denoting metadata as well as the original record content ingressed to DIG /records endpoint. The metadata will include a description for why this record was not valid:

{"TimeStamp":"2020-01-01T01:00:00Z","Cause":"some reason","UserId":"<>", “BaseRecord”: {...}}

Example of a single invalid record:

{

"Error": [],

"Data": {

...,

"InvalidRecords": [

{

"BaseRecord": {

"Code": 255,

"Value": 0,

"DateTime": "2021-01-01T05:00:00Z",

"SerialNo": "dummy",

"Type": "GenericStatusRecord"

},

"Cause": "The record was not accepted because the serial number is unknown or has a formatting issue. This may be an unprovisioned device or we are experiencing a delay to provision.",

"TimeStamp": "2021-07-16T15:00:14.3370137Z",

"UserId": "<snip>"

}

]

}

}

Invalid record expiration & usage expectations

The set of invalid records for your account becomes unavailable once they expire. Typically, invalid records should be available for approximately 48 hours from the time they were identified and processed as invalid records. This expiration is tracked per invalid record.

Workflow walkthrough

The following section displays a more in-depth example of several /invalidrecord requests, demonstrating what we have learned.

At a high level, we will work through the following:

1. As an initial state, no invalid records.
2. Then there are 8 invalid records; query for 5 (get 5).
3. No new invalid records, but still more to get; query for 5 (get 3).
4. We have queried for all the invalid records and there are no new ones; query again for 5 (get 0).
5. Workflow takes a break querying for 1 hour.
6. Resume querying, still no new invalid records.
7. Workflow continues querying for several days, still no new invalid records.
8. Invalid records expire.

Best practices & usage expectations

/authentication/*

Review how your integrations will implement the authentication workflows by understanding the API Authentication Workflow Service Account Best Practices as well as the API Authentication and Token Management Implementation Strategies.

Minimizing redundant & duplicate data into /records

Some devices may output “noisy” (repetitive) data even when there is no change. To scale your integration and achieve optimal performance with the Geotab backend, we recommend minimizing duplicate records from your custom telematics devices.

Records entering the Data Intake Gateway should be for new or changed data from previous record states, with some exceptions such as infrequent status record updates that may necessitate periodic (low frequency) updates.

Having a minimal and lean set of data (and still sufficiently granular for your business needs) is optimal.

Separation of the /records Pipeline & /invalidrecords Processes

Geotab recommends for simplicity that integrators consider the primary inline pipeline path of sending bulk records to DIG to be a separate process from the secondary system that queries periodically for invalid records.

This allows /records requests to “fire and forget” with high performance and scale using Fast ACK, when the response is 202 ACCEPTED. No further auditing or tracking from this portion of the integration would necessarily be required.

A separate secondary process can query /invalidrecords periodically to re-ingest records that were marked as invalid. These records should be saved to storage for human assessment and correction before resending those records for re-processing.

Rate limits

The core DIG API workflow endpoints outlined in this document are authenticated endpoints (protected by JWT bearer token from the Authentication and Token Management workflows), so there are no rate limits for the /records and /invalidrecords endpoints at this time.

Limitations

This section describes any limitations related to this service.

It is currently not possible to insert (import) historical data older than 365 days.

Appendix

Data processing latency for valid and invalid records

If a request to the /records endpoint is accepted (202), your high-performance integration can assume the data was accepted. These records will be processed internally and delivered to the appropriate MyGeotab database(s) as quickly as possible, or be subsequently marked as invalid.

The determination for whether a record is valid or invalid is made as quickly as possible. Due to multiple variables within the system and multiple reasons for records to be marked as invalid, Geotab cannot provide a hard time guarantee.

The /invalidrecords endpoint can be periodically queried to determine if any records were marked as invalid and need attention. Invalid records will be detected, processed, and made available to this endpoint as quickly as possible.

Customers can rest assured that data is processed on high-performance infrastructure and becomes available promptly with minimal latency.

Custom telematics device records

This section outlines any supporting information for custom telematics device records that can be helpful beyond what the OpenAPI yaml specification supplies.

Please adhere to this specification to align with record fields and their object models. To make requests against DIG, this document provides the Records Workflow section above describing how to generally construct API requests to send records. The sections and samples below do not duplicate the record model specification nor iterate the matrix of possible sample requests.

The records workflow and the API specification should provide sufficient documentation to implement a custom telematics device integration with Geotab.

The sections below are additional documentation that goes beyond the specification and DIG API invocation. Where needed for specific record types, we will highlight scenarios and correlate with the MyGeotab platform.

Acceleration record

A record that contains accelerometer data. Axis acceleration units are in milli-g (1000mG = 1G of gravitational force). For more information about g-force, refer to our What is g-force blog article.

Sample Acceleration Record

BinaryRecord: Storage of arbitrary binary/string data

Consider the following sample of arbitrary data that an integrator would like to store in the Geotab ecosystem that falls outside the supported MyGeotab record data schemas. This can be unique per custom telematics device for a particular time and date.

This flexible binary record is useful as well for prototyping solutions integration systems to verify how data can leave and be retrieved.

Data sample

Main Battery=12.89;Backup Battery=0;Box Serial Number=######;Firmware Version=0001_tr00.SE7.G30.NN(Aug 1 2020 14:12:47)#Dis WxC:xC000002 GPS:S5-P4.0;Waypoints Usage=;SIM=XXXXXXX;Cell=;OTAB=N;

Data Base64 encoded

TWFpbiBCYXR0ZXJ5PTEyLjg5O0JhY2t1cCBCYXR0ZXJ5PTA7Qm94IFNlcmlhbCBOdW1iZXI9IyMjIyMjO0Zpcm13YXJlIFZlcnNpb249MDAwMV90cjAwLlNFNy5HMzAuTk4oQXVnICAxIDIwMjAgMTQ6MTI6NDcpI0RpcyBXeEM6eEMwMDAwMDIgR1BTOlM1LVA0LjA7V2F5cG9pbnRzIFVzYWdlPTtTSU09WFhYWFhYWDtDZWxsPTtPVEFCPU47Cg==

DIG API request

This data can be sent using the BinaryRecord record type; set the “Data” field to the arbitrary data, Base64 encoded.

Retrieving the binary data

This data can be retrieved from MyGeotab using the MyGeotab SDK using GetFeed as shown below:

MyGeotab API Request

MyGeotab API Response

DriverChangeRecord: Driver Assignment

This functionality is equivalent to the Driver Assignment feature in MyGeotab and similar to the blogged NFC Driver ID scenario.

For custom telematics device integrations, we can consider the goal of sending a “Driver Assignment” event from a key swipe or other custom hardware or driver operation.

DIG API Request

Integrators can use the DriverChangeRecord for driver assignment.

Supported Key Types

MyGeotab UI Outcomes

The following table outlines outcomes before with no driver assigned.

The following table outlines outcomes after with a new driver assigned.

Fault records

There are different defined protocols by the Society of Automotive Engineers SAE group for fault codes depending on the age and type of asset. The following record types enable the ingestion of fault data records into the Geotab platform through DIG.

There are different input criteria based on the protocol utilized by the connected vehicle, so integrators should utilize as appropriate. Faults captured through these record types are visible by navigating to Engine Faults on MyGeotab’s menu.

Each record model and its properties are denoted in the DIG API on Github: https://github.com/Geotab/data-intake-gateway.

The following outlines the context and specific explanations:

1. A record with isIgnitionOn=false and isGpsValid=true.
2. A record with (a) identical GPS coordinates as previous with (b) a timestamp difference >200 seconds.
3. A record with isIgnitionOn=false and isGpsValid=false to force the live Map to interpret a stop.

The custom telematics device will indicate that it is moving by sending a record with isIgnitionOn=true and isGpsValid=true.

On the Geotab live Map, the Iconography for vehicles states are documented in the Map Icons section.

StatusRecord and GenericStatusRecord

Both StatusRecord and GenericStatusRecord allow status data to be sent to MyGeotab. The difference between the two is that StatusRecords map to legacy third-party source diagnostics, whereas GenericStatusRecords map to telematics device source diagnostics.

1. To find the diagnostic code for GenericStatusRecords, refer to the spreadsheet.
2. To find the diagnostic code for StatusRecords, select Diagnostics from the Engine and Device tab in MyGeotab. Then, choose Third-party for the source filter.

In order for an integration to leverage status records, we recommend reviewing the MyGeotab diagnostics available, comparing and mapping to your system, experimenting/testing, and validating the expected behavior from the source system and ultimately how this is exposed within MyGeotab for your requirements.

See MyGeotab CTD Diagnostics – Status and Fault Codes explained for information on obtaining the correct code value.

MyGeotab CTD Diagnostics – Status and Fault Codes explained

Status data and faults are both considered diagnostics within Geotab systems, and all diagnostics have a unique code value that identifies them. Refer to How to find the Telematics Diagnostic name from the codes | Geotab Knowledge Article for an overview of how the entire list of MyGeotab diagnostics can be exported in a report.

✱ NOTE: Use source Telematics Device.

Use the search bar to filter for diagnostics by their name. Alternatively, lists these items.

As it pertains to custom telematics devices, the code value is applicable for the following.