日本語

SINETStream User Guide

Configuration files

1. Overview
 1.1 Location of configuration files
 1.2 Priority of setting values
2. Common parameters
 2.1 Basic parameters
 2.2 API parameters
 2.3 Parameters for SSL / TLS
 2.4 Parameters for encryption
3. Messaging system-specific parameters
4. Notes
 4.1 Difference between Python API and Java API
 4.2 Unsupported

1. Overview

The parameters to the SINETStream API functions may also be written in the configuration file. By doing so you can make your source code more concise. The configuration file contains the service name and the parameters associated with the service. By specifying the service name in the API function, the parameters specified in the configuration file will be applied.

Publickey cryptography can be used to encrypt confidential information such as passwords and data encryption keys in configuration files.

Using Config Server

There are two ways to use a config server.

  1. download the configuration file from the config server’s Web UI.
  2. download the access info from the config server Web UI and retrieve the configuration file from the SINETStream API.

The first method is, instead of the user starting an editor and writing a configuration file the user simply downloads the configuration file prepared by the administrator.

The second method is to, the config server auth info is downloaded in advance, the configuration file is downloaded from the config server when SINETStream is executed.

The downloaded config server auth info are put in $HOME/.config/sinetstream/auth.json. (In Windows 10, C:\Users\{userXX}\.config\sinetstream\auth.json)

Encryption of secret info with public key cryptography

Limitations: supported public key cryptography is only RSA.

If you put your private key in PEM format (PKCS#1) in ~/.config/sinetstream/private_key.pem, it is referred to when decrypting the secret info in the configuration file.

Caution: the permissions for the private key file must be set so that others cannot read or write to it.

Secret info is the following format:

{parameter}: !sinetstream/encrypted {config value encrypted with pubkey}

The encryption method is SINETStream’s proprietary format.

The plaintext by decryption is handled as follows:

{parameter}: {decrypted config value}

Binary data and attachment

To specify binary data instead of ASCII for parameter, follow the YAML binary type format and specify !!binary followed by a Base64-encoded string. To specify a file as a parameter, use the same way, !!binary followed by a Base64-encoded string of the content.

{parameter}: !!binary {Base64-encoded contents}

Note:

The format of the configuration file

The format of the configuration file is YAML. A configuration file consists of a header section (header:) describing parameters of the configuration file itself and a configuration section (config:) describing services.

header:
  header description block
config:
  service description block 1
  service description block 2
  ...

If the header section doesn’t exist, it is considered to be in conventional format (only the configuration section).

The block describing the header is as follows:

  {header parameter 1}: {value 1}
  {header parameter 2}: {value 2}
  ...

Header section of the configuration file

Header parameters are as follows:

Configuration section of Configuration file

The configuration section consists of service description blocks (each block is described as an entry in a YAML associative array). The block that describes one service is as follows:

{Service Name}:
  type: {type of Message System}
  brokers:
    - {host Name 1}:{port number 1}
    - {host Name 2}:{port number 2}
  {other parameters 1}: {value 1}
  {other parameters 2}: {value 2}

type specifies the type of the messaging system. brokers specify the addresses of the brokers of the messaging system.

The other parameters and values depend on the messaging system. For instance, the following parameters can be specified.

1.1 Location of the configuration file

Configuration file is searched in the following order. Only the first file found will be used.

  1. .sinetstream_config.yml in the current directory
  2. $HOME/.config/sinetstream/config.yml
    • C:\Users\{userXX}\.Config\sinetstream\config.yml on Windows 10

Multiple configuration files cannot be cascaded; e.g., if parameter 1: value 1 is written in .sinetstream_config.yml, and parameter 2: value 2 is written in $HOME/.config/sinetstream/config.yml, the latter will be ignored.

1.2 Priority of setting values

If the parameters specified in the configuration file conflict with the parameters specified in the API functions, or if the common parameters conflict with the messaging system-specific parameters, the first value found in the following order is used.

  1. The messaging system-specific parameter value specified in the API function
  2. The common parameter value specified in the API function
  3. The messaging system-specific parameter value specified in the configuration file
  4. The common parameter value specified in the configuration file

2. Common parameters

Below are the common parameters that can be specified regardless of the messaging system.

2.1 Basic parameters

2.2 API parameters

Set the default value of the parameters of the SINETStream API functions. If a parameter is not specified in the API function, the value specified in the configuration file will be used as the default value.

value_serializer and value_deserializer take precedence over value_type.

If value_deserializer and value_type are specified and value_serializer is not, value_desirializer will be enabled for MessageReader and value_type will be enabled for MessageWriter.

Limitation of Python API: In SINETStream v1.*, value_serializer/value_deserializer can be specified only by the API function parameters and cannot be specified in the configuration file.

2.3 Parameters for SSL/TLS

While the names of the parameters for SSL/TLS differ depending on the messaging system, SINETStream provides a common parameter named tls for specifying them uniformly. SINETStream internally maps them to the messaging system-specific parameters.

The mapping may contain the following keys.

※ TrustStore, trustStoreType, trustStorePassword, keyStore, keyStoreType, keyStorePassword, keyfilePassword are for Java API ony. They cannot be specified with the Python API.

Setting Example

Below is an example of specifying a boolean value for the parameter tls. In this case, a default messaging system-specific value is used as the setting value.

service-tls-1:
  type: mqtt
  brokers: mqtt.example.org
  tls: true

Below is an example of specifying a mapping for the parameter tls.

service-tls-2:
  type: kafka
  brokers:
    - kafka-1:9092
  tls:
    ca_certs: /etc/sinetstream/ca.pem
    certfile: certs/client.pem
    keyfile: certs/client.key

Priority

Instead of using the tls parameter, it is also possible to directly specify messaging system-specific parameters.

If you specify both the tls parameter and a messaging system-specific parameter for a service, the first value found in the following order is used.

  1. The messaging system-specific parameters specified in the API function.
  2. The tls parameter specified in the API function.
  3. The messaging system-specific parameters specified in the configuration file.
  4. The tls parameter specified in the configuration file.

2.4 Parameters for encryption

SINETStream is capable of message content encryption at the frontend, which is independent from the communication encryption by the backend SSL/TLS. By encrypting the message encryption you can protect your information even if a malicious third party peeks into the message stored in the broker.

The mapping may contain the following keys.

Setting Example

Below is an example of setting crypto.

service-aes-1:
  type: kafka
  brokers:
    - kafka0.example.org:9092
  crypto:
    algorithm: AES
    key_length: 256
    mode: EAX
    key_derivation:
      algorithm: pbkdf2
      iteration: 10000
    password: secret-000

3. Messaging system-specific parameters

Parameters specific to the backend messaging system can be specified transparently.

4. Notes

4.1 Difference between the Python API and the Java API

The following parameters are only valid for the Python API. They will be ignored if specified in the Java API.

The following parameters are only valid for the Java API. They will be ignored even if specified in the Python API.

4.2 Unsupported

SINETStream v1.* does not support the following parameters.