Messages

klev is a messaging platform - you send (or publish) and receive (or consume) messages. When you publish a message to klev, it stores it internally. Later on (or immediately) you consume the message. Consuming the message, does not remove it from klev, and you can consume the same message multiple times

A message in klev has of 4 parts:

  • Key - the key is user provided, specific to your application. In some cases, klev uses the message key to deduplicate messages, see tables
  • Value - the value is also user provided, specific to your application.
  • Offset - when publishing, klev assigns an ever increasing number to each message, the offset. Later on, when consuming, you can specify which offset you want to consume, and klev will return the complete message for this offset.
  • Time - this is the time when the message was stored.

Logs

Messages in klev are organized in logs. Logs can be used for many things - for example separating messages by type, by source or client, or anything really. Logs also specify how messages are stored and for how long.

klev supports two types of logs: streams and tables. By default all messages are stored indefinitely. By using streams or tables you define how old messages get deleted.

Streams

Streams are meant for documenting a set of changes, which are temporal in nature. A stream is the default log type on klev. It can be configured to delete messages based on their age or how much data you have in the log. The process of deleting old messages is automatic.

Tables

Tables are a special kind of log, used for tracking a set of changes on a set of keys. With tables klev assigns special meaning to the keys - they are used when deleting old messages. When klev sees multiple messages with the same key, it assures to keep the last message, if configured to delete (or compact) the old ones. In addition, a message without a value is treated as a delete for this key, which klev can be configured to delete (or expire) too.

Offsets

Offsets enable you to store your offset in the log. You can use this to later resume consuming from a certain offset (for example if your consumer dies) or mark important positions in the log.

Tokens

Tokens are used for programmatic access to klev. You provision your first token after registering for klev. Further tokens can be derived from the API.

ACL

When creating a token you could optionally add an ACL to restrict what this token can do. A token without ACL has full access to the API.

ACL items come in the form of subject[:action[:object]] where:

  • logs - controls access to logs. Supports the following actions:
    • list - allows listing logs (including find). Returns only logs that are also allowed by get
    • create - allows creating new logs
    • get[:log_id] - allows getting logs. If log_id is set, ACL only allows getting this log
    • delete[:log_id] - allows deleting logs. If log_id is set, ACL only allows deleting this log
  • messages - controls access to the messages on logs. Supports the following actions:
    • publish[:log_id] - allows publishing messages. If log_id is set, ACL only allows publishing on this log
    • consume[:log_id] - allows consuming messages. If log_id is set, ACL only allows consuming from this log
  • offsets - controls access to offsets. Supports the following actions:
    • list - allows listing offsets (including find). Returns only offsets that are also allowed by get
    • create - allows creating new offsets
    • get[:offset_id] - allows getting offsets. If offset_id is set, ACL only allows getting this offset
    • update[:offset_id] - allows updating offsets. If offset_id is set, ACL only allows updating this offset
    • delete[:offset_id] - allows deleting offsets. If offset_id is set, ACL only allows deleting this offset
  • tokens - controls access to tokens. Supports the following actions:
    • list - allows listing tokens (including find). Returns only tokens that are also allowed by get
    • create - allows creating new tokens
    • get[:token_id] - allows getting tokens. If token_id is set, ACL only allows getting this token
    • delete[:token_id] - allows deleting tokens. If token_id is set, ACL only allows deleting this token
  • ingress_webhooks - controls access to ingress webhooks. Supports the following actions:
    • list - allows listing ingress webhooks (including find). Returns only ingress webhooks that are also allowed by get
    • create - allows creating new ingress webhooks
    • get[:ingress_webhook_id] - allows getting ingress webhooks. If ingress_webhook_id is set, ACL only allows getting this ingress webhook
    • rotate[:ingress_webhook_id] - allows rotating ingress webhooks secrets. If ingress_webhook_id is set, ACL only allows rotating secret for this ingress webhook
    • delete[:ingress_webhook_id] - allows deleting ingress webhooks. If ingress_webhook_id is set, ACL only allows deleting this ingress webhook
  • egress_webhooks - controls access to egress webhooks. Supports the following actions:
    • list - allows listing egress webhooks (including find). Returns only egress webhooks that are also allowed by get
    • create - allows creating new egress webhooks
    • get[:egress_webhook_id] - allows getting egress webhooks. If egress_webhook_id is set, ACL only allows getting this egress webhook
    • rotate[:egress_webhook_id] - allows rotating egress webhooks secrets. If egress_webhook_id is set, ACL only allows rotating secret for this egress webhook
    • status[:egress_webhook_id] - allows getting egress webhooks status. If egress_webhook_id is set, ACL only allows getting status for this egress webhook
    • delete[:egress_webhook_id] - allows deleting egress webhooks. If egress_webhook_id is set, ACL only allows deleting this egress webhook

Webhooks

Webhooks are a way for notifying external applications for something that happened in the application (like new payment or new slack message). klev can be used to both receive and originate these events targeting a concrete log in your account.

Ingress webhooks

Use ingress webhooks to receive notifications from other applications (such as Stripe or Slack). klev will verify the event and then publish a message to the target log setup in your webhook.

Currently, we support the following sources:
  • Github
  • Slack
  • Stripe
  • Klev (messages)
  • Klev (keys)
  • Klev (values)

Egress webhooks

Use egress webhooks to send notifications to other applications about messages published on a log. You can choose if you want the whole message delivered, only its key or its value.

We recommended you validate your webhook signature. To do that for klev:

  1. Start by checking the content type, by extracting the Content-Type http header. It has to match application/json for message payloads or application/octet-stream when key/value payloads are used.
  2. Next, extract the X-Klev-Signature http header. If it is missing or its value is empty, the request was not properly signed
  3. The format for X-Klev-Signature is key=value[;key=value]+, which is a representation of a map. A good way to parse this is by:
    • Splitting by ;, which will give you a single entry key=value
    • Split each entry by = to extract the key and its value
  4. Two keys in X-Klev-Signature have special meaning:
    • t which represents the timestamp at which this request was signed
    • v1 which represents hex encoded value of the signature
    This means that usually the value if this header is in the form of X-Klev-Signature: t=timestamp;v1=hash
  5. The value of t key represents unix timestamp. To prevent request replay, check if this is within a certain predefined interval from now (5m is a good default for this)
  6. Calculate the HMAC value for {timestamp}.{request.body} using your secret. Hex encode the result HMAC and this becomes the request signature
  7. Compare the request signature against every v1 value in the header. If any of them matches, the request was properly signed. If it doesn't match any, discard the request

To see the above algorithm is action (or use in your go project) refer to go api validator

Filters

Use filters to send a subset of the messages from one log to another. They are implemented via CEL expressions. In addition to the standard CEL environment you can use the following:

  • key - a variable, the key of the message
  • value - a variable, the value of the message
  • offset - a variable, the offset of the message
  • time - a variable, the time of the message
  • toJson - a function, allows you to parse the key or value of the message as a JSON and query it via CEL

Examples

Here are some examples of how to use klev