Elixir Wizards S3E4: Justin Schneck and Frank Hunleth on Nerves and Performance How to Use Broadway in Your Elixir Application

Say NO! to montrous Avro schemas

From the very beginning, this library was heavily inspired by avro_turf simplicity and features. Now it’s time to say – Avrora moves one step closer to the feature set avro_turf provides.

The must-have feature inter-schema references comes to Avrora. Now you can split your huge schema into smaller pieces and glue them together via references.

What is a reference?

Reference is a canonical full name of a schema. Accordingly to Avrora name to location rules if you have schema under io/confluent/Message.avsc its full name (namespace + name) will be io.confluent.Message.

How do references work?

Technically Avro specification doesn’t support inter-schema references, only local-schema references. Because of this limitation, inter-schema references implemented via embedding referenced schema into the schema which contains reference and replacing all other references within this schema with local-references.

How to use references?

For example, you have a Messenger schema which contains references to the Message schema:

priv/schemas/io/confluent/Messenger.avsc

{
  "type": "record",
  "name": "Messenger",
  "namespace": "io.confluent",
  "fields": [
    {
      "name": "inbox",
      "type": {
        "type": "array",
        "items": "io.confluent.Message"
      }
    },
    {
      "name": "archive",
      "type": {
        "type": "array",
        "items": "io.confluent.Message"
      }
    }
  ]
}

priv/schemas/io/confluent/Message.avsc

{
  "type": "record",
  "name": "Message",
  "namespace": "io.confluent",
  "fields": [
    {
      "name": "text",
      "type": "string"
    }
  ]
}

Final compiled schema which will be stored and registered in the Confluent Schema Registry, will looks like this:

{
  "type": "record",
  "name": "Messenger",
  "namespace": "io.confluent",
  "fields": [
    {
      "name": "inbox",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "Message",
          "fields": [
            {
              "name": "text",
              "type": "string"
            }
          ]
        }
      }
    },
    {
      "name": "archive",
      "type": {
        "type": "array",
        "items": "Message"
      }
    }
  ]
}

⚠️ In case of avro_turf field archive will keep its canonical items type reference io.confluent.Message instead of local reference Message.