We’re happy to announce today the release of Siren Federate version 28 which adds support for the latest major version of Elasticsearch – version 8.

This is an exciting news for many as Elasticsearch 8 bursts with super interesting features ranging from support for machine learning NLP models to vector searches as well as performance and scalability.

All of the above benefits will soon be available in a Q4 release of the Siren Platform supporting both ES 7.x and ES 8.x – but in the meanwhile Federate is already available to enhance your Elasticsearch 8 applications with extremely scalable join capabilities.

Plenty of enhancements:

Federate 28 specifically supports Elasticsearch 8.2.3 and includes the following enhancements and :

New features:

• Added support for runtime fields defined at query time.
• Added support to join on _id metadata field for semi-join queries. ( not yet for inner joins).

Enhancements:

• Upgraded to Elasticsearch 8.2.3
• Upgraded Arrow to version 7.0.0.
• Added various optimizations to the vectorized processing pipeline. Most noticeably:
  • When processing local transfer requests, data packets are directly serialized using arrowBuf.
  • Improved vectorization performance when projecting single-valued columns.
• Added index-based scan strategy for keyword field type.
• Implemented vectorization of distributed hash semi-join operations.
• Implemented vectorization of distributed inner join operations.
• Added support for a placeholder to represent the Elasticsearch configuration directory in the JDBC properties of a datasource configuration.

Bug fixes:

• Fixed an issue that could prevent queries containing joins to be executed after a redistribution of shards in a cluster.
• Fixed an issue that could cause a node to crash when handling an out of memory error during packet serialization.
• Fixed an issue that could prevent joins on keyword fields across virtual indices from being executed.
• Fixed fingerprinting of search request executions to match the result of explain requests.

Conclusion and what’s next?

Federate is Siren’s unique technology to extend Elasticsearch with real time ultra scalable correlation/join capabilities and is at the heard of the Siren platform and embedded as part of many solutions across sectors.

With this release, we make Federate available for Elasticsearch 8 – with the full Siren Platform compatibility soon to be release.

What’s more exciting is that the work has not only been addressing compatibility, but has focused on scalability and performance.

We’re glad to anticipate that the the next release of Federate will sport up to 100% performance increase on most queries as compared to previous versions thanks to a full vectorization of many operations.

This is the topic of the next blog post and benchmark we’ll be releasing so stay tuned.

Federate is available for download it here