From Database to Events: Sql Listeners
We always try to improve your integration process to Herogi. Because once you have more and more events generated from your services, you can achieve a lot. Today we want to introduce you another open source integration method which we called sql listener.
Sql Listener
Sql Listener is a new open source event generation product which uses sql based databases (Mysql, Oracle, Mssql, Postgresql) as data sources. If you have an access to one of this databases, you can just install our listener to one of your servers to convert your table rows to events easily.
How Does It Work?
Once you start sql listener on a server which can access to your database, it will start running the sql query which you specified in your listener configuration on a timely basis. You can configure time interval so it can run every x seconds/minutes/hours/days. You can specify huge complex queries by joining sevaral tables to get more data as part of configuration.
There is more, you can consume only new records by using dynamic querying functionality. You can use date or id based columns to achieve that.
As other listeners, sql listener provides flexible configuration for data mapping. You can decide which columns from your result set will be part of event by using column information or index number of specific column.
Listeners are also self healing. If it’s killed by system, once you start again it’ll start processing from last record processed. You don’t need to worry about duplicade event sends.
Use Cases
If you have databases populated by other services without code changing, you can just start a sql listener to use data in your Herogi scenarios.
– Bulk customized messages to customers (newsletter, welcome email)
– Implementing data pipelines and transformations
– Listening new records and triggering some other actions on change
– CQRS, Event sourcing
Security
We care about your data and concern about security. You don’t need to worry about using sql listeners because you can install wherever you want to install, behind your firewalls, secure private networks. You don’t need to expose any important password or connection string. You can specify db user and query which has limited access. You have full control over how sql listener connects to your db and what query runs. Lastly, it’s opensource you can go check the source code and use it freely.
What’s Next?
At the moment, Herogi provides file and sql based listeners but we’re planning to have more like Jmx, Kafka listeners for message queues; Aws listeners for cloud support.
If you need different type of listener, you can contact with us. We’ll help you to create a specific listener for your product. Also you can create your own listeners using our open source code. We have extendible core implementation, you can easily tailor it for your specific case.
You can find more technical documentation on our Knowledge Base. You can also download latest release on Github.