{"id":561,"date":"2026-02-04T18:07:50","date_gmt":"2026-02-04T18:07:50","guid":{"rendered":"https:\/\/blog.lottabytes.com\/?p=561"},"modified":"2026-02-04T18:09:08","modified_gmt":"2026-02-04T18:09:08","slug":"parsing-timestamps-with-fluentd-and-opensearch","status":"publish","type":"post","link":"https:\/\/blog.lottabytes.com\/index.php\/2026\/02\/04\/parsing-timestamps-with-fluentd-and-opensearch\/","title":{"rendered":"Parsing Timestamps with Fluentd and Opensearch"},"content":{"rendered":"\n

It’s the seemingly simple things that seem to eat into meaningful productivity, and for me, I’ve struggled on multiple times with Fluentd’s parsers related to non-standard time formats. This post is an attempt to share what I’ve worked through in an attempt to help any others who might be faced with the same issues that I seem to encounter. In this situation, I’m attempting to forward logs from Suricata<\/a>‘s fast.log to an Opensearch <\/a> instance. The logs look like the below, and can be successfully parsed with the following regex.<\/p>\n\n\n\n

\"\"
Suricata Fast Alert Syntax<\/figcaption><\/figure>\n\n\n\n
^(?<timestamp>\\d{2}\\\/\\d{2}\\\/\\d{4}-\\d{2}:\\d{2}:\\d{2}\\.\\d{6})[\\s]*\\[(?<second_field>.*)\\][\\s]*\\[(?<rule_id>\\d*:\\d*:\\d*)]\\s*(?<rule_name>.*)\\[.*\\](?<fifth_field>.*)\\s*\\[Classification:\\s(?<class>.*)\\]\\s*\\[Priority:\\s(?<priority>\\d*)\\]\\s{(?<protocol>.*)}\\s(?<source_ip>[\\d]*\\.[\\d]*\\.[\\d]*\\.[\\d]*):(?<source_port>[\\d]*)\\s->\\s(?<dest_ip>[\\d]*\\.[\\d]*\\.[\\d]*\\.[\\d]*):(?<dest_port>[\\d]*)<\/code><\/pre>\n\n\n\n
In this use case I need the events to be ingested and formatted in a usable manner, with specific types such as IP address, integer, and time being correctly associated with each field.<\/p>\n\n\n\n
\"\"
Working OpenSearch Record Example<\/figcaption><\/figure>\n\n\n\n
If left to it’s own devices however, the Fluentd Agent will not forward the timestamp field, not will it assign any types other than Text. This prohibits functionality that requires integer parsing, or IP address mapping. To resolve this issue, I have manually created my Index inside of Opensearch, specifying all of the fields along with their desired types. This can be done via the UI under Index Management > Indexes > Create Index<\/em>, or via API call. A snippet of that JSON is shown below.<\/p>\n\n\n\n
\"\"
Snippet of OpenSearch Index Creation JSON<\/figcaption><\/figure>\n\n\n\n
During this effort I ran into a secondary issue where the timestamp would refuse to parse or ingest into OpenSearch.  A useful resource for troubleshooting, as well as defining your field types are the Supported Field Types – Date<\/em><\/a>, which outlines what formats can be defined to be accepted by the OpenSearch server.  If your timestamp is coming in as epoch milliseconds, but your server is expecting epoch seconds, this is how you correct that behavior as you see in the above screenshot.<\/p>\n\n\n\n
However, even after looking at length at the Fluentd Parser’s time documentation<\/a> I was still unable to ingest timestamps on events correctly.  As you can see below, I have the time_format<\/em> specified…<\/p>\n\n\n\n
\"\"
Fluentd Configuration that Doesn’t Work<\/figcaption><\/figure>\n\n\n\n
…and if you use the underlying Ruby strptime function<\/a> in a test environment<\/a> it works correctly against the timestamp and the date format.<\/p>\n\n\n\n
\"\"
Ruby Sandbox Shows Code Working in Theory<\/figcaption><\/figure>\n\n\n\n
Yet, a variety of issues arise, from errors like the below (with the above config), all resulting in either an incorrect time being associated, or no time at all.  I have even attempted to build out a custom date regex on the OpenSearch Index matching what I understand the Fluentd agent to be sending.  All to no avail…<\/p>\n\n\n\n
\"\"
Fluentd Ingestion Error Message<\/figcaption><\/figure>\n\n\n\n
At the end of this endeavor I discovered that a combination of things allowed for the proper ingestion of timestamps by a conversion of the timestamp value to epoch seconds.<\/p>\n\n\n\n