2018年6月11日 星期一

NXLog 如何調整Log message內容再送出給Log Server

利用 NXLog 將 Windows DNS Log 送至 OSSIM,卻發現 OSSIM 一直無法將 raw log 進行 parsing,經檢查 OSSIM plugins 設定檔 windns-nxlog.cfg 的 regex 後發現,應該是原本 log 中的中文 (上午|下午) 送到 OSSIM 後無法被辨識。

在嘗試調整 regular expression 無果後,決定從 NXLog 著手,在 NXLog 送出前,先將 (上午|下午) 調整成 (AM|PM),再將 log 送出。

然後,又發現在 SIEM 查詢時,下午的 windns log 都沒有出現,檢查發現在 OSSIM 的 plugin - windns-nxlog.cfg 中,其 date 欄位有先經過 normalize_date() function 處理,而 2018/6/8 06:17:49 PM 這個格式不符其支援的格式,故須修改成 6/8/2018 06:17:49 PM 這種格式。

以下是中文版與英文版所產出之 DNS debugging log。
----------------------------------------------------------------------------------------------------------------------
Windows DNS log from Windows 2008 R2 CHT:

2018/4/16 上午 06:17:49 0B38 PACKET  0000000002134EF0 UDP Rcv 192.168.3.252   3780   Q [0001   D   NOERROR] A      (6)amazon(3)com(0)

2018/4/16 上午 06:17:49 0B38 PACKET  0000000002EA9D30 UDP Snd 168.95.1.1      0fc0   Q [0001   D   NOERROR] A      (6)amazon(3)com(0)

2018/4/16 上午 06:17:49 0B38 PACKET  000000000212DAB0 UDP Rcv 168.95.1.1      0fc0 R Q [8081   DR  NOERROR] A      (6)amazon(3)com(0)

2018/4/16 上午 06:17:49 0B38 PACKET  0000000002134EF0 UDP Snd 192.168.3.252   3780 R Q [8081   DR  NOERROR] A      (6)amazon(3)com(0)
-----------------------------------------------------------------------------------------------------------------------

Windows DNS log from Windows 2008 R2 Eng:

2/19/2015 10:03:57 PM 2AE8 PACKET  00000005CF374F80 UDP Rcv 192.168.0.42    fdd7   Q [0001   D   NOERROR] A      (9)imap-mail(7)outlook(3)com(0)

2/19/2015 10:03:57 PM 2AE8 PACKET  00000005CB426930 UDP Snd 10.255.176.137  0c4c   Q [0001   D   NOERROR] A      (9)imap-mail(7)outlook(3)com(0)

2/19/2015 10:03:57 PM 2AE8 PACKET  00000005D03B4CE0 UDP Rcv 10.255.176.137  0c4c R Q [8081   DR  NOERROR] A      (9)imap-mail(7)outlook(3)com(0)

2/19/2015 10:03:57 PM 2AE8 PACKET  00000005D03B4CE0 UDP Snd 192.168.1.42    fdd7 R Q [8081   DR  NOERROR] A      (9)imap-mail(7)outlook(3)com(0)
-----------------------------------------------------------------------------------------------------------------------


調整 nxlog.conf 設定如下:
======================================================================
< extension json >
    Module      xm_json
< /extension >

## Custom CSV format for the windns-nxlog AlienVault plugin.
< extension transform_alienvault_csv_dns >
    Module          xm_csv
    Fields     $Hostname, $SourceName, $Message
    FieldTypes      string, string, string
    Delimiter       ,
< /extension >

< input dns_logs >
    Module    im_file
    File      "C:\\temp\\DNSLog\\debug.log"
    SavePos  TRUE
    InputType LineBased

    #根據 OSSIM 的 Date Format supported by normalize_date() function,調整 Date Format
    #Convert the Chinese characters into English characters (上午 to AM、下午 to PM)
    #Change the datetime format from "yyyy/mm/dd (上午|下午) hh:mm:ss" to "mm/dd/yyyy hh:mm:ss (AM|PM)"

    Exec if $raw_event =~ /^(\d+)\/(\d+\/\d+)\s(上午\s)(\d+\:\d+\:\d+\s)(.+)/ $raw_event = $2 + '/' + $1 + ' ' + $4 + 'AM ' + $5;
    Exec if $raw_event =~ /^(\d+)\/(\d+\/\d+)\s(下午\s)(\d+\:\d+\:\d+\s)(.+)/ $raw_event = $2 + '/' + $1 + ' ' + $4 + 'PM ' + $5;

    Exec if $raw_event !~ /^\d/ drop();\
    else\
    {\
         $Message = $raw_event;\
         $SourceName = "DNS";\
         $raw_event = to_json();\
    }
< /input >

< output out_alienvault_dns_nxlog >
    Module          om_udp
    Host            192.168.1.2
    Port            514
   
    Exec            if not defined $Message { drop(); }

## Replace newlines, tabs and carriage returns with blanks:
    Exec            $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
    
## Ensure that commonly undefined values are set:
    Exec            if not defined $AccountName { $AccountName = "-"; }
    Exec            if not defined $AccountType { $AccountType = "-"; }
    Exec            if not defined $Domain { $Domain = "-"; }

## Ensure we send in the proper format:
    Exec     $Hostname = hostname_fqdn();
    Exec            transform_alienvault_csv_dns->to_csv(); $raw_event = $Hostname + ' DNS-NXLOG: ' + $raw_event;
< /output >

## Route for dns nxlog logs:
< route route_dns_nxlog >
    Path        DNS_Logs => out_alienvault_dns_nxlog
< /route >

======================================================================


OSSIM中查看rawlog,原本輸出為

Jun  8 00:07:48 DNS.test.corp DNS-NXLOG: "DNS.test.corp","DNS","2018/6/8 ?W?? 12:09:02 0B38 PACKET  00000000029DDCE0 UDP Rcv 192.168.3.31    7f19   Q [0001   D   NOERROR] A      (9)resolver1(5)ofind(6)ctmail(3)com(0)"
Jun  8 00:07:48 DNS.test.corp DNS-NXLOG: "DNS.test.corp","DNS","2018/6/8 ?W?? 12:09:02 0B38 PACKET  00000000029DDCE0 UDP Snd 192.168.3.31    7f19 R Q [8081   DR  NOERROR] A      (9)resolver1(5)ofind(6)ctmail(3)com(0)"


進行 Pre-parsing 後變為

Jun  8 15:43:09 DNS.test.corp DNS-NXLOG: "DNS.test.corp","DNS","6/8/2018 03:42:58 PM 0B34 PACKET  00000000020B9D10 UDP Rcv 192.168.3.31    afdd   Q [0001   D   NOERROR] A      (13)gmail-smtp-in(1)l(6)google(3)com(0)"
Jun  8 15:43:09 DNS.test.corp DNS-NXLOG: "DNS.test.corp","DNS","6/8/2018 03:42:58 PM 0B34 PACKET  00000000020B9D10 UDP Snd 192.168.3.31    afdd R Q [8081   DR  NOERROR] A      (13)gmail-smtp-in(1)l(6)google(3)com(0)"




再回到 OSSIM web介面,ANALYSIS \ SECURITY EVENTS (SIEM) 下,DATA SOURCE 下拉選項出現 windns,表示 DNS Log 已成功寫入資料庫了!!


備註:

OSSIM Date Format supported  by normalize_date() function:
https://www.alienvault.com/documentation/usm-appliance/plugin-management/supported-formats-normalize_date.htm


技術參考來源: http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html

Rewriting and modifying messages
There are many ways to modify log messages. A simple method which does not always work is to modify the $raw_event field (in case of syslog) without parsing the message. This can be done with regular expressions using capturing, for example:

if $raw_event =~ /^(aaaa)(replaceME)(.+)/ $raw_event = $1 + 'replaceMENT' + $3;

The more complex method is to parse the message into fields, modify some fields and finally reconstruct the message from the fields. The conditional rewrite of the syslog facility example shows such a syslog message modification method.