How to Surpress Microsoft Sentinel Log Ingestion

siwei • 2024-07-06 15:02 • 好物分享

You may want to filter your logs collected, or even log content, before the data is ingested into Microsoft Sentinel. For example, you may want to filter out logs that are irrelevant or unimportant to security operations, or you may want to remove unwanted details from log messages. Filtering message content may also be helpful when trying to drive down costs when working with Syslog, CEF, or Windows-based logs that have many irrelevant details.

Filter your logs using one of the following methods:

The Azure Monitor Agent. Supported on both Windows and Linux to ingest Windows security events. Filter the logs collected by configuring the agent to collect only specified events.
Logstash. Supports filtering message content, including making changes to the log messages. For more information, see Connect with Logstash.

More on this page: https://learn.microsoft.com/en-us/azure/sentinel/best-practices-data

Here are four different main scenarios :

On-premises Windows log collection
On-premises Linux log collection
Endpoint solutions
Office data
Cloud platform data

Check Your Sentinel Log Ingestion and Cost

Micosoft Sentinel - Workbooks - Template - Microsoft Sentinel Cost Summary

Log Type Ingestion Details on Each Table

Syslog Forwarder (AMA Installation)

AMA installation can be found in this post: https://blog.51sec.org/2024/04/azure-sentinel-onboarding-log-sources.html

Basically, you deploy your Linux machine (I used Ubuntu Azure VM). In many cases you're machine is on-premises, you need to onboard it using Azure Arc. Mind the pre-requisites (especially network connectivity).

Since you've Arc'd your on-premises collector machine, you can just enable syslog/CEF collection from Sentinel: Stream CEF logs to Microsoft Sentinel with the AMA connector

Sentinel's "Common Event Format (CEF) via AMA" -connector page explains pretty much all you need.

Overview how the data collection works:

Ultimately, all the logs at this point go into CommonSecurityLog table - which is Analytics tier table, there's nothing you can do. Here is an example for enabled local4 facility where my syslog/CEF will flow:

Obviously you need to enable syslog/CEF forwarding in your firewall(s) and make sure it's a) in correct format and b) communication works.

1 rsyslog configuration:

root@NETSEC-syslog:~# cd /etc/rsyslog.d/
root@NETSEC-syslog:/etc/rsyslog.d# ls
05-filterlogs.conf 10-azuremonitoragent-omfwd.conf 10-vcenter.conf.old 20-ufw.conf 50-default.conf
05-filterlogs.conf.20240628.bk 10-azuremonitoragent-omfwd.conf.20240628.bk 11-meraki.conf.old 21-cloudinit.conf 95-omsagent.conf

root@NETSEC-syslog:/etc/rsyslog.d# cat 05-filterlogs.conf
# Forward vCenter Logs to OMS
if ($inputname == 'udp_vcenter' or $inputname == 'tcp_vcenter') then @@127.0.0.1:23033;vcenter
& stop
# Forward Meraki Logs to OMS
if ($inputname == 'udp_meraki' or $inputname == 'tcp_meraki') then @@127.0.0.1:22033;meraki
& stop

# FIlter Fortinet logs
if ($rawmsg contains "traffic:forward accept") then stop
if ($rawmsg contains "traffic:local accept") then stop
if ($rawmsg contains "traffic:forward start") then stop
if ($rawmsg contains "traffic:forward close") then stop
if ($rawmsg contains "traffic:forward server-rst") then stop
if ($rawmsg contains "traffic:forward timeout") then stop
if ($rawmsg contains "traffic:forward client-rst") then stop
if ($rawmsg contains "traffic:local close") then stop
if ($rawmsg contains "traffic:forward ip-conn") then stop
if ($rawmsg contains "traffic:local server-rst") then stop
if ($rawmsg contains "traffic:local client-rst") then stop
if ($rawmsg contains "utm:ssl ssl-anomaly") then stop
if ($rawmsg contains "traffic:local timeout") then stop
if ($rawmsg contains "event:switch-controller") then stop
if ($rawmsg contains "event:connector") then stop
if ($rawmsg contains "event:ha") then stop
if ($rawmsg contains "event:security-rating") then stop
if ($rawmsg contains "event:vpn failure") then stop
if ($rawmsg contains "event:vpn success") then stop
if ($rawmsg contains "event:system clash") then stop
if ($rawmsg contains "event:vpn negotiate_error") then stop
if ($rawmsg contains "event:vpn esp_error") then stop
if ($rawmsg contains "event:system update") then stop
if ($rawmsg contains "event:switch-controller none") then stop
if ($rawmsg contains "event:user logout") then stop
if ($rawmsg contains "utm:anomaly anomaly") then stop
if ($rawmsg contains "event:switch-controller None") then stop
if ($rawmsg contains "event:switch-controller down") then stop
if ($rawmsg contains "event:switch-controller up") then stop
if ($rawmsg contains "event:system success") then stop
if ($rawmsg contains "event:system failed") then stop
if ($rawmsg contains "event:vpn dpd_failure") then stop
root@NETSEC-syslog:/etc/rsyslog.d#

2 Restart rsyslog service and check status

root@NETSEC-syslog:/etc/rsyslog.d# systemctl restart rsyslog
root@NETSEC-syslog:/etc/rsyslog.d# systemctl status rsyslog
● rsyslog.service - System Logging Service
 Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
 Active: active (running) since Sat 2024-07-06 14:45:15 UTC; 12s ago
TriggeredBy: ● syslog.socket
 Docs: man:rsyslogd(8)
 man:rsyslog.conf(5)
 https://www.rsyslog.com/doc/
 Main PID: 3418874 (rsyslogd)
 Tasks: 13 (limit: 9387)
 Memory: 7.0M
 CPU: 122ms
 CGroup: /system.slice/rsyslog.service
 └─3418874 /usr/sbin/rsyslogd -n -iNONE Jul 06 14:45:15 NETSEC-syslog systemd[1]: Starting System Logging Service...
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0]
Jul 06 14:45:15 NETSEC-syslog systemd[1]: Started System Logging Service.
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: rsyslogd's groupid changed to 113
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: rsyslogd's userid changed to 107
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="3418874" x-info="https://www.rsyslog.com"] start
root@NETSEC-syslog:/etc/rsyslog.d#

3 Azuremonitoragent-omfwd.conf

root@NETSEC-syslog:/etc/rsyslog.d# cat 10-azuremonitoragent-omfwd.conf  # Azure Monitor Agent configuration: forward logs to azuremonitoragent template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
# queue.workerThreads sets the maximum worker threads, it will scale back to 0 if there is no activity
# Forwarding all events through TCP port
*.* action(type="omfwd"
template="AMA_RSYSLOG_TraditionalForwardFormat"
queue.type="LinkedList"
queue.filename="omfwd-azuremonitoragent"
queue.maxFileSize="32m"
action.resumeRetryCount="-1"
action.resumeInterval="5"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
queue.size="25000"
queue.workerThreads="100"
queue.dequeueBatchSize="2048"
queue.saveonshutdown="on"
target="127.0.0.1" Port="28330" Protocol="tcp")
root@NETSEC-syslog:/etc/rsyslog.d#

4 Default rules for rsyslog

root@NETSEC-syslog:/etc/rsyslog.d# cat 50-default.conf
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info -/var/log/mail.info
#mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Some "catch-all" log files.
#
#*.=debug;/
# auth,authpriv.none;/
# news.none;mail.none -/var/log/debug
#*.=info;*.=notice;*.=warn;/
# auth,authpriv.none;/
# cron,daemon.none;/
# mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;/
# news.=crit;news.=err;news.=notice;/
# *.=debug;*.=info;/
# *.=notice;*.=warn /dev/tty8
root@NETSEC-syslog:/etc/rsyslog.d#

Rsyslog configuraiton

Commands:

rsyslogd -v
systemctl status rsyslog
systemctl start rsyslog
vi /etc/rsyslog.conf

root@NETSEC-syslog:~# cat /etc/rsyslog.conf
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf

#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
input(type="imudp" port="1514" name="udp_meraki")
input(type="imudp" port="2514" name="udp_vcenter")
input(type="imtcp" port="1514" name="tcp_meraki")
input(type="imtcp" port="2514" name="tcp_vcenter")
$template meraki,"%timestamp% %hostname% %msg%"
$template vcenter,"%timestamp% %hostname% %msg%"
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
root@NETSEC-syslog:~#

Use Data Collection Rules to Filter Logs

Azure Portal - Monitor - Data Collection Rules

Updating Data Collection Rule using Powershell

$ResourceId = "/subscriptions/d95-245-48c-a50-b1a7b9/resourceGroups/SEN-RG/providers/Microsoft.Insights/dataCollectionRules/dcr-dcwindowssecurityeventcommon" # Resource ID of the DCR to edit
$FilePath = "tmp1.dcr" # Store DCR content in this file
$DCR = Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method GET
$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File -FilePath $FilePath

$ResourceId = “/subscriptions//d95-245-48c-a50-b1a7b9/resourceGroups/SEN-RG/providers/Microsoft.Insights/dataCollectionRules/dcr-dcwindowssecurityeventcommon” # Resource ID of the DCR to edit
$FilePath = “tmp1.dcr” # Store DCR content in this file
$DCRContent = Get-Content $FilePath -Raw
Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method PUT -Payload $DCRContent

ResourceID

DCR Details in Export Template

Run in Cloud Shell

Fortinet Devices (FortiGate)

Local Facility: Changed to Local 7

Log Filtering on Syslog Forwarder Server:

Note: "stop" option does not work, but ~ works.

Following configuration file is to stop priority 3 events coming into Sentinel

vi /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf

# Azure Monitor Agent configuration: forward logs to azuremonitoragent
# Filter Fortinet logs
:msg, contains, "traffic:forward accept" ~
:msg, contains, "traffic:local accept" ~
:msg, contains, "traffic:forward start" ~
:msg, contains, "traffic:forward close" ~
:msg, contains, "traffic:forward server-rst" ~
:msg, contains, "traffic:forward timeout" ~
:msg, contains, "traffic:forward client-rst" ~
:msg, contains, "traffic:local close" stop
:msg, contains, "traffic:forward ip-conn" stop
:msg, contains, "traffic:local server-rst" stop
:msg, contains, "traffic:local client-rst" stop
:msg, contains, "utm:ssl ssl-anomaly" stop
:msg, contains, "traffic:local timeout" stop
:msg, contains, "event:switch-controller" stop
:msg, contains, "event:connector" stop
:msg, contains, "event:ha" stop
:msg, contains, "event:security-rating" stop
:msg, contains, "event:vpn failure" stop
:msg, contains, "event:vpn success" stop
:msg, contains, "event:system clash" stop
:msg, contains, "event:vpn negotiate_error" stop
:msg, contains, "event:vpn esp_error" stop
:msg, contains, "event:system update" stop
:msg, contains, "event:switch-controller none" stop
:msg, contains, "event:user logout" stop
:msg, contains, "utm:anomaly anomaly" stop
:msg, contains, "event:switch-controller None" stop
:msg, contains, "event:switch-controller down" stop
:msg, contains, "event:switch-controller up" stop
:msg, contains, "event:system success" stop
:msg, contains, "event:system failed" stop
:msg, contains, "event:vpn dpd_failure" stop
template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
# queue.workerThreads sets the maximum worker threads, it will scale back to 0 if there is no activity
# Forwarding all events through TCP port
*.* action(type="omfwd"
template="AMA_RSYSLOG_TraditionalForwardFormat"
queue.type="LinkedList"
queue.filename="omfwd-azuremonitoragent"
queue.maxFileSize="32m"
action.resumeRetryCount="-1"
action.resumeInterval="5"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
queue.size="25000"
queue.workerThreads="100"
queue.dequeueBatchSize="2048"
queue.saveonshutdown="on"
target="127.0.0.1" Port="28330" Protocol="tcp")

Replace /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf file with above content. Do not forgot to restart rsyslog service:

systemctl start rsyslog
systemctl status rsyslog

Check the count of FortiGate events based on Priority and event name:

let Watchlist = datatable(Priority:string, Activity:string) [
'1','event:system',
'1','event:user',
'1','event:user logon',
'1','event:vpn',
'1','utm:anomaly',
'1','utm:dlp',
'1','utm:dlp dlp-docsource',
'1','utm:dns',
'1','utm:dns dns-query',
'1','utm:dns dns-response',
'1','utm:emailfilter',
'1','utm:emailfilter bannedword',
'1','utm:emailfilter spam',
'1','utm:emailfilter webmail',
'1','utm:ips',
'1','utm:ips botnet',
'1','utm:ips malicious-url',
'1','utm:ips signature',
'1','utm:ssh ssh-channel',
'1','utm:ssh ssh-command',
'1','utm:ssh ssh-hostkey',
'1','utm:waf',
'1','utm:waf waf-address-list',
'1','utm:waf waf-custom-signature',
'1','utm:waf waf-http-constraint',
'1','utm:waf waf-http-method',
'1','utm:waf waf-signature',
'1','utm:waf waf-url-access',
'2','event:cifs-auth-fail',
'2','event:endpoint',
'2','event:rest-api',
'2','event:router',
'2','event:sdwan',
'2','event:sdwan down',
'2','event:sdwan up',
'2','event:webproxy',
'2','event:wireless',
'2','traffic:forward deny',
'2','traffic:ztna',
'2','utm:app-ctrl' ,
'2','utm:app-ctrl port-violation',
'2','utm:app-ctrl protocol-violation',
'2','utm:app-ctrl signature',
'2','utm:file-filter',
'2','utm:virus',
'2','utm:virus analytics',
'2','utm:virus command-blocked',
'2','utm:virus content-disarm',
'2','utm:virus ems-threat-feed',
'2','utm:virus exempt-hash',
'2','utm:virus infected',
'2','utm:virus inline-block',
'2','utm:virus malware-list',
'2','utm:virus outbreak-prevention',
'2','utm:virus oversize',
'2','utm:voip',
'2','utm:webfilter',
'2','utm:webfilter ftgd_blk',
'2','utm:webfilter ftgd_err',
'2','utm:webfilter urlfilter',
'2','utm:webfilter webfilter_command_block',
'3','event:connector',
'3','event:fortiextender',
'3','event:ha',
'3','event:switch-controller',
'3','event:wanopt',
'3','traffic:forward',
'3','traffic:forward accept',
'3','traffic:forward client-rst',
'3','traffic:forward close',
'3','traffic:forward dns',
'3','traffic:forward ip-conn',
'3','traffic:forward server-rst',
'3','traffic:forward timeout',
'3','traffic:local',
'3','traffic:local accept',
'3','traffic:local client-rst',
'3','traffic:local close',
'3','traffic:local deny',
'3','traffic:local server-rst',
'3','traffic:local timeout',
'3','traffic:multicast',
'3','traffic:sniffer',
'3','utm:casb',
'3','utm:emailfilter email',
'3','utm:emailfilter ftgd_err',
'3','utm:forti-switch',
'3','utm:forti-switch fsw-flow',
'3','utm:gtp',
'3','utm:gtp gtp-all',
'3','utm:gtp pfcp-all',
'3','utm:icap',
'3','utm:ssl ssl-anomaly',
'3','utm:ssl ssl-exempt',
'3','utm:ssl ssl-handshake',
'3','utm:ssl ssl-negotiation',
'3','utm:ssl ssl-server-cert-info',
'3','utm:virtual-patch',
'3','utm:virus filename',
'3','utm:virus filetype-executable',
'3','utm:virus mimefragmented',
'3','utm:virus scanerror',
'3','utm:virus switchproto',
'3','utm:webfilter activexfilter',
'3','utm:webfilter antiphishing',
'3','utm:webfilter appletfilter',
'3','utm:webfilter content',
'3','utm:webfilter cookiefilter',
'3','utm:webfilter ftgd_allow',
'3','utm:webfilter ftgd_quota',
'3','utm:webfilter ftgd_quota_counting',
'3','utm:webfilter ftgd_quota_expired',
'3','utm:webfilter http_header_change',
'3','utm:webfilter scriptfilter',
'3','utm:webfilter ssl-exempt',
'3','utm:webfilter unknown-ce',
'3','utm:webfilter urlmonitor',
'3','utm:webfilter videofilter-category',
'3','utm:webfilter videofilter-channel',
'3','event:security-rating'
];
CommonSecurityLog
| where DeviceVendor contains "Forti"
| summarize TotalActivity = count() by Activity
| lookup Watchlist on Activity

Tcpdump command:

sudo tcpdump -i any dst port 28330 -Ann | grep "traffic:forward accept"

--------------------------------------------------------------------

Anotehr way to filter logs:

create 05-logfiler.conf file with following conent:

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:forward accept" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:local accept" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:forward close" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:forward server-rst" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:forward timeout" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:forward client-rst" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:local close" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:forward ip-conn" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:local server-rst" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:local client-rst" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "utm:ssl ssl-anomaly" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "traffic:local timeout" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:switch-controller" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:connector" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:ha" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:security-rating" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:vpn failure" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:vpn success" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:system clash" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:vpn negotiate_error" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:vpn esp_error" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:system update" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:switch-controller none" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:user logout" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "utm:anomaly anomaly" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:switch-controller None" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:switch-controller down" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:switch-controller up" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:system success" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:system failed" ~

:msg, contains, "CEF: 0|Fortinet|Fortigate|" and contains, "event:vpn dpd_failure" ~

SonicWall

Local Facility: Changed to Local 7

Log Filtering on Syslog Forwarder Server:

Note: "stop" option does not work, but ~ works.

Following configuration file is to stop Connection Closed and Connection Opened events coming into Sentinel

vi /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf

:msg, contains, "Connection Opened" ~

:msg, contains, "Connection Closed" ~

Videos

References

Filter & Split Firewall/CEF logs into multiple Sentinel tables (analytics/basic tier) to save in ingestion costs

版权声明：
作者：siwei
链接：https://www.techfm.club/p/144458.html
来源：TechFM
文章版权归作者所有，未经允许请勿转载。

THE END

二维码

搜索内容