- General HealthCommunication debugLogs from devicesLicensingExample debug session on Fortianalyzer
- Show connected to the FAZ devicesGeneral state of FAZ (version, serial, HA status, license status)Performance stats (appliance FAZ will have more data)Running processes and CPU loadLogging devices with quotas for each ADOMRun Linux df -h commandShow disks and partitionsPrint average load, only meaningful for comparingnetstat - Open connections and Listening portsPrint routing tableReal time debug of communication between FAZ and FGT (not much of help)Sniffer of packets in real time
dia sni paWhat type of logs are being received from each deviceLog received stats in last 5, 30, and 60 secondsLogs per second per deviceShow license for VMsThis cheat sheet as PDF: Fortianalyzer diagnose and debug cheat sheet
General Health
| Command | Description |
|---|---|
get sys status | Get general information: firmware version, serial number, ADOMs enabled or not, time and time zone, general license status (Valid or not). |
get sys performance | Detailed performance statistics: CPU load, memory usage, hard disk/flash disk used space and input/output ( |
exe top | Display real time list of running processes with their CPU load. |
diag log device | Shows how much space is used by each device logging to the Fortianalyzer, including quotas. |
exe iotop -b -n 1 | Display and update every 1 second READ/WRITE statistics for all the processes. |
diagnose system print cpuinfo | Display hardware CPU information - vendor, number of CPUs etc. |
diagnose hardware info | Even more hardware-related info. |
diagnose system print df | Show disk partitions and space used. Analog of the Linux |
exe lvm info | Shows disks status and size |
diagnose system print loadavg | Show average system load, analog to the Linux |
diagnose system print netstat | Show established connections to the Fortianalyzer, as well as listening ports. Every logging device can (and usually does) have multiple connections established. |
diagnose system print route | Show routing table of the Fortianalyzer. |
Communication debug
| Command | Description |
|---|---|
diagnose test application oftpd 3 | List all devices sending logs to the Fortianalyzer with their IP addresses, serial numbers, uptime meaning connection establishment uptime, not remote device uptime, and packets received (should be growing). |
diagnose debug application oftpd 8 <Device name> diagnose debug enable | Real time debug of communicating with the Device name device. |
diagnose sniffer packet any "host IP of remote device" | Sniff packets from/to remote device, to make sure they are sending each other packets. The communication is encrypted. |
diagnose sniffer packet any "port 514" | Sniff all packets to/from port 514 used by Fortianalyzer to receive logs from remote devices. |
Logs from devices
| Command | Description |
|---|---|
diagnose test application oftpd 50 | Show log types received and stored for each device. |
diag log device | Shows how much space is used by each device logging to the Fortianalyzer, including quotas. |
diagnose fortilogd lograte | Show in one line last 5/30/60 seconds rate of receiving logs. |
diagnose fortilogd lograte-adom all | Show as table log receiving rates for all ADOMs aggregated per device type (i.e. rate for all Fortigates will be as one data per ADOM). |
diagnose fortilogd lograte-device | Show average logs receive rate per device for the last hour, day, and week. |
diagnose fortilogd lograte-total | Show summary log receive rate for all devices on this Fortianalyzer. |
Licensing
| Command | Description |
|---|---|
diagnose dvm device list | Look for the line There are currently N devices/vdoms count for license. |
diagnose debug vminfo | Show report on Virtual Machine license: whether valid or not, type, licensed storage volume, licensed log receive rate, licensed maximum device count. |
Example debug session on Fortianalyzer
Show connected to the FAZ devices
FAZ-AWS # diagnose test application oftpd 3now = 1713716940(2024/04/21 19:29:00) # DEVICE CONN HOSTNAME IP UPTIME IDLETIME #PKTS BUFSZ (curr,avg,advice)------------------------------------------------------------------------------------------------------------------------------- 1 FGTAWSN1JDGCU42E 65535: 0 FGT-Perimeter 10.100.104.13 3h18m2s 1s 3048 512,0,0 (1) 2 10.100.104.15 6h56m37s 1m42s 2027 Plain-Syslog 512,0,32768 (2)
Fortigate named "FGT-Perimeter" (IP 10.100.104.13) sending logs via OFTP native protocol
Linux server (IP 10.100.104.15), sending its logs via Syslog.
General state of FAZ (version, serial, HA status, license status)
FAZ-AWS # dia deb disFAZ-AWS # get sys statusPlatform Type : FAZVM64-AWSOnDemandPlatform Full Name : FortiAnalyzer-VM64-AWSOnDemandVersion : v7.4.2-build2397 231220 (GA)Serial Number : FAZAWSTA23002441BIOS version : 04000002Hostname : FAZ-AWSMax Number of Admin Domains : 5Admin Domain Configuration : EnabledFIPS Mode : DisabledHA Mode : Stand AloneBranch Point : 2397Release Version Information : GACurrent Time : Sun Apr 21 19:39:34 IDT 2024Daylight Time Saving : YesTime Zone : (GMT+2:00) Jerusalem.x86-64 Applications : YesDisk Usage : Free 70.79GB, Total 78.19GBFile System : Ext4License Status : Valid
Performance stats (appliance FAZ will have more data)
FAZ-AWS # get sys performanceCPU: Used: 5.50% Used(Excluded NICE): 5.50% %used %user %nice %sys %idle %iowait %irq %softirq CPU0 5.05 3.24 0.00 1.80 94.95 0.00 0.00 0.00 CPU1 5.96 4.69 0.00 1.08 94.04 0.00 0.00 0.18Memory: Total: 10,041,896 KB Used: 5,416,028 KB 53.9% Total (Excluding Swap): 7,944,748 KB Used (Excluding Swap): 5,079,124 KB 63.9%Hard Disk: Total: 81,983,896 KB Used: 7,742,552 KB 9.4% Inode-Total: 5,242,880 Inode-Used: 26,347 0.5% IOStat: tps r_tps w_tps r_kB/s w_kB/s queue wait_ms 8.9 3.2 5.7 236.9 312.8 0.2 20.6Flash Disk: Total: 1,006,252 KB Used: 444,916 KB 44.2% Inode-Total: 65,536 Inode-Used: 43 0.1% IOStat: tps r_tps w_tps r_kB/s w_kB/s queue wait_ms 0.1 0.1 0.0 17.5 0.0 0.0 0.8
Running processes and CPU load
top - 19:42:52 up 7:13, 0 user, load average: 0.24, 0.23, 0.19Tasks: 234 total, 1 running, 232 sleeping, 0 stopped, 1 zombie%Cpu(s): 2.3 us, 1.3 sy, 0.0 ni, 96.2 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 stMiB Mem : 7758.5 total, 135.6 free, 7118.2 used, 2722.2 buff/cacheMiB Swap: 2048.0 total, 1707.5 free, 340.5 used. 640.3 avail Mem PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND 9859 root 20 0 177.6m 127.0m 2.0 1.6 7:41.94 S /bin/python /usr/local/lib/python3.11 1727 root 20 0 259.9m 46.9m 0.7 0.6 1:08.81 S /bin/logfwd 7833 postgres 20 0 1436.5m 27.8m 0.7 0.4 0:02.26 S postgres: postgres airflow 127.0.0.1( 9886 root 20 0 176.5m 124.8m 0.7 1.6 1:52.72 S airflow scheduler -- DagFileProcessor 245 root 20 0 95.4m 28.1m 0.3 0.4 0:35.78 S /bin/cmdbsvr 750 redis 20 0 57.6m 9.5m 0.3 0.1 0:51.04 S /bin/redis-server 127.0.0.1:6379 1573 root 20 0 1217.3m 874.5m 0.3 11.3 0:03.68 S scheduled 1579 redis 20 0 131.6m 11.2m 0.3 0.1 0:33.65 S /bin/redis-server 127.0.0.1:6380 1580 redis 20 0 131.6m 10.4m 0.3 0.1 0:26.31 S /bin/redis-server 127.0.0.1:6383 1757 root 20 0 226.5m 39.7m 0.3 0.5 0:09.40 S /bin/clusterd 1785 root 20 0 210.6m 57.8m 0.3 0.7 0:11.08 S /bin/sqlrptcached 1789 root 20 0 283.2m 70.3m 0.3 0.9 0:25.49 S /bin/sqlplugind
Logging devices with quotas for each ADOM
FAZ-AWS # diag log deviceDevice Name Device ID Used Space(logs / quarantine / content / IPS) Allocated Space Used%FGT-Perimeter FGTAWSN1JDGCU42E 4.4MB( 4.4MB/ 0.0KB/ 0.0KB/ 0.0KB) unlimited n/aSYSLOG-Linux SYSLOG-0A64680F 76.0KB( 76.0KB/ 0.0KB/ 0.0KB/ 0.0KB) unlimited n/aTotal: 2 log devices, used=4.5MB quota=unlimitedAdomName AdomOID Type Logs [Retention Quota Used( logs/quaranti/ content/ IPS) Used%] [RetentionFGT-only 193 FGT 365days unlimited 4.4MB( 4.4MB/ 0.0KB/ 0.0KB/ 0.0KB) n/a 92daysFortiAnalyzer 133 FAZ 365days 300.0MB 8.0KB( 8.0KB/ 0.0KB/ 0.0KB/ 0.0KB) 0.0% 60daysFortiAuthenticator 149 FAC 365days 300.0MB 8.0KB( 8.0KB/ 0.0KB/ 0.0KB/ 0.0KB) 0.0% 60daysFortiCache 137 FCH 365days 300.0MB 8.0KB( 8.0KB/ 0.0KB/ 0.0KB/ 0.0KB) 0.0% 60daysFortiCarrier 129 FGT 365days 300.0MB 8.0KB( 8.0KB/ 0.0KB/ 0.0KB/ 0.0KB) 0.0% 60daysFortiClient 139 FCT 365days 300.0MB 8.0KB( 8.0KB/ 0.0KB/ 0.0KB/ 0.0KB) 0.0% 60daysTotal usage: 17 ADOMs, logs=4.6MB(4.6MB/0.0KB/0.0KB/0.0KB) database=181.3MB(ADOMs usage:24.3MB(96.6KB, 0.0KB)Total Quota Summary:*** Warning: Total Allocated Quota is bigger than Total Quota! Please check the quota configuration of ADOMs! Total Quota Allocated Available Allocate% 63.2GB 64.6GB 0.0KB 102.3%System Storage Summary: Total Used Available Use% 78.2GB 7.4GB 70.8GB 9.5 %Reserved space: 15.0GB (19.2% of total space).
Run Linux df -h command
FAZ-AWS # diagnose system print df -hFilesystem Size Used Available Use% Mounted onrootfs 3.8G 1.8G 1.9G 49% /none 3.8G 0 3.8G 0% /devnone 6.9G 1.1M 6.9G 0% /dev/shmnone 64.0M 72.0K 63.9M 0% /tmp/dev/nvme0n1p1 982.7M 434.5M 548.2M 44% /data/dev/mdvg/mdlv 78.2G 7.4G 70.8G 9% /var/dev/mdvg/mdlv 78.2G 7.4G 70.8G 9% /drive0/dev/mdvg/mdlv 78.2G 7.4G 70.8G 9% /Storage/dev/loop0 8.6M 19.0K 8.1M 0% /var/dm/tcl-rootnone 512.0M 0 512.0M 0% /drive0/tmp/sql_batnone 128.0M 0 128.0M 0% /drive0/private/dbcommit
Show disks and partitions
FAZ-AWS # exe lvm infoLVM Status: OKLVM Size: 80GBFile System: ext4 78GBDisk1 : Used 80GBDisk2 : Unavailable 0GBDisk3 : Unavailable 0GB
Print average load, only meaningful for comparing
FAZ-AWS # diagnose system print loadavg0.08 0.19 0.18 1/695 9241
netstat - Open connections and Listening ports
FAZ-AWS # diagnose system print netstatActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 0.0.0.0:7080 0.0.0.0:* LISTENtcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTENtcp 0 0 :::443 :::* LISTENtcp 0 0 :::26443 :::* LISTENtcp 0 0 ::1:8123 :::* LISTENtcp 0 0 ::ffff:10.100.104.17:514 ::ffff:10.100.104.13:4128 ESTABLISHEDtcp 0 0 ::ffff:10.100.104.17:514 ::ffff:10.100.104.15:60170 ESTABLISHEDudp 0 0 127.0.0.1:6001 0.0.0.0:*udp 0 0 127.0.0.1:6003 0.0.0.0:*udp 0 0 0.0.0.0:31167 10.100.0.2:53 ESTABLISHEDudp 0 0 10.100.104.17:52222 10.100.0.2:53 ESTABLISHED
Print routing table
FAZ-AWS # diagnose system print routeDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0 10.100.104.1 0.0.0.0 UG 0 0 0 port10.0.0.0 10.100.104.13 0.0.0.0 UG 1 0 0 port110.100.104.0 0.0.0.0 255.255.255.0 U 0 0 0 port1169.254.169.254 169.254.169.254 255.255.255.255 UGH 0 0 0 port1
Real time debug of communication between FAZ and FGT (not much of help)
FAZ-AWS # diagnose debug application oftpd 8 FGTAWSN1JDGCU42Eoftpd debug filter: filter(string)==FGTAWSN1JDGCU42EFAZ-AWS #FAZ-AWS #[T3993:oftps.c:1933 FGTAWSN1JDGCU42E:10.100.104.13] SSL socket[20] pid[1754] ssl[0x5600c4048980]received [12] bytes:[T3993:main.c:4174 FGTAWSN1JDGCU42E:10.100.104.13] handle KEEPALIVE[T3996:oftps.c:1999 FGTAWSN1JDGCU42E:10.100.104.13] SSL socket[20] pid[1754] ssl[0x5600c4048980]sent [21] bytes:[T3996:oftp_restapi_sched.c:1785] FGTAWSN1JDGCU42E[T3996:oftp_restapi.c:2333 FGTAWSN1JDGCU42E:10.100.104.13] ret = 0.
Sniffer of packets in real time dia sni pa
FAZ-AWS # diagnose sniffer packet any "port 514"interfaces=[any]filters=[port 514]1.383021 10.100.104.13.9334 -> 10.100.104.17.514: udp 6463.640615 10.100.104.13.4128 -> 10.100.104.17.514: psh 2709416742 ack 15567412763.640752 10.100.104.17.514 -> 10.100.104.13.4128: psh 1556741276 ack 27094167763.640870 10.100.104.13.4128 -> 10.100.104.17.514: ack 15567413196.383617 10.100.104.13.9334 -> 10.100.104.17.514: udp 5928.646227 10.100.104.13.4128 -> 10.100.104.17.514: psh 2709416776 ack 15567413198.646360 10.100.104.17.514 -> 10.100.104.13.4128: psh 1556741319 ack 27094168108.646492 10.100.104.13.4128 -> 10.100.104.17.514: ack 1556741362
What type of logs are being received from each device
FAZ-AWS # diagnose test application oftpd 50Showing logtypes of all cached devices ...... SN VDOM RETENTION-HOUR LOGTYPES/ERROR FGTAWSN1JDGCU42E root 2208 app-ctrl|ips|anomaly|dlp|emailfilter|event.system|event.vpn| event.user|event.wireless|event.endpoint|event.ha|event.security-rating|event.connector| traffic.forward|traffic.local|traffic.multicast|traffic.sniffer|virus|voip|webfilter|dns| ssh|ssl|security SYSLOG-0A64680F root 1440 generic
Log received stats in last 5, 30, and 60 seconds
FAZ-AWS # diagnose fortilogd logratelast 5 seconds: 0.0, last 30 seconds: 0.1, last 60 seconds: 0.1
Logs per second per device
FAZ-AWS # diagnose fortilogd lograte-deviceLogs per secondTotals Last Hour Day Week------------------------------------------------------- FGTAWSN1JDGCU42E: 0.16 0.11 0.02 SYSLOG-0A64680F: 0.03 0.02 0.00 FAZAWSTA23002441: 0.00 0.00 0.00
Show license for VMs
FAZ-AWS # diagnose debug vminfoVM license is valid.fds_code: 200Type: FullLicensed GB/Day: 1Max devices: 2Serial Number: FAZAWSTA23772441VM UUID: ec211ef8-3328-358f-f78f-9450cf09a51d
Follow me on https://www.linkedin.com/in/yurislobodyanyuk/ not to miss what I publish on Linkedin, Github, blog, and more.
