ScanBI Support Guide for Diagnostic Investigation and Triage
ScanBI 26.2 Manual Device Integration Diagnostics Purpose & Audience: A technical troubleshooting runbook for ScanBI support engineers to manually verify, diagnose, and isolate data collection issues across airport screening device integrations from the ScanBI/Statseeker CLI. Core Topics & Integration Breakdown General Diagnostics & Safety: CLI prerequisites (nim-api, ping, netcat, tcpdump), clock verification, and baseline device inventory/metric queries in Statseeker. 920CT Baggage Scanner: Validates scanner network mappings, TCP reachability, mutual TLS (mTLS) certificate paths, direct HTTPS API authentication (/dashboard/credentials, /dashboard/network_status), and scanbi_ct_collector logs. QPS Passenger Scanner (201 P1/P2): Validates Samba (SMB port 445) connectivity to /home/scanbi/shared, CSV/statistics file delivery, serial (srNo) matching, queue activity, and scanbi_qps_collector / processor processes. WTMD Walk-Through Metal Detector (900M): Diagnoses single-connection TCP stream behavior on port 10002 via passive tcpdump, base logs, and queue reader. (Note: WTMD 6E is unsupported). RTT Baggage Scanner & Workstations: Checks SFTP file delivery of CSV reports (Sessionlog, FindBag, ThreatDetection, EventLog) to /home/scanbi/rtt/csv_reports, enforcing the 30-minute timestamp/staleness window and serial mapping. 4DX Trace Scanner: Scopes scan metrics by serial and live operator names via nim-api and Rapiscan RCC integration logs. Triage & Escalation: Standardized evidence collection templates, data-state matrix (network vs. file vs. metric vs. identity failures), and escalation criteria.
Audience: ScanBI support engineers
Purpose: Perform direct, manual diagnosis of each supported ScanBI device integration from the ScanBI server.
ScanBi Version: 26.2
Scope: Network, API, SMB, SSH, files, metrics, and collector logs.
1. Case preparation and safety
1.1 Understand and Record these facts first
|
Record |
Why it matters |
|---|---|
|
Airport, Terminal, Screening Point, Lane, device hostname, IP, model, and serial number |
Prevents testing the wrong device or matching data to the wrong serial. |
|
First observed time and time zone |
Required for interpreting logs, SSH logins, and data-file timestamps. |
|
User impact and expected device activity |
A zero scan count can be valid during an idle period. |
|
Recent device, network, certificate, collector, or ScanBI maintenance |
Helps correlate an integration break with a change. |
|
The exact command and output |
Allows escalation without repeating a potentially destructive investigation. |
For security reasons, do not put private keys, API tokens, login details into a support ticket.
1.2 Confirm the ScanBI server time
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
All stale-data conclusions depend on the ScanBI system clock.
date '+%Y-%m-%d %H:%M:%S %Z' uptime
If timestamps look inconsistent, check clock synchronisation under the approved platform procedure before diagnosing a device data feed.
the WTMD 6E is currently not supported by ScanBi 26.2
1.3 Generic direct network checks
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Run the ping and netcat commands from the ScanBI server, replacing the placeholders with the known device address and required service port.
The netcat command below tests if the is open on the target server without sending any data and provides verbose output about the connection attempt. I suggest testing if port 443 (HTTPS) is open.
/sbin/ping -c 4 -W 1000 <device-ip> nc -vz -w 5 <device-ip> <tcp-port>
The ping utility options:
-c count Stop after sending (and receiving) count ECHO_RESPONSE packets. If this option is not specified, ping will operate until interrupted.
-W waittime Time in milliseconds to wait for a reply for each packet sent. If a reply arrives later, the packet is not printed as replied, but considered as replied when calculating statistics.
e.g.
statseeker$ /sbin/ping -c 4 -W 1000 10.2.20.100 PING 10.2.20.100 (10.2.20.100): 56 data bytes 64 bytes from 10.2.20.100: icmp_seq=0 ttl=64 time=0.374 ms 64 bytes from 10.2.20.100: icmp_seq=1 ttl=64 time=0.331 ms 64 bytes from 10.2.20.100: icmp_seq=2 ttl=64 time=0.347 ms 64 bytes from 10.2.20.100: icmp_seq=3 ttl=64 time=0.310 ms --- 10.2.20.100 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.310/0.341/0.374/0.023 ms
The netcat utility options:
-v The -v option tells nc to produce verbose output. -z Specifies that nc should just scan for listening daemons, without sending any data to them. It is an error to use this option in conjunction with the -l option. -w timeout Connections which cannot be established or are idle timeout after timeout seconds. The -w flag has no effect on the -l option, i.e. nc will listen forever for a connection, with or without the -w flag. The default is no timeout.
e.g.
statseeker$ nc -vz -w 5 10.2.20.100 443 Connection to 10.2.20.100 443 port [tcp/https] succeeded!
Interpret them separately:
|
Result |
Meaning |
Next step |
|---|---|---|
|
Ping succeeds; TCP port opens |
Basic route and tested service port are reachable. |
Continue with protocol/API/data checks. |
|
Ping fails; TCP port opens |
ICMP is blocked or filtered, but service reachability exists. |
Do not report a network outage based on ping alone. |
|
Ping succeeds; TCP port fails |
Device answers ICMP but service/port, route ACL, or firewall is unavailable. |
Check device service and firewall policy. |
|
Both fail |
Device, address, VLAN, route, firewall, or power issue. |
Confirm physical state and network path. |
Use packet capture only with the required approval and only long enough to prove connection setup.
tcpdump -D tcpdump -ni <interface> 'host <device-ip> and tcp port <tcp-port>'
The tcpdump utility options
-D --list-interfaces Print the list of the network interfaces available on the system and on which tcpdump can capture packets.
-n Don't convert addresses (i.e., host addresses, port numbers, etc.) to names.
-i interface --interface=interface
If the -D flag is supported, an interface number as printed by that flag can be used as the interface argument.
e.g.
# tcpdump -ni vmx0 'host 10.2.1.98 and tcp port 587' tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on vmx0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 18:02:59.769562 IP 10.2.20.200.62370 > 10.2.1.98.587: Flags [S], seq 132047967, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2493051800 ecr 0], length 0
2. Manual API Statseeker inventory and metric checks
Use nim-api to confirm that Statseeker has the expected device identity before investigating a data mismatch. Run commands as an authorised Statseeker administrative user on the ScanBI server.
2.1 Inventory request
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Create an inventory request that returns hostname, IP, classification, serial, polling state, and average Ping RTT.
cat >/tmp/scanbi-device-inventory.json <<'EOF' { "command": "get", "user": "admin", "objects": [{ "type": "device_equipment", "fields": { "id": {}, "hostname": {}, "ipaddress": {}, "equipmentType": {}, "equipmentStatus": {}, "productLine": {}, "modelName": {}, "serialNumber": {}, "poll": {}, "ping_rtt": { "timefilter": {"query": "range = start_of_today to now; "}, "formats": ["avg"] } }, "limit": 0 }] } EOF /usr/local/statseeker/ss/bin/nim-api -f /tmp/scanbi-device-inventory.json | jq .
Locate the device by hostname/IP/serial and record its numeric id. Remove the temporary request when finished if local operating procedure requires it.
equipmentStatus provides an additional manual triage signal in the Administrator Reference:
|
Status |
Meaning |
|---|---|
|
active |
Equipment has been used within the last five minutes. |
|
standby |
Equipment is online but has not been used within the last five minutes. |
|
network-failed |
Equipment did not respond to ICMP and may still be operational. |
Do not equate network-failed with a confirmed device outage. Confirm the application/data path appropriate to the equipment type.
2.2 Per-device scan metric pattern
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Do not use an unfiltered aggregate when diagnosing one device. Query the native metric with an exact device ID filter.
{ "command": "get", "user": "admin", "objects": [{ "type": "<object-type>", "fields": { "scans": { "timefilter": {"query": "range = start_of_today to now; "}, "formats": ["total"] }, "device_filter": { "filter": {"query": "IN (<device-id>)"}, "hide": true, "field": "id" } } }] }
Use these object types:
|
Equipment |
Object type |
Metric |
|---|---|---|
|
920CT |
device_ct_scanner |
scans |
|
QPS and WTMD |
device_passenger_scanner |
scans |
|
RTT scanner |
device_rtt |
scans(use==(device-id)filter syntax if required by the RTT object) |
|
4DX |
trace_detection_scanner_operator |
traceScannerOperatorScans; see section 8 because it must be scoped by serial and live operator names. |
A number shows a metric was stored for that exact device in the requested time range. A zero can be valid. An empty result is an API/object/filter/data-path problem and must not be reported as a zero.
3. 920CT baggage scanner
3.1 Information required
- Statseeker device ID, hostname, address, and serial number.
- Serial number is mandatory and is a primary key for data being mapped to the device.
- ctScannerNetworkID for the 920CT device.
- Mapped scanner-network address and port.
- Existing CT mutual-TLS certificate and key paths.
3.2 Check the CT device and network mapping
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Query the CT object and scanner networks directly. The target IP/port for the CT API belongs to the scanner-network mapping; it is not necessarily the same as the managed device IP.
cat >/tmp/ct-network-check.json <<'EOF' { "command": "get", "user": "admin", "objects": [ { "type": "device_ct_scanner", "fields": { "id": {}, "hostname": {}, "ipaddress": {}, "serialNumber": {}, "ctScannerNetworkID": {} }, "limit": 0 }, { "type": "scanner_network", "fields": { "id": {}, "scannerNetworkTitle": {}, "scannerNetworkIpaddress": {}, "scannerNetworkPort": {} }, "limit": 0 } ] } EOF /usr/local/statseeker/ss/bin/nim-api -f /tmp/ct-network-check.json | jq .
Match ctScannerNetworkID to scanner_network.id. Record scanner-network title, address, and port. If mapping is absent/incorrect, correct source data through change control before treating the API as failed.
3.3 Check basic connectivity
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
/sbin/ping -c 4 -W 1000 <ct-device-ip> nc -vz -w 5 <scanner-network-ip> <scanner-network-port>
If the TCP check fails, investigate route, firewall, address, port, and CT service state. A successful ping to the CT device does not prove that the scanner-network API endpoint is available. The presentation refers to port 10001, but a configured scanner-network port takes precedence; see section 1.4.
3.4 Locate the existing mutual-TLS material
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
The CT collector’s certificate/key paths are normally defined in scanbi.config.
/usr/local/bin/python3.11 -c 'from scanbi.config import CT_SSL_CRT, CT_SSL_KEY; print("certificate=" + CT_SSL_CRT); print("key=" + CT_SSL_KEY)'
Validate path existence and restricted permission without displaying file contents:
ls -l <certificate-path> <key-path>
Do not copy, print, attach, or alter the private key. A certificate/key read failure is a ScanBI configuration or permission issue, not a CT scanner network failure. The Administrator Reference also requires the ScanBI client certificate to be on the target ConneCT network whitelist; a rejected credentials request can therefore indicate a whitelist or certificate-deployment issue.
3.5 Make a direct 920CT API status request
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
The following standard-library script requests credentials and then network status. It does not print the authentication token.
CT_HOST='<scanner-network-ip>' \ CT_PORT='<scanner-network-port>' \ CT_CERT='<certificate-path>' \ CT_KEY='<key-path>' \ /usr/local/bin/python3.11 - <<'PY' import http.client import json import os import ssl host = os.environ['CT_HOST'] port = int(os.environ['CT_PORT']) context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH) # This deliberately mirrors the installed ScanBI CT collector for a known # scanner-network endpoint. Do not generalise it to unrelated services. context.check_hostname = False context.verify_mode = ssl.CERT_NONE context.load_cert_chain(os.environ['CT_CERT'], os.environ['CT_KEY']) connection = http.client.HTTPSConnection(host, port, context=context, timeout=15) connection.request('POST', '/dashboard/credentials', headers={'Content-Type': 'application/json'}) credentials = connection.getresponse() credential_body = credentials.read().decode('utf-8', errors='replace') print('credentials HTTP', credentials.status, credentials.reason) if not 200 <= credentials.status < 300: raise SystemExit('credentials endpoint rejected the request') token = json.loads(credential_body).get('authToken') if not token: raise SystemExit('credentials response did not contain authToken') connection.close() connection = http.client.HTTPSConnection(host, port, context=context, timeout=15) connection.request('GET', '/dashboard/network_status', headers={ 'Content-Type': 'application/json', 'Authorization': token }) status = connection.getresponse() body = status.read().decode('utf-8', errors='replace') print('network status HTTP', status.status, status.reason) if not 200 <= status.status < 300: raise SystemExit('network-status endpoint rejected the request') payload = json.loads(body) print('networkStatus:', payload.get('networkStatus', '<missing>')) connection.close() PY
|
Observation |
Meaning |
Manual follow-up |
|---|---|---|
|
TCP open, credentials/network-status 2xx, status healthy/ok |
CT API path is working. |
Investigate scan metric and collector data if scans are missing. |
|
TCP fails |
Endpoint is not reachable. |
Check firewall, route, port, address, and device/service state. |
|
TLS/certificate load error |
ScanBI certificate/key path or permission fault. |
Validate deployment/permissions; do not change material without approval. |
|
Credentials 401/403 or other non-2xx |
Endpoint refused authentication/API request. |
Escalate with HTTP code and certificate deployment history; never share token/key. |
|
Network status non-2xx/invalid JSON |
CT API contract or service problem after authentication. |
Capture HTTP code/time and escalate to CT integration/vendor support. |
3.6 Check 920CT data collection and logs
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Query per-device device_ct_scanner.scans using section 2.2. Then inspect collector events:
tail -n 5000 /home/statseeker/base/logs/python.log \ | grep 'scanbi_ct_collector' \ | grep -E 'WARNING|ERROR|FATAL'
Search additionally for serial, hostname, IP, timeout, ssl, certificate, refused, and relevant HTTP codes. Correlate log times with API failures and missing scans.
3.7 CT escalation evidence
- CT device IP/serial and mapped scanner-network ID/title/IP/port.
- Ping and TCP test results.
- Direct API HTTP statuses and returned networkStatus only; never auth token/key.
- Exact per-device scan metric response.
- Sanitised scanbi_ct_collector errors with timestamps.
3.8 CT workstation scope boundary
The supplied presentation states that PVS and SVS CT workstations are supported by ScanBI. This guide deliberately does not provide a manual API check for them: the supplied sources do not define their endpoint, API contract, collector, or metric procedure. Capture the workstation identity and mapped ConneCT network, then obtain the CT workstation-specific runbook from the responsible integration owner.
4. QPS Passenger Scanner (201 P1 / 201 P2)
4.1 Current architecture and support baseline
QPS scan data is delivered to the ScanBI server through the Samba share at:
/home/scanbi/shared
The QPS integration uses two ScanBI processes:
|
Process |
Role |
|---|---|
|
scanbi_qps_collector |
Reads QPS statistics files from /home/scanbi/shared, identifies new scan records, and sends them to the QPS queue. |
|
scanbi_qps_processor |
Reads QPS queue messages, stores raw records in qps_record, and writes timeseries data to device_qps. |
The collector normally checks files every five seconds and bundles new scans into queue messages. The processor normally writes timeseries data every 60 seconds.
4.2 Information to collect before troubleshooting
Record the following before making changes:
- QPS hostname, IP address, equipment ID, model, product line, and serial number.
- Serial number is mandatory and is a primary key for data being mapped to the device.
- The expected QPS share folder.
- The username and password defined for the Samba share
- The latest statistics file and its modification time.
- Whether a controlled scan can be performed.
- ScanBI server time and QPS device time.
- Any recent network, Samba, QPS, or ScanBI configuration changes.
Confirm basic reachability:
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
/sbin/ping -c 4 -W 1000 <qps-ip>
A successful ping only proves IP reachability. It does not confirm that the QPS is connected to the Samba share or writing scan files.
4.3 Verify Samba service and QPS SMB connectivity
First, confirm the Samba service is running through:
Admin Tool -> Service Control
Restarting Samba should only be performed under the approved operational change process.
Confirm that the QPS equipment record has the correct IP address in Statseeker. The QPS IP address is used to permit access to the SMB share. A device that is not configured as QPS equipment, or has an incorrect IP address, will not be allowed to connect.
Check current SMB sessions:
The below command smbstatus can only be run as the ROOT user (root)
/usr/local/bin/smbstatus
A current QPS client session is useful evidence, but no active session does not automatically mean failure. A QPS can disconnect after it has written its statistics file.
The QPS scanner must be able to reach the ScanBI Samba service on TCP port 445. Test from the QPS device itself, or from an approved host on the same network path:
nc -vz -w 5 <scanbi-server-ip> 445
On ScanBI, confirm that Samba is listening:
sockstat -4 -l | grep 445
If the share configuration appears correct but the QPS does not connect, investigate the QPS network and SMB configuration. Do not mount or write to the production QPS share from another device as a workaround.
4.4 Locate the QPS folder and latest statistics file
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
QPS folders normally use the serial number directly, or use a model prefix such as QPSP1_ or QPSP2_.
SERIAL='<qps-serial-number>' find /home/scanbi/shared -maxdepth 1 -type d \ \( -name "$SERIAL" -o -name "QPSP1_$SERIAL" -o -name "QPSP2_$SERIAL" \) \ -print
After identifying the folder, locate the newest data file:
FOLDER='/home/scanbi/shared/QPSP1_<qps-serial-number>' find "$FOLDER" -type f \( -name '*.csv' -o -name 'Statistics_*.txt' \) -print0 \ | xargs -0 ls -1t \ | head -n 1
A current file should appear shortly after a controlled QPS scan completes. Check both the file modification time and timestamps inside the scan data.
ls -lT '<latest-statistics-file>'
The collector’s default file stale threshold is 172800 seconds, or 48 hours. Files older than that threshold are treated as stale and are not processed. Individual scan records more than 120 seconds old are also ignored by the processor by default.
If scan timestamps are significantly wrong, investigate time synchronisation on the QPS device.
4.5 Validate QPS serial identity
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
The serial number in Statseeker must match the srNo field in the statistics data. A mismatch can result in data being associated with the wrong device or excluded from expected results.
/usr/local/bin/python3.11 - "$FOLDER" "$SERIAL" <<'PY' import csv import os import sys folder, expected = sys.argv[1:] expected = expected.strip().casefold() files = [] for root, _, names in os.walk(folder): for name in names: low = name.casefold() if low.endswith(".csv") or ( low.startswith("statistics_") and low.endswith(".txt") ): files.append(os.path.join(root, name)) if not files: raise SystemExit("No recognised statistics files found") failures = 0 for path in sorted(files): values = set() reason = "" try: with open(path, encoding="utf-8-sig", errors="replace", newline="") as source: index = None for line in source: header = next(csv.reader([line]), []) for position, field in enumerate(header): if field.strip().casefold() == "srno": index = position break if index is not None: break if index is None: reason = "srNo column missing" else: for row in csv.reader(source): if len(row) > index and row[index].strip(): values.add(row[index].strip()) if not values: reason = "srNo has no values" except OSError as error: reason = str(error) relative = os.path.relpath(path, folder) if reason: failures += 1 print("INVALID ", relative, ": ", reason, sep="") elif any(value.casefold() != expected for value in values): failures += 1 print("MISMATCH ", relative, ": ", ", ".join(sorted(values)), sep="") else: print("MATCH ", relative, sep="") print("files=", len(files), "failures=", failures) PY
|
Finding |
Meaning |
Required follow-up |
|---|---|---|
|
No matching QPS folder |
Serial mismatch, incorrect folder naming, or QPS has not mounted/written to the share. |
Confirm QPS serial, SMB configuration, and device export settings. |
|
Folder exists but no statistics files |
QPS has access to the share but is not exporting scan data. |
Check QPS operation and data-export configuration. |
|
Latest file is stale |
QPS has not delivered recent data. |
Check QPS time, network, SMB session, storage, and export service. |
|
srNo mismatch |
Data is routed to the wrong folder or the Statseeker serial is incorrect. |
Treat as an identity/routing issue; correct the equipment serial after validation. |
|
srNo missing |
File schema is not suitable for serial validation. |
Preserve the file header and escalate as an integration/schema issue. |
4.6 Check collector and processor status
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Confirm both QPS processes are running:
pgrep -af 'scanbi_qps_(collector|processor)'
Both processes can temporarily enable debug logging:
# Enable debug logging for the selected process. kill -USR2 <pid> # Disable debug logging after the investigation. kill -USR1 <pid>
Use debug logging only for a short, approved investigation window. Always disable it afterwards.
QPS integration logs are stored in:
/home/statseeker/base/logs/python.log
Check recent warnings, errors, and fatal events:
tail -n 5000 /home/statseeker/base/logs/python.log \ | grep -E 'scanbi_qps_(collector|processor)' \ | grep -E 'WARNING|ERROR|FATAL'
During normal processing, collector debug logs should show messages similar to:
Found 4 new lines in /home/scanbi/shared/<folder>/Statistics_<timestamp>.txt pushing 395 bytes
Processor logs should show messages similar to:
Received message with 2 scans Successfully inserted 2 scans into database
Warnings, repeated parsing failures, corrupt-file messages, or missing collector/processor activity during a controlled scan should be escalated to development.
4.7 Observe the QPS queue
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
The collector bundles scans for approximately five seconds before writing them to the queue.
queue_reader -f /home/system/queues/modules/scanbi_qps/scanbi_qps.queue \ -t proto.scanbi_qps_pb2
Use the command briefly during a controlled scan. Do not modify, purge, or manually write to the queue.
A QPS queue message contains:
|
Field |
Description |
|---|---|
|
file |
Path of the statistics file containing the scan data. |
|
version |
Version header from the first line of the file. |
|
header |
CSV column header from the second line of the file. |
|
scans |
Scan lines read from the file. |
Confirm that the message references the expected QPS folder and that new scan records appear shortly after the controlled scan.
4.8 Configuration reference
Default configuration files are located in:
/usr/local/statseeker/scs/etc/
To override a configuration, copy the relevant file to:
/home/statseeker/modules/scanbi/etc/
Do not modify the default file directly. Existing override files should be reviewed before creating another change.
Important collector settings include:
|
Setting |
Default |
Purpose |
|---|---|---|
|
root_path |
/home/scanbi/shared |
QPS Samba-share location. |
|
cache_path |
.../scanbi_qps_collector_cache.json |
Persistent cache of processed file information. |
|
max_message_size |
65536 |
Maximum bundled queue-message size. |
|
stale_threshold |
172800 |
File age in seconds before it is treated as stale. |
|
error_reload_count |
3 |
Consecutive errors before a file is reloaded. |
|
error_reject_count |
9 |
Consecutive errors before a file is considered corrupt and ignored. |
|
max_cache_size |
1000 |
Cache size before stale entries are pruned. |
|
collection_interval |
5 |
File-processing interval in seconds. |
|
collection_offset |
0 |
Offset within the collection interval. |
Important processor settings include:
|
Setting |
Default |
Purpose |
|---|---|---|
|
poll_interval |
60 |
Timeseries insertion interval in seconds. |
|
max_scan_delay |
120 |
Maximum scan age in seconds before the scan is ignored. |
|
fields |
Site/version specific |
CSV headers and column offsets to collect. |
|
separators |
",", `" |
"` |
|
invalid_reasons |
Site/version specific |
Mapping of invalid-reason codes to descriptions. |
4.9 Verify stored QPS data
The processor stores:
|
Data type |
ScanBI object |
|---|---|
|
Raw scan records |
qps_record |
|
QPS timeseries |
device_qps |
Validate that the selected QPS device has current data by querying the applicable QPS timeseries object with an exact device ID. Compare the result with the latest statistics-file timestamp and controlled scan time.
If current CSV data exists but QPS metrics are missing:
- Confirm the collector found the new records.
- Confirm the queue contains the expected message.
- Confirm the processor successfully inserted the scans.
- Check for serial mismatch, stale data, parse errors, or invalid timestamps.
- Escalate with the relevant logs and queue evidence.
4.10 Record retention and manual pruning
QPS scanner records are retained for 90 days by default. The advanced option max_history_scanbi_records sets retention for all ScanBI records, including WTMD data. Records are pruned automatically every four hours.
Manual pruning is available through:
/usr/local/statseeker/scs/bin/scanbi_prune_records -t <unix-timestamp>
This command deletes historical ScanBI records. Do not run it as part of troubleshooting without explicit data-retention and change approval.
4.11 QPS escalation evidence
- QPS hostname, IP, serial number, equipment ID, model, and product line.
- Samba service status and smbstatus result.
- Confirmed SMB/445 connectivity from the QPS network path.
- QPS folder path, latest file name, file timestamp, and controlled scan time.
- srNo validation result.
- Collector and processor process status.
- Relevant Python log entries, including timestamps and severity.
- Queue-reader result for the controlled scan.
- Per-device QPS metric result and expected scan count.
- Any approved configuration overrides and relevant recent changes.
5. WTMD (900M Walk-Through Metal Detector)
5.1 Current architecture and confirmed support baseline
The updated WTMD support note identifies these two local ScanBI processes:
|
Process |
Role |
|---|---|
|
scanbi-wtmd-collector |
Connects to each WTMD over TCP, receives raw scan packets, and forwards data to the WTMD queue. |
|
scanbi-wtmd-processor |
Reads the queue, stores raw records in wtmd_record, and writes WTMD timeseries data to device_wtmd. |
The current support baseline is TCP port 10002. The older presentation’s reference to port 10001 is superseded by the updated WTMD support note unless the installed collector configuration proves a deliberate local override.
Both processes use ScanBI base logs and cannot be externally configured. Do not attempt to alter their configuration or open a second persistent client connection as a workaround.
5.2 Before testing
Record the WTMD IP, serial number, device ID, real-versus-simulated status, first observed time, and whether a controlled scan can be performed. Confirm the equipment record has the correct IP address and that its Statseeker ping state is up.
/sbin/ping -c 4 -W 1000 <wtmd-ip>
A ping failure indicates a device, address, route, firewall, VLAN, or power issue. A successful ping does not prove that the WTMD TCP data stream is healthy.
5.3 Preferred live test: observe collector traffic passively
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Use tcpdump on the ScanBI server to observe the existing collector connection. This is the preferred test because a WTMD only services one active connection; additional connections may complete TCP setup yet be ignored by the device.
tcpdump -ni <interface> -vvv 'host <wtmd-ip> and tcp port 10002'
During a controlled scan, expect to see packets sent from the WTMD to ScanBI. Also expect ScanBI to send a keep-alive approximately once per minute and the WTMD to respond.
|
Observation |
Meaning |
Next action |
|---|---|---|
|
WTMD scan packets and keep-alive responses are visible |
Transport is active. |
Inspect the queue, processor logs, and stored metrics. |
|
Keep-alives leave ScanBI but no device reply |
Address/port path exists but detector is not servicing the collector. |
Confirm no competing client; involve device support before any approved reboot. |
|
No packets in either direction |
Collector may not be attempting the device, route is broken, device may be unresponsive, or another process owns the only connection. |
Verify collector process, correct IP, and existing device connections. |
|
TCP connection succeeds from a test client but no application response |
This is not proof of health; a WTMD can ignore a second connection while another client is active. |
Stop relying on the test connection and use passive capture. |
Do not run a broad or prolonged capture. Start capture immediately before a controlled scan, collect only the required connection evidence, and stop it promptly.
5.5 Check the collector and processor processes
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Confirm both processes exist before escalating a device fault:
pgrep -af 'scanbi-wtmd-(collector|processor)'
The updated support note permits temporary debug logging through signals. Use this only with approval, record the start/end time, and always disable debug logging afterwards.
# Enable debug for the selected collector or processor PID. kill -USR2 <pid> # Disable debug when the short capture window is complete. kill -USR1 <pid>
Signals change logging behaviour; they are not a replacement for an approved service restart and should never be sent to an unverified PID.
5.6 Inspect base logs
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Both WTMD processes log to the ScanBI base logs. Start with bounded searches for the collector and processor, then narrow by IP, device ID, serial, timeout, connection, or a protocol phrase.
tail -n 5000 /home/statseeker/base/logs/base-logd.log \ | grep -E 'scanbi-wtmd-(collector|processor)' \ | grep -E 'WARNING|ERROR|FATAL' grep -E 'scanbi-wtmd-(collector|processor)' /home/statseeker/base/logs/base-logd.log \ | grep -i '<search-term>' \ | tail -n 100
Correlate log timestamps with the packet capture and controlled scan. A clear recent log does not prove that historical errors were absent.
5.7 Observe the WTMD queue
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
The collector writes raw messages to the WTMD queue and the processor consumes them. The updated support note supplies this read-only queue command:
queue_reader -f /home/system/queues/modules/scanbi_wtmd/scanbi_wtmd.queue \ -t proto.scanbi_wtmd_pb
Run it briefly during a controlled scan and stop it when enough evidence is collected. Check queue_reader --help for safe reader options; do not alter or purge the queue during support investigation.
Expected message behaviour:
|
Message |
Timing |
Important fields |
|---|---|---|
|
Individual scan |
Usually delivered within one second of a scan. |
device_id, scan_time, signal_value, scan_zones, alarm, dir(Forward or Reverse). |
|
Minute totals |
Once per minute per scanner. |
device_id, record_time, alarms, rescans, scans. |
Use device_id to confirm that messages belong to the selected WTMD. signal_value and scan_zones are raw fields that ScanBI does not currently use for WTMD monitoring; do not treat their absence from a dashboard as data loss.
5.8 Record retention and pruning
WTMD scanner records are retained for 90 days by default. The advanced option max_history_scanbi_records sets the retention days for all ScanBI records, including QPS. A cron job prunes records every four hours.
The updated support note documents manual pruning with:
/usr/local/statseeker/scs/bin/scanbi_prune_records -t <unix-timestamp>
This command deletes historical records. It is not a troubleshooting command and must not be run during investigation without explicit data-retention/change approval. When used without -t, it applies the configured retention value; if no value is configured, it defaults to 90 days before the time it runs.
5.10 WTMD escalation evidence
- WTMD IP, serial, device ID, real/simulated status, and ScanBI server time.
- Ping result and a short passive TCP/10002 capture summary.
- Whether a single active connection was confirmed or a competing connection is suspected.
- Collector/processor process status and any approved debug window.
- Sanitised collector/processor log lines with timestamp and severity.
- Queue observation showing selected device_id, scan/minute-total timing, or absence of expected messages.
- device_wtmd and device_passenger_scanner metric responses where both were checked.
- Retention configuration only if historical-record availability is relevant.
6. RTT Baggage Scanner
6.1 Current architecture and supported file-delivery method
RTT data is delivered to ScanBI as CSV reports through SFTP.
|
Item |
Value |
|---|---|
|
Fileshare location |
/home/scanbi/rtt/csv_reports |
|
SFTP user |
scanbi |
|
SFTP password |
Configured in ScanBI Administration: OS Configuration -> Passwords |
|
Supported delivery method |
SFTP |
|
Expected delivery interval |
Every 15 minutes |
CSV files may be placed anywhere beneath the csv_reports directory, including user-created subdirectories.
ScanBI recognises these file types, using a case-insensitive filename match:
Sessionlog_YYYYMMDD-HHMMSS.csv ThreatDetection_YYYYMMDD-HHMMSS.csv FindBag_YYYYMMDD-HHMMSS.csv EventLog_YYYYMMDD-HHMMSS.csv
6.2 Information required
- RTT scanner IP address, hostname, serial number, and Statseeker device ID.
- Serial number is mandatory and is a primary key for data being mapped to the device.
- Expected report-generation host and current delivery interval.
- ScanBI server time and timezone.
- Timezone and NTP status of the RTT report-generation system.
- Latest expected data time and whether a controlled scan can be performed.
6.3 Check scanner reachability and scan metrics
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
/sbin/ping -c 4 -W 1000 <rtt-ip>
Query device_rtt.scans for the exact selected RTT device ID.
"device_filter": { "filter": {"query": "==(<rtt-device-id>)"}, "hide": true, "field": "id" }
A reachable RTT scanner can still have an unhealthy central data-export feed. Continue with report-delivery checks when scan data is missing or stale.
6.4 Validate SFTP access and delivery activity
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
The report-generation system should transfer reports using the scanbi SFTP account. The account password is managed through:
ScanBI Administration -> OS Configuration -> Passwords
Check recent successful logins:
grep -E 'Accepted (publickey|password) for scanbi' /var/log/auth.log | tail -n 50 printf 'Today successful scanbi logins: ' grep "$(date '+%b %e')" /var/log/auth.log \ | grep -E 'Accepted (publickey|password) for scanbi' \ | wc -l
The source IP in auth.log belongs to the report-generation or data-dump server, not necessarily an individual RTT scanner.
Check port 22 from the report-generation host, or an approved host on the same network path:
nc -vz -w 5 <scanbi-server-ip> 22
On ScanBI, confirm SSH is listening:
sockstat -4 -l | grep sshd
If authentication is failing, validate the configured scanbi password, source connectivity, SFTP permissions, and report-generation service. Do not reset passwords or alter SSH configuration without approval.
6.5 Find the newest reports
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Reports may exist in subdirectories, so search recursively and case-insensitively.
REPORTS='/home/scanbi/rtt/csv_reports' for type in EventLog FindBag Sessionlog ThreatDetection; do echo "--- $type ---" find "$REPORTS" -type f -iname "${type}_*.csv" \ -exec ls -1t {} + 2>/dev/null | head -n 1 done
Inspect each returned file:
ls -lhT '<report-file>'
A report filename timestamp must use:
YYYYMMDD-HHMMSS
ScanBI ignores files with timestamps more than 30 minutes in the past, except for the expected daylight-saving transition handling. It also ignores a file whose timestamp is older than a previously processed file, except for the one-hour backward daylight-saving change.
A file can be successfully copied to the share but still be ignored because its embedded filename timestamp is old or out of sequence.
6.6 RTT file and row acceptance rules
ScanBI accepts data only when both the file and its rows meet the RTT requirements.
|
Rule |
Behaviour |
|---|---|
|
Filename |
Must match a supported report type and timestamp format, case-insensitively. |
|
File timestamp |
Files more than 30 minutes old are ignored. |
|
File sequence |
A file older than an already processed file is ignored, except for the daylight-saving one-hour rollback. |
|
Row timestamp |
Rows more than 30 minutes old are ignored. |
|
Row ordering |
Rows do not need to be time-sorted within a file. |
|
Row sequence |
Rows older than those in the last processed CSV are ignored. |
|
Required fields |
Rows with missing required columns or empty required values are ignored. |
|
CSV headers |
Required headers are matched case-insensitively. |
Reports may be delivered more frequently than every 15 minutes, such as every 1 or 5 minutes, provided each subsequent report has an increasing timestamp. Reports should not be sent less frequently than every 30 minutes because older data will be ignored.
Data should normally be available in ScanBI dashboards within approximately one minute of a valid report being copied to the SFTP file location.
6.7 Validate time and daylight-saving behaviour
The ScanBI system timezone must match the timezone used in RTT CSV filenames and row timestamps.
For example, if ScanBI uses Australia/Sydney, the report-generation system must produce report times in Australia/Sydney. Both systems should use NTP or an equivalent time-synchronisation mechanism.
ScanBI handles daylight-saving transitions between report files. It assumes all timestamps inside one CSV file use the same daylight-saving state. Do not generate a single report containing rows from both sides of a daylight-saving shift.
If timezone handling is difficult to control, use UTC consistently for both ScanBI and the report-generation system.
6.8 Validate the selected RTT serial in FindBag
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
Use the newest FindBag report and confirm that its Serial No field contains the selected RTT serial number.
FINDBAG=$(find /home/scanbi/rtt/csv_reports -type f -iname 'FindBag_*.csv' \ -exec ls -1t {} + 2>/dev/null | head -n 1) SERIAL='<rtt-serial-number>' /usr/local/bin/python3.11 - "$FINDBAG" "$SERIAL" <<'PY' import csv import re import sys path, wanted = sys.argv[1:] normalise = lambda text: (text or "").strip().casefold() with open(path, encoding="utf-8-sig", errors="replace", newline="") as source: reader = csv.DictReader(source) if not reader.fieldnames: raise SystemExit("FindBag header missing") column = next( ( field for field in reader.fieldnames if field and re.sub(r"[^a-z0-9]", "", field.casefold()) == "serialno" ), None, ) if not column: raise SystemExit("Serial No column missing") count = sum( 1 for row in reader if normalise(row.get(column)) == normalise(wanted) ) print("matching FindBag records:", count) PY
|
Finding |
Meaning |
Manual follow-up |
|---|---|---|
|
Current SFTP logins, fresh reports, matching serial |
RTT data collection is healthy for the selected device. |
Investigate dashboards, metrics, or downstream consumers if impact remains. |
|
No recent scanbi SFTP login |
Report-generation host may be stopped, blocked, or unable to authenticate. |
Check scheduler/service, credentials, network path, and system time. |
|
Report exists but is ignored |
Filename timestamp, file ordering, row timestamps, or required fields are invalid. |
Validate report generation and timestamps. |
|
One or more report types are missing |
Export or upload process is incomplete. |
Check RTT report generation and SFTP transfer configuration. |
|
FindBag lacks selected serial |
Selected RTT has not contributed expected data, or the inventory serial is incorrect. |
Validate serial, scanner activity, report content, and time window. |
6.9 Archive location and retention
ScanBI compresses and archives up to 90 days of RTT CSV reports for support investigation.
/home/scanbi/rtt/csv_reports_archive
Use the archive to investigate historical delivery, timestamp, serial, or parser issues. Do not modify or remove archived files during a support investigation without data-retention approval.
6.10 RTT escalation evidence
- RTT IP, hostname, serial number, and exact per-device scan result.
- Latest successful scanbi SFTP login, source IP, login count, and longest observed gap.
- ScanBI timezone and report-generation-system timezone.
- Newest EventLog, FindBag, Sessionlog, and ThreatDetection filenames, paths, sizes, and timestamps.
- Whether files were in the expected order and within the 30-minute acceptance window.
- FindBag serial-match count and any missing-header or required-column evidence.
- Relevant SFTP, SSH, report-generation, or clock/NTP changes.
7. RTT Workstation
7.1 Manual session-data validation
These types of checks require direct server access to the ScanBI server and also require knowledge and skills working on the CLI and using VI and python.
RTT Workstations are validated against the newest non-empty Sessionlog report. First locate it:
REPORTS='/home/scanbi/rtt/csv_reports' SESSIONLOG=$(find "$REPORTS" -type f -name 'Sessionlog_*.csv' -exec ls -1t {} + 2>/dev/null | head -n 1) printf 'Latest Sessionlog: %s\n' "$SESSIONLOG" ls -lhT "$SESSIONLOG"
Confirm the selected Statseeker serial appears in the Serial Number column and return the latest matching login/logout record:
SERIAL='<rtt-workstation-serial>' /usr/local/bin/python3.11 - "$SESSIONLOG" "$SERIAL" <<'PY' import csv import re import sys from datetime import datetime path, wanted = sys.argv[1:] normalise = lambda text: (text or '').strip().casefold() with open(path, encoding='utf-8-sig', errors='replace', newline='') as source: reader = csv.DictReader(source) if not reader.fieldnames: raise SystemExit('Sessionlog header missing') column = next((field for field in reader.fieldnames if field and re.sub(r'[^a-z0-9]', '', field.casefold()) == 'serialnumber'), None) if not column: raise SystemExit('Serial Number column missing') rows = [row for row in reader if normalise(row.get(column)) == normalise(wanted)] if not rows: raise SystemExit('No matching workstation serial records') def login_time(row): try: return datetime.strptime(row.get('Login Time', ''), '%d/%m/%Y %H:%M:%S') except ValueError: return datetime.min latest = max(rows, key=login_time) print('matching records:', len(rows)) print('latest login:', latest.get('Login Time', '')) print('latest logout:', latest.get('Logout Time', '')) PY
|
Finding |
Meaning |
Manual follow-up |
|---|---|---|
|
Current Sessionlog and matching serial |
Workstation data is reaching the shared RTT reports. |
Compare latest login/logout to reported user activity. |
|
No Sessionlog |
Upstream RTT reporting/dump feed is missing. |
Use RTT scanner section 6. |
|
Header/Serial Number column missing |
File schema changed or is malformed. |
Preserve file name/header; escalate integration/schema issue. |
|
No serial match |
Wrong inventory serial, unused workstation, or missing upstream data. |
Verify physical serial and expected user/session time. |
|
Sessionlog stale |
Upstream report feed delayed. |
Investigate dump cadence and source generation. |
As with RTT scanner reports, a Sessionlog can be in a subdirectory. A manual match in a subdirectory is valid under the Administrator Reference even if the current CGI workstation check does not find it.
8. 4DX trace scanner
8.1 Information required
- Device hostname, IPaddress, serial number, and expected scan activity.
- Serial number is mandatory and is a primary key for data being mapped to the device.
- Current collector deployment/change history.
- The configured Rapiscan Remote Connect Console (RCC) network and the approved credential owner.
The Administrator Reference describes 4DX collection as an integration with Rapiscan RCC API services. It requires one or more defined networks and credentials for those resources. The supplied documents do not define a supported RCC host, port, API path, or authentication exchange for manual replay.
Do not invent an RCC curl, nc, or credential test. Confirm the configured network and credential assignment in the approved ScanBI configuration/change record, then obtain the RCC integration-specific procedure from Rapiscan if a direct endpoint test is required.
8.2 Check reachability and collector errors
/sbin/ping -c 4 -W 1000 <4dx-ip> tail -n 5000 /home/statseeker/base/logs/python.log \ | grep 'scanbi_4dx_collector' \ | grep -E 'WARNING|ERROR|FATAL'
Narrow to a serial, hostname, IP, timeout, connection, authentication, parse, or vendor error code:
grep 'scanbi_4dx_collector' /home/statseeker/base/logs/python.log \ | grep -i '<search-term>' \ | tail -n 100
No recent error does not prove that the 4DX integration is healthy; it only proves that no matching severity event exists in the inspected recent log lines.
8.3 Manually scope 4DX scans by serial
4DX scan totals must be scoped to the scanner serial and the current operator names assigned to that serial. Do not use a hard-coded dashboard/operator list.
First retrieve the live operators for the selected serial:
cat >/tmp/4dx-operators.json <<'EOF' { "command": "get", "user": "admin", "objects": [{ "type": "trace_detection_scanner_operator", "fields": { "traceScannerOperatorName": {}, "serial_filter": { "field": "serial", "filter": {"query": "= '<4dx-serial-number>'"}, "hide": true } }, "limit": 0 }] } EOF /usr/local/statseeker/ss/bin/nim-api -f /tmp/4dx-operators.json | jq .
Then build a metric request using both the same serial filter and an IN ('operator1','operator2',...) filter over the returned live operator names. The metric field is traceScannerOperatorScans, with formats: ["total"] and the time filter range = start_of_today to now;.
Interpretation:
|
Result |
Meaning |
Manual follow-up |
|---|---|---|
|
Operators returned and total positive |
Data is stored for the selected serial today. |
Compare with expected operational activity. |
|
Operators returned and grouped total is null/zero |
Valid query found no scans in range. |
Confirm operating hours before treating as fault. |
|
No operators for serial |
No current operator data can be scoped to serial. |
Validate inventory serial and 4DX source data/collector. |
|
API request failure or no metric row |
Metric/data path is unavailable. |
Capture raw nim-api status and inspect collector logs. |
8.4 4DX escalation evidence
- Device IP/serial and ping output.
- Raw operator lookup/metric response status (sanitised).
- Expected activity and time window.
- Relevant scanbi_4dx_collector warning/error/fatal lines with timestamps.
- Network/device and collector-change history.
- Confirmed RCC network and credential configuration status, without exposing credential values.
9. Manual troubleshooting patterns for all collectors
9.1 Confirm file/directory readability without changing data
ls -ld /home/scanbi/shared /home/scanbi/rtt/csv_reports ls -l /home/statseeker/base/logs/base-logd.log /home/statseeker/base/logs/python.log df -h /home/scanbi/shared /home/scanbi/rtt/csv_reports /home/statseeker/base/logs
Look for missing paths, read/traverse permission issues, full filesystems, unexpected ownership changes, or modification times inconsistent with the incident. Do not add write permission as a diagnostic shortcut.
9.2 Distinguish data states correctly
|
State |
Meaning |
|---|---|
|
Network reachable, no current data |
Device transport is available; source/export/collector/metric may still be broken. |
|
Current file/report, no metric |
Data reached disk but collector/parser/metric ingestion may be failing. |
|
Current metric, no current file |
Metric may come from a separate path; verify device type and data source before assuming contradiction. |
|
Old file/report and stale SSH/SMB session |
Likely upstream export/transfer issue. |
|
Serial mismatch |
Data identity/routing issue; never solve by renaming production folders/files. |
|
Healthy endpoint but no scans |
API/network path works; investigate device activity, collector data, and time window. |
9.3 Evidence template for escalation
Site / terminal / lane: First observed (time zone): User impact / expected activity: Device identity: - Hostname: - IP: - Equipment / model / product line: - Serial: Manual checks: - ScanBI server time: - Ping result: - TCP port result (if applicable): - API / SMB / SSH / report / protocol result: - Exact per-device nim-api metric result: Data/log evidence (sanitised): - Folder/report/file name, size, and modification timestamp: - Serial match/count: - Collector component, severity, timestamp, message: Recent network/device/certificate/collector changes: Requested escalation action:
10. Support hand-off rules
Escalate to the appropriate owner with captured evidence when:
- device power, physical hardware, vendor-specific protocol, or device application action is required;
- a firewall, route, VLAN, DNS, certificate, credential, or server service change is required;
- CT API authentication or API contract is rejected;
- QPS/RTT data has serial mismatches or malformed schema;
- a collector reports repeated error/fatal events or metrics stop despite confirmed source data;
- filesystems, permissions, sudoers, or web/Statseeker runtime paths are missing/incorrect.
Keep manual investigation read-only until the responsible owner approves a corrective change.