# Health & Diagnostics

# NCC & Logs

#### NCC

Run a full health check pass

```bash
ncc health_checks run_all
```

Show installed NCC version

```bash
ncc --version
```

Hihi

```
ncc 1701
```

#### Logs

Create a log bundle for Nutanix support

```bash
logbay collect
```

#### Logs Location

Per-component logs on a CVM (latest file + list all, same pattern per service)

```bash
/home/nutanix/data/logs/zookeeper.out
/home/nutanix/data/logs/genesis.out
/home/nutanix/data/logs/curator.out
/home/nutanix/data/logs/stargate.out
/home/nutanix/data/logs/cerebro.out
/home/nutanix/data/logs/chronos.out
/home/nutanix/data/logs/prism_gateway.log
ls -ltrh /home/nutanix/data/logs/<component>*.*
```

List all FATAL logs across all CVMs

```bash
for i in `svmips` ; do echo "CVM: $i" ; ssh $i "ls -ltr /home/nutanix/data/logs/*.FATAL" ; done
```

Check for Cassandra crashes / restarts

```bash
grep "Logging initialized" /home/nutanix/data/logs/cassandra/*
```

Upgrade history on a CVM

```bash
/home/nutanix/config/upgrade.history
```

# Connectivity & CVM Basics

#### Connectivity

Quick check: which cluster services aren't running

```bash
cluster status | grep -v UP
```

Test network connectivity between CVMs

```bash
ping_cvms
```

#### CVM Basics

Gracefully shut down a CVM

```bash
cvm_shutdown -P now
```

Check whether the metadata store is enabled on a host

```bash
ncli host ls | grep "Metadata store status"
```

Where CVM packet-drop / latency issues get logged

```bash
/home/nutanix/data/logs/sysstats/ping_*.INFO
```

# Component Diagnostics & Ports

#### Component Diagnostics Pages

Each AOS component exposes a local debug page — open it locally on the CVM, or remotely once the port is opened (see below)

```bash
links http://0:<PORT>
links http://<CVM_IP>:<PORT>
```

Ports: 2009 Stargate/Cassandra (+ /latency, /traces), 2010 Curator, 2012 Chronos, 2014 Alert Manager, 2016 Pithos, 2017 Stats Aggregator, 2020 Cerebro, 2025 Arithmos, 2031 Hyperint, 9877 Zookeeper

Open a port on all CVMs to access it remotely (example: 2100, used by Foundation)

```bash
for i in `svmips` ; do ssh $i "sudo iptables -t filter -A WORLDLIST -p tcp -m tcp --dport 2100 -j ACCEPT" ; done
```

Trigger a full Curator scan via its diagnostics page

```bash
http://curator_leader:2010/master/control
```

Alternative: vollen Curator-Scan per API auf allen CVMs anstoßen

```bash
for svm in `svmips`; do wget -O - "http://$svm:2010/master/api/client/StartCuratorTasks?task_type=2"; done
```

# Arithmos

#### Arithmos (Stats)

How much space Arithmos is using on all nodes

```bash
allssh du -sh ~/data/cassandra/data/arithmos*
```

Stop/start Arithmos cluster-wide

```bash
allssh '~/cluster/bin/genesis stop arithmos'
cluster start
```

Reset Arithmos statistics (stops Arithmos, Hyperint and Prism, clears stats, restarts)

```bash
allssh '~/cluster/bin/genesis stop arithmos'
allssh '~/cluster/bin/genesis stop hyperint'
allssh '~/cluster/bin/genesis stop prism'
allssh 'rm ~/data/arithmos/arithmos_per*'
cluster start
```

# General Process Troubleshooting

#### General Process Troubleshooting

Network connections, port state, PID and program name

```bash
netstat -tulpn
```

Find the working directory of a running process

```bash
ls -l /proc/<PID>/cwd
pwdx <PID>
```

Watch a component's PID to catch crash-restarts (PID changes = it crashed and came back)

```bash
watch pgrep <component>
```

Find which process opened a given TCP port

```bash
fuser 7000/tcp
```

Check if a port is a known/registered service

```bash
cat /etc/services
```

Find which web server is answering on a URL

```bash
curl -I <website>
```

Last cluster/CVM shutdown date and time

```bash
last -x | grep shutdown | head -1
```

# Panacea CLI – Setup & SSH

#### Setup

Scripts live in `$TOP/panacea/nutanix_log_analyzer/tools/` in the repo, or `<panacea install dir>/tools` once installed. Not on PATH by default (avoids name collisions with existing cluster scripts).

List available tools / help

```bash
panacea_cli --help
panacea_cli -l
```

Activate the tools environment on a live cluster (puts every tool on PATH)

```bash
source panacea_tools_env
```

Activate the tools environment for an extracted log bundle (sets TZ and LB\_ROOT\_DIR)

```bash
source log_bundle_env -p <path to extracted log bundle> -t <timezone>
```

#### Parallel SSH

allssh with parallel execution (live cluster only)

```bash
cssh <command>
OUTSYNC=0 cssh <command>   # faster, interleaved per-host output
```

hostssh with parallel execution (live cluster only)

```bash
hssh <command>
```

# Panacea CLI – Tools

#### Tools (panacea\_cli &lt;tool&gt;)

SVM, host and IPMI IPs from the Zeus config

```bash
show_all_ips
```

Formatted bash history across all CVMs

```bash
show_bash_history
```

Cassandra leader status (live cluster only)

```bash
show_cassandra_leadership
```

Formatted Cassandra nodetool ring status

```bash
show_cassandra_ring
```

Formatted Cassandra status history

```bash
show_cassandra_status_history
```

Controller latencies from Arithmos for the last 10 minutes, or live every 5 seconds (live cluster only)

```bash
show_controller_latencies
show_controller_latencies --live
```

Reboot history per node

```bash
show_reboots
```

Configured snapshot schedules (live cluster only)

```bash
show_snapshot_schedules
```

Formatted zeus\_config dump

```bash
show_zeus_config
```

Service leaders in the cluster (NCC 4.4+)

```bash
show_leaders
```

Node list from the Zeus config (NCC 4.4+)

```bash
show_node_list
```

Disk list from the Zeus config (NCC 4.4+)

```bash
show_disk_list
```

Cassandra token/shard mapping per node (NCC 4.4+)

```bash
show_cassandra_node_config
```

More tools show up in `panacea_cli -l` but aren't individually documented in this KB — check `--help` per tool: `ping_ips`, `fping_ips`, `collect_calm_info.py`, `show_storage_pool_usage`

# Panacea CLI – Analyzers & Formatters

#### Analyzer Tools

Kompletten Report für ein Log-Bundle generieren — übersichtlich nach Services und CVMs sortiert zum Durchklicken

```bash
panacea run
```

Detailed storage usage breakdown for a container (garbage/overhead, VM/VG/image usage, snapshots)

```bash
panacea_cli summ.py --container_name=<container> [--mode calculate_usage|ui|skip_cache] [--days <n>]
```

Root-cause analysis for failed AHV workflows: AHV upgrade, VM migration, AOS upgrade, Backplane Network Segmentation

```bash
panacea_cli ahv_analyzer.py --workflow=<ahv_upgrade|vm_migrate|aos_upgrade|ns_backplane_segmentation> --env=live
panacea_cli ahv_analyzer.py --workflow=<...> --env=log --path=<log bundle path>
```

Automated checks to help root-cause a degraded node (runs a sequence of NCC checks)

```bash
panacea_cli dnd_analyzer.py
```

Real-time and historic CPU/memory/IO stats for the running cluster (NCC 4.6+)

```bash
panacea_cli narf.py
```

#### Formatters (tools/formatters/)

Work on a live cluster or a log bundle, usually piped after the raw command

```bash
<command> | format_<name>
```

SVM/host/IPMI IP table from a Zeus config excerpt

```bash
awk '/<timestamp>/,/^$/' data/logs/sysstats/zeus_config.INFO | format_all_ips
```

Formats `all_ssh.sh` output per host

```bash
TARGET=svms OUTSYNC=0 all_ssh.sh <command> | format_all_ssh
```

Bash history with readable timestamps

```bash
format_bash_history ~/.bash_history
```

Stargate activity traces (Op + parameters on one line)

```bash
links -dump "http://0:2009/h/traces?low=0&high=&expand=&a=completed&c=all" | format_activity_traces
```

Arithmos time-range stats with readable timestamps

```bash
arithmos_cli master_get_time_range_stats ... | format_arithmos_time_range_stats
```

Audit log with readable timestamps

```bash
sudo cat /home/log/audit/audit.log | format_audit_logs
```

Cassandra status history from a Zeus config sample

```bash
zeus_config_printer | format_cassandra_status_history
```

dmesg with readable timestamps

```bash
dmesg | format_dmesg_logs
```

Sysstat logs (12h to 24h, timestamp prepended)

```bash
cat data/logs/sysstats/cassandra_gc_stats.INFO | format_sysstat_logs
```

zeus\_config with helpful annotations

```bash
zeus_config_printer | format_zeus_config
```