Article #000015802

Last modified on Dec 28th 2023

Summary:

LCM Auto Inventory Task may not be triggered if the Auto Inventory Schedule is configured is between 12 and 12:30 am cluster time.

Versions affected:

ALL AOS Version

LCM

Description:

When LCM Auto Inventory run time is scheduled between 12:00 am and 12:30 am cluster time, the Auto Inventory task may not be triggered. 

The LCM Auto Inventory script runs every 60 minutes to check if the configured next update time falls under a certain time range.
A thread in LCM runs every hour at a certain minute of the hour (+/- 30 minutes from the scheduled time) to check if the current time (including the current date falls) under the scheduled time range generated.  

For example, in the below case, the next configured run is stored as 00:02, meaning 12:02 AM cluster time (CVM time):

nutanix@CVM:~$ configure_lcm -p | grep next
next_update: 00:02

For the above example, +/- 30 minutes from the scheduled time will be from the previous day 11:32 pm, to the current day, 12:32 am.
There might be an instance where the time range-based frequency check might run before 12:00 am (let’s say the script invokes every hour at the 58th minute, you can see below every 58th minute we see the script running)

Finding the LCM leader:

nutanix@CVM:~$ lcm_leader
Lcm leader is at <CVM_IP>

This prints the IP of the LCM leader.

Below are the genesis logs from the lcm_leader:

nutanix@CVM:~$less  ~/data/logs/genesis.out |grep  betweenscheduled
2023-11-09 18:58:46,918Z INFO 83000816 auto_inventory.py:61 Not triggering auto inventory: current time is not betweenscheduled time range
2023-11-09 19:58:46,926Z INFO 83000816 auto_inventory.py:61 Not triggering auto inventory: current time is not betweenscheduled time range
2023-11-09 20:58:46,934Z INFO 83000816 auto_inventory.py:61 Not triggering auto inventory: current time is not betweenscheduled time range
2023-11-09 21:58:46,942Z INFO 83000816 auto_inventory.py:61 Not triggering auto inventory: current time is not betweenscheduled time range
2023-11-09 22:58:46,950Z INFO 83000816 auto_inventory.py:61 Not triggering auto inventory: current time is not betweenscheduled time range
2023-11-09 23:58:46,950Z INFO 83000816 auto_inventory.py:61 Not triggering auto inventory: current time is not betweenscheduled time range

Even though 23:58 falls between +/- 30mins from the scheduled time, Because of the Date issue, we never fall within the same date range as of LCM auto Inventory schedule. Hence, LCM auto inventory will not be triggered.  

Solution:

Nutanix Engineering is aware of the issue and will be fixed in future versions.

Workaround: 
Nutanix recommends avoiding scheduling Auto Inventory between 12:00 and 12:30 am cluster time.

Adjust the auto_inventory script run time not to be later than at the 58th minute.

For example, at the 9th minute of every hour (i.e., anytime between the 1st Minute and 29th minute of the hour). This can be achieved by restarting genesis cluster-wide anytime between 12:01 am and 12:29 am so that the auto_inventory script runs every 09th minute (2 mins for LCM to start up). After this, inventory would run at 12:09 am.

nutanix@cvm$ allssh genesis restart

The workflow may get overwritten if genesis is restarted cluster-wide again. 
For example, if genesis is restarted at the 40th minute of the hour, then the issue will resurface.

SOURCE: https://portal.nutanix.com/page/documents/kbs/details?targetId=kA07V0000010zc2SAA