sensor: Implement sensor "ASYNC_READ_TIMEOUT"
This change will prevent sensors from blocking all other sensor reads
and D-Bus if they do not report failures quickly enough.
If "ASYNC_READ_TIMEOUT" environment variable is defined in the
sensor's config file for a key_type, the sensor read will be
asynchronous with timeout set in milliseconds.
For example for "sensor1":
ASYNC_READ_TIMEOUT_sensor1="1000" // Timeout will be set to 1 sec
If the read times out, the sensor read will be skipped and the
sensor's functional property will be set to 'false'. Timed out futures
will be placed in a map to prevent their destructor from running and
blocking until the read completes (limitation of std::async).
Tested: This patch has been running downstream for over a year to
solve a slow I2C sensor reads causing IPMI slowdown.
Change-Id: I3d9ed4d5c9cc87d3196fc281451834f3001d0b48
Signed-off-by: Brandon Kim <brandonkim@google.com>
diff --git a/mainloop.cpp b/mainloop.cpp
index 75c8c12..0b6d070 100644
--- a/mainloop.cpp
+++ b/mainloop.cpp
@@ -34,6 +34,7 @@
#include <cassert>
#include <cstdlib>
#include <functional>
+#include <future>
#include <iostream>
#include <memory>
#include <phosphor-logging/elog-errors.hpp>
@@ -242,7 +243,7 @@
{
// Add status interface based on _fault file being present
sensorObj->addStatus(info);
- valueInterface = sensorObj->addValue(retryIO, info);
+ valueInterface = sensorObj->addValue(retryIO, info, _timedoutMap);
}
catch (const std::system_error& e)
{
@@ -484,10 +485,28 @@
// RAII object for GPIO unlock / lock
auto locker = sensor::gpioUnlock(sensor->getGpio());
- // Retry for up to a second if device is busy
- // or has a transient error.
- value = _ioAccess->read(sensorSysfsType, sensorSysfsNum, input,
+ // For sensors with attribute ASYNC_READ_TIMEOUT,
+ // spawn a thread with timeout
+ auto asyncReadTimeout =
+ env::getEnv("ASYNC_READ_TIMEOUT", sensorSetKey);
+ if (!asyncReadTimeout.empty())
+ {
+ std::chrono::milliseconds asyncTimeout{
+ std::stoi(asyncReadTimeout)};
+ value = sensor::asyncRead(
+ sensorSetKey, _ioAccess, asyncTimeout, _timedoutMap,
+ sensorSysfsType, sensorSysfsNum, input,
+ hwmonio::retries, hwmonio::delay);
+ }
+ else
+ {
+ // Retry for up to a second if device is busy
+ // or has a transient error.
+ value =
+ _ioAccess->read(sensorSysfsType, sensorSysfsNum, input,
hwmonio::retries, hwmonio::delay);
+ }
+
// Set functional property to true if we could read sensor
statusIface->functional(true);