kwutil.util_resources module

Utilities to configure / inspect system resources

TODO Reorganization:

  • Add class called Software to mirror Hardware that lets you inspect and perhaps modify your software environmental configuration. Could also name this Runtime or something like that.

kwutil.util_resources.request_nofile_limits(request_rlimit_nofile='auto')[source]

Attempts to increase the limit of open file descriptors on the system.

Does nothing on non-linux operating systems.

Parameters:

request_rlimit_nofile (int | str) – attempt to increase rlimit_nofile to this number of open files, if auto, uses a heuristic, which currently defaults to 8192.

Example

>>> from kwutil import util_resources
>>> util_resources.request_nofile_limits()
kwutil.util_resources.get_shm_size()[source]

Get the size of the shared memory filesystem.

Returns:

the size of shared memory in bytes

Return type:

int

Raises:

IOError – if /dev/shm does not exist

kwutil.util_resources.check_shm_limits(threshold_gibibytes=1)[source]

When running training in docker you may need more shared memory than its default provide. Warns if shared memory is not above a threshold.

Parameters:

threshold_gibibytes (float) – The amount of shared memory we want to have access to.

Notes

Running docker run with --shm-size=32768m increases shm to 32GB

Raises:

IOError – if the system does not have a /dev/shm device.

References