This is a depressing – and all-too-common – scenario: a user runs the free command and opens a support case because he sees zero “free” memory on the server.

For years I’ve been trying to explain to users – developers, application engineers – how memory allocation works in Linux. It’s an utterly futile endeavor. Sometimes I just want to soft-link the free command to exit.

Starting with RHEL 7, we’ve been blessed with an updated version of free that now adds the Available column to the output. This helps sometimes, but users are still confused as they don’t understand the difference between “free” and “available”.

Here’s output of free on RHEL 6:

root@hecuba# free -h
             total       used       free     shared    buffers     cached
Mem:           62G        61G       1.1G       677M       951M        39G
-/+ buffers/cache:        21G        41G
Swap:         8.0G       804M       7.2G

And here’s the output of free on RHEL 7:

[root@kassandra~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        478M        400M        9.5M        6.7G        6.8G
Swap:          2.0G          0B        2.0G

I thought this was a problem of presentation and so I set out to write a quick script that would detail current system memory utilization in a way that your average developer can understand.

It also occurred to me that instead of relying on local system tools to gather this information, I should use snmpwalk. This would allow for wider access and more standardized presentation. And so I picked a major monitoring software vendor – Solarwinds – and decided to stick with its standards.

According to Solarwinds documentation, the percentage of “used memory” is calculated like so:

Used Memory = (Total RAM in machine - Total RAM Available - Cached Memory - RAM Buffered) / Total RAM * 100

Here’s a practical example showing how to calculate the percentage of used memory using SNMP data:

h=<target_host>
s=<snmp_community_string>

# Define relevant OIDs
total_ram='1.3.6.1.4.1.2021.4.5'
total_ram_available='1.3.6.1.4.1.2021.4.6'
total_ram_buffered='1.3.6.1.4.1.2021.4.14'
total_cached_memory='1.3.6.1.4.1.2021.4.15'

# Configure snmpwalk function
walk() {
  snmpwalk -v 2c -c $s $h $@
}

# Configure output formatting
chomp() {
  awk -F: '{print $NF}' | grep -oP '[0-9]{1,}'
}

[root@devsw01 ~]#
echo "Used Memory: $(echo "scale=0;($(walk ${total_ram} | chomp)\
> -$(walk ${total_cached_memory} | chomp)\
> -$(walk ${total_ram_buffered} | chomp)) * 100 / $(walk ${total_ram} | chomp)"|bc -l)%"

Used Memory: 32%

The script below (also on GitHub) shows reasonably detailed memory utilization information that can be used to calm a panicked mind. If this doesn’t help, check with the management to see if the mind in question can be better utilized elsewhere.

#!/bin/bash
#
#                                      |
#                                  ___/"\___
#                          __________/ o \__________
#                            (I) (G) \___/ (O) (R)
#                                   Igor Os
#                           igor@comradegeneral.com
#                                 2020-04-13
# ----------------------------------------------------------------------------
# Retrieve a server's memory utilization information via SNMP
# ----------------------------------------------------------------------------
# CHANGE CONTROL
# ----------------------------------------------------------------------------
# 2020-04-13  igor  wrote this script

h=""
if [ -z "${h}" ]
then
  echo "You need to supply this command with a valid hostname. Exiting..."
  exit 300
else
  ping_status=$(timeout 6 ping -c 2 -W 1 ${h} >/dev/null 2>&1 ; echo $?)
  if [ -z "${ping_status}" ] || [ ${ping_status} -ne 0 ]
  then
    echo "Unable to ping ${h}. Exiting..."
    exit 310
  fi
fi

configure() {
  total_ram='1.3.6.1.4.1.2021.4.5'
  total_ram_available='1.3.6.1.4.1.2021.4.6'
  total_ram_buffered='1.3.6.1.4.1.2021.4.14'
  total_cached_memory='1.3.6.1.4.1.2021.4.15'
  # Update the SNMP community string. Make sure the script has appropriate permissions
  s="*********************"

  RED='3[1;31m'
  GREEN='3[1;32m'
  YELLOW='3[1;33m'
  NC='3[0m'

  hfqdn="$(nslookup $h 2>/dev/null | grep -m1 '^Name:' | awk '{print $NF}')"
  if [ -z "${hfqdn}" ]
  then
    hfqdn="${h}"
  fi
}

walk() {
  snmpwalk -v 2c -c $s $h $@
}

chomp() {
  awk -F: '{print $NF}' | grep -oP '[0-9]{1,}'
}

mem_check() {
  echo "-------------------------------------"
  echo "${hfqdn}"
  echo "-------------------------------------"
  echo -e "Total RAM installed:\t ~$(echo "scale=0;($(walk ${total_ram} | chomp))/1024/1024"|bc -l)GB"
  echo -e "Allocated memory:\t ~$(echo "scale=0;($(walk ${total_ram} | chomp)\
  -$(walk ${total_ram_available} | chomp))/1024/1024"|bc -l)GB"
  echo -e "Unallocated memory:\t ~$(echo "scale=0;($(walk ${total_ram_available} | chomp))/1024/1024"|bc -l)GB"

  echo -e "Utilized memory:\t ~$(echo "scale=0;($(walk ${total_ram} | chomp)\
  -$(walk ${total_cached_memory} | chomp)\
  -$(walk ${total_ram_buffered} | chomp))/1024/1024"|bc -l)GB ($(echo "scale=0;($(walk ${total_ram} | chomp)\
  -$(walk ${total_cached_memory} | chomp)\
  -$(walk ${total_ram_buffered} | chomp)) * 100 / $(walk ${total_ram} | chomp)"|bc -l)%)"

  available_pct="$(echo "scale=0;($(walk ${total_ram_available} | chomp)\
  +$(walk ${total_cached_memory} | chomp)\
  +$(walk ${total_ram_buffered} | chomp)) * 100 / $(walk ${total_ram} | chomp)"|bc -l)"

  if [ ${available_pct} -gt 40 ]
  then
    highlight="${GREEN}"
  elif [ ${available_pct} -le 40 ] && [ ${available_pct} -gt 15 ]
  then
    highlight="${YELLOW}"
  elif [ ${available_pct} -le 15 ]
  then
    highlight="${RED}"
  fi

  echo ""
  echo -e "${highlight}Available memory:\t ~$(echo "scale=0;($(walk ${total_ram_available} | chomp)\
  +$(walk ${total_cached_memory} | chomp)\
  +$(walk ${total_ram_buffered} | chomp))/1024/1024"|bc -l)GB ($(echo "scale=0;($(walk ${total_ram_available} | chomp)\
  +$(walk ${total_cached_memory} | chomp)\
  +$(walk ${total_ram_buffered} | chomp)) * 100 / $(walk ${total_ram} | chomp)"|bc -l)%)${NC}"

  echo ""
  echo "Available memory composition:"
  echo -e "\tCached:   \t ~$(echo "scale=0;($(walk ${total_cached_memory} | chomp))/1024/1024"|bc -l)GB"
  echo -e "\tBuffered:\t ~$(echo "scale=0;($(walk ${total_ram_buffered} | chomp))/1024/1024"|bc -l)GB"
  echo -e "\tUnallocated:\t ~$(echo "scale=0;($(walk ${total_ram_available} | chomp))/1024/1024"|bc -l)GB"
  echo "-------------------------------------"
}

# ----------------------------------------------------------------------------
# RUNTIME
# \(^_^)/                                      __|__
#                                     __|__ *---o0o---*
#                            __|__ *---o0o---*
#                         *---o0o---*
# ----------------------------------------------------------------------------
configure
mem_check

And sample output:

[root@devsw01~]# memwalk hecuba
-------------------------------------
hecuba.kc16.local
-------------------------------------
Total RAM installed:     ~62GB
Allocated memory:        ~62GB
Unallocated memory:      ~0GB
Utilized memory:         ~22GB (35%)

Available memory:        ~41GB (65%)

Available memory composition:
        Cached:          ~40GB
        Buffered:        ~0GB
        Unallocated:     ~0GB
-------------------------------------