System with 1.25 TB memory OS only seeing 1.00 TB ram

Question

I have a System with 1.25 TB of ram the Mother board sees it , lshw see's it , but only 1.0 TB is useable, Is there a kernel Parameter I need to recognize more memory ?

It is an Asus Mother board RS700-E9 , with 2 Intel XEON Gold 6150 CPU's

[root@xtc ~]# lshw -C memory
  *-firmware
       description: BIOS
       vendor: American Megatrends Inc.
       physical id: 0
       version: 0403
       date: 07/04/2017
       size: 64KiB
       capacity: 16MiB
       capabilities: pci apm upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi
  *-memory
       description: System Memory
       physical id: 31
       slot: System board or motherboard
       size: 1280GiB                 <------------- memory in system 
       capabilities: ecc
       configuration: errordetection=ecc
     *-bank:0
          description: DIMM DDR4 Synchronous 2400 MHz (0.4 ns)
          product: HMAA8GL7MMR4N-UH
          vendor: Hynix
          physical id: 0
          serial: 32BF004A
          slot: DIMM_A1
          size: 64GiB
          width: 64 bits
          clock: 2400MHz (0.4ns)
.
.
.

but can only 1 TB

[root@xtc ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.0T         10G        994G         19M        346M        993G
Swap:          4.0G          0B        4.0G
[root@xtc ~]#

[root@xtc ~]# dmidecode -t memory
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.

Handle 0x0031, DMI type 16, 23 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: Single-bit ECC
        Maximum Capacity: 768 GB     <---- thats Per CPU  
        Error Information Handle: Not Provided
        Number Of Devices: 6

Handle 0x0033, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0031
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 64 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_A1
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MT/s
        Manufacturer: Hynix
        Serial Number: 32BF004A
        Asset Tag: DIMM_A1_AssetTag
        Part Number: HMAA8GL7MMR4N-UH
        Rank: 4
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V

Handle 0x0035, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0031
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 64 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_A2
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2666 MT/s
        Manufacturer: Micron
        Serial Number: 353C140B
        Asset Tag: DIMM_A2_AssetTag
        Part Number: 72ASS8G72LZ-2G6J1
        Rank: 4
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V

Handle 0x0037, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0031
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 64 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_B1
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MT/s
        Manufacturer: Hynix
        Serial Number: 32BF00A6
        Asset Tag: DIMM_B1_AssetTag
        Part Number: HMAA8GL7MMR4N-UH
        Rank: 4
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V

Handle 0x0039, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0031
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 64 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_B2
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2666 MT/s
        Manufacturer: Micron
        Serial Number: 353C2294
        Asset Tag: DIMM_B2_AssetTag
        Part Number: 72ASS8G72LZ-2G6J1
        Rank: 4
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V

any ideas ????

****** adding the imxi -m output

 RAM: total: 1005.33 GiB used: 12.41 GiB (1.2%)
  Array-1: capacity: 768 GiB slots: 6 EC: Single-bit ECC
  Device-1: DIMM_A1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-2: DIMM_A2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-3: DIMM_B1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-4: DIMM_B2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-5: DIMM_C1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-6: DIMM_C2 type: no module installed
  Array-2: capacity: 768 GiB slots: 6 EC: Single-bit ECC
  Device-1: DIMM_D1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-2: DIMM_D2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-3: DIMM_E1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-4: DIMM_E2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-5: DIMM_F1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-6: DIMM_F2 type: no module installed
  Array-3: capacity: 768 GiB slots: 6 EC: Single-bit ECC
  Device-1: DIMM_G1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-2: DIMM_G2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-3: DIMM_H1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-4: DIMM_H2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-5: DIMM_J1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-6: DIMM_J2 type: no module installed
  Array-4: capacity: 768 GiB slots: 6 EC: Single-bit ECC
  Device-1: DIMM_K1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-2: DIMM_K2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-3: DIMM_L1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-4: DIMM_L2 type: DDR4 size: 64 GiB speed: spec: 2666 MT/s
  Device-5: DIMM_M1 type: DDR4 size: 64 GiB speed: 2400 MT/s
  Device-6: DIMM_M2 type: no module installed

adding proc/meminfo

# cat /proc/meminfo
MemTotal:       1054167812 kB
MemFree:        1040596332 kB
MemAvailable:   1040131352 kB
Buffers:            1400 kB
Cached:          1305460 kB
SwapCached:            0 kB
Active:          1495260 kB
Inactive:         452332 kB
Active(anon):     643724 kB
Inactive(anon):    33940 kB
Active(file):     851536 kB
Inactive(file):   418392 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      980955128 kB
SwapFree:       980955128 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        638656 kB
Mapped:           166396 kB
Shmem:             36088 kB
Slab:             268216 kB
SReclaimable:      90680 kB
SUnreclaim:       177536 kB
KernelStack:       22864 kB
PageTables:        43764 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    1508039032 kB
Committed_AS:    5051712 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     2269720 kB
VmallocChunk:   34081574908 kB
Percpu:            97280 kB
HardwareCorrupted:     0 kB
AnonHugePages:    221184 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      320044 kB
DirectMap2M:     9787392 kB
DirectMap1G:    1063256064 kB

Welcome to AskUbuntu! Could you [edit] your question to include: (0) the version of Ubuntu you’re running (1) the terminal output of `sudo dmidecode -t memory` (2) the brand and model of your CPU and motherboard. With this, it may be possible to identify why only 1TB is accessible — matigo, Nov 16 '22 at 21:04
Is the RAM seated in slots 0-19, or did you split it 0-9, then 12-21? This looks like one socket has only 512GB (as per DIMM layout), which will have the motherboard identify all of it, but the OS “balance” it by going with the minimum assigned to a given physical socket — matigo, Nov 16 '22 at 22:54
I think at least part of the discrepancy is that there is a difference between what free and lshw measure (GiB vs GB) - one binary, the other decimal. Have a look at the comments in https://askubuntu.com/questions/1130621/why-do-free-and-lshw-show-different-ram-sizes — Will, Nov 16 '22 at 23:34
Try looking at it with `free --si -h` and then `free -h` to see how much difference is in reporting between GB vs GiB — Terrance, Nov 17 '22 at 01:01
Even at terabyte level, the difference between TB and TiB is less than 10%, so it can't account for a 25% difference. At gigabyte level the difference is less than 7.5%. — thomasrutter, Nov 17 '22 at 02:33
It could be a bug in the `free` command as well. What is the output of `sudo inxi -m`? — Terrance, Nov 17 '22 at 05:12
aded the imxi -m output , that is very interesting it shows all the memory dimm, but reports 1 TB of ram — John Goeller, Nov 17 '22 at 17:07
how many processors (physical CPUs, not cores, etc.) are on this board? There's a correlation between number of processors and usable ram on some boards, and i think anything over a TB needs to be at least dual-processors to work with more than 1TB of RAM on some boards — Thomas Ward, Nov 17 '22 at 17:08
Instead of using `free -h` could you use simply `free` in bytes? There's a slight chance it's a rounding bug in how `free` reports memory and 1.2TiB ends up being incorrectly rounded down to 1.0TiB. — Matias N Goldberg, Nov 17 '22 at 17:16
I re-arranged the DImm's and either the system wouldn't boot , or got the same result — John Goeller, Nov 17 '22 at 17:41
Your configuration on the DIMMs is correct. https://dlcdnets.asus.com/pub/ASUS/server/RS700-E9-RS4/Manual/E19796_RS700-E9-RS4_RS12_UM_V7_WEB.pdf Maybe a BIOS update as your BIOS version is quite old from 2017 https://servers.asus.com/products/Servers/Rack-Servers/RS700-E9-RS12/#Resources — Terrance, Nov 17 '22 at 19:30
OK so I booted a USB stick contain CENTOS and gotthe same results. I also fully populatedthe machine to 1.5 TB , but still only have access to 1.tb . also took memory out ofthis machine and put it in another machine o to make it 1.25 TB and that Machine worked. that machne was a super micro , this one is an Asus . so It makes me believe it is some setting in the Bios — John Goeller, Nov 30 '22 at 15:38

System with 1.25 TB memory OS only seeing 1.00 TB ram

0 Answers0