This article provides information on how to calculate actual amount of RAM being used by the /tmp and /mfs directories.
The /tmp and /mfs directories are RAM drives, which are known as memory disk (MD for short). As per PR81098/427302/697096/703437, there is a known memory leak bug in MFS (memory-based file system), which causes the physical pages to hold the contents of files that will not get deleted, when deleted.
For example, when the show system core-file core-file-info <> command is abnormally terminated, the temporary files created in /tmp will not be cleaned up. This results in the following issues:
- Presence of stale files in /tmp (until system is rebooted).
- Further allocation of memory to MFS on execution of the same command due to the presence of stale files (and memory not being re-usable).
Note: ‘show system core-files core-file-info <>’ will no longer cause this behavior. From Junos10.4R12 and 11.4R6, the working directory of such CLI will be changed from ‘/tmp’ (memory) to ‘/var/tmp’ (HD).
The output of show system storage does not show the actual used memory being mapped to /tmp and /mfs; whereas memory utilization is high, show chassis routing-engine is executed.
1. Obtain the unit IDs for the /tmp and /mfs directories; in this example, they are 8 and 9:
> show system storage | match "md[0-9]" Filesystem Size Used Avail Capacity Mounted on /dev/md0 34M 34M 0B 100% /packages/mnt/jbase /dev/md1 9.8M 9.8M 0B 100% /packages/mnt/jkernel-10.4R8.5 /dev/md2 79M 79M 0B 100% /packages/mnt/jpfe-T-10.4R8.5 /dev/md3 5.8M 5.8M 0B 100% /packages/mnt/jdocs-10.4R8.5 /dev/md4 69M 69M 0B 100% /packages/mnt/jroute-10.4R8.5 /dev/md5 21M 21M 0B 100% /packages/mnt/jcrypto-10.4R8.5 /dev/md6 39M 39M 0B 100% /packages/mnt/jpfe-common-10.4R8.5 /dev/md7 295M 295M 0B 100% /packages/mnt/jruntime-10.4R8.5 /dev/md8 2.0G 12K 1.8G 0% /tmp /dev/md9 2.0G 20M 1.8G 1% /mfs
2. Check the page size being allocated by the root account under the shell; here it is 4096:
% sysctl -a | grep pagesize hw.pagesize: 4096 p1003_1b.pagesize: 4096
3. Write a live kernel coredump via the root account under the shell:
% sysctl -w kern.live_core_dump=1
4. Check the active pages being mapped to the /tmp (unit 8) and /mfs (unit 9) directories by decoding the kernel coredump:
(kgdb) set print pretty
(kgdb) p *md_softc_list->lh_first
$1 = {
unit = 9,
list = {
le_next = 0xc7b4c800,
le_prev = 0xc0c21c88
},
bio_queue = {
queue = {
tqh_first = 0x0,
tqh_last = 0xc7b4c00c
},
last_offset = 0,
insert_point = 0xd7fe7528
},
queue_mtx = {
mtx_object = {
lo_name = 0xc0bb7675 "md bio queue",
lo_type = 0xc0bb7675 "md bio queue",
lo_flags = 16973824,
lo_list = {
tqe_next = 0x0,
tqe_prev = 0x0
},
lo_witness = 0x0
},
mtx_lock = 4,
mtx_recurse = 0
},
dev = 0x0,
type = MD_SWAP,
mediasize = 2147475456,
sectorsize = 512,
opencount = 1,
fwheads = 0,
fwsectors = 0,
flags = 0,
name = "md9", '\0' <repeats 16 times>,
procp = 0xc7dab678,
gp = 0xc7a2fd70,
pp = 0xc7d53a20,
start = 0xc0500349 <mdstart_swap>,
indir = 0x0,
uma = 0x0,
pl_ptr = 0x0,
pl_len = 0,
vnode = 0x0,
file = '\0' <repeats 1023 times>,
cred = 0xc7738ee0,
sc_compdata = 0x0,
sc_offset = 0,
object = 0xc7dba100
}
(kgdb)
(kgdb) p *md_softc_list->lh_first->list->le_next
$2 = {
unit = 8,
list = {
le_next = 0xc7bd7000,
le_prev = 0xc7b4c004
},
bio_queue = {
queue = {
tqh_first = 0x0,
tqh_last = 0xc7b4c80c
},
last_offset = 0,
insert_point = 0xd69b6108
},
queue_mtx = {
mtx_object = {
lo_name = 0xc0bb7675 "md bio queue",
lo_type = 0xc0bb7675 "md bio queue",
lo_flags = 16973824,
lo_list = {
tqe_next = 0x0,
tqe_prev = 0x0
},
lo_witness = 0x0
},
mtx_lock = 4,
mtx_recurse = 0
},
dev = 0x0,
type = MD_SWAP,
mediasize = 2147475456,
sectorsize = 512,
opencount = 1,
fwheads = 0,
fwsectors = 0,
flags = 0,
name = "md8", '\0' <repeats 16 times>,
procp = 0xc7a568a0,
gp = 0xc7a2f5a0,
pp = 0xc7d53ba0,
start = 0xc0500349 <mdstart_swap>,
indir = 0x0,
uma = 0x0,
pl_ptr = 0x0,
pl_len = 0,
vnode = 0x0,
file = '\0' <repeats 1023 times>,
cred = 0xc7738ee0,
sc_compdata = 0x0,
sc_offset = 0,
object = 0xc7dba200
}
}
(kgdb) p *$1->object
$3 = {
mtx = {
mtx_object = {
lo_name = 0xc0bcfb35 "vm object",
lo_type = 0xc0bd04c0 "standard object",
lo_flags = 21168128,
lo_list = {
tqe_next = 0x0,
tqe_prev = 0x0
},
lo_witness = 0x0
},
mtx_lock = 4,
mtx_recurse = 0
},
object_list = {
tqe_next = 0xc7daf900,
tqe_prev = 0xc7db0020
},
shadow_head = {
lh_first = 0x0
},
shadow_list = {
le_next = 0xc7dba800,
le_prev = 0xc7dbae2c
},
memq = {
tqh_first = 0xc45795f8,
tqh_last = 0xc457c228
},
root = 0xc4a34b30,
size = 524286,
generation = 35851,
ref_count = 1,
shadow_count = 0,
type = 1 '\001',
flags = 8576,
pg_color = 8,
paging_in_progress = 0,
resident_page_count = 35850,
backing_object = 0x0,
backing_object_offset = 0,
pager_object_list = {
tqe_next = 0x0,
tqe_prev = 0x0
},
handle = 0x0,
un_pager = {
vnp = {
vnp_size = 0
},
devp = {
devp_pglist = {
tqh_first = 0x0,
tqh_last = 0x0
}
},
swp = {
swp_bcount = 0
}
}
}
(kgdb) p *$2->object
$4 = {
mtx = {
mtx_object = {
lo_name = 0xc0bcfb35 "vm object",
lo_type = 0xc0bd04c0 "standard object",
lo_flags = 21168128,
lo_list = {
tqe_next = 0x0,
tqe_prev = 0x0
},
lo_witness = 0x0
},
mtx_lock = 4,
mtx_recurse = 0
},
object_list = {
tqe_next = 0xc7db0000,
tqe_prev = 0xc7d60da0
},
shadow_head = {
lh_first = 0x0
},
shadow_list = {
le_next = 0xc1448300,
le_prev = 0xc14483a8
},
memq = {
tqh_first = 0xc4596260,
tqh_last = 0xc4596190
},
root = 0xc4598198,
size = 524286,
generation = 265719,
ref_count = 1,
shadow_count = 0,
type = 1 '\001',
flags = 8576,
pg_color = 13,
paging_in_progress = 0,
resident_page_count = 265718,
backing_object = 0x0,
backing_object_offset = 0,
pager_object_list = {
tqe_next = 0x0,
tqe_prev = 0x0
},
handle = 0x0,
un_pager = {
vnp = {
vnp_size = 0
},
devp = {
devp_pglist = {
tqh_first = 0x0,
tqh_last = 0x0
}
},
swp = {
swp_bcount = 0
}
}
}
(kgdb)
/mfs (unit 9) is 35850 pages * 4k = 146,841,600 bytes = 140 MB
/tmp (unit 8) is 265718 pages * 4096 = 1,088,380,928 bytes = 1 GB