Life After Heap Exhaustion #
hacking, June 4th 2006
SBCL has historically had only one way of dealing with heap exhaustion: instant death.
This is about to change:
(defvar *list* nil)
(defun cram (n)
(push (make-array n :element-type '(unsigned-byte 8)) *list*))
;;;; Allocate 10Mb chunks in a loop, and see what happens...
* (loop (cram (* 10 1024 1024)))
Heap exhausted during allocation: 8589312 bytes available, 10485768 requested.
debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread
#<THREAD "initial thread" {A68D5B9}>:
Heap exhausted: 8589312 bytes available, 10485768 requested. PROCEED WITH CAUTION!
restarts (invokable by number or by possibly-abbreviated name):
0: [ABORT] Exit debugger, returning to top level.
(SB-KERNEL::HEAP-EXHAUSTED-ERROR 8589312 10485768)
;;;; Kazonk! An allocation failed. Let's see what the backtrace looks like...
0] ba 10
0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8589312 10485768)
1: ("foreign function: call_into_lisp")
2: ("foreign function: funcall2")
3: ("foreign function: gc_find_freeish_pages")
4: ("foreign function: gc_alloc_large")
5: ("foreign function: alloc")
6: ("foreign function: alloc_overflow_ecx")
7: (CRAM 10485760)
8: (NIL)
9: (SB-INT:EVAL-IN-LEXENV (LOOP (CRAM (* 10 1024 1024))) #)
;;;; Presto! Selecting the ABORT restart gives up the failing allocation
;;;; — and are still alive and well.
0] 0
* (length *list*)
48
;;;; Let's try with smaller chunks
* (loop (cram (* 1024 1024)))
Heap exhausted during allocation: 167936 bytes available, 1048584 requested.
debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread
#<THREAD "initial thread" {A68D5B9}>:
Heap exhausted: 167936 bytes available, 1048584 requested. PROCEED WITH CAUTION!
restarts (invokable by number or by possibly-abbreviated name):
0: [ABORT] Exit debugger, returning to top level.
(SB-KERNEL::HEAP-EXHAUSTED-ERROR 167936 1048584)
0] 0
;;;; We managed to allocate a few more, as expected.
* (length *list*)
56
;;;; ...let's try smallish chunks instead of these megabyte monsters...
* (loop (cram (* 8 1024)))
Heap exhausted during allocation: 8120 bytes available, 8200 requested.
debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread
#<THREAD "initial thread" {A68D5B9}>:
Heap exhausted: 8120 bytes available, 8200 requested. PROCEED WITH CAUTION!
Heap exhausted during allocation: 112 bytes available, 528 requested.
Heap exhausted during allocation: 0 bytes available, 8 requested.
Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB LUB !move Alloc Waste Trig WP GCs Mem-age
0: 131071 0 131063 0 201 0 2089 7683 2563 40785336 64072 33464640 0 1 0.0000
1: 0 0 0 0 3 0 0 46098 2563 188751280 78416 190751280 1 1 0.0000
2: 5863 5853 0 0 75 3 0 69147 2605 283429280 116320 2000000 29 0 0.0000
3: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
4: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
5: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
6: 0 0 0 0 5771 0 0 0 0 23638016 0 2000000 5670 0 0.0000
Total bytes allocated=536603912
fatal error encountered in SBCL pid 3613(tid 3085402240):
Heap exhausted, game over.
LDB monitor
;;;; OOOPS! What happened? We ran totally out of memory — and died — while
;;;; printing out the debugger message, it seems. Note the nested calls to
;;;; HEAP-EXHAUSTED-ERROR in the backtrace. The possibility of this happening is the
;;;; reason for the apparently redundant error messages printed to STDOUT that you
;;;; see before the debugger messages ("Heap exhausted during allocation: ...").
ldb> ba 35
Backtrace:
0: Foreign function (null), fp = 0xb7a7e1b8, ra = 0x80557ab
1: Foreign function ldb_monitor, fp = 0xb7a7e288, ra = 0x805531e
2: Foreign function lose, fp = 0xb7a7e2a8, ra = 0x80523ee
3: Foreign function gc_find_freeish_pages, fp = 0xb7a7e2d8, ra = 0x805e21b
4: Foreign function (null), fp = 0xb7a7e308, ra = 0x806064d
5: Foreign function alloc, fp = 0xb7a7e338, ra = 0x805f8db
6: Foreign function alloc_overflow_edx, fp = 0xb7a7e3b4, ra = 0x806283f
7: (SB-C::TL-XEP COMMON-LISP::INVOKE-DEBUGGER)
8: (SB-C::TL-XEP COMMON-LISP::ERROR)
9: (SB-C::TL-XEP SB-KERNEL::HEAP-EXHAUSTED-ERROR)
10: Foreign function call_into_lisp, fp = 0xb7a7e490, ra = 0x8062651
11: Foreign function funcall2, fp = 0xb7a7e4b0, ra = 0x805cf1d
12: Foreign function gc_find_freeish_pages, fp = 0xb7a7e4e0, ra = 0x805e251
13: Foreign function (null), fp = 0xb7a7e510, ra = 0x806064d
14: Foreign function alloc, fp = 0xb7a7e540, ra = 0x805f8db
15: Foreign fp = 0xb7a7e578, ra = 0x28ffff29
16: (SB-C::TL-XEP SB-FORMAT::INTERPRET-FORMAT-LOGICAL-BLOCK)
17: (SB-C::TL-XEP SB-FORMAT::<-FORMAT-DIRECTIVE-INTERPRETER)
18: (SB-C::TL-XEP SB-FORMAT::INTERPRET-DIRECTIVE-LIST)
19: (SB-C::HAIRY-ARG-PROCESSOR SB-FORMAT::%FORMAT)
20: (SB-C::TL-XEP COMMON-LISP::FORMAT)
21: (SB-C::TL-XEP SB-DEBUG::%INVOKE-DEBUGGER)
22: (COMMON-LISP::LAMBDA ())
23: (SB-C::TL-XEP SB-IMPL::CALL-WITH-SANE-IO-SYNTAX)
24: (SB-C::TL-XEP SB-IMPL::%WITH-STANDARD-IO-SYNTAX)
25: (SB-C::TL-XEP COMMON-LISP::INVOKE-DEBUGGER)
26: (SB-C::TL-XEP COMMON-LISP::ERROR)
27: (SB-C::TL-XEP SB-KERNEL::HEAP-EXHAUSTED-ERROR)
28: Foreign function call_into_lisp, fp = 0xb7a7e9d4, ra = 0x8062651
29: Foreign function funcall2, fp = 0xb7a7e9f4, ra = 0x805cf1d
30: Foreign function gc_find_freeish_pages, fp = 0xb7a7ea24, ra = 0x805e251
31: Foreign function (null), fp = 0xb7a7ea54, ra = 0x806064d
32: Foreign function alloc, fp = 0xb7a7ea84, ra = 0x805f8db
33: Foreign function alloc_overflow_ecx, fp = 0xb7a7eb04, ra = 0x8062827
34: COMMON-LISP-USER::CRAM
So obviously this is by no means a full solution, but a damn sight better then nothing. If you allocate big objects this should work perfectly well, as there will still be plenty of memory left when the first allocation fails. However, if you manage to fill the heap close enough to the limit, then you are simply out of luck. Running out of heap during GC is also not dealt with.
I hope that this will be of use despite the limitations.
Coming soon to a CVS near you.