Re: question about "cr_save_mmaps_data" function

From: Balazs Gerofi (
Date: Thu Mar 25 2010 - 23:40:34 PDT

  • Next message: TK: "Re: question about "cr_save_mmaps_data" function"
    Sorry, I was wrong :)
    So the call stack is like this:
    cr_dump_self() -> cr_do_dump() -> cr_do_vmadump() -> cr_freeze_threads() ->
    vmadump_freeze_proc() -> and there if you are the leader then the non-shared
    mappings are also saved (in vmadump4/vmadump_common.c).
    Sorry again,
    On Fri, Mar 26, 2010 at 3:08 PM, Balazs Gerofi <> wrote:
    > Hi Tao,
    > if you go to cr_dump_self() you will see a call to cr_do_dump() after the
    > leader
    > thread is chosen. cr_do_dump() calls cr_do_vmadump() which calls a couple
    > of vmadump functions.
    > cr_save_mmaps_maps() is the one where non-shared mappings are dumped.
    > I recommend you to use cscope or any other source code tagging package,
    > so that you can easily follow the call stack.
    > Regards,
    > Balazs
    > On Fri, Mar 26, 2010 at 2:34 PM, Tao Ke <cartoon.ke_at_gmail_dot_com> wrote:
    >> Thank you so much for your patient and detailed explanation. It is very
    >> helpful to me.
    >> I have tracked the call path from the beginning. And it seems to me that
    >> the context of the process if saved inside cr_save_mmaps_data, and the
    >> checkpoint looks end here. I am confused here about when vmadump4 is used.
    >> From you explanation, vmadump can be used to handle a single thread. I found
    >> that blcr module initialize vmadump module, but I can not find when the
    >> vmadump is used else where. Could you please give me some hints about when
    >> vmadump module is used?
    >> Thank you again for you time.
    >> On Thu, Mar 25, 2010 at 10:58 PM, Paul H. Hargrove <PHHargrove_at_lbl_dot_gov>wrote:
    >>> TK,
    >>> It is not my intent to be rude or condescending but I don't have the time
    >>> to describe everything that takes place in a checkpoint.
    >>> The simple answer is that "the whole story" is in the source code - which
    >>> you have available to examine.
    >>> You have correctly determined that a checkpoint begins with an ioctl()
    >>> that invokes cr_dump_self(), and you should be able to trace the rest using
    >>> the source code.  I have not memorized which functions call which others in
    >>> what order, even though I wrote most of it.  To give you the "whole story" I
    >>> would have to take the time to read through the sources and trace the calls.
    >>>  Instead, I encourage you to read them.  Doing so is likely to give you a
    >>> deeper understanding than if I were to try to do it for you.  If after that
    >>> you have some specific questions about "how" or "why" things are done, I may
    >>> be able to help.
    >>> You may want to look at tools like "cflow" to build a call graph for you,
    >>> though I cannot be certain they work well with Linux kernel code.
    >>> I CAN summarize the distinction between the code in cr_module/ and
    >>> vmadump4/, which appears to be a significant point of your question.   The
    >>> vmadump code is a heavily modified version of software from the BProc
    >>> project that predates BLCD (and comes from a different organization).  It
    >>> was never able to deal with shared memory, files or multiple processes; nor
    >>> does it have the callback mechanisms of BLCR.  So the BLCR project began
    >>> with the intent of keeping the changes made to files in vmadump to a minimum
    >>> and building the other functionality (e.g. shared memory, files and multiple
    >>> processes) separately.  That is why you will find that vmadump handles
    >>> "anonymous" pages and non-shared mappings, while the cr_save_mmaps code
    >>> handles the shared mappings.
    >>> I hope my answer helps you some, even if I can't provide the answer you
    >>> may have been looking for.
    >>> -Paul
    >>> TK wrote:
    >>>> Thanks.
    >>>> But when a checkpoint request is issued with "cr_checkpoint" command, a
    >>>>  ioctl request is made to /proc/checkpoint/ctrl. I suppose it will be  the
    >>>> "CR_OP_HAND_CHKPT" request. Then "cr_dump_self"  will be called, and finally
    >>>> cr_save_mmaps_data will be called, and the memory will be saved here. Am I
    >>>> correct? If so, when is  the whole story of checkpoint? When the "vmadump"
    >>>> module is used then ?
    >>>> Thank you very much.
    >>>> On 03/25/2010 07:20 PM, Paul H. Hargrove wrote:
    >>>>> TK,
    >>>>> I am sorry I didn't get the chance to answer this one when you asked me
    >>>>> directly 2 days ago - I am up against some deadlines right now.
    >>>>> To answer your question:
    >>>>> In the function you ask about we are dealing only with memory regions
    >>>>> created by mmap() of a file.  Therefore all the "clean" pages already exist
    >>>>> somewhere on disk in the file that has been mmap()ed.  This includes the
    >>>>> executable file and shared libraries that were mmap()ed in prior to the
    >>>>> start of main().  As with open files, BLCR makes the (optimistic) assumption
    >>>>> that the file will still exist, unmodified, at the time of the restart.
    >>>>>  However, one can ensure that even the "clean" pages will be stored with the
    >>>>> checkpoint by passing --save-all.
    >>>>> -Paul
    >>>>> TK wrote:
    >>>>>> Hi , all. I am trying to adding my own code into BLCR for some
    >>>>>> experiments.
    >>>>>> When I was reading the code of "cr_save_mmaps_data" function in
    >>>>>> cr_module/cr_mmaps.c, I found the comment /* dump the dirty pages */ . I am
    >>>>>> wondering you dump only the dirty pages only? It will not be enough info for
    >>>>>> restart. Or the other pages are dumped else where? If so, where is it?
    >>>>>> Thank you.
    >>> --
    >>> Paul H. Hargrove                          PHHargrove_at_lbl_dot_gov
    >>> Future Technologies Group                 Tel: +1-510-495-2352
    >>> HPC Research Department                   Fax: +1-510-486-6900
    >>> Lawrence Berkeley National Laboratory

  • Next message: TK: "Re: question about "cr_save_mmaps_data" function"