Re: question about "cr_save_mmaps_data" function

From: TK (cartoon.ke_at_gmail_dot_com)
Date: Fri Mar 26 2010 - 06:58:22 PDT

  • Next message: Leonardo Fialho: "Re: restart failed:Device or resource busy,found pid 4818 in use"
    Thanks very much, Balazs. I looked into the call graph again, and a lot 
    of calls to other
    functions in vmadump4 as well.
    Thank you very much for you help.
          Tao Ke
    On 03/26/2010 01:40 AM, Balazs Gerofi wrote:
    > Sorry, I was wrong :)
    > So the call stack is like this:
    > cr_dump_self() -> cr_do_dump() -> cr_do_vmadump() -> 
    > cr_freeze_threads() ->
    > vmadump_freeze_proc() -> and there if you are the leader then the 
    > non-shared
    > mappings are also saved (in vmadump4/vmadump_common.c).
    > Sorry again,
    > Balazs
    > On Fri, Mar 26, 2010 at 3:08 PM, Balazs Gerofi 
    > < <>> 
    > wrote:
    >     Hi Tao,
    >     if you go to cr_dump_self() you will see a call to cr_do_dump()
    >     after the leader
    >     thread is chosen. cr_do_dump() calls cr_do_vmadump() which calls a
    >     couple
    >     of vmadump functions.
    >     cr_save_mmaps_maps() is the one where non-shared mappings are dumped.
    >     I recommend you to use cscope or any other source code tagging
    >     package,
    >     so that you can easily follow the call stack.
    >     Regards,
    >     Balazs
    >     On Fri, Mar 26, 2010 at 2:34 PM, Tao Ke <
    >     <>_at_gmail_dot_com <>> wrote:
    >         Thank you so much for your patient and detailed explanation.
    >         It is very helpful to me.
    >         I have tracked the call path from the beginning. And it seems
    >         to me that the context of the process if saved inside
    >         cr_save_mmaps_data, and the checkpoint looks end here. I am
    >         confused here about when vmadump4 is used. From you
    >         explanation, vmadump can be used to handle a single thread. I
    >         found that blcr module initialize vmadump module, but I can
    >         not find when the vmadump is used else where. Could you please
    >         give me some hints about when vmadump module is used?
    >         Thank you again for you time.
    >         On Thu, Mar 25, 2010 at 10:58 PM, Paul H. Hargrove
    >         <PHHargrove_at_lbl_dot_gov <mailto:PHHargrove_at_lbl_dot_gov>> wrote:
    >             TK,
    >             It is not my intent to be rude or condescending but I
    >             don't have the time to describe everything that takes
    >             place in a checkpoint.
    >             The simple answer is that "the whole story" is in the
    >             source code - which you have available to examine.
    >             You have correctly determined that a checkpoint begins
    >             with an ioctl() that invokes cr_dump_self(), and you
    >             should be able to trace the rest using the source code.  I
    >             have not memorized which functions call which others in
    >             what order, even though I wrote most of it.  To give you
    >             the "whole story" I would have to take the time to read
    >             through the sources and trace the calls.  Instead, I
    >             encourage you to read them.  Doing so is likely to give
    >             you a deeper understanding than if I were to try to do it
    >             for you.  If after that you have some specific questions
    >             about "how" or "why" things are done, I may be able to help.
    >             You may want to look at tools like "cflow" to build a call
    >             graph for you, though I cannot be certain they work well
    >             with Linux kernel code.
    >             I CAN summarize the distinction between the code in
    >             cr_module/ and vmadump4/, which appears to be a
    >             significant point of your question.   The vmadump code is
    >             a heavily modified version of software from the BProc
    >             project that predates BLCD (and comes from a different
    >             organization).  It was never able to deal with shared
    >             memory, files or multiple processes; nor does it have the
    >             callback mechanisms of BLCR.  So the BLCR project began
    >             with the intent of keeping the changes made to files in
    >             vmadump to a minimum and building the other functionality
    >             (e.g. shared memory, files and multiple processes)
    >             separately.  That is why you will find that vmadump
    >             handles "anonymous" pages and non-shared mappings, while
    >             the cr_save_mmaps code handles the shared mappings.
    >             I hope my answer helps you some, even if I can't provide
    >             the answer you may have been looking for.
    >             -Paul
    >             TK wrote:
    >                 Thanks.
    >                 But when a checkpoint request is issued with
    >                 "cr_checkpoint" command, a  ioctl request is made to
    >                 /proc/checkpoint/ctrl. I suppose it will be  the
    >                 "CR_OP_HAND_CHKPT" request. Then "cr_dump_self"  will
    >                 be called, and finally cr_save_mmaps_data will be
    >                 called, and the memory will be saved here. Am I
    >                 correct? If so, when is  the whole story of
    >                 checkpoint? When the "vmadump" module is used then ?
    >                 Thank you very much.
    >                 On 03/25/2010 07:20 PM, Paul H. Hargrove wrote:
    >                     TK,
    >                     I am sorry I didn't get the chance to answer this
    >                     one when you asked me directly 2 days ago - I am
    >                     up against some deadlines right now.
    >                     To answer your question:
    >                     In the function you ask about we are dealing only
    >                     with memory regions created by mmap() of a file.
    >                      Therefore all the "clean" pages already exist
    >                     somewhere on disk in the file that has been
    >                     mmap()ed.  This includes the executable file and
    >                     shared libraries that were mmap()ed in prior to
    >                     the start of main().  As with open files, BLCR
    >                     makes the (optimistic) assumption that the file
    >                     will still exist, unmodified, at the time of the
    >                     restart.  However, one can ensure that even the
    >                     "clean" pages will be stored with the checkpoint
    >                     by passing --save-all.
    >                     -Paul
    >                     TK wrote:
    >                         Hi , all. I am trying to adding my own code
    >                         into BLCR for some experiments.
    >                         When I was reading the code of
    >                         "cr_save_mmaps_data" function in
    >                         cr_module/cr_mmaps.c, I found the comment /*
    >                         dump the dirty pages */ . I am wondering you
    >                         dump only the dirty pages only? It will not be
    >                         enough info for restart. Or the other pages
    >                         are dumped else where? If so, where is it?
    >                         Thank you.
    >             -- 
    >             Paul H. Hargrove PHHargrove_at_lbl_dot_gov
    >             <mailto:PHHargrove_at_lbl_dot_gov>
    >             Future Technologies Group                 Tel: +1-510-495-2352
    >             HPC Research Department                   Fax: +1-510-486-6900
    >             Lawrence Berkeley National Laboratory

  • Next message: Leonardo Fialho: "Re: restart failed:Device or resource busy,found pid 4818 in use"