Re: File checkpointing in BLCR

From: Paul H. Hargrove (PHHargrove_at_lbl_dot_gov)
Date: Tue Jan 22 2008 - 13:25:15 PST

  • Next message: Neal Becker: "BLCR 0.6.3 a few rpmlint errors"
    Abhinav Jha wrote:
    > Sir,
    > We want to give the user a flexibility of choosing different file
    > checkpointing methods, like : saving the name of the file, contents of the
    > file, etc. depending on the file size, file properties, etc. Thus, we
    > would like to add on to the file checkpointing facilities that you have
    > provided in the latest release of BLCR. We were thinking of making changes
    > to the VMADump module code in order to do this. Is that right ? If not,
    > kindly suggest which part of the code we need to make changes to. We feel
    > lost :(
    I am sorry that you are feeling lost.  The BLCR source code is not very 
    easy to find one's way through initially, but it should get easier as 
    you get more familiar with it.
    The VMADump code (originally from the BProc project) is used by BLCR to 
    save/restore the memory, signal handlers and a few other 
    architecture-independent elements of the process state, plus all the 
    architecture-specific bits (the registers).  While VMADump handles most 
    mmap()ed files, it does not handle the files that are open(), and in the 
    future BLCR will take over handling of many/most mmap()ed files from 
    VMADump as support is added for inclusion of the executable and shared 
    libs in the checkpoint context file.
    The code that handles the save of open files (only "by reference" 
    currently) is in the function cr_save_open_file() in the file 
    cr_module/cr_dump_self.c.  The corresponding restore code is in 
    cr_restore_open_file() in cr_module/cr_rstrt_req.c.  It is hoped that in 
    the future these two functions will move from their current locations 
    into a single file with a name like cr_module/cr_files.c.  Those 
    functions, plus possibly their callers or callees, are where you are 
    likely to make your modifications.
    When looking to save the entire contents of the file, look at how 
    unlinked (deleted) files are currently handled in 
    cr_{save,restore}_open_file().  There are two things to be aware of:
    1) The function cr_copyfile() is used to insert the entire file into the 
    checkpoint context file.  The setting of the "unlinked" flag in the 
    cr_file_info structure causes a corresponding cr_mkunlinked() call at 
    restart time to reconstruct the file.  While you may not wish to restore 
    your files in an unlinked state, the logic you may require should be 
    very similar to this case.
    2) Note that cr_{save,restore)_open_file() both make use of calls to our 
    object-map code (cr_insert_object() and cr_find_object()) to determine 
    if the save/restore of the file should take place or not.  This is to 
    ensure that if multiple processes have the same file open, we will save 
    its contents to the context file exactly once.  At checkpoint time the 
    use of the object map is trivial (inserting a map from the inode address 
    to itself) and serves only as an indication that the specific file has 
    been saved.  At restart time, however, the mapping used is from the 
    "file_id" to the "struct file" of the new instance.  That ensures that 
    while only one copy of the file is restored, all other references will 
    reopen that same instance (which is done by cr_filp_reopen() in the case 
    of unlinked files, since they don't have a path by which to open them 
    We look forward to seeing your work completed and, if possible, 
    contributed for inclusion in BLCR where the entire community can benefit 
    from your work.  When ready to contribute patches to BLCR, we will need 
    a Developer's Certificate of Origin, a.k.a a "Signed-off-by" line (see 
    the "Sign Your Work" section of the file Documentation/SubmittingPatches 
    in any recent Linux kernel sources).  This allows us to satisfy our 
    organization's requirement to redistribute your contribution to others.
    I am pleased to know that somebody is pursuing this work and am happy to 
    provide what assistance I can by e-mail.  Please don't hesitate to ask 
    more questions using the checkpoint_at_lbl_dot_gov list address.  You may also 
    wish to subscribe to that list to ensure you see the answers and any 
    follow-ups to them (just e-mail "subscribe checkpoint" in the message 
    /body/ and without the quotes to majordomo_at_lbl_dot_gov 
    Paul H. Hargrove                          PHHargrove_at_lbl_dot_gov
    Future Technologies Group
    HPC Research Department                   Tel: +1-510-495-2352
    Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900

  • Next message: Neal Becker: "BLCR 0.6.3 a few rpmlint errors"