Re: Problems with --enable-restore-ids

From: Paul H. Hargrove (PHHargrove_at_lbl_dot_gov)
Date: Wed Feb 18 2009 - 15:16:12 PST

  • Next message: Ted Cabeen: "Re: Problems with --enable-restore-ids"
    Ted,
      Thanks for your patience.  The restore-ids code itself seems to be 
    doing just what it should: dropping the root privilege before performing 
    any fs permission checks, preventing use of a maliciously modified 
    checkpoint context file as a way to access otherwise inaccessible 
    files.  However, there seem to be problems with files (ncsd is just one 
    example) that were originally openned *with* some privilege.  Anything 
    setup by the batch system is a candidate for such problems.
    
      For the new problem I can see the problem, but don't know enough to 
    suggest a solution.
      The term "external pipe" means that at the time the checkpoint was 
    taken there was a pipe that had only one endpoint within the scope of 
    the checkpoint, while the other was not.  In a batch scheduled 
    environment this is often the case for the std{in,out,err} descriptors, 
    but in your case the error says fd=16, so it must be something else.
      When an "external pipe" is encountered in the context file at restart 
    time, BLCR's behavior is to try to connect this fd to same file as the 
    stdin or stdout (depending on which end of the pipe is external) of the 
    cr_restart process.  In your case the user has insufficient permission 
    to do so, most likely because the cr_restart was launched as root and 
    root owns the file (or device) that are used for std{in,out}.
    
      Since I don't know what fd 16 was being used for,  I can't be certain 
    that connecting it to stdin or stdout is even the right thing to do.  If 
    it is the right thing, then I will need to go back to looking at the 
    BLCR source code and determine if the permission checks being performed 
    (the ones that yield the first error) are required for 
    correctness/security.  My initial thought is that if the stdin/out of 
    cr_restart have not been marked close-on-exec, then any child it creates 
    potentially has access to those fds as its own stdin/out and bypassing 
    fs permissions sounds like the right thing to do.
    
      So, it is possible that BLCR needs to be doing something different 
    with respect to the permissions when reopenning external pipes.  I will 
    look into that and get back to you.  However, I'd appreciate it if you 
    could also be looking into figuring out what fd 16 is being used for.  
    It is entirely possible that it is something that will require a 
    different approach.
    
    -Paul
    
    Ted Cabeen wrote:
    > All right.  I've disabled nscd, and we're on to the next problem. 
    > (Sorry I missed that note in the FAQ)  When I restart a job with 
    > enable-restore-ids, I get the following error:
    > - Error -13 from cr_filp_reopen() while restoring external pipe
    > - cr_restore_all_files [3766]:  Unable to restore fd 16 (type=4,err=-13)
    > - cr_rstrt_child [3766]:  Unable to restore files!  (err=-13)
    > Restart failed: Permission denied
    >
    > My suspended jobs start fine when complied without enable-restore-ids.
    >
    > Thoughts?
    >
    > --Ted
    >
    > Paul H. Hargrove wrote:
    >> Ted,
    >>
    >>  You are right about the nscd cache file being opened as root (or 
    >> other "system" id).  The program acquires the file descriptor via fd 
    >> passing from a privileged daemon process. Since we can't safely 
    >> reopen this file as the user and we are equally unable to reproduce 
    >> the descriptor passing from the daemon, BLCR is incompatible with 
    >> nscd (see FAQ:  http://mantis.lbl.gov/blcr/doc/html/FAQ.html#nscd ).  
    >> If you were to perform the restart as the original user you would 
    >> encounter this problem regardless of --enable-restore-ids of not.  I 
    >> am afraid the only known solution is to disable nscd.
    >>
    >> -Paul
    >>
    >> Ted Cabeen wrote:
    >>> I'm having problems with 0.8.0 with --enable-restore-ids.  When I 
    >>> try to restart a checkpointed job, I get the following error:
    >>> - open('/var/cache/nscd/passwd', 0x0) failed: -13
    >>> - mmap failed: /var/cache/nscd/passwd
    >>> - thaw_threads returned error, aborting. -13
    >>> Restart failed: Permission denied
    >>>
    >>> If I recompile 0.8.0 without restore-ids, it doesn't have this 
    >>> error.  I think that the problem may be that the nscd cache is 
    >>> opened on behalf of the program by libc as root, but when BLCR tries 
    >>> to restart the checkpointed program as the original user, it can't 
    >>> open the nscd cache.  Is there a way to fix this?
    >>>
    >>> --Ted
    >>
    >>
    
    
    -- 
    Paul H. Hargrove                          PHHargrove_at_lbl_dot_gov
    Future Technologies Group
    HPC Research Department                   Tel: +1-510-495-2352
    Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900
    

  • Next message: Ted Cabeen: "Re: Problems with --enable-restore-ids"