o +kes@s.ddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z e Z e dZ ddlmZmZdd lmZmZmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZmZmZddlmZddlm Z ddlm!Z!ddlm"Z"ddlm#Z#m$Z$ddlm%Z%m&Z&ddlm'Z'ddlm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9eddZ:Gd d!d!Z;d"d#Zd(d)Z?d*d+Z@Gd,d-d-ZAGd.d/d/ZBGd0d1d1ZCGd2d3d3eCZDGd4d5d5eCZEdS)7N) unhexlify) namedtuple) perf_counter) create_loggerzborg.debug.files_cache) CACHE_READMEFILES_CACHE_MODE_DISABLED) ChunkIndexChunkIndexEntryCacheSynchronizer)Location)Error)Manifest) get_cache_dirget_security_dir) int_to_bigint bigint_to_int bin_to_hexparse_stringified_list)format_file_size)safe_ns)yes)remove_surrogates)ProgressIndicatorPercentProgressIndicatorMessage)set_ec EXIT_WARNING) safe_unlink)msgpack) ArchiveItemChunkListEntry) PlaintextKey)IntegrityCheckedFileDetachedIntegrityCheckedFileFileIntegrityError)Lock)SaveFile)cache_if_remote)LIST_SCAN_LIMITFileCacheEntryzage inode size cmtime chunk_idsc@seZdZdZddZedddZddZd d Zd d Z dd dZ dddZ dddZ ddddddZ dddZddZdS)SecurityManageras Tracks repositories. Ensures that nothing bad happens (repository swaps, replay attacks, unknown repositories etc.). This is complicated by the Cache being initially used for this, while only some commands actually use the Cache, which meant that other commands did not perform these checks. Further complications were created by the Cache being a cache, so it could be legitimately deleted, which is annoying because Borg didn't recognize repositories after that. Therefore a second location, the security database (see get_security_dir), was introduced which stores this information. However, this means that the code has to deal with a cache existing but no security DB entry, or inconsistencies between the security DB and the cache which have to be reconciled, and also with no cache existing but a security DB entry. cCsV||_t|j|_t||_tj|jd|_tj|jd|_ tj|jd|_ dS)Nzkey-typelocationzmanifest-timestamp) repositoryrid_strdir cache_dirospathjoin key_type_file location_filemanifest_ts_file)selfr,r7.usr/lib/python3.10/site-packages/borg/cache.py__init__?s   zSecurityManager.__init__NcCs,|pt|j}tj|rt|dSdS)z:destroy the security dir for ``repository`` or at ``path``N)rr-r0r1existsshutilrmtreer,r1r7r7r8destroyGs zSecurityManager.destroycCstdd|j|j|jfDS)Ncss|] }tj|VqdSN)r0r1r:).0fr7r7r8 Osz(SecurityManager.known..)allr3r4r5r6r7r7r8knownNszSecurityManager.knownc Cs|sdSz$t|j}|}|t|jkWdWS1s#wYWdStyC}z td|WYd}~dSd}~ww)NFz&Could not read/parse key type file: %s) rEopenr3readstrTYPEOSErrorloggerwarning)r6keyfdtypeexcr7r7r8 key_matchesRs  (zSecurityManager.key_matchescCstd|jj|j|jj}td|tdt|jtd|j t |j  }| |Wdn1s;wYt |j }| t|jWdn1sXwYt |j}| |j WddS1stwYdS)Nz#security: saving state for %s to %szsecurity: current location %szsecurity: key type %szsecurity: manifest timestamp %s)rKdebugr,r-r. _locationcanonical_pathrHrI timestampr&r4writer3r5)r6manifestrMZcurrent_locationrNr7r7r8save\s      "zSecurityManager.savec Cs\z!t|j }|}Wdn1swYtd|Wn+ty3td|jd}YntyL}ztd|d}WYd}~nd}~ww|r`|jr`||jkr`|j}td||j j }|r||krd ||d}t |ddd d d sttd t|j }||Wdn1swY|r|dSdSdSdS) Nz#security: read previous location %rz-security: previous location file %s not found)Could not read previous location file: %sz.security: using previous_location of cache: %rzDWarning: The repository at location {} was previously located at {} zDo you want to continue? [yN] Aborting.Invalid answer, aborting.FZ BORG_RELOCATED_REPO_ACCESS_IS_OKZ false_msgZ invalid_msgretryZenv_var_overridezrErQrXrerkrnrursrrr7r7r7r8r*+s    r*cCst|}|j||j|ddS)Nrq)r*rurM)r,rWrpZsmr7r7r8rusrucCs|j}t|}t|}|j|jkrG|j|jkrG|j|jkrG|j|jkrG|jrG|jrG|jdrG|jdrG|jdd|jddkrG|S|S)Nz/~/z/./) rSrTr protouserhostportr1 startswith)Zcache_locationr,Z repo_locationZrlZclr7r7r8 recanonicalize_relative_locations 0  rcCs|p tjt|jSr?)r0r1r2rr-r=r7r7r8r/r/cCstjdd}|r d|SdS)NZBORG_FILES_CACHE_SUFFIXrffiles.files)r0environgetsuffixr7r7r8files_cache_namesrcCsddt|DdS)NcSs"g|] }|dks |dr|qS)rr)rr@fnr7r7r8 s"z-discover_files_cache_name..r)r0listdir)r1r7r7r8discover_files_cache_namesrc@s`eZdZdddZddZddZdd Zd d Zd d ZddZ dddZ ddZ ddZ dS)rtNcCs4||_t|||_tj|jd|_d|_||_dS)Nconfig)r,r/r1r0r2 config_pathlockrp)r6r,r1rpr7r7r8r9s   zCacheConfig.__init__cCs ||Sr?)rFrDr7r7r8 __enter__szCacheConfig.__enter__cC |dSr?closer6exc_typeZexc_valZexc_tbr7r7r8__exit__ zCacheConfig.__exit__cCstj|jSr?)r0r1r:rrDr7r7r8r:szCacheConfig.existscCs|rJtjdd}|d|ddd|dd|jj|ddd|d|dddt|j}| |WddS1sJwYdS) N interpolationcacheversion1r,rWrf integrity) r: configparser ConfigParser add_sectionsetr,r-r&rrV)r6rrNr7r7r8creates      "zCacheConfig.createcCs.ttj|jdd|jd|_|dS)NrT) exclusivetimeout)r%r0r1r2rpacquirerloadrDr7r7r8rFs" zCacheConfig.opencCstjdd|_t|j}|j|Wdn1swY||j|jdd|_t |jdd|_ |jjdddd|_ |jjdddd|_ t t|jjddd d|_t t|jjdd d d|_z t|jd |_|jdd|jdkri|_td Wntjytd i|_Ynw|jjdddd}|rt||j|_nd|_|j dd|jjdS)Nrrr,rWrU)fallbackrlignored_featuresrfmandatory_featuresrzHCache integrity data not available: old Borg version modified the cache.zTCache integrity: No integrity data found (files, chunks). Cache is from old version.r_)rr_configrFr read_file_check_upgraderidr manifest_idrUrlrrrrdictitemsrpoprKrLNoSectionErrorrRrr,r_rSrT)r6rNr_r7r7r8rs4     zCacheConfig.loadcCs|rS|jdd|j|jdd|j|jddd|j|jddd|j|jds8|jd|j D] \}}|jd||q=|jdd|j|r`|jddt |j t |j}|j|WddS1swwYdS) NrrWrUr,rrrl)rrr-rUr2rr has_sectionrrrrHrIr&rrV)r6rWrMfileintegrity_datarNr7r7r8rX3s   "zCacheConfig.savecCs"|jdur|jd|_dSdSr?)rreleaserDr7r7r8rCs   zCacheConfig.closecCsbz|jdd}d}||kr|td|||fWdStjy0|td|dw)Nrrrz0%s has unexpected cache version %d (wanted: %d).z#%s does not look like a Borg cache.)rgetintr Exceptionrr)r6rZ cache_versionZwanted_versionr7r7r8rHszCacheConfig._check_upgrade)NN) rwrxryr9rrr:rrFrrXrrr7r7r7r8rts    rtc @seZdZdZGdddeZGdddeZGdddeZGdd d eZGd d d eZ e dd dZ e dddZ d dddd de ddf ddZd S)razClient Side cache c@eZdZdZdS)zCache.RepositoryIDNotUniquez`Cache is newer than repository - do you have multiple, independently updated repos with same ID?Nrwrxryrzr7r7r7r8riXric@r)zCache.RepositoryReplayzCache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)Nrr7r7r7r8rj[rrjc@r)zCache.CacheInitAbortedErrorzCache initialization abortedNrr7r7r7r8rv^rrvc@r)zCache.RepositoryAccessAbortedzRepository access abortedNrr7r7r7r8rbarrbc@r)zCache.EncryptionMethodMismatchzLRepository encryption method changed since last access, refusing to continueNrr7r7r7r8rmdrrmNcCs(t||}ttj|ddddS)NrT)r)r/r%r0r1r2 break_lockr=r7r7r8rgs zCache.break_lockcCsL|p tjt|j}tj|d}tj|r$t|t|dSdS)z3destroy the cache for ``repository`` or at ``path``rN) r0r1r2rr-r:remover;r<)r,r1rr7r7r8r>ls   z Cache.destroyTFc  s f dd} fdd}| s | St}|rL||jjk}Wdn1s=wY|rLtd| Std|S)Nc st d S)N) r,rMrWr1syncroprogressiecrp cache_modeconsider_part_files) LocalCacher7 rrrrMrprWr1rr,rror7r8localys zCache.__new__..localcstdS)N)r,rMrWrprr) AdHocCacher7)rrrMrprWr,r7r8adhoc~s zCache.__new__..adhocz%Cache: choosing local cache (in sync)zKCache: choosing ad-hoc cache (local cache does not exist or is not in sync))rtr:rrrKrR)clsr,rMrWr1rrorrpZpermit_adhoc_cacherrrrrrcZ cache_in_syncr7rr8__new__us    z Cache.__new__r?)rwrxryrzr rirjrvrbrmr{rr>rrr7r7r7r8raUs  rac@sHeZdZdZdddZddZedgdZd d Zd d Z d dZ dS)CacheStatsMixinzAll archives: {0.total_size:>20s} {0.total_csize:>20s} {0.unique_csize:>20s} Unique chunks Total chunks Chunk index: {0.total_unique_chunks:20d} {0.total_chunks:20d}FcCs||_i|_dSr?)r pre12_meta)r6rr7r7r8r9s zCacheStatsMixin.__init__cCs|j|Sr?) str_formatr` format_tuplerDr7r7r8__str__szCacheStatsMixin.__str__Summary) total_size total_csize unique_size unique_csizetotal_unique_chunks total_chunksc Csddlm}|j\}}}}}}d\}}|jjD] }||j|j|j||jd} | j |dd} || j 7}|| j 7}q| |||||| } | S)Nr)Archive)rr)rF)Z want_unique)archiverchunksZ summarizerWarchivesr,rMrZ calc_statsZosizecsizer_asdict) r6rrrrrrr archive_namerstatsr7r7r8rs     zCacheStatsMixin.statscCs8|}dD] }t|||jd||<q|jdi|S)N)rrrrr7)rrrr)r6rfieldr7r7r8rszCacheStatsMixin.format_tuplecCs |dS)Nr)rrDr7r7r8chunks_stored_sizerz"CacheStatsMixin.chunks_stored_sizeN)F) rwrxryrr9rrrrrrr7r7r7r8rs  rc@seZdZdZdddddeddfddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZddZddZddZdd Zd!d"Zd/d#d$Zd0d%d&Zd1d'd(Zd2d)d*Zd+d,Zd-d.ZdS)3rz0 Persistent, local (client-side) cache. NTFc Cstj|| d||_||_||_||_| |_| |_d|_d|_ t |||_ t ||_ t|j|j ||_tj |j sF|j |||||z1|j j|||jd|s]|||rv|jj|jjkry||WdSWdSWdS|)aO :param warn_if_unencrypted: print warning if accessing unknown unencrypted repository :param lock_wait: timeout for lock acquisition (int [s] or None [wait forever]) :param sync: do :meth:`.sync` :param cache_mode: what shall be compared in the file stat infos vs. cached stat infos comparison rNF)rc)rr9r,rMrWrrrrU txn_activer/r1r*security_managerrtrcr0r:rrrrFrucheck_cache_compatibility wipe_cacheupdate_compatibilityrrrcommitr) r6r,rMrWr1rrorrprrrr7r7r8r9s8    zLocalCache.__init__cC|Sr?r7rDr7r7r8rzLocalCache.__enter__cCrr?rrr7r7r8rrzLocalCache.__exit__cCst|jttj|jdd }|tWdn1s!wY|jt tj|jdttj|jdt tj|jt ddWdn1sZwYt tj|jdd d}t j |j|d d WddS1swYdS) z0Create a new empty cache at `self.path` READMEwNrchunks.archive.dTbinary pre12-metaFindent)r0makedirsr1rFr2rVrrcrr r&rjsondumprr6rNr7r7r8rs   "zLocalCache.createc Cs|jttj|jdd|jjdd}t ||_ Wdn1s(wYd|j vr6d|_ n| z%ttj|jd}t||_WdWdS1sXwYWdSttjfylYdSw)NrFr1rVrdr)rcrr"r0r1r2rrr rGrrr _read_filesrFrrr^JSONDecodeErrorrr7r7r8_do_open s"   &zLocalCache._do_opencCs2tj|jstd|j|j|dS)Nz"%s Does not look like a Borg cache)r0r1isdirrrcrFrollbackrDr7r7r8rFs  zLocalCache.opencCsjttj|jdd}tj|j|ddWdn1swY|jdur3|jd|_dSdS)Nrrrr) rFr0r1r2rrrrcrrr7r7r8r s   zLocalCache.closec Csi|_d|_tdtdd}zrttj|jt d|j j t dT}t jdd} |d}|s7n=||z|D]\}}t|}t |j|jdd |j|<q?Wnttfyr}z d t|}WYd}~nd}~wwq/Wdn1s~wYWn/ty}z d t|}WYd}~nd}~wty} z d t| }WYd} ~ nd} ~ ww|durt|td i|_tdt|jdS)NzReading files cache ...zFILES-CACHE-LOAD: starting...FrT)Zuse_listir)agez#The files cache seems invalid. [%s]z#The files cache can't be read. [%s]z"The files cache is corrupted. [%s]z:Continuing without files cache - expect lower performance.z.FILES-CACHE-LOAD: finished, %d entries loaded.)r_newest_cmtimerKrRfiles_cache_loggerr"r0r1r2rrcrrrZUnpackerrGfeedr)packb_replacer TypeError ValueErrorrHrJr$rLlen) r6rdrNudata path_hashitementryrPfier7r7r8r(sT           zLocalCache._read_filesc Cstdd}tj|jd}t||dttj|jd||dttj|jd||dzttj|jt|Wn%t ymt tj|td d Wdn1sfwYYnwt tj|jdtj|jd d |_ | dS) Nzcache.begin_transactionmsgidtxn.tmpz.Initializing cache transaction: Reading configrz.Initializing cache transaction: Reading chunksrz-Initializing cache transaction: Reading filesTr txn.active)rr0r1r2mkdiroutputr;copyrr^r&renamerfinish)r6pitxn_dirr7r7r8 begin_txnIs*       zLocalCache.begin_txncCs|jsdS|j|j|jtdd}|jdur|jdur d|_tt j dd}| dt dtt j|jtd d >}d }|jD].\}}tt|}|jd kr`t|j|jksj|jd krv|j|krvt||f||d 7}qHWdn1swYt d |t d|jt d||j|jjt<| dtt j|jdd d }|j|Wdn1swY|j|jjd<| d|j|j|jt t j|jdt j|jdt!"t j|jdd|_|#dS)zCommit transaction Nz cache.commitrlZBORG_FILES_CACHE_TTLzSaving files cachezFILES-CACHE-SAVE: starting...Tr1rVrrz>FILES-CACHE-KILL: removed all old entries with age >= TTL [%d]zCFILES-CACHE-KILL: removed all current entries with newest cmtime %dz7FILES-CACHE-SAVE: finished, %d remaining entries saved.zSaving chunks cacherzSaving cache configrrF)$rrrXrWrMrrrintr0rrrrrRr"r1r2rrr)runpackbrrcmtimepackrrcrrrVrr;r<r)r6rZttlrNZ entry_countr r rr7r7r8r]sL          zLocalCache.commitcCstjtj|jdrttj|jdtj|jd}tj|rnttj|d|jttj|d|jttj|t||jt|tj|jdtjtj|jdrnttj|jdd|_ | dS)z3Roll back partial and aborted transactions rrrrFN) r0r1r:r2r;r<rrrrr)r6rr7r7r8rs  zLocalCache.rollbackcs`tjjdtd ddd d dfdd fddfdd fd d d fd d  fdd fdd fdd fddfdd  fdd}fdd}jtjjf t j j d|tj _|j_WddS1swYdS)!aRe-synchronize chunks cache with repository. Maintains a directory with known backup archive indexes, so it only needs to fetch infos from repo and build a chunk index once per backup archive. If out of sync, missing archive indexes get added, outdated indexes get removed and a new master chunks index is built by merging all archive indexes. rrrfcst|}tj||}|Sr?)rr0r1r2)rrZid_hexr1) archive_pathr7r8mkpathszLocalCache.sync..mkpathcs2jrt}dd|Ddd|DBStS)NcSs h|] }t|dkrt|qS)@)r rrr7r7r8 s z;LocalCache.sync..cached_archives..cSs2h|]}t|dkr|drt|ddqS)H.compactNr$)r endswithrrr7r7r8r%s2)do_cacher0rr)fnsr"r6r7r8cached_archivess    z(LocalCache.sync..cached_archivescsddjjDS)NcSsh|]}|jqSr7)r)r@infor7r7r8r%sz9LocalCache.sync..repo_archives..)rWrlistr7rDr7r8 repo_archivesrz&LocalCache.sync..repo_archivescs|D]}|qdSr?r7)idsr)cleanup_cached_archiver7r8cleanup_outdateds z)LocalCache.sync..cleanup_outdatedTcszt|t|dWn tyYnw|s dSzt|ddt|dddWdStyAYdSw)Nz .integrityr'r)r0unlinkr^)rcleanup_compact)r#r7r8r1s  z/LocalCache.sync..cleanup_cached_archivecs|}g}tr4|D]&}|}|r-||j|jd}|j|jks(Jd|||<q ||q n|}t|j |D] \}}||jt|d}|||<||<d7t|7q?dS)z Archives created with AdHocCache will have csize=0 in all chunk list entries whose chunks were already in the repository. Scan *chunk_idx* for entries where csize=0 and fill in the correct information. )rzChunk size mismatchrN) Zzero_csize_idsr rrrsizeappendzipr,get_many) chunk_idxZall_missing_idsZ fetch_idsid_Zalready_fetched_entryrr )chunks_fetched_size_indexdecrypted_repositoryfetched_bytes_for_csizefetched_chunks_for_csizer7r8fetch_missing_csizes&     z,LocalCache.sync..fetch_missing_csizec s||\}}||dt||jj|dd\}}}t|d}|jdkr*tdt|}t |j | |j D]\} \}}|| dt||t|7d7| |q8j rf|||dSdS)NrT)Zforce_tam_not_required)Z internal_dictz Unknown archive metadata version)raddr rMZunpack_and_verify_archiverrrr r7rr8rr)) archive_idr<r9rr rZverified_rZitem_id)r?processed_item_metadata_bytesprocessed_item_metadata_chunksr6write_archive_indexr7r8fetch_and_build_idxs      z,LocalCache.sync..fetch_and_build_idxcs|7|dd}|dd}z"t|dt|dd }||Wdn1s.wYWntyBt|YdSwt||dS)Nr'rz.tmpT)r1rVfilename)compactr#rrVrrr0r)rAr9rZfn_tmprN)"compact_chunks_archive_saved_spacer#r7r8rEs      z,LocalCache.sync..write_archive_indexc s|}td|zPz)t|ddd}tj|dd}Wdn1s&wY|dd|WWStyXt|dd }t|}Wdn1sQwYYnwWn"ty|}ztd|||tt WYd}~dSd}~wwt d |||||S) Nz-Reading cached archive chunk index for %s ...r'FrT)Zpermit_compact)r4z1Cached archive chunk index of %s is corrupted: %sz6Found non-compact index for %s, converting to compact.) rKr-r#r rGr^r$errorrrrR)rArZarchive_chunk_idx_pathrNarchive_chunk_idxr)r1r#rEr7r8read_archive_indexs6      z+LocalCache.sync..read_archive_indexcsDi}jjD] }|j|vr|j||j<qt|t|ks J|Sr?)rWrr.rnamer ) archive_idsZ archive_namesr-rDr7r8get_archive_ids_to_names/s  z1LocalCache.sync..get_archive_ids_to_namesc std} }tdt|t|t||t|||||t j}|rƈ js6dnt|d}tt|dddd}|}|D]S\}}|j t |gd jr||vro ||}|duro| |||vrtd |t}||td | |qM|pt|d}td |||qM js|| td ttd t  tdttd|S)NzSynchronizing chunks cache...zIArchives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.Zusableg?z3%3.0f%% Syncing chunks cache. Processing archive %sz cache.sync)totalsteprdr)r-z.Fetching and building archive index for %s ...z$Merging into master chunks index ...z!Fetching archive index for %s ...zlCache sync: had to fetch %s (%d chunks) because no archive had a csize set for them (due to --no-cache-sync)z0Cache sync: processed %s (%d chunks) of metadataz;Cache sync: compact chunks.archive.d storage saved %s byteszDone.)rKr-r clearr,r)r rrshowrrmergerrRr) r9Z cached_idsrNZmaster_index_capacityrZarchive_ids_to_namesrArrK)r,r2rIr<rFr?r=r>rOrCrDrLr/r6r7r8create_master_idx;s^             z*LocalCache.sync..create_master_idxcsfz ttjjdWnYz ttjjdWnYztWdSYdS)z?bring old cache dirs into the desired state (cleanup and adapt)zchunks.archivezchunks.archive.tmpN)r0r3r1r2rr7r+r7r8legacy_cleanupnsz'LocalCache.sync..legacy_cleanup)Zdecrypted_cacheN)rf)T)r0r1r2r rWZcheck_repository_compatibilityrZ OperationZREADrr'r,rMrr)r)r6rVrWr7)r"r,r;r1r2rIr<rFr?r=r>rOr#rCrDrLr/r6rEr8rs4  ! & 3"zLocalCache.synccCs*tj}|jj|@r dS|jj|ksdSdS)NFT)rSUPPORTED_REPO_FEATURESrcrr)r6 my_featuresr7r7r8rs   z$LocalCache.check_cache_compatibilitycCstdtj|jd}tj|r)ttj|jdttj|jdt |_ t tj|jt ddWdn1sEwYd|j _|j jdddt|j _t|j _dS)Nz9Discarding incompatible cache and forcing a cache rebuildrTrrfrrW)rKrLr0r1r2rr;r<rr rr&rrcrrrrr)r6r"r7r7r8rs   zLocalCache.wipe_cachecCsZ|j}tj}t}|D] \}}||q|jj|||jj ||@dSr?) rWZget_all_mandatory_featuresrrXrrupdatercrr)r6Zoperation_to_features_maprYZ repo_featuresZ operationfeaturesr7r7r8rs  zLocalCache.update_compatibilityc Cs|js|t|}|||}|r|s|||S|j|}t|} |jj|||d|j |d|| | || | t ||| S)Nwaitr) rrr  seen_chunk chunk_increfrMencryptr,putrr@rZr r6rchunkrZ overwriter]r5refcountr rr7r7r8 add_chunks    zLocalCache.add_chunkcCsJ|j|tddd\}}}|dur#|dur#||kr#td|||f|S)Nrz@chunk has same id [%r], but different size (stored: %d new: %d)!)rrr r)r6rr5rdZ stored_sizerBr7r7r8r^s zLocalCache.seen_chunkcCs>|js||j|\}}}|j||d|dt|||SNFpart)rrrincrefrZr r6rrr5rhcount_sizerr7r7r8r_s  zLocalCache.chunk_increfcCt|js||j|\}}}|dkr-|j|=|jj||d|j| | d|ddS|j| | d|ddSNrr\TrgF)rrrdecrefr,deleterZr6rrr]rhrkr5rr7r7r8 chunk_decrefzLocalCache.chunk_decrefcCs.t|jsdS|j}d|vrtddSd|vr!tddS|j|}|s1td|dStt |}d|vrJ|j |j krJtd|d Sd |vr\|j |jkr\td |d Sd |vrpt|j|jkrptd |d Sd|vrt|j|jkrtd|d St |j|jdd|j|<d|jfS)aj Check if we know the file that has this path_hash (know == it is in our files cache) and whether it is unchanged (the size/inode number/cmtime is same for stuff we check in this cache_mode). :param hashed_path: the file's path as we gave it to hash(hashed_path) :param path_hash: hash(hashed_path), to save some memory in the files cache :param st: the file's stat() result :return: known, ids (known is True if we have infos about this file in the cache, ids is the list of chunk ids IF the file has not changed, otherwise None). FNrzUNKNOWN: files cache disabledrzUNKNOWN: rechunking enforcedz*UNKNOWN: no file metadata in cache for: %rsz(KNOWN-CHANGED: file size has changed: %r)TNiz0KNOWN-CHANGED: file inode number has changed: %rcz)KNOWN-CHANGED: file ctime has changed: %rmz)KNOWN-CHANGED: file mtime has changed: %rr)inoderT)statS_ISREGst_moderrrRrrr)rrr5st_sizerzst_inorr  st_ctime_ns st_mtime_nsrr chunk_ids)r6 hashed_pathr strrr7r7r8file_known_and_unchangeds8          z#LocalCache.file_known_and_unchangedc Cst|jsdS|j}d|vrtddSd|vr"d}t|j}nd|vr.d}t|j}nd}t|j}t d|j |j t ||d}t ||j|<t|jpOd||_td |jd t|jd ||dS) Nrz*FILES-CACHE-NOUPDATE: files cache disabledrxctimerymtimer)rrzr5r rz)FILES-CACHE-UPDATE: put %r [has %s] <- %rz [%d entries])r)r{r|r}rrrRrrrr)rr~rrrrrgrrr r) r6rr rr0rZ cmtime_typeZ cmtime_nsrr7r7r8 memorize_file s*     zLocalCache.memorize_fileFTr?NFTF)rwrxryrzrr9rrrrrFrrrrrrrrrrer^r_rrrrr7r7r7r8rs6 ,!)s    4rc@seZdZdZdZ  dddZdd Zd d ZdZd Z d dZ ddZ d ddZ d!ddZ d"ddZd#ddZddZddZddZdS)$ra Ad-hoc, non-persistent cache. Compared to the standard LocalCache the AdHocCache does not maintain accurate reference count, nor does it provide a files cache (which would require persistence). Chunks that were not added during the current AdHocCache lifetime won't have correct size/csize set (0 bytes) and will have an infinite reference count (MAX_VALUE). zAll archives: unknown unknown unknown Unique chunks Total chunks Chunk index: {0.total_unique_chunks:20d} unknownTNFcCs\tj||d||_||_||_||_d|_t||_|jj |||di|_ t ddS)NrFrqz1Note: --no-cache-sync is an experimental feature.) rr9r,rMrWr _txn_activer*rrurrKrL)r6r,rMrWrorprrr7r7r8r95s zAdHocCache.__init__cCrr?r7rDr7r7r8rFrzAdHocCache.__enter__cCdSr?r7rr7r7r8rIrzAdHocCache.__exit__rcCstddS)Nz$UNKNOWN: files cache not implementedrt)rrR)r6rr rr7r7r8rOs z#AdHocCache.file_known_and_unchangedcCrr?r7)r6rr rr0r7r7r8rSrzAdHocCache.memorize_filec Cs|rJd|js |t|}|||}|r!|j|||dS|j|}t|} |jj|||d|j |d|| | || | t ||| S)NuHAdHocCache does not permit overwrites — trying to use it for recreate?r5r\r) rrr r^r_rMr`r,rarr@rZr rbr7r7r8reVs    zAdHocCache.add_chunkcCsL|js||j|tddd}|jr#|r#|js#|j|d|j|<|jS)Nrr)rrrrr rdr5r)r6rr5rr7r7r8r^es zAdHocCache.seen_chunkcCsN|js||j|\}}}|p|}|sJ|j||d|dt|||Srf)rrrrirZr rjr7r7r8r_ps zAdHocCache.chunk_increfcCrmrn)rrrror,rprZrqr7r7r8rr{rszAdHocCache.chunk_decrefcCs&|jsdS|j|j|jd|_dSr)rrrXrWrMrDr7r7r8rs zAdHocCache.commitcCsd|_|`dSr)rrrDr7r7r8rszAdHocCache.rollbackc Csd|_t|j}t|dd|_t|ddd}t}d}d} |jjt|d}|d 7}|s.n|j t|d |d }t tj ddd }|D]}||j|<qDqt|j|ksVJ|j|j j =t|pbd } |td|| |t|d| dS)NTg?rPz!Downloading chunk list... %3.0f%%zcache.download_chunks)rQrdrr)limitmarkerr)Zincrease)rdr5rg{Gz?zBAdHocCache: downloaded %d chunk IDs in %.2f s (%d requests), ~%s/s")rr r,r rrrr.r(rTr Z MAX_VALUErWZ MANIFEST_IDrrKrRr) r6Z num_chunksrt0Z num_requestsrresultZ init_entryr:Zdurationr7r7r8rs6   zAdHocCache.begin_txn)TNFFrr?rr)rwrxryrzrr9rrrrrrrer^r_rrrrrr7r7r7r8r%s&       rr?)Frrr0r;r{binasciir collectionsrtimerrKrr constantsrrZ hashindexr r r helpersr r rrrrrrrrrrrrrrrrrr rr Z crypto.keyr!Zcrypto.file_integrityr"r#r$Zlockingr%platformr&Zremoter'r,r(r)r*rurr/rrrtrarrrr7r7r7r8sd                   % dA0c