o  cJ@sddlZddlZddlmZddlmZmZddlmZm Z m Z m Z m Z ddl mZmZmZmZddlmZmZmZmZddlmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"m#Z#e$d Z%e&Z'e'(e)d       d&de*de+de+de,de e e-de e e-de.de.defddZ/      d&de de+de+de,de e e-de e e-de.de.defddZ0      d&d d!de+de+de,de e e-de e e-de.de.defd"d#Z1     d'd d!de+de+de,de e e-de e e-de.defd$d%Z2dS)(N)PathLike)basenamesplitext)AnyBinaryIOListOptionalSet)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bom is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s皙?TF sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc- Cst|ttfstdt||rtj}tt t t t |} | dkrGt d|r;tt t |p9tjtt|dddgdgS|dur]tt d d |d d |D}ng}|durutt d d |dd |D}ng}| ||krtt d||| d}| }|dkr| ||krt| |}t |tk} t |tk} | rtt d| n | rtt d| g} |rt|nd} | dur| | tt d| t}g}g}d}d}d}t}t|\}}|dur| |tt dt ||| dd| vr| d| tD]}|r!||vr!q|r+||vr+q||vr2q||d}||k}|oCt|}|dvrU|sUtt d|qzt|}Wnt t!fyott d|Yqwz9| r|durt"|dur|dtdn |t |td|dnt"|dur|n|t |d|d}Wn+t#t$fy}zt|t$stt d|t"|||WYd}~qd}~wwd}|D] }t%||rd}nq|rtt d||qt&|sdnt || t| |}|o|duot || k} | rtt d|tt |d}!t'|!d }!d}"d}#g}$g}%z9t(||||||||| D]*}&|$|&|%t)|&||%d!|krY|"d7}"|"|!ksf|rh|durhnq?Wn!t#y}ztt d"|t"||!}"d}#WYd}~nd}~ww|#s| r|sz|td#dj*|d$d%Wn#t#y}ztt d&|t"|||WYd}~qd}~ww|%rt+|%t |%nd}'|'|ks|"|!kr||tt d'||"t,|'d(d)d*|dd| fvr|#st|||dg|}(|| kr|(}n |dkr|(}n|(}qtt d+|t,|'d(d)d*|s2t-|})nt.|})|)rEtt d,|t"|)g}*|dkre|$D]}&t/|&d-|)r[d.|)nd}+|*|+qNt0|*},|,rvtt d/|,||t|||'||,||| ddfvr|'d-krt d0||rtt t |t||gS||krt d1||rtt t |t||gSqt |dkr&|s|s|rtt d2|rt d3|j1||n2|r|dus|r |r |j2|j2ks|durt d4||n |r&t d5|||r8t d6|3j1t |dnt d7|rJtt t ||S)8ae Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z4Expected object of type bytes or bytearray, got: {0}rz[zfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSr,r-r.r/r2r2r3r4fr5z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_32utf_16z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelrlendebug removeHandlerloggingWARNINGrrlogjoinintrrrappendsetrraddrrModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorrrangemaxrrdecodesumroundr r r rr9Z fingerprintbest)-r!r"r#r$r%r&r'r(Zprevious_logger_levellengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specifiedresultsZ sig_encodingZ sig_payloadZ encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decodereZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratioschunkZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedr2r2r3 from_bytes#s                                                          rhfpc Cst||||||||S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rhread)rir"r#r$r%r&r'r(r2r2r3from_fpsrkpathz PathLike[Any]c CsDt|d}t||||||||WdS1swYdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openrk) rlr"r#r$r%r&r'r(rir2r2r3 from_paths $roc Cstdtt|||||||}t|}tt|} t|dkr'td || } | dd| j 7<t d t ||d| d} | | Wd| S1sZwY| S) zi Take a (text-based) file path and try to create another file next to it, this time using UTF-8. z2normalize is deprecated and will be removed in 3.0rz;Unable to normalize "{}", no encoding charset seems to fit.-z{}r+wbN)warningswarnDeprecationWarningrorlistrrNIOErrorrGrcr9rnr[replacerTwriteoutput) rlr"r#r$r%r&r'refilenameZtarget_extensionsresultrir2r2r3 normalizes@    r|)rrr NNTF)rrr NNT)3rQrrosros.pathrrtypingrrrrr cdr r r rZconstantrrrrZmdrZmodelsrrutilsrrrrrrr getLoggerrI StreamHandlerrL setFormatter FormatterrErUfloatr[boolrhrkror|r2r2r2r3s  $    G