Class Snappy

java.lang.Object
io.netty.handler.codec.compression.Snappy

public final class Snappy extends Object
Uncompresses an input ByteBuf encoded with Snappy compression into an output ByteBuf. See snappy format.
  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    private static enum 
     
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    private static final int
     
    private static final int
     
    private static final int
     
    private static final boolean
     
    private static final FastThreadLocal<short[]>
     
    private static final int
     
    private static final int
     
    private static final int
     
    private static final int
     
    private static final int
     
    private final boolean
     
    private Snappy.State
     
    private byte
     
    private int
     
  • Constructor Summary

    Constructors
    Constructor
    Description
     
    Snappy(boolean reuseHashtable)
     
  • Method Summary

    Modifier and Type
    Method
    Description
    private static int
    bitsToEncode(int value)
    Calculates the minimum number of bits required to encode a value.
    (package private) static int
    Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
    (package private) static int
    calculateChecksum(ByteBuf data, int offset, int length)
    Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
    void
     
    private static int
    decodeCopyWith1ByteOffset(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
    Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
    private static int
    decodeCopyWith2ByteOffset(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
    Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
    private static int
    decodeCopyWith4ByteOffset(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
    Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
    (package private) static int
    decodeLiteral(byte tag, ByteBuf in, ByteBuf out)
    Reads a literal from the input buffer directly to the output buffer.
    void
    encode(ByteBuf in, ByteBuf out, int length)
     
    private static void
    encodeCopy(ByteBuf out, int offset, int length)
    Encodes a series of copies, each at most 64 bytes in length.
    private static void
    encodeCopyWithOffset(ByteBuf out, int offset, int length)
     
    (package private) static void
    encodeLiteral(ByteBuf in, ByteBuf out, int length)
    Writes a literal to the supplied output buffer by directly copying from the input buffer.
    private static int
    findMatchingLength(ByteBuf in, int minIndex, int inIndex, int maxIndex)
    Iterates over the supplied input buffer between the supplied minIndex and maxIndex to find how long our matched copy overlaps with an already-written literal value.
    private short[]
    getHashTable(int hashTableSize)
    Returns a short[] to be used as a hashtable
    static short[]
    Returns a short[] from a FastThreadLocal, zeroing for correctness creating a new one and resizing it if necessary
    (package private) int
    Get the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
    private static int
    hash(ByteBuf in, int index, int shift)
    Hashes the 4 bytes located at index, shifting the resulting hash into the appropriate range for our hash table.
    (package private) static int
    maskChecksum(long checksum)
    From the spec: "Checksums are not stored directly, but masked, as checksumming data and then its own checksum can be problematic.
    private static int
    Reads the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
    void
     
    (package private) static void
    validateChecksum(int expectedChecksum, ByteBuf data)
    Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
    (package private) static void
    validateChecksum(int expectedChecksum, ByteBuf data, int offset, int length)
    Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
    private static void
    validateOffset(int offset, int chunkSizeSoFar)
    Validates that the offset extracted from a compressed reference is within the permissible bounds of an offset (0 invalid input: '<' offset invalid input: '<' Integer.MAX_VALUE), and does not exceed the length of the chunk currently read so far.
    static Snappy
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • MAX_HT_SIZE

      private static final int MAX_HT_SIZE
      See Also:
    • MIN_COMPRESSIBLE_BYTES

      private static final int MIN_COMPRESSIBLE_BYTES
      See Also:
    • PREAMBLE_NOT_FULL

      private static final int PREAMBLE_NOT_FULL
      See Also:
    • NOT_ENOUGH_INPUT

      private static final int NOT_ENOUGH_INPUT
      See Also:
    • LITERAL

      private static final int LITERAL
      See Also:
    • COPY_1_BYTE_OFFSET

      private static final int COPY_1_BYTE_OFFSET
      See Also:
    • COPY_2_BYTE_OFFSET

      private static final int COPY_2_BYTE_OFFSET
      See Also:
    • COPY_4_BYTE_OFFSET

      private static final int COPY_4_BYTE_OFFSET
      See Also:
    • HASH_TABLE

      private static final FastThreadLocal<short[]> HASH_TABLE
    • DEFAULT_REUSE_HASHTABLE

      private static final boolean DEFAULT_REUSE_HASHTABLE
    • reuseHashtable

      private final boolean reuseHashtable
    • state

      private Snappy.State state
    • tag

      private byte tag
    • written

      private int written
  • Constructor Details

    • Snappy

      public Snappy()
    • Snappy

      Snappy(boolean reuseHashtable)
  • Method Details

    • withHashTableReuse

      public static Snappy withHashTableReuse()
    • reset

      public void reset()
    • encode

      public void encode(ByteBuf in, ByteBuf out, int length)
    • hash

      private static int hash(ByteBuf in, int index, int shift)
      Hashes the 4 bytes located at index, shifting the resulting hash into the appropriate range for our hash table.
      Parameters:
      in - The input buffer to read 4 bytes from
      index - The index to read at
      shift - The shift value, for ensuring that the resulting value is within the range of our hash table size
      Returns:
      A 32-bit hash of 4 bytes located at index
    • getHashTable

      private short[] getHashTable(int hashTableSize)
      Returns a short[] to be used as a hashtable
      Parameters:
      hashTableSize - the size for the hashtable
      Returns:
      An appropriately sized empty hashtable
    • getHashTableFastThreadLocalArrayFill

      public static short[] getHashTableFastThreadLocalArrayFill(int hashTableSize)
      Returns a short[] from a FastThreadLocal, zeroing for correctness creating a new one and resizing it if necessary
      Parameters:
      hashTableSize -
      Returns:
      An appropriately sized empty hashtable
    • findMatchingLength

      private static int findMatchingLength(ByteBuf in, int minIndex, int inIndex, int maxIndex)
      Iterates over the supplied input buffer between the supplied minIndex and maxIndex to find how long our matched copy overlaps with an already-written literal value.
      Parameters:
      in - The input buffer to scan over
      minIndex - The index in the input buffer to start scanning from
      inIndex - The index of the start of our copy
      maxIndex - The length of our input buffer
      Returns:
      The number of bytes for which our candidate copy is a repeat of
    • bitsToEncode

      private static int bitsToEncode(int value)
      Calculates the minimum number of bits required to encode a value. This can then in turn be used to calculate the number of septets or octets (as appropriate) to use to encode a length parameter.
      Parameters:
      value - The value to calculate the minimum number of bits required to encode
      Returns:
      The minimum number of bits required to encode the supplied value
    • encodeLiteral

      static void encodeLiteral(ByteBuf in, ByteBuf out, int length)
      Writes a literal to the supplied output buffer by directly copying from the input buffer. The literal is taken from the current readerIndex up to the supplied length.
      Parameters:
      in - The input buffer to copy from
      out - The output buffer to copy to
      length - The length of the literal to copy
    • encodeCopyWithOffset

      private static void encodeCopyWithOffset(ByteBuf out, int offset, int length)
    • encodeCopy

      private static void encodeCopy(ByteBuf out, int offset, int length)
      Encodes a series of copies, each at most 64 bytes in length.
      Parameters:
      out - The output buffer to write the copy pointer to
      offset - The offset at which the original instance lies
      length - The length of the original instance
    • decode

      public void decode(ByteBuf in, ByteBuf out)
    • readPreamble

      private static int readPreamble(ByteBuf in)
      Reads the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
      Parameters:
      in - The input buffer to read the preamble from
      Returns:
      The calculated length based on the input buffer, or 0 if no preamble is able to be calculated
    • getPreamble

      int getPreamble(ByteBuf in)
      Get the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
      Parameters:
      in - The input buffer to get the preamble from
      Returns:
      The calculated length based on the input buffer, or 0 if no preamble is able to be calculated
    • decodeLiteral

      static int decodeLiteral(byte tag, ByteBuf in, ByteBuf out)
      Reads a literal from the input buffer directly to the output buffer. A "literal" is an uncompressed segment of data stored directly in the byte stream.
      Parameters:
      tag - The tag that identified this segment as a literal is also used to encode part of the length of the data
      in - The input buffer to read the literal from
      out - The output buffer to write the literal to
      Returns:
      The number of bytes appended to the output buffer, or -1 to indicate "try again later"
    • decodeCopyWith1ByteOffset

      private static int decodeCopyWith1ByteOffset(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
      Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
      Parameters:
      tag - The tag used to identify this as a copy is also used to encode the length and part of the offset
      in - The input buffer to read from
      out - The output buffer to write to
      Returns:
      The number of bytes appended to the output buffer, or -1 to indicate "try again later"
      Throws:
      DecompressionException - If the read offset is invalid
    • decodeCopyWith2ByteOffset

      private static int decodeCopyWith2ByteOffset(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
      Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
      Parameters:
      tag - The tag used to identify this as a copy is also used to encode the length and part of the offset
      in - The input buffer to read from
      out - The output buffer to write to
      Returns:
      The number of bytes appended to the output buffer, or -1 to indicate "try again later"
      Throws:
      DecompressionException - If the read offset is invalid
    • decodeCopyWith4ByteOffset

      private static int decodeCopyWith4ByteOffset(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
      Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
      Parameters:
      tag - The tag used to identify this as a copy is also used to encode the length and part of the offset
      in - The input buffer to read from
      out - The output buffer to write to
      Returns:
      The number of bytes appended to the output buffer, or -1 to indicate "try again later"
      Throws:
      DecompressionException - If the read offset is invalid
    • validateOffset

      private static void validateOffset(int offset, int chunkSizeSoFar)
      Validates that the offset extracted from a compressed reference is within the permissible bounds of an offset (0 invalid input: '<' offset invalid input: '<' Integer.MAX_VALUE), and does not exceed the length of the chunk currently read so far.
      Parameters:
      offset - The offset extracted from the compressed reference
      chunkSizeSoFar - The number of bytes read so far from this chunk
      Throws:
      DecompressionException - if the offset is invalid
    • calculateChecksum

      static int calculateChecksum(ByteBuf data)
      Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
      Parameters:
      data - The input data to calculate the CRC32C checksum of
    • calculateChecksum

      static int calculateChecksum(ByteBuf data, int offset, int length)
      Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
      Parameters:
      data - The input data to calculate the CRC32C checksum of
    • validateChecksum

      static void validateChecksum(int expectedChecksum, ByteBuf data)
      Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
      Parameters:
      expectedChecksum - The checksum decoded from the stream to compare against
      data - The input data to calculate the CRC32C checksum of
      Throws:
      DecompressionException - If the calculated and supplied checksums do not match
    • validateChecksum

      static void validateChecksum(int expectedChecksum, ByteBuf data, int offset, int length)
      Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
      Parameters:
      expectedChecksum - The checksum decoded from the stream to compare against
      data - The input data to calculate the CRC32C checksum of
      Throws:
      DecompressionException - If the calculated and supplied checksums do not match
    • maskChecksum

      static int maskChecksum(long checksum)
      From the spec: "Checksums are not stored directly, but masked, as checksumming data and then its own checksum can be problematic. The masking is the same as used in Apache Hadoop: Rotate the checksum by 15 bits, then add the constant 0xa282ead8 (using wraparound as normal for unsigned integers)."
      Parameters:
      checksum - The actual checksum of the data
      Returns:
      The masked checksum