Class BasePackFetchConnection

All Implemented Interfaces:
AutoCloseable, Connection, FetchConnection
Direct Known Subclasses:
InternalFetchConnection, TransportGitAnon.TcpFetchConnection, TransportGitSsh.SshFetchConnection, TransportHttp.SmartHttpFetchConnection, TransportLocal.ForkLocalFetchConnection

public abstract class BasePackFetchConnection extends BasePackConnection implements FetchConnection
Fetch implementation using the native Git pack transfer service.

This is the canonical implementation for transferring objects from the remote repository to the local repository by talking to the 'git-upload-pack' service. Objects are packed on the remote side into a pack file and then sent down the pipe to us.

This connection requires only a bi-directional pipe or socket, and thus is easily wrapped up into a local process pipe, anonymous TCP socket, or a command executed through an SSH tunnel.

If BasePackConnection.statelessRPC is true, this connection can be tunneled over a request-response style RPC system like HTTP. The RPC call boundary is determined by this class switching from writing to the OutputStream to reading from the InputStream.

Concrete implementations should just call BasePackConnection.init(java.io.InputStream, java.io.OutputStream) and BasePackConnection.readAdvertisedRefs() methods in constructor or before any use. They should also handle resources releasing in close() method if needed.

  • Field Details

    • MAX_HAVES

      private static final int MAX_HAVES
      Maximum number of 'have' lines to send before giving up.

      During negotiate(ProgressMonitor) we send at most this many commits to the remote peer as 'have' lines without an ACK response before we give up.

      See Also:
    • MIN_CLIENT_BUFFER

      protected static final int MIN_CLIENT_BUFFER
      Amount of data the client sends before starting to read.

      Any output stream given to the client must be able to buffer this many bytes before the client will stop writing and start reading from the input stream. If the output stream blocks before this many bytes are in the send queue, the system will deadlock.

      See Also:
    • OPTION_INCLUDE_TAG

      public static final String OPTION_INCLUDE_TAG
      Include tags if we are also including the referenced objects.
      Since:
      2.0
      See Also:
    • OPTION_MULTI_ACK

      public static final String OPTION_MULTI_ACK
      Multi-ACK support for improved negotiation.
      Since:
      2.0
      See Also:
    • OPTION_MULTI_ACK_DETAILED

      public static final String OPTION_MULTI_ACK_DETAILED
      Multi-ACK detailed support for improved negotiation.
      Since:
      2.0
      See Also:
    • OPTION_THIN_PACK

      public static final String OPTION_THIN_PACK
      The client supports packs with deltas but not their bases.
      Since:
      2.0
      See Also:
    • OPTION_SIDE_BAND

      public static final String OPTION_SIDE_BAND
      The client supports using the side-band for progress messages.
      Since:
      2.0
      See Also:
    • OPTION_SIDE_BAND_64K

      public static final String OPTION_SIDE_BAND_64K
      The client supports using the 64K side-band for progress messages.
      Since:
      2.0
      See Also:
    • OPTION_OFS_DELTA

      public static final String OPTION_OFS_DELTA
      The client supports packs with OFS deltas.
      Since:
      2.0
      See Also:
    • OPTION_SHALLOW

      public static final String OPTION_SHALLOW
      The client supports shallow fetches.
      Since:
      2.0
      See Also:
    • OPTION_NO_PROGRESS

      public static final String OPTION_NO_PROGRESS
      The client does not want progress messages and will ignore them.
      Since:
      2.0
      See Also:
    • OPTION_NO_DONE

      public static final String OPTION_NO_DONE
      The client supports receiving a pack before it has sent "done".
      Since:
      2.0
      See Also:
    • OPTION_ALLOW_TIP_SHA1_IN_WANT

      public static final String OPTION_ALLOW_TIP_SHA1_IN_WANT
      The client supports fetching objects at the tip of any ref, even if not advertised.
      Since:
      3.1
      See Also:
    • OPTION_ALLOW_REACHABLE_SHA1_IN_WANT

      public static final String OPTION_ALLOW_REACHABLE_SHA1_IN_WANT
      The client supports fetching objects that are reachable from a tip of a ref that is allowed to fetch.
      Since:
      4.1
      See Also:
    • OPTION_FILTER

      public static final String OPTION_FILTER
      The client specified a filter expression.
      Since:
      5.0
      See Also:
    • walk

      private final RevWalk walk
    • reachableCommits

      private RevCommitList<RevCommit> reachableCommits
      All commits that are immediately reachable by a local ref.
    • REACHABLE

      final RevFlag REACHABLE
      Marks an object as having all its dependencies.
    • COMMON

      final RevFlag COMMON
      Marks a commit known to both sides of the connection.
    • STATE

      private final RevFlag STATE
      Like COMMON but means its also in pckState.
    • ADVERTISED

      final RevFlag ADVERTISED
      Marks a commit listed in the advertised refs.
    • multiAck

    • thinPack

      private boolean thinPack
    • sideband

      private boolean sideband
    • includeTags

      private boolean includeTags
    • allowOfsDelta

      private boolean allowOfsDelta
    • noDone

      private boolean noDone
    • noProgress

      private boolean noProgress
    • lockMessage

      private String lockMessage
    • packLock

      private PackLock packLock
    • maxHaves

      private int maxHaves
    • state

      private TemporaryBuffer.Heap state
      RPC state, if BasePackConnection.statelessRPC is true or protocol V2 is used.
    • pckState

      private PacketLineOut pckState
    • filterSpec

      private final FilterSpec filterSpec
      Either FilterSpec.NO_FILTER for a filter that doesn't filter anything, or a filter that indicates what and what not to send to the server.
  • Constructor Details

    • BasePackFetchConnection

      public BasePackFetchConnection(PackTransport packTransport)
      Create a new connection to fetch using the native git transport.
      Parameters:
      packTransport - the transport.
  • Method Details

    • fetch

      public final void fetch(ProgressMonitor monitor, Collection<Ref> want, Set<ObjectId> have) throws TransportException
      Fetch objects we don't have but that are reachable from advertised refs.

      Only one call per connection is allowed. Subsequent calls will result in TransportException.

      Implementations are free to use network connections as necessary to efficiently (for both client and server) transfer objects from the remote repository into this repository. When possible implementations should avoid replacing/overwriting/duplicating an object already available in the local destination repository. Locally available objects and packs should always be preferred over remotely available objects and packs. Transport.isFetchThin() should be honored if applicable.

      Specified by:
      fetch in interface FetchConnection
      Parameters:
      monitor - progress monitor to inform the end-user about the amount of work completed, or to indicate cancellation. Implementations should poll the monitor at regular intervals to look for cancellation requests from the user.
      want - one or more refs advertised by this connection that the caller wants to store locally.
      have - additional objects known to exist in the destination repository, especially if they aren't yet reachable by the ref database. Connections should take this set as an addition to what is reachable through all Refs, not in replace of it.
      Throws:
      TransportException - objects could not be copied due to a network failure, protocol error, or error on remote side, or connection was already used for fetch.
    • fetch

      public final void fetch(ProgressMonitor monitor, Collection<Ref> want, Set<ObjectId> have, OutputStream outputStream) throws TransportException
      Fetch objects we don't have but that are reachable from advertised refs.

      Only one call per connection is allowed. Subsequent calls will result in TransportException.

      Implementations are free to use network connections as necessary to efficiently (for both client and server) transfer objects from the remote repository into this repository. When possible implementations should avoid replacing/overwriting/duplicating an object already available in the local destination repository. Locally available objects and packs should always be preferred over remotely available objects and packs. Transport.isFetchThin() should be honored if applicable.

      Specified by:
      fetch in interface FetchConnection
      Parameters:
      monitor - progress monitor to inform the end-user about the amount of work completed, or to indicate cancellation. Implementations should poll the monitor at regular intervals to look for cancellation requests from the user.
      want - one or more refs advertised by this connection that the caller wants to store locally.
      have - additional objects known to exist in the destination repository, especially if they aren't yet reachable by the ref database. Connections should take this set as an addition to what is reachable through all Refs, not in replace of it.
      outputStream - OutputStream to write sideband messages to
      Throws:
      TransportException - objects could not be copied due to a network failure, protocol error, or error on remote side, or connection was already used for fetch.
    • didFetchIncludeTags

      public boolean didFetchIncludeTags()
      Did the last FetchConnection.fetch(ProgressMonitor, Collection, Set) get tags?

      Some Git aware transports are able to implicitly grab an annotated tag if TagOpt.AUTO_FOLLOW or TagOpt.FETCH_TAGS was selected and the object the tag peels to (references) was transferred as part of the last FetchConnection.fetch(ProgressMonitor, Collection, Set) call. If it is possible for such tags to have been included in the transfer this method returns true, allowing the caller to attempt tag discovery.

      By returning only true/false (and not the actual list of tags obtained) the transport itself does not need to be aware of whether or not tags were included in the transfer.

      Specified by:
      didFetchIncludeTags in interface FetchConnection
      Returns:
      true if the last fetch call implicitly included tag objects; false if tags were not implicitly obtained.
    • didFetchTestConnectivity

      public boolean didFetchTestConnectivity()
      Did the last FetchConnection.fetch(ProgressMonitor, Collection, Set) validate graph?

      Some transports walk the object graph on the client side, with the client looking for what objects it is missing and requesting them individually from the remote peer. By virtue of completing the fetch call the client implicitly tested the object connectivity, as every object in the graph was either already local or was requested successfully from the peer. In such transports this method returns true.

      Some transports assume the remote peer knows the Git object graph and is able to supply a fully connected graph to the client (although it may only be transferring the parts the client does not yet have). Its faster to assume such remote peers are well behaved and send the correct response to the client. In such transports this method returns false.

      Specified by:
      didFetchTestConnectivity in interface FetchConnection
      Returns:
      true if the last fetch had to perform a connectivity check on the client side in order to succeed; false if the last fetch assumed the remote peer supplied a complete graph.
    • setPackLockMessage

      public void setPackLockMessage(String message)
      Set the lock message used when holding a pack out of garbage collection.

      Callers that set a lock message must ensure they call FetchConnection.getPackLocks() after FetchConnection.fetch(ProgressMonitor, Collection, Set), even if an exception was thrown, and release the locks that are held.

      Specified by:
      setPackLockMessage in interface FetchConnection
      Parameters:
      message - message to use when holding a pack in place.
    • getPackLocks

      public Collection<PackLock> getPackLocks()
      Specified by:
      getPackLocks in interface FetchConnection
      Returns:
      collection (possibly empty) of locks created by the last call to fetch. The caller must release these after refs are updated in order to safely permit garbage collection.
    • clearState

      private void clearState()
    • doFetch

      protected void doFetch(ProgressMonitor monitor, Collection<Ref> want, Set<ObjectId> have, OutputStream outputStream) throws TransportException
      Execute common ancestor negotiation and fetch the objects.
      Parameters:
      monitor - progress monitor to receive status updates. If the monitor is the NullProgressMonitor.INSTANCE, then the no-progress option enabled.
      want - the advertised remote references the caller wants to fetch.
      have - additional objects to assume that already exist locally. This will be added to the set of objects reachable from the destination repository's references.
      outputStream - ouputStream to write sideband messages to
      Throws:
      TransportException - if any exception occurs.
      Since:
      3.0
    • doFetchV2

      private void doFetchV2(ProgressMonitor monitor, Collection<Ref> want, OutputStream outputStream) throws IOException, BasePackFetchConnection.CancelledException
      Throws:
      IOException
      BasePackFetchConnection.CancelledException
    • sendNextHaveBatch

      private boolean sendNextHaveBatch(BasePackFetchConnection.FetchStateV2 fetchState, PacketLineOut output, ProgressMonitor monitor) throws IOException, BasePackFetchConnection.CancelledException
      Sends the next batch of "have"s and terminates the output.
      Parameters:
      fetchState - is updated with information about the number of items written, and whether to expect a packfile next
      output - to write to
      monitor - for progress reporting and cancellation
      Returns:
      true if a "done" was written and we should thus expect a packfile next
      Throws:
      IOException - on errors
      BasePackFetchConnection.CancelledException - on cancellation
    • readAcknowledgments

      private boolean readAcknowledgments(BasePackFetchConnection.FetchStateV2 fetchState, PacketLineIn input, ProgressMonitor monitor) throws IOException, BasePackFetchConnection.CancelledException
      Reads and processes acknowledgments, adding ACKed objects as "have"s to the global state TemporaryBuffer.
      Parameters:
      fetchState - to update
      input - to read from
      monitor - for progress reporting and cancellation
      Returns:
      true if a "ready" was received and a packfile is expected next
      Throws:
      IOException - on errors
      BasePackFetchConnection.CancelledException - on cancellation
    • close

      public void close()

      Close any resources used by this connection.

      If the remote repository is contacted by a network socket this method must close that network socket, disconnecting the two peers. If the remote repository is actually local (same system) this method must close any open file handles used to read the "remote" repository.

      If additional messages were produced by the remote peer, these should still be retained in the connection instance for Connection.getMessages().

      AutoClosable.close() declares that it throws Exception. Implementers shouldn't throw checked exceptions. This override narrows the signature to prevent them from doing so.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Connection
      Overrides:
      close in class BasePackConnection
    • getFetchConfig

    • maxTimeWanted

      private int maxTimeWanted(Collection<Ref> wants)
    • markReachable

      private void markReachable(Set<ObjectId> have, int maxTime) throws IOException
      Throws:
      IOException
    • parseReachable

      private void parseReachable(ObjectId id)
    • sendWants

      private boolean sendWants(Collection<Ref> want, PacketLineOut p) throws IOException
      Throws:
      IOException
    • getCapabilitiesV2

      private Set<String> getCapabilitiesV2(Set<String> advertisedCapabilities) throws TransportException
      Throws:
      TransportException
    • enableCapabilities

      private String enableCapabilities() throws TransportException
      Throws:
      TransportException
    • negotiate

      Throws:
      IOException
      BasePackFetchConnection.CancelledException
    • negotiateBegin

      private void negotiateBegin() throws IOException
      Throws:
      IOException
    • markRefsAdvertised

      private void markRefsAdvertised()
    • markAdvertised

      private void markAdvertised(AnyObjectId id)
    • markCommon

      private void markCommon(RevObject obj, PacketLineIn.AckNackResult anr, boolean useState) throws IOException
      Throws:
      IOException
    • receivePack

      private void receivePack(ProgressMonitor monitor, OutputStream outputStream) throws IOException
      Throws:
      IOException
    • onReceivePack

      protected void onReceivePack()
      Notification event delivered just before the pack is received from the network. This event can be used by RPC such as TransportHttp to disable its request magic and ensure the pack stream is read correctly.
      Since:
      2.0